JP7450241B2 - Information processing devices, information processing systems, and learning devices - Google Patents

Description

The present invention relates to a technique for suggesting a clothing item suitable for a customer from among a plurality of clothing items.

In stores that sell clothing items, many clothing items are displayed, and it is not easy for prospective purchasers to find the clothing item they are looking for.

For example, Japanese Patent Application Publication No. 2017-215667 (Patent Document 1) makes it possible to easily recommend recommended products to customers who visit the store based on the items the customer owns or the sales products the customer is looking at. In order to solve the problem of not having a product, we asked customers who visited the store to take photos of the items they were wearing and the products on sale displayed at the customer's location in the store. To disclose a configuration for proposing product information regarding a type of recommended product according to owner information of an article or the like shown in a photograph.

International Publication No. 2003/069526 (Patent Document 2) discloses a first database device that outputs data on fashion contents that match the physical characteristics when physical characteristics are input; A fashion advising system is disclosed that includes a second database device that outputs data on stores that provide fashion content based on data on fashion content.

Japanese Patent Publication No. 2001-502090 (Patent Document 3) relates to a method for fashion shopping by customers, and specifically discloses a method for helping customers select appropriate fashion items to purchase based on data about the customer. do.

JP2017-215667A International Publication No. 2003/069526 Special Publication No. 2001-502090

The configuration disclosed in Patent Document 1 is a product that is different from the type of product shown in the photo, for example, a product whose design elements such as color, shape, and pattern match the product, or whose owner information is a second owner. Recommended products in the case of customer information are, for example, products that are the same type as the product in the photo, and that have design elements such as color, shape, and pattern that match the product. It is not something that recommends products according to people's tastes.

The configuration disclosed in Patent Document 2 focuses on determining fashion content that suits the customer's physical characteristics when the customer's physical characteristics are input, and provides fashion that matches the customer's tastes. It's not something.

The configuration disclosed in Patent Document 3 acquires personal information including bust, hip, waist, arm length, height, and front-centered measurements to assist in selecting clothing items to purchase. , Personal information is required to suggest clothing items, and the configuration is not suitable for general-purpose operation.

An object of the present invention is to provide a technology that proposes a clothing item suitable for a customer from among a plurality of clothing items with higher accuracy based on feature quantities representing the customer's characteristics.

According to an aspect of the present invention, there is provided an information processing device that proposes a clothing item suitable for a customer from among a plurality of clothing items based on a feature quantity representing the customer's characteristics. The information processing device includes a camera for taking an image of the customer, a microphone for collecting voice, and a facial area representing the customer's face and a facial area representing the customer's body in the image obtained by taking the customer with the camera. an image feature extractor for extracting a first feature from the face region of the image and a second feature from the body region of the image; an audio feature extraction unit for extracting a third feature from a portion of the collected audio that corresponds to the customer's utterance; , a trained estimation model that outputs the probability that each of a plurality of clothing items is a clothing item that should be proposed as an estimation result, and a system that displays clothing items according to the customer based on the estimation results. and a display section. The estimation model is generated by a learning process using a training data set, and the training data set is based on images obtained by capturing images of other customers and voices uttered by the other customers. Contains multiple learning data labeled with clothing items purchased by customers.

Before the voice is collected by the microphone, the display section displays a list of categories indicating the classification of clothing items, and also displays a message prompting the customer to select one of the categories displayed on the list by voice. You may also do so.

The area specifying unit may specify a part representing clothing worn by the customer as the body area.

Each of the plurality of clothing items may belong to one of a plurality of predetermined categories. The information processing device may further include a voice analysis unit for identifying a category selected by the customer from among a plurality of categories based on the voice uttered by the customer. The display unit is configured to display clothing items that belong to the category specified by the voice analysis unit and clothing items that do not belong to the specified category in different display modes among the clothing items displayed based on the estimation results. You may also do so.

An information processing system according to another aspect of the present invention is an information processing device that inputs feature quantities representing characteristics of a customer into a trained estimation model to suggest clothing items suitable for the customer from among a plurality of clothing items. , and a learning device for generating an estimated model. The information processing device includes a camera for capturing an image of the customer, a microphone for collecting voice, and a facial area representing the customer's face and a facial area representing the customer's body in the input image obtained by capturing the customer using the camera. an image feature extraction unit for extracting a first feature amount from the face region of the input image and a second feature amount from the body region of the input image; , and an audio feature extracting unit for extracting a third feature amount from a portion of the audio collected by the microphone that corresponds to the customer's utterance. The estimation model receives input of the first feature, the second feature, and the third feature, and outputs the probability that each of the plurality of clothing items is the clothing item to be proposed as an estimation result. This is how it is learned. The information processing device further includes a display unit for displaying clothing items appropriate for the customer based on the estimation result. The learning device includes an acquisition unit for acquiring a learning data set. The training data set is training data in which clothing items purchased by the other customer are labeled based on the training image obtained by imaging the other customer and the training voice uttered by the other customer. Contains multiple. The learning device further includes an area identifying unit for identifying a face area representing the face of another customer and a body area representing the body of the other customer in the learning image, and an area identifying unit for identifying a face area representing the face of another customer and a body area representing the body of the other customer in the learning image; an image feature extraction unit for extracting a first learning feature and a second learning feature from the body region of the learning image; A voice feature extraction unit for extracting a third learning feature from speech, and a model for estimating the first learning feature, second learning feature, and third learning feature extracted from the learning data. and a learning unit for optimizing the estimation model so that the estimation result inputted and output from the learning data approaches the purchase history of the clothing item labeled in the learning data.

According to still another aspect of the present invention, learning is performed to generate an estimation model used to propose a clothing item suitable for the customer from among a plurality of clothing items upon receiving input of feature values representing the customer's characteristics. Equipment is provided. The learning device includes an acquisition unit for acquiring a learning data set. The learning data set includes a plurality of pieces of learning data in which clothing items purchased by the customer are labeled with respect to images obtained by imaging the customer and voices uttered by the customer. The learning device includes an area identifying unit for identifying a facial area representing a customer's face and a body area representing a customer's body in the image, and a region identifying unit that extracts a first feature amount from the facial area of the image. an image feature extractor for extracting a second feature from the body region of the customer; an audio feature extractor for extracting a third feature from a portion of the voice that corresponds to the customer's utterance; and learning data. The estimation result that is output by inputting the first feature amount, second feature amount, and third feature amount extracted from the estimation model to the estimation model approaches the purchase history of the clothing item labeled in the training data. and a learning unit for optimizing the estimation model.

According to still another aspect of the present invention, there is provided a trained estimation model that is used to propose a clothing item suitable for the customer from among a plurality of clothing items upon receiving input of feature quantities representing the customer's characteristics. Ru. The estimated model is generated by a learning process using a learning data set. The learning data set includes a plurality of pieces of learning data in which clothing items purchased by the customer are labeled with respect to images obtained by imaging the customer and voices uttered by the customer. The learning process includes the steps of identifying, for each of the learning data, a facial area representing the customer's face and a body area representing the customer's body in the image, and extracting a first feature amount from the facial area of the image. In addition, a step of extracting a second feature amount from the body region of the image, a step of extracting a third feature amount from a portion of the audio corresponding to the customer's utterance, and a step of extracting the first feature amount, the second feature amount, and The method includes a step of optimizing the estimation model so that the estimation result outputted by inputting the third feature amount into the estimation model approaches the purchase history of the clothing item labeled in the learning data.

According to still another aspect of the present invention, learning is used for learning an estimation model used to propose a clothing item suitable for the customer from among a plurality of clothing items upon receiving input of feature values representing the customer's characteristics. A method for collecting data is provided. The method of collecting data for training involves the steps of capturing an image of a customer and acquiring audio including the customer's utterances, and inputting multiple features extracted from the image and audio into a trained estimation model. generating a proposal for a clothing item according to the customer; generating identification information; and publishing a medium that encourages the purchase of the clothing item, including the generated proposal for the clothing item and the generated identification information. a step of associating the generated identification information with an image and a sound; a step of obtaining the identification information included in the medium and the clothing item purchased by the customer; and the step of associating the generated identification information with the image and audio; and the step of obtaining the identification information acquired from the medium and the clothing item purchased by the customer. and a step of associating the image and voice with the clothing item purchased by the customer using identification information as a key, and storing the association as learning data used for learning the estimation model.

According to the present invention, it is possible to suggest a clothing item suitable for a customer from among a plurality of clothing items with higher accuracy based on feature amounts representing the customer's characteristics.

FIG. 1 is a schematic diagram showing an example of the appearance of a store where the clothing suggestion system according to the present embodiment is installed. FIG. 3 is a diagram for explaining processing at a display terminal that constitutes the clothing suggestion system according to the present embodiment. FIG. 3 is a diagram for explaining processing at a display terminal that constitutes the clothing suggestion system according to the present embodiment. FIG. 2 is a diagram for explaining a store visitor who uses a coupon output from a display terminal that constitutes the clothing suggestion system according to the present embodiment. FIG. 3 is a diagram for explaining a learning data set generation process in the clothing proposal system according to the present embodiment. FIG. 1 is a schematic diagram showing an example of a system configuration of a clothing proposal system according to the present embodiment. FIG. 1 is a schematic diagram showing an example of the hardware configuration of a display terminal that constitutes the clothing suggestion system according to the present embodiment. FIG. 1 is a schematic diagram showing an example of the hardware configuration of a POS terminal that constitutes the clothing proposal system according to the present embodiment. FIG. 2 is a schematic diagram showing an example of the hardware configuration of a management device that constitutes the clothing proposal system according to the present embodiment. FIG. 1 is a schematic diagram showing an example of the functional configuration of a display terminal that constitutes the clothing suggestion system according to the present embodiment. FIG. 3 is a diagram for explaining processing contents in a proposed item estimation function of a display terminal that constitutes the clothing suggestion system according to the present embodiment. FIG. 12 is a diagram for explaining area identification processing by the area identification module shown in FIG. 11; FIG. 12 is a diagram for explaining the process of specifying a section by the section specifying module shown in FIG. 11; 12 is a schematic diagram showing an example of a network configuration of the estimation model shown in FIG. 11. FIG. FIG. 2 is a diagram for explaining processing contents in a display control function 150 and a coupon issuance control function of a display terminal that constitutes the clothing suggestion system according to the present embodiment. FIG. 7 is a diagram for explaining the processing contents in the image and audio storage function 170 of the display terminal that constitutes the clothing suggestion system according to the present embodiment. It is a flowchart which shows the processing procedure of the item estimation process in the display terminal which comprises the clothing suggestion system according to this Embodiment. FIG. 1 is a schematic diagram showing an example of the functional configuration of a POS terminal that constitutes the clothing proposal system according to the present embodiment. FIG. 6 is a diagram for explaining the processing contents in the sales information storage function 250 of the POS terminal that constitutes the clothing proposal system according to the present embodiment. It is a flowchart showing the processing procedure of sales management processing in the POS terminal that constitutes the clothing proposal system according to the present embodiment. FIG. 2 is a diagram for explaining an overview of a learning phase in the clothing suggestion system according to the present embodiment. FIG. 2 is a schematic diagram showing an example of the functional configuration of a management device that constitutes the clothing suggestion system according to the present embodiment. FIG. 7 is a diagram for explaining processing contents in a learning data set generation function 350 of the management device that constitutes the clothing suggestion system according to the present embodiment. FIG. 7 is a diagram for explaining processing contents in a learning function 360 of the management device that constitutes the clothing suggestion system according to the present embodiment. It is a flowchart which shows the processing procedure of the learning process in the management device which constitutes the clothing suggestion system according to this embodiment. FIG. 2 is a schematic diagram showing an example of a system configuration of a clothing proposal system according to a first modification of the present embodiment. FIG. 7 is a diagram for explaining an item suggestion screen displayed on a display terminal of the clothing suggestion system according to a second modification of the present embodiment. It is a figure for explaining the processing content in the display control function and coupon issuing control function of the display terminal which constitutes the clothing proposal system according to the modification 2 of this embodiment. FIG. 12 is a diagram for explaining processing contents in a suggested item estimation function of a display terminal that constitutes a clothing suggestion system according to a third modification of the present embodiment. FIG. 12 is a diagram for explaining processing contents in a proposed item estimation function of a display terminal that constitutes a clothing suggestion system according to a fourth modification of the present embodiment. It is a schematic diagram which shows the example of use of the clothing suggestion system according to the modification 5 of this Embodiment. It is a schematic diagram which shows the example of implementation of the clothing suggestion system according to the modification 5 of this Embodiment.

Embodiments of the present invention will be described in detail with reference to the drawings. Note that the same or corresponding parts in the figures are designated by the same reference numerals, and the description thereof will not be repeated.

<A. Overview of the clothing suggestion system>
First, as a typical example of an information processing system according to the present invention, an outline of a clothing proposal system 1 according to the present embodiment will be explained.

As used herein, "clothing" refers to all clothing (clothing) and accessories (accessories) worn by people. "Clothing item" is a term indicating any product included in clothing. For ease of explanation, "clothing items" may be simply referred to as "items."

In this specification, the term "customer" refers to a general user who has some intention of purchasing clothing items. In the following explanation, customers who visit the store are also referred to as "visitors." Further, a customer who uses the system according to this embodiment via a mobile terminal is also referred to as a "net user."

FIG. 1 is a schematic diagram showing an example of the appearance of a store in which a clothing proposal system 1 according to the present embodiment is installed. 2 and 3 are diagrams for explaining processing at display terminal 100 that constitutes clothing proposal system 1 according to the present embodiment.

As shown in FIG. 1, it is assumed that a customer (hereinafter also referred to as "visitor 40") enters the store 30. A display terminal 100, which is an example of an information processing device, is arranged near the entrance of the store 30. Display terminal 100 includes a relatively large display 102, a human sensor 128, a camera 130, and a microphone 132 arranged near the display 102. A printer 120 is arranged below the display 102.

When the store visitor 40 approaches the display terminal 100 (FIG. 2(a)), the human sensor 128 detects the approach, and the category selection reception screen 50 is displayed on the display 102 (FIG. 2(b)). . In this state, the store visitor 40 is imaged by the camera 130 of the display terminal 100. That is, the display terminal 100 acquires an image showing the store visitor 40 (hereinafter also referred to as "captured image 136").

The category selection reception screen 50 displays a list of one or more categories. At the same time, a message "Please select a category by voice" is displayed to encourage the visitor 40 to speak.

Thereafter, the microphone 132 of the display terminal 100 starts collecting voices, and when the visitor 40 utters a voice indicating the desired category (in the example shown in FIG. 2, "jacket") (FIG. 2(c)), the display An item suggestion screen 52 is output to 102 (FIG. 3(a)). At this time, the display terminal 100 acquires the voice uttered by the store visitor 40 (hereinafter also referred to as "collected voice 138").

In this way, before the microphone 132 collects the audio, the display 102 displays a list of categories indicating the classification of clothing items, and also prompts the store visitor to select one of the categories displayed on the list by voice. A message prompting the person 40 is displayed.

The item suggestion screen 52 includes a list display 54 of clothing items that are estimated to be "recommended" according to the preferences of the store visitor 40. The items displayed in a list on the item suggestion screen 52 are determined based on estimation results obtained by executing item estimation processing using a learned model as described below. In this way, the display terminal 100, which is an example of an information processing device, selects a customer from among a plurality of clothing items based on the feature values representing the customer's characteristics (typically, the captured image 136 and the collected audio 138). Suggest appropriate clothing items.

The item suggestion screen 52 further includes a coupon issue button 56. In response to pressing the coupon issue button 56, the coupon 10 is output from the printer 120.

In addition to the discount amount display 12, the coupon 10 output from the printer 120 also displays a list display 14 corresponding to the list display 54 included in the item suggestion screen 52, and a list display 14 corresponding to the list display 54 included in the item suggestion screen 52, and a list display 14 indicating where each item included in the list display 14 is located in the store. and a map 16 indicating the location (FIG. 3(b)).

Further, the coupon 10 includes an identification image 18 such as a QR code (registered trademark) indicating a coupon ID, which will be described later. By using the coupon ID indicated by the identification image 18, a learning data set used for learning the estimation model is generated.

FIG. 4 is a diagram for explaining a store visitor 40 who uses the coupon 10 output from the display terminal 100 that constitutes the clothing suggestion system 1 according to the present embodiment. The visitor 40 can enjoy shopping while referring to the content printed on the coupon 10 (FIG. 4(a)). Since the discount is applied by presenting the coupon 10, the visitor 40 usually presents the coupon 10 output from the display terminal 100 at the time of checkout (FIG. 4(b)).

FIG. 5 is a diagram for explaining the training data set generation process in the clothing proposal system 1 according to the present embodiment. Referring to FIG. 5, the captured image 136 and collected audio 138 acquired by the display terminal 100 and the information on the purchased item (hereinafter also referred to as "sales information 218") are the coupon 10 (more precisely, , coupon ID 166) as an intermediary. In this way, the associated captured image 136, collected audio 138, and sales information 218 are used as a learning data set for learning the estimation model.

In this way, the clothing suggestion system 1 according to the present embodiment not only proposes items based on the preferences of the visitor 40 when entering the store, but also proposes items based on the preferences of the visitor 40 using information on items actually purchased by the visitor 40. Estimation models for making suggestions can be learned.

<B. Hardware configuration example of clothing proposal system>
Next, an example of the system configuration of the clothing proposal system 1 according to the present embodiment will be described. First, an example of the overall configuration of the clothing proposal system 1 will be described, and then an example of the hardware configuration of the main devices included in the clothing proposal system 1 will be described.

(b1: System configuration example)
FIG. 6 is a schematic diagram showing an example of the system configuration of the clothing proposal system 1 according to the present embodiment. Referring to FIG. 6, clothing proposal system 1 includes one or more display terminals 100, one or more POS terminals 200, and management device 300, which are connected via local network 2.

The display terminal 100 is typically placed near the entrance of the store 30 and suggests clothing according to the tastes of the store visitor. More specifically, the display terminal 100 acquires an image by capturing an image of a visitor, and also collects the sounds uttered by the visitor. The display terminal 100 inputs images (hereinafter also referred to as "captured images") and sounds (hereinafter also referred to as "collected sounds") of the store visitor into a trained model, thereby adjusting the preferences of the store visitor. The degree of suitability (hereinafter also referred to as "score") is calculated for each item being sold. The display terminal 100 suggests items with high scores to the store visitor. The display terminal 100 can also issue a coupon on which the proposed item is printed to the visitor.

The display terminal 100 can also transmit captured images and collected audio to the management device 300 upon request.

The POS terminal 200 performs checkout processing for items that a visitor wishes to purchase. The POS terminal 200 can generate information on purchased items (sales information) and also transmit it to the management device 300 in response to a request.

The management device 300 is in charge of managing and updating learned models used by the display terminal 100. More specifically, the management device 300 acquires captured images and collected audio from the display terminal 100, and also acquires sales information from the POS terminal 200. Then, the management device 300 generates a learning data set from the acquired captured images and collected sounds, and the acquired sales information. The management device 300 executes learning of the learned model (which may include both new learning and additional learning) using the generated learning data set.

The learned model generated or updated by the management device 300 is transmitted to the display terminal 100.

(b2: display terminal 100)
FIG. 7 is a schematic diagram showing an example of the hardware configuration of the display terminal 100 that constitutes the clothing proposal system 1 according to the present embodiment. Display terminal 100 may be realized using a general-purpose computer.

Referring to FIG. 7, the display terminal 100 includes a display 102, a processor 104, a memory 106, a network controller 108, a storage 110, a printer 120, an optical drive 122, and a touch panel as main hardware elements. It includes a detection unit 126, a human sensor 128, a camera 130, and a microphone 132.

The display 102 outputs a category selection reception screen 50, an item proposal screen 52, and the like. The display 102 is configured with, for example, an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence) display.

The processor 104 is an arithmetic main body that executes processes necessary for realizing the display terminal 100 by executing various programs as described below.The processor 104 includes, for example, one or more CPUs (Central Processing Unit) It consists of a GPU (Graphics Processing Unit), etc. A CPU or GPU having multiple cores may also be used.

The memory 106 provides a storage area for temporarily storing program codes, work memory, etc. when the processor 104 executes a program. As the memory 106, for example, a volatile memory device such as a DRAM (Dynamic Random Access Memory) or an SRAM (Static Random Access Memory) may be used.

The network controller 108 transmits and receives data to and from arbitrary information processing devices including the management device 300 via the local network 2. The network controller 108 may be compatible with any communication method such as, for example, Ethernet (registered trademark), wireless LAN (Local Area Network), or Bluetooth (registered trademark).

The storage 110 stores an OS (Operating System) 112 executed by the processor 104, an application program 114 for realizing a functional configuration as described below, a learned model 116, and an item image for generating the item suggestion screen 52. 118 etc. are stored. As the storage 110, for example, a nonvolatile memory device such as a hard disk or a solid state drive (SSD) may be used. Further, the storage 110 may store captured images obtained by capturing a visitor to the store and collected sounds that are sounds uttered by the visitor.

A part of the library or function module required when the application program 114 is executed by the processor 104 may be a library or function module provided as a standard by the OS 112. In this case, although the application program 114 alone does not include all of the program modules necessary to implement the corresponding functions, by being installed under the execution environment of the OS 112, the functions described below can be implemented. The configuration can be realized. Therefore, even a program that does not include such a part of the library or functional module may be included in the technical scope of the present invention.

The printer 120 issues a coupon on which the proposed item is printed to the visitor. As the printer 120, any printing method such as an electrophotographic method, an inkjet method, or a thermal paper method can be adopted.

The optical drive 122 reads information such as programs stored on an optical disc 124 such as a CD-ROM (Compact Disc Read Only Memory) or a DVD (Digital Versatile Disc). The optical disc 124 is an example of a non-transitory recording medium, and is distributed in a state in which an arbitrary program is stored in a non-volatile manner. Display terminal 100 according to this embodiment can be configured by optical drive 122 reading a program from optical disk 124 and installing it in storage 110. Therefore, the subject matter of the present invention may be the program itself installed in the storage 110 or the like, or a recording medium such as the optical disk 124 that stores a program for realizing the functions and processing according to the present embodiment.

FIG. 7 shows an optical recording medium such as an optical disk 124 as an example of a non-transitory recording medium, but the present invention is not limited to this, and the present invention is not limited to semiconductor recording media such as flash memory, magnetic recording media such as hard disks, or storage tapes. , a magneto-optical recording medium such as MO (Magneto-Optical disk) may be used.

Alternatively, the program for realizing the display terminal 100 may not only be stored and distributed in any recording medium as described above, but also may be distributed by being downloaded from a server device or the like via the Internet or an intranet. .

The touch detection unit 126 is arranged in association with the display 102 and detects an input operation to the display 102. As the touch detection unit 126, any detection method such as a capacitance method, a resistive film method, or an ultrasonic surface acoustic wave method can be adopted.

The human sensor 128 detects the approach of a visitor to the display terminal 100 using infrared rays or the like.

The camera 130 is a device that captures an image of the store visitor, and is arranged near the display area of the display 102 and is configured to include the store visitor directly facing the display 102 in its field of view. The camera 130 may continuously take images of the field of view at a predetermined period, or may take images in response to a command issued from the processor 104 or the like.

The microphone 132 is a device for collecting sounds, and is placed near the display area of the display 102, where it can collect sounds emitted by visitors. The microphone 132 preferably collects only the voices of visitors directly facing the display 102, and therefore preferably has sharp directivity.

FIG. 7 shows a configuration example in which the display terminal 100 is realized by a general-purpose computer (processor 104) executing the application program 114. It may also be implemented using hard-wired circuits such as integrated circuits. For example, it may be realized using an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or the like.

(b3: POS terminal 200)
FIG. 8 is a schematic diagram showing an example of the hardware configuration of the POS terminal 200 that constitutes the clothing proposal system 1 according to the present embodiment. Referring to FIG. 8, a POS terminal 200 includes a display 202, a processor 204, a memory 206, a network controller 208, a storage 210, a printer 220, an optical drive 222, and a touch panel as main hardware elements. It includes a detection section 226, an optical reader 228, an input section 230, and a payment processing section 232.

The display 202 displays information necessary for accounting for items. The display 202 is composed of, for example, an LCD or an organic EL display.

The processor 204 is a computation main body that executes processes necessary for realizing the POS terminal 200 by executing various programs as described below.The processor 204 is composed of, for example, one or more CPUs. . A CPU having multiple cores may also be used.

The memory 206 provides a storage area for temporarily storing program codes, work memory, etc. when the processor 204 executes a program. As the memory 206, for example, a volatile memory device such as DRAM or SRAM may be used.

The network controller 208 transmits and receives data to and from any information processing device including the management device 300 via the local network 2 . The network controller 208 may be compatible with any communication method such as, for example, Ethernet, wireless LAN, or Bluetooth.

The storage 210 stores an OS 212 executed by the processor 204, an application program 214 for realizing a functional configuration as described below, item information 216 including price and attribute information of each item necessary for accounting processing, and information on purchased items. Stores sales information 218, which is information on items purchased. As the storage 210, for example, a nonvolatile memory device such as a hard disk or SSD may be used.

Some of the libraries and function modules required when the application program 214 is executed by the processor 204 may be provided as standard by the OS 212. In this case, although the application program 214 alone does not include all the program modules necessary to implement the corresponding functions, by being installed under the execution environment of the OS 212, the functions described below can be implemented. The configuration can be realized. Therefore, even a program that does not include such a part of the library or functional module may be included in the technical scope of the present invention.

The printer 220 issues a printed receipt containing the results of the transaction. As the printer 220, any printing method such as an electrophotographic method, an inkjet method, or a thermal paper method can be adopted.

The optical drive 222 reads information such as programs stored on an optical disk 224 such as a CD-ROM or DVD. The optical disc 224 is an example of a non-transitory recording medium, and is distributed in a state in which an arbitrary program is stored in a non-volatile manner. The POS terminal 200 according to this embodiment can be configured by the optical drive 222 reading the program from the optical disk 224 and installing it in the storage 210. Therefore, the subject matter of the present invention may be the program itself installed in the storage 210 or the like, or a recording medium such as the optical disk 224 storing the program for realizing the functions and processing according to the present embodiment.

Although FIG. 8 shows an optical recording medium such as an optical disk 224 as an example of a non-transitory recording medium, the present invention is not limited to this, and the present invention is not limited to semiconductor recording media such as flash memory, magnetic recording media such as hard disks, or storage tapes. , MO, or other magneto-optical recording media may be used.

Alternatively, the program for implementing the POS terminal 200 may not only be stored and distributed in any recording medium as described above, but also may be distributed by being downloaded from a server device or the like via the Internet or an intranet. .

The touch detection unit 226 is arranged in association with the display 202 and detects an input operation to the display 202. As the touch detection unit 226, any detection method such as a capacitance method, a resistive film method, or an ultrasonic surface acoustic wave method can be adopted.

The optical reader 228 optically reads information on item tags attached to items, QR codes included in coupons, and the like. As the optical reader 228, any detection method such as a laser scanning method or an image sensing method can be adopted.

The input unit 230 accepts input operations such as amount and type. As the input unit 230, for example, a register key, a keyboard, a mouse, a touch panel, a pen, etc. may be used.

The payment processing unit 232 includes a mechanism necessary for cash payment and a mechanism necessary for electronic payment such as a credit card. More specifically, regarding cash payments, the payment processing unit 232 includes a cash storage unit for storing banknotes and coins, a sales management unit for managing sales amounts, and the like. The payment processing unit 232 includes a mechanism for reading information stored in a credit card and exchanging payment information with a payment center and the like regarding electronic payment.

FIG. 8 shows a configuration example in which a general-purpose computer (processor 204) executes an application program 214 to realize a POS terminal 200. It may also be implemented using hardwired circuits such as integrated circuits. For example, it may be realized using ASIC, FPGA, or the like.

(b4: management device 300)
FIG. 9 is a schematic diagram showing an example of the hardware configuration of the management device 300 that constitutes the clothing proposal system 1 according to the present embodiment. Referring to FIG. 9, management device 300 includes a display 302, a processor 304, a memory 306, a network controller 308, a storage 310, and an input unit 330 as main hardware elements.

The display 302 displays information necessary for processing by the management device 300. The display 302 is composed of, for example, an LCD or an organic EL display.

The processor 304 is a calculation main body that executes processes necessary for realizing the management device 300 by executing various programs as described below.The processor 304 is composed of, for example, one or more CPUs or GPUs. be done. A CPU or GPU having multiple cores may also be used. In the management device 300, it is preferable to employ a GPU or the like suitable for learning processing for generating a trained model.

The memory 306 provides a storage area for temporarily storing program codes, work memory, etc. when the processor 304 executes a program. As the memory 306, for example, a volatile memory device such as DRAM or SRAM may be used.

The network controller 308 transmits and receives data to and from arbitrary information processing devices including the display terminal 100 and the POS terminal 200 via the local network 2 . The network controller 308 may be compatible with any communication method such as, for example, Ethernet, wireless LAN, or Bluetooth.

The storage 310 includes an OS 312 executed by the processor 304, an application program 314 for realizing a functional configuration as described below, and preprocessing for generating a learning data set 324 from image/audio information 320 and sales information 322. A program 316, a learning program 318 for generating a trained model 326 using a learning data set 324, and the like are stored.

The image/audio information 320 consists of a captured image 136 and collected audio 138 acquired from the display terminal 100. Sales information 322 consists of sales information 218 acquired from POS terminal 200. The acquisition process of image/audio information 320 and sales information 322 will be described in detail later.

The learning data set 324 is a training data set in which sales information 322 is added to image/audio information 320 as a label (or tag). The learned model 326 is an estimated model obtained by performing a learning process using the learning data set 324.

As the storage 310, for example, a nonvolatile memory device such as a hard disk or SSD may be used.

Some of the libraries and function modules required when the application program 314, preprocessing program 316, and learning program 318 are executed by the processor 304 may be made to use libraries or function modules provided as standard by the OS 312. . In this case, each of the application program 314, preprocessing program 316, and learning program 318 alone does not include all the program modules necessary to realize the corresponding functions, but under the execution environment of the OS 312, By installing this software, it is possible to realize the functional configuration described below. Therefore, even a program that does not include such a part of the library or functional module may be included in the technical scope of the present invention.

The application program 314, the preprocessing program 316, and the learning program 318 are stored in an optical recording medium such as an optical disk, a semiconductor recording medium such as a flash memory, a magnetic recording medium such as a hard disk or a storage tape, and a magneto-optical recording medium such as an MO. It may be stored and distributed in a non-transitory recording medium and installed in the storage 310. Therefore, the subject matter of the present invention may be the program itself installed in the storage 310 or the like, or a recording medium storing a program for realizing the functions and processing according to the present embodiment.

Alternatively, the program for realizing the management device 300 may not only be stored and distributed in any recording medium as described above, but also may be distributed by being downloaded from a server device or the like via the Internet or an intranet. .

Input unit 330 accepts various input operations. As the input unit 330, for example, a keyboard, a mouse, a touch panel, a pen, etc. may be used.

FIG. 9 shows a configuration example in which a general-purpose computer (processor 304) executes an application program 314, a preprocessing program 316, and a learning program 318 to realize the management device 300. All or part of the required functionality may be implemented using hardwired circuits such as integrated circuits. For example, it may be realized using ASIC, FPGA, or the like.

(b5: Integrated configuration/cloud configuration)
As a typical example, FIGS. 6 to 9 illustrate a configuration in which each of the display terminal 100, POS terminal 200, and management device 300 has a processor to realize the functions it is responsible for, but the configuration is not limited to this. , an integrated configuration may be adopted that implements the functions necessary to implement the clothing proposal system 1 with fewer calculations.

As an example of such an integrated configuration, the functions handled by the display terminal 100 and the POS terminal 200 are realized in the management device 300, and the display terminal 100 and the POS terminal 200 have only a user interface like a thin client. may be provided.

Furthermore, the management device 300 may also be realized by a plurality of computers connected via a computer network explicitly or implicitly working together. When multiple computers work together, some of the computers may be unspecified computers on a network, called cloud computers.

Those skilled in the art will be able to realize the clothing suggestion system 1 according to the present embodiment by appropriately using techniques appropriate to the era in which the present invention is implemented.

<C. Functions and processing of display terminal 100>
Next, the functions and processing of the display terminal 100 that constitutes the clothing proposal system 1 according to the present embodiment will be explained. In the clothing proposal system 1, the display terminal 100 is in charge of the operation phase of proposing clothing using a trained model (estimated model), and is also in charge of a part of the learning phase for constructing the trained model. Become.

(c1: Functional configuration of display terminal 100)
FIG. 10 is a schematic diagram showing an example of the functional configuration of the display terminal 100 that constitutes the clothing proposal system 1 according to the present embodiment. Each function shown in FIG. 10 may typically be realized by the processor 104 of the display terminal 100 executing the OS 112 and the application program 114 (both shown in FIG. 7).

Referring to FIG. 10, display terminal 100 has a suggested item estimation function 140, a display control function 150, a coupon issuance control function 160, and an image/audio storage function 170 as functional configurations.

The suggested item estimation function 140 receives as input a captured image 136 obtained by capturing an image of a store visitor using the camera 130, and a collected voice 138 obtained by collecting the voice uttered by the store visitor using the microphone 132. By inputting it to the learned model 116, the estimation result is output.

The display control function 150 receives the estimation results from the suggested item estimation function 140 and generates a screen that proposes clothing according to the preferences of the store visitor.

The coupon issue control function 160 receives information about the item suggested to the visitor by the display control function 150, generates a coupon ID 166, and issues a coupon 10 on which the proposed item and the coupon ID 166 are printed.

The image and audio storage function 170 assigns a coupon ID 166 generated by the coupon issuance control function 160 to the captured image 136 and collected audio 138 received as input by the suggested item estimation function 140 and stores them. The captured image 136 and collected audio 138 (to which a coupon ID 166 is assigned) saved by the image/audio storage function 170 are sent to the management device 300 and undergo learning processing to generate a trained model, as described later. used for.

(c2: Suggested item estimation function 140)
Next, details of the suggested item estimation function 140 of the display terminal 100 shown in FIG. 10 will be described.

FIG. 11 is a diagram for explaining the processing content in the suggested item estimation function 140 of the display terminal 100 that constitutes the clothing suggestion system 1 according to the present embodiment. Referring to FIG. 11, display terminal 100 includes a region identification module 141, size adjustment modules 142, 143, section identification module 144, and resampling module 145 as suggested item estimation function 140.

The area specifying module 141 analyzes the subject (visitor) included in the captured image 136 and specifies the face area and body area. That is, the area specifying module 141 specifies a facial area representing the customer's face and a body area representing the customer's body in an image obtained by capturing an image of the customer with the camera 130. The region identification module 141 extracts and outputs a face region partial image 147 and a body region partial image 148 corresponding to the identified face region and body region from the captured image 136.

Typically, the region identification module 141 identifies facial regions and body regions by extracting facial features such as eyes and nose, as well as skeletal features such as limbs. At this time, the area specifying module 141 may specify a part representing the clothing worn by the customer as the body area.

FIG. 12 is a diagram for explaining area identification processing by the area identification module 141 shown in FIG. 11. Referring to FIG. 12, area identification module 141 extracts an area including the face of the store visitor as facial area partial image 147, and extracts an area below the visitor's face as body area partial image 148.

The face area partial image 147 is considered to include attribute information such as the visitor's gender and age, and the body area partial image 148 contains information regarding the visitor's current clothing (that is, information indicating trends in clothing preferences). It is considered to include.

Referring again to FIG. 11, the facial area partial image 147 extracted from the captured image 136 by the area identification module 141 is output to the size adjustment module 142. Similarly, a body region partial image 148 extracted from the captured image 136 by the region identification module 141 is output to the size adjustment module 143.

In the size adjustment modules 142 and 143, the face region partial image 147 and the body region partial image 148 are converted into feature amounts (feature amount vectors) having predetermined dimensions and provided to the estimation model 1400. Here, since the image sizes of the face area partial image 147 and the body area partial image 148 extracted by the area identification module 141 may vary, the size adjustment modules 142 and 143 standardize the image sizes.

More specifically, the size adjustment module 142 adjusts the facial area partial image 147 from the area identification module 141 into an image with a predetermined number of pixels, and then adjusts the pixel value of each pixel constituting the adjusted image. is input to the estimation model 1400 as the face region feature amount 1410.

Similarly, the size adjustment module 143 adjusts the body region partial image 148 from the region identification module 141 into an image with a predetermined number of pixels, and then adjusts the pixel value of each pixel constituting the adjusted image to the body region partial image 148. It is input to the estimation model 1400 as a feature quantity 1420.

In this way, the size adjustment modules 142 and 143 extract the facial area feature amount 1410 (first feature amount) from the facial area partial image 147 (the facial area of the image), and also extract the facial area feature amount 1410 (first feature amount) from the facial area partial image 148 (the facial area of the image). ) is extracted from the body region feature amount 1420 (second feature amount).

The section identifying module 144 identifies the section of the voice uttered by the store visitor included in the collected voice 138, extracts and outputs the specific section voice 149. Typically, the section identifying module 144 analyzes temporal changes in the voice indicated by the collected voice 138 and identifies a section in which the amplitude or frequency has changed with respect to noise components surrounding the display terminal 100. In this way, the specific section audio 149 is extracted.

FIG. 13 is a diagram for explaining the process of specifying a section by the section specifying module 144 shown in FIG. 11. Referring to FIG. 13, the section identification module 144 identifies, among the temporal changes in the voice indicated by the collected speech 138, a section that shows a significant change with respect to the temporal change before and after, as an utterance section by the store visitor. Then, it is extracted as specific section audio 149.

The specific section audio 149 is the audio in which the visitor utters the desired category, and therefore includes information for specifying the desired category. Furthermore, the specific section audio 149 is considered to include information indicating the current feeling (mood) of the visitor.

In this manner, the section identification module 144 and the resampling module 145 extract the audio feature amount 1430 (third feature amount) from the portion of the audio collected by the microphone 132 that corresponds to the customer's utterance.

Referring again to FIG. 11, specific section audio 149 extracted from collected audio 138 by resampling module 145 is output to resampling module 145. In the resampling module 145, the specific section audio 149 is converted into a feature amount (feature amount vector) having a predetermined dimension, and is provided to the estimation model 1400. Here, since the audio time length of the specific section audio 149 specified by the section specifying module 144 may vary, the resampling module 145 normalizes the number of audio samples.

More specifically, the resampling module 145 samples the time waveform of the audio indicated by the specific section audio 149 from the section identification module 144 with a predetermined number of samples, and calculates the amplitude value at each sampling point into the audio. It is input to the estimation model 1400 as a feature quantity 1430.

Estimated model 1400 is constructed based on trained model 116 that defines the network structure and corresponding parameters. By inputting the face region feature amount 1410, the body region feature amount 1420, and the audio feature amount 1430 to the estimation model 1400, the calculation process defined by the estimation model 1400 is executed, and a score for each item is calculated as the estimation result 1450. be done. Here, the score for each item is a value indicating the probability that each clothing item is a clothing item that should be proposed.

The estimated model 1400 is generated by a learning process using a learning data set as described below. As will be described later, the training data set is a training data set in which clothing items purchased by the other customer are labeled based on images obtained by imaging the other customer and audio uttered by the other customer. Contains multiple data.

In this way, the estimated model 1400, which is a trained estimation model, includes a face area feature 1410 (first feature), a body area feature 1420 (second feature), and a voice feature 1430 (third feature). In response to the input, the probability (score) of each of the plurality of clothing items being the clothing item to be proposed is output as an estimation result 1450.

(c3: Estimated model 1400)
FIG. 14 is a schematic diagram showing an example of the network configuration of the estimation model 1400 shown in FIG. 11. Referring to FIG. 14, estimation model 1400 is a network classified as a DNN (Deep Neural Network). Estimation model 1400 includes preprocessing networks 1460, 1470, and 1480 classified as CNNs (Convolutional Neural Networks), an intermediate layer 1490, an activation function 1492 corresponding to an output layer, and a Softmax function 1494.

The preprocessing networks 1460, 1470, and 1480 extract effective feature amounts for calculating the estimation result 1450 from the face region feature amount 1410, the body region feature amount 1420, and the voice feature amount 1430, which have relatively large degrees. It is planned to function as a kind of filter. Each of the preprocessing networks 1460, 1470, and 1480 has a configuration in which convolution layers (CONV) and pooling layers (Pooling) are arranged alternately. Note that the number of convolutional layers and pooling layers may not be the same, and an activation function such as ReLU (rectified linear unit) is placed on the output side of the convolutional layer.

More specifically _{, the preprocessing network 1460 receives input of facial area feature amounts 1410 (x 11 ,} Constructed to output features. The preprocessing network 1470 receives the input of the body region feature amounts 1420 (x ₂₁ , x ₂₂ , . . . , x _2s ) and generates information regarding the visitor's current clothing (that is, information indicating clothing preferences). It is constructed to output internal features that indicate . The preprocessing network 1480 receives the input of the audio features 1430 (x ₃₁ , x ₃₂ , . . . , x _3t ) and inputs information for specifying the category and the current feeling of the store visitor. It is constructed to output internal features that indicate information.

The intermediate layer 1490 consists of a fully connected network having a predetermined number of layers, and sequentially combines the outputs from each of the preprocessing networks 1460, 1470, and 1480 node by node using weights and biases determined for each node. do.

An activation function 1492 such as ReLU is placed on the output side of the intermediate layer 1490, and finally, after being normalized to a probability distribution by a Softmax function 1494, the estimation result 1450 (y ₁ , y ₂ , · ..., y _N ) is output.

In the learning phase as described below, the parameters of each element that constructs the network of estimation model 1400 are optimized.

(c4: Display control function 150 and coupon issue control function 160)
Next, details of the display control function 150 and the coupon issuance control function 160 of the display terminal 100 shown in FIG. 10 will be explained.

FIG. 15 is a diagram for explaining processing contents in the display control function 150 and the coupon issuing control function 160 of the display terminal 100 that constitute the clothing proposal system 1 according to the present embodiment. Referring to FIG. 15, display terminal 100 includes a display control module 152 as display control function 150.

The display control module 152 receives the estimation result 1450 calculated by the suggested item estimation function 140 and generates the item proposal screen 52 using the item image 118 corresponding to the item having the highest score in the estimation result 1450. The display control module 152 outputs the generated item suggestion screen 52 to the display 102. That is, the display 102 displays clothing items appropriate for the customer based on the estimation result 1450.

The item image 118 includes an image of each item associated with item identification information. The display control module 152 extracts necessary images from among the images included in the item image 118 based on the estimation result 1450.

The display terminal 100 also includes a coupon issue control module 162 and a coupon ID generation module 164 as the coupon issue control function 160. The coupon issue control module 162 receives the proposed item from the display control module 152 and the coupon ID 166 from the coupon ID generation module 164, and issues the coupon 10 on which these pieces of information are printed from the printer.

Coupon ID generation module 164 generates coupon ID 166, which is unique identification information, using any method. Note that the coupon ID 166 may be printed on the coupon 10 in the form of a QR code or the like. In this case, the coupon ID generation module 164 may randomly generate a predetermined number of character strings, and may also generate a QR code corresponding to the generated character strings. As will be described later, the coupon ID 166 is used as a key for generating the learning data set 324.

(c5: Image and audio storage function 170)
Next, details of the image and audio storage function 170 of the display terminal 100 shown in FIG. 10 will be explained.

FIG. 16 is a diagram for explaining processing contents in the image and audio storage function 170 of the display terminal 100 that constitutes the clothing proposal system 1 according to the present embodiment. Referring to FIG. 16, display terminal 100 includes an association module 172 and an image/audio storage section 174 as an image/audio storage function 170.

In response to the coupon issuing control module 162 (see FIG. 15) issuing the coupon 10, the association module 172 receives the coupon ID 166 given to the issued coupon 10, and uses the coupon ID 166 to issue the coupon 10. The received coupon ID 166 is associated with the captured image 136 and collected audio 138. The association module 172 stores the associated coupon ID 166, the captured image 136, and the collected audio 138 together in the image/audio storage unit 174.

The image and audio storage unit 174 is realized using at least a part of the storage area provided by the memory 106 or the storage 110 (both shown in FIG. 7). The image/audio storage unit 174 stores data in units of data sets including the coupon ID 166, the captured image 136, and the collected audio 138.

(c6: Processing procedure)
Next, the item estimation process executed in the display terminal 100 that constitutes the clothing proposal system 1 will be described.

FIG. 17 is a flowchart showing the processing procedure of the item estimation process in the display terminal 100 that constitutes the clothing suggestion system 1 according to the present embodiment. Each step shown in FIG. 17 may typically be realized by the processor 104 of the display terminal 100 executing the OS 112 and the application program 114 (both shown in FIG. 7).

Referring to FIG. 17, first, display terminal 100 determines whether a visitor is detected (step S100). In step S100, it is typically determined whether a visitor is present based on the detection result from the human sensor 128 (see FIG. 7). If no visitor is detected (NO in step S100), the process of step S100 is repeated.

When a visitor is detected (YES in step S100), display terminal 100 displays a category selection reception screen (see FIG. 2) on display 102 (step S102).

Subsequently, the display terminal 100 acquires a captured image 136 by capturing an image of the visitor directly facing the display terminal 100 using the camera 130 (step S104). At the same time, the display terminal 100 starts collecting audio (step S106). Then, the display terminal 100 determines whether or not the visitor's utterance is detected based on the collected audio (step S108). In step S108, as shown in FIG. 13, it is determined whether a section (utterance section) showing a significant change with respect to the temporal change in the voice has appeared. If the visitor's utterance is not detected (NO in step S108), the processes of steps S106 and S108 are repeated. In this way, the display terminal 100 executes the process of acquiring the captured image 136 obtained by capturing the customer and the collected audio 138 including the customer's utterances.

When the visitor's utterance is detected (YES in step S108), the display terminal 100 extracts the face area partial image 147 and the body area partial image 148 from the captured image 136 (step S110). Then, the display terminal 100 adjusts the size of the extracted face area partial image 147 and body area partial image 148 to extract the face area feature amount 1410 and the body area feature amount 1420 (step S112).

In addition, the display terminal 100 extracts the speech section included in the collected speech 138 as the specific section speech 149 (step S114). Then, the display terminal 100 resamples the extracted specific section audio 149 and extracts the audio feature amount 1430 (step S116).

The display terminal 100 inputs the face area feature amount 1410 and the body area feature amount 1420 generated in step S112, and the audio feature amount 1430 generated in step S116 to the estimation model 1400 to generate an estimation result 1450 (step S118 ).

In this way, the display terminal 100 inputs a plurality of feature quantities extracted from the captured image 136 and the collected audio 138 into the trained estimation model 1400, and executes the process of generating a proposal for clothing items according to the customer. do.

The display terminal 100 generates and outputs the item suggestion screen 52 based on the item with the highest score in the estimation result 1450 generated in step S118 (step S120).

The display terminal 100 determines whether the coupon issue button has been pressed (step S122). If the coupon issue button is not pressed (NO in step S122), the process of step S122 is repeated.

When the coupon issue button is pressed (YES in step S122), the display terminal 100 generates the coupon ID 166 and issues the coupon 10 on which the suggested item and the coupon ID 166 are printed (step S124). That is, the display terminal 100 executes the process of generating the coupon ID 166, which is the identification information, and also executes the process of issuing the coupon 10, which is the medium. The coupon 10 includes the generated clothing item suggestion and the generated coupon ID 166 (identification information), and displays the details of the discount to encourage the purchase of the clothing item.

Finally, the display terminal 100 stores the captured image 136 and the collected audio 138 in association with the coupon ID 166 (step S126). That is, the display terminal 100 executes a process of associating the generated coupon ID 166 (identification information) with the captured image 136 and the collected audio 138.

With the above, the item estimation process for one store visitor is completed.
(c7: summary)
The display terminal 100 according to the present embodiment has a facial area feature amount 1410 generated from a facial area partial image 147 of the store visitor, a body area feature amount 1420 generated from the store visitor's body area partial image 148, and a facial area feature amount 1420 generated from the store visitor's body area partial image 148. By feeding the estimation model 1400 with the audio feature amount 1430 generated from the specific section audio 149, it is possible to propose clothing according to the store visitor's tastes.

Further, by issuing the coupon 10 including the coupon ID 166, the display terminal 100 according to the present embodiment can provide a platform for generating a learning data set used in a learning phase as described below.

<D. Functional configuration of POS terminal 200>
Next, the functions and processing of the POS terminal 200 that constitutes the clothing proposal system 1 according to the present embodiment will be explained. In the clothing proposal system 1, the POS terminal 200 is mainly responsible for part of the learning phase for constructing a trained model.

(d1: Functional configuration of POS terminal 200)
FIG. 18 is a schematic diagram showing an example of the functional configuration of the POS terminal 200 that constitutes the clothing proposal system 1 according to the present embodiment. Each function shown in FIG. 18 may typically be realized by the processor 204 of the POS terminal 200 executing the OS 212 and the application program 214 (both shown in FIG. 8).

Referring to FIG. 18, POS terminal 200 has an accounting function 240 and a sales information storage function 250 as functional configurations.

The accounting function 240 is in charge of payment processing when a visitor purchases an item. More specifically, the accounting function 240 calculates the amount of the item to be purchased, the discount amount, the payment amount, etc. based on the information on the item tag 20 attached to the item to be purchased and the coupon ID 166 read from the coupon. At the same time, payment processing is executed. Accounting function 240 outputs sales information 218 indicating items for which payment has been processed.

The sales information storage function 250 adds the coupon ID 166 read from the coupon 10 to the sales information 218 output from the accounting function 240 and stores the same. The sales information 218 (to which the coupon ID 166 is assigned) stored by the sales information storage function 250 is transmitted to the management device 300 and used in a learning process to generate a trained model, as described later.

(d2: Sales information storage function 250)
Next, details of the sales information storage function 250 of the POS terminal 200 shown in FIG. 18 will be explained.

FIG. 19 is a diagram for explaining processing contents in the sales information storage function 250 of the POS terminal 200 that constitutes the clothing proposal system 1 according to the present embodiment. Referring to FIG. 19, POS terminal 200 includes a correlation module 252 and a sales information storage section 254 as a sales information storage function 250.

In response to input of the sales information 218 and the coupon ID 166 from the accounting function 240 (see FIG. 18), the association module 252 determines whether the coupon 10 was attached to the coupon 10 used when purchasing the item related to the sales information 218. The coupon ID 166 is accepted and associated with the sales information 218. The association module 252 stores the associated coupon ID 166 and sales information 218 together in the sales information storage unit 254.

The sales information storage unit 254 is realized using at least a part of the storage area provided by the memory 106 or the storage 110 (both shown in FIG. 7). The sales information storage unit 254 stores data in units of data sets consisting of coupon IDs 166 and sales information 218.

The sales information 218 typically stores the number of items sold for each item type (item 1, item 2, . . . , item N).

(d3: Processing procedure)
Next, sales management processing executed in the POS terminal 200 that constitutes the clothing proposal system 1 will be explained.

FIG. 20 is a flowchart showing the processing procedure of sales management processing in the POS terminal 200 configuring the clothing proposal system 1 according to the present embodiment. Each step shown in FIG. 20 may typically be realized by the processor 204 of the POS terminal 200 executing the OS 212 and the application program 214 (both shown in FIG. 8).

Referring to FIG. 20, POS terminal 200 first determines whether coupon 10 has been read by optical reader 228 (see FIG. 8) (step S200). When the coupon 10 is read (YES in step S200), the POS terminal 200 acquires the coupon ID 166 of the read coupon 10 (step S202). On the other hand, if the coupon 10 is not read (NO in step S200), the process of step S202 is skipped.

Subsequently, the POS terminal 200 determines whether the item tag 20 attached to the item to be purchased has been read by the optical reader 228 (see FIG. 8) (step S204). When the item tag 20 is read (YES in step S204), the POS terminal 200 adds the item information of the read item tag 20 to the sales information 218 (step S206).

Then, the POS terminal 200 determines whether an instruction to finish reading the item tag has been given (step S208). If the instruction to finish reading the item tag has not been given (NO in step S208), the processes from step S204 onwards are repeated.

When the instruction to finish reading the item tag is given (YES in step S208), the POS terminal 200 calculates the payment amount based on the presence or absence of the coupon 10 and the current sales information 218 (step S210). Then, the POS terminal 200 executes a payment process for the payment amount calculated in step S210 (step S212).

Subsequently, the POS terminal 200 determines whether the coupon ID 166 has been acquired (step S214). That is, in step S200, it is determined whether the coupon 10 has been read.

If the coupon ID 166 has been acquired (YES in step S214), the POS terminal 200 stores the sales information 218 in association with the coupon ID 166 (step S216). On the other hand, if the coupon ID 166 has not been acquired (NO in step S214), the process of step S216 is skipped. With the above steps, the sales management process for one store visitor is completed.

(d4: summary)
The POS terminal 200 according to this embodiment executes payment processing for items purchased by a store visitor, reads the coupon ID 166 given to the coupon 10 presented at that time, and associates it with the purchased item. save. This saved purchased item information (sales information 218) is used to generate a learning data set used in a learning phase as described below.

<E. Overview of learning phase>
Next, an overview of the learning phase in the clothing proposal system 1 according to the present embodiment will be explained.

The clothing proposal system 1 according to this embodiment associates the captured image 136 and collected audio 138 stored in the display terminal 100 with the sales information 218 stored in the POS terminal 200 regarding the same visitor. A learning data set 324 is generated, and an estimation model is learned using the generated learning data set 324.

FIG. 21 is a diagram for explaining an overview of the learning phase in the clothing proposal system 1 according to the present embodiment. Referring to FIG. 21, display terminal 100 transmits captured image 136 and collected audio 138 associated with coupon ID 166, which were acquired during execution of the item estimation process, to management device 300 (sequence SQ1). Similarly, the POS terminal 200 transmits the sales information 218 associated with the coupon ID 166, which was acquired when executing the sales management process, to the management device 300 (sequence SQ2).

The management device 300 creates the learning data set 324 by associating the captured image 136 and collected audio 138 transmitted from the display terminal 100 with the sales information 218 transmitted from the POS terminal 200 using the coupon ID 166 as a key. generate (sequence SQ3). That is, sequence SQ3 corresponds to preprocessing for generating the learning data set 324.

The management device 300 uses the generated learning data set 324 to learn or additionally learn the estimated model, thereby generating a learned model 326 (sequence SQ4). Then, the management device 300 transmits the generated trained model 326 to each of the display terminals 100 (sequence SQ5). The display terminal 100 stores the learned model 326 sent from the management device 300 as the learned model 116. That is, the learned model 116 of the display terminal 100 is set or updated.

As shown in FIG. 21, in the clothing proposal system 1 according to the present embodiment, the information acquired by the display terminal 100 and the POS terminal 200 can be combined using the coupon ID 166 given to the coupon 10. , the learning data set 324 for improving the estimation accuracy of the estimation model can be easily generated without imposing a burden on the store visitors.

<F. Functional configuration of management device 300>
Next, the functions and processing of the management device 300 that constitutes the clothing proposal system 1 according to this embodiment will be explained. In the clothing proposal system 1, the management device 300 is mainly responsible for a part of the learning phase for constructing a trained model.

(f1: Functional configuration of management device 300)
FIG. 22 is a schematic diagram showing an example of the functional configuration of the management device 300 that constitutes the clothing proposal system 1 according to the present embodiment. Each function shown in FIG. 22 is typically realized by the processor 304 of the management device 300 executing the OS 312, application program 314, preprocessing program 316, and learning program 318 (all shown in FIG. 9). It's okay.

Referring to FIG. 22, the management device 300 has a captured image/collected audio/sales information acquisition function 340, a learning data set generation function 350, and a learning function 360 as functional configurations.

The captured image/collected audio/sales information acquisition function 340 retrieves the captured image 136 and collected audio 138 associated with the coupon ID 166 that are stored in the display terminal 100 and the coupon ID 166 that is stored in the POS terminal 200. obtains the sales information 218 associated with. These data will be used as a learning dataset. That is, the captured image/collected audio/sales information acquisition function 340 of the management device 300 corresponds to a configuration for acquiring a learning data set.

As a method for acquiring data from the display terminal 100 and the POS terminal 200, for example, some instructions may be given to the display terminal 100 and the POS terminal 200 so that the display terminal 100 and the POS terminal 200 respectively transmit data. Alternatively, the management device 300 may access the display terminal 100 and the POS terminal 200 to obtain data from each. Alternatively, the display terminal 100 and the POS terminal 200 may transmit data to the management device 300 at predetermined intervals.

The learning data set generation function 350 generates captured images 136 and collected audio 138 associated with the coupon ID 166 acquired from the display terminal 100, and sales information 218 associated with the coupon ID 166 acquired from the POS terminal 200. From this, a learning data set 324 is generated.

The learning function 360 uses the learning data set 324 generated by the learning data set generation function 350 to generate a learned model 326 by learning the estimated model. The generated learned model 326 is transmitted to the display terminal 100.

(f2: Learning dataset generation function 350)
Next, details of the learning data set generation function 350 of the management device 300 shown in FIG. 22 will be described.

FIG. 23 is a diagram for explaining the processing contents in the learning data set generation function 350 of the management device 300 that constitutes the clothing proposal system 1 according to the present embodiment. Referring to FIG. 23, regarding the learning data set generation function 350, the management device 300 acquires the captured image 136 and collected audio 138 associated with the coupon ID 166 acquired from the display terminal 100, and the acquired voice 138 acquired from the POS terminal 200. The sales information 218 that has been associated with the coupon ID 166 is compared with the sales information 218 that has been associated with the coupon ID 166, and the data having the same coupon ID 166 are associated with each other.

As an example, FIG. 23 shows a data set of a captured image 136 and a collected audio 138 to which "01", "02", and "03" are assigned as coupon IDs 166, respectively, and "02" and "03" are assigned as coupon IDs 166, respectively. 03" and "08" are shown, respectively. Sales information 218 is shown. Of these, the captured image 136, collected audio 138, and sales information 218 are all available for data to which the coupon ID 166 is assigned "02" and "03." These three types of data can be determined as learning data (the relationship between the input information and the correct value of the estimation result). By generating learning data for each of the plurality of coupon IDs 166, the learning data set 324 can be generated.

At this time, the sales information 218 is used as a label (tag) in order to adapt the learning process to be described later. In other words, the learning data set 324 is configured based on the captured image 136 (learning image) obtained by capturing an arbitrary customer and the collected audio 138 (learning audio) uttered by the arbitrary customer. Clothing items (sales information 218) purchased by the customer are labeled.

(f3: learning function 360)
Next, details of the learning function 360 of the management device 300 shown in FIG. 22 will be described.

FIG. 24 is a diagram for explaining the processing content in the learning function 360 of the management device 300 that constitutes the clothing proposal system 1 according to the present embodiment. Referring to FIG. 24, management device 300 includes, as learning function 360, region identification module 141, size adjustment modules 142 and 143, section identification module 144, and resampling module 145. These modules are substantially the same as the modules that the display terminal 100 has as the suggested item estimation function 140. Therefore, a detailed description of these modules will not be repeated.

Furthermore, the management device 300 includes a parameter optimization module 362 as a learning function 360. Parameter optimization module 362 generates trained model 326 by optimizing model parameters 364 for defining estimated model 1400.

The parameter optimization module 362 optimizes the model parameters 364 using each set (learning data) of the captured image 136, collected audio 138, and sales information 218 included in the learning data set 324.

More specifically, the parameter optimization module 362 calculates a face region feature amount 1410, a body region feature amount 1420, and a voice feature amount 1430 from each set of captured images 136 and collected audio 138 included in the learning data set 324. The estimation result 1450 is calculated by generating and inputting it to the estimation model 1400. Then, the parameter optimization module 362 calculates an error by comparing the estimation result 1450 output from the estimation model 1400 and the corresponding sales information 218 (correct label), and adjusts the model parameters 364 according to the calculated error. Optimize (adjust) values.

That is, the parameter optimization module 362 corresponds to a learning unit, and is based on the facial area feature amount 1410 (the first The estimation result 1450 that is output by inputting the body region feature amount 1420 (second feature amount), and the audio feature amount 1430 (third feature amount) to the estimation model 1400 is labeled with the training data. The estimation model 1400 is optimized so as to approximate the purchase history (sales information 218) of clothing items. In other words, the parameter optimization module 362 extracts features from the captured image 136 and collected audio 138 included in the learning data and inputs them into the estimation model 1400. Adjust model parameters 364 to match .

In the same procedure, the model parameters 364 of the estimation model 1400 are repeatedly optimized based on each learning data (captured image 136, collected audio 138, and sales information 218) included in the learning data set 324. A completed model 326 is generated.

Parameter optimization module 362 may use any optimization algorithm to optimize the values of model parameters 364. More specifically, as an optimization algorithm, for example, SGD (Stochastic Gradient Descent), Momentum SGD (Inertial term added SGD), AdaGrad, RMSprop, AdaDelta, Adam (Adaptive moment estimation), etc. A gradient method can be used.

Note that when outputting each element of the estimation result 1450 output from the estimation model 1400 as a normalized probability, the item type (item 1, item 2, ..., item N) included in the sales information 218 It is also preferable to normalize the number of sales for each period (see FIG. 19).

The estimated model 1400 whose model parameters 364 have been optimized by the parameter optimization module 362 corresponds to the learned model 326 and is transmitted to the display terminal 100.

(f4: processing procedure)
Next, a learning process executed in the management device 300 that constitutes the clothing proposal system 1 will be described.

FIG. 25 is a flowchart showing the processing procedure of the learning process in the management device 300 that constitutes the clothing suggestion system 1 according to the present embodiment. Each step shown in FIG. 25 is typically realized by the processor 304 of the management device 300 executing the OS 312, application program 314, preprocessing program 316, and learning program 318 (all shown in FIG. 9). It's okay.

Referring to FIG. 25, management device 300 acquires captured image 136 and collected audio 138 to which coupon ID 166 is attached from display terminal 100 (step S300). At the same time, the management device 300 acquires the sales information 218 to which the coupon ID 166 is attached from the POS terminal 200 (step S302). That is, the management device 300 executes a process of acquiring the coupon ID 166 (identification information) included in the coupon 10 as a medium and the clothing item (sales information 218) purchased by the customer.

Then, the management device 300 generates a learning data set 324 by associating the captured image 136 and collected audio 138 with the sales information 218 using the coupon ID 166 as a key (step S304). That is, the management device 300 executes a process of associating the coupon ID 166 (identification information) acquired from the coupon 10 that is the medium with the clothing item (sales information 218) purchased by the customer, and further performs an image capturing process using the coupon ID 166 as a key. A process of associating the image 136, the collected audio 138, and the sales information 218 with each other and storing the association as learning data used for learning the estimation model 1400 is executed.

The management device 300 selects one data set (learning data) from the generated learning data set 324 (step S306).

The management device 300 extracts the face area partial image 147 and the body area partial image 148 from the captured image 136 of the selected data (step S308). Then, the management device 300 adjusts the size of the extracted face area partial image 147 and body area partial image 148 to extract the face area feature amount 1410 and the body area feature amount 1420 (step S310).

In this way, the management device 300 executes the process of identifying the face area representing the customer's face and the body area representing the customer's body in the captured image 136 of each learning data. Then, the management device 300 extracts a face region feature amount 1410 (first feature amount) from the face region of the captured image 136, and extracts a body region feature amount 1420 (second feature amount) from the body region of the captured image 136. Execute the processing to be performed.

In addition, the management device 300 extracts the speech section included in the collected speech 138 of the selected data as the specific section speech 149 (step S312). Then, the management device 300 resamples the extracted specific section audio 149 and extracts the audio feature amount 1430 (step S314). In this way, the management device 300 executes the process of extracting the voice feature amount 1430 (third feature amount) from the voice of the part of the collected voice 138 that corresponds to the customer's utterance.

The management device 300 inputs the face region feature amount 1410 and the body region feature amount 1420 generated in step S310, and the voice feature amount 1430 generated in step S314 to the estimation model 1400 to generate an estimation result 1450 (step S316 ).

The management device 300 optimizes the model parameters 364 of the estimation model based on the error between the sales information 218 of the selected data and the estimation result 1450 generated in step S316 (step S318).

In this way, the management device 300 inputs the face region feature amount 1410 (first feature amount), the body region feature amount 1420 (second feature amount), and the voice feature amount 1430 (third feature amount) to the estimation model 1400. A process of optimizing the estimation model 1400 is executed so that the estimation result 1450 outputted from the training data approaches the purchase history (sales information 218) of the clothing item labeled in the learning data.

Then, the management device 300 determines whether all the learning data sets 324 generated in step S304 have been processed (step S320). If all of the learning data set 324 has not been processed (NO in step S320), the processes from step S306 onwards are repeated.

If all of the learning data set 324 has been processed (YES in step S320), the management device 300 transmits the learned model 326 defined by the current model parameters 364 to each display terminal 100 (step S322). . With the above, the learning process is completed.

(f5: summary)
The management device 300 according to the present embodiment learns by associating the captured image 136 and collected audio 138 acquired from the display terminal 100 with the sales information 218 acquired from the POS terminal 200 using the coupon ID 166 as a key. data set 324 can be easily generated. By using such a training data set 324, it is possible to construct an estimation model or perform additional learning on the trained model 326. This makes it possible to improve the accuracy of clothing suggestions.

<G. Modified example>
In the above-described embodiment, the clothing suggestion system 1 in which the display terminal 100, the POS terminal 200, and the management device 300 are arranged in a single store 30 is illustrated as a typical example, but the present invention is not limited to this, and various modifications can be made. is possible. Some modified examples will be described below.

(g1: Multiple store collaboration: Modification example 1)
As a variation, the management device 300 may manage multiple stores.

FIG. 26 is a schematic diagram showing an example of the system configuration of the clothing proposal system 1A according to the first modification of the present embodiment. Referring to FIG. 26, one or more display terminals 100 and one or more POS terminals 200 are arranged in each of store 30A and store 30B. Each store 30 is connected to the same management device 300 via the wide area network 4.

The management device 300 acquires necessary information (captured images 136, collected audio 138, and sales information 218) from the display terminal 100 and POS terminal 200 of the store 30A, and also acquires necessary information from the display terminal 100 and POS terminal 200 of the store 30B. Get information. Then, the management device 300 generates a learned model common to both stores or a learned model for each store based on the collected information.

By adopting the configuration shown in FIG. 26, the number of management devices 300 to be arranged can be reduced, and more learning data sets can be acquired, so the estimation accuracy of the trained model can be improved.

(g2: Item proposal by category: Modification example 2)
Since the estimation model 1400 (see FIG. 11) according to the above-described embodiment is inputted with the audio feature amount 1430 corresponding to any of the categories displayed on the category selection reception screen 50, basically, the estimated model 1400 (see FIG. 11) is In the estimation result 1450, items belonging to the uttered category will have relatively high scores. Note that each of the plurality of clothing items belongs to one of a plurality of predetermined categories (product categories).

However, if there are many items belonging to other categories that were purchased at the same time as the item belonging to the selected category, the estimation result 1450 may include items belonging to other categories having relatively high scores. obtain. In such a case, items belonging to categories other than the selected category will also be proposed on the item suggestion screen 52.

FIG. 27 is a diagram for explaining an item suggestion screen displayed on the display terminal 100 of the clothing suggestion system 1 according to the second modification of the present embodiment. As shown in FIG. 27(a), when an item belonging to another category has a relatively high score in the estimation result 1450, the list display 54 of the item suggestion screen 52 shows the item belonging to the other category. (Symbol 54M) will be included.

The item suggestion screen 52 may be displayed which may include items belonging to other categories, but as shown in FIG. 27(b), items belonging to the selected category and other categories The items to which it belongs may be proposed in a different display mode.

That is, the item suggestion screen 52 shown in FIG. 27(b) includes a list display 54 consisting of items belonging to the category selected by the store visitor 40 by voice, and a list display 54 consisting of items belonging to the category other than the category selected by the store visitor 40 by voice. A list display 55 is included. The list display 55 also displays a message such as ``How about this one too?'' indicating that the item is suitable for the proposal based on past sales results, although it is in a different category than the selected category.

FIG. 28 is a diagram for explaining the processing contents in the display control function 150A and the coupon issuing control function 160 of the display terminal 100 that constitute the clothing suggestion system 1 according to the second modification of the present embodiment. Referring to FIG. 28, display terminal 100 has display control module 152A, audio analysis module 154, and category/item correspondence information 156 as display control function 150A.

The voice analysis module 154 performs voice analysis on the collected voice 138 uttered by the store visitor 40 to identify the category selected by the store visitor 40 by voice. That is, the voice analysis module 154 identifies the category selected by the customer from among a plurality of categories based on the voice uttered by the customer. Note that any known algorithm can be used as the voice analysis method by the voice analysis module 154. The categories identified by the audio analysis module 154 are provided to the display control module 152A.

The display control module 152A receives the estimation result 1450 calculated by the suggested item estimation function 140, and identifies the item having the highest score in the estimation result 1450. The display control module 152A refers to the category/item correspondence information 156 and determines whether each item having a high score in the estimation result 1450 belongs to the category specified by the audio analysis module 154. Then, the display control module 152A refers to the item image 118, adds the image to the list display 54 for items belonging to the category specified by the audio analysis module 154, and adds the image to the list display 54, For items belonging to other categories, the item suggestion screen 52A is generated by adding the image to the list display 55. The generated item suggestion screen 52A is displayed on the display 102.

By executing the processing in the display control module 152A as described above, an item suggestion screen 52A as shown in FIG. 27(b) is provided. That is, the display 102 displays clothing items that belong to the category specified by the voice analysis module 154 and clothing items that do not belong to the specified category among the clothing items displayed based on the estimation result 1450 in different display modes. Display in . By adopting such a display mode, it is possible to encourage purchase of items other than the category selected by the store visitor 40.

Other processes and functions are substantially the same as those described with reference to FIG. 15, so detailed descriptions will not be repeated.

(g3: Network: Modification 3)
In the embodiment described above, the estimation model 1400 to which the face region feature amount 1410, the body region feature amount 1420, and the voice feature amount 1430 are input is illustrated, but an estimation model that can input additional information is also adopted. Good too.

FIG. 29 is a diagram for explaining the processing contents in the suggested item estimation function 140 of the display terminal 100 that constitutes the clothing suggestion system 1 according to the third modification of the present embodiment. FIG. 29 shows an estimation model 1400A that accepts meteorological information such as weather and temperature as an input feature amount 1440. In this way, by adding input information, estimation accuracy can be improved.

When a feature quantity input to the estimation model 1400A is added, the information included in the learning data set used for learning the estimation model 1400A is also increased in accordance with the input feature quantity.

Note that although weather information is shown as a typical example in FIG. 29, additional information to be input is not limited to this, and any information that is presumed to have some relevance to the decision of the item to be proposed may be used. Information can be adopted. For example, other weather information such as wind speed and sunshine hours, time information such as date and time and day of the week, and information such as how crowded the store is may be used.

(g4: Network: Modification 4)
In the embodiment described above, the estimation model 1400 to which the face region feature amount 1410, the body region feature amount 1420, and the voice feature amount 1430 are inputted is illustrated, but an estimation model that substitutes some information may be adopted. It's okay.

FIG. 30 is a diagram for explaining the processing contents in the suggested item estimation function 140 of the display terminal 100 that constitutes the clothing suggestion system 1 according to the fourth modification of the present embodiment. FIG. 30 shows a configuration example in which an input feature amount 1442 indicating a category is input instead of the audio feature amount 1430. The input feature amount 1442 may be generated by analyzing the collected speech 138 uttered by the store visitor 40 and specifying the category selected by the store visitor 40 by voice.

Alternatively, if the store visitor 40 selects a category by touching the part corresponding to the category on the category selection reception screen 50 displayed on the display terminal 100, the category selected by the touch operation is input to the feature amount 1442. You may also input it as .

Note that although FIG. 30 shows an example in which the input feature amount 1442 indicating the category is input, by adopting the configuration shown in FIG. good.

(g5: Item proposal using mobile terminal: Modification example 5)
As a variation, the above-described item suggestion may be made on a personally owned mobile terminal instead of at a physical store.

FIG. 31 is a schematic diagram showing an example of use of the clothing suggestion system 1B according to the fifth modification of the present embodiment. Referring to FIG. 31, by installing an application on mobile terminal 500, the same functions as display terminal 100 can be realized on mobile terminal 500. The net user executes an application on the mobile terminal 500, takes an image of himself using the camera installed in the mobile terminal 500, and speaks the desired category, thereby receiving the above-mentioned clothing suggestions. be able to.

Any implementation may be adopted to implement the item estimation process according to this embodiment on mobile terminal 500.

FIG. 32 is a schematic diagram showing an implementation example of the clothing suggestion system according to the fifth modification of the present embodiment.

FIG. 32(a) shows an implementation example in which the item estimation process is implemented by the mobile terminal 500 alone. As shown in FIG. 32(a), an application 510 is installed from the server device 400 onto the mobile terminal 500. The application 510 has a suggested item estimation function 512, a display control function 514, and a coupon issuance control function 516. The suggested item estimation function 512, display control function 514, and coupon issuance control function 516 are substantially the same as the suggested item estimation function 140, display control function 150, and coupon issuance control function 160 of the display terminal 100 (all see FIG. 10). Execute the process.

In the implementation example shown in FIG. 32A, a trained model 518 (substantially the same as the trained model 116 located in the display terminal 100) is located in the mobile terminal 500, so communication with the server device 400 is Even if it cannot be performed, item estimation processing can be performed.

FIG. 32(b) shows an implementation example in which the server device 400 and the mobile terminal 500 cooperate to realize item estimation processing. As shown in FIG. 32(b), an application 520 is installed on the mobile terminal 500 from the server device 400. The application 520 has a feature generation function 522 and a display function 524. The feature amount generation function 522 extracts a face area feature amount 1410 and a body area feature amount 1420 from a captured image obtained by capturing an image of a net user, and also generates a voice feature amount 1430 from the collected voice 138 uttered by the net user. and transmits it to the server device 400.

The display function 524 outputs the display content from the server device 400 to the display of the mobile terminal 500.

On the other hand, the server device 400 has a suggested item estimation function 412, a display control function 414, and a coupon issuance control function 416. The suggested item estimation function 412 corresponds to the part of the suggested item estimation function 140 (see FIG. 11) of the display terminal 100 excluding the function of extracting feature amounts. The display control function 414 and the coupon issuance control function 416 execute substantially the same processing as the display control function 150 and the coupon issuance control function 160 (both see FIG. 10) of the display terminal 100.

In the example of implementation shown in FIG. It becomes possible to update the model 518 as appropriate. Moreover, since the mobile terminal 500 only needs to extract the feature amount, the consumption of resources on the mobile terminal 500 side can be reduced.

FIG. 32(c) shows an implementation example in which the server device 400 and the mobile terminal 500 cooperate to realize item estimation processing. As shown in FIG. 32(c), an application 530 is installed on the mobile terminal 500 from the server device 400. The application 530 has an image/audio transmission function 532 and a display function 524. The image/audio transmission function 532 transmits to the server device 400 a captured image obtained by capturing an image of a net user and a collected voice 138 uttered by the net user.

The display function 534 outputs the display content from the server device 400 to the display of the mobile terminal 500.

On the other hand, the server device 400 has a suggested item estimation function 412, a display control function 414, and a coupon issuance control function 416. The suggested item estimation function 412, the display control function 414, and the coupon issuance control function 416 execute substantially the same processing as the display control function 150 and the coupon issuance control function 160 (all see FIG. 10) of the display terminal 100.

In the implementation example shown in FIG. It becomes possible to update the model 518 as appropriate. Furthermore, since the mobile terminal 500 only needs to transmit the captured image 136 and the collected audio 138 to the server device 400 as they are, the consumption of resources on the mobile terminal 500 side can be reduced.

(g6:Other)
It is obvious that various modifications can be made in accordance with the spirit of the present invention, without being limited to the above-mentioned modifications. Furthermore, one or more of the above-mentioned modifications can be arbitrarily combined.

<H. Summary＞
According to the clothing proposal system 1 according to the present embodiment, the facial area feature amount 1410 generated from the facial area partial image 147 of the store visitor, and the body area feature amount 1420 generated from the store visitor's body area partial image 148. By using the audio feature amount 1430 generated from the visitor's specific section speech 149 as input information, it is possible to propose clothing according to the visitor's taste with higher accuracy.

Further, according to the clothing suggestion system 1 according to the present embodiment, input information acquired from the store visitor (captured image 136 and collected audio 138) and items actually purchased by the store visitor are attached to the coupon 10. A learning data set 324 can be generated by associating the coupons with each other using the coupon IDs 166. By learning the estimation model using such a training data set 324, estimation accuracy can be continuously improved, and even when new items are added, the estimation model can be adapted. be able to.

Furthermore, since the clothing proposal system 1 according to the present embodiment issues a coupon 10 with which the payment amount is discounted, the store visitor has an incentive to actively use the coupon 10. As a result, the possibility of collecting information for generating the learning data set 324 can be increased.

The embodiments disclosed this time should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is indicated by the claims rather than the description of the embodiments described above, and it is intended that all changes within the meaning and range equivalent to the claims are included.

1, 1A, 1B Clothing proposal system, 2 Local network, 4 Wide area network, 10 Coupon, 12 Discount amount display, 14, 54, 55 List display, 16 Map, 18 Identification image, 20 Item tag, 30, 30A, 30B Store , 40 visitor, 50 category selection reception screen, 52, 52A item suggestion screen, 56 coupon issue button, 100 display terminal, 102, 202, 302 display, 104, 204, 304 processor, 106, 206, 306 memory, 108, 208,308 Network controller, 110,210,310 Storage, 112,212,312 OS, 114,214,314 Application program, 116,326,518 Trained model, 118 Item image, 120,220 Printer, 122,222 Optics Drive, 124,224 Optical disk, 126,226 Touch detection unit, 128 Human sensor, 130 Camera, 132 Microphone, 136 Captured image, 138 Collected audio, 140,412,512 Suggested item estimation function, 141 Area identification module, 142 , 143 size adjustment module, 144 section identification module, 145 resampling module, 147 face area partial image, 148 body area partial image, 149 specific section audio, 150, 150A, 414, 514 display control function, 152, 152A display control module , 154 Voice analysis module, 156 Category/item correspondence information, 160,416,516 Coupon issue control function, 162 Coupon issue control module, 164 Coupon ID generation module, 166 Coupon ID, 170 Image and audio storage function, 172,252 Correspondence module, 174 image and audio storage unit, 200 POS terminal, 216 item information, 218, 322 sales information, 228 optical reader, 230, 330 input unit, 232 payment processing unit, 240 accounting function, 250 sales information storage function, 254 sales information storage unit, 300 management device, 316 preprocessing program, 318 learning program, 320 audio information, 324 learning dataset, 340 sales information acquisition function, 350 learning dataset generation function, 360 learning function, 362 parameter optimization module, 364 model parameter, 400 server device, 500 mobile terminal, 510, 520, 530 application, 522 feature generation function, 524, 534 display function, 532 image and audio transmission function, 1400, 1400A estimation model, 1410 face area feature , 1420 Body region feature amount, 1430 Audio feature amount, 1440, 1442 Input feature amount, 1450 Estimation result, 1460, 1470, 1480 Preprocessing network, 1490 Middle layer, 1492 Activation function, 1494 Softmax function.

Claims (7)

An information processing device that proposes a clothing item suitable for a customer from among a plurality of clothing items based on a feature amount representing the customer's characteristics based on an estimation result of a learned estimation model, the information processing device comprising:
a camera for capturing an image of the customer;
A first feature vector from a face region representing the customer's face in an image obtained by imaging the customer with the camera, and a second feature vector from a body region representing the customer's body in the image. and an image feature extraction unit for extracting the
In machine learning for the estimation model, a purchased clothing item among the plurality of clothing items is used as the correct answer for the output of the estimation model,
The estimation model receives the first feature vector and the second feature vector and outputs the probability that each of the plurality of clothing items is a clothing item to be proposed as the estimation result. , information processing equipment. an area specifying unit for specifying the face area and the body area in the image;
The information processing device according to claim 1, further comprising a display unit for displaying clothing items appropriate for the customer based on the estimation result. The information processing device according to claim 2, wherein the area specifying unit specifies a part representing clothing worn by the customer as the body area. An information processing device that proposes a clothing item suitable for a customer from among a plurality of clothing items based on a feature amount representing the customer's characteristics based on an estimation result of a learned estimation model, the information processing device comprising:
a camera for capturing an image of the customer;
Extracting a first feature amount from a face region representing the customer's face in an image obtained by imaging the customer with the camera and a second feature amount from a body region representing the customer's body in the image. an image feature extraction unit for
an area specifying unit for specifying the face area and the body area in the image;
a display unit for displaying clothing items according to the customer based on the estimation result;
Equipped with a microphone for collecting audio,
The estimation model receives the first feature amount and the second feature amount, and outputs each possibility that each of the plurality of clothing items is a clothing item to be proposed as the estimation result;
The area specifying unit specifies a part representing clothing worn by the customer as the body area,
The display unit displays a list of categories indicating classifications of clothing items, and displays a message prompting the customer to select one of the categories displayed in the list by voice. Each of the plurality of clothing items belongs to one of a plurality of predetermined categories,
The information processing device further includes a voice analysis unit for identifying a category selected by the customer from among the plurality of categories based on the voice uttered by the customer,
The display section displays, among the clothing items displayed based on the estimation results, clothing items that belong to the category specified by the voice analysis section and clothing items that do not belong to the specified category in different display modes. The information processing device according to claim 4, which displays the information. An information processing system,
an information processing device that inputs feature quantities representing characteristics of a customer into a learned estimation model and proposes a clothing item suitable for the customer from among a plurality of clothing items based on the estimation result of the estimation model;
and a learning device for generating the estimated model,
The information processing device includes:
a camera for capturing an image of the customer;
A first feature vector from a face region representing the customer's face in an input image obtained by imaging the customer with the camera, and a second feature vector from a body region representing the customer's body in the input image. an image feature extraction unit for extracting the quantity vector;
The estimation model receives the first feature vector and the second feature vector and outputs the probability that each of the plurality of clothing items is a clothing item to be proposed as the estimation result. It is learned that
The learning device includes:
The learning data set includes an acquisition unit for acquiring a learning data set, and the learning data set includes a learning image obtained by capturing images of other customers. A first learning feature vector from a face region representing the other customer's face in the learning image, and a first learning feature vector from a face region representing the other customer's face in the learning image; an image feature extraction unit for extracting a second learning feature vector from a body region representing the body of the other customer in the learning image;
The first learning feature vector and the second learning feature vector extracted from the learning data are input to the estimation model and an estimation result is outputted by labeling the learning data. and a learning unit for optimizing the estimation model so as to approximate the purchase history of clothing items. A learning device for generating an estimation model used to propose a clothing item suitable for the customer from among a plurality of clothing items upon receiving an input of feature values representing the customer's characteristics, the learning device comprising:
The learning data set includes an acquisition unit for acquiring a learning data set, and the learning data set selects the clothing items purchased by the customer from among the plurality of clothing items with respect to the image obtained by capturing the customer. It includes a plurality of learning data associated as correct answers of the output of the estimation model , and further includes a first feature vector from a face region representing the customer's face in the image and a first feature vector from a body region representing the customer's body in the image. an image feature extraction unit for extracting a second feature vector;
The first feature vector and the second feature vector extracted from the learning data are input to the estimation model, and the estimation result output is the estimation result of the clothing item labeled in the learning data. A learning device, comprising: a learning unit for optimizing the estimation model so as to approximate the purchase record.

Images