JP2024043774A - Information processing device, information processing method, and information processing program - Google Patents

Abstract

[Problem] To provide an information processing device, an information processing method, and an information processing program that can improve user convenience. [Solution] The information processing device according to the present application includes an image processing unit and an output unit. The image processing unit generates a composite image by combining a user face image, which is a facial image of the user, with a hair image selected by the user. The output unit outputs the composite image generated by the image processing unit to a hair salon professional. [Selected Figure] FIG.

Description

The present invention relates to an information processing device, an information processing method, and an information processing program.

2. Description of the Related Art Conventionally, a technique is known in which a user's face image and hair image are combined and presented to the user. For example, Patent Document 1 proposes a technique that performs image processing using hair simulation to change hairstyles and hair colors, and displays the results of such image processing.

JP 2013-178789 A

However, the above-mentioned conventional technology only performs image processing for hair simulation, and there is room for improvement in terms of improving user convenience.

The present application has been made in consideration of the above, and aims to provide an information processing device, an information processing method, and an information processing program that can improve user convenience.

An information processing device according to the present application includes an image processing section and an output section. The image processing unit generates a composite image by combining a user face image, which is a user's face image, and a hair image selected by the user. The output unit outputs the composite image generated by the image processing unit to the hair practitioner.

According to one aspect of the embodiment, user convenience can be improved.

FIG. 1 is a diagram illustrating an example of information processing according to an embodiment. FIG. 2 is a diagram illustrating an example of the configuration of an information processing system according to the embodiment. FIG. 3 is a diagram illustrating a configuration example of an information providing device according to an embodiment. FIG. 4 is a diagram illustrating an example of a user information table stored in the user information storage unit of the information providing device according to the embodiment. FIG. 5 is a diagram illustrating an example of a posted information table stored in the posted information storage unit of the information providing device according to the embodiment. FIG. 6 is a diagram illustrating an example of a model information table stored in the model information storage unit of the information providing device according to the embodiment. FIG. 7 is a diagram illustrating an example of a content table stored in the content storage unit of the information providing device according to the embodiment. FIG. 8 is a diagram illustrating an example of the configuration of a terminal device according to the embodiment. FIG. 9 is a diagram illustrating an example of the configuration of the reception unit of the terminal device according to the embodiment. FIG. 10 is a diagram illustrating an example of the configuration of an image processing unit of a terminal device according to the embodiment. FIG. 11 is a diagram illustrating an example of a reservation setting screen displayed on the display unit by the output unit of the information providing device according to the embodiment. FIG. 12 is a diagram illustrating an example of a keyword designation screen displayed on the display unit by the output unit of the information providing device according to the embodiment. FIG. 13 is a diagram for explaining a process of acquiring a similar hair image from a specified keyword by the processing unit of the information providing device according to the embodiment. FIG. 14 is a diagram showing another example of the reservation setting screen displayed on the display unit by the output unit of the information providing device according to the embodiment. FIG. 15 is a diagram showing still another example of the reservation setting screen displayed on the display unit by the output unit of the information providing device according to the embodiment. FIG. 16 is a diagram illustrating an example of a composite image change screen displayed on the display unit by the output unit of the information providing device according to the embodiment. FIG. 17 is a diagram for explaining a process of estimating a treatment content from a user image and a composite image by an estimation unit of an information providing device according to an embodiment. FIG. 18 is a diagram showing an example of a face type selection screen displayed on the display unit by the output unit of the information providing device according to the embodiment. FIG. 19 is a flowchart illustrating an example of information processing by the processing unit of the terminal device according to the embodiment. FIG. 20 is a flowchart illustrating an example of a reservation process by the processing unit of the terminal device according to the embodiment. FIG. 21 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information providing device and the terminal device according to the embodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS Below, an information processing apparatus, an information processing method, and a form for implementing an information processing program (hereinafter referred to as an "embodiment") according to the present application will be described in detail with reference to the drawings. Note that the information processing apparatus, information processing method, and information processing program according to the present application are not limited to this embodiment. Moreover, each embodiment can be combined as appropriate within the range that does not conflict with the processing contents. Further, in each of the embodiments below, the same parts are given the same reference numerals, and redundant explanations will be omitted.

[1. An example of information processing]
First, an example of information processing according to the embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of information processing according to the embodiment. In the following, a case will be described in which the information processing device according to the embodiment is the terminal device 4 shown in Fig. 1, but the information processing device may be the information providing device 1 shown in Fig. 1, or may be a combination of the terminal device 4 and the information providing device 1.

The information providing device 1 according to the embodiment shown in FIG. 1 is, for example, an information processing device that provides a service such as SNS (Social Networking Service). For example, the information providing device 1 receives posted information posted by a poster. The information providing device 1 distributes the posted information received from the poster to the terminal device 4 of the user U who becomes the viewer.

The poster may be, for example, a hair salon, a hair practitioner, a person receiving a treatment, etc., and may include user U. A hair salon is a beauty parlor (beauty salon) or a barber shop (hairdressing parlor). A hair practitioner is a practitioner who styles hair at a hair salon, and is also called a hair stylist. A person receiving a treatment is a user who has received a hair treatment by a hair practitioner.

The posted information posted by the poster includes, for example, a posted image that is a captured image of the poster obtained by capturing an image of the poster, information indicating a posted text that is text posted by the poster, and the like. The posted image includes, for example, a captured image of the poster's head including the poster's hair and face.

The posted text also includes, for example, the poster's comments on the posted image. The poster's comments on the posted image include, for example, evaluation information indicating the poster's evaluation of the posted image. In the following, the poster's comments on the posted image may be referred to as poster comments.

The information providing device 1 also receives evaluation information indicating evaluations of other users (an example of viewers) of posted images included in the posted information and comments from other users on the posted information. Then, the information providing device 1 distributes the above-mentioned posted information including the received evaluation information of other users and comments of other users to the viewer's device. The viewer's device is, for example, the user U's terminal device 4 or another user's device. Below, comments from other users may be referred to as viewer comments.

The evaluation of the posted information may be, for example, a positive evaluation (e.g., "good") or a negative evaluation (e.g., "bad"), but may also include an intermediate evaluation (e.g., "average"). In addition, the information providing device 1 may distribute information such as the number of times the posted information has been added to the timeline by other users as an evaluation of the posted information to the terminal device 4 of the user U who is a viewer.

The information providing device 1 transmits information on multiple types of learning models to the terminal device 4 (step S1). The multiple types of learning models include a first learning model, a second learning model, a third learning model, and a fourth learning model.

First, the first learning model will be described. The first learning model is a model that has learned the relationship between keywords and hair images, and is a learning model that inputs one or more keywords and outputs hair images. The first learning model is generated by machine learning using learning data that includes multiple combinations of keywords and hair images.

The first learning model may be a learning model that inputs one or more keywords and outputs hair image identification information. The hair image identification information is information that identifies a hair image.

For example, the information providing device 1 extracts one or more keywords included in a poster's comment or text, and generates the above-mentioned data including a plurality of combinations of the one or more keywords and the posted image that is the subject of the comment. generated as training data.

In addition, when the information providing device 1 cannot extract keywords from the poster's comments or the poster's text, it extracts one or more keywords contained in the viewer's comments and generates data containing multiple combinations of the one or more keywords and the posted image that was the subject of the comment as the learning data described above.

Further, the information providing device 1 extracts one or more keywords included in both the poster comment or text and the viewer comment, and combines the one or more keywords with the posted image that is the subject of the comment. It is also possible to generate data including a plurality of . as the learning data described above.

The learning data may be data that includes multiple combinations of posted images and keywords assigned by an operator for each posted image. Furthermore, instead of content such as posted information, the learning data may be images included in content such as web pages, or images provided by a third party. Posted information is an example of posted content.

For example, the information providing device 1 can obtain content that includes text and images of people's faces and hair via a network, and extract keywords from the text included in the content. In this case, the information providing device 1 generates data as learning data that includes multiple combinations of keywords extracted from the text included in the content and images included in the content.

Next, the second learning model will be explained. The second learning model is a learning model that has learned the relationship between a face image and information indicating facial features, and is a learning model that takes an image including a facial image as input and outputs information indicating facial features. .

The second learning model is generated by machine learning using learning data including a plurality of combinations of images including facial images and information indicating facial features. Information indicating facial features is, for example, information indicating the type of face shape, and information indicating the size, shape, and position of one or more of the eyebrows, eyes, nose, mouth, and ears on the face. It may also be information indicating characteristics.

In addition, the information indicating facial features may be information indicating the positions of the eyebrows, eyes, nose, mouth, and ears on the face, or information indicating the positions and sizes of the eyebrows, eyes, nose, mouth, and ears on the face.

Next, the third learning model will be explained. The third learning model is a model that has learned the relationship between information indicating facial features and hairstyles that have been evaluated as suitable for the face, and is estimated to be suitable for the face using the information indicating facial characteristics as input. This is a learning model whose output is information indicating the hairstyle to be used.

The third learning model is generated by machine learning using learning data that includes multiple combinations of information indicating facial features, hair images, and information indicating evaluations of combinations of information indicating facial features and hair images.

The information providing device 1 collects a plurality of pieces of posted information each including posted images, viewer comments, evaluation information, and the like. The facial image includes a facial image and a hair image, and the evaluation information is information indicating the evaluation of the posted image by other users who are viewers, and includes information indicating the evaluation included in the poster's comment. Good too.

The information providing device 1 extracts facial features indicated by face images included in each piece of collected post information. The information providing device 1 then generates data including a hair image included in the posted image of the post information, information indicating the extracted facial features, and evaluation information as the learning data described above.

In the third learning model, the information indicating facial features is, for example, information indicating the type of face shape, and the size and shape of one or more of the eyebrows, eyes, nose, mouth, and ears on the face. , may be information indicating the characteristics of each part, such as its position. Further, the information indicating the facial features may be information indicating the positions of the eyebrows, eyes, nose, mouth, and ears on the face. It may also be information indicating the position and size.

Next, the fourth learning model will be explained. The fourth learning model is a learning model that has learned the relationship between the hair images before and after the treatment and the treatment content, and is a learning model that uses the hair images before and after the treatment as input, and outputs information indicating the estimated treatment content. be. The hair images before and after the hair treatment include a hair image before the hair treatment and a hair image after the hair treatment.

The details of the treatment include, for example, whether or not to cut, the amount of cut, whether or not to bleach, the number of times to bleach, whether or not to color, the type of color, whether or not to perm, the type of perm, whether or not to straighten the hair, etc. Information indicating the details of the treatment includes score information for each of whether or not to cut, the amount of cut, whether or not to bleach, the number of times to bleach, whether or not to color, the type of color, whether or not to perm, the type of perm, and whether or not to straighten the hair.

In addition, the information indicating the treatment contents may include score information for each combination of whether or not a cut was performed, the amount of cut, whether or not bleaching was performed, the number of times bleaching was performed, whether or not coloring was performed, the type of coloring, whether or not perming was performed, the type of perming, and whether or not hair straightening was performed.

The fourth learning model is generated by machine learning using learning data including multiple combinations of hair images before and after a treatment and information indicating the treatment content. The information providing device 1 collects treatment information including hair images before and after a treatment and information indicating the treatment content from, for example, a device of a hair salon or a hair stylist. Then, the information providing device 1 generates learning data including multiple combinations of hair images before and after a treatment and information indicating the treatment content based on the collected treatment information.

Any type of model can be employed as each of the learning models described above. For example, each learning model is a model generated by machine learning, such as a DNN (Deep Neural Network), a GBDT (Gradient Boosting Decision Tree), or an SVM (Support Vector Machine). The DNN is, for example, a CNN (Convolutional Neural Network) or an RNN (Recurrent Neural Network). Further, the RNN may be an LSTM (Long short-term memory) or the like. Further, each learning model may be a model realized by combining a plurality of models, such as a model combining CNN and RNN.

Next, the terminal device 4 receives information on the multiple types of learning models transmitted from the information providing device 1, and stores the received information on the multiple types of learning models in an internal memory unit (step S2).

A user U of the terminal device 4 may consider what hairstyle to have before making a reservation for a hair treatment at a hair salon. The user U may consider a hairstyle from, for example, information found in a web search or information published in a magazine.

For example, the user U can cause the terminal device 4 to display hairstyle-related images obtained through a keyword search, web search, or the like. The hairstyle-related image is an image that allows the user U to confirm the hairstyle, and is an image that includes at least a hair image and a face image.

Here, we will explain keyword search. The terminal device 4 displays multiple keywords on the display unit (step S3). These multiple keywords are multiple predetermined keywords, but they may also be multiple keywords randomly extracted from the multiple keywords or multiple keywords that satisfy predetermined conditions.

The multiple keywords that satisfy the predetermined conditions are, for example, a predetermined number of keywords that are most frequently selected by user U. The keywords that are candidates for extraction are, for example, keywords that are to be input to the first learning model.

When one or more keywords are specified by the user U, the terminal device 4 receives the one or more keywords specified by the user U, and stores the hair images corresponding to the one or more keywords using the first learning model. and specify it (step S4).

In step S4, the terminal device 4 inputs one or more keywords specified by the user U into the first learning model, and inputs one or more keywords specified by the user U based on information output from the first learning model. Hair images corresponding to the above keywords are identified.

For example, when the first learning model outputs a hair image, the terminal device 4 identifies the hair image output from the first learning model as a hair image corresponding to one or more keywords specified by the user U. do.

In addition, when the first learning model outputs hair image specific information, the terminal device 4 acquires the hair image specific information output from the first learning model, and stores the hair image specified by the acquired hair image specific information. The image is acquired from the internal storage unit or the information providing device 1. In this case as well, the terminal device 4 identifies the acquired hair image as a hair image corresponding to one or more keywords specified by the user U.

Then, the terminal device 4 acquires a plurality of similar hair images, which are images including hair images similar to the hair image identified in step S4, and displays the acquired similar hair images on the display unit (step S5). In step S5, the terminal device 4 transmits a similar image search request including the hair image identified in step S4 to a similar image search server or information providing device 1 (not shown).

The similar image search server or information providing device 1 searches for multiple similar hair images based on the hair image included in the similar image search request from the terminal device 4, and transmits one or more similar hair images obtained by the search to the terminal device 4. The terminal device 4 acquires the multiple similar hair images transmitted from the similar image search server or information providing device 1 in response to the similar image search request.

In this way, the terminal device 4 displays multiple similar hair images on the display unit based on one or more keywords specified by the user U. This allows the user U to check the hair images to be considered simply by specifying a keyword.

In the above example, the user U designates one or more keywords from among the multiple keywords presented by the terminal device 4, but the designation of one or more keywords by the user U is not limited to this example. For example, the user U can designate one or more keywords by operating the terminal device 4 to input one or more keywords into the terminal device 4, or can designate one or more keywords by inputting one or more keywords into the terminal device 4 by voice.

In addition to or instead of a keyword search, user U can also consider hairstyles based on user U's facial features. Consideration of hairstyles based on user U's facial features will be described below.

The terminal device 4 determines the facial features of the user U based on the user-captured image specified by the user U (step S6). The user-captured image is an image of the head including the user U's hair and face, captured by an imaging unit (not shown) of the terminal device 4 or an imaging unit of a device (not shown), and is an example of a user image. The terminal device 4 acquires the user-captured image based on, for example, an operation by the user U.

For example, the user U can operate the terminal device 4 to capture an image of the user U's head, including the hair and face, with the imaging unit of the terminal device 4, and have the terminal device 4 acquire multiple user-captured images. The user U can operate the terminal device 4 to specify a desired user-captured image from the multiple user-captured images acquired by the terminal device 4.

In step S6, the terminal device 4 determines the facial features of the user U shown in the user captured image, for example, from among a plurality of predetermined types of facial features. For example, the terminal device 4 inputs the user captured image to the second learning model described above, and determines the facial features of the user U based on the information indicating the facial features output from the second learning model. .

Further, the terminal device 4 may have information indicating each of a plurality of types of facial features. In this case, the terminal device 4 may have information indicating each type of facial feature and a user-captured image. The facial features of the user U are determined by pattern matching technology with information indicating the facial features of the user U.

The terminal device 4 can also display facial feature line drawings, which are line drawings showing multiple types of facial features (e.g., face shapes), on the display unit, and allow the user U to select the facial feature line drawing that is closest to his or her own face shape from among these multiple types of facial feature line drawings.

Next, the terminal device 4 uses the third learning model described above to identify a hairstyle that is estimated to suit the face of the user U from the facial features of the user U determined in step S6 (step S7). As described above, the third learning model is a model that learns the relationship between information indicating the facial features and a hairstyle that is evaluated as suiting the face.

In step S7, the terminal device 4 inputs the information indicating the facial features of the user U determined in step S6 to the third learning model, and based on the hairstyle information output from the third learning model, A hairstyle that is estimated to suit user U's face is identified. The information on the hairstyle is a hair image, and the terminal device 4 identifies the hair image output from the third learning model as a hairstyle that is estimated to suit the face of the user U.

Subsequently, the terminal device 4 generates a composite image by combining the user face image that is the face image of the user U and the hair image selected by the user U, and displays the generated composite image on the display unit (step S8 ). The user face image is, for example, the above-mentioned user captured image or an image included in the user captured image.

The hair image selected by the user U is, for example, one or more images selected by the user U from among the hairstyle images specified in step S8 or the plurality of similar hair images obtained by keyword search.

The terminal device 4 accepts the selection of one or more hair images by the user U from among the specific hair image, which is an image of the hairstyle specified in step S8, or a plurality of similar hair images obtained by keyword search. Further, the terminal device 4 can also accept a selection of one or more hair images by the user U from among a plurality of hair images included in a plurality of web pages viewed by the user U, for example.

When two or more hair images are selected by the user U, the terminal device 4 generates a composite image by combining the composite hair image obtained by combining the two or more hair images with the user's face image.

The hair image selected by user U may be, for example, a captured image captured by an imaging unit (not shown) in terminal device 4. Such a captured image is, for example, an image captured from a photo of a person's face and hair published in a magazine. In this case, terminal device 4 generates a composite image by, for example, replacing the facial image of the person included in the captured image selected by user U of the captured image captured by terminal device 4 with the facial image of user U included in the user facial image.

In addition, in a captured image taken from a photo published in a magazine, the face of a person included in the captured image may not be facing forward. In this case, the terminal device 4 generates a composite image by replacing the facial image included in the captured image obtained by projectively transforming the captured image in a direction in which the person's face faces forward with the user's facial image. In addition, when multiple captured images are selected by the user U, the terminal device 4 generates a composite image by replacing the facial image included in the composite captured image obtained by combining the multiple captured images with the user's facial image.

Furthermore, when the hair image selected by the user U from a plurality of hair images is changed based on the user's operation, the terminal device 4 combines the hair image with the hair image whose selection was changed by the user's U operation. A modified composite image is generated as a composite image, and the generated modified composite image is displayed on a display unit.

For example, each time a hair image selected by the user U from among the multiple hair images is changed based on a first change operation by the user U, the terminal device 4 generates a corresponding changed composite image and displays the generated changed composite image on the display unit. The first change operation is, for example, a scroll operation in the left/right direction or a swipe operation in the left/right direction.

Further, the terminal device 4 can also change at least one of the hair length and color of the hair image included in the composite image based on the user U's second changing operation. For example, the terminal device 4 changes at least one of the length and color of the hair of the hair image included in the composite image every time the second changing operation is performed. The first change operation is, for example, a scroll operation in the up and down direction or a swipe operation in the up and down direction.

Moreover, when the terminal device 4 receives the processing of the hair image selected by the user U, the terminal device 4 can also generate a processed hair image by performing the accepted processing on the hair image. The hair image processing is, for example, processing that changes at least one of the length and color of the hair in the hair image. In this case, in step S8, the terminal device 4 generates a composite image by combining the user face image and the processed hair image, and displays the generated composite image on the display unit.

Next, the terminal device 4 uses the fourth learning model described above to estimate the treatment contents from the pre-treatment image of the user U and the composite image generated in step S8 (step S9). The pre-treatment image of the user U is, for example, the user-captured image described above. The treatment contents estimated in step S9 are the treatment contents necessary to achieve the hairstyle shown in the composite image generated in step S8, and may be referred to as estimated treatment contents below.

In step S9, the terminal device 4 indicates the treatment details output from the fourth learning model when the user U's pre-treatment image and the composite image generated in step S8 are input to the fourth learning model. Based on the information, the estimated treatment details are determined.

Furthermore, the terminal device 4 determines the treatment cost, which is the cost required for the treatment with the estimated treatment content, based on the determined estimated treatment content (step S10). For example, the terminal device 4 can determine the treatment cost for each hair salon based on treatment cost information for each hair salon that indicates the relationship between treatment details and treatment costs. The terminal device 4 can acquire treatment cost information for each hair salon from the information providing device 1, for example.

When the terminal device 4 receives a treatment content designated by the user U, it compares the user-specified treatment content, which is the treatment content that has been received, with the estimated treatment content, and generates comparison information indicating the comparison result (step S11). The comparison information includes, for example, information indicating the difference between the user-specified treatment content and the estimated treatment content. For example, when the estimated treatment content includes, in addition to the user-specified treatment content, one more bleach, the terminal device 4 generates comparison information including information indicating that the number of bleaches is increased by one compared to the user-specified treatment content.

Subsequently, the terminal device 4 displays treatment-related information including information indicating the treatment content estimated in step S9, treatment cost information that is information on the treatment cost determined in step S10, and comparison information generated in step S11. (Step S12). As a result, the user U can grasp the treatment details and treatment costs estimated to be necessary to achieve the hairstyle shown in the selected hair image, and the terminal device 4 can provide user U with convenience. can be improved.

Then, the terminal device 4 receives an acceptance/rejection response, which is a response indicating whether the user U accepts or rejects the treatment content estimated in step S9 (step S13). If the treatment content estimated in step S9 is acceptable, the user U operates the terminal device 4 to respond indicating acceptance, and if the treatment content estimated in step S9 is not acceptable, the user U operates the terminal device 4 to respond indicating rejection.

Next, if the accepted permission/denial response is a response indicating approval, the terminal device 4 outputs a reservation request including the specific image and treatment content information to the hair practitioner (step S14). The specific image includes, for example, the composite image generated in step S8 or a hair image selected by the user U, and a user-captured image including a hair image of the user U before the treatment. The treatment content information includes at least one of information indicating the treatment content designated and accepted by the user U and information indicating the treatment content estimated in step S9.

The above-mentioned reservation request is outputted to the hair practitioner's store device 5 via the information providing device 1, for example, but is not outputted to the hair practitioner's store device 5 via SNS or a mail server (not shown). Good too. The hair practitioner's store device 5 receives the reservation request output from the terminal device 4 and displays the specific image and treatment content information included in the received reservation request. Thereby, the hair practitioner can confirm the specific image and treatment content information included in the reservation request.

The hair practitioner can operate the store device 5 to modify the specific image included in the reservation request or the treatment content indicated by the treatment content information. In this case, the store device 5 outputs to the user U modification information including a modified specific image that is a modified specific image, modified treatment content information that is information indicating the modified treatment content, and the like. The modification information is output to the terminal device 4 via the information providing device 1, for example, but may also be output to the terminal device 4 via an SNS or a mail server (not shown).

The hair practitioner can operate the store device 5 to input a comment to the store device 5 as to whether or not the hairstyle shown in the composite image included in the reservation request can be performed. In this case, the store device 5 outputs a practitioner comment, which is a comment as to whether or not the hairstyle shown in the composite image can be performed, to the user U. The comment is output to the terminal device 4 via the information providing device 1, for example, but may also be output to the terminal device 4 via SNS or a mail server (not shown).

In addition, if there is no need to modify the specific image included in the reservation request or the treatment content indicated by the treatment content information included in the reservation request, the hair practitioner operates the in-store device 5 to output reservation acceptance information from the in-store device 5 to the user U, indicating that the reservation has been accepted.

When the terminal device 4 acquires the correction information and the practitioner's comments from the in-store device 5, it displays the acquired correction information and the practitioner's comments on the display unit (step S15). This allows the user U to check the correction suggestions and comments made by the hair practitioner, making it easy to communicate with the hair practitioner.

As described above, terminal device 4, which is an example of an information processing device, generates a composite image by combining a user face image, which is a facial image of user U, with a hair image selected by user U, and outputs the generated composite image to the hair practitioner. This allows user U to present a composite image by combining a desired hair image with the user face image to the hair practitioner, and allows the desired hairstyle to be specifically communicated to the hair practitioner, compared to a case where the desired hairstyle is vaguely communicated to the hair practitioner. Therefore, terminal device 4 can improve the convenience of user U.

Furthermore, terminal device 4, which is an example of an information processing device, generates a composite image by combining a user face image, which is a facial image of user U, with a hair image selected by user U, and estimates the treatment content from the pre-treatment image of user U and the composite image using a learning model that has learned the relationship between the hair images before and after the treatment and the treatment content. This allows user U to, for example, understand the treatment content to achieve a desired hairstyle, and also to specifically communicate the treatment content to the hair practitioner to achieve the desired hairstyle. Therefore, terminal device 4 can improve the convenience for user U.

Further, the terminal device 4, which is an example of an information processing device, accepts the specification of one or more keywords by the user U, and uses a learning model that receives the one or more keywords as input and outputs a hair image. A hair image is identified from one or more keywords that are accepted. Then, the terminal device 4 acquires a similar hair image, which is an image including a hair image similar to the specified hair image. Thereby, the user U can easily collect hair images corresponding to a desired hairstyle simply by specifying a keyword. Therefore, the terminal device 4 can improve convenience for the user U.

Furthermore, terminal device 4, which is an example of an information processing device, identifies a hairstyle that is estimated to suit user U's face using a learning model that is a model that has learned the relationship between information indicating facial features and hairstyles that are evaluated to suit the face, and acquires posted images including a specific hair image that is an image of the identified hairstyle. Then, based on the posted images acquired by the acquisition unit and the facial image of user U, terminal device 4 generates a composite image by combining the image of the hairstyle identified by the identification unit and the facial image of user U. This allows user U to easily acquire an image of what it would be like if the hairstyle estimated to suit user U's face was applied to himself or herself. Therefore, terminal device 4 can improve the convenience of user U.

Furthermore, in the above-described example, various processes are performed by the terminal device 4, but some or all of the above-described processes of the terminal device 4 may be performed mainly by the information providing device 1. For example, the information providing device 1 provides the terminal device 4 with information generated based on information input from the terminal device 4 via an interface such as an API (Application Programming Interface), and the terminal device 4 The above-mentioned information may be displayed based on information provided by the user. Further, although FIG. 1 shows a case where the terminal device 4 and the information providing device 1 are separate devices, the terminal device 4 and the information providing device 1 may be integrated as an information processing device.

The following provides a detailed explanation of the configuration of the information processing system that includes the information providing device 1, terminal device 4, and store device 5 that perform this processing.

[2. Information processing system configuration]
FIG. 2 is a diagram illustrating an example of the configuration of the information processing system according to the embodiment. As shown in FIG. 2, the information processing system 100 according to the embodiment includes an information providing device 1, a plurality of terminal devices 4, and a plurality of store devices 5.

Each of the multiple terminal devices 4 is used by a different user U. Each of the multiple in-store devices 5 is installed, for example, in a different hair salon and used by hair practitioners at the different hair salons.

The information providing device 1, the terminal device 4, and the store device 5 are connected to each other via a network N so that they can communicate with each other by wire or wirelessly. Note that the information processing system 100 shown in FIG. 2 may include multiple information providing devices 1. The network N is, for example, a local area network (LAN) or a wide area network (WAN) such as the Internet.

The information providing device 1 is an information processing device that provides various types of information. The information providing device 1 provides online services through SNS sites, web search sites, content distribution sites, and the like. The SNS site accepts posted information sent from the terminal device 4, the store device 5, and the like, and transmits the posted information to the terminal device 4 or the store device 5 based on a request from the user U, a hair practitioner, and the like. The information providing device 1 may also be an information processing device that acquires information such as posted information from the above-mentioned SNS sites, and the like.

Further, content distribution sites are sites that provide content distribution services, and include video/music distribution sites, map sites, route search sites, route guidance sites, route information sites, operation information sites, weather forecast sites, and the like. Further, the information providing device 1 can also provide online services through various online sites such as, for example, an electronic payment site, an online game site, an online banking site, or an accommodation/ticket reservation site.

Each of the terminal device 4 and the store device 5 is, for example, a desktop PC (Personal Computer), a notebook PC, a tablet terminal, a smartphone, a mobile phone, or a PDA (Personal Digital Assistant). Note that each of the terminal device 4 and the store device 5 is not limited to the above example, and may be, for example, a smart watch or a wearable device.

In addition, each terminal device 4 can connect to the network N via a wireless communication network such as LTE (Long Term Evolution), 4G (4th Generation), or 5G (5th Generation: 5th generation mobile communication system), or via short-range wireless communication such as Bluetooth (registered trademark) or wireless LAN, and communicate with the information providing device 1 and each store device 5.

In addition, each store device 5 can connect to the network N via a wireless communication network such as LTE, 4G, or 5G, or a short-range wireless communication such as Bluetooth or wireless LAN, and communicate with the information providing device 1 and each terminal device 4.

3. Configuration of information providing device 1
Next, a configuration example of the information providing device 1 will be described with reference to Fig. 3. Fig. 3 is a diagram showing a configuration example of the information providing device 1 according to an embodiment. As shown in Fig. 3, the information providing device 1 has a communication unit 10, a storage unit 11, and a processing unit 12.

[3.1. Communication department 10]
The communication unit 10 is realized by, for example, a NIC (Network Interface Card). The communication unit 10 is connected to a network by wire or wirelessly, and transmits and receives information to and from the terminal device 4, store device 5, etc. via the network N.

[3.2. Storage unit 11
The storage unit 11 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. , a posted information storage unit 21, a model information storage unit 22, and a content storage unit 23.

[3.2.1. User information storage unit 20]
The user information storage unit 20 stores information regarding the user U. FIG. 4 is a diagram showing an example of a user information table stored in the user information storage unit 20 of the information providing device 1 according to the embodiment.

As shown in FIG. 4, the user information table stored in the user information storage unit 20 includes items such as "User ID (IDentifier)," "Attribute information," and "History information." "User ID" is identification information that identifies user U.

"Attribute information" is information regarding user attributes that are attributes of the user U corresponding to the "user ID", and includes information on psychographic attributes, information on demographic attributes, and the like. Demographic attributes include, for example, gender, age, place of residence, and occupation, and psychographic attributes include objects of interest such as travel, clothes, cars, and religion, lifestyle, thoughts, and tendencies of thought.

"History information" is history information that includes information such as user U's service usage history corresponding to the "user ID", and includes, for example, user U's posting history information, user U's search history information, and user U's Contains browsing history information, etc. User U's posting history information includes information posted by user U to SNS sites, comments by user U on posted information by other users U, information indicating user U's evaluation of posted information by other users U, etc. included.

The search history information of the user U is, for example, information on the search history of posted information on SNS sites, information on the search history on various websites, and the like. The browsing history information of the user U is, for example, information such as information on the browsing history by the user U of posted information on SNS sites, information on browsing history on various websites, and the like.

3.2.2. Posted information storage unit 21
The posted information storage unit 21 stores various types of information posted by each user U. Fig. 5 is a diagram showing an example of a posted information table stored in the posted information storage unit 21 of the information providing device 1 according to the embodiment. As shown in Fig. 5, the posted information table stored in the posted information storage unit 21 includes items such as "posting ID", "posting date and time", "posting user", "posted text", "posted image", "rating information", "number of views", and "comments".

“Post ID” is identification information that identifies posted information. Posted information is an example of posted content. “Posting date and time” is information indicating the posting date and time of the posted information corresponding to the “posting ID”. The "posting user" is information indicating the user U who posted the posted information corresponding to the "posting ID", and in the example shown in FIG. 5, is the user ID. The "posted text" is information indicating text included in the posted information corresponding to the "posted ID", and is represented by a character string, but may also include a stamp or the like.

A "posted image" is a posted image included in the posted information corresponding to a "post ID." "Evaluation information" is evaluation information indicating the evaluation of other users U on the posted information corresponding to a "post ID," but may also include information indicating the poster's evaluation of the posted information corresponding to a "post ID." "Number of views" is information indicating the number of times the posted information corresponding to a "post ID" has been viewed.

A "comment" is a comment from another user U on the posted information corresponding to the "posting ID", and is indicated by, for example, a character string, but may also include stamps, etc. Note that a "comment" may also include a comment from the posting user on the posted information corresponding to the "posting ID".

The evaluation information shown in FIG. 5 is information indicating a positive evaluation (e.g., "good") or a negative evaluation (e.g., "bad"), but may also include information indicating an intermediate evaluation (e.g., "average").

[3.2.3. Model information storage unit 22]
The model information storage unit 22 stores information on various learning models. Fig. 6 is a diagram showing an example of a model information table stored in the model information storage unit 22 of the information providing device 1 according to the embodiment.

As shown in FIG. 6, the model information table stored in the model information storage unit 22 includes items such as "model ID" and "model information." "Model ID" is identification information that identifies a learning model. "Model information" is information on a learning model that corresponds to the "model ID." The information on a learning model is information such as parameters of the learning model, but is not limited to such an example and may be information for using the learning model in the terminal device 4.

The learning models whose information is included in the model information table include, for example, the above-described first learning model, second learning model, third learning model, and fourth learning model. The first learning model is a model that has learned the relationship between keywords and hair images, and is a learning model that receives one or more keywords as input and outputs hair images or hair image specific information.

The second learning model is a learning model that has learned the relationship between a face image and facial features, and is a learning model that takes an image including a face image as input and outputs information indicating the facial features. The third learning model is a model that has learned the relationship between information indicating facial features and hairstyles that have been evaluated as suitable for the face, and is estimated to be suitable for the face using the information indicating facial characteristics as input. This is a learning model that outputs information indicating hairstyles.

The fourth learning model is a learning model that has learned the relationship between the hair images before and after the treatment and the treatment content, and is a learning model that uses the hair images before and after the treatment as input, and outputs information indicating the estimated treatment content. be. Note that the learning models whose information is included in the model information table are not limited to the example described above.

Each of the above-mentioned learning models can be any type of model. For example, each learning model is a model generated by machine learning, such as DNN, GBDT, SVM, etc. The DNN is, for example, CNN or RNN. The RNN may be, for example, LSTM. Each learning model may also be a model realized by combining multiple models, such as a model that combines CNN and RNN.

[3.2.4. Content storage unit 23]
The content storage unit 23 contains information on content. FIG. 7 is a diagram showing an example of a content table stored in the content storage unit 23 of the information providing device 1 according to the embodiment. The content table shown in FIG. 7 includes items such as "content ID" and "content".

"Content ID" is an identifier that identifies content. “Content” is information regarding the content associated with “Content ID”. Specifically, the information regarding the content is, for example, information regarding the contents of the content or information about the address of the content.

In the example shown in FIG. 7, the content with content ID "C1" is content CNT1, the content with content ID "C2" is content CNT2, and the content with content ID "C3" is content CNT3. Note that the content storage unit 23 is not limited to the above, and may store various types of information depending on the purpose.

[3.3. Processing unit 12]
The processing unit 12 is a controller, and for example, processes various programs (information processing) stored in a storage device inside the information providing device 1 by a processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). (corresponding to an example of a program) is executed using a RAM or the like as a work area.

Further, the processing unit 12 is a controller, and may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a GPGPU (General Purpose Graphic Processing Unit). .

As shown in FIG. 3, the processing section 12 includes an acquisition section 30, a learning section 31, and a providing section 32, and realizes or executes information processing functions and operations described below. Note that the internal configuration of the processing unit 12 is not limited to the configuration shown in FIG. 3, and may be any other configuration as long as it performs information processing to be described later.

[3.3.1. Acquisition unit 30
The acquisition unit 30 acquires various information. The acquisition unit 30 acquires various information from the storage unit 11. The acquisition unit 30 acquires various information from the user information storage unit 20, the posted information storage unit 21, the model information storage unit 22, Also, various information is acquired from the content storage unit 23 and the like.

The acquisition unit 30 also acquires information about the user U (e.g., attribute information and behavioral history information about the user U) from, for example, the terminal device 4, the store device 5, or an external device, and stores the acquired information about the user U in the user information storage unit 20. The acquisition unit 30 also acquires various types of content from, for example, an external device, and stores the acquired content in the content storage unit 23.

The acquisition unit 30 also acquires posted information from a device such as the terminal device 4 or the store device 5 via the communication unit 10. The acquisition unit 30 adds the acquired posted information to the posted information table stored in the posted information storage unit 21.

The acquisition unit 30 also acquires an acquisition request for posted information from the terminal device 4 via the communication unit 10. For example, the acquisition unit 30 notifies the provision unit 32 of information related to the acquired acquisition request.

The acquisition unit 30 also acquires various information such as a reservation request from the terminal device 4 via the communication unit 10, stores various information such as the acquired reservation request in the storage unit 11, or stores the acquired reservation request and other various information in the storage unit 11. It notifies the providing unit 32 of various information such as requests. The acquisition unit 30 acquires various types of information such as modification information from the store device 5 via the communication unit 10 and notifies the provision unit 32 of various types of information such as the acquired modification information.

The acquisition unit 30 also acquires learning data for generating the above-mentioned learning model, and notifies the acquired learning data to the learning unit 31. For example, the acquisition unit 30 acquires information stored in the memory unit 11 from the memory unit 11, and notifies the acquired information to the learning unit 31 as learning data.

The learning data for the first learning model is, for example, learning data that includes multiple combinations of keywords and hair images. Also, the learning data for the second learning model is, for example, learning data that includes multiple combinations of images including facial images and information indicating facial features.

The learning data for the third learning model includes, for example, a plurality of combinations of information indicating facial features, hair images, and information indicating evaluations of combinations of information indicating facial features and hair images. This is data for use. The evaluation of the combination of the information indicating facial features and the hair image is, for example, information indicating a positive evaluation or information indicating a negative evaluation, and is used as a label. The learning data for the fourth learning model is, for example, learning data that includes a plurality of combinations of hair images before and after the treatment and information indicating the treatment details.

The acquisition unit 30 can also generate the above-mentioned learning data based on the information stored in the storage unit 11. The acquisition unit 30 can also acquire or generate learning data other than the above-mentioned.

3.3.2. Learning unit 31
The learning unit 31 generates various learning models through machine learning using the various learning data notified by the acquisition unit 30, and stores information on the various learning models generated in the posted information storage unit 21.

For example, the learning unit 31 generates a first learning model using the learning data for the first learning model, and generates a second learning model using the learning data for the second learning model. In addition, the learning unit 31 generates a third learning model using the learning data for the third learning model, and generates a fourth learning model using the learning data for the fourth learning model.

Note that the learning unit 31 can also generate learning models other than the first learning model, the second learning model, the third learning model, and the fourth learning model.

[3.3.3. Providing unit 32
The providing unit 32 starts the communication unit 10 and provides various contents to the terminal device 4 of the user U and the shop device 5 of the hair salon. For example, the providing unit 32 transmits the following contents to the terminal device 4 and the shop device 5: Based on the request, the browser generates browsing content including the SNS posted information, and provides the browsing content to the terminal device 4 and the store device 5 via the communication unit 10 and the network N, etc.

The providing unit 32 also provides various pieces of information notified by the acquiring unit 30 to the terminal device 4 or the store device 5, etc., via the communication unit 10 and the network N, etc. For example, the providing unit 32 provides information from the terminal device 4 notified by the acquiring unit 30 to the store device 5, etc., via the communication unit 10 and the network N, etc., and provides information from the store device 5 notified by the acquiring unit 30 to the terminal device 4, etc., via the communication unit 10 and the network N, etc.

Further, the providing unit 32 provides information on various learning models generated by the learning unit 31 to the terminal device 4 via the communication unit 10, the network N, and the like. For example, the providing unit 32 transmits the first learning model, second learning model, third learning model, and fourth learning model generated by the learning unit 31 to the communication unit 10, the network N, etc. Provided via.

Further, the providing unit 32 provides an application program (hereinafter referred to as an application) used in the terminal device 4 to the terminal device 4 via the communication unit 10, the network N, and the like. For example, such an application is installed on the terminal device 4 and executes the above-described processing on the terminal device 4. For example, with the functions of such an application, the terminal device 4 can generate the above-mentioned composite image or acquire various information using the above-mentioned learning model.

[4. Terminal device 4]
Fig. 8 is a diagram showing an example of the configuration of the terminal device 4 according to the embodiment. As shown in Fig. 8, the terminal device 4 according to the embodiment includes a communication unit 40, a display unit 41, an operation unit 42, an imaging unit 43, a sensor unit 44, a storage unit 45, and a processing unit 46.

4.1. Communication unit 40
The communication unit 40 is realized by, for example, a NIC. The communication unit 40 is connected to the network N by wire or wirelessly, and transmits and receives information between the information providing device 1, the store device 5, and the like via the network N. The communication with the store device 5 is performed, for example, via the information providing device 1, an SNS, or a mail server (not shown), but the communication with the store device 5 may be performed directly without going through these.

[4.2. Display unit 41
The display unit 41 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display.

[4.3. Operation unit 42
The operation unit 42 includes, for example, a keyboard including keys for inputting letters, numbers, and spaces, an enter key, arrow keys, and the like, a mouse, a power button, and the like. If the display unit 41 is a touch panel compatible display, The operation unit 42 includes a touch panel.

[4.4. Imaging unit 43]
The imaging unit 43 is an image sensor (camera) that photographs a subject. For example, the imaging unit 43 is a CMOS (Complementary Metal Oxide Semiconductor) image sensor, a CCD (Charge-Coupled Device) image sensor, or the like. Note that the imaging unit 43 is not limited to a built-in camera, and may be an external camera such as a wireless camera that can communicate with the terminal device 4 or a web camera.

4.5. Sensor unit 44
The sensor unit 44 includes a position detection unit, a gyro sensor, etc. The position detection unit detects, for example, the position of the terminal device 4, which is the current position of the user U, and outputs information indicating the detected current position of the user U to the processing unit 46.

The position detection unit receives a plurality of positioning signals transmitted from a plurality of positioning satellites in a GNSS (Global Navigation Satellite System), and detects the current position of the user U based on the plurality of received positioning signals. The gyro sensor is a sensor that detects postures such as tilt and rotation of the terminal device 4.

[4.6. Storage unit 45]
The storage unit 45 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk.

The storage unit 45 stores, for example, information transmitted from the information providing device 1, the store device 5, etc. and acquired by the processing unit 46 via the network N and the communication unit 40, captured images that are images captured by the imaging unit 43, and detection information that is information detected by the sensor unit 44. For example, the storage unit 45 stores information on apps and various learning models transmitted from the information providing device 1.

4.7. Processing Unit 46
The processing unit 46 is a controller, and is realized, for example, by a CPU or an MPU, etc., executing various programs (corresponding to an example of an information processing program) stored in a storage device inside the terminal device 4 using a RAM as a working area. The processing unit 46 may be realized in part or in whole by an integrated circuit, for example, an ASIC or an FPGA.

The processing unit 46 includes a reception unit 50, an acquisition unit 51, an identification unit 52, an image processing unit 53, an estimation unit 54, a determination unit 55, a comparison unit 56, and an output unit 57. The processing unit 46 functions as a functional unit including the reception unit 50, the acquisition unit 51, the identification unit 52, the image processing unit 53, the estimation unit 54, the determination unit 55, the comparison unit 56, and the output unit 57, for example, by executing the above-mentioned application on the OS (Operating System).

Note that a plurality of functions including the reception section 50, the acquisition section 51, the identification section 52, the image processing section 53, the estimation section 54, the determination section 55, the comparison section 56, and the output section 57 are pre-installed in the processing section 46. It's okay.

4.7.1. Reception unit 50
The reception unit 50 receives various requests and information. The reception unit 50 receives requests and information transmitted from the terminal device 4 and the in-store device 5.

For example, the reception unit 50 receives correction information transmitted from the in-store device 5. The correction information is information indicating correction suggestions made by the hair practitioner. The reception unit 50 also receives reservation reception information transmitted from the in-store device 5. The reservation reception information includes information indicating that a reservation for the treatment content, etc. has been accepted in the reservation request.

The reception unit 50 also receives various operations by the user U using the operation unit 42, etc. For example, the reception unit 50 receives various operations such as the selection of a hair image by the user U, processing of a hair image by the user U, specification of treatment details by the user U, approval or disapproval of the estimated treatment details by the user U, and specification of treatment details by the user U.

The reception unit 50 also receives an imaging operation by the user U. When accepting the imaging operation by the user U, the reception unit 50 operates the imaging unit 43 to cause the imaging unit 43 to perform the imaging operation.

The receiving unit 50 also receives a first hair image changing operation by the user U, a second hair image changing operation by the user U, and the like. The first modification operation is an operation for modifying the composite image by modifying the hair image for generating the composite image. The second changing operation is an operation for changing at least one of the length and color of the hair in the hair image.

FIG. 9 is a diagram illustrating an example of the configuration of the reception unit 50 of the terminal device 4 according to the embodiment. As shown in FIG. 9, the reception section 50 includes a selection reception section 60, a processing reception section 61, a specification reception section 62, a modification proposal reception section 63, and a permission acceptance section 64.

The selection accepting unit 60 accepts various selections made by the user U. For example, the selection accepting unit 60 accepts the selection of a hair image by the user U from among multiple types of hair images. The user U can perform a hair image selection operation by operating the operation unit 42, and the selection reception unit 60 accepts the hair image selection based on the hair image selection operation.

The hair image selected by the user U is included in the hairstyle image specified by the specifying unit 52, the plurality of similar hair images obtained by the obtaining unit 51, and the captured image captured by the imaging unit 43, for example. One or more hair images, such as hair images.

Note that the hair image selected by user U is not limited to the example described above. For example, the hair images selected by the user U may include the hair images selected by the processing unit 46 when the automatic hair image selection mode by the processing unit 46 is selected by the user U's operation.

The processing reception unit 61 accepts processing of the hair image. For example, the processing reception unit 61 accepts a selection of at least one of the hair length and color of the hair image. The hair in the hair image is the hair shown in the hair image, and the user U can perform processing operations on the hair image by operating the operation unit 42, and the processing reception unit 61 accepts processing of the hair image based on the processing operations of the hair image.

The designation accepting unit 62 accepts various designations by the user U. For example, the specification reception unit 62 accepts the specification of the treatment content by the user U. The user U can specify the treatment content by operating the operation unit 42, and the specification reception unit 62 receives the specification of the treatment content based on the operation for specifying the treatment content.

The specification receiving unit 62 also receives a specification of a user-captured image, which is an image captured by the imaging unit 43 and includes a hair image, based on an operation by the user U on the operation unit 42.

The designation reception unit 62 also receives the user U's approval or disapproval of the estimated treatment contents. The user U can perform an approval operation or a disapproval operation for the estimated treatment contents by operating the operation unit 42, and the designation reception unit 62 accepts approval or disapproval of the treatment contents based on the approval operation or the disapproval operation.

Further, the specification accepting unit 62 accepts the specification of one or more keywords by the user U. For example, the designation accepting unit 62 accepts designation by the user U of one or more keywords from among a plurality of keywords. The user U can specify one or more keywords by operating the operation unit 42, and the specification accepting unit 62 accepts the specification of one or more keywords based on the operation of specifying the one or more keywords.

The revision proposal receiving unit 63 receives revision proposals made by the hair practitioner to the estimated treatment contents. For example, the revision proposal receiving unit 63 receives revision proposal information, which is information indicating revision proposals made by the hair practitioner to the estimated treatment contents, from the store device 5 as revision proposals made by the hair practitioner to the estimated treatment contents.

The revision proposal receiving unit 63 also receives revision proposals made by the hair practitioner to the hairstyle shown in the composite image. For example, the revision proposal receiving unit 63 receives revision proposal information, which is information indicating revision proposals made by the hair practitioner, from the in-store device 5 as revision proposals for the hairstyle made by the hair practitioner.

The permission acceptance unit 64 accepts permission or disapproval from the user U to the estimated treatment content, which is the treatment content estimated by the estimation unit 54. The user U can perform a permission operation indicating permission for the estimated treatment contents or a non-permission operation indicating disapproval for the estimated treatment contents by operating the operation unit 42, and the permission reception unit 64 performs a permission operation or non-permission operation. Based on the operation, user U's consent or disapproval to the estimated treatment content is accepted.

[4.7.2. Acquisition unit 51]
The acquisition unit 51 acquires various types of information. For example, the acquisition unit 51 acquires various information from the storage unit 45. The acquisition unit 51 acquires information on the learning model stored in the storage unit 45.

The acquisition unit 51 also acquires a captured image, which is an image captured by the imaging unit 43, and stores the acquired captured image in the storage unit 45. The acquisition unit 51 also acquires detection information, which is information detected by the sensor unit 44, and stores the acquired detection information in the storage unit 45.

The acquisition unit 51 acquires information on apps and learning models transmitted from the information providing device 1 via the network N and the communication unit 40, and stores the acquired information on apps and learning models in the storage unit 45. The acquisition unit 51 acquires multiple pieces of posted information each including a specific hair image from the information providing device 1, and stores the acquired pieces of posted information in the storage unit 45.

The acquisition unit 51 also acquires a practitioner's comment and stores the acquired comment in the storage unit 45. The practitioner's comment is a comment from the hair practitioner as to whether or not the hairstyle shown in the composite image can be performed.

Further, the acquisition unit 51 acquires information according to the operation on the operation unit 42 by the user U. For example, the acquisition unit 51 acquires, from the storage unit 45, an image captured by the imaging unit 43 and including a hair image, based on an operation by the user U.

Further, the acquisition unit 51 acquires a plurality of similar hair images, which are images including hair images similar to the specific hair image, and causes the storage unit 45 to store the acquired plurality of similar hair images. The specific hair image is an image of a hairstyle specified by the specifying unit 52. For example, the acquisition unit 51 transmits a similar image search request including a specific hair image to a search server or information providing device 1 (not shown), and multiple images sent from the search server or information providing device 1 in response to the similar image search request. Get similar hair images.

Further, the acquisition unit 51 transmits a search request including information indicating the hairstyle specified by the specifying unit 52 to a search server or information providing device 1 (not shown), and responds to the search request to the search server or the information providing device 1. A plurality of specific hair images transmitted from the device 1 are acquired.

The acquisition unit 51 also acquires treatment cost information for each hair salon, which indicates the relationship between the treatment content and the treatment cost, from, for example, the information providing device 1 or the store device 5.

[4.7.3. Identification unit 52
The identification unit 52 uses the first learning model to identify a hair image from one or more keywords whose designation is accepted by the acceptance unit 50. As described above, the first learning model is a model for identifying a hair image from a keyword and a hair image. This is a model that has learned the relationship between the above and the keywords, and takes one or more keywords as input and outputs a hair image or hair image specification information.

When the first learning model outputs a hair image, the identifying unit 52 identifies the hair image output from the first learning model as a hair image corresponding to one or more keywords specified by the user U.

Further, when the first learning model outputs hair image specific information instead of a hair image, the specifying unit 52 inputs one or more keywords specified by the user U to the first learning model, and inputs one or more keywords specified by the user U to the first learning model. Obtain hair image specific information output from the learning model. The hair image specific information is information that specifies a hair image, and is, for example, a score for each hair image. The specifying unit 52 specifies, for example, the hair image with the highest score among the scores for each hair image as the hair image corresponding to one or more keywords specified by the user U.

The specifying unit 52 obtains the hair image specified by the image specifying information from the storage unit 45 or the information providing device 1, and selects the obtained hair image as a hair image corresponding to one or more keywords specified by the user U. Identify.

Further, the specifying unit 52 uses the third learning model to specify a hairstyle that is estimated to suit the face of the user U. For example, the specifying unit 52 uses the third learning model based on the facial features of the user U specified by the user U or the facial features of the user U determined by the determining unit 55 to determine which model suits the user U's face. Identify the hairstyle that is estimated to be.

As described above, the third learning model is a model that learns the relationship between information indicating facial features and hairstyles that are evaluated as suiting the face, and is a model that receives information indicating facial features as input and outputs information indicating hairstyles that are estimated to suit the face.

The identification unit 52 inputs information indicating the facial features of the user U into the third learning model, and identifies a hairstyle estimated to suit the face of the user U based on the hairstyle information output from the third learning model. The hairstyle information output from the third learning model is, for example, a score for each hair image, and the identification unit 52 identifies the hairstyle shown in the hair image with the highest score among the scores for each hair image as the hairstyle estimated to suit the face of the user U. The hairstyle information output from the third learning model may also be a hair image. The information indicating the facial features of the user U is, for example, information indicating the facial features of the user U specified by the user U or information indicating the facial features of the user U determined by the determination unit 55.

Facial features are, for example, information indicating the type of face shape, but may also be information indicating the features of each part of the face, such as the size, shape, and position of one or more of the eyebrows, eyes, nose, mouth, and ears. For example, information indicating facial features may be information indicating the positions of the eyebrows, eyes, nose, mouth, and ears, or information indicating the positions and sizes of the eyebrows, eyes, nose, mouth, and ears.

4.7.4. Image Processing Unit 53
The image processing unit 53 performs various types of image processing. For example, the image processing unit 53 generates a composite image by combining a user face image, which is a facial image of the user U, with a hair image selected by the user U. Hereinafter, the hair image selected by the user U may be referred to as a selected hair image.

The selected hair image is a hair image whose selection has been accepted by the selection accepting unit 60, but is not limited to this example. For example, the selected hair image may include a hair image selected by the processing unit 46 when an automatic hair image selection mode by the processing unit 46 is selected by an operation of the user U. The user face image is, for example, a user captured image whose designation has been accepted by the designation accepting unit 62 or a face image of the user U included in the user captured image.

For example, the image processing unit 53 generates a composite image by combining the selected hair image with the user's face image. The image processing unit 53 generates a composite image by replacing the hair image of the user U included in the user face image with the selected hair image.

The image processing unit 53 extracts, for example, the feature amount of the user's face image and the feature amount of the selected hair image. The image processing unit 53 generates a composite image by replacing the feature amount of the hair image included in the user face image among the feature amounts of the user face image with the feature amount of the selected hair image.

The image processing unit 53 can also generate a composite image by deleting the hair image of user U included in the user face image and superimposing a selected hair image on the user face image from which the hair image of user U has been deleted.

In addition, when generating a composite image by combining a selected image that is an image that includes a selected hair image and a user image that is an image that includes a user's face image, the image processing unit 53 generates a user's face image that is included in the user image. You can also replace it with a selected hair image.

When two or more hair images are selected by the selection receiving unit 60, the image processing unit 53 generates a composite hair image by combining the two or more selected hair images. Then, the image processing unit 53 generates a composite image by combining the generated composite hair image with the user's face image.

For example, the image processing unit 53 generates a composite hair image by extracting feature amounts of each of the two or more selected hair images and synthesizing the extracted feature amounts of the two or more selected hair images. For example, the image processing unit 53 generates a composite hair image by synthesizing the feature amounts of the two or more selected hair images by weighted addition, but the synthesis method is not limited to this example. For example, the image processing unit 53 has a synthesis model that is a learning model for synthesizing two or more images and is generated by machine learning, and the two or more selected hair images can be input to the synthesis model to obtain a composite hair image output from the synthesis model.

When the image processing unit 53 generates a composite image by combining a selected image, which is an image including a selected hair image, with a user image (e.g., a user-captured image), which is an image including a user's face image, it can also replace the face image of a person included in the selected image with the user's face image included in the user image.

For example, assume that the selected image is a selected image captured by the imaging unit 43. In this case, the image processing unit 53 generates a composite image by replacing the face image of the person included in the selected captured image with the user face image included in the user image in the selected captured image.

When generating a composite image by combining a selected image that is an image that includes a selected hair image and a user image that is an image that includes a user face image, the image processing unit 53 generates a user face that is included in the user image in the user image. The image can also be replaced with a facial image of a person included in the selected image.

For example, assume that the selected image is a selected image captured by the imaging unit 43. In this case, the image processing unit 53 generates a composite image by replacing the user face image in the user image with the face image of the person included in the selected captured image.

When there are multiple selected images, the image processing unit 53 synthesizes the multiple selected images to generate a composite selected image. For example, the image processing unit 53 uses a specific selected image as a reference to synthesize the multiple selected images to generate a composite selected image. The specific selected image is, for example, the initially selected selected image or a selected image specified by the user U.

For example, the image processing unit 53 extracts features of the hair images of each of the multiple selected images, synthesizes the features of the multiple extracted hair images to generate a composite hair image, and replaces the hair image of a specific selected image with the composite hair image to generate a composite selected image. Alternatively, the image processing unit 53 has a synthesis model that is a learning model for synthesizing multiple images and is generated by machine learning, and can obtain a composite hair image output from the synthesis model by inputting the multiple selected images to the synthesis model.

In addition, when the processing reception unit 61 receives a request to process a hair image included in the composite image, the image processing unit 53 generates a user face image including a processed hair image obtained by processing the hair image received by the processing reception unit 61.

For example, when the processing reception unit 61 receives a request to process a hair image, the image processing unit 53 generates a processed hair image obtained by processing the hair image received by the processing reception unit 61. Then, the image processing unit 53 generates a composite image by combining the processed hair image with the user's face image.

The image processing unit 53 generates a processed hair image by, for example, extracting feature amounts of a hair image and adjusting the extracted feature amounts, but is not limited to this example. Further, when the processing acceptance unit 61 accepts processing of the hair image in the composite image, the image processing unit 53 extracts the feature amount of the hair image in the composite image, and adjusts the extracted feature amount to create a composite image. You can also process hair images.

The image processing unit 53 can also perform projective transformation on the selected image. FIG. 10 is a diagram showing an example of the configuration of the image processing section 53 of the terminal device 4 according to the embodiment. As shown in FIG. 10, the image processing section 53 includes a converting section 65 and a combining section 66. The conversion unit 65 projectively transforms the selected image. The composition unit 66 generates a composite image by replacing the face image of the person included in the selected image or the face image of the person included in the selected image projectively transformed by the conversion unit 65 with the user face image.

The conversion unit 65 can also perform projection transformation on a user image or a user face image in place of the selected image. In this case, the synthesis unit 66 generates a composite image by replacing the face image of a person included in the selected image or the face image of a person included in the selected image projection transformed by the conversion unit 65 with the face image of user U included in the user image projection transformed by the conversion unit 65 or the user face image projection transformed by the conversion unit 65.

Furthermore, the combining unit 66 converts the face image of the user U included in the user image projectively transformed by the converting unit 65 into the face image of the person included in the selected image or the selected image projectively transformed by the converting unit 65. A composite image can also be generated by replacing it with a person's face image.

[4.7.5. Estimation unit 54
The estimation unit 54 estimates the treatment contents from the image before the treatment of the user U and the composite image. For example, the estimation unit 54 estimates the treatment contents from the image before the treatment of the user U and the composite image by using the fourth learning model. The treatment details are estimated from this.

The fourth learning model is a learning model that has learned the relationship between the hair images before and after the treatment and the treatment content, and is a learning model that uses the hair images before and after the treatment as input, and outputs information indicating the estimated treatment content. be.

The user U's pre-treatment image is, for example, the above-mentioned user-captured image. The treatment details estimated by the estimation unit 54 are the estimated treatment details described above, and are the treatment details necessary to achieve the hairstyle shown in the composite image generated by the image processing unit 53.

When the estimation unit 54 inputs the pre-treatment image of the user U and the composite image generated by the image processing unit 53 into the fourth learning model, it determines the estimated treatment content based on the information indicating the treatment content output from the fourth learning model.

The information indicating the treatment content outputted from the fourth learning model includes, for example, the presence or absence of a cut, the amount of cut, the presence or absence of bleaching, the number of bleaching, the presence or absence of color, the type of color, the presence or absence of perm, the type of perm, Contains information on each score for whether or not hair is straightened. The estimation unit 54 determines the estimated treatment content based on each score output from the fourth learning model.

For example, the estimation unit 54 includes a cut in the estimated treatment contents when the score of whether or not a cut is performed is equal to or greater than a threshold, and does not include a cut in the estimated treatment contents when the score of whether or not a cut is performed is less than the threshold. Furthermore, the estimation unit 54 includes, for example, an amount of cut proportional to the score of the amount of cut in the estimated treatment contents. Furthermore, the estimation unit 54 includes, for example, colors with scores equal to or greater than a threshold among the scores of each type of color in the estimated treatment contents.

Further, the information indicating the treatment contents from the fourth learning model may be information indicating the treatment contents including treatment items and treatment order. In this case, the fourth learning model is formed to output scores for multiple treatment methods, each including treatment items and treatment orders, and determines the treatment content indicated by the treatment method with the highest score as the estimated treatment content. do.

[4.7.6. Determination unit 55
The determination unit 55 determines the treatment cost, which is the cost required for the treatment with the estimated treatment content, based on the estimated treatment content determined by the estimation unit 54.

For example, the determination unit 55 has treatment cost information for each hair salon that indicates the relationship between the treatment content and the treatment cost, and determines the treatment cost for each hair salon based on the treatment cost information. The treatment cost information for each hair salon is information acquired by the acquisition unit 51 as described above, but is not limited to this example.

Further, the determination unit 55 determines, for example, the facial features of the user U shown in the user captured image. For example, the determination unit 55 inputs the captured image of the user into the second learning model described above, and determines the facial features of the user U based on the information indicating the facial features output from the second learning model. .

The information indicating the facial features output from the second learning model is, for example, a score for each facial feature, and in this case, the determining unit 55 selects the face with the highest score among the scores for each facial feature. The feature is determined as a feature of the user U's face.

In addition, the information indicating the facial features output from the second learning model may be a score for each feature (e.g., eyebrows, eyes, nose, mouth, ears). In this case, the determination unit 55 determines the facial feature including the feature with the highest score among the scores for each feature (combination of position, size, and shape) as the facial feature of the user U.

In addition, the information indicating the facial features output from the second learning model may be a score for each feature of the size and shape of each feature (e.g., eyebrows, eyes, nose, mouth, ears). In this case, the determination unit 55 determines the facial feature including the feature with the highest score among the scores for each feature of the size and shape of the feature as the facial feature of the user U.

Further, the determining unit 55 may have information indicating each of a plurality of types of facial features, and in this case, the determining unit 55 may have information indicating each type of facial feature and the user U's image indicated by the user captured image. The facial features of the user U are determined by pattern matching technology with the facial features.

In addition, the determination unit 55 can display facial feature line drawings, which are line drawings showing multiple types of facial features (e.g., face shapes), on the display unit, and allow the user U to select the facial feature line drawing that is closest to his or her own face shape from among these multiple types of facial feature line drawings.

[4.7.7. Comparison unit 56
The comparison unit 56 compares a user-designated service content, which is the service content designated by the designation receiving unit 62 , with an estimated service content, which is the service content estimated by the estimation unit 54 .

For example, the comparison unit 56 compares the user-specified treatment content and the estimated treatment content, and generates comparison information indicating the comparison result. The comparison information includes, for example, information indicating a difference between the user-specified treatment content and the estimated treatment content.

For example, if the estimated treatment content includes one more bleaching than the user-specified treatment content, the comparison unit 56 generates comparison information including information indicating that the number of bleachings is increased by one compared to the user-specified treatment content.

In addition, when the estimated treatment content or the user-specified treatment content includes a treatment order, and if a part of the treatment order is different, the comparison unit 56 performs a comparison including information indicating a different order among the treatment orders. Generate information.

4.7.8. Output Unit 57
The output unit 57 outputs, as output information, information indicating the identification result by the identification unit 52, information indicating the result of image processing by the image processing unit 53, information indicating the estimation result by the estimation unit 54, information indicating the judgment result by the judgment unit 55, and information indicating the comparison result by the comparison unit 56, etc.

The output unit 57 can also output, as output information, information indicating the reception result by the reception unit 50 and information indicating the acquisition result by the acquisition unit 51. For example, the output unit 57 can output the output information to the display unit 41 to display the output information on the display unit 41, or output the output information to the hair practitioner via the communication unit 40 and the network N. The output to the hair practitioner is performed, for example, via the information providing device 1, SNS, or a mail server (not shown), but may also be output directly to the store device 5.

The output unit 57 outputs information on a reservation setting screen for reserving a hair treatment at a hair salon as output information. The reservation setting screen is a GUI (Graphical User Interface) screen for setting a user image, hair image, hair salon, date and time for reservation, etc., and will be described in detail later.

The output unit 57 outputs the reservation request including the treatment-related information to the hair practitioner, for example, by sending the reservation request including the treatment-related information to the hair practitioner.

The treatment-related information includes, for example, a user image including a hair image of user U before the treatment, a composite image generated by the image processing unit 53, information indicating the treatment details whose designation has been accepted by the designation reception unit 62, and the estimation unit 54. This includes information indicating the details of the treatment estimated by.

The treatment-related information may include a selected hair image, which is a hair image whose selection has been accepted by the selection reception unit 60, instead of or in addition to the composite image generated by the image processing unit 53.

The output unit 57 also outputs multiple keywords so that they can be specified by the user U. For example, the output unit 57 outputs output information including the multiple keywords to the display unit 41, thereby causing the display unit 41 to display the multiple keywords so that they can be specified by the user U.

Further, the output unit 57 outputs the practitioner comment acquired by the acquisition unit 51. For example, the output unit 57 causes the display unit 41 to display the practitioner comment by outputting the practitioner comment acquired by the acquisition unit 51 to the display unit 41.

Further, the output unit 57 outputs modification proposal information that is information indicating the modification proposal by the hair practitioner accepted by the modification proposal reception unit 63. The modification information is, for example, information indicating a modification to the estimated treatment details or information indicating a modification to the hairstyle shown in the composite image. For example, the output unit 57 outputs the modification proposal information accepted by the modification proposal reception unit 63 to the display unit 41, thereby displaying the modification proposal by the hair practitioner on the display unit 41.

In addition, if the permission acceptance unit 64 accepts permission, the output unit 57 outputs the above-mentioned reservation request to the hair practitioner, and if the permission acceptance unit 64 accepts denial, the output unit 57 does not output the above-mentioned reservation request to the hair practitioner.

Note that, when displaying the output information on the display section 41, the output section 57 converts the output information into display information by, for example, rendering, and outputs the converted output information to the display section 41. Further, if the display section 41 has a rendering function, the output section 57 outputs the output information as is to the display section 41 and causes the display section 41 to display the output information.

FIG. 11 is a diagram showing an example of a reservation setting screen displayed on the display unit 41 by the output unit 57 of the information providing device 1 according to the embodiment. As shown in FIG. 11, the reservation setting screen 70 includes a user image selection area 71, a hair image selection area 72, a hair salon selection area 73, and a reservation target date and time setting area 74.

The appointment setting screen 70 is displayed on the display unit 41 by the output unit 57, for example, when the operation by the user U on the operation unit 42 is a specific operation, that is, an appointment setting operation. Although not shown in FIG. 11, the appointment setting screen 70 also includes, for example, a treatment content input area, a treatment content estimation button, and an appointment button.

The user image selection area 71 includes a start shooting button 711 and a folder button 712. The start shooting button 711 is a button for driving an imaging application program (hereinafter, sometimes referred to as an imaging app) that captures an image with the imaging unit 43. The user U can operate the start shooting button 711 to display the imaging app screen on the display unit 41 and operate the imaging app by operating the operation unit 42. The user U can capture an image of an area including his or her own face by operating the imaging app. As a result, the acquisition unit 51 acquires a user image.

The folder button 712 is a button for opening a specific folder (for example, a photo folder into which images captured by the imaging unit 43 are placed). User U can display a plurality of captured images included in a specific folder on display unit 41 by operating folder button 712. By operating the operation unit 42, the user U can select a user image from a plurality of captured images included in a specific folder. Thereby, the user image is acquired by the acquisition unit 51.

Hair image selection area 72 includes a keyword setting button 721, a shooting start button 722, and a folder button 723. The keyword setting button 721 is a button for displaying on the display unit 41 a keyword specification screen for the user U to specify one or more desired keywords from among a plurality of keywords.

The start shooting button 722 is a button for driving the imaging app. The user U can operate the start shooting button 722 to display the imaging app screen on the display unit 41 and operate the imaging app by operating the operation unit 42. By operating the imaging app, the user U can capture an image of an area including a desired hair image (for example, an area including an image of a hair model published in a hair catalogue or fashion magazine). This causes the acquisition unit 51 to acquire the selected image.

The folder button 723 is a button for opening a specific folder (such as a photo folder in which images captured by the imaging unit 43 are placed). The user U can operate the folder button 723 to display multiple captured images contained in the specific folder on the display unit 41. The user U can select a desired image from the multiple captured images contained in the specific folder by operating the operation unit 42. This causes the acquisition unit 51 to acquire the selected image.

The hair salon selection area 73 includes a search button 731 around the current location and a keyword search button 732. The current location vicinity search button 731 is a button for displaying hair salons located around the current location (for example, an area within a predetermined distance from the current location) on the display unit 41 in a selectable manner. User U can select a desired hair salon from among the hair salons displayed on display section 41 by operating operation section 42 .

The keyword search button 732 is a button for displaying a keyword search screen on the display unit 41 for performing a keyword search. By operating the operation unit 42, the user U can select a desired hair salon from among the hair salons searched for on the keyword search screen and displayed on the display unit 41.

The reservation target date and time setting area 74 is an area for setting the reservation target date and time. The user U can set the date and time at which the user wishes to receive the hair treatment in the reservation target date and time setting area 74 by operating the operation unit 42.

FIG. 12 is a diagram showing an example of a keyword specification screen displayed on the display unit 41 by the output unit 57 of the information providing device 1 according to the embodiment. As shown in FIG. 12, the keyword specification screen 80 includes keyword specification button groups 81, 82, 83, and 84, and a search button 85.

The keyword specification button group 81 includes a number of buttons for specifying keywords corresponding to hair length. The keyword specification button group 81 shown in FIG. 12 includes a button for specifying the keyword "short", a button for specifying the keyword "medium", a button for specifying the keyword "long", etc.

The keyword specification button group 82 includes a plurality of buttons for specifying keywords corresponding to hair colors. The keyword specification button group 82 shown in FIG. 12 includes a button for specifying the keyword "brown," a button for specifying the keyword "beige," a button for specifying the keyword "pink," and a button for specifying the keyword "black." Including.

The keyword specification button group 83 includes a number of buttons for specifying keywords corresponding to atmospheres. The keyword specification button group 83 shown in FIG. 12 includes a button for specifying the keyword "cute", a button for specifying the keyword "cool", a button for specifying the keyword "pop", a button for specifying the keyword "natural", and the like.

The keyword specification button group 84 includes a plurality of buttons for specifying keywords corresponding to TPO (Time Place Occasion). The keyword specification button group 84 shown in FIG. 12 includes a button for specifying the keyword "Harajuku", a button for specifying the keyword "Ginza", a button for specifying the keyword "Shinjuku", a button for specifying the keyword "Roppongi", etc.

The multiple keywords indicated by the keyword designation buttons 81, 82, 83, and 84 are multiple predetermined keywords, but may also be multiple keywords randomly extracted from the multiple keywords or multiple keywords that satisfy predetermined conditions. The multiple keywords that satisfy the predetermined conditions are, for example, a predetermined number of keywords that are ranked in order of frequency of selection by user U. The keywords that are candidates for extraction are, for example, keywords that are to be input to the first learning model.

On the keyword specification screen 80, the user U can specify hair length, hair color, mood, time, occasion, etc. by operating the operation unit 42 and pressing a button corresponding to a desired keyword. Then, the user U operates the operation unit 42 to select a search button 85, which causes the processing unit 46 to execute a process of acquiring similar hair images from the specified keyword. Note that the multiple keywords included in the keyword specification screen 80 are not limited to the above-mentioned examples.

Figure 13 is a diagram for explaining the process of acquiring similar hair images from a specified keyword by the processing unit 46 of the information providing device 1 according to the embodiment. In the example shown in Figure 13, the keywords specified by the user U are "long", "pink", "cute", and "Harajuku".

In this case, the reception unit 50 accepts keywords specified by the user U such as "long", "pink", "cute", and "Harajuku". The identification unit 52 uses the first learning model to identify hair images from the keywords “long”, keyword “pink”, keyword “cute”, and keyword “Harajuku” whose designations are accepted by the reception unit 50.

Then, the acquisition unit 51 transmits a similar image search request including the specific hair image to a search server or information providing device 1 (not shown), and multiple images sent from the search server or information providing device 1 in response to the similar image search request. Get similar hair images.

Fig. 14 is a diagram showing another example of a reservation setting screen displayed on the display unit 41 by the output unit 57 of the information providing device 1 according to the embodiment. In the reservation setting screen 70 shown in Fig. 14, a user image selected by the user U is set in a user image selection area 71, and a plurality of similar hair images corresponding to one or more keywords designated by the user U are set in a hair image selection area 72.

By operating the operation unit 42, the user U can select one or more similar hair images from among the multiple similar hair images set in the hair image selection area 72 as selected images.

FIG. 15 is a diagram showing still another example of the reservation setting screen displayed on the display unit 41 by the output unit 57 of the information providing device 1 according to the embodiment. The reservation setting screen 70 shown in FIG. 15 is created when the user U selects one similar hair image among the plurality of similar hair images shown in FIG. 14 by operating the operation unit 42 on the reservation setting screen 70 shown in FIG. is displayed on the display section 41.

The reception unit 50 receives the selection of a similar hair image by the user U as a selected image. The image processing unit 53 generates a composite image by combining the similar hair image selected and received by the reception unit 50 with the user image selected by the user U, and places the generated composite image on the reservation setting screen 70 as shown in FIG. 15. This allows the user U to check the composite image displayed on the display unit 41, and to more realistically check whether the hairstyle shown in the selected hair image suits him or her.

In addition, the user U can select a composite image on the reservation setting screen 70 by operating the operation unit 42, thereby displaying the composite image on the display unit 41 while changing the selected image that generates the composite image.

FIG. 16 is a diagram showing an example of a composite image change screen displayed on the display unit 41 by the output unit 57 of the information providing device 1 according to the embodiment. A composite image change screen 90 shown in FIG. 16 includes a length selection button 91, a color selection button 92, and a composite image display area 93.

The length selection button 91 is a button for changing the target to be changed by swiping (or scrolling) up or down to the hair length. The color selection button 92 is a button for changing the target to be changed by swiping (or scrolling) up or down to the hair color. Swiping (or scrolling) up or down is an example of a second change operation for the hair image.

In the example shown in FIG. 16, the length selection button 91 is selected, and the user U operates the operation unit 42 to swipe (or scroll) up and down to select the length of the hair in the composite image. Can be changed. The image processing unit 53 changes the length of the hair in the composite image when the reception unit 50 accepts a swipe (or scroll) in the vertical direction, and displays the composite image with the changed hair length on the display unit 41. Display.

Further, the user U operates the operation unit 42 to select the color selection button 92, and then operates the operation unit 42 to swipe (or scroll) up and down to change the color of the hair in the composite image. can do. The image processing unit 53 changes the hair color of the composite image when the reception unit 50 accepts the swipe in the vertical direction, and causes the display unit 41 to display the composite image with the changed hair color.

Furthermore, the user U can display on the display unit 41 a composite image in which the selected image has been changed by swiping (or scrolling) in the left/right direction. A swipe (or scroll) in the left/right direction is an example of the first changing operation of the hair image. When the reception unit 50 accepts a swipe (or scroll) in the left/right direction, the image processing unit 53 changes the selected image to generate a composite image, and displays the composite image with the changed selected image on the display unit 41. let

As described above, the reservation setting screen 70 includes a treatment content input area (not shown). By operating the operation unit 42, the user U can input the treatment content specified or desired by the user U into the treatment content input area.

Further, as described above, the reservation setting screen 70 includes a treatment content estimation button, a reservation button, etc. (not shown). The user U determines a composite image, inputs the treatment content specified or desired by the user U into the treatment content input area, selects a hair salon, and sets the date and time for reservation. Then, the user U operates the operation unit 42 and selects the reservation button, so that the output unit 57 transmits a reservation request to the hair practitioner. As a result, a reservation request is output to the hair practitioner.

The reception unit 50 receives selection of the treatment content estimation button by the user U. When the selection of the treatment content estimation button is accepted by the reception unit 50, the estimating unit 54 estimates the treatment content from the user image and the composite image shown on the reservation setting screen 70. The user image is an example of an image of the user U before the treatment. When the user image and the composite image are input to the fourth learning model, the estimation unit 54 determines the estimated treatment content based on information indicating the treatment content output from the fourth learning model.

FIG. 17 is a diagram for explaining the process of estimating the treatment contents from the user image and the composite image by the estimation unit 54 of the information providing device 1 according to the embodiment. In the example shown in FIG. 17, the estimation unit 54 determines "cut, bleach once, color bleach twice, and perm once" as the estimated treatment contents from the user image and the composite image.

When the reception unit 50 accepts the selection of the reservation button, the output unit 57 outputs the user image, the composite image, the information indicating the treatment content input in the treatment content input area, the information indicating the estimated treatment content, and the reservation target date and time. A reservation request including information indicating such information is output to the hair practitioner. Further, when the reception unit 50 accepts selection of a specific button (not shown), the output unit 57 causes the display unit 41 to display a screen indicating the end of the operation, for example. User U's selection of the reservation button is an example of a response indicating user U's consent to the estimated treatment content, and user U's selection of a specific button is an example of a response indicating user U's disapproval of the estimated treatment content. .

In addition, the output unit 57 can output a face shape selection screen for allowing the user U to select the face shape of the user U to the display unit 41 in response to an operation of the operation unit 42 by the user U, and can display the face shape selection screen on the display unit 41.

FIG. 18 is a diagram showing an example of a face type selection screen displayed on the display unit 41 by the output unit 57 of the information providing device 1 according to the embodiment. As shown in FIG. 18, the face type selection screen 95 includes a plurality of face types. The user U can operate the operation unit 42 to select the face type closest to the user U's face type among the face types.

The reception unit 50 receives a face type selection by the user U from among multiple face types included in the face type selection screen 95. The identification unit 52 regards the face type selected by the user U as the facial features of the user U designated by the user U, and identifies a hairstyle that is estimated to suit the face of the user U using a third learning model from the facial features of the user U.

The identifying unit 52 inputs information indicating the face type selected by the user U into the third learning model, and determines which hairstyle suits the user U's face based on the information on the hairstyle output from the third learning model. Identify the estimated hairstyle.

5. Processing Procedure
Next, a procedure of information processing by the processing unit 46 of the terminal device 4 according to the embodiment will be described. Fig. 19 is a flowchart showing an example of information processing by the processing unit 46 of the terminal device 4 according to the embodiment.

As shown in FIG. 19, the processing unit 46 of the terminal device 4 determines whether learning model information has been acquired from the information providing device 1 (step S20). When the processing unit 46 determines that the learning model information has been acquired (step S20: Yes), the processing unit 46 causes the storage unit 45 to store the acquired learning model information (step S21).

When the process of step S21 is completed or when it is determined that the learning model information has not been acquired (step S20: No), the processing unit 46 determines whether there is a reservation setting operation by the user U ( Step S22). If the processing unit 46 determines that there is a reservation setting operation (step S22: Yes), it executes reservation processing (step S23). The reservation process in step S23 is the process in steps S40 to S49 shown in FIG. 20, and will be described in detail later.

When the processing of step S23 is completed or when it is determined that no reservation setting operation has been performed (step S22: No), the processing unit 46 determines whether or not there are any revision suggestions from the hair practitioner (step S24). When it is determined that there are any revision suggestions (step S24: Yes), the processing unit 46 causes the display unit 41 to display the revision suggestions from the hair practitioner (step S25).

When the processing of step S25 is completed or when it is determined that there is no modification proposal (step S24: No), the processing unit 46 determines whether or not there is a practitioner comment (step S26). If the processing unit 46 determines that there is a practitioner comment (step S26: Yes), the processing unit 46 displays the practitioner comment on the display unit 41 (step S27).

When the processing of step S27 ends or when it is determined that there is no practitioner comment (step S26: No), the processing unit 46 determines whether or not there is an image synthesis operation (step S28). When it is determined that there is an image synthesis operation (step S28: Yes), the processing unit 46 performs synthesis processing (step S29). The synthesis processing of step S29 is the same as steps S40 to S47 shown in FIG. 20.

When the processing of step S29 is completed or when it is determined that there is no image synthesis operation (step S28: No), the processing unit 46 determines whether or not the operation end timing has arrived (step S30). For example, the processing unit 46 determines that the operation end timing has arrived when the power supply of the information providing device 1 is turned off or when it is determined that an end operation has been performed by operating an operation unit (not shown) of the information providing device 1.

If the processing unit 46 determines that the operation end timing has not come (step S30: No), the process proceeds to step S20, and if it determines that the operation end timing has come (step S30: Yes), the process proceeds to FIG. The indicated process ends.

FIG. 20 is a flowchart illustrating an example of a reservation process by the processing unit 46 of the terminal device 4 according to the embodiment. As shown in FIG. 20, the processing unit 46 receives the selection of a user image, and acquires the selected user image (step S40).

Next, the processing unit 46 accepts the designation of one or more keywords (step S41). The processing unit 46 identifies a hair image from the one or more keywords designated in step S41 (step S42). Then, the processing unit 46 acquires a similar hair image that is an image similar to the hair image identified in step S42 (step S43).

Subsequently, the processing unit 46 combines the similar hair image selected from among the similar hair images acquired in step S42 and the user's face image (step S44). Then, the processing unit 46 causes the display unit 41 to display the composite image synthesized in step S44 (step S45).

Subsequently, the processing unit 46 determines whether there is a change operation (step S46). The changing operation is, for example, the first hair image changing operation or the second hair image changing operation described above. If the processing unit 46 determines that there is a changing operation (step S46: Yes), it performs image processing according to the changing operation to change the composite image (step S47), and moves the process to step S46.

When the processing unit 46 determines that no change operation has been performed (step S46: No), it estimates the treatment contents based on the pre-treatment image and the composite image (step S48). The processing unit 46 determines that no change operation has been performed when, for example, the user U selects the reservation button.

Then, the processing unit 46 outputs a reservation request including the pre-treatment image, the composite image, the estimated treatment contents, and the designated treatment contents to the hair treatment person (step S49). The specified treatment content is the treatment content input by the user U. When the processing in step S49 is completed, the processing unit 46 ends the processing shown in FIG. 20.

[6. Modified example]
In the above example, the processing unit 46 causes the display unit 41 to display a composite image in which the selected image has been changed by the user's operation in the left and right directions, and displays the composite image in which the length or color of the hair has been changed by the user's operation in the vertical direction. Although the image is displayed on the display unit 41, the present invention is not limited to this example.

For example, the processing unit 46 can cause the display unit 41 to display a composite image in which the selected image has been changed by a user operation in the up or down direction, and can cause the display unit 41 to display a composite image in which the hair length or color has been changed by a user operation in the left or right direction.

Furthermore, the processing unit 46 can also cause the display unit 41 to display a composite image in which the selected image and the hair length, color, etc. have been changed by the user's operation in the diagonal direction. Note that the user operation is a swipe operation or a scroll operation, but is not limited to such an example, and may be, for example, a tap operation, a click operation, a pinch-in operation, a pinch-out operation, or the like.

Furthermore, the hairstyle information output from the third learning model is not limited to hair images. For example, the hairstyle information output from the third learning model may be hair image specifying information that is information for specifying a hair image. The information for specifying the hair image is, for example, identification information of the hair image or a keyword for specifying the hair image.

In this case, the third learning model is a learning model that includes multiple combinations of information indicating facial features, hair image specific information, and information indicating an evaluation of the combination of information indicating facial features and hair image specific information. Generated using data for

Moreover, each learning model including the first learning model, the second learning model, the third learning model, etc. may be a learning model for each user U. For example, by setting the hair image used to generate each of the first learning model and the third learning model to be the hair image included in the posted image viewed by user U, the first learning model and the third learning model can be generated. Each of the learning models can be generated for each user U.

In addition, by setting the facial image used to generate the second learning model to a hair image included in the posted images viewed by user U, the second learning model can be generated for each user U.

The information providing device 1 can execute some or all of the functions of the acquisition unit 51, the identification unit 52, the image processing unit 53, the estimation unit 54, the determination unit 55, the comparison unit 56, and the output unit 57 in cooperation with the terminal device 4, and can function as a part of an information processing device. Note that hereinafter, a configuration including the above-mentioned terminal device 4 and some or all of the information providing device 1 may be referred to as an information processing device.

[7. Hardware configuration]
Each of the information providing device 1 and the terminal device 4 according to the embodiments described above is realized by, for example, a computer 200 configured as shown in FIG. 21. FIG. 21 is a hardware configuration diagram showing an example of a computer 200 that implements the functions of the information providing device 1 and the terminal device 4 according to the embodiment. The computer 200 includes a CPU 201, a RAM 202, a ROM (Read Only Memory) 203, an HDD (Hard Disk Drive) 204, a communication interface (I/F) 205, an input/output interface (I/F) 206, and a media interface (I/F). )207.

The CPU 201 operates based on a program stored in the ROM 203 or the HDD 204, and controls each component. The ROM 203 stores a boot program executed by the CPU 201 when the computer 200 is started, and programs that depend on the hardware of the computer 200.

The HDD 204 stores programs executed by the CPU 201, data used by the programs, and the like. The communication interface 205 receives data from other devices via the network N (see FIG. 2) and sends it to the CPU 201, and sends data generated by the CPU 201 to the other devices via the network N.

The CPU 201 controls output devices such as a display and a printer, and input devices such as a keyboard and a mouse via an input/output interface 206. The CPU 201 obtains data from an input device via the input/output interface 206. Further, the CPU 201 outputs the generated data to an output device via the input/output interface 206.

The media interface 207 reads a program or data stored in the recording medium 208 and provides it to the CPU 201 via the RAM 202. The CPU 201 loads the program from the recording medium 208 onto the RAM 202 via the media interface 207 and executes the loaded program. The recording medium 208 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

For example, when the computer 200 functions as the information providing device 1 or the terminal device 4 according to the embodiment, the CPU 201 of the computer 200 controls the functions of the processing unit 12 and the processing unit 46 by executing a program loaded on the RAM 202. Realize. Furthermore, data in the storage unit 11 or the storage unit 45 is stored in the HDD 204 . The CPU 201 of the computer 200 reads these programs from the recording medium 208 and executes them, but as another example, these programs may be acquired from another device via the network N.

[8. others〕
Further, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or the processes described as being performed manually can be performed manually. All or part of the process can also be performed automatically using known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above documents and drawings may be changed arbitrarily, unless otherwise specified. For example, the various information shown in each figure is not limited to the illustrated information.

Furthermore, each component of each device shown in the drawings is functionally conceptual, and does not necessarily need to be physically configured as shown in the drawings. In other words, the specific form of distributing and integrating each device is not limited to what is shown in the diagram, and all or part of the devices can be functionally or physically distributed or integrated in arbitrary units depending on various loads and usage conditions. Can be integrated and configured.

For example, the information providing device 1 described above may be realized by multiple server computers, and depending on the function, it may be realized by calling an external platform using API or network computing, so that the configuration can be changed flexibly. .

Furthermore, the embodiments and modifications described above can be combined as appropriate within a range that does not conflict with the processing contents.

9. Effects
As described above, the information processing device according to the embodiment includes the image processing unit 53 and the output unit 57. The image processing unit 53 generates a composite image by combining a user face image, which is a facial image of the user U, with a hair image selected by the user U. The output unit 57 outputs the composite image generated by the image processing unit 53 to the hair practitioner. This allows the user U to present the composite image, which is a composite image of a desired hair image and the user face image, to the hair practitioner, and allows the user U to specifically communicate the desired hairstyle to the hair practitioner, compared to a case where the user U vaguely communicates the desired hairstyle to the hair practitioner. Therefore, the information processing device can improve the convenience of the user U.

The information processing device also includes a selection receiving unit 60 that receives a hair image selected by the user U from among multiple types of hair images. The image processing unit 53 generates a composite image by combining the hair image received by the selection receiving unit 60 with the user's face image. This allows the information processing device to improve the convenience of the user U.

The information processing device also includes a designation receiving unit 62 that receives, based on an operation by the user U, a designation of a captured image that is captured by the imaging unit 43 and includes a hair image. The image processing unit 53 creates a composite image by replacing the face image of the person included in the captured image with the face image of the user U included in the user face images based on the captured image whose designation has been accepted by the specification reception unit 62. generate. Thereby, the information processing device can improve convenience for the user U.

The image processing unit 53 also includes a conversion unit 65 that performs projective transformation on the captured image, and a synthesis unit 66 that generates a composite image by replacing the facial image included in the captured image that has been projectively transformed by the conversion unit 65 with the user's facial image. This allows the information processing device to improve the convenience of the user U.

In addition, when two or more hair images are selected by the selection receiving unit 60, the image processing unit 53 generates a composite image by combining the two or more hair images to obtain a composite hair image with the user's face image. This allows the information processing device to improve the convenience of the user U.

When the specification receiving unit 62 receives a specification of multiple captured images, the image processing unit 53 generates a composite image by replacing the facial image included in the composite captured image obtained by combining the multiple captured images with the user's facial image. This enables the information processing device to improve the convenience of the user U.

The information providing device 1 also includes a processing reception unit 61 that receives processing of a hair image. The image processing unit 53 generates a composite image by combining the processed hair image obtained by processing the hair image received by the processing reception unit 61 with the user's face image. Thereby, the information processing device can improve convenience for the user U.

Further, the output unit 57 outputs a user image including the user U's pre-treatment hair image to the hair treatment person. Thereby, the information processing device can improve convenience for the user U.

The information processing device also includes a designation reception unit 62 that receives designation of the treatment content by the user U. The output unit 57 outputs, to the hair practitioner, a reservation request that includes information indicating the treatment content whose designation has been accepted by the designation accepting unit 62, the image of the user U, and the composite image. Thereby, the information processing device can improve convenience for the user U.

The information processing device also includes an acquisition unit 51 that acquires comments from hairdressers about whether the hairstyle shown in the composite image can be achieved. The output unit 57 outputs the comments acquired by the acquisition unit 51. This allows the information processing device to improve the convenience of the user U.

The information processing device also includes a modification proposal reception unit 63 that receives modification proposals by a hair practitioner for the hairstyle shown in the composite image. The output unit 57 outputs information indicating the modification proposal accepted by the modification proposal reception unit 63. Thereby, the information processing device can improve convenience for the user U.

The above describes the embodiments of the present application in detail based on the drawings, but this is merely an example, and the present invention can be implemented in other forms that incorporate various modifications and improvements based on the knowledge of those skilled in the art, including the aspects described in the disclosure section of the invention.

In addition, the above-mentioned "section, module, unit" can be read as "means" or "circuit." For example, an acquisition unit can be read as an acquisition means or an acquisition circuit.

LIST OF SYMBOLS 1 Information providing device 4 Terminal device 5 Store device 10, 40 Communication unit 11, 45 Memory unit 12, 46 Processing unit 20 User information memory unit 21 Posted information memory unit 22 Model information memory unit 23 Content memory unit 30, 51 Acquisition unit 31 Learning unit 32 Providing unit 41 Display unit 42 Operation unit 43 Imaging unit 44 Sensor unit 50 Reception unit 52 Identification unit 53 Image processing unit 54 Estimation unit 55 Determination unit 56 Comparison unit 57 Output unit 60 Selection reception unit 61 Processing reception unit 62 Designation reception unit 63 Revision proposal reception unit 64 Acceptance/rejection reception unit 65 Conversion unit 66 Synthesis unit 100 Information processing system

Claims (13)

an image processing unit that generates a composite image by combining a user face image that is a user's face image and a hair image selected by the user;
An information processing device comprising: an output unit that outputs the composite image generated by the image processing unit to a hair practitioner. a selection receiving unit that receives a selection of a hair image by the user from among a plurality of types of hair images;
The image processing unit includes:
The information processing apparatus according to claim 1 , wherein the composite image is generated by combining the hair image accepted by the selection accepting unit with the user's face image. a designation receiving unit that receives a designation of a captured image including a hair image, the captured image being captured by an imaging unit, based on an operation by the user;
The image processing unit includes:
The information processing device according to claim 1, characterized in that the composite image is generated by replacing a facial image of a person contained in the captured image with a facial image of the user contained in the user facial image, based on the captured image for which the designation is accepted by the designation accepting unit. The image processing unit includes:
A transformation unit that performs projective transformation on the captured image;
The information processing apparatus according to claim 3 , further comprising: a synthesis unit configured to generate the synthetic image by replacing the facial image included in the captured image projected by the transformation unit with the user's facial image. The image processing unit includes:
When two or more hair images are selected by the selection reception unit, the synthesized hair image obtained by synthesizing the two or more hair images is synthesized with the user face image to generate the synthesized image. The information processing device according to claim 2. The image processing unit includes:
4. The information processing device according to claim 3, wherein when the designation receiving unit receives designation of a plurality of the captured images, the composite image is generated by replacing the facial image included in a composite captured image obtained by combining the plurality of the captured images with the user's facial image. comprising a processing reception unit that receives processing of the hair image,
The image processing unit includes:
The combined image is generated by combining a processed hair image obtained by performing the processing of the hair image received by the processing reception unit with the user face image. The information processing device according to any one of the above. The output section is
The information processing device according to any one of claims 1 to 6, wherein a user image including an image of the user's hair before treatment is output to the hair treatment person. comprising a designation reception unit that accepts designation of treatment details by the user;
The output section is
Claim 2 or 4, characterized in that a reservation request including information indicating the treatment content whose designation has been accepted by the designation reception unit, the image of the user, and the composite image is output to the hair treatment person. 6. The information processing device according to 5 or 6. an acquisition unit that acquires a comment from the hairdresser as to whether the hair style shown in the composite image is possible;
The output unit is
The information processing apparatus according to claim 9 , further comprising: an output section configured to output the comment acquired by the acquisition section. comprising a modification proposal reception unit that receives a modification proposal by the hair practitioner for the hairstyle shown in the composite image,
The output section is
The information processing device according to claim 9, wherein information indicating the modification proposal received by the modification proposal reception unit is output. An information processing method performed by a computer, the method comprising:
an image processing step of generating a composite image by combining a user face image that is a user's face image and a hair image selected by the user;
An information processing method comprising: an output step of outputting the composite image generated by the image processing step to a hair practitioner. an image processing procedure for generating a composite image by combining a user face image, which is a face image of a user, with a hair image selected by the user;
and an output step of outputting the composite image generated by the image processing step to a hair salon.

Images