JP6857638B2 - Recommender system based on user's physical characteristics - Google Patents

Description

Those who are interested in buying clothing and accessories (such as eyeglasses) usually need to visit the store directly and try on to see if the item fits well. Clothing and accessories typically have both functional and aesthetic purposes, which can influence the selection process. It can take some time to look around and try on a product, and most of the time there is no prior knowledge of the product's suitability. Personalized stylists can assist in the decision process by proposing products. However, personalized stylists are limited by their bias and their relatively narrow familiarity with the available inventory. Traditional product recommendation algorithms are usually simple and provide suggestions based on a primitive evaluation of the user's browsing history.

Various embodiments of the present invention are disclosed in the following detailed description and accompanying drawings.

A flowchart showing an embodiment of a process for recommending a product based on a user's image.

A block diagram showing an embodiment of a system for making recommendations based on the physical characteristics of a user.

A block diagram showing an example of a system for making recommendations based on the physical characteristics of the user.

The block diagram which shows one Embodiment of a recommendation engine.

A flowchart showing an embodiment of a process for 3D modeling of a user's face.

A flowchart showing an embodiment of a process for 3D modeling of a user's face.

The figure which shows an example of the input to the principal component analysis and the output from the principal component analysis for generating a 3D morphing model of a user's face.

The figure which shows an example of the reference point obtained from one set of images of a user's face.

The figure which shows an example of the reference point obtained from one set of images of a user's face.

The figure which shows an example of the visualization of a predetermined 3D face in a 3D space.

A flowchart showing an embodiment of a process for determining a description of a recommendation.

The block diagram which shows one Embodiment of a face data scoring system.

A flowchart showing an embodiment of a process for determining the characteristics of facial features using a classifier.

The figure which shows an example of the graphical user interface (GUI) of the product catalog for browsing an inventory item.

FIG. 13A is a diagram showing an example of a GUI showing a product displayed according to a user interaction with the GUI of FIG. 13A.

FIG. 13A is a diagram showing an example of a GUI showing a product displayed according to a user interaction with the GUI of FIG. 13A.

The figure which shows an example of GUI of the product catalog for browsing an inventory item.

FIG. 14A is a diagram showing an example of a GUI displayed according to the user interaction with the GUI.

The figure which shows an example of GUI of the product catalog for browsing an inventory item.

FIG. 15A is a diagram showing an example of a GUI displayed according to the user interaction with the GUI.

The figure which shows an example of GUI which shows the output of the recommendation based on the physical characteristic of a user.

The present invention presents the processing, equipment, systems, composition of materials, computer program products embodied on computer-readable storage media, and / or processors (stored in and / or memory in memory connected to the processor). It can be implemented in various forms, including a processor configured to execute the instructions provided). As used herein, these implementations or any other form that the invention may take is referred to as technology. In general, the order of the disclosed processing steps may be changed within the scope of the present invention. Unless otherwise stated, components such as processors or memory that are described as being configured to perform a task are either as general components that are temporarily configured to perform a task at a given time, or It may be implemented as a specific component manufactured to perform a task. As used herein, the term "processor" refers to a processing core configured to process data such as one or more devices, circuits, and / or computer program instructions.

Hereinafter, one or more embodiments of the present invention will be described in detail with reference to the drawings showing the principles of the present invention. The present invention has been described in connection with such embodiments, but is not limited to any of the embodiments. The scope of the invention is limited only by the claims, and the invention includes many alternatives, variants, and equivalents. In the following description, many specific details are provided to provide a complete understanding of the present invention. These details are for purposes of illustration only, and the present invention can be carried out in accordance with the claims without some or all of these specific details. For the sake of simplicity, the technical matters well known in the technical fields related to the present invention are not described in detail so as not to make the present invention unnecessarily confusing.

An embodiment of a recommender system based on the physical characteristics of the user is disclosed. In various embodiments, physical characteristics (also referred to as "biometric characteristics") are determined and used to make appropriate product recommendations to the user. Physical features can represent the 3D surface of the user's face, such as facial structure or proportions, facial shape, eye color and shape, hair color, skin color, skin pattern, etc. In various embodiments, the physical characteristics of the user are determined based at least in part on the image of the user. A correlation between physical characteristics and the product is determined, and product recommendations are generated based at least in part on the determined correlation. The user's image may be a still image or video frame of the user's front, side, or images in between, as described in detail later.

The correlation between physical characteristics and merchandise may be determined based on the degree of similarity between users (such as the 3D surface of a similar face) or the fit of the merchandise to the user. Unlike traditional stylists or fit advisors who provide recommendations based on user-reported 2D image attributes or characteristics, in various embodiments, systematically determined 3D physical features such as 3D surface features of the user's face. Recommendations are made based on characteristics.

FIG. 1 is a flowchart showing an embodiment of a process for recommending a product based on a user's image. Process 100 may be performed at least partially on one or more components of system 200 of FIG. In some embodiments, the process 200 is performed by the recommendation engine 314 of FIG.

The process determines physical characteristics based on the image (step 102). A physical feature is an attribute of an object in an image. For example, for a user's image, physical features include facial 3D surface, facial structure or proportions, face shape, eye color, eye shape, hair color, skin color, skin pattern, etc. Including features. As detailed with respect to FIGS. 8A and 8B, physical features such as the distance between the eyes, the distance from the bridge of the nose to the tip of the nose, etc. can be measured by extracting reference points obtained from a set of images. In various embodiments, the image includes other sensor data (such as a depth sensor) that can facilitate the determination of physical characteristics.

The image may be captured by a camera (such as the camera associated with client device 204 in FIG. 2). Other sensor data may be captured by various sensors (such as depth sensors). In some embodiments, a single, substantially frontal image of the user is sufficient to make a product recommendation. In some embodiments, multiple images of the user, such as images of the user's head taken from different angles, frames forming the video, determine physical characteristics as detailed herein. Used for The user may be provided with instructions on how to take the image. For example, the user may be instructed to be in front of the camera and rotate his head as the camera captures a video frame. In some embodiments, the user may be instructed to look to the right after looking to the left. In some embodiments, the instructions may be provided in the form of demo video clips or animations. The number of video frames captured may vary. The camera may be instructed by the processor to capture the user's head in a series of videos or snapshots (eg, a series of images with delays between captures). In some embodiments, the camera captures an image of the user's head in continuous capture mode, where the frame rate may be lower than the video capture. The processor may be local or remote (eg, on a server). The captured image determines the physical characteristics of the object and / or the 3D model of the user's face on which the 3D model of the selected spectacle frame is placed / tried on, as detailed below. Can be used to.

In some embodiments, physical characteristics are determined based on an intermediate model of the user's head constructed from images. The intermediate model may be a 3D model of the head described by vectors and polygons that encode information about facial features and head proportions on the 3D surface of the face, as detailed herein. Physical features may be extracted from the 3D model. In some embodiments, the 3D model is generated by the 3D model engine 306 shown in FIG.

The process determines the correlation between physical characteristics and merchandise (step 104). The correlation can be a relationship between users, such as the similarity of facial features or the similarity of a face on a 3D surface. The correlation may be the relationship between the user and the product, such as the degree of fit of the product to the user. As used herein, fit refers to a functional fit, such as how well the spectacles fit on the user's face, depending on factors such as comfort and the quality of vision that the spectacles provide to the user. Can be shown. As used herein, fit also refers to an aesthetic fit, such as how nice the glasses on the user's face look, and scores the ratio of the frame to the eyes and the ratio of other faces. Can be indicated by

The process recommends the product (step 106). Recommendations may be generated based at least in part on the determined correlation. In various embodiments, recommendations are determined using deep neural networks, content-based filtering, collaborative filtering, or a combination thereof.

Content-based filtering is the search for products that are similar to other products. Using the eyeglass example, similarity may be assessed based on attributes such as shape, color, material (plastic or wireframe), price, and compatibility with lens formulations. By searching by the similarity of products to each other, it is possible to find eyeglasses that are similar to other eyeglasses because they have shared or similar attributes. A user who is interested in a certain product may be interested in a similar product.

Collaborative filtering is finding products based on purchase history. It is assumed that the first user purchases eyeglasses A, B, and C, and the second user purchases eyeglasses B, C, and D. Since both purchase the glasses B and C, the purchase histories of the first user and the second user are similar to each other. In some embodiments, collaborative filtering of this data allows eyeglasses A to be recommended to a second user and eyeglasses D to be recommended to a first user because the preferences of the first and second users are similar. Suggest that you can.

In various embodiments, the results of content-based filtering or collaborative filtering may be adjusted based on other factors, such as user-expressed preferences. By default, a frame with a color that contrasts with the user's skin tone is recommended, but it is assumed that the user prefers a frame with a color that matches the user's skin tone. Recommendations may be fine-tuned according to user suggestions for a preference for frames that are in harmony with the skin tone. If there is user preference information, the recommendation system can be updated to recommend products that are more suitable for the user's preference.

In one aspect, preferences can change over time. For example, user preferences can change with changes in style or preferences. Recommendation engines can adapt to changes by learning and adapting to them.

In some embodiments, the product recommendation includes an explanation of why the recommendation was made. In contrast to traditional techniques where recommender systems present recommendations to users without explaining why the product was selected, the recommendations made here are facts and explanations of why and how the system made the selection. May include. The description of the recommendation may be made according to the process shown in FIG. Correlation and recommendation processing will be further described herein with respect to the system shown in FIG.

FIG. 2 is a block diagram showing an embodiment of a system for making recommendations based on the physical characteristics of the user. In this example, system 200 includes client device 204, network 206, and server 208. The client device 204 is connected to the server 208 via the network 206. Network 206 may include high speed data networks and / or telecommunications networks. User 202 may interact with the client device to provide useful information for making recommendations and to receive recommendations according to the methods detailed herein.

Client device 204 is configured to provide a user interface to user 202. For example, client device 204 may receive input or observe user interaction by user 202 with client device. Recommendations are printed to the user based on at least some of the information collected by the client device. In some embodiments, recommendations are made in collaboration with server 208 as detailed herein.

In various embodiments, the client device comprises input components such as a camera, a depth sensor, other sensors, or a combination of a plurality of sensors. The camera may be configured to observe and / or capture an image of the user that may be the basis for determining physical characteristics. The user may be instructed to activate or pose towards the camera, as detailed herein. The information collected by the input components may be utilized and / or stored for making recommendations.

The server 108 is configured to determine the physical characteristics from the input image, determine the correlation between the physical characteristics and the product, and recommend the product. The server may make correlations and recommendations based on various collected data such as user images, product images, user browsing habits, user profiles, etc., as detailed herein. Making recommendations based on the physical characteristics of the user has many uses. Examples of applications for making recommendations based on the user's physical characteristics include the virtual fitting of facial accessories such as eyeglasses, cosmetics, and jewelery.

FIG. 3 is a block diagram showing an example of a system for making recommendations based on the physical characteristics of the user. The system 300 may be included in the server 208 of FIG. In this example, the system 300 includes a sensor data storage 316, a product information storage 312, a user information storage 318, a recommendation engine 314, and an output engine 312.

The sensor data storage 316 is configured to store sensor data such as image / depth data, still images, video frames, and related data from various sensors. The sensor data may be relevant to various users. In some embodiments, image storage stores images recorded by a camera or otherwise captured. The object of the image may be an object such as a (human) user. Images may be stored, for example, in association with a particular user or user group associated with a user profile. The image stored in the sensor data storage 316 may be retrieved and processed according to the methods described herein to determine the physical characteristics of the object in the image.

The product information storage 312 is configured to store information about inventories. The inventor may include a set of merchandise available to the user, such as an eyeglass frame that can be purchased. For example, product features such as color, shape, and size may be stored. A digital representation (such as the 3D coordinates of the vertices) representing the 3D model of the product may be stored. Each item may be stored with one or more predetermined labels. As detailed herein, a label is a description of the type or characteristic of a product and may allow the product to be grouped with other products. Examples of eyeglass frame labels are characteristics of physical attributes such as "plastic" or "metal", or style / human perception-based classifications such as "scholarly", "serious", and "sporty". including.

In various embodiments, product sales information may be stored. Sales can be related to the popularity of a product or product group from a particular user or user group.

The user information storage 318 is configured to store information related to a specific user. For example, a user profile describing the user may be stored, such as the physical properties of the object, past purchases, preferences, and history of interactions with the user interface (eg, web browsing).

The recommendation engine 314 is configured to make recommendations based on at least a portion of the information stored by the sensor data storage 316, the product information storage 312, and / or the user information storage 314. In various embodiments, the recommendation engine is configured to determine a user's physical characteristics based at least in part on the user's image, such as an image of the user's face or head. Images may be extracted in real time from the client device and / or from the sensor data storage 316. In some embodiments, physical characteristics are determined directly from the image. In some embodiments, the physical characteristics are determined from the 3D model or the user's head, where the 3D model is drawn from a set of images as detailed herein. Product recommendations are generated, at least in part, based on the determined physical characteristics. The recommendation engine may be configured to make recommendations based on several factors, such as the similarity of physical characteristics of different users and the similarity of customer preferences.

In various embodiments, the recommendation engine makes recommendations based on the similarity between the physical characteristics of different users. For example, if one user purchases or is interested in a particular product, he or she can recommend the same product to another user with similar physical characteristics.

In various embodiments, the recommendation engine makes recommendations by adapting to the tastes of the customer. Adaptation may be real tom. For example, the recommendations displayed to a user may be updated when the user scrolls down a web page or navigates from one web page to another. When a user browses a web page, information about the user (such as a reaction to a product) may be collected. In some embodiments, this information may be used immediately to update content that the user expects to access later, such as a portion of a web page that is not yet visible to the user. In some embodiments, the information collected about the user is stored for later use, such as by associating it with a user profile. Examples of updating the graphical user interface (GUI) according to customer preferences are shown in FIGS. 13A-13C, 14A, 14B, 15A, and 15B.

In some embodiments, a group of styles (also referred to as "style clusters") is recommended to the user. Depending on user feedback about the style group, the products in the style group may be displayed. For example, scholarly and sporty style recommendations can be made to the user. In various embodiments, categories are determined for individual product groups (eg, for each e-commerce site). Categories can be pre-created using product characteristics (eg, Material: Metal / Plastic, Style: Rectangle / Oval, Color, Size). The product characteristic forms a multidimensional space, in which a clustering technique (for example, K-means clustering) is applied to find a cluster of products. The style expert then assigns each cluster a label (human-friendly in various embodiments): "scholarly", "sporty", "serious" and so on. When the clusters are presented to the customer, the recommendation engine (eg, the recommendation engine 314 in FIG. 3) chooses the most recommended glasses of N samples (eg, N = 30), and which cluster has these N samples. Search for and order clusters based on score. The score is calculated, for example, by calculating the number of glasses derived from N samples in each cluster, and can be calculated based on a weighted sum or the like.

If the user chooses a sporty style, the products in the "sporty style" group that suit the user are displayed (eg, as determined by the methods detailed herein). Style cluster labels (eg, "scholarly") may be created by humans or computers. In one aspect, by presenting some aggregated products rather than individual products, users can focus on a group of products without overwhelming the user with too many individual products. By making it possible, the user experience can be improved.

The recommendation engine may be trained to improve recommendation decisions over time. For example, a good recommendation is one that the user tends to follow. Over time, the recommendation engine may be trained to make recommendations that the user is likely to follow. An example of the process for training the recommendation engine is shown in FIG.

The output engine 312 is configured to provide output (recommendations, etc.). The output engine may include an explanation of why the individual recommendations were made, such as the factors that caused the particular recommendations to be made, as detailed herein. The output engine 312 may provide output in a variety of media, such as the user interface of a client device (eg, client device 204 in FIG. 2).

In some embodiments, the system 300 comprises a 3D model engine 306. The morphing model engine 306 is configured to generate a three-dimensional (3D) model from the input image. In various embodiments, the 3D model engine is configured to generate a 3D model of the user's head from a set of images. In various embodiments, the 3D model engine is configured to generate a 3D model of the product from the product description or image. The morphing model may be used to extract the physical characteristics of the user, as detailed herein.

3D models (also referred to as "morphing models") may be generated according to various techniques. For example, a cost function in which an input image of the user's face to be modeled is received and a set of parameters to be input into the statistical model to correspond to the input image is determined based at least in part on the optical flow. Generated by evaluating candidate parameter values using. A further example of 3D model generation will be described with reference to FIGS. 5-7.

In various embodiments, a subset of PCA components define various physical characteristics. For example, one subset of PCA components defines the shape of the face, and another subset (which may or may not overlap the first subset) defines the proportion of the face.

FIG. 4 is a block diagram showing an embodiment of the recommendation engine. In this example, the recommendation engine 400 includes a face data scoring system 402, a browsing data scoring system 404, a user profile scoring system 406, and a correlator 410.

The face data scoring system 402 is configured to provide scores based on the user's physical characteristics (such as facial features). The score may be used to determine a recommendation for the user. As detailed herein, facial features include facial structure or proportions, facial shape, iris color, hair color, skin color, and skin pattern. The recommendation may be score S1.

The browsing data scoring system 404 is configured to provide recommendations based on user behavior such as interaction with the website, purchase history, browsing behavior, and the like. For example, when a user stays in a particular product for a long time, that behavior can suggest that he or she is interested in that product. This interest may be associated with the user and a product similar to that product may be recommended to the user.

The user profile scoring system 406 is configured to provide recommendations based on customer attributes that are not physical characteristics. Examples of attributes include demographic data such as location, income, education, age, and gender. Non-biometric attributes may influence the recommendation. For example, a user living in a sunny state is more likely to want sunglasses and the right product may be recommended to the user.

The correlator 410 is configured to determine the correlation between the user's features (facial features, user history, etc.) and the product. The correlator may determine the correlation based on product characteristics such as price and color. The correlator may determine the correlation based on one or more of the following factors: facial similarity (content-based filtering), site interaction, and / or in-store purchase history ( Collaborative filtering) etc. The correlator may combine the scores output by each of the scoring systems 402-406. In some embodiments, the scores are weighted when combined. The weighting of the system may be dynamically adjusted or predetermined. For example, the user may want to emphasize style, facial enhancement, fit, and so on. The weights assigned to the scores S1, S2, and S3 may be adjusted accordingly.

In some embodiments, the face data (score S1) is weighted heavier than the other scores, as the face data can be considered more important for making recommendations. For example, 50% is allocated to face data and the remaining 50% is distributed between user data and user profile data.

The weighting of the face data may be adjusted. For example, if the user is not satisfied with the recommendation, the weight given to the face data is reduced. In some embodiments, the weights are dynamically adjusted in response to the user's interaction with the web page, such as after the user clicks on the product (which may indicate that they are interested in the product). Will be done. In some embodiments, the weights are dynamically adjusted in response to the user session. For example, information collected during a user's browsing session (such as while the user is visiting a website for 15 minutes) is processed to adjust the weights based on the user's interaction with the website. Staying on the part of a web page where a user displays a particular product can suggest that they are interested in those product.

In some embodiments, a 3D model of the user's face and a 3D model of the product are used to perform the correlation. For each of the face and the product, the 3D morphing techniques described herein are applied to extract a 3D model containing the PCA components of each of the face and the product. Each vertex and PCA component of the face and goods and purchase history is provided to the correlator to obtain recommendations for the goods.

In various embodiments, the recommendation engine is configured to output a list of recommended products. In some embodiments, the list is a ranking list of merchandise, within which the merchandise may be displayed in the order in which they are ranked. The recommendation engine may be configured to apply one or more rules to further filter recommendations. For example, the recommendation engine may be configured to display a particular brand, goods in a certain price range, and the like.

In some embodiments, deep neural networks may be used to find the correlation between the user's face and the purchase. Data related to the user's face, purchase history, and product information (product features, etc.) are stored. Deep neural networks may be trained to determine facial and product information and purchase history combinations (relationships, correlations). In other words, a data model (deep neural network) may be constructed to describe which face buys which product. The data model may be used to make recommendations to users who have no or short purchase history. For example, a known face similar to the face of a new user is identified, and a product is recommended for the face of the new user based on the purchase history of the known face. Products may be recommended based on a history of user interactions with the website.

FIG. 5 is a flowchart showing an embodiment of a process for 3D modeling of a user's face. Process 500 may be performed at least partially on one or more components of system 200 of FIG. In some embodiments, process 500 is performed by the 3D model engine 306 of FIG. In some embodiments, at least a portion of the process of FIG. 5 is included in step 102 of FIG.

In step 502, the input image of the modeled user's face is received. The input image of the user's face includes a captured or recorded image of the user's face. In various embodiments, the input image of the user's face may include a frontal image of the user's face, a side image of the user's face, or an image of the user's face at an angle between the front and side images.

In step 504, a set of parameter values to the statistical model corresponding to the input image is generated by evaluating the candidate parameter values using a cost function determined at least partially based on the optical flow. In various embodiments, one or more shape components and one or more to generate a 3D model of a face that matches the user's face in the input image (eg, optimally or within a given tolerance). The parameter value input to the PCA model including the texture component of is determined. In some embodiments, in addition to the parameter values entered into the PCA model, the focal length of the camera, the rotation of the camera (eg, in 3D space), and the camera (eg, in 3D space). Additional parameters related to parallel movement are also determined. In various embodiments, the parameter values and camera parameter values input to the PCA model are iterated through various candidate sets of parameter values to generate a 3D model of the face using the candidate sets of parameter values to generate a candidate composite image. Is determined by projecting a 3D model of the generated face onto a 2D surface to generate and calculating the cost between each candidate composite image and the input image. In various embodiments, the cost between each candidate composite image and the input image is determined based on the optical flow from the candidate composite image to the input image.

FIG. 6 is a flowchart showing an embodiment of a process for 3D modeling of a user's face. Process 600 may be performed at least partially on one or more components of system 200 of FIG. In some embodiments, process 600 is performed by the 3D model engine 306 of FIG. In some embodiments, at least a portion of the process of FIG. 6 is included in step 102 of FIG.

In step 602, a plurality of 3D scans of the user's face are input into principal component analysis (PCA). PCA is applied to 3D scans of the faces of many (eg, 200) different users.

In step 604, a PCA model containing the plurality of components and the parameter value ranges corresponding to each of the plurality of components is received from the PCA. PCA is a statistical method for identifying a set of linearly uncorrelated variables (called "principal components") in input data. Each principal component represents the variance in the input data and is orthogonal to each of the other principal components determined from the same input data. In various embodiments, the resulting PCA model comprises a set of shape components and a set of texture components that describe the variance of the shape and texture in the input 3D scan of the user's face, respectively. The combination of shape and texture components forms a 3D morphing model of the face. In some embodiments, each shape component is a vector of 40,000 (x, y, z) coordinates describing the 40,000 vertices of a 3D model of the user's face. In some embodiments, each texture component is a vector of 40,000 (r, g, b) values that describes the colors of the 40,000 corresponding vertices of the 3D model of the user's face. In addition to the set of shape components and the set of texture components, the PCA also outputs a range corresponding to each shape component or each coefficient associated with the texture component. One set of shape and texture coefficients produces a particular face using a 3D morphing model, and the second set of shape and texture coefficients generate another specific face, which the model says " That's why it's called "morphable".

In various embodiments, different values are assigned to the coefficients associated with each shape component or texture component, the scaled shape components are combined to generate a combined face shape, and the scaled texture components are combined to create a combined face texture. And by combining the combined face shape and the combined face texture, one different 3D model of the face may be generated. The shape and texture coefficients corresponding to the shape and texture components of the PCA are included in the set of parameter values determined in the various embodiments described herein. In some embodiments, the PCA model is configured to include 20 shape components and 20 texture components. Therefore, 20 shape coefficients and 20 texture coefficients are included in the set of parameter values determined in some embodiments described herein.

FIG. 7 is a diagram showing an example of an input to the principal component analysis and an output from the principal component analysis for generating a 3D morphing model of the user's face. As described in process 600 of FIG. 6, various 3D scans of different user faces 704 are input to principal component analysis 706 and principal component analysis 706 is performed using one or more software and hardware. Good. Principal component analysis 706 then produces output 708. The output 708 contains a set of shape components S ₁ , ..., _SN , where S _i = (x ₁ , y ₁ , z ₁ , ..., X _n , y _n , z. _n ), where x, y, and z are the 3D positions of the vertices. Output 708 is further set texture component _T 1 of the, ..., it includes a _{T N, where, T i = (r 1,} g 1, b 1, ···, r n, g n , B _n ), and collectively r, g, and b are the colors of the vertices. In some embodiments, both N and n are configurable values. In some embodiments, N = 20 and n = 40,000. _{The output 708 further has a range of each shape coefficient α 1} , ···, α _N corresponding to each shape component and a range of each texture coefficient β ₁ , ···, β _{N corresponding to each texture component.} And include.

In some embodiments, the average shape _Savg (eg, a vector with 3D coordinates describing the 3D positions of 40,000 vertices) is a variety of different user faces 704, apart from principal component analysis 706. Generated from a 3D scan. In some embodiments, the average texture _Tavg (eg, a vector with r, g, and b values describing the colors of 40,000 vertices) is of a different user, separate from the principal component analysis 706. Generated from various 3D scans of the face.

In various embodiments, the shape and texture components of output 208 may be combined with the average shape and texture components to generate a new 3D face shape and new 3D face texture based on the following equation: :

S _new = S _avg + Σ _i α _i S _i (1)

T _new = T _avg + Σ _i β _i T _i (2)

_{Here, S new new} represents a new 3D face _{shape, S avg} represent the average shape, alpha _i represents the shape factor corresponding to _{S i, S i} represents the PCA shape _{component, T new new} represents a new 3D facial _{texture, T avg} represents the average texture, beta _i represents a texture coefficient corresponding to _{T i, T i} denotes the PCA texture component.

S _new and T _new may then be combined into a mesh to generate a new 3D model of the face. For _{example, S new new} and _{T new is} the vertices of _{S new new,} may be combined by assigning _{T new} color corresponding to the same vertex.

In some embodiments, the shape coefficients α ₁ , ···, α _{N = 20} and the texture coefficients β ₁ , ···, β _{N = 20} of the PCA model are included in the set of parameter values to be determined.

FIG. 8A shows an example of reference points obtained from a set of images of the user's face. Reference points are used to define the location of various facial features and generate a 3D model of the user's face. FIG. 8A shows image 800, where the user is facing forward, reference point 802 is at the inner corner of the user's right eye, and coordinates (x0, y0) are assigned. The eye reference point may further include a point 804 at the inner corner of the left eye at coordinates (x1, y1) and a point 806 at the outer corner of the left eye at coordinates (x2, y2). The bridge distance 808 can be determined from the two reference points of the right inner canthus 802 and the left inner canthus 804. In some embodiments, the point 804 at the inner corner of the left eye and the point 806 at the outer corner of the left eye can be used to determine the lens distance (resulting in a distance of 810).

FIG. 8B shows an example of reference points obtained from a set of images of the user's face. FIG. 8B shows another image (ie, right side image 820) containing the user's face in a different orientation, at reference point 822 and coordinates (y3, z3) at the right corner of the eye at coordinates (y4, z4). Includes an upper point 824 where the helix connects to the head. From the profile picture, the z coordinate can be added to the 3D model of the user's face. From the 2D video frame of the user's face, the 3D model of the user's face can be determined.

In some embodiments, reference points from 2D images (eg, those shown in FIGS. 8A and 8B) are combined to create a 3D model of the user's face. Corresponding reference points are combined to create a set of (x, y, z) coordinates representing each position of a reference point (eg, the position of a facial feature) on the user's face / head. For example, the (x, y) coordinates of the right outer corner 812 from the forward-looking video frame 800 of FIG. 8A are combined with the z-coordinate of the reference point 822 of the right outer corner of the right eye in the right-facing image / video frame 820 of FIG. You can get the reference points. The 3D model of the user's face includes 3D points (corresponding (x, y, z) coordinates) such as points inside the eyelashes, outer corners of the eyes, inner corners of the eyes, joints of the ears, cheekbones, and tips of the nose.

FIG. 9 is a diagram showing an example of visualization of a predetermined 3D face in 3D space. In some embodiments, a given 3D face is: Includes obj file. In the example of FIG. 9, the predetermined 3D face 900 includes a triangular polygon that defines the shape of the facial features of the predetermined 3D face. In some embodiments, each polygon contains an associated color that represents the color of that area of the face. In the example of FIG. 9, the predetermined 3D face 900 includes triangular polygons, but in an actual embodiment, the predetermined 3D face may include polygons of other shapes and / or other number of vertices.

FIG. 10 is a flowchart showing an embodiment of a process for determining the description of the recommendation. Process 1000 may be performed at least partially on one or more components of system 200 of FIG. In some embodiments, the process 200 is performed by the recommendation engine 314 and / or the output engine 312 of FIG. In some embodiments, at least a portion of the process of FIG. 10 is included in step 104 and / or step 106 of FIG. An example of the output is shown in FIG.

The process performs a 3D reconstruction of the face based on one or more images (step 1002). The 3D reconstruction may be performed according to the method described herein. For example, a 3D model of the head may be generated from the image. The pose of the face may be determined from the rotation matrix and the translation vector. Pose evaluation may be performed on the face based on the 3D model and the camera's unique parameters.

The process determines the description of the physical characteristics based on the 3D reconstruction (step 1004). Facial features such as face shape, facial structure or proportions, eye color, hair color, skin color, and / or texture may be determined. The description of choosing a spectacle frame with a particular nasal pad structure may be explained by the facial features of a relatively narrow and short nasal bridge. The particular nasal pad structure of the spectacle is suitable for that type of nasal bridge (eg, a good fit) and can combine comfort with an aesthetic fit centered on the eye within the frame.

The process outputs a description (step 1006). The description may be output with the recommendation in order to provide the user with more content and information as to why the recommendation was made. Returning to the example of a nasal pad suitable for a relatively narrow and short nasal bridge, this physical attribute is output in the GUI to inform the user that the frame is suitable for the user because of the structure of the user's nasal bridge. Good.

FIG. 11 is a block diagram showing an embodiment of a face data scoring system. In this example, the face data scoring system 1100 comprises classifiers 1102-1106. Each of the classifiers may be configured to output a score based on the received features. For example, the classifier 1102 receives the feature 1 and outputs the score of the feature, the classifier 1104 receives the feature 2 and outputs the score of the feature, and the classifier 1106 receives the feature 3. Outputs the score of the feature. In various embodiments, the classifier is implemented by a machine learning model so that more training data is input to the classifier and the scoring improves over time.

As shown in the figure, various numbers of classifiers may be provided to the system, each adapted to handle its own feature type. Each feature type includes face shape, skin color, eye color, hair color, face proportions, and the like. In some embodiments, one or more feature types may be determined based on the 3D surface of the face.

The shape of the face may be detected as follows. A classifier is configured to determine the shape of the face based on the features received. The classifier determines the contour of the face, where the contour is a set of points. The contour may be determined using a library or conventional methods. A classifier is configured to assign a score to each face shape. The score vector describes the contour of the face. In some embodiments, the shape of the face is described by the extent to which the face falls under a given category or label, such as circular, oval, heart-shaped, triangular, and square. For example, the face may be 40% quadrangular and 60% oval and need not be classified as 100% oval. The classifier is configured to assign scores by determining the degree of coincidence or similarity between a set of facial contour points and a set of known points. If the degree of coincidence or similarity is within the threshold value, this means that the shape is an ellipse at a predetermined ratio, a quadrangle at a predetermined ratio, and the like.

In various embodiments, areas of the skin define the contours of the face. For example, the area of the skin is a determination of where the hair ends and the skin begins, where the background ends and the skin begins, and so on.

In some embodiments, facial features are extracted directly from the image. In some embodiments, facial features are extracted from the morphing model as detailed herein. The detected facial contours may be input to the machine learning model associated with face shape detection.

The skin pattern is a measurement of skin texture, bruises, texture roughness, and the like. Product recommendations may be adapted to the user's skin pattern. For example, a unique skin pattern may influence the selection of recommended products. In various embodiments, the pattern of the skin is determined by sampling pixels from the area representing the skin. In some embodiments, the 3D morphing head provides information about the skin area and narrows the area where the pixels are sampled to determine the skin pattern. For example, the image is divided into skin and non-skin areas (eg, hair, clothing, background, etc.), and then the skin areas may be sampled to determine the pattern of the skin.

In various embodiments, a classifier is configured to determine skin color. Skin color describes the color of at least a portion of the user's face. The skin color may be described in various color spaces such as RGB and CIELAB. Skin color may be determined by sampling colors from different areas of the user's face, at least partially based on a set of images of the user's face.

The skin color may be determined taking into account the lighting environment in which the image was recorded. In various embodiments, for each image in the image set associated with the user's face, the morphed 3D general model transformed with the set of ancillary information associated with that particular image is associated with the particular image. A similarly transformed 3D spectacle model is fitted with a set of part information, and the shadow cast by the transformed 3D spectacle model onto the transformed 3D general model is simulated in a given lighting model. In some embodiments, the lighting model is determined based on the user's particular lighting environment in the set of images. For example, a 3D spectacle model relates to spectacles selected to virtually try on the user's face. In various embodiments, a shaded 2D face image is generated that includes the shadow cast by the spectacle model on the transformed 3D general model. In various embodiments, for each image in the image set associated with the user's face, at least in part, based on the difference between the corresponding 2D face image of each image and the corresponding shaded 2D face image. The corresponding shadow transformation is generated. In various embodiments, a shadow transform corresponding to an image in the image set associated with the user's face is applied to the image to render / simulate the shadow cast by the selected eyeglasses on the user's face in the image. can do. The image of the user's face with the corresponding shadow transform applied, along with a 2D image overlay of the selected eyeglasses, to simulate the virtual fitting of the selected eyeglasses to the user's face, along with the user's specific lighting. The selected eyeglasses can be presented with a real shadow cast on the user's face, as simulated using an environment-based lighting model.

In some embodiments, facial features are extracted directly from the image. In some embodiments, facial features are extracted from the morphing model as detailed herein. For example, the skin color may be determined from the morphing model based on the color of the polygons. The sampled color may be input to a machine learning model involved in determining skin color.

In various embodiments, a classifier is configured to determine eye color. Eye color may be determined by sampling colors from various areas of the user's eyes (such as the iris), at least in part based on a set of images of the user's face. In some embodiments, facial features are extracted directly from the image. In some embodiments, facial features are extracted from the morphing model as detailed herein. For example, a 3D morphing head may be used to narrow the area where pixels are sampled to determine the color of the eye by locating the eye. The sampled colors may be input to the machine learning model involved in determining eye color.

In various embodiments, a classifier is configured to determine hair color. Hair color may be determined by sampling colors from different areas of the user's hair, at least partially based on a set of images of the user's face. In some embodiments, facial features are extracted directly from the image. In some embodiments, facial features are extracted from the morphing model as detailed herein. The sampled color may be input to a machine learning model involved in determining hair color.

In various embodiments, the classifier is configured to determine the structure of the face (also referred to as the face ratio). The face ratio may be determined by extracting the ratio between at least two facial feature distances, at least partially based on a set of images of the user's face. For example, the classifier may receive other measurements or ratios of facial width and facial features. For example, (i) the ratio between the distance between the outer corners of the eyes and (ii) the distance from the tip of the nose to the bridge line of the nose, where the bridge line is the line that crosses the nose and connects to the inner corner of the eye. ..

In some embodiments, facial features are extracted directly from the image. In some embodiments, facial features are extracted from the morphing model as detailed herein. The sampled colors may be input to the machine learning model involved in determining the face ratio.

In various embodiments, the classifier is configured to determine the 3D surface of the face. The 3D surface of the face is a description of the facial structure measured by obtaining positions or vertices from the user's facial morphing model. Various features may be determined from the 3D model of the face. For example, the shape of the face may be determined by extracting features from a 3D morphing model of the head. For example, the bridge of the nose, cheekbones, pantoscopic tilt (angle of the lens with respect to the face), and other features may be determined from the 3D morphing model. The fit of the spectacles and how the spectacles look on the face of a particular user may be evaluated based on these factors. Eyeglasses that don't fit well or don't look good on the user's face may or may not be prioritized when making a recommendation list.

In some embodiments, the apex coordinates and colors of the 3D surface of the face may be extracted from the morphing head to train the classifier to identify facial features assuming a 3D morphing head. The coordinates of the vertices may be polygons forming the morphing head. In some embodiments, the morphing head may include a number of coordinates forming the polygon (eg, on the order of 20,000 points). Each polygon may have a related color. Coordinates are supplied to a data model or classifier (which may be implemented in a machine learning model). The coordinates of the vertices and the label of the goods are supplied to the classifier. The label may be pre-determined, assigned, or automatically determined with the goods as a representation. The classifier determines the correlation between vertices and labels.

In some embodiments, the 3D morphing head is rendered from several angles in order to train the classifier to identify facial features assuming a 3D morphing head. For example, a library of images may be created from heads rendered from different angles. Contours may be detected from the head rendered at various angles. Face shape or face shape scoring may be assigned based on the detected contours.

The 3D surface of the face may be used to determine the degree of similarity between the faces. For example, two users may share features with relatively wide eyes, suggesting that both fit a particular type of eyeglasses.

The acquired positions or vertices may be input to the machine learning model involved in determining the 3D surface of the face. In some embodiments, the correlation between the shape of the spectacles and the 3D surface of the face may be determined. Correlation may be used to recommend eyeglasses to the user or other users with similar 3D facial surfaces.

In various embodiments, the output of the combined model is generated from the output of classifiers 1102-1106. The corresponding outputs of each of the multiple trained opportunity learning models associated with multiple facial feature types are combined. The combined output may be correlated with multiple products to select a subset of the products for further analysis. In one aspect, the recommended potential products are narrowed down by correlation processing. In some embodiments, additional types of recommendations, collaborative filtering, etc. may be used to determine the recommendations. An example of the correlation is shown in FIG. A part of the product may be presented.

FIG. 12 is a flowchart showing an embodiment of a process for determining the characteristics of facial features using a classifier. Process 1200 may be performed at least partially on one or more components of system 200 of FIG. In some embodiments, the process 200 is performed by the recommendation engine 314 of FIG. 3 and / or the classifier 1102-1106 of FIG. In some embodiments, at least a portion of the process of FIG. 12 is included in step 104 of FIG.

The process extracts facial features corresponding to the type of facial features based on the received image (step 1202). Facial feature extraction may be based on images and / or other sensor data (such as depth). In various embodiments, facial feature extraction may be performed for several types of facial features. For example, 704 may be repeated for each facial feature until no untreated facial features remain.

The process inputs the extracted facial features into a classifier corresponding to the type of facial features (step 1204). The classifier may be implemented by a trained machine learning model. For example, face features related to face shape are input to the face shape classifier. The face shape may be systematically determined by a classifier that is trained on the face shape data and can automatically detect the face shape based on the input data.

The process obtains output from the classifier corresponding to the type of facial feature (step 1206). Returning to the face shape example again, the classifier outputs the percentage of the input face that fits into various shape categories (circle, ellipse, rectangle, diamond, etc.).

Although described mainly using the example of eyeglasses, the techniques described herein have applications in various environments such as virtual fitting of earrings, hats, makeup, shoes, clothing and the like.

FIG. 13A is an example of a GUI of a product catalog for browsing inventory items. The GUI 1300 includes an optional navigation area 1302, which may include menus, filters, etc. to assist in navigating the inventory displayed on the GUI. In this example, the GUI 1300 displays a product catalog that includes many spectacle frames. Here you can see six eyeglass frames (models AA, AB, AC, BE, BX, and CD). The user can interact with the GUI 1300 to review the goods and details. For example, additional eyeglasses may become visible on the GUI 1300 as the page scrolls down using the scroll bar 1304. One or more additional eyeglass frames may be loaded as the user scrolls down the page. In various embodiments, the spectacle frame loaded in response to the user scrolling down the page may be determined at least in part based on the techniques described herein. For example, eyeglass recommendations may be adapted to the customer's preferences, and at least a portion of the preferences may be learned based on the customer's browsing habits.

FIG. 13B is an example of a GUI showing a product displayed according to the user interaction with the GUI of FIG. 13A. When the user scrolls down by using the scroll bar 1304 of the GUI 1300, the group 1330 of the six spectacle frames (models DA, DB, DR, EA, EF, and EL) may be displayed. Group 1330 may include one or more recommended products selected at least in part based on the techniques described herein, such as products selected by the recommendation engine 314 of FIG. In some embodiments, the group 1330 is preloaded (eg, buffered) to facilitate rapid loading of goods in anticipation of the user scrolling down the GUI 1300.

FIG. 13C is an example of a GUI showing a product displayed according to the user interaction with the GUI of FIG. 13A. When the user scrolls down, such as by using the scroll bar 1304 of the GUI 1300, the group 1350 of the six spectacle frames (models AD, AG, AL, RV, RM, and RP) may be displayed. In some embodiments, the group 1350 is preloaded (eg, buffered) to facilitate rapid loading of goods in anticipation of the user scrolling down the GUI 1300.

Group 1350 may include one or more recommended products selected at least in part based on the techniques described herein, such as the products selected by the recommendation engine 314 of FIG. In this example, group 1350 includes recommendations made in response to user selection of model AC (shown in GUI 1300). For example, the user can click on the model AC to load the product page associated with that model and return to the catalog page 1300. The group 1350 may be loaded so that when the user returns to the catalog page 1300, the group 1350 is displayed when the user scrolls down the GUI 1300. The products of Group 1350 may be selected based on the model AC according to the techniques described herein. For example, group 1350 includes rectangular frame eyeglasses similar to model AC. In one aspect, the assembly of groups 1330 and 1350 provides a seamless user experience as the user is unaware that the goods are re-sorted or updated as they are viewed. As the user scrolls down the web page, hidden products (eg, products at the bottom of the product list) can be updated to dynamically adapt to the user's preferences as the user browses.

FIG. 14A is an example of a GUI of a product catalog for browsing inventory items. The GUI 1400 includes an optional navigation area 1402, which may include menus, filters, etc. to assist in navigating the inventory displayed in the GUI. In this example, the GUI 1400 displays a product catalog that includes many spectacle frames. Here, six eyeglass frames (models A, B, C, D, E, and G) are shown. The user can interact with the GUI 1400 to review the goods and details. For example, the product displayed on the GUI 1400 may be updated according to the user selection of the model C.

FIG. 14B is an example of a GUI displayed according to the user interaction with the GUI of FIG. 14A. For example, the user can click model C (displayed in GUI1400) to load the product page associated with that model and return to the catalog page. When the user returns to the catalog page, GUI1450 may be loaded to display the updated product recommendations. Here, the models M, K, and X are replaced with the models C, D, and G of the GUI 1400 according to the user selection of the model C. Models M, K, and X may be selected based on model C according to the techniques described herein. For example, the model CD is a wireframe. Models M, K, and X are wireframes that replace the plastic frame eyeglasses of models C, D, and G. In various embodiments, a portion of the area of GUI1450 is designated to accept replacement or updated product recommendations. These areas are also referred to herein as "recommended blocks". For example, recommendation blocks 1452, 1454, and 1456 are used for updating, but goods (such as model D) displayed in other areas remain. In some embodiments, a more visible area is selected to accept product recommendation updates. In other words, the product recommendation can be incorporated into the product displayed in the GUI and dynamically updated according to the customer's preference.

FIG. 15A is an example of a GUI of a product catalog for browsing inventory items. The GUI 1500 includes an optional navigation area 1502, which may include menus, filters, etc. to assist in navigating the inventory displayed in the GUI. The GUI 1500 includes a recommendation block 1504 on which the recommended products (here, four eyeglass models) are displayed. In this example, the GUI 1500 displays a product catalog that includes many spectacle frames. Here, six eyeglass frames (models A, B, C, D, E, and G) are shown. In addition, four spectacle frames (models L, P, Y, and A) are displayed on the recommendation block 1504.

The recommendation block can be updated to display other recommended products that match the user's tastes. For example, the product displayed in the recommendation block 1504 may be updated according to the user selection of the model C.

FIG. 15B is an example of a GUI displayed according to the user interaction with the GUI of FIG. 15A. For example, the user can click model C (displayed in GUI 1500) to load the product page associated with that model and return to the catalog page. When the user returns to the catalog page, the GUI 1550 may be loaded to display the updated product recommendations in recommendation block 1504. Here, the models L, P, Y, and A of the GUI 1500 are replaced with the models X, H, Q, and S shown in the GUI 1550, depending on the user selection of the model C. Models X, H, Q, and S may be selected based on model C according to the techniques described herein. For example, model C is a rectangular frame. Models X, H, Q, and S are rectangular frame eyeglasses that users interested in model C may be interested in.

FIG. 16 is an example of a GUI showing the output of recommendations based on the physical characteristics of the user. The GUI 1600 displays a captured image of the user's face and a recommendation. Here, the recommendation is a round frame. A description of this recommendation is displayed. Here, the reason for recommending the round frame is that the main face shape of the user is a quadrangle. That is, the user's face may be displayed in various shapes, the main shape being a quadrangle. The round spectacle frame in this example looks best on a square face.

Although the above embodiments have been described in some detail for ease of understanding, the present invention is not limited to the details provided. There are many alternative ways to carry out the present invention. The disclosed embodiments are exemplary and are not intended to be limiting. For example, it can be implemented as the following application example.
[Application example 1]
A system with a processor
The processor
Determining the physical characteristics of the user, at least partially based on the user's image,
Determine the correlation between the physical characteristics and the product and
A system configured to generate product recommendations based at least in part on the determined correlation.
[Application Example 2] The system according to Application Example 1, wherein the determination of the physical characteristics includes a fit of the product to the user.
[Application Example 3] The system according to Application Example 1, wherein the physical characteristics include a facial shape, and the shape is defined with respect to the degree to which the shape fits into a plurality of categories.
[Application Example 4] The system according to Application Example 1, wherein the physical characteristics include a facial ratio.
[Application Example 5] In the system according to Application Example 1, the physical feature includes at least one of eye color, skin color, and hair color, and the color is at least the above. A system in which one image is at least partially normalized to the captured lighting environment.
[Application Example 6] The system according to Application Example 1, wherein the determination of the correlation is based on at least one weighted combination of the physical characteristics, user browsing data, and user profile data.
[Application Example 7] In the system according to the application example 6, the weight assigned to at least one of the physical characteristics, the user browsing data, and the user profile data is output from the machine learning model. Based system.
[Application Example 8] The system according to the application example 6, wherein the physical characteristics are weighted more heavily than at least one of the user browsing data and the user profile data.
[Application Example 9] The system according to Application Example 6, wherein the physical characteristics, the user browsing data, and the weights assigned to the user profile data are adjustable.
[Application Example 10] The system according to Application Example 6, wherein the processor is further configured to adjust the weights used in the weighted combination in response to user feedback regarding the product recommendation.
[Application Example 11] The system according to Application Example 1.
The processor further obtains a set of parameter values for the statistical model corresponding to the image of the user by evaluating the candidate parameter values using a cost function determined at least partially based on the optical flow. It is configured to generate a three-dimensional (3D) model of the user's head based on the image of the user, including generating.
The physical characteristics are determined, at least in part, by extracting parameters from the 3D model.
[Application 12] In the system according to Application 11, the statistical model is derived from principal component analysis based at least in part on a 3D scan of the user's face, and the statistical model is based on shape components and A system that contains texture components.
[Application Example 13] The system according to Application Example 11, wherein the 3D model is a mesh model including polygons that define at least one of a facial feature shape and a head shape.
[Application Example 14] The system according to the application example 11, wherein the determination of the physical characteristics includes determining the skin color based on the polygon color of the 3D model.
[Application Example 15] In the system according to the application example 1, the generation of the product recommendation is determined based on the attribute of the product, and the attribute of the product is determined at least partially based on the 3D model of the product. system.
[Application Example 16] The system according to the application example 15, wherein the generation of the product recommendation is based on the fit of the product to the user's 3D model.
[Application Example 17] In the system according to Application Example 1, the product recommendation includes one group of products.
The processor is further configured to update the product recommendation in response to a user selection of one product in the group.
[Application Example 18] The system according to Application Example 1, wherein the product recommendation includes a description of the product recommendation.
[Application example 19] This is a method.
The process of determining the physical characteristics of the user, at least in part, based on the image of the user.
The process of determining the correlation between the physical characteristics and the product,
With the step of generating product recommendations based at least in part on the determined correlation
How to prepare.
[Application 20] A computer program product embodied in a persistent computer-readable storage medium.
Computer instructions for determining the user's physical characteristics, at least in part based on the user's image.
Computer instructions to determine the correlation between the physical characteristics and the product,
With computer instructions to generate product recommendations based at least in part on the determined correlation
Computer program products equipped with.

Claims (20)

A system with a processor
The processor
Determining the physical characteristics of the user, at least partially based on the user's image,
Determine the correlation between the physical characteristics and the product and
It is configured to generate product recommendations based at least in part on the determined correlation.
The determination of the correlation, the physical characteristics, the user browsing data, and the user profile data,-out based on the weighted combination,
Determining the correlation between the physical characteristics and the goods includes determining the degree of fit of the goods to the user based at least in part on the physical characteristics, the degree of fit. , A system that includes the size of the product size relative to the ratio of the user's face. The system according to claim 1, wherein the determination of the physical characteristics includes a fit of the product to the user. The system according to claim 1, wherein the physical features include the shape of a face, the shape being defined with respect to the extent to which the shape fits into a plurality of categories. The system according to claim 1, wherein the physical feature includes a face ratio. The system of claim 1, wherein the physical feature comprises at least one of eye color, skin color, and hair color, which color is captured by the at least one image. A system that is at least partially normalized to the lighting environment. The system according to claim 1, wherein the weight assigned to at least one of the physical characteristics, the user browsing data, and the user profile data is based on the output from the machine learning model. The system according to claim 1, wherein the physical characteristics are weighted more heavily than at least one of the user browsing data and the user profile data. The system according to claim 1, wherein the physical characteristics, the user browsing data, and the weights assigned to the user profile data are adjustable. The system according to claim 1, wherein the processor is further configured to adjust the weights used in the weighted combination in response to user feedback regarding the product recommendation. The system according to claim 1.
The processor further obtains a set of parameter values for the statistical model corresponding to the image of the user by evaluating the candidate parameter values using a cost function determined at least partially based on the optical flow. It is configured to generate a three-dimensional (3D) model of the user's head based on the image of the user, including generating.
The physical characteristics are determined, at least in part, by extracting parameters from the 3D model. The system of claim 10, wherein the statistical model is derived from principal component analysis based at least in part on a 3D scan of the user's face, and the statistical model is a system that includes shape and texture components. .. The system according to claim 10, wherein the 3D model is a mesh model including polygons that define at least one of a facial feature shape and a head shape. The system according to claim 10, wherein the determination of the physical characteristics includes determining the skin color based on the color of the polygon of the 3D model. The system according to claim 1, wherein the generation of the product recommendation is determined based on the attributes of the product, and the attributes of the product are determined at least partially based on the 3D model of the product. The system according to claim 14, wherein the generation of the product recommendation is based on the fit of the product to the user's 3D model. The system according to claim 1, wherein the product recommendation includes one group of products.
The processor is further configured to update the product recommendation in response to a user selection of one product in the group. The system according to claim 1, wherein the product recommendation includes a description of the product recommendation. It's a method
The process by which the computer determines the physical characteristics of the user, at least in part, based on the image of the user.
The process by which the computer determines the correlation between the physical characteristics and the product,
The computer comprises a step of generating a product recommendation based on at least partly based on the determined correlation, the step of determining the correlation of the physical characteristics, user browsing data, and user profile data. Based on the weighted combination, the correlation was determined and
The step of determining the correlation between the physical characteristics and the goods comprises determining the degree of fit of the goods to the user based at least in part on the physical characteristics of the fit. The degree includes the size of the product size relative to the ratio of the user's face to the method. A computer program, embodied in a persistent computer-readable storage medium,
Computer instructions for determining the user's physical characteristics, at least in part based on the user's image.
Computer instructions to determine the correlation between the physical characteristics and the product,
With computer instructions to generate product recommendations, at least in part based on the determined correlation.
Wherein the computer instructions for determining a correlation, the physical characteristics, the user browsing data, and the user profile data, based on the weighted combination, Ri instructions der to determine the correlation,
Computer instructions that determine the correlation between the physical characteristics and the goods include determining the degree of fit of the goods to the user, at least in part, based on the physical characteristics. The degree of is a computer program that includes the size of the product size relative to the ratio of the user's face. The system according to claim 1.
The processor is further configured to generate a three-dimensional (3D) model of the user's head, at least partially based on the user's image.
A system in which the physical characteristics of the user are determined at least in part based on the generated 3D model.

Images