KR20210091874A - Cosmetics matching service providing system using augmented reality and method thereof - Google Patents

Abstract

A method for providing a cosmetics matching service using augmented reality according to one embodiment of the present invention comprises the steps of: providing a management application to provide a cosmetics color matching test and a similar product purchase service from a management server to a user terminal connected through a communication network; requesting to select an image for testing in the management application executed in the user terminal; selecting a photo including a lip image to be tested from a user album stored in the user terminal; inputting, by the user terminal, a desired color position in the lip image included in the selected photo to request a color development test on the corresponding position; and carrying out, by the management server, coloring of the corresponding position of the actual lip image with a color on the corresponding position in a shape view when the color development test on the selected position is requested and displaying the same on a screen of the user terminal.

Description

Cosmetics matching service providing system using augmented reality and method thereof

The present invention relates to a system and method for providing a cosmetic matching service, and more particularly, to a system and method for providing a cosmetic matching service using augmented reality capable of recommending by color development test and product matching on the lips.

It can be seen that the value of conventional marketing is decreasing in today's flooded with advertisements, unlike in the past.

Therefore, corporate marketing emphasized differentiation from other companies, and 'experiential marketing' focused on the consumer's experience rapidly emerged.

On the other hand, women's 'desire to be beautiful' has been a top concern for all centuries and the world. Recently, as Hallyu has spread beyond entertainment such as dramas and K-pop to K-beauty (Korean cosmetics and cosmetics), 'K-beauty' is positioned as the core of the evolution of Hallyu. The K-beauty craze, which started blowing in China as a base, has now become a global 'phenomena'. Accordingly, there has been a demand for many cosmetics, and the technology and marketing have also developed accordingly. The marketing method of a representative cosmetic company is the aforementioned experiential marketing.

For example, giving a sample to a customer who purchases a product or trying a tester placed in front of the product.

However, New York City dermatologist Karen Burke warns that at cosmetic stores, customers are weakened by contaminated samples, and testers are 100% contaminated on the busiest weekends. In addition, the Korea Consumer Agency, together with the Ministry of Food and Drug Safety, investigated 42 tester cosmetics (16 eyeshadow, 10 mascara, 16 lip products) in 16 cosmetic stores located in places with high floating population. Among 14 products (33.3%), microorganisms exceeding the standard were detected.

In particular, it was found that most of the tester products were exposed for a long time without a lid, and the opening date was not indicated. Through this, it is necessary to further strengthen hygiene management, and there is a need for a function to test and purchase a product without visiting a store.

Virtual/Augmented Reality is suitable as one of the means to realize experiential marketing that meets the requirements to test products and make purchases without visiting a store.

Examples of popular augmented reality include mobile games such as Pokemon GO, and augmented reality adds virtual objects or information to the real environment and synthesizes them as if they exist in the real world to provide various information and to interact with reality and virtual reality. Because it can stimulate the five senses, it has been attracting attention as a technology that will have a big impact in the future and is being actively researched.

In addition, since small electronic devices such as cell phones are distributed to people and can be easily accessed without the need for a special device, there is a big advantage of augmented reality different from virtual reality that requires VR devices, so the spread of augmented reality is faster and the proportion This is expected to increase, and attempts are being made to apply it to cosmetics.

Representatively, in Korea, virtual makeup applications include 'YouCam Makeup' and 'Makeup Plus'. Both of these apps were combined as a single marketing element by allowing users to not only experience domestic and foreign cosmetics they want, but also to purchase them right away.

However, limitations of conventional applications include that only limited makeup products and limited colors within the application can be used, and the interface is relatively complex. It is said that the application that the user can directly apply the desired lip makeup color and experience it is practically insufficient.

Therefore, in order to solve the above problem, the user can check the state of lip makeup virtually through a color test using augmented reality that implements makeup on the lip area, and augmentation including a management application that can purchase products of a desired color A study on a system and method for providing a cosmetic matching service using reality became necessary.

Korean Patent Registration No. 10-1948040 (Registered on February 08, 2019)

An object of the present invention is that a user can check the makeup state virtually through a color development test using augmented reality that implements makeup on the lips directly, and performs a color development test by receiving a product list of a desired color by matching a similar product group, , to provide a system and method for providing a cosmetic matching service using augmented reality in which the product can be purchased through a link to a purchase site.

A method for providing a cosmetic matching service using augmented reality according to an embodiment of the present invention includes: providing a management application for providing a cosmetic color matching test and a similar product purchase service from a management server to a user terminal connected through a communication network; requesting an image selection for a test in the management application executed in the user terminal; selecting a photo including a lip image to be tested from a user album stored in the user terminal; The user terminal inputs a desired color position in the lip image included in the selected photo to request a color development test for that position; When the management server requests a color development test for the color of the selected position, the corresponding position in the shape view screen and implementing color development at the corresponding position of the actual lip image where the position color is displayed, and displaying the color on the screen of the user terminal.

In the above, the management server steps in conjunction with the product purchase site in advance to build a database storing a plurality of product information and product list; Requesting the product to be tested through the user terminal through a management application; The user The terminal step of confirming the product information of the product selected by the request through the management application; The user terminal is provided with a list of products similar to the selected product from the management server and confirming; On the product selected from the product list of the user terminal When a test request is made by inputting a test button, the management server further includes a step of displaying the color image of the corresponding product on a specific lip part selected from the actual image through the shape view screen.

In the above, the steps of determining the purchase of the product after confirming the color of the product through the lip portion display screen in the user terminal; receiving a product information confirmation window for confirming product information of the selected product when the user terminal requests a product for which a purchase decision has been made; The user terminal requests a purchase decision, and the management server connects to a purchase site for purchasing the product.

In the above, the user terminal receiving a product list similar to the product selected for purchase decision from the management server; further comprises.

In the above, the product information is characterized in that it includes a product image, brand name, unique identification code, cosmetic type, color code, product name, product selling price, manufacturer, manufacturing date, and a link to a purchase site for the product.

The cosmetic matching service providing system using augmented reality according to an embodiment of the present invention provides a management application for providing a cosmetic color matching test and a similar product purchase service to a user terminal connected through a communication network, and in the user terminal When a photo including a lip image to be tested is selected from a stored user album to request an image selection for testing in the executed management application, input the desired color position from the lip image included in the selected photo from the user terminal a management server that requests a color development test for a location, and when a color test is requested for a color at the selected location, the color of the location on the shape view screen implements color development at the location of the actual lip image, and displays on the screen of the user terminal; includes

In the above, the management server establishes a database storing a plurality of product information and product list in advance in conjunction with the product purchase site, requests the product to be tested through the management application in the user terminal, and the product of the product After confirming the information, when a color test is requested for the color of the product, the color image of the product is displayed on the lips selected from the actual image through the shape view screen.

The cosmetic matching service providing system and method using augmented reality of the present invention has the advantage that the user can check the desired color through a color test using augmented reality in his or her lip image, so that the user can directly check the color that suits him. .

In addition, by receiving a product list that matches a product group for a specific color and conducting a color test, you can virtually check whether the product is suitable, and help you make a purchase, as well as visit the purchase site for the product. It has the advantage of being able to purchase by linking.

1 is a block diagram showing the overall configuration of a cosmetic matching service providing system using augmented reality according to an embodiment of the present invention.
2 is a flowchart illustrating a user album image selection process according to an embodiment of the present invention.
3 is a flowchart illustrating an algorithm process for recommending a product of a similar color according to an embodiment of the present invention.
4 is a flowchart illustrating a product color development test process of a method for providing a cosmetic matching service using augmented reality according to an embodiment of the present invention.
5 is a flowchart illustrating a product purchase process after testing of a method for providing a cosmetic matching service using augmented reality according to an embodiment of the present invention.
6 is a diagram for explaining a lip detection algorithm in the present invention.
7 is a view exemplarily showing a product data set applied to the present invention.
8 is a diagram exemplarily showing a color data set applied to the present invention.
9 is an exemplary view showing a color development confirmation screen of a touched lip position in a lip shape view using augmented reality in the present invention.
10 is a view showing an exemplary color test screen selected from the palette image in the present invention.
11 is a view exemplarily showing a screen for recommending a product similar to an image selected in the present invention.

Hereinafter, specific embodiments of the present invention will be described in detail with reference to the drawings. However, the spirit of the present invention is not limited to the embodiments presented, and those skilled in the art who understand the spirit of the present invention may add, change, delete, etc. other components within the scope of the same spirit, through addition, change, deletion, etc. Other embodiments included within the scope of the invention may be easily proposed, but this will also be included within the scope of the invention. In addition, components having the same function within the scope of the same idea shown in the drawings of each embodiment will be described using the same reference numerals.

1 is a block diagram showing the overall configuration of a cosmetic matching service providing system using augmented reality according to an embodiment of the present invention.

The cosmetic matching service providing system using augmented reality includes a management server 300 that is connected by a user terminal 100 accessible through a communication network 200 such as the Internet and provides a cosmetic matching test and similar product purchase service.

The user terminal 100 is a terminal of a mobile platform that interworks with a management application provided by the management server 300 , and may be, for example, a smart phone, a phablet phone, or a tablet.

The management server 300 may provide a cosmetic matching test and a similar product purchase service through a management application to which the user terminal 100 is connected.

In particular, in order to recommend cosmetics suitable for the user, it may include a function of extracting feature points of a specific lip region and directly checking the color development test of the lip region or recommending a similar product group.

In addition, the cosmetic matching service providing system using augmented reality of the present invention can detect various specific lip parts and match a product group of the corresponding part to make recommendations, and in an embodiment of the present invention, lipstick is detected by detecting lips with a specific part. Initiate a service that recommends product groups.

In the present invention, the lip detection algorithm can be made as follows.

First, it uses the Hyperlandmark library to detect lip landmarks, taking 106 landmarks on a person's face from the chin to the eyebrows and detecting them in real time. The location and value of landmarks are always fixed. The shape of the lips is detected using points that recognize the lips among the landmarks of the face.

After loading the lip detection library in the FaceTracking class and detecting the landmark, when the lip is detected, the lip landmark information is stored in the Face object. It then uses it to track the movement of the lips in real time.

The points array is a space to store the x and y coordinates of 106 landmark points. Get the coordinate information of each point from the face object r where the landmark information stored in the FaceTracking class is stored and input it into the points array.

In addition, the algorithm for the lip color test may be performed as follows.

The basic algorithm method is to put the points on the edge of the lips in a buffer to create a shape, then receive the color selected by the user and fill the shape of the lips with a triangle. Draw lips on the camera in real time using Open Graphics Library (OpenGL).

The following is the basic principle of the algorithm for drawing a triangle using OpenGL ES. The process of drawing a triangle is called the Rendering Pipeline, and to use this pipeline, you need to use a shader. The shader tells the graphics processing unit (GPU) how to draw the data. Define and use two vertex shaders and fragment shaders.

The Vertex Shader creates the final position for each vertex and is executed once for each vertex. Once the final position is known, OpenGL takes a set of visible vertices and turns them into points, lines, and triangles.

6 is an image showing the landmark number and position of the lip part by way of example. The location and order of landmarks do not always change, but since there are no rules, each number is extracted.

To extract only the lips from the points array that stores the x and y coordinates of 106 lip landmarks, only the landmark coordinate data can be input into the points array only in the landmark turn corresponding to the lip position.

To make a triangle with these data, a line array was created by arranging them in an adjacent order. After that, the data of the line array is passed to the vertex shader as a vertex value. Also, the color data received from the previous activity is transferred to the fragment shader as a color value. A fragment is a small, rectangular area of a single color that resembles a pixel on a computer screen.

Draw a triangle through the glDrawArrays function. The first argument is how to draw the vertex information received through the Vertex Shader. The second argument is the vertex to start drawing from the vertex array loaded into the memory, and the last argument determines how many vertices to draw the screen using. Therefore, 51 vertex values are drawn in the method using GL_TRIANGLES, and GL_TRIANGLES is the method of drawing a triangle. There must be 3 vertices of a triangle, and a triangle is drawn by pairing 3 sequentially in the vertex array. One or two remaining vertices are not drawn.

For example, use the points function to put the landmark values of the lips in the order of small values, or put the index of the value you want to get from the points array in the line function in order. You can also store the points values in the index values fetched from the line in sequence.

When all the landmark values of lips are stored in the points array, that is, when i==103, get the index of the points function from the line array and store the x and y coordinate values of the points at the corresponding index in the lines array.

The mPoints class is a class that creates 3D lip rendering objects using the OpenGL library, and mPoints.setPoints(lines) stores the values of vertices in the vertex shader.

Also, glDrawArrays(GL_TRIANGLES, 0, 51) can be implemented in the drawPoints function by using mPoints.drawPoints().

Furthermore, according to the present invention, the algorithm for recommending a product of a similar color may use the following method.

The basic idea is to calculate the distance between a color and another color in the RGB color space to find products of similar color. After importing the red, green, and blue color data of the standard product, the distance in Euclidean 3D is calculated and the data is stored. Then, using quick sort, the colors are sorted according to the closest distance.

The alignment method will be described in detail with reference to FIG. 3 , and first, R, G, and B color values selected by the user are extracted ( S301 ).

The next product list is loaded, and the distance value is stored in the ArrayList by calculating the distance between the RGB values of each product and the R, G, B values selected by the user and the R, G, B values selected by the user. Using quick sort, the distances are sorted in descending order (S302 to S304).

Then, the distance of the color to be compared with the reference color is calculated in the three-dimensional color space. This is obtained using the Euclidean distance calculation method. Euclidean distance calculation between two points in 3D is as follows [Equation 1].

Furthermore, lipstick product data used in the management application provided by the management server 300 is collected by crawling a site selling the product. After analyzing the web page structure, product data corresponding to product information in the website source is extracted using the Python library Beautiful Soup as shown in FIGS. 7 and 8 . That is, the product image, brand name, product name, product selling price, and purchase site link of each product are extracted from the purchase site that sells the product. Then, RGB codes are extracted from the product images to store the color data of each product.

2 is a flowchart illustrating a user album image selection process of a method for providing a cosmetic matching service using augmented reality according to an embodiment of the present invention.

First, an image button is touched in order to request an image selection for a test in the management application executed in the user terminal 100 (S201). The management application may be a management application provided by the management server 300 to provide a cosmetic color matching test and a similar product purchase service.

A photo including a lip image to be tested is selected from the user album stored in the user terminal 100 (S202).

Here, the user album may list, store, and manage photos including lip images preferred by the user in the form of a favorite through the user terminal 100 .

A lip position, which is a desired color position in the lip image included in the selected photo, is touched (S203). One's own lip image is stored and registered in the management server 300 in advance, and thereafter, upon request, management such as image editing or image replacement may be performed.

Furthermore, you can choose the palette menu to choose your own color rather than the user's album. That is, as shown in FIG. 10 , when a desired color is selected from the palette image by selecting the palette menu, the lipstick color of the corresponding color image is selected.

As shown in FIG. 9 after step S203, when a test for the selected color is requested, the color development of the touched lip position can be checked on the lip shape view screen (S204, S205, S206), and through this, the selected color is selected using augmented reality. The virtual lipstick color image is implemented and displayed so that it is overlaid on the actual lip image.

4 is a flowchart illustrating a product color development test process of a method for providing a cosmetic matching service using augmented reality according to an embodiment of the present invention.

The management server 300 builds a database in which a plurality of product information and product lists are stored in advance in conjunction with a product purchase site, and the user terminal 100 uses a management application as shown in FIG. 11 , the product desired to be tested. Touch to request (S401).

When the product information of the product selected by the user terminal 100 is requested from the management server 300, the product information stored in the database of the management server 300 is read and provided, and the user terminal 100 confirms it (S402). Here, the product information may include a product image, a brand name, a unique identification code, a cosmetic type, a color code, a product name, a product selling price, a manufacturer, a manufacturing date, a link to a purchase site of the product, and the like.

The user terminal 100 may receive and confirm a product list similar (similar to) to the selected product by product matching using the database of the management server 300 (S403). Here, the product matching criteria for providing a list of similar products may be products with a color similar to the selected product, a similar product type, or a product with a similar purchase history.

When the user terminal 100 requests a test by inputting a test button for a product selected from the product list, the color image of the product is displayed on the lip image corresponding to the selected specific part of the actual lip part through the lip shape view screen. It is displayed as lips (S404).

5 is a flowchart illustrating a product purchase process after testing of a method for providing a cosmetic matching service using augmented reality according to an embodiment of the present invention.

The user terminal 100 determines the intention to purchase the product through the lip display screen with the color of the product ( S501 ).

That is, when the user terminal 100 touches a product on which a purchase decision has been made, a product information confirmation window for confirming product information of the selected product is provided ( S502 ).

In addition, the user terminal 100 is provided with a product list similar to the selected product from the management server 300 (S503).

The user terminal 100 requests a purchase decision, and is connected to a purchase site for purchasing a corresponding product from the management server 300 (S504).

A corresponding product purchase process may be performed in the user terminal 100 accessing the purchase site (S505).

100 ; user terminal
200 ; communications network
300 ; management server

Claims (9)

providing a management application for providing a cosmetic color matching test and a similar product purchase service to a user terminal connected through a communication network in the management server;
requesting to select an image for testing in the management application executed in the user terminal;
selecting a photo including a lip image to be tested from the user album stored in the user terminal;
inputting a desired color position in the lip image included in the selected photo by the user terminal, and requesting a color development test for that position;
When the management server requests a color test for the color of the selected position, the color of the corresponding position on the shape view screen is implemented at the corresponding position of the actual lip image, and displaying on the screen of the user terminal;
A method of providing a cosmetic matching service using augmented reality comprising a. According to claim 1,
establishing, by the management server, a database storing a plurality of product information and product list in advance in conjunction with a product purchase site;
The user terminal requesting a product to be tested through a management application;
confirming, by the user terminal, product information of a product selected by a request through a management application;
confirming, by the user terminal, receiving a list of products similar to the selected product from the management server;
When a test is requested by inputting a test button for a product selected from the product list of the user terminal, the management server displays the color image of the product in a specific lip area selected from the actual image through the shape view screen. step;
A method of providing a cosmetic matching service using augmented reality further comprising a. 3. The method of claim 2,
determining, in the user terminal, to purchase the product after confirming it through a display screen of a specific lip area in which the color of the product is developed;
receiving a product information confirmation window for confirming product information of the selected product when the user terminal requests a product for which a purchase decision has been made;
connecting the user terminal to a purchase site for purchasing a corresponding product from the management server by requesting a purchase decision;
A method of providing a cosmetic matching service using augmented reality further comprising a. 4. The method of claim 3,
receiving, by the user terminal, a product list similar to the product selected for purchase decision from the management server;
A method of providing a cosmetic matching service using augmented reality further comprising a. 3. The method of claim 2,
Cosmetic matching using augmented reality, characterized in that the product information includes product image, brand name, unique identification code, cosmetic type, color code, product name, product selling price, manufacturer, date of manufacture, and a link to the purchase site of the product HOW TO PROVIDE SERVICES. A computer program that performs the method of providing a cosmetic matching service using the augmented reality of any one of claims 1 to 5, and stores the method in a computer-readable recording medium. Provides a management application for providing cosmetic color matching tests and similar product purchase services to user terminals connected through a communication network,
When a photo including a lip image to be tested is selected from a stored user album to request an image selection for testing in the management application executed in the user terminal, a desired color position in the lip image included in the selected photo from the user terminal to request a color test for the corresponding location by inputting , and when a color test for the color of the selected location is requested, the color of the location on the shape view screen implements the color development at the location of the actual lip image, and the display management server;
Cosmetic matching service providing system using augmented reality, including a. 8. The method of claim 7,
The management server
By linking with the product purchase site in advance, we build a database that stores a number of product information and product lists,
If the user terminal requests a product to be tested through the management application, and requests a color test for the color of the product after checking the product information of the product, a specific lip part selected from the actual image of the user through the shape view screen Cosmetic matching service providing system using augmented reality, characterized in that the color image of the product is displayed and implemented. 9. The method of claim 8,
Cosmetic matching using augmented reality, characterized in that the product information includes product image, brand name, unique identification code, cosmetic type, color code, product name, product selling price, manufacturer, date of manufacture, and a link to the purchase site of the product service delivery system.

Images