KR20220040051A - Apparel wearing system based on face application, and methoe thereof - Google Patents

Abstract

The present invention relates to a clothing wearing system based on face application and a method thereof, and more specifically, to a clothing wearing system based on face application and a method thereof. Although a user is provided with the experience of wearing clothes through a service provided in an existing augmented reality field, the actual offline wear and experience of clothes are provided at a level of about 60%. Therefore, the system improves an active appearance model algorithm used within existing augmented reality, and through a direct volumetric (DVM) process, the system provides customer showvata composite images based on augmented reality, which is a 3D avatar based on ultra-high-definition images.

Description

Apparel wearing system based on face application, and methoe thereof

The present invention relates to a face application-based clothing wearing system and method. More specifically, a user is provided with a clothing wearing experience through a service provided in the existing augmented reality field, but the actual offline clothing fit and experience is provided at the level of 60%, so the "AAM (Active Appearance Model) algorithm" used in the existing augmented reality is improved, and through the "DVM (Direct Volumetric) process", augmented reality that is a 3D avatar based on ultra-high-resolution images It relates to a face application-based clothing wearing system and method for providing a composite image of customer Shobata.

In the online e-commerce market, the online conversion of retailers through on-offline convergence is being carried out.

In other words, as retail distribution companies, which have difficulty in operating offline stores, start selling online, the combination between on-line and offline sales channels is accelerating.

In particular, due to the novel coronavirus infection, the global economy is stagnating, and non-face-to-face consumption is receiving a lot of attention. Due to these consumer movements, the domestic e-commerce market has grown to a size of $84,677 million in 2019. The share of the e-commerce market in the domestic distribution market is also increasing at 28.2% every year.

As non-face-to-face consumption becomes the “new normal”, the introduction of innovative technologies to provide an optimal consumer experience is accelerating. In order to compensate for this, innovative information technology (IT) is being actively introduced into the online commerce market.

As the trend of untact consumption in the distribution field is prolonged, it is recognized as a new standard. In this trend, global IT companies are actively investing in startups that have these technologies to secure augmented/virtual reality (AR, VR) technologies.

Due to the performance of a general office PC, it is difficult to properly express 3D virtual reality images, and in particular, when using a head-mounted display (HMD) device that is worn directly on the head, the wearer may cause dizziness.

The non-HMD method, which looks at objects in 3D by wearing special glasses, must be worn in the same way as the HMD to be used. Of course, compared to the HMD, the weight of the wearing device is remarkably small, so the convenience is improved, but there is a limit in the part that the equipment needs to be worn.

On the other hand, the cost of creating a virtual reality environment is quite high, and the application efficiency within the industry is low. If a 3D model made with CAD is not available in the company, most of the production process must be implemented by manual labor-intensive virtual reality. This includes the fact that it is difficult to start without a clear profit model or investment return.

As of 2020, current AR technologies are implemented by wearing wearable devices, so the problems of inconvenience and high equipment prices have not been resolved yet.

These problems are that they do not provide the practical value of the awkwardness and inconvenience in front of the workers in the industrial field. This is expected to be possible only when conservative practitioners have a clear sense of experience, advantages, and further benefits through new technologies.

Republic of Korea Patent Application No. 10-2009-0047947 (June 1, 2009) "A simulated clothing wearing system based on a simulated clothing wearing device and a method of providing a simulated clothing wearing service using the same (VIRTUAL CLOTHES WEARING SYSTEM BASE ON VIRTUAL CLOTHES WEARING) APPARATUS AND METHOD FOR PROVIDING VIRTUAL CLOTHES WEARING SERVICE USING THEREOF)

The present invention is to solve the above problems, and although the user is provided with a clothing wearing experience through the service provided in the existing augmented reality field, the actual offline clothing fit and experience is provided at a level of about 60%. To improve the "AAM (Active Appearance Model) algorithm" used in augmented reality, and to provide an augmented reality based 3D avatar based on augmented reality through the "DVM (Direct Volumetric) process" An object of the present invention is to provide a face application-based clothing wearing system and a method therefor.

However, the objects of the present invention are not limited to the above-mentioned objects, and other objects not mentioned will be clearly understood by those skilled in the art from the following description.

In order to achieve the above object, a face application-based clothing wearing system according to an embodiment of the present invention is a user smart terminal group 100g consisting of a plurality of user smart terminals 100, a network 200, and a face application-based clothing In the face application-based clothing wearing system (1) including the wearing server (300), the face application-based clothing wearing server (300) is a member according to the access of each user's smart terminal (100) through the network (200). an information collection module 321 for controlling the transceiver 310 to perform a subscription procedure and provide shopping app data to the smart terminal 100 through the network 200; And the user's face taken from the user's user smart terminal 100 according to the facial feature point extraction technique, which applies the improved Active Appearance Model (AAM) algorithm and machine learning technique to the user's face shooting information and body size information a 3D face image generation module 322 that reconstructs a 2D face image by using the information and forms a reference value by manually processing facial feature elements for a reconstruction & visualization operation; It may be characterized in that it includes.

In order to achieve the above object, a method for wearing clothes based on face application according to an embodiment of the present invention includes: a first step of performing a login process by executing a pre-set shopping app by the user smart terminal 100; And when the user smart terminal 100 is logged in is completed, after receiving information according to the relay of the face application-based clothing wearing server 300 for clothes registered by the Shobata shopping mall server 500 executing the Shobata API, a second step of outputting a Shobata execution selection screen; It may be characterized in that it includes.

In the face application-based clothing wearing system and method according to an embodiment of the present invention, a user is provided with a clothing wearing experience through a service provided in the existing augmented reality field, but the actual offline clothing fit and experience is reduced by 60% As it is provided at the level of precision, we improve the "AAM (Active Appearance Model) algorithm" used in the existing augmented reality, and through the "DVM (Direct Volumetric) process", a 3D avatar based on ultra-high-resolution images, augmented reality-based customers It has the effect of being able to provide a Shobata composite image.

1 is a diagram illustrating a face application-based clothing wearing system 1 according to an embodiment of the present invention.
2 is a block diagram illustrating components of a face application-based clothing wearing server 300 of the face application-based clothing wearing system 1 according to an embodiment of the present invention.
3 and 4 are for explaining a process for providing a face application-based clothing wearing service of the face application-based clothing wearing server 300 among the face application-based clothing wearing system 1 according to an embodiment of the present invention It is a drawing.
5 is a flowchart illustrating a process of providing a face application-based clothing wearing service according to an embodiment of the present invention.
6 is a reference diagram illustrating a process of providing a face application-based clothing wearing service according to an embodiment of the present invention, focusing on components.

Hereinafter, a detailed description of a preferred embodiment of the present invention will be described with reference to the accompanying drawings. In the following description of the present invention, if it is determined that a detailed description of a related well-known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

In the present specification, when any one component 'transmits' data or signal to another component, the component may directly transmit the data or signal to another component, and through at least one other component This means that data or signals can be transmitted to other components.

1 is a diagram illustrating a face application-based clothing wearing system 1 according to an embodiment of the present invention. Referring to FIG. 1 , the face application-based clothing wearing system 1 includes a user smart terminal group 100g consisting of a plurality of user smart terminals 100 , a network 200 , a face application-based clothing wearing server 300 , It may include a big data server 400 , a Shobata shopping mall server 500 , and an SNS server 600 .

As a dictionary definition related to the present invention, "SHOVATAR" is a compound word of Shopping + Avatar, based on the existing sources used in the augmented reality technology field and the 5G network environment , refers to the shape of a person specialized in the clothing industry and produced by augmented reality.

On the other hand, although the user is provided with a clothing experience through the service provided in the existing augmented reality field, the actual offline clothing fit and experience is provided at a level of about 60%. Accordingly, the face application-based clothing wearing server 300 according to the present invention improves the "AAM (Active Appearance Model) algorithm" used in the existing augmented reality, and through the "DVM (Direct Volumetric) process", an ultra-high-resolution image It is possible to provide a composite image of customer Shobata based on augmented reality, which is a 3D avatar based.

2 is a block diagram illustrating components of a face application-based clothing wearing server 300 of the face application-based clothing wearing system 1 according to an embodiment of the present invention. 3 and 4 are for explaining a process for providing a face application-based clothing wearing service of the face application-based clothing wearing server 300 among the face application-based clothing wearing system 1 according to an embodiment of the present invention It is a drawing.

Referring to FIG. 2 , the face application-based clothing wearing server 300 includes a transceiver 310 , a controller 320 and a database 330 , and the controller 320 includes an information collection module 321 , a 3D face By including the image generating module 322, the image synthesis module 323, the Shobata providing module 324 and the additional information providing module 325, 3D based augmented reality based to give a more realistic feel than the existing app Rather than focusing on virtual implementation of the wearing state of clothes by deep learning as While rotating 360 degrees or moving arms and legs, it is possible to know the texture of clothes, the state of wearing clothes, etc. in detail.

The information collection module 321 performs a membership registration procedure according to the access of each user's smart terminal 100 through the network 200 and transmits/receives the shopping app data to the smart terminal 100 through the network 200 310 can be controlled

The 3D face image generation module 322 is the user's user smart terminal 100 according to the facial feature point extraction technique, which applies the improved AAM (Active Appearance Model) algorithm and machine learning technique to the user's face photographing information and body size information. ) to reconstruct a 2D face image by using the user's face photographing information obtained in ), and for visualization (Reconstruction & Visualization) work, facial feature elements can be manually processed to form a reference value. Here, the 3D face image generating module 322 may improve the extraction speed based on the APF.

Here, "AAM (Active Appearance Model)" is a generalization of the ASM (Active Shape Model) approach. It is a technique that uses information from all image areas, not just the body edges, in the modeled part of the body. As shown in 3, a 360° camera (not shown) formed separately from the user smart terminal 100, and other face application-based clothing wearing servers 300 operators of 4K image quality collected through a separately provided 360° camera (not shown) After controlling the transceiver 310 so that the 3D face image generating module 322 receives the 2D live image from the user smart terminal 100, the 4K live image group photographed by the 360 ° camera is stored in the database 330 can

The 3D face image generation module 322 is a process of matching the differences in brightness, color, saturation, etc. of the image acquired according to 360° rotation in the shooting stage according to the digital intermediate operation for the 4K real image group As a result, the corrected live-action video image can be completed by performing the overall correction including color correction and correction of color values and exposure.

Here, "AAM (Active Appearance Model)" used by the 3D face image generation module 322 is not only the body edge in each live-action video image, but also each 4K live-action image that forms a 4K live-action image group shot by a 360° camera. The information (color, saturation, brightness) of all image areas in each 4K photorealistic image is set to the same coordinate value for the overlapping area, and each x, y, z axis coordinate is converted to metadata according to 360° rotation of the body edge (contour line). information) and area information can be matched and extracted.

That is, the 3D face image generation module 322 can extract an overlapping coordinate region based on change predictability corresponding to the machine learning technique from each 4K live-action image of one angle and another neighboring angle.

More specifically, the 3D face image generation module 322 accesses the big data server 400 through the network 200 and stores the contour of the body part according to the change in the angle of the human body stored in the big data server 400 . After extracting information on changes in straight lines and curves when moving left and right and up and down based on one central point, as well as changes in color, saturation, and brightness, it is applied to the overlapping coordinate area in each 4K actual image of one angle and another neighboring angle. The body edge of the non-overlapping coordinate area by excluding the body edge (outline information) and area information of the overlapping coordinate area through the secondary confirmation process through the 3D image of the extracted coordinate area (Contour line information) and only area information can be acquired.

Meanwhile, for the body part, the 3D face image generating module 322 may extract the body part by comparing the 2D image with the body part pattern information stored in the big data server 400 . To this end, various body part pattern information is stored in the big data server 400, and the 3D face image generating module 322 includes information on the various body part patterns itself or a preset angle of the various body part pattern information. Body parts can be identified by comparing 2D images with those made of

In the secondary confirmation process, the 3D face image generation module 322 performs decoding on each 4K actual image corresponding to the 2D image to generate a decoded-image and a polygon, which is a basic unit for expressing in 3D shape. After creating a set of , texture mapping is performed to attach each decoded image to the set of polygons to form a 3D image, and then, based on the overlapping line corresponding to the changing constant, the overlapping line that matches the upper and lower left and right is extracted, , changes in straight lines and curves when moving left and right and up and down based on one central point of the contour, which is the parameter of change of body parts according to the coordinate area overlapping based on the overlapping line and the angle change of the human body extracted first, and the color and saturation , after extracting the change information of the brightness, if the overlapping coordinate area in each 4K actual image of one angle and another neighboring angle is matched, the overlapping coordinate area is extracted based on the overlapping line, and if not, the coordinate area is changed The process of re-extracting the movement change for the overlapping line corresponding to the constant based on machine learning can be performed.

Here, the machine learning performed by the 3D face image generation module 322 is performed based on the analysis/control program, and the collected data for each body part distributed and stored in the DCS DB by the distributed file program on the big data server 400 is machined. The overlapping area can be extracted by analyzing it through a learning algorithm and setting an overlapping line. More specifically, the machine learning algorithm used in the analysis/control program may be one of a decision tree (DT) classification algorithm, a random forest classification algorithm, and a support vector machine (SVM) classification algorithm.

The analysis/control program analyzes the collected data distributed and stored in the DCS DB for each body part on the big data server 400 by the distributed file program, and the analysis result is based on one overlapping line of the 3D image for each body part As a result, it is possible to extract a plurality of feature information according to different angle changes, learn the extracted feature information using at least one of a plurality of machine learning algorithms, and determine whether the matching state is a result of the learning.

That is, the analysis/control program can apply an ensemble structure composed of a number of complementary machine learning algorithms to improve the accuracy of the status determination result.

The decision tree classification algorithm learns in a tree structure and derives results, so it is easy to interpret and understand the results, the data processing speed is fast, and rules can be derived based on the search tree. RF can be applied as a method to improve the low classification accuracy of DT. The random forest classification algorithm slaughters the results of learning multiple DTs as an ensemble, and it is difficult to understand the results than DT, but the accuracy of the results may be higher than that of DT. SVM can be applied as a way to improve overfitting that can occur through DT or RF learning. The SVM classification algorithm classifies data belonging to different classifications in a plane-based manner, and generally has high accuracy and may have low sensitivity to structural overfitting.

The 3D face image generation module 322 may reconstruct a 3D image through "Direct Volumetric technology" as shown in FIG. 4 based on the extracted feature points.

More specifically, the 3D face image generation module 322 is a convolutional neural network (CNN) between each coordinate of the body edge (contour line information) of the coordinate region among the reconstructed 2D face image information having information on each coordinate at 360°. ), "DVM (Direct Volumetric)" that creates and extracts 3D coordinates from 2D pixels, and uses 3D coordinates to match and apply area information to create a 2D face image as a 3D face image technique can be used.

That is, the face image generation module 322 converts 2D face and body image into 3D form through machine learning to express as a shopping avatar optimized for the e-commerce industry. Through the "DVM (Direct Volumetric)" process based on "Volumetric Capture Video", the texture of the clothing and the actual user's fit (fit, below) are expressed, and the actual offline clothing fit is more than 90% the same in the online channel. An “ultra-high-definition AR shopping avatar” that provides an experience can be provided.

The image synthesis module 323 may perform face change synthesis with the 2D image of the shopping mall fitting model stored in the database 330 corresponding to the dynamic object on the 3D face image generated through the “Direct Volumetric (DVM)” technique. there is.

More specifically, the image synthesizing module 323 receives the clothes selected by the user from the Shobata shopping mall server 500 in advance and stores the 3D face of the user with respect to the 2D image of the shopping mall fitting model stored in the database 330 . Through the "AAM (Active Appearance Model)" technique by the image generating module 322, not only the body edges in each live-action video image, but also all images of each 4K live-action image forming a 4K live-action image group shot by a 360° camera With the area information (color, saturation, lightness) as the same coordinate value for the overlapping area in each 4K real-world image, each x, y, and z axis coordinate as metadata as metadata for 360° rotation of the body edge (outline information) and After matching and extracting area information, a 3D image shape can be generated by using a 2D image of a shopping mall fitting model in a manner that generates a 3D image shape.

Thereafter, the image synthesis module 323 synthesizes the values of the extracted facial features of the dynamic object and the 3D facial feature values of the user image in which the values of lighting, image quality, camera performance, etc., which are different according to the external environment, are standardized to synthesize 3D synthesis. Create a contour surface. Thereafter, the image synthesis module 323 may generate a customer Shobata composite image by synthesizing and processing area information for each 3D coordinate according to the facial tracking data.

That is, there was difficulty (accuracy) in extracting or synthesizing facial feature points in the conventional image synthesis dynamic 3D object. It is to produce an improved and developed form of Shobata.

After that, the Shobata providing module 324 synthesizes the customer Shobata synthesis image on the background screen when the background screen provided from the user terminal 100 is provided from the Shobata shopping mall server 500 connected to the network 200 . By creating a composite image of customer Shobata based on matching images and providing it to the user smart terminal 100 through the Shobata shopping mall server 500 connected to the network 200, or through the network 200, the user smart terminal ( 100) may control the transceiver 310 to directly provide the augmented reality-based customer Shobata composite image.

That is, the face application-based clothing wearing server 300 receives the fitting model data that has passed through the processing unit-synthesis unit through the enterprise open API optimized for “Shobata” in the form of a service module for the implementation of an online commerce platform to a user corresponding to a mobile device. The smart terminal 100 is to be implemented in augmented reality.

In the past, when creating augmented reality content using high-quality (large-capacity) images, there are limitations such as reduced augmented reality image quality and reduced realism when driving a smartphone according to the large capacity of the content. Most augmented reality contents produce AR contents using regular 2D images or 3D object files, which have a small capacity compared to high-quality images. can Accordingly, the "super-realistic augmented reality" provided in the present invention is focused on using high-quality (large-capacity) images to make it available on smartphones. In particular, through the self-developed “video and audio compression technology” and “content standardization technology”, you can experience hyper-realistic augmented reality with your smartphone.

Accordingly, online clothing commerce companies that operate the Shobata shopping mall server 500, based on the user's UX / UI, freely implement "Shobata" as augmented reality on the smartphone corresponding to the user's smart terminal 100. You can move and experience it.

5 is a flowchart illustrating a service providing process, which is a method for wearing clothes based on face application according to an embodiment of the present invention. 6 is a reference diagram illustrating a process of providing a face application-based clothing wearing service according to an embodiment of the present invention, focusing on components.

Referring to FIG. 5 , the user smart terminal 100 executes a preset shopping app to perform a login process ( S11 ).

After the step (S11), when the login according to the step (S11) is completed, the user smart terminal 100 applies a face application-based clothing server 300 for the clothes registered by the Shobata shopping mall server 500 executing the Shobata API. ), after receiving the information according to the relay, it is possible to output the Shobata execution selection screen (S12).

Here, Shobata API is an open API optimization integrated operating platform for businesses that can be used in online commerce malls. For B2B services, the face application-based clothing wearing server 300 and the customer Shobata shopping mall server 500 are B2B With the API that connects to the channel, you can manage member management, product information and management, and billing.

After the step (S12), the user's smart terminal 100 takes a picture of the user's face in the "Shobatta setting page" and sets the body size if selected, depending on whether the Shobata execution selection screen is selected (S13) and can be stored (S14).

On the other hand, the user smart terminal 100 provides the same product search service as in a general shopping mall from the Shobata shopping mall server 500 to the network 200 depending on whether the Shobata execution selection screen is selected (S13), if not selected. It can be provided through (S15).

Next, the Shobata shopping mall server 500 accesses the face application-based clothing wearing server 300 through the network 200 to perform the Shobata API setting through a request to the face application-based clothing wearing server 300. (S21), depending on whether Shobata API is set (S22), if not set, it can return to step (S21) and retry for Shobata API setting can be performed, and if set conversely, step (S31) By proceeding to , it is possible to receive the Shobata API module from the face application-based clothing wearing server 300 and provide the Shobata service for the user smart terminal 100 based on the Shobata API module, and provide the Shobata API module and payment for the face application-based clothing wearing server 300 for use (S31).

Thereafter, when Shobata shopping mall server 500 executes Shobata in the Shobata execution selection screen on the shopping app of the user smart terminal 100 through the network 200, the user smart terminal using the Shobata API module and processing the request data of the shopping app through the network 200 with the shopping app of (100) (S23).

According to the shooting by the user smart terminal 100 in step S14, the Shobata shopping mall server 500 receives the user's face shooting information and body size information from the user smart terminal 100, and then After performing the storage (S24), the user approval is performed based on the user's face photographing information (S25), and if the approval result is inappropriate for approval, the process returns to step (S12), and if the approval result is approved, the steps described below ( In order to proceed to S32), the user's face photographing information and body size information may be provided to the face application-based clothing wearing server 300 through the network 200 .

On the other hand, the Shobata shopping mall server 500 provides augmentation provided when an augmented reality-based customer Shobata composite image is generated and provided by the face application-based clothing wearing server 300 for the user's face photographing information and body size information. A Shobata showroom may be provided to the user smart terminal 100 by using the reality-based customer Shobata composite image (S27).

Next, the face application-based clothing wearing server 300 not only performs payment for the face application-based clothing wearing server 300 for providing and using the Shobata API module for the Shobata shopping mall server 500 (S31) ), when the approval result based on the user's face photographing information for the user's smart terminal 100 by the Shobata shopping mall server 500 is suitable for approval, the user's face photographing information and body size from the Shobata shopping mall server 500 Information may be received through the network 200 and stored in the database 330 (S32).

Thereafter, the face application-based clothing wearing server 300 generates a customer Shobata by using the user's face photographing information and body size information (S33), and receives it from the Shobata shopping mall server 500 and is placed on the database 330. After synthesizing the stored shopping mall fitting model image and customer Shobata (S34), and generating the synthesized image as an augmented reality image (S35), it is provided to the Shobata shopping mall server 500 through the network 200 Thus, it is possible to provide a Shobata showroom to the user smart terminal 100 by utilizing the augmented reality-based customer Shobata composite image by the Shobata shopping mall server 500 (S27).

On the other hand, the additional information providing module 325 is the SNS server 600 according to the request of the user smart terminal 100 for the augmented reality-based customer Shobata composite image generated for the user who operates each user smart terminal 100 . ), information can be transmitted to the smart terminal 100 of the user who has requested a recommendation for the most recommended clothing through the transmission and recommendation functions to the smart terminal 100 of another user registered as an acquaintance.

The present invention can also be implemented as computer-readable codes on a computer-readable recording medium. The computer-readable recording medium includes all types of recording devices in which data readable by a computer system is stored.

Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. also includes

In addition, the computer-readable recording medium is distributed in network-connected computer systems, and computer-readable codes can be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the present invention can be easily inferred by programmers in the technical field to which the present invention pertains.

As described above, preferred embodiments of the present invention have been disclosed in the present specification and drawings, and although specific terms are used, these are only used in a general sense to easily explain the technical content of the present invention and to help the understanding of the present invention. , it is not intended to limit the scope of the present invention. It will be apparent to those of ordinary skill in the art to which the present invention pertains that other modifications based on the technical spirit of the present invention can be implemented in addition to the embodiments disclosed herein.

1: Clothing wearing system based on face application
100: user smart terminal
100g: User smart terminal group
200: network
300: face application-based clothing wearing server
400: big data server
500: Shobata shopping mall server
600: SNS server

Claims (2)

In the face application-based clothing wearing system (1) comprising a user smart terminal group (100g) consisting of a plurality of user smart terminals (100), a network (200), and a face application-based clothing wearing server (300), face application Base clothing wearing server 300,
Information for controlling the transceiver 310 to perform a membership registration procedure according to the access of each user's smart terminal 100 through the network 200 and provide shopping app data to the smart terminal 100 through the network 200 collection module 321; and
User's face photographing information obtained from the user's user smart terminal 100 according to the facial feature point extraction technique, which applies the improved AAM (Active Appearance Model) algorithm and machine learning technique to the user's face photographing information and body size information a 3D face image generating module 322 that reconstructs a 2D face image using Face application-based clothing wearing system comprising a
A first step of the user smart terminal 100 executing a preset shopping app to perform a login process; and
When the user smart terminal 100 is logged in, after receiving information according to the relay of the face application-based clothing wearing server 300 for the clothes registered by the Shobata shopping mall server 500 executing the Shobata API, the shock absorber a second step of outputting another execution selection screen; Face application-based clothing wearing method comprising a.

Images