KR20220095483A - System for providing real-time costume replacement contents with AR beam project - Google Patents

Abstract

The present invention relates to a real-time clothes change content providing system. The real-time clothes change content providing system includes: a user terminal in which a dedicated application including a character clothes drawing function to draw user character clothes is installed; a generation server, when receiving user picture data drawn by a user through the character clothes drawing function of the dedicated application from the user terminal, generating an artificial intelligence-based three-dimensional clothes model about the received user picture data, and providing the generated three-dimensional clothes model to an AR beam projector; and the AR beam projector projecting an augmented reality (AR) screen on a projection area of content formed as an actual object, while projecting the AR screen employing the three-dimensional clothes model received from the generation server. In accordance with the present invention, since a two-dimensional original picture inputted directly from a user is generated into a three-dimensional model to be applied to AR content in real time, the user can have more interest, and an AR content-related education function can be enhanced.

Description

Real-time costume replacement contents providing system having AR beam project {System for providing real-time costume replacement contents with AR beam project}

The present invention relates to a technology for providing content for replacing clothes, and more particularly, to a technology for providing content for replacing clothes with an AR beam project for projecting an AR (Augmented Reality) image.

AR (Augmented Reality) is one of virtual reality that combines the real world seen by the user and the virtual world with additional information as one image. AR, a concept that complements the real world with the virtual world, uses a virtual environment created with computer graphics, but the main character is the real environment. Computer graphics serve to provide additional information necessary for the real environment and blur the distinction between the real environment and the virtual screen by overlapping the 3D virtual image on the actual image viewed by the user.

Whereas virtual reality uses virtual images rather than reality for both the self (object), the background, and the environment, augmented reality (AR) is a technology that shows a 3D virtual image overlaid on a real image or background as a single image.

Augmented reality is also called Mixed Reality (MR), and augmented reality was first introduced to the world around 1990 when Boeing, an airplane manufacturer, added virtual images to the assembly process of airplanes.

Augmented reality and virtual reality (VR) seem similar to each other, but they are clearly differentiated depending on whether the subject is a virtual image or a real image. For example, in a virtual reality fighting game, a character that replaces me confronts a virtual enemy in a virtual space, but in an augmented reality fighting game, a real person competes with a virtual enemy in a real space. Therefore, augmented reality is characterized by superior realism compared to virtual reality.

In addition, virtual reality is generally used only in special environments such as movies and video fields, but augmented reality is now popular enough to be widely used by the general public.

For example, map search and location search through the Internet are also included in augmented reality in a broad sense.

In general, infants and children grow up by reading a storybook or jigsaw puzzles or building blocks as play equipment. As such, children's books or rides are an important medium that allows children to naturally acquire various information and knowledge, and children's books, puzzles, or blocks help develop children's emotions, imagination, and creativity, and provide social, language, and It has an even effect on cognitive development.

According to the 2015 National Reading Survey Report, communication through books (reading books, playing books, reading conversations, etc.) during the growth process plays an important role in fostering intimacy with reading and fostering reading habits.

As such, according to recent research results, it has been found that children who read a lot have higher thinking and creativity, as well as academic performance. In addition, it is a situation that requires a response to the rapidly changing reading environment and the decrease in the reading population due to the development of digital media. Accordingly, the government has established a basic plan for the promotion of reading culture and is carrying out projects to promote reading and spread the culture of reading.

As such, as the infant market emerges from a niche market to a core (angel) market, the learning game-type mobile edutainment content consumption market targeting mobile kids is growing.

And, as virtual reality (VR) emerges as a blue ocean to lead the future industry, application cases are increasing in various fields such as games, medical care, education, and exhibitions, but it is necessary to discover killer contents.

Recently, Digilog Book technology, a kind of next-generation e-book based on augmented reality that combines analog and digital information, is attracting attention. It provides a variety of augmented reality-based multimedia effects through natural markers while touching and feeling the real-life printed on paper.

In addition, it has interesting features that develop the story in various directions through easy interaction with the user. Various studies are being conducted to combine analog and digital information, but among them, digital information is projected onto a real object through a projection method to obtain the effect of operating various multimedia contents on a real object. This has the advantage of increasing the level of immersion by being able to experience realistic additional information while watching the real thing.

However, the projection method can provide realistic content by harmonizing the printed information with the projected digital information only when the real object is accurately positioned in the projection area. There is a problem that is not easy.

The existing 2D image-based 3D character creation technology selects and transforms data most similar to the image drawn by the user from the database through cloud search, etc. As it is being serviced in the form of being applied to a 3D model, strictly speaking, it can be said that it provides clothes for other characters that do not completely match the user's intentions.

Republic of Korea Patent Publication 10-2020-0137766

The present invention was devised to solve the above problems, and using artificial intelligence technology to generate a 2D original image input by a user as a 3D model as it is, and to provide a real-time clothes replacement content providing system that can be applied to the content in real time but it has a purpose.

In addition, the present invention provides a desk on which an AR beam projector capable of accurately positioning an object printed with analog information in a projection area (projection area) of an AR beam projector that projects digital information using AR projection technology is installed. It has a different purpose.

Objects of the present invention are not limited to the objects mentioned above, and other objects not mentioned will be clearly understood by those skilled in the art from the following description.

The present invention for achieving the above object relates to a system for providing real-time clothes replacement content, wherein a user terminal having a dedicated application including a character costume drawing function capable of drawing a user character costume is installed, and the dedicated application from the user terminal When receiving user picture data drawn by the user through the character costume drawing function of and an AR beam project for irradiating an AR (Augmented Reality) screen to the projection area of the content composed of real objects, and applying the 3D character costume model received from the generation server.

The generation server can generate a 3D character costume model through a training data construction step for generating an artificial intelligence-based 3D character costume model and a preparation step for 3D costume generation using the user figure data input from the user terminal. .

In the learning data building step, the generation server includes modeling learning data composed of a basic costume model that can be worn on a character model, texture learning data composed of texture, color, and UV maps of model data for each costume composed of the basic costume model, and content The basic costume data construction process, which is the process of building rigging learning data that can support animation in real time, the process of constructing clothes data specialized for the content, the process of defining the representative clothes for the specialized clothes in the content, and the transformation of each representative clothes The learning model extension process can be performed to extend the learning model by changing the clothes by designating an area.

According to the present invention, the user's interest can be increased by generating a 2D original image directly input by the user as a 3D model and applying it to AR contents in real time, and there is an effect that can increase the AR contents related education function.

In addition, since the desk on which the AR beam projector of the present invention is installed puts the real object in the insertion groove, the real object is precisely positioned in the projection area of the AR beam projector, so infants and children can easily place the real object in the correct place There is this.

1 and 2 are diagrams schematically showing the configuration of a system for providing real-time clothes replacement content according to an embodiment of the present invention.
3 to 5 are views showing a desk on which an AR beam projector according to an embodiment of the present invention is installed.
6 is a view showing a state of adjusting the position of the rear panel on the desk where the AR beam projector is installed according to an embodiment of the present invention.
7 is a view showing a state in which the top cover is removed from the insertion groove of the desk on which the AR beam projector is installed according to an embodiment of the present invention.
8 is an example of a screen of a user terminal according to an embodiment of the present invention.
9 exemplifies data transmitted from the user terminal to the providing server in the present invention.
10 to 13 illustrate a learning data construction process for generating an artificial intelligence 3D character costume model in the present invention.
14 is an illustration of a representative costume model definition in the present invention.
15 and 16 are diagrams illustrating manufacturing according to the definition of a representative costume model in the present invention.
17 illustrates a process of performing local transformation of representative clothes in the present invention.
18 is an exemplary diagram of a character model according to an embodiment of the present invention.
19 to 21 are exemplary views of a UV texture map of a character in the present invention.
22 and 23 illustrate a final output image according to an embodiment of the present invention.

Advantages and features of the embodiments disclosed herein, and methods for achieving them will become apparent with reference to the embodiments described below in conjunction with the accompanying drawings. However, the embodiments to be proposed in the present disclosure are not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the present embodiments are provided to those of ordinary skill in the art. It is only provided to fully indicate the category.

Terms used in this specification will be briefly described, and the disclosed embodiments will be described in detail.

The terms used in this specification have been selected as currently widely used general terms as possible while considering the functions of the disclosed embodiments, but may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, and the like. In addition, in a specific case, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the detailed description part of the corresponding specification. Therefore, the terms used in the present disclosure should be defined based on the meaning of the term and the contents of the entire specification, rather than the simple name of the term.

References in the singular herein include plural expressions unless the context clearly dictates the singular.

In the entire specification, when a part "includes" a certain element, this means that other elements may be further included, rather than excluding other elements, unless otherwise stated. Also, as used herein, the term “unit” refers to a hardware component such as software, FPGA, or ASIC, and “unit” performs certain roles. However, "part" is not meant to be limited to software or hardware. A “unit” may be configured to reside on an addressable storage medium and may be configured to refresh one or more processors. Thus, by way of example, “part” includes components such as software components, object-oriented software components, class components and task components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. The functionality provided within components and “parts” may be combined into a smaller number of components and “parts” or further divided into additional components and “parts”.

In addition, in the description with reference to the accompanying drawings, the same components are given the same reference numerals regardless of the reference numerals, and the overlapping description thereof will be omitted. In describing the present invention, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

The present invention is a technology that generates a 3D model of the 2D original image input by a user by using artificial intelligence technology and applies it to content in real time. suggest

1 and 2 are diagrams schematically showing the configuration of a real-time clothes replacement content providing system according to an embodiment of the present invention.

1 and 2, the real-time clothes replacement content providing system of the present invention includes a generating server 100, a user terminal 200, and an AR beam projector (B).

The user terminal 200 is a terminal in which a dedicated application including a character costume drawing function capable of drawing a user character costume is installed. For example, the user terminal 200 is a terminal capable of communicating by connecting to a mobile communication network and installing a dedicated application, and may include a smartphone, a tablet PC, and the like.

When the generation server 100 receives the user picture data drawn by the user through the character costume drawing function of the dedicated application from the user terminal 200, it generates an artificial intelligence-based 3D character costume model with respect to the received user picture data, and generates The 3D character costume model is provided to the AR beam project (B).

The AR beam project (B) irradiates an AR (Augmented Reality) screen to the projection area of the content composed of real objects, but serves to investigate the AR screen to which the 3D character costume model received from the generation server 100 is applied.

As shown in FIG. 2 , in the present invention, the content is implemented as a real object with pictures, for example, the content may be implemented as a storybook, an AR finger story book, and the like.

The generation server 100 creates a 3D character costume model through a learning data construction step for generating an artificial intelligence-based 3D character costume model and a preparation step for 3D costume creation using the user picture data input from the user terminal 200 can create

In the learning data building stage, the generation server 100 includes modeling learning data composed of a basic costume model that can be worn on a character model, texture learning data composed of texture, color, and UV maps of model data for each costume composed of a basic costume model, and The basic costume data construction process, which is the process of building rigging learning data that can support animation in real time within the content, the costume data construction process specialized for the content content, the process of defining the representative clothes for the specialized clothes in the content, and the process of each representative costume The learning model extension process can be performed to extend the learning model by changing the clothes by designating the transformation area.

8 is an example of a screen of a user terminal according to an embodiment of the present invention.

Referring to FIG. 8 , the user terminal 200 provides a character costume drawing function through a dedicated application (app). In the dedicated app, a user picture input by the user through the user terminal 200 is received as an input. Then, the user picture is divided into a base color and a pattern and transmitted to the generating server 100 (Step1).

9 exemplifies data transmitted from the user terminal to the providing server in the present invention.

Referring to FIG. 9 , in the present invention, since the costume drawn by the user must be made by inferring a 3D model using artificial intelligence technology with an app that receives a user picture as an input, the dress base color layer and the pattern layer corresponding to the circular texture of the dress are separated. It is separated and transmitted to the generation server 100 .

Next, the generation server 100 requests identification number information for identifying the user and the user picture drawn by the user (Step2). For example, the user terminal 200 generates data such as a user ID (name), a template ID (format index), image 1 (dress base color), image 2 (dress pattern), and the like. ) is sent to

Next, the generation server 100 gives an identification number including a request result and a token ID to the user terminal 200 (Step3).

The generation server 100 generates a 3D character costume model based on artificial intelligence technology based on the image transmitted from the user terminal 200 (Step4).

The generation server 100 performs a 3D character costume through a preparation stage for creating a 3D costume using a user figure image input from the user terminal 200 and building learning data in order to generate an artificial intelligence-based 3D character costume model. create a model

The generation server 100 performs 1) basic clothes data construction, 2) clothes data specialized for content content, 3) representative clothes definition for specialized clothes in content, and 4) learning model expansion process in the learning data construction stage.

First, 1) The basic clothes data construction process among the learning data is as follows.

For real-time content service using 3D costume modeling technology based on artificial intelligence technology, a learning data set suitable for application service is required, and modeling learning data composed of basic clothes models that can be worn by character models are built.

Then, texture learning data composed of texture, color, and UV map of model data for each costume composed of basic clothes model is constructed.

And, rigging learning data that can support animation movement in real time within the content is built.

Next, 2) The process of constructing clothes data specialized for content content is as follows.

In order to more accurately reproduce the user-drawn costume image in 3D, separate learning data is built for the clothes that a specific character mainly wears. For example, learning data such as a dress, which is an Alice costume for "Alice in Wonderland", is constructed.

10 to 13 illustrate a learning data construction process for generating an artificial intelligence 3D character costume model in the present invention.

In FIG. 10, learning data that can be worn with female models such as short-sleeved, long-sleeved female tops and shorts, and long pants female bottoms are exemplified.

In FIG. 11, the texture and UV map of the model for each costume are exemplified.

In FIG. 12 , rigging data for animation support is exemplified.

In FIG. 13, the clothes category learning data required for wearing a character in the content is exemplified.

Next, 3) The process of defining the representative costume for the specialized costume in the content is as follows.

For the learning data to be applied to the characters in the content, a database is secured by defining a representative costume model for each clothing category to cover the diversity of user drawings.

14 is an illustration of a representative costume model definition in the present invention.

In FIG. 14 , in order to define a representative costume model for a sleeveless dress, a representative costume model is defined among various component data such as neckline shape, waist line, skirt hem, skirt width, and skirt length/others.

15 and 16 are diagrams illustrating production according to the definition of a representative costume model in the present invention.

Next, 4) In the learning model expansion process, for the expansion of the learning data, the learning model is extended in the form of transforming the clothes by designating the transformation area of each representative costume and moving the Control Points. 17 illustrates a process of performing local transformation of a representative costume in the present invention.

Then, the generation server 100 erases the texture of the part corresponding to the costume from the UV texture map of the model of the character to be used in the content in the preparation stage for 3D costume generation using the input image of the user picture app, and then separates the By indexing with a template, the user's image can be transformed and processed.

18 is an exemplary view of a character model according to an embodiment of the present invention, and FIGS. 19 to 21 are exemplary views of a UV texture map of a character in the present invention.

19 illustrates a UV texture map for the character model of FIG. 18 .

And, FIG. 20 is an example of making an area corresponding to clothes in a UV texture map as a separate template and specifying a different gray scale for each area to be used as an index.

21 shows a texture map in which the clothing part is omitted from the character's UV texture map.

The user terminal 200 transmits the user ID and identification number to the AR beam project B in charge of content play. In the present invention, the user terminal 200 transmits the identification number given from the generation server 100 to the AR beam project B so that the user can immediately check when the 3D model creation is completed based on the picture drawn by the user ( Step5). For example, the user terminal 200 may deliver the user ID and token ID to the AR beam project (B).

Next, in the AR beam project (B), a request is sent to the generation server 100 to check whether the clothes corresponding to the received identification number have been produced or not (Step6). For example, the AR beam project B may transmit an Inquiry (progress status) and a token ID to the generation server 100 .

When the 3D clothes model generation is completed, the generation server 100 provides the generated 3D clothes model to the AR beam project B (Step7). At this time, the generation server 100 generates a mesh of the 3D model that fits the user's drawing, delivers the character model containing the rigging information to the AR beam project (B), and generates a UV map suitable for the newly created character mesh. It is automatically generated and delivered along with the content. For example, the generation server 100 may transmit 3D costume model information including an image (texture), generation modeling (Fbx), a character model image, and the like to the AR beam project (B).

In AR Beam Project (B), newly created clothes are applied to the content in real time and played. For example, AR finger story content can be played in real time by applying the artificial intelligence character model received from the generation server 100 to the content. 22 and 23 illustrate a final output image according to an embodiment of the present invention.

In one embodiment of the present invention, an AR beam projector is fixed at a predetermined position, and a dedicated desk for optimally implementing the real-time clothes replacement content providing system of the present invention, such as a space for accommodating actual content, is proposed.

Hereinafter, an embodiment of a desk on which the AR beam projector according to the present invention is installed will be described in detail with reference to the accompanying drawings.

3 to 5 are views showing a desk on which an AR beam projector according to an embodiment of the present invention is installed.

And, Figure 6 is a view showing the state of adjusting the position of the rear panel on the desk on which the AR beam projector is installed according to an embodiment of the present invention.

7 is a view showing a state in which the top cover is removed from the insertion groove of the desk on which the AR beam projector is installed according to an embodiment of the present invention.

The desk on which the AR beam projector according to the present invention is installed includes a leg unit 10 and a top plate 20 installed on the upper end of the leg unit 10 .

The upper plate 20 has a fixing part 21 installed on its upper surface, an insertion groove 22 having a predetermined depth is formed, and a receiving hole 23 is formed.

The fixing part 21 sets the installation position of the AR beam projector B that forms a projection area (projection area) on the upper surface of the upper plate 20, and moves in a direction closer to or away from the insertion groove 22. installed possible. This fixing part 21 is composed of a pair of side panels (21a) and a rear panel (21b) connected to the rear side of the side panel (21a).

The side panel 21a is in close contact with both sides of the lower end of the AR beam projector B installed on the rear side of the upper surface of the upper plate 20 . A front panel 21c is installed at the front end of the side panel 21a to prevent the AR beam projector from moving forward.

In addition, a plurality of ventilation holes 21d are formed in the side panel 21a to dissipate heat generated by the AR beam projector to the outside, and the sound emitted through the speaker of the AR beam projector is delivered to the user without degrading the sound quality.

Both ends of the rear panel 21b are fixed by being bolted to the side panel 21a, and are in close contact with the lower rear surface of the AR beam projector B.

Therefore, the side panel 21a and the front panel 21c prevent the AR beam projector B from moving laterally and forward, and the rear panel 21b prevents the AR beam projector B from moving backward. , the installation position of the AR beam projector B is firmly maintained.

On the other hand, the AR beam projector (B) forms a projection area that provides digital information to the real object (A) by irradiating the AR (Augmented Reality) screen to the real object (A) from the front constant distance line, The starting line of the projection area may vary depending on the model of the AR beam projector (B).

Since the starting line of the projection area of the AR beam projector (B) is different for each model, it is necessary to slightly adjust the installation position of the AR beam projector (B) back and forth. In the present invention, a long hole 24 is formed in the upper plate 20 to adjust the installation position of the AR beam projector B, and the hand screw 25 moving in the long hole 24 is connected to the rear panel 21b and was made to contract.

In other words, the long hole 24 is formed in the upper plate 20 long in the direction toward the AR beam projector B, that is, in the front-rear direction, and the hand screw 25 moves along the long hole 24 under the upper plate 20 . After that, it is fastened with the rear panel 21b at a specific position. When the rear panel 21b is moved in the front-rear direction and the AR beam projector B is positioned at a desired location, the hand screw 25 is fastened to the rear panel 21b to fix the position of the rear panel 21b.

The insertion groove 22 is where the real object A, such as a board or paper, on which analog information such as characters or pictures is printed is inserted. The insertion groove 22 is formed to have a depth equal to or similar to the thickness of the actual object (A), so that one sheet of the actual object (A) is inserted into the insertion groove (22).

In addition, a pressing groove 22a deeper than the insertion groove 22 is formed at one end of the insertion groove 22 while being connected to the insertion groove 22 . Therefore, when the real object (A) is inserted into the insertion groove (22), one end of the real object (A) is located above the pressing groove (22a).

Since one end of the real object (A) is located above the pressing groove (22a), when one end of the real object (A) is pressed in the pressing groove (22a), the other end of the real object (A) is lifted upward. It is possible to easily take out the actual object (A) from the insertion groove (22) by holding the other end.

And, when the real object (A) is not inserted into the insertion groove (22) because the AR beam projector (B) is not used, the top cover (C) is installed in the insertion groove (22) to prevent foreign substances in the insertion groove (22). prevent entry

The accommodating hole 23 is formed on the upper surface of the upper plate 20, and a plurality of actual objects A are stored therein. When the real object A is inserted into the accommodation hole 23 , the upper part of the real object protrudes out of the accommodation hole 23 .

On the other hand, a plurality of drawers 30 are provided under the top plate 20 .

The drawer 30 is divided by use. That is, the alphabet block is stored in the leftmost drawer in the drawing, the Hangul block is stored in the drawer next to the right side, and the number block is stored in the drawer next to the right side again. Finally, the real object (A) is stored in the rightmost drawer. Each of the drawers 30 with different uses is provided slidably below the top plate 20 .

In addition, the lower portion of the drawer 30 is provided with a cradle 40 on which a plurality of real objects are seated.

The cradle 40 is provided below the rightmost drawer for accommodating the real object (A). In addition, the cradle 40 is located below the storage hole 23 in which a plurality of real objects (A) are inserted. Therefore, on the right side of the desk, the storage hole 23 for inserting the real object (A), the drawer for accommodating the real object (A), and the cradle 40 for seating the real object (A) are formed up and down, so the real object (A) is formed up and down (A) can be made easier.

The present invention has been described above using several preferred embodiments, but these embodiments are illustrative and not restrictive. Those of ordinary skill in the art to which the present invention pertains will understand that various changes and modifications can be made without departing from the spirit of the present invention and the scope of the rights set forth in the appended claims.

100: providing server 200: user terminal
10: leg 20: top plate
21: fixed part 21a: side panel
21b: rear panel 21c: front panel
21d: ventilation hole 22: insertion groove
22a: press groove 23: storage hole
24: long hole 25: hand screw
30: drawer 40: cradle
A: Real thing B: AR beam projector
C: top cover

Claims (3)

a user terminal in which a dedicated application including a character costume drawing function capable of drawing a user character costume is installed;
When receiving the user picture data drawn by the user through the character costume drawing function of the dedicated application from the user terminal, an AI-based 3D character costume model is generated with respect to the received user picture data, and the generated 3D character costume model is AR A generation server that provides the beam project; and
AR beam project for irradiating an AR (Augmented Reality) screen in the projection area of the content composed of real objects, but applying the 3D character costume model received from the generation server
A real-time clothing replacement content provision system that includes.
The method according to claim 1,
The generation server generates a 3D character costume model through a learning data construction step for generating an artificial intelligence-based 3D character costume model and a preparation step for 3D costume creation using the user figure data input from the user terminal A system for providing real-time clothing replacement content with
3. The method according to claim 2,
The generating server in the learning data building step,
Modeling training data consisting of basic clothing models that can be worn on character models, texture training data consisting of texture, color, and UV maps of model data for each costume consisting of basic clothing models, and rigging learning that can support animation in real time within content basic clothing data construction process, which is the process of building data;
The process of constructing clothes data specialized for content content;
The process of defining the representative costume for the specialized costume in the content; and
A real-time clothing replacement content providing system, characterized in that the learning model extension process is performed to extend the learning model by changing the costume by designating the transformation area of each representative costume.

Images