KR102667880B1 - beauty educational content generating apparatus and method therefor - Google Patents

Abstract

The present invention relates to an apparatus and method for generating beauty education content, which uses a plurality of depth cameras to photograph the movement of at least one person performing a beauty act in all directions to obtain image data and a transmission/reception time corresponding to the image data. a motion information collection unit that collects motion information including data; The image data for each depth camera included in the motion information is input into an artificial neural network-based image analysis model to generate first behavior information including the movement line, movement speed, and movement angle for each body part, and included in the motion information. By analyzing the transmission and reception time data for each depth camera, a plurality of consecutive depth data, speed data, and amplitude data are generated, and noise is removed using the cumulative distribution function and noise removal function according to the size of the amplitude using the amplitude data. a behavior information generator that generates second behavior information by removing the noise and averaging the depth data, speed data, and amplitude data for each body part generated for each depth camera to an intermediate value; The first behavior information and the second behavior information are matched based on time to calculate an error, and the calculated error is removed using a continuous weighted median filter to generate final behavior information. wealth; The movement line, movement speed, and movement angle of each body part over time included in the final action information, as well as the image data, are input into the motion image generation model to generate 3D composite image data about the person's movement that can be rotated 360 degrees. A 3D image generator that generates; Beauty content that generates informational text by analyzing information on the movement line, movement speed, and movement angle of each body part, and creates beauty content by adding subtitles to the 3D synthetic image data generated based on the generated informational text. It may include a creation unit.

Description

Beauty educational content generating apparatus and method therefor}

The present invention relates to a technology for generating beauty education content. More specifically, the beauty education content created as a 360-degree 3D image is automatically generated by analyzing the shooting data by filming the beauty movements of a beauty expert through a plurality of depth cameras. In addition, beauty education content is a 3D image created in the shape of a person by connecting body parts so that actions can be clearly identified, and subtitles provide informational text created by analyzing information on the movement line, movement speed, and movement angle of each body part. The purpose is to provide an automatic beauty content generation device and method that provides beauty education content so that users who watch the content can easily learn beauty movements consisting of complex movements.

As Korean dramas and movies, led by K-POP, are gradually receiving love from people around the world, interest in the field of beauty, including Korean makeup methods, massage methods, and beauty treatments called K-Beauty, is gradually increasing.

In order to perform activities included in this field of beauty, mastery of beauty movements is a basic prerequisite, and in order to learn these beauty movements, the only existing method was to visit an educational institution and spend a lot of money to take a course offline. .

However, this method is not only a realistically difficult alternative for consumers in rural areas or overseas, but also requires relatively high costs, and there are many limitations, especially in the current environment where social distancing due to the coronavirus has been strengthened.

To overcome these limitations, a lot of video content about beauty education has been uploaded on various video-based social networks such as YouTube. There is also the inconvenience of having to create and apply it separately.

Therefore, the need for technology that allows beauty experts to easily create beauty content without additional help from others is gradually increasing.

Republic of Korea Patent Publication No. 10-2020-0089640 (2020.07.27.)

The purpose of the present invention is to provide automatic beauty content creation technology that allows beauty experts to easily create beauty content without any additional help from others. All that is required is for a beauty expert to install a plurality of depth cameras and perform beauty actions. Multiple depth cameras film the person performing the movement, analyze the captured image data and transmission/reception data, and use behavioral information including the movement line, movement speed, and movement angle for each body part to produce 3D composite image data of the person's movement. By creating and analyzing information on movement lines, movement speeds, and movement angles for each body part, the information-providing text created is subtitled to create beauty-related video content in which beauty experts implement beauty movements in 3D images without the help of others. It can be provided to trainees.

According to an embodiment of the present invention, an apparatus for automatically generating beauty content uses a plurality of depth cameras to photograph the movements of at least one person performing a beauty act in all directions and generates image data and transmission/reception time data corresponding to the image data. A motion information collection unit that collects motion information including; The image data for each depth camera included in the motion information is input into an artificial neural network-based image analysis model to generate first behavior information including the movement line, movement speed, and movement angle for each body part, and included in the motion information. By analyzing the transmission and reception time data for each depth camera, a plurality of consecutive depth data, speed data, and amplitude data are generated, and noise is removed using the cumulative distribution function and noise removal function according to the size of the amplitude using the amplitude data. a behavior information generator that generates second behavior information by removing the noise and averaging the depth data, speed data, and amplitude data for each body part generated for each depth camera to an intermediate value; The first behavior information and the second behavior information are matched based on time to calculate an error, and the calculated error is removed using a continuous weighted median filter to generate final behavior information. wealth; The movement line, movement speed, and movement angle of each body part over time included in the final action information, as well as the image data, are input into the motion image generation model to generate 3D composite image data about the person's movement that can be rotated 360 degrees. A 3D image generator that generates; and Beauty, which generates informational text by analyzing information on the movement line, movement speed, and movement angle of each body part, and creates beauty content by adding subtitles to the 3D synthetic image data generated based on the generated informational text. It may include a content creation unit.

According to one embodiment of the present invention, the behavior information generator includes an artificial neural network-based image analysis model formed by a feature point extraction module and a behavior information analysis module, and inputs the image data to the feature point extraction module to extract feature points, The extracted feature points are labeled according to preset body parts, and the feature points and image data for each body part are input into the behavior information analysis module to generate a feature map including the movement line, movement speed, and movement angle for each feature point, and the generated a first behavior information generator that outputs first behavior information by applying an output activation function to the feature map; and analyzing the transmission and reception time data for each depth camera to generate preset continuous depth data, speed data, and amplitude data for each body part, and using the amplitude data to generate a cumulative distribution function and a noise function according to the size of the amplitude. The median value of the depth data, speed data, and amplitude data for each body part generated by each depth camera was normalized to a constant amplitude regardless of the amount of light absorption on the surface using a continuous probability distribution function. It may further include a second behavior information generator that generates second behavior information by averaging it using a filter.

According to one embodiment of the present invention, the final action information generator inputs the first action information and the second action information into an artificial neural network-based final action information generation model, and inputs each of the first action information and the second action information included in the first action information and the second action information. The difference arising from the comparison of the movement line, movement speed, and movement angle of each body part is calculated as an error, and the error is removed by applying a weight to each data in which the error occurred and averaging the calculated values to the median value. Using an artificial neural network-based judgment model that generates final action information with movement line, movement speed, and movement angle, and evaluates accuracy by comparing 3D images and image data generated according to the final action information, the final action information A first expectation value is set for the reference behavior information to determine that the 3D image generated according to the above can be evaluated as being close enough to be recognized as matching the image data and behavior, and the 3D image generated according to the final behavior information and The difference between the first expected values is calculated as the first difference value, and the 3D image generated according to the final action information is used for reference action information to determine that it cannot be evaluated as being close enough to be recognized as matching the image data and action. Set a second expected value, calculate the difference between the 3D image generated according to the final behavior information and the second expected value as a second difference value, and calculate a second difference value based on the sum of the first difference value and the second difference value. Calculate the classification loss value of the human neural network constituting the final behavior information generation model, fix the weight so that the classification loss value is minimal, and update it with the weight applied to each data in which an error occurred in the final behavior information generation model. You can.

According to one embodiment of the present invention, the 3D image generator is implemented as a convolutional artificial neural network consisting of a plurality of computational layers and includes a motion image generation model including a background screen generation module, a motion image generation module, and a 3D image synthesis module. When the image data is input, the 3D image synthesis module extracts a plurality of background images excluding objects included in the image data, performs preprocessing by matching the extracted plurality of background data to the reference image, and preprocessing is performed. Descriptor data is set for each section by analyzing the plurality of background data, deriving descriptor data shared between spatially connected image data among the plurality of image data, and matching the plurality of images based on the derived descriptor data. This generates background screen image data that can be rotated 360 degrees, and the motion image generation module connects each body part around a preset body part to create a 3D human shape, and converts the generated human shape into the final image. Motion image data is generated for each body part according to the movement line, movement speed, and movement angle of each body part included in the behavior information, and the 3D image synthesis module displays the motion of the 3D virtual character on the background screen that can rotate 360 degrees. Synthetic image data can be created by combining images according to location and angle.

According to an embodiment of the present invention, the motion information collection unit performs a two-stage depth measurement when collecting transmission and reception time data, and the first depth measurement measures the depth using a low modulation frequency to detect a preset body part. Measurements are performed with low measurement quality for each region of interest, and the second depth measurement measures depth using a high modulation frequency based on the measurement results for the region of interest for each body part of the first depth measurement with high measurement quality. To increase measurement precision, the low modulation frequency is selected based on Equation 1,

[Equation 1]

The high modulation frequency is selected as a value inversely proportional to the standard deviation measured when using the low modulation frequency, and if the standard deviation is less than a preset threshold, it is determined that the signal-to-noise ratio is high and the standard deviation is preset. The frequency can be relatively higher than when it is greater than the limit value, and measurement precision can be increased through repeated measurements by performing the second depth measurement multiple times.

By using the automatic beauty content generation device and method implemented in accordance with an embodiment of the present invention, you can set the positions of a plurality of depth cameras that can capture the beauty movements of a beauty expert using a stand, and the plurality of depth cameras can be created without additional help from others. By analyzing the image data captured and transmitted and received data, 3D synthetic image data of human movement is generated using behavioral information including the movement line, movement speed, and movement angle of each body part, and the movement line and movement of each body part are generated. By applying subtitles to the information provision text created by analyzing information on speed and movement angle, beauty experts create beauty-related video content that embodies beauty movements in 3D video and provide it to trainees, enabling the creation of beauty content quickly and easily. This is possible, and not only is it a 3D video that allows identification of movements by detailed body part, rather than a video of a person's actual motion, but it also provides informational text with subtitles, which has the effect of allowing trainees to perceive the training content more clearly. .

1 is a configuration diagram of an automatic beauty content generation device implemented according to an embodiment of the present invention.
FIG. 2 is a detailed configuration diagram of the behavior information generator shown in FIG. 1.
Figure 3 is a diagram showing an image analysis model implemented according to an embodiment of the present invention.
Figure 4 is a diagram showing an image analysis model including a feature point extraction module and a behavior information analysis module implemented according to an embodiment of the present invention.
Figure 5 is a diagram showing detailed modules of a motion image generation model implemented according to an embodiment of the present invention.
Figure 6 is a diagram showing a motion image generation model implemented according to an embodiment of the present invention.
Figure 7 is a diagram showing a motion image generation model including a background screen creation module, a motion image creation module, and a 3D image synthesis module implemented according to an embodiment of the present invention.
Figure 8 is a flowchart of a method for automatically generating beauty content according to an embodiment of the present invention.

Below, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein.

The terms used in the present invention are only used to describe specific embodiments and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise.

In the present invention, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by a person of ordinary skill in the technical field to which the present invention pertains.

Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and unless clearly defined in the present invention, should not be interpreted in an idealized or excessively formal sense. No.

It will also be understood that each block of the drawings and the combinations of the flowchart drawings can be performed by computer program instructions, and these computer program instructions can be mounted on a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing equipment. Therefore, the instructions executed through a processor of a computer or other programmable data processing equipment create a means of performing the functions described in the flowchart block(s).

These computer program instructions may also be stored in computer-usable or computer-readable memory that can be directed to a computer or other programmable data processing equipment to implement a function in a particular manner, so that the computer-usable or computer-readable memory The instructions stored in may also produce manufactured items containing instruction means that perform the functions described in the flow diagram block(s).

Computer program instructions can also be mounted on a computer or other programmable data processing equipment, so that a series of operational steps are performed on the computer or other programmable data processing equipment to create a process that is executed by the computer, thereby generating a process that is executed by the computer or other programmable data processing equipment. Instructions that perform processing equipment may also provide steps for executing the functions described in the flow diagram block(s).

Additionally, each block may represent a module, segment, or portion of code that includes one or more executable instructions for executing specified logical function(s).

It should also be noted that in some alternative embodiments it is possible for the functions mentioned in the blocks to occur out of order. For example, it is possible for two blocks shown in succession to be performed substantially at the same time, or it is possible for the blocks to be performed in reverse order depending on the corresponding function.

At this time, the term '~unit' used in this embodiment refers to software or hardware components such as FPGA (field-programmable gate array) or ASIC (Application Specific Integrated Circuit), and '~unit' refers to what role perform them.

However, '~part' is not limited to software or hardware. The '~ part' may be configured to reside in an addressable storage medium and may be configured to reproduce on one or more processors.

Therefore, as an example, '~ part' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and 'parts' may be combined into a smaller number of components and 'parts' or may be further separated into additional components and 'parts'. Additionally, components and 'parts' may be implemented to regenerate one or more CPUs within a device or a secure multimedia card.

In explaining the embodiments of the present invention in detail, the main focus will be on examples of specific systems, but the main point claimed in this specification is that the scope disclosed in this specification is applicable to other communication systems and services with similar technical background. It can be applied within a range that does not significantly deviate, and this can be done at the discretion of a person with skilled technical knowledge in the relevant technical field.

Hereinafter, an apparatus and method for automatically generating beauty content according to an embodiment of the present invention will be described with reference to the drawings.

Figure 1 is a configuration diagram of an automatic beauty content generation device 10 implemented according to an embodiment of the present invention.

Referring to Figure 1, the beauty content automatic generation device 10 includes a motion information collection unit 100, a behavior information generation unit 200, a final behavior information generation unit 300, a 3D image generation unit 400, and beauty content generation. It may include unit 500.

The motion information collection unit 100 uses a plurality of depth cameras to photograph the movements of at least one person performing a beauty treatment in all directions, including image data and preset transmission/reception time data for each body part corresponding to the image data. Movement information can be collected.

According to one embodiment of the present invention, the depth camera fires a pulse-modulated infrared (IR) beam from the origin of the spherical coordinate system to the target and scans it horizontally (pan, ϕ) and up and down (tilt, θ) to detect the sphere surface (sphere). 3-dimensional image of the target in the background based on the time it is reflected and returned to the origin by back scattering that occurs in the point-wise distribution of different discontinuous points (г, θ, ϕ) on the surface, that is, the transmission and reception time A camera capable of acquiring image information may be used.

According to an embodiment of the present invention, a plurality of depth cameras may be installed based on a certain angle to capture the movement of at least one person in all directions, and image data and a preset body corresponding to the image data and time may be installed. Transmission and reception time data for each part can be collected.

Here, the preset body parts refer to human body parts that can identify human movements, such as the fingertips and each joint of both hands, wrists, elbows, shoulders, neck, both ends of each facial feature, and hip joints. , it can be set mainly around the extremities of the body and movable joints, such as knees, ankles, the tips of the toes of both feet, and each joint.

According to one embodiment of the present invention, the motion information collection unit 100 may perform two-stage depth measurement when collecting transmission and reception time data.

According to one embodiment of the present invention, the two-stage depth measurement can be divided into a first depth measurement stage and a second depth measurement stage.

The reason why depth measurement is performed in two stages as in the above embodiment is that when depth measurement is performed using a low modulation frequency when using a single frequency, the maximum range is widened but the measurement quality is lowered, and conversely, , When depth measurement is performed using a high modulation frequency, measurement quality can be improved, but there is a disadvantage in that the maximum range is narrowed. This is to compensate for this disadvantage as much as possible.

According to an embodiment of the present invention, the first depth measurement may measure depth using a low modulation frequency and perform measurement with low measurement quality for a preset region of interest for each body part.

According to an embodiment of the present invention, the second depth measurement measures depth using a high modulation frequency based on the measurement results for the region of interest for each body part of the first depth measurement, thereby increasing measurement precision with high measurement quality. there is.

According to one embodiment of the present invention, the first depth measurement using a low modulation frequency to have a wide maximum depth range can measure the approximate depth over a wide depth range, and the measurement quality at this time is proportional to the modulation frequency. Therefore, the progress measured through the first depth measurement may provide low measurement quality for a wide area of interest.

*According to one embodiment of the present invention, the maximum depth range of the second depth measurement may be set based on the precision of the first depth measurement, and a relatively higher frequency may be selected to provide high measurement quality for a narrow region of interest. By doing so, the error in the first depth measurement result can be compensated.

According to one embodiment of the present invention, a low modulation frequency can be selected based on Equation 1.

here means low modulation frequency, is the speed of light, may mean the maximum depth range.

According to one embodiment of the present invention, a high modulation frequency can be selected as a value inversely proportional to the standard deviation measured when using a low modulation frequency, and if the standard deviation is less than a preset threshold, this means that the signal-to-noise ratio is high. By judging, a relatively higher frequency can be selected than when the standard deviation is greater than a preset limit value.

where standard deviation ( ) can be calculated based on Equation 2.

here may mean the standard deviation to be measured, dp may mean the depth measured through the first depth measurement, μ means the average value of dp for the region of interest (RoI), and N is the number of pixels in the region of interest (RoI). The number can be a natural number.

According to an embodiment of the present invention, measurement precision can be increased through repeated measurements by performing multiple first and second depth measurement steps.

The behavior information generator 200 inputs the image data for each depth camera included in the motion information into an artificial neural network-based image analysis model to generate first behavior information including the movement line, movement speed, and movement angle for each body part. can do.

According to an embodiment of the present invention, an artificial neural network-based image analysis model can receive image data for each depth camera into an input layer and output movement line, movement speed, and movement angle information for each preset body part, which can be used to output information about each body part. It can be learned to generate first action information by grouping by part.

In addition, by analyzing the transmission and reception time data for each depth camera included in the motion information, multiple consecutive depth data, speed data, and amplitude data are generated, and the cumulative distribution function and noise removal function are calculated according to the size of the amplitude using the amplitude data. Second behavioral information can be generated by removing noise and averaging the depth data, speed data, and amplitude data for each depth camera from which the noise has been removed to an intermediate value.

According to an embodiment of the present invention, continuous depth data, speed data, and amplitude data can be generated for each body part over time by analyzing transmission and reception time data for each body part generated for each depth camera.

Here, depth data may refer to data that can recognize the three-dimensional location of the area of interest, and speed data may refer to the movement speed calculated through changes in location over time based on depth data. , Amplitude data may refer to the light intensity of the surface resulting from reflection of damaged light because the light intensity of the surface is different depending on the amplitude of the surface of the light transmitted by the depth camera.

According to an embodiment of the present invention, noise can be removed using amplitude data using a cumulative distribution function and a noise function depending on the size of the amplitude.

According to the above embodiment, the amplitude data may indicate how bright the light reflected by the depth camera is, and this is based on the characteristic that the light intensity of the surface appears differently depending on the amplitude of the surface.

According to an embodiment of the present invention, filtering can be performed using a cumulative distribution function and a noise function to remove noise caused by a non-ideal waveform used in a depth camera and noise generated at the boundary of an object.

According to one embodiment of the present invention, the cumulative distribution function can be performed based on Equation 3.

Φ(x) represents the cumulative distribution function and NF may represent the noise function.

According to one embodiment of the present invention, the noise function (NF) can be performed based on Equation 4.

The behavior information generator 200 will be described in more detail with reference to FIG. 2 .

The final action information generator 300 calculates an error by matching the first action information and the second action information based on time, and removes the error using a continuous weighted median filter for the calculated error to produce the final action. It can be created with information.

According to an embodiment of the present invention, the first behavior information and the second behavior information are input into an artificial neural network-based final behavior information generation model to determine the movement line, movement speed, and The difference that occurs in comparison with the movement angle is calculated as an error, and a weight is applied to each data in which the error occurs, and the calculated values are averaged to the middle value to remove the error, thereby generating depth data, speed data, and amplitude data for each body part. Final action information with can be generated.

According to one embodiment of the present invention, the final behavior information generation model may include a motion analysis module and a matching module.

According to an embodiment of the present invention, the motion analysis module receives depth data, speed data, and amplitude data for each body part generated for each depth camera included in the second behavior information, and receives the movement line, movement speed, and movement angle for each body part. Information can be printed.

According to an embodiment of the present invention, the matching module inputs the movement line, movement speed, and movement angle for each body part included in the first behavior information and the movement line, movement speed, and movement angle for each body part included in the second behavior information. The difference that occurs can be calculated as an error.

According to the above embodiment, the movement line, movement speed, and movement angle for each body part can be output by applying a weight to each data in which an error occurred and averaging the calculated values to a median value to remove the error. Behavioral information can be generated.

Here, weights are given to each body part of the data in which an error occurred, and weights are also given to each first and second behavior information, and the final behavior information can be generated by performing averaging with the median value of the values calculated by applying the weights. And, according to one embodiment of the present invention, the weight values can be updated to relatively improve accuracy by an artificial neural network-based judgment model.

According to an embodiment of the present invention, an artificial neural network-based decision model that evaluates accuracy by comparing 3D images generated according to final action information and image data can be used.

According to the above embodiment, the first reference behavioral information for determining that the 3D image generated according to the final behavioral information using an artificial neural network-based judgment model can be evaluated as being close enough to be recognized as matching the image data and behavior. An expected value can be set, and the difference between the 3D image generated according to the final behavior information and the first expected value can be calculated as the first difference value.

In addition, a second expected value for the standard behavior information is set to determine that the 3D image generated according to the final behavior information cannot be evaluated as being close enough to be recognized as matching the image data and behavior, and is generated according to the final behavior information. The difference between the generated 3D image and the second expected value can be calculated as a second difference value.

According to the above embodiment, the sum of the first difference value and the second difference value is calculated as the segmentation loss value of the human neural network constituting the final behavioral information generation model based, the weight is fixed so that the segmentation loss value is minimal, and the final behavior information generation model is calculated. It can be updated with the weight applied to each data in which an error occurred in the behavioral information generation model.

The 3D image generator 400 inputs the movement line, movement speed, movement angle, and image data for each body part over time included in the final action information into the motion image generation model to determine the movement of the person capable of 360 degree rotation. 3D synthetic image data can be generated.

According to an embodiment of the present invention, the movement line, movement speed, and movement angle of each body part over time included in the final action information are entered into the motion image generation model, each body part is used as a starting point, and these are connected to create a human A 3D image can be created in the shape of , and a 3D image that can represent a person's movement can be created by reflecting values that change over time.

According to an embodiment of the present invention, background screen image data can be generated by inputting image data into a motion image generation model and using a background image included in the image data.

The 3D image generator 400 will be described in more detail with reference to FIG. 5 .

The beauty content creation unit 500 generates informational text by analyzing information on the movement line, movement speed, and movement angle of each body part, and adds subtitles to the 3D synthetic image data generated based on the generated informational text. You can create beauty content.

According to one embodiment of the present invention, the informational text may be information in the form of text created to match the image over time by analyzing information on the movement line, movement speed, and movement angle of each body part, and the body part may be Arithmetic values of the direction, speed, and angle of movement may be converted into text and provided.

FIG. 2 is a detailed configuration diagram of the behavior information generator 200 shown in FIG. 1.

The behavior information generation unit 200 may include a first behavior information generation unit 210 and a second behavior information generation unit 220, and the first behavior information generation unit 210 may include a feature point extraction module and a behavior information analysis unit. It may include an artificial neural network-based image analysis model formed as a module.

According to an embodiment of the present invention, the first behavior information generator 210 may input image data into a feature point extraction module to extract feature points, and label the extracted feature points for each preset body part.

According to one embodiment of the present invention, the feature point extraction module may be a model learned to extract feature points that can specify a preset body part from an image.

According to an embodiment of the present invention, at least one feature point among the extracted feature points can be matched to a body part and labeled.

Additionally, by inputting labeled feature points and image data into the behavior information analysis module, a feature map including the movement line, movement speed, and movement angle for each feature point is generated, and an output activation function is applied to the generated feature map to generate first behavior information. Can be printed.

According to an embodiment of the present invention, the feature map may be data generated by grouping the movement line, movement speed, and movement angle for each feature point to create a plurality of groups and sorting them into body parts matching the feature points.

The second behavior information generator 220 may analyze transmission and reception time data for each depth camera and generate preset continuous depth data, speed data, and amplitude data for each body part.

According to one embodiment of the present invention, the second behavior information generator 220 uses amplitude data to remove detected noise using a cumulative distribution function and a noise function according to the size of the amplitude, and uses a continuous probability distribution function. Second behavioral information can be generated by averaging the depth data, speed data, and amplitude data for each body part generated by each depth camera, which are normalized to a constant amplitude regardless of the amount of light absorption on the surface, using a median filter.

According to one embodiment of the present invention, the cumulative distribution function can be performed based on Equation 3.

According to one embodiment of the present invention, the noise function (NF) can be performed based on Equation 4 below.

According to an embodiment of the present invention, the median filter may refer to a filter that can remove errors by applying weights to each data in which errors occur and averaging the calculated values to a median value.

Here, weights are given to each body part of the data in which an error occurred, and weights are also given to each first and second behavior information, and the final behavior information can be generated by performing averaging with the median value of the values calculated by applying the weights. And, according to one embodiment of the present invention, the weight values can be updated to relatively improve accuracy by an artificial neural network-based judgment model.

According to an embodiment of the present invention, an artificial neural network-based decision model that evaluates accuracy by comparing 3D images generated according to final action information and image data can be used.

According to the above embodiment, the first reference behavioral information for determining that the 3D image generated according to the final behavioral information using an artificial neural network-based judgment model can be evaluated as being close enough to be recognized as matching the image data and behavior. An expected value can be set, and the difference between the 3D image generated according to the final behavior information and the first expected value can be calculated as the first difference value.

In addition, a second expected value for the standard behavior information is set to determine that the 3D image generated according to the final behavior information cannot be evaluated as being close enough to be recognized as matching the image data and behavior, and is generated according to the final behavior information. The difference between the generated 3D image and the second expected value can be calculated as a second difference value.

According to the above embodiment, the sum of the first difference value and the second difference value is calculated as the segmentation loss value of the human neural network constituting the final behavioral information generation model based, the weight is fixed so that the segmentation loss value is minimal, and the final behavior information generation model is calculated. It can be updated with the weight applied to each data in which an error occurred in the behavioral information generation model.

Figure 3 is a diagram showing an image analysis model implemented according to an embodiment of the present invention.

Referring to FIG. 3, an image analysis model implemented according to an embodiment of the present invention is shown. The image analysis model can be formed as a convolutional network, and a plurality of image data collected by each depth camera is input to the input layer. It is possible to receive input and output first action information.

Figure 4 is a diagram showing an image analysis model including a feature point extraction module and a behavior information analysis module implemented according to an embodiment of the present invention.

Referring to FIG. 4, the data flow of the feature point extraction module and the behavioral information analysis module included in the image analysis model implemented according to an embodiment of the present invention is shown.

According to an embodiment of the present invention, when a plurality of image data collected from each depth camera is input to the feature point extraction module, feature points that can specify a preset body part are extracted from the image, and at least one feature point among the extracted feature points is extracted from the image. Labeled feature point information may be generated by matching and labeling body parts.

According to an embodiment of the present invention, when labeled feature point information and a plurality of image data collected from each depth camera are input to the behavior information analysis module, first behavior information including the movement line, movement speed, and movement angle for each feature point is generated. It can be.

Figure 5 is a diagram showing detailed modules of a motion image generation model implemented according to an embodiment of the present invention.

Referring to FIG. 5, the 3D image generator 400 is implemented as a convolutional artificial neural network consisting of a plurality of computational layers and is capable of creating a motion image generation model including a background screen creation module, a motion image generation module, and a 3D image synthesis module. there is.

According to one embodiment of the present invention, when the image data is input, the 3D image synthesis module extracts a plurality of background images excluding objects included in the image data, and performs preprocessing by matching the extracted plurality of background data to the reference image. can do.

Here, the reference image may refer to a reference image whose position and size are preset to match multiple generated background images and combine them into an omnidirectional image.

According to the above embodiment, descriptor data is set for each section by analyzing a plurality of background data on which preprocessing has been performed, descriptor data shared between spatially connected image data among the plurality of image data is derived, and the derived descriptor data is By matching multiple images as a standard, background screen image data that can be rotated 360 degrees can be generated.

Here, descriptor data may refer to data about numbers that can represent the features calculated by converting meaningful features in the image into appropriate numbers in order to measure the similarity between two images.

According to an embodiment of the present invention, characteristic parts (scale-space extrema detection) for each image part can be found for a plurality of acquired background data and classified into feature points, and among the classified feature points, the core is the final reliable feature point. By selecting the feature points, the final feature points can be selected based on the pixel value (Intensity) of the key feature points, the location or size of the corner of the object among the key feature points, etc.

According to the above embodiment, the gradient is calculated for the surrounding area of the final feature point, the direction in which the pixels in the surrounding area are pointing as a whole is obtained, and the direction in which the pixels in the surrounding area are pointing is rotated to 0 degrees, and the part hit by the surrounding area is set as descriptor data. You can.

According to an embodiment of the present invention, the descriptor data may include pixel values in the surrounding area of the final feature point, and histogram information based on direction information of pixel values in the surrounding area based on the final feature point is also included, so the descriptor data can be used to By comparing a plurality of image data that changes depending on the shooting angle and identifying a target point representing the same section, neighboring image data can be accurately matched to generate background screen image data.

According to an embodiment of the present invention, the motion image generation module connects each body part around a preset body part to generate a 3D human shape, and matches the generated human shape to each body part included in the final action information. Motion image data can be generated for each body part according to the movement line, movement speed, and movement angle.

According to one embodiment of the present invention, the 3D image synthesis module can generate synthesized image data by synthesizing the motion image of the 3D virtual character on the background screen that can rotate 360 degrees according to the position and angle.

Figure 6 is a diagram showing a motion image generation model implemented according to an embodiment of the present invention.

Referring to Figure 6, a motion image generation model implemented based on a convolutional neural network according to an embodiment of the present invention is shown, and the image data for each depth camera and the movement line, movement speed, and movement angle information for each body part are used for operation. Once input is received from the image model, 3D synthetic image data can be output.

Figure 7 is a diagram showing a motion image generation model including a background screen creation module, a motion image creation module, and a 3D image synthesis module implemented according to an embodiment of the present invention.

Referring to FIG. 7, the data flow between the background screen creation module, the motion image creation module, and the 3D image synthesis module included in the motion image generation model implemented according to an embodiment of the present invention is shown.

According to an embodiment of the present invention, image data for each depth camera is input into the background image generation module to output background image data, and input movement line, movement speed, and movement angle information for each body part into the motion image generation module. Motion video data can be output.

According to the above embodiment, the background screen image data and the motion image data output from the background screen generation module and the motion image generation module, respectively, can be input to the 3D image synthesis module to output 3D composite image data.

Figure 8 is a flowchart of a method for automatically generating beauty content according to an embodiment of the present invention.

The movement of at least one person performing a beauty act is photographed in all directions to collect motion information including image data and transmission/reception time data corresponding to the image data (S10).

According to an embodiment of the present invention, the movements of at least one person performing a beauty act are photographed in all directions using a plurality of depth cameras, including image data and preset transmission/reception time data for each body part corresponding to the image data. Movement information can be collected.

According to an embodiment of the present invention, the depth camera fires a pulse-modulated infrared (IR) beam from the origin of the spherical coordinate system to the target and scans it horizontally (pan, ϕ) and up and down (tilt, θ) to capture the sphere surface (sphere). A three-dimensional image of a target in the background based on the time it reflects and returns to the origin due to back scattering that occurs in the point wise distribution of different discontinuous points (г, θ, ϕ) on the surface, that is, the transmission and reception time. A camera capable of acquiring information may be used.

According to an embodiment of the present invention, a plurality of depth cameras may be installed based on a certain angle to capture the movement of at least one person in all directions, and image data and a preset body corresponding to the image data and time may be installed. Transmission and reception time data for each part can be collected.

According to one embodiment of the present invention, when collecting transmission and reception time data, two-stage depth measurement can be performed.

*According to one embodiment of the present invention, the two-stage depth measurement can be divided into a first depth measurement stage and a second depth measurement stage.

The reason why depth measurement is performed in two stages as in the above embodiment is that when depth measurement is performed using a low modulation frequency when using a single frequency, the maximum range is widened but the measurement quality is lowered, and conversely, , When depth measurement is performed using a high modulation frequency, measurement quality can be improved, but there is a disadvantage in that the maximum range is narrowed. This is to compensate for this disadvantage as much as possible.

According to an embodiment of the present invention, the first depth measurement may measure depth using a low modulation frequency and perform measurement with low measurement quality for a preset region of interest for each body part.

According to an embodiment of the present invention, the second depth measurement measures depth using a high modulation frequency based on the measurement results for the region of interest for each body part of the first depth measurement, thereby increasing measurement precision with high measurement quality. there is.

According to one embodiment of the present invention, the first depth measurement using a low modulation frequency to have a wide maximum depth range can measure the approximate depth over a wide depth range, and the measurement quality at this time is proportional to the modulation frequency. Therefore, the progress measured through the first depth measurement may provide low measurement quality for a wide area of interest.

According to one embodiment of the present invention, the second depth measurement may have a maximum depth range set based on the precision of the first depth measurement, by selecting a relatively higher frequency to provide high point quality for a narrow region of interest. Errors in the first depth measurement result may be compensated.

According to one embodiment of the present invention, a low modulation frequency can be selected based on Equation 1.

According to one embodiment of the present invention, a high modulation frequency can be selected as a value inversely proportional to the standard deviation measured when using a low modulation frequency, and if the standard deviation is less than a preset threshold, this means that the signal-to-noise ratio is high. By judging, a relatively higher frequency can be selected than when the standard deviation is greater than a preset limit value.

where standard deviation ( ) can be calculated based on Equation 2.

According to an embodiment of the present invention, measurement precision can be increased through repeated measurements by performing multiple first and second depth measurement steps.

The image data for each depth camera included in the motion information is input into an artificial neural network-based video analysis model to generate first behavior information including the movement line, movement speed, and movement angle of each body part (S20).

According to an embodiment of the present invention, the image data for each depth camera included in the motion information is input into an artificial neural network-based image analysis model to generate first behavior information including the movement line, movement speed, and movement angle for each body part. According to one embodiment, an artificial neural network-based image analysis model can receive image data for each depth camera into an input layer and output movement line, movement speed, and movement angle information for each preset body part, which can be used for each body part. It can be learned to generate first action information by grouping by body part.

Transmission and reception time data included in the motion information is analyzed to generate a plurality of consecutive depth data, speed data, and amplitude data, noise is removed, and then averaged to an intermediate value to generate second behavior information (S30).

*According to one embodiment of the present invention, the transmission and reception time data for each depth camera included in the motion information is analyzed to generate a plurality of consecutive depth data, speed data, and amplitude data, and the amplitude data is used to determine the size of the amplitude. Noise can be removed using a cumulative distribution function and a noise removal function, and the depth data, speed data, and amplitude data for each depth camera from which the noise has been removed can be averaged to a median value to generate second behavioral information.

According to an embodiment of the present invention, continuous depth data, speed data, and amplitude data can be generated for each body part over time by analyzing transmission and reception time data for each body part generated for each depth camera.

According to an embodiment of the present invention, noise can be removed using amplitude data using a cumulative distribution function and a noise function depending on the size of the amplitude.

According to the above embodiment, the amplitude data may indicate how bright the light reflected by the depth camera is, and this is based on the characteristic that the light intensity of the surface appears differently depending on the amplitude of the surface.

According to an embodiment of the present invention, filtering can be performed using a cumulative distribution function and a noise function to remove noise caused by a non-ideal waveform used in a depth camera and noise generated at the boundary of an object.

According to one embodiment of the present invention, the cumulative distribution function can be performed based on Equation 3.

According to one embodiment of the present invention, the noise function (NF) can be performed based on Equation 4.

The first behavior information and the second behavior information are matched based on time to calculate the error, and the calculated error is removed using a continuous weighted median filter to generate final behavior information (S40).

According to an embodiment of the present invention, the first behavior information and the second behavior information are matched based on time to calculate the error, and the calculated error is removed using a continuous weighted median filter to produce the final behavior information. It can be created with

According to an embodiment of the present invention, the first behavior information and the second behavior information are input into an artificial neural network-based final behavior information generation model to determine the movement line, movement speed, and The difference occurring in comparison with the movement angle is calculated as an error, a weight is applied to each data in which the error occurred, and the calculated values are averaged to the middle value to remove the error, resulting in depth data, speed data, and amplitude data for each body part. Final action information with can be generated.

According to one embodiment of the present invention, the final behavior information generation model may include a motion analysis module and a matching module.

According to an embodiment of the present invention, the motion analysis module receives depth data, speed data, and amplitude data for each body part generated for each depth camera included in the second behavior information, and receives the movement line, movement speed, and movement angle for each body part. Information can be printed.

According to an embodiment of the present invention, the matching module inputs the movement line, movement speed, and movement angle for each body part included in the first behavior information and the movement line, movement speed, and movement angle for each body part included in the second behavior information. The difference that occurs can be calculated as an error.

According to the above embodiment, the movement line, movement speed, and movement angle for each body part can be output by applying a weight to each data in which an error occurred and averaging the calculated values to a median value to remove the error. Behavioral information can be generated.

Here, weights are given to each body part of the data in which an error occurred, and weights are also given to each first and second behavior information, and the final behavior information can be generated by performing averaging with the median value of the values calculated by applying the weights. And, according to one embodiment of the present invention, the weight values can be updated to relatively improve accuracy by an artificial neural network-based judgment model.

According to an embodiment of the present invention, an artificial neural network-based decision model that evaluates accuracy by comparing 3D images generated according to final action information and image data can be used.

According to the above embodiment, the first reference behavioral information for determining that the 3D image generated according to the final behavioral information using an artificial neural network-based judgment model can be evaluated as being close enough to be recognized as matching the image data and behavior. An expected value can be set, and the difference between the 3D image generated according to the final behavior information and the first expected value can be calculated as the first difference value.

In addition, a second expected value for the standard behavior information is set to determine that the 3D image generated according to the final behavior information cannot be evaluated as being close enough to be recognized as matching the image data and behavior, and is generated according to the final behavior information. The difference between the generated 3D image and the second expected value can be calculated as a second difference value.

According to the above embodiment, the sum of the first difference value and the second difference value is calculated as the segmentation loss value of the human neural network constituting the final behavioral information generation model based, the weight is fixed so that the segmentation loss value is minimal, and the final behavior information generation model is calculated. It can be updated with the weight applied to each data in which an error occurred in the behavioral information generation model.

The movement line, movement speed, and movement angle of each body part over time included in the final action information, as well as the image data, are input into the motion image generation model to generate 3D synthetic image data (S50).

According to an embodiment of the present invention, the movement line, movement speed, movement angle, and image data for each body part over time included in the final action information are input into a motion image generation model to determine the movement of a person capable of 360 degree rotation. 3D synthetic image data can be generated.

According to an embodiment of the present invention, the movement line, movement speed, and movement angle of each body part over time included in the final action information are entered into the motion image generation model, each body part is used as a starting point, and these are connected to create a human A 3D image can be created in the shape of , and a 3D image that can represent a person's movement can be created by reflecting values that change over time.

According to an embodiment of the present invention, background screen image data can be generated by inputting image data into a motion image generation model and using a background image included in the image data.

Information on the movement line, movement speed, and movement angle of each body part is analyzed to generate informational text, and beauty content is created by adding subtitles to the 3D composite image data based on the informational text (S60).

According to an embodiment of the present invention, information on the movement line, movement speed, and movement angle of each body part is analyzed to generate informational text, and subtitles are added to the 3D synthetic image data generated based on the generated informational text. You can create beauty content.

According to one embodiment of the present invention, the informational text may be information in the form of text created to match the image over time by analyzing information on the movement line, movement speed, and movement angle of each body part, and the body part may be Arithmetic values of the direction, speed, and angle of movement may be converted into text and provided.

The embodiments of the present invention are not implemented only through the apparatus and/or method described above. Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto and is subject to the following claims. Various modifications and improvements made by those skilled in the art using the basic concept of the present invention defined in also fall within the scope of the present invention.

Claims (2)

a motion information collection unit that collects motion information including image data and transmission/reception time data corresponding to the image data by photographing the movements of at least one person performing a beauty treatment in all directions using a plurality of depth cameras;
The image data for each depth camera included in the motion information is input into an artificial neural network-based image analysis model to generate first behavior information including the movement line, movement speed, and movement angle for each body part, and included in the motion information. By analyzing the transmission and reception time data for each depth camera, a plurality of consecutive depth data, speed data, and amplitude data are generated, and noise is removed using the cumulative distribution function and noise removal function according to the size of the amplitude using the amplitude data. a behavior information generator that generates second behavior information by removing the noise and averaging the depth data, speed data, and amplitude data for each body part generated for each depth camera to an intermediate value;
The first behavior information and the second behavior information are matched based on time to calculate an error, and the calculated error is removed using a continuous weighted median filter to generate final behavior information. wealth;
The movement line, movement speed, and movement angle of each body part over time included in the final action information, as well as the image data, are input into the motion image generation model to generate 3D composite image data about the person's movement that can be rotated 360 degrees. A 3D image generator that generates; and
Beauty content that generates informational text by analyzing information on the movement line, movement speed, and movement angle of each body part, and creates beauty content by adding subtitles to the 3D synthetic image data generated based on the generated informational text. Includes a generating unit,
The behavioral information generator,
It includes an artificial neural network-based image analysis model formed by a feature point extraction module and a behavioral information analysis module, inputs the image data to the feature point extraction module to extract feature points, and labels the extracted feature points by preset body parts, By inputting the feature points and image data for each body part into the behavior information analysis module, a feature map including the movement line, movement speed, and movement angle for each feature point is generated, and an output activation function is applied to the generated feature map to generate first behavior information. a first behavior information generator that outputs; and
By analyzing the transmission and reception time data for each depth camera, continuous depth data, speed data, and amplitude data for each body part that are preset are generated, and the amplitude data is used to use a cumulative distribution function and a noise function according to the size of the amplitude. The detected noise is removed, and the depth data, speed data, and amplitude data for each body part generated by each depth camera are normalized to a constant amplitude regardless of the amount of light absorption on the surface using a continuous probability distribution function. It includes a second behavior information generator that generates second behavior information by averaging,
The second behavior information generator performs filtering using a cumulative distribution function and a noise function to remove noise caused by the non-ideal waveform used in the depth camera and noise generated at the boundary of the object, and the cumulative distribution function is calculated using a mathematical formula. It is performed based on equation 3,
[Equation 3]

The final action information generator,
The first and second action information are input into an artificial neural network-based final action information generation model, and the movement line, movement speed, and movement angle of each body part included in the first and second action information are compared. The difference is calculated as an error, a weight is applied to each data in which the error occurred, and the calculated values are averaged to the median to remove the error, thereby generating final behavioral information with the movement line, movement speed, and movement angle for each body part. do,
Using an artificial neural network-based judgment model that evaluates accuracy by comparing the 3D image and image data generated according to the final behavioral information,
Setting a first expected value for the standard behavior information to determine that the 3D image generated according to the final behavior information can be evaluated as being close enough to be recognized as matching the image data and behavior, and generating according to the final behavior information Calculate the difference between the 3D image and the first expected value as the first difference value,
Setting a second expected value for the standard behavior information to determine that the 3D image generated according to the final behavior information cannot be evaluated to be approximate enough to be recognized as matching the image data and behavior, and generated according to the final behavior information The difference between the 3D image and the second expected value is calculated as a second difference value,
The sum of the first difference value and the second difference value is calculated as a classification loss value of the human neural network constituting the final action information generation model based, the weight is fixed so that the division loss value is minimal, and the final action information generation model is calculated. It is updated with the weight applied to each data in which an error occurred in the information generation model,
The operation information collection unit,
In collecting transmission and reception time data, two-stage depth measurement is performed using two modulation frequencies,
The first depth measurement measures depth using a relatively low modulation frequency of the two modulation frequencies and performs measurement with low measurement quality for a preset region of interest for each body part,
The second depth measurement measures depth using the relatively higher modulation frequency of the two modulation frequencies based on the measurement results for the area of interest for each body part of the first depth measurement, increasing measurement precision with high measurement quality. ,
The low modulation frequency is selected based on Equation 1,
[Equation 1]

The high modulation frequency is selected as a value inversely proportional to the standard deviation measured when using the low modulation frequency, and if the standard deviation is less than a preset threshold, it is determined that the signal-to-noise ratio is high and the standard deviation is preset. It can be at a relatively higher frequency than when it is greater than the limit value,
A device for automatically generating beauty content, characterized in that the second depth measurement is performed multiple times to increase measurement precision through repeated measurements.
The 3D image generator of claim 1,
It is implemented as a convolutional artificial neural network consisting of a plurality of computational layers and includes a motion image generation model including a background screen generation module, a motion image generation module, and a 3D image synthesis module,
When the 3D image synthesis module inputs the image data, it extracts a plurality of background images excluding objects included in the image data, performs pre-processing by matching the extracted plurality of background data to the reference image, and performs pre-processing By analyzing the background data, descriptor data is set for each section, descriptor data shared between spatially connected image data among the plurality of image data is derived, and the plurality of images are matched based on the derived descriptor data to create a 360 Create background image data that can be rotated,
The motion image generation module connects each body part around a preset body part to create a 3D human shape, and converts the generated human shape into the movement line, movement speed, and movement speed for each body part included in the final action information. Generates moving motion image data for each body part according to the movement angle,
The 3D image synthesis module is an automatic beauty content generation device, wherein the 3D image synthesis module synthesizes the motion image of the 3D virtual character on the background screen that can be rotated 360 degrees according to the position and angle to generate synthesized image data.