KR20210157271A - Dynamic projection mapping interactive gesture recognition media content prodcution method and apparatus - Google Patents

Abstract

According to an embodiment of the present invention, a method for creating gesture recognition interactive media content in connection with dynamic projection mapping, which is performed by a device, comprises the steps of: (a) reading a configuration file matched with a predetermined gesture with respect to a media particle effect, and generating a motion history image based on a gesture image collected from a camera that photographs the gesture of a user; (b) inputting the motion history image to a gesture inference model, and determining a gesture type of the current state of the user based on the gesture type information output from the gesture inference model; and (c) performing projection mapping on the media particle effect corresponding to each gesture type determined according to changes in the gesture of the user after tracking the location of the user, based on the spatial mapping information between the camera and a projector. The gesture inference model is trained through a convolutional neural network (CNN). Therefore, the method enables a user with no programming experience to easily create interactive media content.

Description

DYNAMIC PROJECTION MAPPING INTERACTIVE GESTURE RECOGNITION MEDIA CONTENT PRODCUTION METHOD AND APPARATUS

The present invention relates to a method and apparatus for producing interactive media content with gesture recognition linked to dynamic projection mapping.

Because projection mapping can create a virtual stage space on the plane of the projected object, it is possible to develop a story through rapid stage scene change. In traditional theatrical performances and stage production of various large events, modern high-tech technology is a key factor in the production of such content. Compared to the existing stage production method, the stage production is free according to the creator's intention and there are less physical and economic limitations. One of the factors that strengthens the interest and immersion of the audience is that the stage can be changed quickly without a break according to the story. Accordingly, convergence contents using mapping are spreading in various fields such as public space displays, exhibition spaces, performance stages, and fashion shows, including marketing fields such as digital signage.

In particular, as a representative interactive media content type, continuous attempts are being made to produce “gesture-based dynamic projection mapping content” that tracks the location and motion of objects or people in real time and generates appropriate effects.

However, dynamic projection mapping content creation requires a lot of preparation in advance and is difficult to implement, and the existing paid commercial projection mapping tools do not support gesture-based dynamic projection mapping, and it is impossible for developers to directly modify the tool. . Therefore, there is a demand for an “interactive content creation framework” that enables efficient media effects according to the performer's motion to be easily performed without a complicated programming process.

In order to solve the above problems, the present invention provides a method and apparatus for producing interactive media content with dynamic projection mapping interlocking gesture recognition that allows a user without programming experience to easily produce interactive media content that responds to a user's gesture. There is a purpose.

However, the technical problems to be achieved by the present embodiment are not limited to the technical problems described above, and other technical problems may exist.

As a technical means for achieving the above-mentioned technical problem, the method for producing interactive media content with dynamic projection mapping linked gesture recognition performed by the device according to the present invention is (a) a configuration file in which a predetermined gesture and media particle effect are matched in advance. generating a motion history image based on a gesture image collected from a camera for reading and photographing a user's gesture; (b) inputting a motion history image to a gesture inference model, and determining a gesture type of a user's current state based on gesture type information output from the gesture inference model; And (c) based on the spatial mapping information between the camera and the projector, the media particle effect corresponding to each gesture type determined according to the user's gesture change by tracking the user's location and projection mapping; is learned through a convolutional neural network (CNN).

Step (a) generates a motion history image by sequentially accumulating continuous frame images from before a predetermined frame to the present with respect to the gesture image based on the current frame time.

When generating a motion history image, the user's silhouette region is extracted for each frame image, the extracted user silhouette information is stored in a circular queue, and the silhouette information from the past time to the present time is sequentially retrieved from the circular queue. It is read and compressed into a grayscale image, but each frame image consists of a silhouette area filled with pixel values matching the time stamp and the remaining background area filled with black pixel values.

In step (b), a gesture queue is configured with the gesture type information output from the gesture inference model, and noise for each frame image is removed using the gesture queue.

A dynamic projection mapping-linked gesture recognition interactive media content production apparatus includes: a memory in which a dynamic projection mapping-linked gesture recognition interactive media content production method program is stored; A processor for executing a program stored in the memory, wherein the processor reads a configuration file in which a predetermined gesture and media particle effect are matched in advance by the execution of the program, and records the gesture image collected from a camera that captures the user's gesture. Generate a motion history image based on the motion history image, input the motion history image to the gesture inference model, determine the gesture type of the user's current state based on the gesture type information output from the gesture inference model, and spatial mapping between the camera and the projector Based on the information, the media particle effect corresponding to each gesture type determined according to the user's gesture change is projected by tracking the user's location, but the gesture inference model is learned through a convolutional neural network (CNN) .

The processor generates a motion history image by sequentially accumulating continuous frame images from before a predetermined frame to the present with respect to the gesture image based on the current frame time point.

When generating a motion history image, the processor extracts the user's silhouette region for each frame image, stores the extracted user silhouette information in a circular queue, and collects the silhouette information from the past time to the present time from the circular queue. It is read sequentially and compressed into a grayscale image, but each frame image is composed of a silhouette area filled with pixel values matching the time stamp and the remaining background area filled with black pixel values.

When determining the gesture type, the processor configures a gesture queue with gesture type information output from the gesture inference model, and removes noise for each frame image by using the gesture queue.

The computer-readable recording medium stores a computer program for performing a method for creating interactive media content for dynamic projection mapping-linked gesture recognition.

The present invention is intended to solve the problems of the prior art, and the present invention can provide a content creation framework that enables a user without programming experience to easily produce interactive media content that responds to a user's gesture.

In addition, the user can create interactive media content that responds to user gestures without a separate programming process by expressing gestures and media effects numerically and defining gestures and corresponding media effects in a text-based configuration file.

1 is a diagram illustrating the configuration of a dynamic projection mapping-linked gesture recognition interactive media content creation system according to an embodiment of the present invention.
2 is a detailed structural diagram of a proposed interactive media content production system according to an embodiment of the present invention.
3 illustrates an algorithm for removing noise and finally determining a gesture type using a gesture queue according to an embodiment of the present invention.
4 is a diagram illustrating an arrangement of motion history images for a 'kick-left' motion according to an embodiment of the present invention.
5 illustrates a 3-layer synthetic convolutional neural network including 2-layer all-connection layers according to an embodiment of the present invention.
6 is a diagram illustrating a 5-layer synthetic convolutional neural network including a 1-layer all-connection layer according to an embodiment of the present invention.
7 illustrates motion history images for each target gesture according to an embodiment of the present invention.
8 illustrates an interaction particle effect according to each target gesture according to an embodiment of the present invention.
9 is a view showing motion history images of erroneous operations learned by classifying into three types according to an embodiment of the present invention.
10 is a graph of the experimental results of [Table 4] - 8 filters.
11 is a graph showing the experimental results of [Table 5] - condition3.
12 illustrates a screen for applying a media effect according to a gesture according to an embodiment of the present invention.
13 is a flowchart illustrating a method for creating interactive media content for gesture recognition linked to dynamic projection mapping according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. However, the present invention may be embodied in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the present specification, when a part is "connected" with another part, it is not only "directly connected" but also "electrically connected" with another element interposed therebetween. include

Throughout the present specification, when a member is said to be “on” another member, this includes not only a case in which a member is in contact with another member but also a case in which another member is present between the two members.

Throughout the present specification, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated. As used throughout this specification, the terms "about", "substantially", etc. are used in or close to the numerical value when manufacturing and material tolerances inherent in the stated meaning are presented, and are used in the meaning of the present invention. It is used to prevent an unscrupulous infringer from using the disclosure in which exact or absolute figures are mentioned for better understanding. As used throughout the present specification, the term "step for (to)" or "step for" does not mean "step for".

The above description of the present invention is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

On the other hand, looking at the existing deep learning-based approaches for gesture recognition, gesture recognition is one of gesture recognition technologies that interacts with a computer by recognizing the movement of a user's body through various sensors such as infrared or image sensors. For gesture recognition technology, a sensor capable of receiving a user's gesture is required, and it is divided into two types. There is a contact-type method in which a user directly touches a sensor or device with a body to obtain data, and a non-contact method in which data is obtained using remote and near-field sensors. Recently, as large-scale data set collection becomes easy and high-performance computing using GPUs becomes common, studies to recognize gestures using deep learning technology are continuing.

[Table 1] Deep Learning Approaches for Gesture and Motion Recognition

For example, deep learning approaches for gesture recognition are roughly divided into three categories as shown in [Table 1]. The first approach uses a three-dimensional filter in a convolutional neural network to extract discriminative features in spatial and temporal dimensions. The second approach is to calculate in advance motion features such as a two-dimensional optical flow map and use them as input to the network. The third approach is a temporal method that utilizes sequence data, such as combining a Recurrent Neural Network (RNN) and a Long Short Term Memory (LSTM).

Also, looking at the existing interactive media art based on dynamic projection mapping, projection mapping is a media art technique of an immersive interface that projects an image of content suitable for the surface of an actual building or object. In the field of media art, the projection-based spatial augmented reality technique is called projection mapping. Projection mapping is used as a device to map colorful images on large buildings such as media façades or to increase the interest and immersion of audiences or customers in exhibitions, performances, and marketing fields. In general, the projection mapping technology seen in the performing arts expresses visual augmentation by mapping the surface of the indoor environment of the performance hall or objects used, such as art tools used for stage sets or directing. Recently, the technology has advanced to the extent that it can be used as a screen beyond 3D and 4D to a 360-degree dome, and at the same time, it is combined with VR/AR, and the possibility of its use is growing. Among them, dynamic projection mapping is one type of representative interactive media content. Various interaction effects are provided by applying the projection technique to dynamic objects, such as moving positions or changing shapes on stage. The strength of dynamic projection mapping is that it can increase the audience's immersion because the mapped image changes according to the movement of an object.

1 is a diagram illustrating the configuration of a dynamic projection mapping-linked gesture recognition interactive media content creation system according to an embodiment of the present invention.

Referring to FIG. 1 , the present invention includes a dynamic projection mapping-linked gesture recognition interactive media content creation apparatus 100 , a camera 10 , and a projector 20 .

Hereinafter, for convenience of description, the apparatus 100 for producing interactive media content with dynamic projection mapping linked gesture recognition according to an embodiment of the present invention will be briefly referred to as a 'media content production apparatus 100'.

The media content production apparatus 100 may include a memory 110 , a communication module 120 , a processor 130 for executing a program (or an application), and a display unit 140 . Here, the processor 130 may perform various functions according to the execution of the program stored in the memory 110 , and the processor 130 may include detailed modules according to each function.

The communication module 120 processes data communication with the camera 10 and the projector 20, respectively. The communication module 120 provides a communication interface necessary to provide signals transmitted and received to and from the camera 10 and the projector 20 in the form of packet data by interworking with a communication network. Here, the communication module 120 may be a device including hardware and software necessary for transmitting and receiving signals such as control signals or data signals through wired/wireless connection with other network devices.

The memory 110 stores a dynamic projection mapping-linked gesture recognition interactive media content production method program for providing an interactive media content creation service, and the dynamic projection mapping-linked gesture recognition interactive media stored in the memory 110 The content creation method program may be driven by the processor 130 .

In addition, the memory 110 performs a function of temporarily or permanently storing data processed by the processor 130 . Here, the memory 110 may include a volatile storage medium or a non-volatile storage medium, but the scope of the present invention is not limited thereto.

The memory 110 may store a separate program such as an operating system for processing and controlling the processor 130 , or may perform a function for temporarily storing input or output data.

The memory 110 may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg, SD or XD memory, etc.), RAM , ROM may include at least one type of storage medium.

The processor 130 executes the program stored in the memory 110, and each operation of the program for the dynamic projection mapping interlocking gesture recognition interactive media content production method processed through the processor of the camera 10 and the projector 20 to be described below processing corresponding to

To this end, the processor 130 may be implemented including at least one processing unit (CPU, micro-processor, DSP, etc.), a random access memory (RAM), a read-only memory (ROM), and the like, and the memory 110 . The program stored in the RAM may be read into the RAM and executed through at least one processing unit. Also, according to an embodiment, the term 'processor' may be interpreted as the same meaning as terms such as 'controller', 'arithmetic unit', and 'controller'.

The processor 130 reads a configuration file in which a predetermined gesture and media particle effect are matched in advance, and generates a motion history image based on the gesture image collected from the camera 10 for photographing the user's gesture, and the motion history image is input to the gesture inference model, and based on the gesture type information output from the gesture inference model, the gesture type of the user's current state is determined, and based on spatial mapping information between the camera 10 and the projector 20, the user The projection mapping of the media particle effect corresponding to each gesture type determined according to the gesture change of the user by tracking the location of the user, the gesture inference model can be learned through a convolutional neural network (CNN).

Illustratively, the predetermined gesture includes 'gathering wind', 'shooting long wind-left', 'shooting long wind-right', 'kicking-left', and 'kicking-right', and detailed descriptions of each gesture will be described later with reference to FIG.

The processor 130 may generate a motion history image by sequentially accumulating continuous frame images from before a predetermined frame to the present with respect to the gesture image based on the current frame time.

When the motion history image is generated, the processor 130 extracts the user's silhouette region for each frame image, stores the extracted user silhouette information in a circular queue, and from the circular queue from the past time to the present time The silhouette information can be sequentially read and compressed into a gray scale image. In this case, each frame image may include a silhouette area filled with pixel values matching the time stamp and the remaining background area filled with black pixel values.

When determining the gesture type, the processor 130 may configure a gesture queue with gesture type information output from the gesture inference model, and remove noise for each frame image by using the gesture queue.

Hereinafter, with reference to FIG. 2 , a method for creating interactive media content with dynamic projection mapping-linked gesture recognition according to an embodiment of the present invention will be described.

2 is a detailed structural diagram of a proposed interactive media content production system according to an embodiment of the present invention.

For example, referring to FIG. 2 , the apparatus 100 for producing media content may be implemented with C++ and Python-based Tensorflow.

The processor 130 may include a first module 210 , a second module 220 , a third module 230 , a fourth module 240 , and a fifth module 250 performing respective functions. The first module 210 reads the connection content between the specified gesture and the media content effect from the configuration file (configuration file reader), and based on the camera image (user gesture image data) input in real time, motion history An image (MHI) can be created. Subsequently, it may serve as a main control class (ofApp) that manages a state transition according to the recognized user gesture type and transmits it to the third module 230 .

The second module 220 receives the motion history image from the first module 210 and transmits it to the gesture inference model of TensorFlow, receives the gesture type inferred for the current frame from the gesture inference model, and configures a gesture queue, According to the configuration of the queue, it can serve as a class (recognizer) that finally determines the current gesture type.

The third module 230 receives, from the first module 210, camera-projector spatial mapping information for dynamically projection mapping media effects to the current state and user location according to gesture changes, and renders appropriate particle effects. Note can play the role of a particle system class (Particle System).

The fourth module 230 is a pre-processing step before real-time gesture recognition processing, which reads the motion history image dataset collected for each gesture and performs learning to generate a PB (Protobuf) file for gesture recognition. It is a convolutional neural network model (Gesture Training).

Finally, the fifth module 250 is a TensorFlow module (Gesture Inference) that infers a matching gesture type by receiving a motion history image in real time based on the generated PB file.

The processor 130 may generate a single motion history image MHI by processing a frame image from 30 frames before the current frame time.

Here, for the motion history image, the user's silhouette is extracted from each frame image (user's gesture image data) (body tracking unit), the user silhouette area is filled with pixel values matching the current timestamp, and the background area is black in pixels. It can be generated by accumulating the result of filling the values in the motion history image completed up to the current frame. On the other hand, gesture recognition performed through a convolutional neural network based on the motion history expressed in one sheet does not show a recognition rate of 100%.

Accordingly, the processor 130 configures a gesture queue without directly determining the result value obtained through the TensorFlow inference model as the current gesture type in order to perform stable gesture recognition based on the real-time input image. , it is possible to filter the situation recognized as noise (error situation) for each frame. In this case, the gesture queue can be repeatedly performed whenever a result value is returned from the session.

3 illustrates an algorithm for removing noise and finally determining a gesture type using a gesture queue according to an embodiment of the present invention.

Algorithm 1 shown in FIG. 3 is an algorithm for removing noise using a gesture queue and finally determining a gesture type.

First, in line 1, the gesture type returned through the TensorFlow module is continuously pushed to the gesture queue. If you look at lines 2 and 11, the return value varies depending on the size of the queue. If the size of the gesture queue is less than 4, it returns a value irrelevant to the gesture and does not provide the gesture type and starts when the size of the gesture queue becomes 4. Among them, the process of finding the most returned gesture type is performed. Declare an int array arr with the same number and size of target gesture types, and initialize all values to 0. In lines 4 and 5, the number of each gesture type in the gesture queue is counted. The variable i used in the for statement in lines 6 to 9 is the index value of the target gesture. Line 7 means that the value of arr[i] occupies a majority of the gesture queue. In lines 8 to 9, one element of the gesture queue is popped and the gesture type i, which is more than half of the gesture queue, is returned. In line 9, one element of the gesture queue is popped. Proceeding to line 10 means that the gesture type set as the target did not occupy a majority in the gesture queue. return

Interactive media content linking gestures and media effects may be defined in a configuration file composed of text.

For example, dynamic media effects matching each motion can be created based on openFramework. openFrameworks is an open source C++ toolkit that helps content creation through an intuitive framework. [Table 2] shows the rules for defining each gesture and particle effect matching corresponding to each gesture, such as the total number of gestures and the number of particle effects in the configuration file, and a directory that stores the result values of the determined gestures.

[Table 2] Example of configuration file rules for matching gestures and particle effects

There are a total of four particle effects, and the first particle effect is used for mapping by determining the presence of a user in front of the camera rather than a gesture matching effect.

4 shows a list of motion history images for a 'kick-left' motion according to an embodiment of the present invention.

Referring to FIG. 4 , the present invention uses a motion history image as input data, and the motion history image refers to a static image template that can check the progress path of a movement or gesture as a silhouette image at a glance. In general, since continuous frame images are used for gesture recognition, the size of memory used is larger and the amount of calculation is increased than in the case of object recognition using 2D images. 4 is a sequential list of motion history images of the 'kick' operation in which the learning experiment was conducted in the present invention. To create a motion history image, there are two steps. First, a user's silhouette is extracted for each frame. The user silhouette image is stored in a circular queue that stores user silhouette information of 30 frames. Second, the silhouette images are sequentially read from the past time point to the present time point of the silhouette circular cue, the pixel value of the silhouette area is read, and the pixel value is replaced with the pixel value of the matching pixel in the motion history image space. The motion sequence expressed using the user's silhouette compresses and expresses motion information into a low-capacity gray-scale image.

5 illustrates a 3-layer synthetic convolutional neural network including 2-layer all-connection layers according to an embodiment of the present invention.

6 is a diagram illustrating a 5-layer synthetic convolutional neural network including a 1-layer all-connection layer according to an embodiment of the present invention.

For example, in the case of designing a convolutional neural network for gesture recognition according to an embodiment of the present invention, gesture recognition learning was performed through a gesture learning module implemented with Python-based TensorFlow. The experiment was conducted in a GPU environment of GeForce GTX 1080 Ti with Windows 10 operating system, CPU RAM 64.0GB and NVIDIA 11GByte memory. The Python version is set to 3.7. The convolutional neural network used in the experiment is a model composed of a 3-layer convolutional layer and 2-layer fully-connected layer in FIG. 5, and a 5-layer convolutional layer and a fully-connected layer in FIG. -connected) layers were constructed and tested. In the three-layer synthetic neural network, the number of filters in the convolutional layer is denoted as ‘X’, and learning was carried out in three versions in which 32, 16, and 8 filters were applied in each layer. And a 5x5 filter was applied to all layers.

In the case of a 5-layer convolutional neural network, the stride size is applied to all convolutional layers as 2x2 and the number of filters in each layer is set to 16, 32, 64, 128, and 256 in sequence. 6 shows the structure of a 5-layer convolutional neural network applied to the proposed platform.

The media content production apparatus 100 may include a display unit 140 that receives an image from the camera 10 , generates a motion history image, and displays an interface for rendering a media party effect.

For example, in the media content production apparatus 100 according to an embodiment of the present invention, a module for generating a motion history image by receiving an image from the camera 10 and an overall interface for rendering a media particle effect may be implemented in C++ have. A suitable gesture type recognition module according to one motion history image is implemented in TensorFlow, and the ‘boost Python’ library can be used to connect the C++ module and the TensorFlow module. In addition, the media content production apparatus 100 may be performed by importing the gesture learning result in a pb file from C++ through the 'boost Python' library to a node graph and connecting it. Boost can be performed by setting 1.69ver, Conda 4.6.15ver, Python 3.7.3ver, and openFramework to 0.9.8ver.

Hereinafter, an application experiment of the apparatus 100 for producing media content according to an embodiment of the present invention will be described.

For the application experiment, the target gestures were composed of intuitive gestures that appear frequently in the production of actual projection mapping content. The first is a gesture to collect the stress hidden inside me, the second is a gesture to let the accumulated stress out, and the third is a gesture to kick the leftover emotions. In the case of long wind shooting and kicking gestures, considering that the direction of movement differs depending on the user, image data of a total of five gestures was collected by dividing the left and right sides.

7 illustrates motion history images for each target gesture according to an embodiment of the present invention.

Referring to FIG. 7 , each target gesture is performed, the motion flow of the target gesture to be recognized is expressed as a motion history image, and descriptions of each gesture are shown.

Illustratively, each target gesture is a gesture of raising the arms and slowly drawing a circle while raising the arms, the gesture of 'Shoot long wind - left', extending the arm to the left as if pushing something out of the palm, and 'Shoot long wind - right'. These include the gesture of extending the arm to the right as if pushing something from the palm, the 'kick-left' gesture of extending the foot to the left, and the 'kick-right' gesture of extending the foot to the right.

8 is an interaction particle effect according to each target gesture according to an embodiment of the present invention. As an example, particle effects were created based on openFrameworks.

On the other hand, as important as intuitive gestures that are easy to understand are the interaction effects for each gesture. Appropriate interaction effect further emphasizes the meaning of the gesture itself and makes the message that the performer wants to convey.

Exemplarily, when the user stands in a designated position, the camera 10 recognizes a person, and the projector 20 sprays a particle effect corresponding to the user recognition of FIG. 8 around the user. When you start the ‘Raise Raise’ gesture, the effect in which the light emitted from the surroundings gathers at the user's center is mapped. ‘Filling the long wind’ gives the same effect as if a beam is actually fired from the user’s hands by firing a mass of particles from the center of the performer’s hands in the direction they reach out. ‘Kick’ is also an effect that explodes by firing particles in the form of flames in the direction the toes are facing at the end of the movement. In fact, it gives the intuitive effect of popping something from the toe, helping the audience to understand it quickly.

Illustratively, when explaining the construction of a learning and test dataset according to an embodiment of the present invention, a motion history image-based gesture recognition experiment was performed using a dataset produced by the 5-layered CNN. The collected motion history images were captured at 30 fps and stored as 3 channels of 320x240 pixel size. Image data for each gesture was obtained for a total of 7 participants. A total of 8 types of gesture image data were collected by dividing them into 5 types of target gestures and 3 types of error motions. Table 3 shows the composition of the dataset used in the experiment. The ratio of the training data set and the validation data set is about 7:3.

[Table 3] Composition of training and experimental datasets

Error gestures other than the five kinds of gesture image data are data on variable actions of users that appear in various ways during application experience.

9 is a motion history image of erroneous operations that are learned by classifying them into three types according to an embodiment of the present invention.

The first error gesture was ‘Black’, and black screens with little motion history were collected. This was added to rule out the possibility that the user may react even though the user has not yet appeared. The second error gesture is ‘etc’, which contains motion history images of large movements that are not related to the target gesture. As the classification with the largest range, most meaningless gestures that occur naturally when a user uses the platform proposed in the present invention are included. Finally, the third error gesture is ‘ready’. It is a collection of small actions that are not related to the target gesture, and mainly includes fine gestures that appear before performing the target gesture. Since these gestures are movements that are highly likely to lead to meaningful target motions, they are error data that should be paid attention to when classifying data.

When explaining the experimental environment and experimental results, the PC environment of the gesture recognition learning and proposal platform experiment is carried out by fixing the distance between the experimenter and the camera at 2.25 m and the distance between the camera and the projector at 1.35 m through the CNN designed in Chapter 3 did The height of the camera was fixed at 0.8m to accommodate as many different physical conditions as possible. The user participating in the experiment conducts the experiment in a position where the camera faces the front. For gesture recognition and effect confirmation experiments through the proposed platform, a depth camera, kinect V2, and a high-light beam projector with a resolution of 5000 ANSI were used.

When explaining the experimental results for each CNN structure, [Table 4] shows the results showing the highest performance for each number of filters in an experiment conducted using a 3-layered CNN. When it was 8 filter, high test accuracy was obtained, but in order to reduce the error with the average performance of train accuracy and the speed of runtime, the experiment was performed again with the 5-layered CNN version.

[Table 4] Recognition rate and runtime result of 3-layered CNN

[Table 5] below shows the results of learning experiments by changing conditions (conditions, con) such as stride size and filter size to reduce the runtime speed in the 5-layered CNN structure.

[Table 5] Recognition rate and runtime result of 5-layered CNN according to stride size, filter size, learning rate and batch size

As a result of the experiment, it was confirmed that the runtime was noticeably reduced when the stride size was changed from 1x1 to 2x2 in the 4th and 5th convolution layers (conv). After that, when a part of the filter size was changed with full stride applied and a re-experiment was performed, a slightly reduced runtime and more stable test accuracy and train accuracy than before were obtained.

10 is a graph of the experimental results of [Table 4] - 8 filters.

11 is a graph showing the experimental results of [Table 5] - condition3.

10 and 11 are graphs of the learning experiment results of the 3-layered CNN of [Table 4] - 8 filters and the 5-layered CNN to which the hyper parameter of [Table 5] condition 3 (con.3) is applied, respectively. In the case of 3-layered CNN, there is a large variation in accuracy, so even if the test accuracy is high, there is a high possibility of error when applied to an application. Therefore, in the present invention, the hyper parameter of condition 3 of 5-layered CNN with relatively small variation in accuracy was finally selected and applied.

12 illustrates a screen for applying a media effect according to a gesture according to an embodiment of the present invention.

When explaining the experimental results of applying media effects according to gestures, FIG. 12 shows the results of interactive media contents implemented using the proposed framework, dataset, and CNN structure described above. If the camera succeeds in user recognition immediately after a person enters an empty space, the effect corresponding to 'user recognition' in FIG. 8 is mapped. After that, when raising, shooting, and kicking are performed, the particle effect corresponding to each gesture number is mapped onto the user's body as shown below. As such, in the CNN structure used for gesture learning, dynamic projection mapping content experiments were conducted by applying the result value with the maximum stable performance and fast speed in consideration of the accuracy variation and run time even in the 5-layered CNN structure. As a result, it was confirmed that the appropriate particle effect was mapped in real time according to the user's gesture.

Through the above-described experimental results, the apparatus 100 for producing media content according to an embodiment of the present invention provides a simple and intuitive framework for implementing interactive content based on gesture recognition for creators who feel difficult in production. can Accordingly, even those who are using the media content creation apparatus 100 for the first time can connect a dataset of easy-to-understand gestures and particle effects suitable for each gesture, and create a numbered configuration file so that they can be read and processed quickly in a program.

Hereinafter, a description of a configuration performing the same function among the configurations shown in FIGS. 1 to 12 described above will be omitted.

13 is a flowchart illustrating a method for creating interactive media content for gesture recognition linked to dynamic projection mapping according to an embodiment of the present invention.

Referring to FIG. 13 , in the method for producing interactive media content with dynamic projection mapping interlocking gesture recognition according to the present invention, a gesture image collected from a camera that reads a configuration file in which a predetermined gesture and media particle effect are matched in advance, and captures a user's gesture generating a motion history image based on (S110), inputting the motion history image to a gesture inference model, and determining the gesture type of the user's current state based on the gesture type information output from the gesture inference model ( Based on the spatial mapping information between S120) and the camera 10 and the projector 20, the media particle effect corresponding to each gesture type determined according to the user's gesture change is projected by tracking the user's location (S130) Including, the gesture inference model may be learned through a convolutional neural network (CNN).

In step S110, a motion history image may be generated by sequentially accumulating continuous frame images from before a predetermined frame to the present with respect to the gesture image based on the current frame time.

When generating a motion history image, the user's silhouette region is extracted for each frame image, the extracted user silhouette information is stored in a circular queue, and the silhouette information from the past time to the present time is sequentially retrieved from the circular queue. It is read and compressed into a grayscale image, but each frame image may be composed of a silhouette area filled with pixel values matching the time stamp and the remaining background area filled with black pixel values.

In step S120, a gesture queue may be configured with gesture type information output from the gesture inference model, and noise for each frame image may be removed using the gesture queue.

The above description of the present invention is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

10: camera
20: projector
100: media content creation device
110: memory
120: communication module
130: processor
140: display unit

Claims (9)

A method for producing dynamic projection mapping-linked gesture recognition interactive media content performed by a device, the method comprising:
(a) reading a configuration file in which a predetermined gesture and media particle effect are matched in advance, and generating a motion history image based on a gesture image collected from a camera that captures a user's gesture;
(b) inputting the motion history image to a gesture inference model, and determining a gesture type of the user's current state based on gesture type information output from the gesture inference model; and
(c) based on the spatial mapping information between the camera and the projector, projecting mapping the media particle effect corresponding to each gesture type determined according to the change of the user's gesture by tracking the position of the user;
The gesture inference model is to be learned through a convolutional neural network (CNN),
A method for creating interactive media content with gesture recognition in conjunction with dynamic projection mapping.
The method of claim 1,
The step (a) is
Generating the motion history image by sequentially accumulating continuous frame images from before a predetermined frame to the present with respect to the gesture image based on the current frame time point,
A method for creating interactive media content with gesture recognition in conjunction with dynamic projection mapping.
3. The method of claim 2,
When generating the motion history image,
The user's silhouette area is extracted for each frame image, but the extracted user silhouette information is stored in a circular queue,
From the circular cue, the silhouette information is sequentially read from the past time to the present time and compressed into a gray scale image,
Each frame image is composed of the silhouette area filled with pixel values matching the time stamp and the remaining background area filled with black pixel values,
A method for creating interactive media content with gesture recognition in conjunction with dynamic projection mapping.
4. The method of claim 3,
Step (b) is
constructing a gesture queue with the gesture type information output from the gesture inference model, and removing noise for each frame image by using the gesture queue,
A method for creating interactive media content with gesture recognition in conjunction with dynamic projection mapping.
A dynamic projection mapping interlocking gesture recognition interactive media content production apparatus, comprising:
Dynamic projection mapping interlocking gesture recognition method for producing interactive media content; a memory in which a program is stored;
A processor for executing the program stored in the memory;
The processor reads a configuration file in which a predetermined gesture and media particle effect are matched in advance by executing the program, and generates a motion history image based on a gesture image collected from a camera that captures a user's gesture,
inputting the motion history image to a gesture inference model, and determining a gesture type of the user's current state based on gesture type information output from the gesture inference model;
Based on the spatial mapping information between the camera and the projector, the media particle effect corresponding to each gesture type determined according to the change of the user's gesture is projected by tracking the location of the user,
The gesture inference model is to be learned through a convolutional neural network (CNN),
Dynamic projection mapping interlocking gesture recognition interactive media content creation device.
6. The method of claim 5,
the processor is
Generating the motion history image by sequentially accumulating continuous frame images from before a predetermined frame to the present with respect to the gesture image based on the current frame time point,
Dynamic projection mapping interlocking gesture recognition interactive media content creation device.
7. The method of claim 6,
the processor is
When generating the motion history image,
The user's silhouette area is extracted for each frame image, but the extracted user silhouette information is stored in a circular queue,
From the circular cue, the silhouette information is sequentially read from the past time to the present time and compressed into a gray scale image,
Each frame image is composed of the silhouette area filled with pixel values matching the time stamp and the remaining background area filled with black pixel values,
Dynamic projection mapping interlocking gesture recognition interactive media content creation device.
8. The method of claim 7,
the processor is
When the gesture type is determined, a gesture queue is configured with the gesture type information output from the gesture inference model, and noise for each frame image is removed using the gesture queue,
Dynamic projection mapping interlocking gesture recognition interactive media content creation device.
A non-transitory computer-readable recording medium having recorded thereon a computer program for performing the method of claim 1 , wherein the dynamic projection mapping-linked gesture recognition interactive media content production method is performed.

Images