KR20210098220A - Content Transforming Apparatus Responsive to Audience Reaction - Google Patents

Abstract

Disclosed is a technology related to automatic editing of multimedia content. A reaction clip, which is part of the multimedia content originally produced as one-way content, is converted into a customized reaction clip determined according to a viewer reaction. The viewer reaction is detected, and accordingly, a customized reaction is determined. The reaction clip, which is part of the multimedia content, is converted into the customized reaction clip according to the determined customized reaction. The viewer reaction may be detected by analyzing a facial expression, a change in a posture, or a voice of the viewer. The conversion to the customized reaction may include a process of changing a reaction movement or a reaction facial expression without changing an identity of a character.

Description

Content Transforming Apparatus Responsive to Audience Reaction

Disclosed is a technique for automatic editing of multimedia content.

Korean Patent No. 992,509, filed on April 24, 2009 and registered on November 1, 2010, discloses the idea of providing the voice or action/expression of a character in content by modulating it into the voice or action/expression of an acquaintance such as a code or a brother, and there is. However, it does not suggest a feasible method for a specific method of modulating the voice, and does not suggest a specific modulation method in relation to the modulation of motion or expression.

Developing at this level of merely presenting ideas, recent deep learning-based technologies are presenting innovative technologies in the field of motion, voice, and face transformation. GAN (Generative-Adversarial Network) technology makes it possible to change the facial expression and change the posture in a photo delicately. Liqian Ma, et al. “Pose Guided Person Image Generation”, 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, 25 May 2017 The paper uses a form of this GAN network to create images based on pose information. A technique for changing the posture of a person in the body is disclosed. The application of VLSI technology is also expanding in artificial intelligence engines, enabling real-time implementation of these technologies.

On the other hand, the production of user-responsive multimedia content is practically impossible because the number of various cases for each scene must be considered. Recently, artificial intelligence robots with a limited range that appropriately select and reproduce stored motion clips while communicating with humans in natural language have been proposed, but learning by a large amount of data for each application field should be preceded.

It is an object of the proposed invention to present a new technology for changing the operation of an avatar reproduced according to a viewer's reaction when playing multimedia content.

Furthermore, the proposed invention aims to convert multimedia content that is not produced in a user responsive manner into a user responsive one.

According to an aspect of the proposed invention, a reaction clip, which is a part of multimedia content originally produced as one-way content, is converted into a customized reaction clip determined according to a viewer reaction. A viewer reaction is detected, and a customized reaction is determined in response thereto. A reaction clip that is part of the multimedia content is converted into a customized reaction clip according to the determined customized reaction.

According to a further aspect, the viewer's reaction may be detected by a cloud server. Furthermore, the cloud server may determine the viewer reaction by detecting and synthesizing the reaction of each viewer from a plurality of viewer images received through the network.

According to a further aspect, the viewer reaction may be detected by classifying the facial expression of the viewer into one of predetermined categories. Further, according to another aspect, the viewer's reaction may be detected by classifying a motion from a change in the viewer's posture into one of predetermined categories. Further, according to another aspect, the viewer's reaction may be detected by classifying the viewer's voice into one of predetermined categories.

According to a further aspect, the custom reaction may include a reaction action. According to yet another aspect, the customized reaction may include a reaction expression.

According to a further aspect, the viewer reaction may be processed by an artificial intelligence engine that receives a viewer image, determines and outputs one of a range of responses.

According to the proposed invention, it is possible to provide a responsive service that responds to a viewer's reaction with multimedia content that was originally produced one-way. For example, in the case of Internet lectures, the same content is provided to many viewers and does not respond at all to the viewers' reactions, which reduces concentration. It is possible to increase the concentration on the recorded lecture by capturing the degree of concentration of the viewer or the instantaneous reaction and transforming the instructor to take an appropriate action accordingly.

1 illustrates an example of a digital broadcasting system to which a content conversion apparatus according to an embodiment is applied.
2 is a block diagram illustrating a configuration of a content conversion apparatus according to an embodiment.
FIG. 3 is a block diagram illustrating the configuration of an embodiment of the reaction detection unit of FIG. 2 .
4 is a block diagram illustrating a configuration of a content conversion apparatus according to another embodiment.
5 is a block diagram illustrating a configuration of a content conversion apparatus according to another embodiment.
6 is a block diagram illustrating a configuration of a content conversion apparatus according to another embodiment.

The foregoing and additional aspects are embodied through the embodiments described with reference to the accompanying drawings. It should be understood that various combinations are possible with each other, as long as there is no contradiction between the elements or the elements that are different or additionally provided in each embodiment compared to the other embodiments. The claims are intended to cover various embodiments derived by combinations between elements of the disclosed embodiments.

1 shows an example of a multimedia service system to which a content conversion apparatus according to an embodiment of the proposed invention is applied. The illustrated multimedia service system may include content service servers 30 and 31 , a public network 50 and receiving end systems 11 to 17 . The content service servers 30 and 31 may be a broadcast transmission system that provides a streaming service of digital content according to a broadcast schedule. As another example, the content service servers 30 and 31 may be an on-demand content service system. Contents stored in the content database 31 may be reproduced by a reproduction server and provided streaming service to clients by the streaming server. A load balancing system can distribute the load of client connections. The public network 50 may be wired, wireless, or a combination thereof. These may include wired networks of Internet service providers or mobile communication networks of mobile communication operators, and may include wired and wireless communication networks in buildings. In the illustrated system, the receiving end system may include a set-top box 11 , for example, an over-the-top (OTT) box, and a display 13 . As another example, the receiving end system may be a mobile communication terminal such as a tablet 15 or a smart phone 17 .

According to one aspect, the content conversion apparatus according to the proposed invention may be implemented as a part of the receiving end system, for example, the set-top box 11 . As another example, the content conversion device according to the proposed invention is a peripheral device equipped with an application program included in the smart phone 17 or tablet 15, and dedicated hardware that cooperates with this application, for example, an artificial intelligence engine. can be implemented as As another example, the content conversion device according to the proposed invention is an application included in the smart phone 17 or tablet 15, and the content conversion server 50 that cooperates with this application, for example, an artificial intelligence program. It can be implemented as a high-performance graphics server running at high speed. A reaction clip that is a part of one-way content transmitted by streaming from the content service server 30 may be converted into a customized reaction clip determined according to a viewer's reaction according to the content conversion device.

Although the embodiments described below are described as an apparatus, when implemented in an information processing system, hardware is implemented according to the order of information flow, and in some cases, it may be implemented as software, that is, a program executable on a computer. In particular, algorithms with high complexity, such as artificial intelligence algorithms, are often implemented with dedicated hardware focusing on areas where general-purpose processors cannot keep up with the required execution speed. often meaningless. The illustrated embodiments are described from an apparatus point of view but they themselves support the appended method claims.

According to an aspect of the proposed invention, a reaction clip, which is a part of multimedia content originally produced as one-way content, is converted into a customized reaction clip determined according to a viewer reaction. A viewer reaction is detected, and a customized reaction is determined in response thereto. A reaction clip that is part of the multimedia content is converted into a customized reaction clip according to the determined customized reaction.

2 is a block diagram illustrating a configuration of a content conversion apparatus according to an embodiment to which an aspect of the proposed invention is applied. The content conversion apparatus may include a memory 230 and an information processing system 10 that reads control data stored in the memory 230 and processes input data. For example, the information processing system 10 includes a general-purpose microprocessor that controls the entire device, a plurality of artificial intelligence engines implemented with dedicated hardware and each having a structure optimized for the execution of a specific deep learning algorithm, and a variable structure. and may include a processor array with a re-configurable structure that processes speed-insensitive AI algorithms or parts thereof.

The multimedia content 250 , which is an input of the information processing system 10 , may be, for example, content that the terminal receives by streaming through a communication module, reproduces it through a codec, and outputs the content. As another example, it may be content that the terminal reads from the memory 230, reproduces it through a codec, and outputs the content.

According to an embodiment, the content conversion apparatus may include a reaction detecting unit 100 , a custom reaction determining unit 300 , and a reaction clip converting unit 500 . The reaction detection unit 100 detects a viewer reaction to the output multimedia content.

According to one aspect, the viewer's reaction may be detected by a cloud server. In this embodiment, the response detection unit 100 of the information processing system 10 transmits to the cloud server at least some frames of the image received from the camera 210 or partial regions such as the face and torso extracted by preprocessing among the frames. and receive the detected reaction information. Furthermore, the cloud server may determine the viewer reaction by detecting and synthesizing the reaction of each viewer from a plurality of viewer images received through the network. That is, the cloud server 50 of FIG. 1 receives the viewer's video, audio, sensor information, etc. from the plurality of information processing systems 10 and detects the reaction of each viewer. Thereafter, they may be combined to determine, for example, the most reaction among the viewer reactions as the final viewer group reaction.

3 is a block diagram illustrating the configuration of the reaction detection unit 100 according to an exemplary embodiment. According to an aspect of the proposed invention, the reaction detection unit 100 may include an expression-based reaction detection unit 110 . The expression-based reaction detection unit 110 may detect a viewer reaction by classifying the viewer's facial expression into one of predetermined categories. The expression-based reaction detection unit 110 extracts a face image by first detecting a face region from the input image. Thereafter, for example, the expression-based reaction detection unit 110 analyzes the extracted face image to detect the viewer's reaction. According to an aspect, the expression-based reaction detection unit 110 may determine the viewer's reaction by analyzing the expression in the face image. In one embodiment, the expression-based reaction detection unit 110 is, for example, an artificial intelligence engine that determines a reaction in one of a range of categories such as {boredness, expressionlessness, surprise, joy, interest}, for example, deep It can be implemented as a deep-layered convolutional neural network model. In addition to the approach of classifying emotions into one of a number of different categories of emotions, there is also known an approach that continuously quantifies the degree of excitement or positive/negative, for example. For example, the AffectNet dataset is publicly available for training artificial intelligence engines for such emotion recognition.

According to an aspect of the proposed invention, the reaction detection unit 100 may include a motion-based reaction detection unit 130 . The motion-based reaction detector 130 may classify a motion into one of a predetermined category from a change in the viewer's posture to detect a reaction. The motion-based reaction detection unit 130 detects a pose from feature points such as joints constituting a skeleton of a viewer's head, torso, and arms in the video, and sets a change in this posture, ie, a movement, into a predetermined category. You can classify one of them to determine the reaction.

According to an aspect of the proposed invention, the reaction detection unit 100 may include a voice-based reaction detection unit 150 . The voice-based reaction detector 150 may detect a viewer reaction by classifying the viewer's facial expression into one of predetermined categories. Speech-based emotion recognition technology that exists sporadically in multimedia content is a classical method that divides and classifies factors of similar emotions in speech according to a hierarchical classification methodology using multiple classifiers, for example, Z. Xiao, Dellandrea, L. Techniques such as those disclosed in Chen, W. Dou, “Recognition of emotions in speech by a hierarchical approach,” ACII 2009, 2009, pp.401-408. are known. As another example, speech-based emotion recognition technologies using deep learning, for example, Jiwon Lee et al., “Voice emotion recognition using multi-task-based convolutional neural network”, 2017 Summer Conference of the Korean Telecommunications Society, 2017. 6. Such techniques are also known.

The synthesis reaction determining unit 170 finally determines the reaction by synthesizing the output information of each of the reaction detecting units 110 to 150 . For example, the overall reaction determination unit 170 may select an emotion value with the highest probability in consideration of the emotion value that is the output of each of the reaction detection units 110 to 150 and the probability value of the emotion value.

In FIG. 3 , the reaction detection unit 100 is illustrated as including all of the expression-based reaction detection unit 110 , the motion-based reaction detection unit 130 , and the voice-based reaction detection unit 150 , but the proposed invention includes one of them. All three examples are also included. In this case, the overall reaction determining unit 170 may be omitted. The proposed invention is another embodiment, and includes a combination or all three of the expression-based reaction detection unit 110, the motion-based reaction detection unit 130, and the voice-based reaction detection unit 150, and outputs each All examples that collectively determine the response are included.

For example, multimodal deep learning-based emotion recognition technology that judges emotions by simultaneously processing audio and video captured by the viewer, for example, Z. Xiao, Dellandrea, L. Chen, W. Dou, “Recognition of emotions in speech by a hierarchical approach,” ACII 2009, 2009, pp.401-408. can be applied to the proposed invention.

As described above, in the embodiment of FIG. 3 , at least a part of the processing of each of the reaction detection units 110 , 130 , and 150 may be executed in the cloud server.

Returning to FIG. 2 , the customized reaction determining unit 300 determines a customized reaction to the multimedia content according to the viewer's reaction detected by the reaction detecting unit 100 . For example, a custom response could be a few gestures to call attention, a playful expression, a back-and-forth movement, and a few poses. Determination of a custom response can be implemented simply by a mapping function. For example, a custom reaction that is considered empirically appropriate to the viewer reaction may be mapped. Since the customized response that is uniformly mapped to the detected viewer response is monotonous and easy to lose interest, either randomly select a customized response mapped to one viewer response value from a plurality of values, or consider other additional variable values such as time value together can be decided by

According to an additional aspect, the customized reaction determiner 300 may determine a customized reaction corresponding to the reaction clip of the multimedia content according to the detected viewer reaction and the content of the reaction clip. In one embodiment, the content of the reaction clip corresponding to the part to be converted in the multimedia content 250 is, for example, among a set category such as {in high difficulty logic explanation, simple knowledge transfer, interest induction, pause} one can be determined. These can be referenced by index.

In an embodiment to which an additional aspect has been applied, the custom response may be implemented as a mapping function such as f(r,c). Here, r is the viewer reaction index, and c is the content index of the reaction clip to be converted. That is, in this embodiment, the customized reaction may be determined by being mapped according to the viewer reaction and the content of the reaction clip. Such a mapping may be determined according to a planner's intention to determine a content transformation concept or direction.

Referring to FIG. 2 , the reaction clip conversion unit 500 converts the reaction clip of the output multimedia content into a customized reaction clip reflecting the determined customized reaction, but changes the reaction without changing the identity of the character. In one embodiment, the reaction clip conversion unit 500 receives the reaction clip of the multimedia content in units of image frames, and converts it into a customized reaction clip in which the determined customized reaction is reflected. The reactive clip conversion unit 500 generates still images converted to the target posture. Rather than transforming a reactive clip for all frames of a moving picture, a moving picture may be generated by converting only sampled frames and then estimating frames between them through another image processing engine.

In the embodiment shown in FIG. 2 , the response detection unit 100 may be implemented as a convolutional neural network (CNN) having a depth of 6 layers. Such artificial intelligence circuits can be implemented as graphics processing circuits with multiple cores. As another example, it may be implemented as a gate array-based dedicated circuit designed for real-time implementation. In addition, the custom response determining unit 300 may be implemented as a computer program running on a microprocessor that determines a rule-based custom response. In addition, the reactive clip conversion unit 500 may be implemented including a deep convolutional GAN (DCGAN) model. This DCGAN can be designed and implemented with dedicated hardware tailored to the target processing speed. In another example, at least a part of the processing of the reactive clip converting unit 500 may be executed in a cloud server.

Several implementation techniques are known for each functional block according to the design purpose, specification, and process technology selection, and a complex algorithm such as the proposed invention can be implemented in one signal processing processor or general-purpose microprocessor, or Implementing it on a single server computer doesn't seem easy at the moment.

4 is a block diagram illustrating a configuration of a content conversion apparatus according to another embodiment. According to an aspect, a reaction clip that is a part of multimedia content originally produced as one-way content is replaced with a customized reaction clip determined according to a viewer reaction. In the embodiment shown in FIG. 4 , the reactive clip converting unit 500 may include a reactive clip generating unit 510 and a reactive clip replacing unit 530 . The reaction clip generation unit 510 may generate and output a reaction clip included in the multimedia content based on an artificial intelligence algorithm and a customized reaction clip in which a customized reaction is reflected from the determined customized reaction information. The reaction clip replacement unit 530 may replace and output the reaction clip included in the multimedia content with the generated custom reaction clip.

According to a further aspect of the proposed invention, the custom reaction may comprise a reaction action. In the illustrated embodiment, the customized reaction determination unit 300 may include a customized reaction action determination unit 310 . The customized reaction action may be designated by an index as one of actions of a predetermined category, such as {hand gesture 1, hand gesture 2, head movement, getting up}. As another example, the customized response motion information may be a set of coordinate values of each joint point listed in time series order. The order of each joint point of the human body graph model can be defined, and the position of each joint point at a certain time can be expressed as a coordinate value. The motion information of the character can be expressed by arranging ordered pairs representing the coordinates of each joint point at one time in chronological order.

The customized reaction action determining unit 310 maps and determines the customized reaction action according to the detected viewer reaction. Since the reaction action uniformly mapped to the detected viewer reaction is monotonous and easy to lose interest, a customized reaction action mapped to one viewer reaction value is randomly selected from among a plurality of responses, or other additional variable values such as time values are used together. can be decided by considering. According to an additional aspect, the customized reaction action determining unit 310 may reflect the content of the reaction clip to be converted in the multimedia content 250 to the determination of the customized reaction. In one embodiment, the content of the reaction clip corresponding to the part to be converted in the multimedia content 250 is, for example, among a set category such as {in high difficulty logic explanation, simple knowledge transfer, interest induction, pause} one can be determined. These can be referenced by index.

According to an aspect, the reactive clip generating unit 510 may include a reactive clip operation converting unit 511 . The reaction clip action conversion unit 511 converts the reaction clip of the output multimedia content into a customized reaction clip in which the determined customized reaction action is reflected, but changes the action without changing the identity of the character. In one embodiment, generative adversarial networks (GANs) may be used to transform reactive clips. Liqian Ma, et al. “Pose Guided Person Image Generation”, 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, 25 May 2017 The paper discloses a novel pose and a technology for generating a target image from an input image. are doing This device has a two-level structure. Step 1 includes a convolutional autoencoder with a structure similar to that of U-Net, and integrates the input image with a new pose expressed by the coordinates of 18 key points to integrate the target and the basic to produce a somewhat blurry image with the same color and posture. Step 2 involves a variant of conditional deep convolution GAN (DCGAN), which refines the rather faint image input through adversarial training.

In the illustrated embodiment, the reactive clip replacement unit 530 replaces the reactive clip constituting a part of the multimedia content with the customized reactive clip generated by the reactive clip action conversion unit 511 and outputs it.

According to a further aspect of the proposed invention, the customized reaction may comprise a reaction expression. 5 is a block diagram illustrating a configuration of a content conversion apparatus according to another embodiment. In the illustrated embodiment, the customized reaction determination unit 300 may include a customized reaction expression determination unit 330 . The customized reaction expression may be designated by an index as one of the actions of a predetermined category, such as {big smile, smile, frown, comedy 1, comedy 2, angry expression}. As another example, the customized response motion information may be a set of coordinate values of each feature point that is arranged in a time series order. An order can be defined for each feature point of the graph model of the face, and the position of each feature point at a certain point of view can be expressed as a coordinate value. Facial expression information can be expressed by arranging ordered pairs representing the coordinates of each joint point at one time in chronological order.

The customized reaction expression determining unit 330 maps and determines the customized reaction expression according to the detected viewer reaction. Since the reaction expression uniformly mapped to the detected viewer reaction is monotonous and easy to lose interest, a customized reaction expression mapped to one viewer reaction value is randomly selected from a plurality of responses, or other additional variable values such as time values are used together. can be decided by considering. According to an additional aspect, the customized reaction expression determining unit 330 may reflect the content of the reaction clip to be converted in the multimedia content 250 to the determination of the customized reaction. In one embodiment, the content of the reaction clip corresponding to the part to be converted in the multimedia content 250 is, for example, among a set category such as {in high difficulty logic explanation, simple knowledge transfer, interest induction, pause} one can be determined. These can be referenced by index.

According to an aspect, the reactive clip generating unit 510 may include a reactive clip expression converting unit 513 . The reaction clip expression conversion unit 513 converts the reaction clip of the output multimedia content into a customized reaction clip reflecting the determined customized reaction expression, but changes the expression without changing the identity of the character. In one embodiment, generative adversarial networks (GANs) may be used to transform reactive clips. A facial expression translation technology that converts a target expression to a target expression while maintaining the identity of a character based on Generative Adversarial Networks (GANs) is known. Hao Tang et al., “Expression Conditional GAN for Facial Expression-to-Expression Translation”, ICIP 2019 paper, EC-GAN (Expression Conditional GAN) mapping from one image domain to another based on additional expression attributes GAN) technology is disclosed.

In the illustrated embodiment, the reaction clip replacement unit 530 replaces the reaction clip constituting a part of the multimedia content with the customized reaction clip generated by the reaction clip expression conversion unit 513 and outputs it.

6 is a block diagram illustrating a configuration of a content conversion apparatus according to another embodiment. In the illustrated embodiment, the customized reaction determination unit 300 includes a customized reaction expression determination unit 330 and a customized reaction action determination unit 310 . In addition, the reactive clip conversion unit 500 includes a reactive clip expression conversion unit 513 , a reactive clip action conversion unit 511 , and a reactive clip replacement unit 530 .

The customized reaction expression determining unit 330 is similar to the corresponding configuration of the embodiment shown in FIG. 5 , and the customized reaction action determining unit 310 is similar to the corresponding configuration of the embodiment shown in FIG. 4 . The reaction clip expression conversion unit 513 is similar to the corresponding configuration of the embodiment shown in FIG. 5 . The reaction clip motion conversion unit 511 is similar to the corresponding configuration of the embodiment shown in FIG. 4 except that the input is an image generated by the reaction clip expression conversion unit 513 .

Although the present invention has been described above with reference to the accompanying drawings, the present invention is not limited thereto, and it should be construed to encompass various modifications that can be apparent from those skilled in the art. The claims are intended to cover such modifications.

11: set-top box 13: display
15: tablet 17: smartphone
30: content service server 31: content database
50: public network
10: information processing system
100: reaction detection unit
110: expression-based reaction detection unit 130: motion-based reaction detection unit
150: voice-based reaction detection unit
210: camera 230: memory
250: multimedia content 270: microphone
300: custom reaction determining unit
310: customized reaction action determination unit 330: customized reaction expression determination unit
500: reaction clip conversion unit
510: reaction clip generation unit
511: reactive clip action conversion unit 513: reactive clip expression conversion unit
530: reaction clip expression conversion unit

Claims (14)

memory;
at least one information processing system that reads control data stored in the memory and processes input data, the information processing system comprising:
a reaction detection unit for detecting a viewer reaction to the output multimedia content;
a customized response determining unit for determining a customized response to the multimedia content according to the viewer's reaction detected by the reaction detecting unit;
a reaction clip conversion unit for converting the outputted multimedia content reaction clip into a customized reaction clip reflecting the determined customized reaction, changing the reaction without changing the identity of the character;
A multimedia content conversion device comprising a. The method according to claim 1, wherein the reaction detection unit:
an expression-based reaction detection unit for classifying a viewer's facial expression into one of a predetermined category to detect a reaction;
a motion-based reaction detection unit for classifying a motion into one of a predetermined category from a change in the viewer's posture and detecting a reaction;
a voice-based reaction detector for classifying the viewer's voice into one of a predetermined category and detecting a reaction;
Multimedia content conversion device comprising at least one of. The method according to claim 1, wherein the reaction detection unit
A multimedia content conversion apparatus for detecting and synthesizing each viewer's reaction from a plurality of viewer's images received through a network to determine the viewer's reaction. The method according to claim 1,
The customized reaction determining unit determines a customized reaction corresponding to the reaction clip of the multimedia content according to the detected viewer reaction and the content of the reaction clip. The method according to claim 1, Reactive clip conversion unit:
a reaction clip generation unit for generating and outputting a reaction clip included in the multimedia content based on an artificial intelligence algorithm and a customized reaction clip in which a customized reaction is reflected from the determined customized reaction information;
a reactive clip replacement unit for replacing and outputting a reactive clip included in the multimedia content with a generated custom reactive clip;
A multimedia content conversion device comprising a. The method according to claim 1,
The customized reaction determination unit includes a customized reaction action determination unit that determines a customized reaction action of the character according to the detected viewer reaction,
The reaction clip conversion unit converts the reaction clip of the output multimedia content into a customized reaction clip in which the determined customized reaction action is reflected, the multimedia content conversion apparatus including a reaction clip action conversion unit that changes the operation without changing the identity of the character. The method according to claim 1,
The customized reaction determination unit further comprises a customized reaction expression determination unit for determining a customized reaction expression of the character according to the detected viewer reaction,
The reaction clip conversion unit converts the reaction clip of the output multimedia content into a customized reaction clip reflecting the determined customized reaction expression, and a reaction clip expression conversion unit for changing the reaction expression without changing the identity of the character. . memory; In the multimedia content conversion method executed in at least one information processing system that reads the control data stored in the memory and processes the input data,
a reaction detection step of detecting a viewer reaction to the output multimedia content;
a custom reaction determining step of determining a customized reaction to the multimedia content according to the viewer reaction detected in the reaction detecting step;
a reaction clip conversion step of converting the reaction clip of the output multimedia content into a customized reaction clip reflecting the determined customized reaction, changing the reaction without changing the identity of the character;
A multimedia content conversion method comprising a. The method of claim 8 , wherein detecting the reaction comprises:
an expression-based reaction detection step of classifying a viewer's facial expression into one of a predetermined category to detect a reaction;
a motion-based reaction detection step of classifying a motion into one of a predetermined category from a change in a viewer's posture and detecting a reaction;
a voice-based reaction detection step of classifying a viewer's voice into one of a predetermined category to detect a reaction;
Multimedia content conversion method comprising at least one of. The method according to claim 1, wherein the reaction detection step detects and synthesizes each viewer's reaction from a plurality of viewer images received through a network to determine the viewer's reaction. 9. The method of claim 8,
The step of determining the customized reaction includes determining a customized reaction corresponding to the reaction clip of the multimedia content according to the detected viewer reaction and the content of the reaction clip. The method of claim 8 , wherein the step of converting reactive clips comprises:
a reaction clip generation step of generating and outputting a reaction clip included in the multimedia content based on an artificial intelligence algorithm, and a customized reaction clip in which a customized reaction is reflected from the determined customized reaction information;
Responsive clip replacement step of replacing the reaction clip included in the multimedia content with the generated custom reaction clip and outputting;
A multimedia content conversion method comprising a. 9. The method of claim 8,
The determining of a customized reaction includes a determination of a customized reaction action of determining a customized reaction action of the character according to the detected viewer reaction,
The reaction clip conversion step converts a reaction clip of the output multimedia content into a customized reaction clip in which the determined customized reaction action is reflected, multimedia content conversion including a reaction clip action conversion step of changing the action without changing the identity of the character method. 9. The method of claim 8,
The customized reaction determination step further includes a customized reaction expression determination step of determining a customized reaction expression of the character according to the detected viewer reaction,
The reaction clip conversion step converts the reaction clip of the output multimedia content into a customized reaction clip reflecting the determined customized reaction expression, but the reaction clip expression conversion step of changing the reaction expression without changing the identity of the character conversion method.

Images