CN116342763A - Intelligent multi-mode animation creation system and creation method - Google Patents

Abstract

The present application discloses an intelligent multi-modal animation creation system and creation method, wherein the system includes: a situation design module, which is used to generate creation operation instructions according to the creator's creation intention, and build animation story occurrence scenarios according to the creation operation instructions; The building module is used to create narrative content in the animation story occurrence situation according to the key elements of the narrative and the narrative paradigm; the multi-modal driving module is used to collect the multi-modal action interaction information of the creator and map the multi-modal action interaction information to In the state of body movement of the characters and objects in the scene of the animation story, the creation of intelligent multi-modal paintings is completed. Therefore, the use of gamification design methods and multi-modal interaction technology has enhanced the understanding and interest of young people in animation art, and they can quickly make animations without the limitation of time and space without the experience of using professional tools.

Description

Intelligent multi-modal animation creation system and creation method

technical field

This application relates to the technical field of artificial intelligence and multimodal interaction, and in particular to an intelligent multimodal animation creation system and creation method.

Background technique

WebGL (Web Graphics Library) is a 3D drawing protocol. This drawing technology standard allows the combination of JavaScript and OpenGL ES 2.0. By adding a JavaScript binding of OpenGL ES 2.0, WebGL can provide hardware 3D accelerated rendering for HTML5 Canvas , so that web developers can use the system graphics card to more smoothly display 3D scenes and models in the browser, and create complex navigation and data visualization.

Posenet is a real-time pose detection technique that detects human poses in images or videos. It works in both cases as single mode (single human pose detection) and multi-pose detection (multiple human pose detection). As a deep learning TensorFlow model, it estimates the pose of the human body by detecting body parts such as elbows, hips, wrists, knees, and ankles, and forms the skeleton structure of the pose by connecting these points. PoseNet is trained on the MobileNet architecture. MobileNet is a convolutional neural network developed by Google. It is trained on the ImageNet dataset and is mainly used for image classification and target estimation in categories. is a lightweight model that uses depthwise separable convolutions to deepen the network and reduce parameters, computational cost, and improve accuracy. The pre-trained model runs in the browser, which is what differentiates posenet from other API-dependent libraries. Posenet provides a total of 17 keypoints that can be used, from eyes to ears to knees and ankles.

Game engine refers to the core components of some well-written editable computer game systems or some interactive real-time graphics applications. These systems provide game designers with various tools needed to write games. The purpose is to allow game designers to easily and quickly make game programs without starting from scratch. Most of them support multiple operating platforms, such as Linux, Mac OS X, and Microsoft Windows. The game engine includes the following systems: rendering engine ("renderer", including two-dimensional image engine and three-dimensional image engine), physics engine, collision detection system, sound effects, script engine, computer animation, artificial intelligence, network engine and scene management. A game engine is a collection of codes (instructions) designed for a machine that runs a certain type of game and can be recognized by the machine. It is like an engine that controls the operation of the game. A game work can be divided into two parts: game engine and game resources. Game resources include images, sounds, animations and other parts, and a formula is listed: game=engine (program code)+resources (images, sounds, animations, etc.). The game engine calls these resources sequentially according to the requirements of the game design.

At present, computer animation production software has been developed relatively maturely, but these software require users to have professional artistic accomplishment and a lot of software operation experience, and young users often cannot quickly master them. How to reduce the difficulty of animation production and improve the efficiency of animation production is an urgent problem for those skilled in the art.

Contents of the invention

This application provides an intelligent multi-modal animation creation system and creation method, adopts gamification design method and multi-modal interaction technology, and enhances young people's understanding and interest in animation art. In the case of no experience in using professional tools, you can quickly create animations without limitation of time and space.

The embodiment of the first aspect of the present application provides an intelligent multi-modal animation creation system, including: a scenario design module, which is used to generate creation operation instructions according to the creation intention of the creator, and build animation story occurrence scenarios according to the creation operation instructions; The building module is used to create narrative content in the animation story occurrence situation according to the narrative key elements and the narrative paradigm; the multimodal driving module is used to collect the multimodal action interaction information of the creator and convert the multimodal The dynamic action interaction information is mapped to the body movement state of the character object in the animation story scene, and the intelligent multi-modal painting creation is completed.

Optionally, in one embodiment of the present application, the scenario design module includes: a 3D scene rendering unit, configured to construct a 3D rendering space, and present the animation story scenario; a scene object selection unit, configured to perform the 3D rendering The scene objects in the rendering space are classified, and the scene objects corresponding to the creation intention are selected by the user; the scene object operation unit is used to provide a manipulation interaction interface to adjust the spatial position of the scene objects in the three-dimensional rendering space .

Optionally, in one embodiment of the present application, the scene object selection unit is specifically configured to divide the scene objects into background objects, foreground objects, character objects and prop objects, and the background objects are located farthest from the scene. The optional texture at the location; the foreground object is a plurality of graphic textures smaller than the background size; the character object is a movable character image in the narrative content; the prop object is a scene object with its own special function.

Optionally, in one embodiment of the present application, the narrative construction module includes: a comic-style mirroring unit, used to divide the animation segment into multiple mirroring intervals based on the narrative paradigm; a role deployment unit, used to Set character objects and background layouts in each segment; the recording unit is used to record corresponding animations for character objects in each segment.

Optionally, in an embodiment of the present application, the role deployment unit is further configured to divide the existence space of the character object in the scene object into an on-field mode and an off-field mode, and through the on-field mode and the off-field mode Mode switching, adding and deleting the character object in the scene object.

Optionally, in an embodiment of the present application, the multimodal driving module includes: an AI recognition unit, configured to recognize the character object animation recorded by the creator, and obtain the creator's multimodal action interaction Information; a motion redirection unit, configured to match the multimodal action interaction information with the action of the character object, so as to map the action in the multimodal action interaction information to the action specified by the character object Behavior: the animation driving unit is used to determine the relationship between the skeleton and the movement of the character object, so as to drive the movement of the skeleton according to the action behavior of the character object.

Optionally, in one embodiment of the present application, the motion redirection unit is further used to define the bone structure of the character object, and define the connection line between the key point of the character object and its child node and parent node as two bones, and calculate The plane angle of the two bones, and transfer the plane angle value to the local rotation value of the target joint, wherein the bone structure of the character object is a patch body construction mode.

Optionally, in an embodiment of the present application, the animation driving unit is further configured to perform facial expression mapping of characters through image facial recognition and emotion classification recognition.

The embodiment of the second aspect of the present application provides an intelligent multi-modal animation creation method, which includes the following steps: generating creation operation instructions according to the creation intention of the creator, and building an animation story occurrence situation according to the creation operation instructions; according to the narrative key elements and The narrative paradigm creates narrative content in the animation story occurrence situation; collects the multi-modal action interaction information of the creator, and maps the multi-modal action interaction information to the character object body movement state of the animation story occurrence situation , complete the creation of intelligent multi-modal paintings.

Optionally, in an embodiment of the present application, generating the creation operation instruction according to the creation intention of the creator includes: selecting a material with the creation intention from a preset material library according to the creation intention of the creator. For the corresponding scene objects, the creation operation instruction is generated by freely arranging or collaging the scene objects.

The intelligent multi-modal animation creation system and creation method proposed in the embodiment of the present application use gamification design method and multi-modal interaction technology to enhance the understanding and interest of young people in animation art. Target users can quickly produce animation works without the limitation of time and space without professional tool experience.

Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application.

Description of drawings

The above and/or additional aspects and advantages of the present application will become apparent and easy to understand from the following description of the embodiments in conjunction with the accompanying drawings, wherein:

Fig. 1 is an example diagram of an intelligent multimodal animation creation system according to an embodiment of the present application;

FIG. 2 is a schematic diagram of animation resource encoding and classification based on natural interaction according to an embodiment of the present application;

3 is a schematic diagram of the functional relationship between the animation production module and other external modules according to an embodiment of the present application;

Fig. 4 is a flow chart of an intelligent multi-modal animation creation method provided according to an embodiment of the present application.

Detailed ways

Embodiments of the present application are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary, and are intended to explain the present application, and should not be construed as limiting the present application.

FIG. 1 is an example diagram of an intelligent multi-modal animation creation system according to an embodiment of the present application.

As shown in FIG. 1 , the intelligent multimodal animation creation system 10 includes: a scenario design module 100 , a narrative construction module 200 and a multimodal driving module 300 .

The scenario design module 100 is used to generate creation operation instructions according to the creator's creation intention, and build animation story occurrence scenarios according to the creation operation instructions. The narrative construction module 200 is used for creating narrative content in the animation story occurrence situation according to the key elements of the narrative and the narrative paradigm. The multi-modal driving module 300 is used to collect multi-modal action interaction information of the creator, and map the multi-modal action interaction information to the body motion state of the character object in the scene where the animation story occurs, so as to complete the creation of intelligent multi-modal paintings.

The scenario design module can quickly build an animation narrative scenario for creators. The creation system provides a large number of picture materials and collage resources, allowing creators to freely place collages in the scene to achieve the state of mind. The scenario design module is intuitive to operate, and the experiencer can create unique animation scenes without drawing skills or professional software skills.

Narrative building blocks can be used by creators to configure key narrative elements such as characters, items, story chains, and actions, and create narrative content in design situations. The animation creation system adopts a streamlined design method, using the narrative paradigm of four-frame comics and stage plays to guide creators to create storylines.

The multi-modal drive module captures and records creators' multi-modal interaction information based on body expression, which comes from the creator's intuitive movements and can most truly represent their description of expected motion. After the multi-modal drive module obtains the data, it is further cleaned and screened, and mapped to the body movement state of the animated character.

Optionally, in one embodiment of the present application, the scenario design module includes: a 3D scene rendering unit, configured to construct a three-dimensional rendering space, and present an animation story situation; a scene object selection unit, configured to select scene objects in the three-dimensional rendering space Classify and allow the user to select the scene object corresponding to the creation intention; the scene object operation unit is used to provide a manipulation interaction interface to adjust the spatial position of the scene object in the three-dimensional rendering space.

The 3D scene rendering unit is a three-dimensional rendering space constructed based on WebGL technology, and all dynamic graphic content is presented in the three-dimensional rendering space. The scene object selection unit divides the elements in the 3D rendering space into four categories, and the creator selects the scene objects that meet the creative intention from the database and drags them into the scene; the scene object manipulation unit uses a specially designed manipulation interface, Adjust the left, right, up and down, and depth of the scene objects, and manipulate and place the spatial positions of each object in the scene, so as to realize the fine control of the creator on the scene objects.

The 3D scene rendering unit includes an orthographic projection mode, an oblique camera depth display mode and a scene layering mode. Orthogonal projection is the projection mode used in this system. Orthogonal projection avoids the difference between near and far objects in the scene, and the user's cognitive attention is concentrated on the left and right positions of the same level, which simplifies the user's Cognitive load on scene complexity. The left and right direction of the projection space is the X direction of the renderer's world coordinates, the up and down direction of the projection space is the Y direction of the renderer's coordinates, and the front and rear depth directions of the projection space are the Z direction of the renderer's world coordinates. On the basis of the aforementioned orthogonal projection mode, this system adjusts the pitch angle of the renderer camera to -20 degrees, and the vertical angles of all scene objects corresponding to it are also deflected by -20 degrees with the ground as the axis to ensure an orthogonal view Throwing down can show the perspective difference of up and down direction and depth performance at the same time. The background of the layered scene system is composed of two mutually perpendicular pictures, one vertically represents the distant view of the sky, and the other horizontally represents the close view of the ground. At the same time, the background also consists of types without horizontal ground close-ups, such as sky and water scenes. In this scene, characters and props have no gravity effect. The foreground is distributed in various positions in the scene, and the creator can freely specify and edit the animation.

As shown in Figure 2, the scene object selection unit divides the scene objects into four categories: background objects, foreground objects, character objects and prop objects. The background object is an optional texture located at the farthest point of the scene, and there is only one background object in a scene. Foreground objects are a series of graphic textures that are smaller in size than the background. They can be selected arbitrarily, dragged and dropped into the scene multiple times to decorate the scene, and their depth positions can be adjusted to create a sense of depth in the scene. A character object is an animated character image that can be driven by the creator. Character objects can be placed arbitrarily in the scene without restriction; prop objects are scene objects that can be driven by creators and have their own special functions. Prop objects can be placed arbitrarily in the scene without restriction.

Optionally, in one embodiment of the present application, the narrative construction module includes: a comic-style mirroring unit, used to divide the animation segment into multiple mirroring intervals based on the narrative paradigm; Set the character object and background layout in the mirror interval; the recording unit is used to record the corresponding animation for the character object in each mirror interval.

The comic-style storyboarding unit divides the animation clip into four fixed mirroring intervals based on the narrative mode of the four-frame comic. Each mirroring interval is arranged by the creator to play the character object. This part is realized by the character deployment unit. After the creator arranges the background layout and playing characters for each storyboard, the corresponding animation can be recorded for a single storyboard, which is completed by the recording unit.

In order to simplify the narrative design process, the cartoon storyboard unit is based on the four familiar and favorite cartoons of young people as the paradigm of animation storyboard design, and four storyboard units are defined for users. This restrictive design focuses the user's attention on how to design an ingenious and interesting storyline, and can pay less attention to the problem of storyboard montage. The selection and switching of storyboards in this unit is based on the visual design mode of index stickers, which is convenient for selecting different storyboards in the same window, and can view the differences in characters and scenes between different storyboards.

Optionally, in one embodiment of the present application, the role deployment unit is further used to divide the existence space of the role object in the scene object into an on-field mode and an off-field mode, and through switching between the on-field mode and the off-field mode, the Scene objects are added and deleted.

The role deployment unit divides the existence space of characters in the scene into two systems: on-field and off-field. Its purpose is to ① realize the appearance and disappearance of character objects and prop objects; ② to enable creators to better perceive the dynamic environment of the scene; ③ to facilitate the reuse of scene objects and the management of addition and deletion. The character in the waiting area can click the play button to appear in the scene, otherwise it will return to the waiting area. The character's on and off behavior can be recorded by the animation system to realize the appearance and disappearance of the object. When a character in the waiting area is deleted, its stored animation content will also be deleted. Under the new storyboard, all the characters in the original storyboard returned to the waiting area to wait for the creator to dispatch.

The recording unit imitates the UI layout of video games, avoids complex functional areas and nested panels, and only superimposes a flat UI on top of the real-time rendered animation scene. Its core recording start button is located in the lower left corner, and the camera captures the content in the upper right corner. The animation creation system simplifies the animation recording process. For any character in a single scene, the system does not provide the editing function of the recording result, and the creator can directly delete and restart the recording.

Optionally, in one embodiment of the present application, the multimodal driving module includes: an AI recognition unit, used to recognize the character object animation recorded by the creator, and obtain the creator’s multimodal action interaction information; a motion redirection unit , used to match the multi-modal action interaction information with the action of the character object, so as to map the action in the multi-modal action interaction information to the action behavior specified by the character object; the animation drive unit is used to determine the bone of the character object and Relationships between motions to drive bone motion based on the character object's motion behavior.

The AI recognition unit can recognize the creator's body movements to express information. This system regards the creator's body multimodal expression as the main data source for driving animation. The motion redirection unit processes and maps the data acquired by the AI recognition unit to specified actions. The animation drive unit determines the skeleton and motion relationship of the character object, which serves as the technical basis for multi-modal information to drive the virtual character's motion.

The AI recognition unit includes two modules: intelligent algorithm model and data cleaning. The intelligent algorithm model part mainly uses the PoseNet gesture recognition model. PoseNet is a TensorFlow-based machine learning model that can perform real-time human pose estimation (Pose Estimation) in the browser. PoseNet can estimate both a single pose and multiple poses. Pose estimation is performed in two stages: ① input RGB image and parse it through convolutional neural network. ② Use a single-pose or multi-pose decoding algorithm to output decoded poses (Decode Poses), pose confidence scores (Pose ConfidenceScrore), keypoint positions (Keypoint Position) and keypoint confidence scores (Keypoint ConfidenceScores). When estimating the pose of the human body, PoseNet selects 17 key points for the human body, basically covering all the movable joints of the animated character. The key point coordinate data directly output by the PoseNet model is affected by the quality of the camera, which will cause obvious jitter and data loss. The original data is first cleared, and the key points with a confidence score lower than 0.2 are discarded, so that most of the overlapping and occluded point data can be filtered out. On this basis, delay smoothing is performed on the filtered discrete coordinate data (currently using the Lerp smoothing algorithm).

Optionally, in one embodiment of the present application, the motion redirection unit is further used to define the bone structure of the character object, and define the connection line between the key point of the character object and its child node and parent node as two bones, and calculate the two bones The plane angle of the bone, transfer the plane angle value to the local rotation value of the target joint, where the bone structure of the character object is a patch body construction mode.

The motion redirection unit includes object bone structure and data mapping modules. The skeleton structure of the object defines the patch-type body construction mode. The characteristics of this scheme are: ①Compared with the rigid-body skeleton, the 3D skeleton supports the distortion and bending of the picture. ② All body parts are distributed on the same texture without overlapping. The goal is to realize that the same set of bones can be adapted to multiple character images as much as possible. The first type of character image has poor adaptability, and its skinning area is limited to the area where the body part of the character itself is located. Texture replacement can only replace the head and body. The second type of skeleton model is aimed at a specific type of character. The main body of these character images is a specific object, such as garbage bags, steamed buns, fried dough sticks, vegetables, fruits, etc., and the limbs are composed of matchsticks. A stickman skeleton system is designed, and the designer only needs to replace the content of a specific object in the texture to complete the replacement. For the difference in the position of the hands and feet caused by objects of different shapes, the stickman system can set different distances between the hands and feet in the engine character construction without changing the skin of the model. The third bone model will adapt to most bipedal human or animal images in the future, and can support larger images and clothing changes. Rigid body bone mode is the most general character based on rigid body bone mode, which has no specific requirements on the size, length and width of skinned bones and limbs. Therefore, the texture is directly bound to the bone without skinning, and limbs with any aspect ratio can be naturally matched to the character system.

Data mapping processes the smoothed keypoint data and can be utilized in different interactive applications. Key point data is currently used for ① motion mapping of a single point. ② Overall mapping of human joints. The original key point coordinates are the projection of the user's body in the camera screen coordinate system, and the bone length ratio of different characters in the animation system is inconsistent. In order to obtain a normalized body motion posture, so that user actions can be redirected to any character image, the connection line between the key point and its child nodes and parent nodes is defined as two bones, and the plane angle of these two bones is calculated. Pass this angle value to the target joint's local rotation value.

The animation driver unit includes three modules: model skinning, skeletal animation, and animation state machine. The model skinning mod does not contain facial organ components. Elbow and knee skins are continuous, but shoulder and hip joints are disconnected. Under the condition that the bone structure remains unchanged, directly replacing the texture can ensure that the character system can switch skins without errors. However, only a part of the area of each component of the body map can be effectively drawn, and the part beyond the area will not be displayed. Among them, the left and right sides of the hands and feet are symmetrical, and the texture of the same area is used. The overall skeletal structure of the biped skeleton is close to that of the model humanoid model, but the model requires its natural posture to be slightly deflected to the right, so that the character object can switch to the left and right. The tail part extends extra bones from the waist joint, and the ears can also set bone nodes. The bone spring system realizes the elastic effect of ears and tail.

Optionally, in one embodiment of the present application, the animation driving unit is further configured to perform facial expression mapping of characters through image facial recognition and emotion classification recognition.

The bones of character objects in the scene can be moved and transformed under the influence of user interaction. This system designs a set of composite motion capture scheme. For facial organs (eyes, mouth, nose, eyebrows, etc.), two input methods, image-based facial recognition (continuous) and emotion classification recognition (state), are used to complete the mapping of character facial expressions. At the character action level, the three levels of interactive input jointly realize the character movement. The basic layer is the basic displacement and animation (state) such as walking, jumping, and squatting generated by keyboard and mouse operations. This movement method is similar to the character control in the game, and the user can quickly generate familiarity with the animation situation. The second layer is trajectory animation based on image recognition. The algorithm recognizes that the motion trajectory of the finger is mapped to the object, and the motion data of this layer will overload the motion data of the previous layer. The third layer is image-based pose recognition. The algorithm recognizes the movement trajectory of the body skeleton and maps it to the character object. The motion data at this layer is remapped with the user experience as the core. The final motion operation mode of the animation system is: the user operates the animation character to walk freely in the scene by pressing the up, down, left, and right buttons. Actions will be performed physically on the animated character.

The system of the present application enhances young people's understanding and interest in animation art by adopting gamification design method and multi-modal interaction technology. In the case of no experience in using professional tools, you can quickly create animations without limitation of time and space.

With reference to FIG. 3 , the authoring system of the present application will be described in detail through a specific embodiment.

1) Project creation

Before using the system, you should first click the New button to create a new project instance (the background is switched to blue sky), or click the Read button to read the project instance currently stored in the cloud. Starting animation editing without clicking the New or Load button will cause a system error. At the same time, you cannot read the cloud project based on the newly created project. You need to refresh the page and then click Read. At any time during project editing, click the save button to save the current progress to the cloud, and then click read to continue editing the saved content. At present, the multi-user system has not been built yet, and there is only one storage copy in the cloud, and your storage items may be overwritten by other testers.

2) Resource browsing system

The resource browsing system includes four options, namely background, foreground, character and props. Click the background button to switch the background style directly. Click the Foreground button to select the desired foreground pattern. Click the role button to select the desired role. Click the item button to select the item you want.

Once the object is spawned in the scene, clicking on the object activates the movement controls (yellow circles). Among them, the large yellow circle indicates the position of the object on the virtual flat ground, the small green circle directly above can adjust the height of the object, and the small blue circle on the side can adjust the left and right position of the object.

3) Split-mirror window system

The storyboard interface generates four storyboards by default, and each storyboard can be switched by the upper loose-leaf button. Each storyboard can select a set of user-defined background + foreground combination. This feature is not yet complete, all storyboards currently share a set of background + foreground combinations. Click the camera button in the lower right corner to enter the single mirror interface, and click again to switch back to the split mirror interface.

4) Time axis system

The time axis of the multi-shot interface is composed of 4 equal-width areas. The maximum interval of each area is mapped to the system default maximum time (10s) of a single shot. If the animation time of a shot does not exceed 10s, it will be displayed as The green rectangular area is overlaid and displayed. Users can freely drag the timeline in the green area to view the animation of each shot. Click the play button on the right to play/pause the animation.

The time axis of the single-shot interface is composed of a completion timeline. From left to right is the system default maximum time of a single shot (10s). If the animation time of a shot does not exceed 10s, the user can only Drag the timeline within the time range of movie recording. Click the play button on the right to play/pause the animation.

5) Waiting/Entry System

In order to facilitate the control of each character in the scene, a waiting/entering system is designed to select the character object to be used in each storyboard. Only character and prop objects can be managed by the waiting system. In an animation project, multiple character objects of the same type can be created. As shown in the figure above, two Peppa Pig objects are created in the project. As independent actors, they can appear in each scene from the beginning to the end, and the actors who do not appear will appear in a translucent state. However, during the animation, characters who are not in the scene will not appear. In the UI interface of the waiting system, you can see the list of all actors participating in the performance and their waiting conditions.

As shown in the picture above, there are two piglets starring in the animation project, the first of which is in the first storyboard, and the second is in the second storyboard. Actors who play will be highlighted in red in the waiting area list. Click the admission button of their avatar to switch between waiting and admission status. Clicking the delete button on the avatar will remove the actor, and the animation performance data in each storyboard will also be deleted.

6) Animation recording system

Click the camera button in the lower right corner of the window to enter the animation editing interface in the specified storyboard. The functions of the right UI are: pose (pose) animation recording button, motion (move) animation recording button, storage animation and discard animation button.

The pose animation recording process of a character in a certain scene is as follows: ① Select the animation object ② Drag and drop it to the time point you want to record the animation. The starting point of the animation must be determined within the time range of the recorded animation, that is, within the range that the time axis can be dragged. If the start time point is somewhere in the middle of the recorded animation, the new animation clip will overwrite the old animation clip. ③Click gesture animation recording. Button to start recording After clicking this button, the time axis will move with time, and the gesture movement captured by the camera will be recorded in the temporary animation data object (animationdata_new). ④ Click the pose animation recording button again to stop recording. Another click of the gesture animation record button by the user will stop the recording.

According to the intelligent multi-modal animation creation system proposed in the embodiment of the present application, the gamification design method and multi-modal interaction technology are adopted to enhance the understanding and interest of young people in animation art. Target users can quickly produce animation works without the limitation of time and space without professional tool experience.

Next, the intelligent multi-modal animation creation method proposed according to the embodiment of the present application will be described with reference to the accompanying drawings.

Fig. 4 is a flowchart of an intelligent multi-modal animation creation method according to an embodiment of the present application.

As shown in Figure 4, the intelligent multimodal animation creation method includes the following steps:

In step S101, a creation operation instruction is generated according to the creator's creation intention, and an animation story occurrence scenario is built according to the creation operation instruction.

Optionally, in one embodiment of the present application, generating the creation operation instruction according to the creator's creation intention includes: selecting a scene object corresponding to the creation intention in the preset material library according to the creator's creation intention, and Objects are freely arranged and/or collaged to generate creative operation instructions.

Based on the gamified interactive interface, users can quickly design and build the animation story scene imagined in their minds.

Step S102, creating narrative content in the animation story occurrence situation according to the key narrative elements and narrative paradigm.

The narrative key elements include at least one of character objects, items, story chains, and character actions. Narrative paradigms include at least one of four-panel comics and stage plays.

The embodiment of the present application provides a set of interactive metaphors based on four-panel comics and stage performances, helping users construct storylines and narrative chains in situations.

Step S103, collecting the creator's multi-modal action interaction information, and mapping the multi-modal action interaction information to the body movement state of the character object in the scene where the animation story takes place, to complete the intelligent multi-modal painting creation.

The embodiment of the present application converts the user's body expression into character movement through a series of multi-modal interactive interfaces, and realizes the recording of animation narration.

It should be noted that the foregoing explanations for the embodiment of the intelligent multi-modal animation creation system are also applicable to the intelligent multi-modal animation creation method of this embodiment, and will not be repeated here.

According to the intelligent multi-modal animation creation method proposed in the embodiment of the present application, the gamification design method and multi-modal interaction technology are adopted to enhance the understanding and interest of young people in animation art. Target users can quickly produce animation works without the limitation of time and space without professional tool experience.

In the description of this specification, descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific examples", or "some examples" mean that specific features described in connection with the embodiment or example , structure, material or characteristic is included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Moreover, the described specific features, structures, materials or characteristics may be combined in any one or N embodiments or examples in an appropriate manner. In addition, those skilled in the art can combine and combine different embodiments or examples and features of different embodiments or examples described in this specification without conflicting with each other.

In addition, the terms "first" and "second" are used for descriptive purposes only, and cannot be interpreted as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of these features. In the description of the present application, "N" means at least two, such as two, three, etc., unless otherwise specifically defined.

Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing a custom logical function or step of a process , and the scope of preferred embodiments of the present application includes additional implementations in which functions may be performed out of the order shown or discussed, including in substantially simultaneous fashion or in reverse order depending on the functions involved, which shall It should be understood by those skilled in the art to which the embodiments of the present application belong.

Claims (10)

1. An intelligent multimodal animation authoring system comprising:

the situation design module is used for generating an authoring operation instruction according to the authoring intention of an author and building an animation story generating situation according to the authoring operation instruction;

the narrative construction module is used for creating narrative content according to the narrative key elements and the narrative paradigm in the animation story occurrence situation;

and the multi-mode driving module is used for collecting multi-mode action interaction information of the creator and mapping the multi-mode action interaction information into the body motion state of the character object in the animation story occurrence situation to complete the creation of the intelligent multi-mode painting.

2. The system of claim 1, wherein the context design module comprises:

the 3D scene rendering unit is used for constructing a three-dimensional rendering space and presenting the animation story situation;

the scene object selecting unit is used for classifying the scene objects in the three-dimensional rendering space and providing users with the scene objects corresponding to the creation intention;

and the scene object operation unit is used for providing a manipulation interaction interface so as to adjust the spatial position of the scene object in the three-dimensional rendering space.

3. The system according to claim 2, wherein the scene object selection unit is specifically configured to divide the scene objects into a background object, a foreground object, a character object and a prop object, the background object being a selectable texture located farthest from the scene; the foreground object is a plurality of graphic textures smaller than the background size; the character objects are movable character images in the narrative content; the prop object is a scene object with a special function.

4. The system of claim 1, wherein the narrative construction module comprises:

the cartoon type mirror dividing unit is used for dividing the animation segments into a plurality of mirror dividing sections based on the narrative paradigm;

a character deployment unit for setting character objects and background layouts in each of the scope sections;

and the recording unit is used for recording corresponding animations for the character objects in each of the subarea intervals.

5. The system of claim 4, wherein the character deployment unit is further configured to divide a presence space of the character object in a scene object into an upper scene mode and a lower scene mode, and to add and delete the character object in the scene object by switching the upper scene mode and the lower scene mode.

6. The system of claim 1, wherein the multi-modal driver module comprises:

the AI identification unit is used for identifying the character object animation recorded by the creator to obtain multi-mode action interaction information of the creator;

a motion redirection unit, configured to match the multi-modal interaction information with the actions of the character object, so as to map the actions in the multi-modal interaction information to action behaviors specified by the character object;

and the animation driving unit is used for determining the relation between the bones and the motions of the character objects so as to drive the bones to move according to the action behaviors of the character objects.

7. The system of claim 6, wherein the motion redirection unit is further configured to define a character object skeleton structure and define a connection of a character object key point to its child and parent nodes as two skeletons, calculate a plane angle of the two skeletons, and transfer the plane angle value to a local rotation value of a target joint, wherein the character object skeleton structure is a facial body building model.

8. The system of claim 6, wherein the animation drive unit is further configured to map a facial expression of a character through facial recognition and emotion classification recognition of an image.

9. An intelligent multi-modal animation authoring method comprising the steps of:

generating an authoring operation instruction according to the authoring intention of an author, and building an animation story generating situation according to the authoring operation instruction;

creating narrative content in the animated story occurrence context according to the narrative key elements and the narrative paradigm;

and collecting multi-modal action interaction information of the creator, and mapping the multi-modal action interaction information into the body motion state of the character object in the occurrence situation of the animation story, so as to complete the creation of the intelligent multi-modal painting.

10. The method of claim 9, wherein generating the authoring operation instructions from the author's authoring intent comprises:

and selecting a scene object corresponding to the authoring intention from a preset material library according to the authoring intention of the creator, and generating the authoring operation instruction by freely placing or and/or collaging the scene object.

Images