WO2020151273A1 - 一种情绪刺激虚拟现实场景自动生成系统及方法 - Google Patents

一种情绪刺激虚拟现实场景自动生成系统及方法 Download PDF

Info

Publication number
WO2020151273A1
WO2020151273A1 PCT/CN2019/112884 CN2019112884W WO2020151273A1 WO 2020151273 A1 WO2020151273 A1 WO 2020151273A1 CN 2019112884 W CN2019112884 W CN 2019112884W WO 2020151273 A1 WO2020151273 A1 WO 2020151273A1
Authority
WO
WIPO (PCT)
Prior art keywords
virtual reality
background music
model
module
material library
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2019/112884
Other languages
English (en)
French (fr)
Inventor
徐向民
李振豪
舒琳
晋建秀
胡国华
路程栋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
South China University of Technology SCUT
Original Assignee
South China University of Technology SCUT
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by South China University of Technology SCUT filed Critical South China University of Technology SCUT
Publication of WO2020151273A1 publication Critical patent/WO2020151273A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G06F16/332—Query formulation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—Two-dimensional [2D] image generation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics

Definitions

  • the invention relates to the research field of information technology and cognitive psychology, in particular to a system and method for automatically generating emotional stimulation virtual reality scenes.
  • Emotions are the psychological and physical states produced by a person's multiple feelings, thoughts and behaviors. As an advanced function of the human brain, it guarantees the survival and adaptation of organisms. As emotions affect people's behavior, learning, memory, and decision-making to varying degrees, emotion triggering and emotion evaluation combined with information technology, game design, experimental research, and psychotherapy are increasingly becoming important development directions in the future.
  • Virtual reality technology is a collection of simulation technology and computer graphics man-machine interface technology, multimedia technology, sensor technology, network technology, and other technologies. It mainly includes simulated environment, perception, natural skills, and sensor equipment.
  • the simulation environment is a real-time dynamic three-dimensional realistic image generated by a computer.
  • Perception means that the ideal VR should have the perception that all people have.
  • there are perceptions such as hearing, touch, force, movement, and even smell and taste, which are also called multiple perceptions.
  • Natural skills refer to a person's head turning, eyes, gestures, or other human behaviors.
  • the computer processes the data suitable for the participant's actions, and responds to the user's input in real time, and feeds back to the user's five senses. .
  • Sensing equipment refers to three-dimensional interactive equipment.
  • the virtual reality emotion trigger scene comprehensively uses the two stimulating emotion channels of vision and hearing, and adds the rendering of light effects and the use of lens dynamics. It is an effective emotion trigger tool.
  • the virtual reality emotion trigger scene overcomes the traditional The shortcomings of emotional triggering materials that are not strong enough for immersion are more effective in triggering emotions.
  • a tagged material library has been created.
  • the materials in the material library have three-dimensional emotional quantification tags, which are more conducive to users' choices during the construction process.
  • the main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and provide an emotional stimulation virtual reality scene automatic generation system. Combining machine learning algorithms and natural language processing technology, it provides a new way to build virtual reality emotional stimulation scenes. Compared with the existing technology, the present invention has no technical requirements for users and can generate a large number of different types in a short time. The virtual reality scene with emotional stimulation effect is simple and convenient to operate, saving a lot of human resources and time costs. In addition, the present invention also constructs a tagged material library, and the materials in the material library have three-dimensional emotional quantification tags, which is more conducive to user selection during the construction process.
  • Another object of the present invention is to provide a method for automatically generating emotional stimulation virtual reality scenes.
  • An emotional stimulation virtual reality scene automatic generation system which is characterized by comprising a tag system module, a material library module, a human-computer interaction module, and a virtual reality scene automatic generation module;
  • the label system module is used to label the three-dimensional model; the material library module is used to provide materials for the three-dimensional model and background music; the human-computer interaction model is used for the user to input emotions and settings; the virtual reality scene automatically
  • the generation module is used to create a virtual reality scene in accordance with the emotions and settings input by the human-computer interaction module, combined with the materials provided by the material library module and the tags provided by the label system module.
  • the tag system module includes tree structure tags, emotion tags, and attribute tags;
  • the tree structure tag is used to determine the objective classification of the three-dimensional model and background music to facilitate manual indexing;
  • the emotion tag is used to evaluate the three-dimensional model and the background music and to quantify the three-dimensional model;
  • the attribute tag is used to record the three-dimensional Characteristics of the model;
  • the tree structure label includes four layers, namely: the first layer divides the material into three categories: the overall 3D model, the single 3D model, and the background music; the second layer separates the three categories of the first layer Divided into several major categories; the third level divides each of the second level of several major categories into several categories; the fourth level divides each of the third level of several medium categories into several small categories;
  • the emotion tags include pleasure, arousal, and dominance;
  • the attribute tags include public attributes and unique attributes;
  • the public attributes include 3D model color, 3D model size, background music language, and background music style;
  • the unique attributes include Gender, occupation, expression, and clothing style of the human model in the 3D model;
  • the tree structure tag refers to the classification system of the ImageNet picture library;
  • the emotion tag adds emotion tags to the three-dimensional model from IAPS and CAPS, and adds emotion tags to the background music from CADS, and the emotion tags cover emotions.
  • the eight hexagram limits of high/low pleasure, high/low arousal, and high/low dominance in the VAD space are: HVHAHD, HVHALD, HVLAHD, HVLALD, LVHAHD, LVHALD, LVLAHD, LVLALD; each dimension
  • the high and low discrimination threshold of the above value is 5;
  • the material library module includes a 3D model material library and a background music material library
  • the three-dimensional model material library is used to provide suitable three-dimensional models
  • the background music material library is used to provide suitable background music
  • the material library module requires a tag system module to determine the environment and background of the three-dimensional scene when building the material library, that is, the overall three-dimensional model, the single three-dimensional model, and the background music, and all need to have tree structure tags and emotion tags. , Attribute label;
  • the background music material library When collecting background music material in the background music material library, first establish the background music material corresponding to the tree structure label, and then rank the background music from three dimensions of pleasure, arousal and dominance from 1 to 9, that is, with the help of emotional SAM.
  • Table evaluation experiment allowing users to evaluate the VAD value of the material after listening to the background music. After performing the experiment with at least L users, the three-dimensional average value of the VAD value is obtained, which is close to the expected value, and then it is placed in the background music material library;
  • the setting of the expected value refers to the three-dimensional emotion recognition scale (VAD) and the scale conversion from the interval (-1,1) to (1,9);
  • S1 referring to the classification system of ImageNet image library, establish a complete label system including tree structure labels, emotion labels, and attribute labels, that is, build a label system module;
  • the tree structure tags of the tag system collect and filter or build 3D models and background music by yourself, and classify them in a folder tree; refer to IAPS and CAPS to add emotion tags to the 3D model, and refer to CADS to add emotions to background music Tags; and add necessary attribute tags according to the objective attributes of objects and background music, and correspond to the tag system to build a complete 3D model material library and background music material library, that is, build a material library module;
  • the user inputs emotions and sets parameters through the human-computer interaction module
  • the virtual reality scene automatic generation module based on the emotions and settings input by the human-computer interaction module, combines the materials provided by the material library module and the tags provided by the label system module to create a virtual reality scene that matches the mood, and records panoramic videos.
  • the user selects a single model to place it in an appropriate position in the determined three-dimensional model, and selects appropriate background music to perfect the virtual reality scene;
  • the virtual reality scene includes three creation methods:
  • the first is that the user selects the material through the selection of the material, and the virtual reality scene automatic generation module selects it from the material library and generates the overall 3D model, single 3D model, and background music at a specific location according to the clicked emotion tag and attribute tag, thereby building a Corresponding to a virtual reality scene with emotional stimulation effects, where only one overall 3D model and background music can exist at the same time, and the number of single models is unlimited;
  • the second method is to perform natural language processing based on the text entered by the user in the human-computer interaction, extract the user's needs for the scene emotion and object types, select from the material library and generate background models, single models, and background music at specific locations. So as to build a scene with corresponding emotional stimulation effects, in which only one background model and background music can exist at the same time, and the number of single models is not limited;
  • the third is to use the association rule algorithm to obtain the weight network of the relationship between different emotions and objects from the existing complete emotional scene, and then use the machine learning algorithm to obtain the absolute position of the object model.
  • the relative position relationship of different object models is based on this.
  • select objects with higher weights from the relationship weight network to randomly select several categories, and generate a model based on the obtained position relationship, add corresponding background music, so as to obtain several sets of scenes that meet the emotional requirements ;
  • a panoramic video is recorded along the camera path attached to the selected background model to obtain a virtual reality emotional stimulation scene video.
  • the present invention has the following advantages and beneficial effects:
  • the invention provides a new way to build virtual reality emotional stimulation scenes, can generate a large number of virtual reality scenes with different emotional stimulation effects in a short time, is simple and convenient to operate, and saves a lot of human resources and time costs.
  • the present invention also constructs a tagged material library, and the materials in the material library have three-dimensional emotional quantification tags, which is more conducive to user selection during the construction process.
  • Fig. 1 is a structural block diagram of an emotional stimulation virtual reality scene automatic generation system according to the present invention
  • Figure 2 is a flow chart of SAM scale evaluation in Example 1 of the present invention.
  • Fig. 3 is a method flow chart of the method for automatically generating emotional stimulation virtual reality scenes according to the present invention.
  • An emotional stimulation virtual reality scene automatic generation system includes a tag system module, a material library module, a human-computer interaction module, and a virtual reality scene automatic generation module;
  • the label system module is used to label the three-dimensional model; the material library module is used to provide materials for the three-dimensional model and background music; the human-computer interaction model is used for the user to input emotions and settings; the virtual reality scene automatically
  • the generation module is used to create a virtual reality scene in accordance with the emotions and settings input by the human-computer interaction module, combined with the materials provided by the material library module and the tags provided by the label system module.
  • the tag system module includes tree structure tags, emotion tags, and attribute tags;
  • the tree structure tag is used to determine the objective classification of the 3D model and background music;
  • the emotion tag is used to evaluate the 3D model and the background music and to quantify the 3D model;
  • the attribute tag is used to record the characteristics of the 3D model;
  • the tree structure label includes four layers, namely: the first layer divides the material into three categories: the overall 3D model, the single 3D model, and the background music; the second layer separates the three categories of the first layer Divided into several major categories; the third level divides each of the second level of several major categories into several categories; the fourth level divides each of the third level of several medium categories into several small categories;
  • the emotion tags include pleasure, arousal, and dominance;
  • the attribute tags include public attributes and unique attributes;
  • the public attributes include 3D model color, 3D model size, background music language, and background music style;
  • the unique attributes include Gender, occupation, expression, and clothing style of the human model in the 3D model;
  • the tree structure tag refers to the classification system of the ImageNet picture library;
  • the emotion tag adds emotion tags to the three-dimensional model from IAPS and CAPS, and adds emotion tags to the background music from CADS, and the emotion tags cover emotions.
  • the eight hexagram limits of high/low pleasure, high/low arousal, and high/low dominance in the VAD space are: HVHAHD, HVHALD, HVLAHD, HVLALD, LVHAHD, LVHALD, LVLAHD, LVLALD; each dimension
  • the high and low discrimination threshold of the above value is 5;
  • the material library module includes a 3D model material library and a background music material library
  • the three-dimensional model material library is used to provide suitable three-dimensional models
  • the background music material library is used to provide suitable background music
  • the material library module requires a tag system module to determine the environment and background of the three-dimensional scene when building the material library, that is, the overall three-dimensional model, the single three-dimensional model, and the background music, and all need to have tree structure tags and emotion tags. , Attribute label;
  • 3D model materials in the 3D model material library (1) Collect from a formal channel or make it by yourself, without involving commercial copyright and intellectual property rights; (2) According to the label system, the materials should conform to the corresponding label description and have the same specific tree structure
  • the tag corresponds to at most one material, and there is no limit to emotion tags and attribute tags; when collecting 3D model materials, first establish the 3D model material corresponding to the tree structure tag, and then perform the three dimensions of pleasure, arousal and dominance on the 3D model from 1 to A rating of 9, that is, with the aid of the emotional SAM scale evaluation experiment, as shown in Figure 2, let users evaluate the VAD value of the material after full observation through HMD. After at least 50 users are tested, the three-dimensional variance is less than 3 The 3D model is rated and put into the 3D model material library;
  • the background music material library When collecting background music material in the background music material library, first establish the background music material corresponding to the tree structure label, and then rank the background music from three dimensions of pleasure, arousal and dominance from 1 to 9, that is, with the help of emotional SAM. Table evaluation experiment, allowing users to evaluate the VAD value of the material after listening to the background music. After conducting at least 100 user experiments, the three-dimensional average value of the VAD value is obtained, which is close to the expected value, and then placed in the background music material library; The setting of the expected value refers to the three-dimensional emotion recognition scale (VAD) and performs the scale conversion from the interval (-1,1) to (1,9).
  • VAD three-dimensional emotion recognition scale
  • a method for automatically generating emotional stimulation virtual reality scenes includes the following steps:
  • the first step is to refer to the classification system of the ImageNet image library and establish a complete label system including tree structure labels, emotion labels, and attribute labels, that is, to build a label system module;
  • the second step according to the tree structure tags of the tag system, collect and filter or build 3D models and collect background music by yourself, and sort them through the folder tree; refer to IAPS and CAPS to add emotional tags to the 3D model, and refer to CADS as background music Add emotion tags; and add necessary attribute tags according to the objective attributes of objects and background music, and correspond with the tag system to build a complete 3D model material library and background music material library, that is, build a material library module;
  • the third step the user through the human-computer interaction module, select the material from the drop-down box to combine the scene, enter the text requirements and emotions, select a scene creation method, and set the parameters;
  • the fourth step the virtual reality scene automatic generation module, according to the emotions and settings input by the human-computer interaction module, combines the materials provided by the material library module and the tags provided by the label system module to create a virtual reality scene that matches the mood, and the generated scene is reasonable If the objects do not overlap or interlace in space, they should be on the ground except for specific items, and no unreasonable combination, such as wild animals appearing in the city; after generating a reasonable virtual reality scene, record a panoramic video; the shooting track is determined by The virtual reality scene is directly related, and the captured video has a high definition and frame rate to ensure that the user does not cause dizziness when watching.
  • the user selects a single model to place it in an appropriate position in the determined three-dimensional model, and selects appropriate background music to perfect the virtual reality scene;
  • the virtual reality scene includes three creation methods:
  • the first is that the user selects the material through the selection of the material, and the virtual reality scene automatic generation module selects it from the material library and generates the overall 3D model, single 3D model, and background music at a specific location according to the clicked emotion tag and attribute tag, thereby building a Corresponding to a virtual reality scene with emotional stimulation effects, where only one overall 3D model and background music can exist at the same time, and the number of single models is unlimited;
  • the second method is to perform natural language processing based on the text entered by the user in the human-computer interaction, extract the user's needs for the scene emotion and object types, select from the material library and generate background models, single models, and background music at specific locations. So as to build a scene with corresponding emotional stimulation effects, in which only one background model and background music can exist at the same time, and the number of single models is not limited;
  • the third is to use the association rule algorithm to obtain the weight network of the relationship between different emotions and objects from the existing complete emotional scene, and then use the machine learning algorithm to obtain the absolute position of the object model.
  • the relative position relationship of different object models is based on this.
  • select objects with higher weights from the relationship weight network to randomly select several categories, and generate a model based on the obtained position relationship, add corresponding background music, so as to obtain several sets of scenes that meet the emotional requirements ;
  • a panoramic video is recorded along the camera path attached to the selected background model to obtain a virtual reality emotional stimulation scene video.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Computer Hardware Design (AREA)
  • Computer Graphics (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Processing Or Creating Images (AREA)

Abstract

一种情绪刺激虚拟现实场景自动生成系统，包含标签体系模块、素材库模块、人机交互模块、虚拟现实场景自动生成模块；所述标签体系模块用于对三维模型进行标签；所述素材库模块用于对三维模型和背景音乐提供素材；所述人机交互模型用于用户输入情绪和设定；所述虚拟现实场景自动生成模块用于根据人机交互模块输入的情绪和设定，结合素材库模块提供的素材及标签体系模块提供的标签，创符合情绪的建虚拟现实场景；为搭建虚拟现实情绪刺激场景提供了一种新的途径，能够在短时间内生成大量具有不同情绪刺激效果的虚拟现实场景，操作简单便捷，节约人力资源和时间成本；还构建了标签化的素材库，在搭建过程更有利于用户的选择。

Description

一种情绪刺激虚拟现实场景自动生成系统及方法 技术领域
本发明涉及信息技术和认知心理学交叉的研究领域，特别涉及一种情绪刺激虚拟现实场景自动生成系统及方法。
 背景技术
情绪是人的多种感觉、思想和行为综合产生的心理和生理状态。它作为人脑的高级功能，保证着有机体的生存和适应。由于情绪在不同程度上影响着人的行为、学习、记忆与决策，与信息技术相结合的情绪触发及情感评估游戏设计、实验研究、心理治疗等日渐成为未来的重要发展方向。
虚拟现实技术是仿真技术与计算机图形学人机接口技术多媒体技术传感技术网络技术等多种技术的集合，主要包括模拟环境、感知、自然技能和传感设备等方面。模拟环境是由计算机生成的、实时动态的三维立体逼真图像。感知是指理想的VR应该具有一切人所具有的感知。除计算机图形技术所生成的视觉感知外，还有听觉、触觉、力觉、运动等感知，甚至还包括嗅觉和味觉等，也称为多感知。自然技能是指人的头部转动，眼睛、手势、或其他人体行为动作，由计算机来处理与参与者的动作相适应的数据，并对用户的输入作出实时响应，并分别反馈到用户的五官。传感设备是指三维交互设备。虚拟现实情绪触发场景综合运用了视觉和听觉这两种刺激情绪通道，并加入了光效的渲染和镜头动态的运用，是一种有效的情绪触发工具，此外虚拟现实情绪触发场景还克服了传统情绪触发材料的沉浸感不强的缺点，更能有效地进行情绪触发。
目前已出现了利用虚拟现实场景来刺激人类情绪的研究，但发展缓慢，主要的一个原因是传统的人工搭建虚拟现实情绪刺激场景的方法过于复杂，具备虚拟现实场景制作专业技术的人才少，且制作时间长，因此采用传统的方式制作情绪刺激虚拟现实场景需要大量的资金成本和时间成本。
 技术问题
需要解决的技术问题：
克服了传统情绪触发材料的沉浸感不强的缺点，更能有效地进行情绪触发；
传统的人工搭建虚拟现实情绪刺激场景的方法过于复杂，具备虚拟现实场景制 作专业技术的人才少，且制作时间长，节约大量的人力资源、资金成本和时间 成本；
传统的人工搭建虚拟现实情绪刺激场景，无法在短时间内生成，而本申请的技术方案对用户无技术要求，操作简单便捷，能够在短时间内生成大量具有不同情绪刺激效果的虚拟现实场景；
创建了标签化素材库，素材库中的素材均具有三维情感量化标签，在搭建过程更有利于用户的选择。
 技术解决方案
本发明的主要目的在于克服现有技术的缺点与不足，提供一种情绪刺激虚拟现实场景自动生成系统。结合了机器学习算法和自然语言处理技术，为搭建虚拟现实情绪刺激场景提供了一种新的途径，与现有技术相比，本发明对用户无技术要求，能够在短时间内生成大量具有不同情绪刺激效果的虚拟现实场景，操作简单便捷，节约了大量的人力资源和时间成本。此外，本发明还构建了一个标签化的素材库，该素材库中的素材均具有三维情感量化标签，在搭建过程更有利于用户的选择。
本发明的另一目的在于提供一种情绪刺激虚拟现实场景自动生成方法。
本发明的主要目的通过以下的技术方案实现：
一种情绪刺激虚拟现实场景自动生成系统，其特征在于，包含标签体系模块、素材库模块、人机交互模块、虚拟现实场景自动生成模块；
所述标签体系模块用于对三维模型进行标签；所述素材库模块用于对三维模型和背景音乐提供素材；所述人机交互模型用于用户输入情绪和设定；所述虚拟现实场景自动生成模块用于根据人机交互模块输入的情绪和设定，结合素材库模块提供的素材及标签体系模块提供的标签，创符合情绪的建虚拟现实场景。
进一步地，所述标签体系模块包括树形结构标签、情绪标签、属性标签；
所述树形结构标签，用于确定三维模型及背景音乐的客观分类，便于人工索引；所述情绪标签用于评估三维模型及背景音乐并对三维模型进行量化；所述属性标签用于记录三维模型的特征；
进一步地，所述树形结构标签包含四层，分别为：第一层将素材分为整体三维模型、单体三维模型、背景音乐三个类；第二层将第一层的三个类分别分为若干大类；第三层将第二层的若干大类的每个大类分为若干种类；第四层将第三层的若干中类的每个中类分为若干小类；所述情绪标签包含愉悦度、唤醒度、支配度；所述属性标签包含公共属性、特有属性；所述公共属性包含三维模型颜色、三维模型大小、背景音乐语言、背景音乐风格；所述特有属性包含三维模型中人模型性别、职业、表情、衣服风格；
进一步地，所述树形结构标签参考ImageNet图片库的分类系统；所述情绪标签从IAPS和CAPS中为三维模型添加情绪标签，从CADS中为背景音乐添加情绪标签，所述情绪标签涵盖了情绪VAD空间中高/低愉悦度、高/低唤醒度、高/低支配度的八个卦限，八个卦限分别为：HVHAHD、HVHALD、HVLAHD、HVLALD、LVHAHD、LVHALD、LVLAHD、LVLALD；各维度上数值的高低区分阈值为5；
进一步地，所述素材库模块包含三维模型素材库、背景音乐素材库；
所述三维模型素材库用于提供合适的三维模型；所述背景音乐素材库用于提供合适的背景音乐；
进一步地，所述素材库模块，在搭建素材库时需要标签体系模块确定三维场景的环境及背景，即整体三维模型、单体三维模型、背景音乐，并且都需要具有树形结构标签、情绪标签、属性标签；
在三维模型素材库收集三维模型素材时，先建立树形结构标签对应的三维模型素材，然后对三维模型进行愉悦度、唤醒度、支配度三个维度1到9的评级，即借助情绪SAM量表评估实验，让使用者通过HMD充分观察后对素材进行VAD值评价，进行至少K名使用者实验后，得到三个维度方差小于3的三维模型为通过评级，放入三维模型素材库；
在背景音乐素材库收集背景音乐素材时，先建立树形结构标签对应的背景音乐素材，然后对背景音乐进行愉悦度、唤醒度、支配度三个维度1到9的评级，即借助情绪SAM量表评估实验，让使用者收听背景音乐后对素材进行VAD值评价，进行至少L名使用者实验后，得到VAD值三个维度平均值，与预期值接近，则放入背景音乐素材库；所述预期值的设定参照三维情绪识别量表(VAD)并做从区间（-1,1）到（1,9）尺度转换；
进一步地，所述K＞50；所述L＞100。
一种情绪刺激虚拟现实场景自动生成方法，其特征在于，包括以下步骤：
S1、参考ImageNet图片库的分类系统，建立包含树形结构标签、情绪标签、属性标签的一套完整标签体系，即搭建标签体系模块；
S2、根据标签体系的树形结构标签，收集并筛选或自行搭建三维模型以及收集背景音乐，并通过文件夹树状分类；参考IAPS、CAPS为三维模型添加情绪标签，参考CADS为背景音乐添加情绪标签；并根据物体和背景音乐的客观属性添加必要的属性标签，与标签体系进行对应关系，搭建完善的三维模型素材库和背景音乐素材库，即搭建素材库模块；
S3、用户通过人机交互模块，输入情绪，并设定参数；
S4、虚拟现实场景自动生成模块根据人机交互模块输入的情绪和设定，结合素材库模块提供的素材及标签体系模块提供的标签，创建符合情绪的虚拟现实场景，并录制全景视频。
进一步地，还包括，用户选择单体模型在确定的三维模型中进行恰当位置摆放，并选择合适的背景音乐，完善虚拟现实场景；
进一步地，所述虚拟现实场景包含三种创建方法：
第一种、用户通过筛选素材，虚拟现实场景自动生成模块根据点击的情绪标签和属性标签，从素材库中选择并在特定位置生成整体三维模型、单体三维模型、背景音乐、从而搭建一个具有对应情绪刺激效果的虚拟现实场景，其中，整体三维模型和背景音乐都只能同时存在一个，单体模型数量不限；
第二种、根据用户在人机交互输入的文字，进行自然语言处理，提取用户对场景情绪和物体种类的需求，从素材库中选择并在特定位置生成背景模型、单体模型、背景音乐，从而搭建一个具有对应情绪刺激效果的场景，其中背景模型与背景音乐都只能同时存在一个，单体模型数量不限；
第三种、运用关联规则算法，从已有的完整情绪场景中得到不同情绪与物体的关系权重网，再运用机器学习算法得出物体模型绝对位置不同物体模型自己的相对位置关系，在这基础上，根据用户输入的单一情绪要求，从关系权重网中选择权重较高的物体随机抽取若干类，并根据得出的位置关系生成模型，添加对应背景音乐，从而得到若干组符合情绪要求的场景；
最后，沿选取的背景模型所附带的摄像机路径录制全景视频，得到虚拟现实情感刺激场景视频。
 有益效果
本发明与现有技术相比，具有如下优点和有益效果：
本发明为搭建虚拟现实情绪刺激场景提供了一种新的途径，能够在短时间内生成大量具有不同情绪刺激效果的虚拟现实场景，操作简单便捷，节约了大量的人力资源和时间成本。此外，本发明还构建了一个标签化的素材库，该素材库中的素材均具有三维情感量化标签，在搭建过程更有利于用户的选择。
 附图说明
图1是本发明所述一种情绪刺激虚拟现实场景自动生成系统的结构框图；
图2是本发明实施例1中SAM量表评估流程图；
图3是本发明所述一种情绪刺激虚拟现实场景自动生成方法的方法流程图。
 本发明的实施方式
下面结合实施例及附图对本发明作进一步详细的描述，但本发明的实施方式不限于此。
实施例1：
一种情绪刺激虚拟现实场景自动生成系统，如图1所示，包含标签体系模块、素材库模块、人机交互模块、虚拟现实场景自动生成模块；
所述标签体系模块用于对三维模型进行标签；所述素材库模块用于对三维模型和背景音乐提供素材；所述人机交互模型用于用户输入情绪和设定；所述虚拟现实场景自动生成模块用于根据人机交互模块输入的情绪和设定，结合素材库模块提供的素材及标签体系模块提供的标签，创符合情绪的建虚拟现实场景。
进一步地，所述标签体系模块包括树形结构标签、情绪标签、属性标签；
所述树形结构标签，用于确定三维模型及背景音乐的客观分类；所述情绪标签用于评估三维模型及背景音乐并对三维模型进行量化；所述属性标签用于记录三维模型的特征；
进一步地，所述树形结构标签包含四层，分别为：第一层将素材分为整体三维模型、单体三维模型、背景音乐三个类；第二层将第一层的三个类分别分为若干大类；第三层将第二层的若干大类的每个大类分为若干种类；第四层将第三层的若干中类的每个中类分为若干小类；所述情绪标签包含愉悦度、唤醒度、支配度；所述属性标签包含公共属性、特有属性；所述公共属性包含三维模型颜色、三维模型大小、背景音乐语言、背景音乐风格；所述特有属性包含三维模型中人模型性别、职业、表情、衣服风格；
进一步地，所述树形结构标签参考ImageNet图片库的分类系统；所述情绪标签从IAPS和CAPS中为三维模型添加情绪标签，从CADS中为背景音乐添加情绪标签，所述情绪标签涵盖了情绪VAD空间中高/低愉悦度、高/低唤醒度、高/低支配度的八个卦限，八个卦限分别为：HVHAHD、HVHALD、HVLAHD、HVLALD、LVHAHD、LVHALD、LVLAHD、LVLALD；各维度上数值的高低区分阈值为5；
进一步地，所述素材库模块包含三维模型素材库、背景音乐素材库；
所述三维模型素材库用于提供合适的三维模型；所述背景音乐素材库用于提供合适的背景音乐；
进一步地，所述素材库模块，在搭建素材库时需要标签体系模块确定三维场景的环境及背景，即整体三维模型、单体三维模型、背景音乐，并且都需要具有树形结构标签、情绪标签、属性标签；
在三维模型素材库收集三维模型素材，（1）从正规途径收集或自行制作，不涉及商业版权和知识产权；（2）根据标签体系，素材应符合所对应标签描述，且同一特定树形结构标签对应至多一个素材，情绪标签和属性标签无限制；收集三维模型素材时，先建立树形结构标签对应的三维模型素材，然后对三维模型进行愉悦度、唤醒度、支配度三个维度1到9的评级，即借助情绪SAM量表评估实验，如图2所示，让使用者通过HMD充分观察后对素材进行VAD值评价，进行至少50名使用者实验后，得到三个维度方差小于3的三维模型为通过评级，放入三维模型素材库；
在背景音乐素材库收集背景音乐素材时，先建立树形结构标签对应的背景音乐素材，然后对背景音乐进行愉悦度、唤醒度、支配度三个维度1到9的评级，即借助情绪SAM量表评估实验，让使用者收听背景音乐后对素材进行VAD值评价，进行至少100名使用者实验后，得到VAD值三个维度平均值，与预期值接近，则放入背景音乐素材库；所述预期值的设定参照三维情绪识别量表(VAD)并做从区间（-1,1）到（1,9）尺度转换。
实施例2：
一种情绪刺激虚拟现实场景自动生成方法，如图3所示，包括以下步骤：
第一步、参考ImageNet图片库的分类系统，建立包含树形结构标签、情绪标签、属性标签的一套完整标签体系，即搭建标签体系模块；
第二步、根据标签体系的树形结构标签，收集并筛选或自行搭建三维模型以及收集背景音乐，并通过文件夹树状分类；参考IAPS、CAPS为三维模型添加情绪标签，参考CADS为背景音乐添加情绪标签；并根据物体和背景音乐的客观属性添加必要的属性标签，与标签体系进行对应关系，搭建完善的三维模型素材库和背景音乐素材库，即搭建素材库模块；
第三步、用户通过人机交互模块，从下拉框选择素材自行组合场景，输入文字要求和情绪，选择一种场景创建方法，并设定参数；
第四步、虚拟现实场景自动生成模块根据人机交互模块输入的情绪和设定，结合素材库模块提供的素材及标签体系模块提供的标签，创建符合情绪的虚拟现实场景，生成的场景具有合理性，如物体不在空间上重叠或交错，除特定物品外均应在地上，不生成不合理的组合，如野生动物出现在城市里；生成合理的虚拟现实场景后，录制全景视频；拍摄轨迹由虚拟现实场景直接相关，且拍摄的视频具有较高的清晰度和帧率，以保证用户在观看时不造成眩晕。
进一步地，还包括，用户选择单体模型在确定的三维模型中进行恰当位置摆放，并选择合适的背景音乐，完善虚拟现实场景；
进一步地，所述虚拟现实场景包含三种创建方法：
第一种、用户通过筛选素材，虚拟现实场景自动生成模块根据点击的情绪标签和属性标签，从素材库中选择并在特定位置生成整体三维模型、单体三维模型、背景音乐、从而搭建一个具有对应情绪刺激效果的虚拟现实场景，其中，整体三维模型和背景音乐都只能同时存在一个，单体模型数量不限；
第二种、根据用户在人机交互输入的文字，进行自然语言处理，提取用户对场景情绪和物体种类的需求，从素材库中选择并在特定位置生成背景模型、单体模型、背景音乐，从而搭建一个具有对应情绪刺激效果的场景，其中背景模型与背景音乐都只能同时存在一个，单体模型数量不限；
第三种、运用关联规则算法，从已有的完整情绪场景中得到不同情绪与物体的关系权重网，再运用机器学习算法得出物体模型绝对位置不同物体模型自己的相对位置关系，在这基础上，根据用户输入的单一情绪要求，从关系权重网中选择权重较高的物体随机抽取若干类，并根据得出的位置关系生成模型，添加对应背景音乐，从而得到若干组符合情绪要求的场景；
最后，沿选取的背景模型所附带的摄像机路径录制全景视频，得到虚拟现实情感刺激场景视频。
上述实施例为本发明较佳的实施方式，但本发明的实施方式并不受上述实施例的限制，其他的任何未背离本发明的精神实质与原理下所作的改变、修饰、替代、组合、简化，均应为等效的置换方式，都包含在本发明的保护范围之内。

Claims (10)

  1. 一种情绪刺激虚拟现实场景自动生成系统，其特征在于，包含标签体系模块、素材库模块、人机交互模块、虚拟现实场景自动生成模块；
    所述标签体系模块用于对三维模型进行标签；所述素材库模块用于对三维模型和背景音乐提供素材；所述人机交互模型用于用户输入情绪和设定；所述虚拟现实场景自动生成模块用于根据人机交互模块输入的情绪和设定，结合素材库模块提供的素材及标签体系模块提供的标签，创符合情绪的建虚拟现实场景。
  2. 根据权利要求1所述的一种情绪刺激虚拟现实场景自动生成系统，其特征在于，所述标签体系模块包括树形结构标签、情绪标签、属性标签；
    所述树形结构标签，用于确定三维模型及背景音乐的客观分类；所述情绪标签用于评估三维模型及背景音乐并对三维模型进行量化；所述属性标签用于记录三维模型的特征。
  3. 根据权利要求2所述的一种情绪刺激虚拟现实场景自动生成系统，其特征在于，所述树形结构标签包含四层，分别为：第一层将素材分为整体三维模型、单体三维模型、背景音乐三个类；第二层将第一层的三个类分别分为若干大类；第三层将第二层的若干大类的每个大类分为若干种类；第四层将第三层的若干中类的每个中类分为若干小类；所述情绪标签包含愉悦度、唤醒度、支配度；所述属性标签包含公共属性、特有属性；所述公共属性包含三维模型颜色、三维模型大小、背景音乐语言、背景音乐风格；所述特有属性包含三维模型中人模型性别、职业、表情、衣服风格。
  4. 根据权利要求3所述的一种情绪刺激虚拟现实场景自动生成系统，其特征在于，所述树形结构标签参考ImageNet图片库的分类系统；所述情绪标签从IAPS和CAPS中为三维模型添加情绪标签，从CADS中为背景音乐添加情绪标签，所述情绪标签涵盖了情绪VAD空间中高/低愉悦度、高/低唤醒度、高/低支配度的八个卦限，八个卦限分别为：HVHAHD、HVHALD、HVLAHD、HVLALD、LVHAHD、LVHALD、LVLAHD、LVLALD；各维度上数值的高低区分阈值为5。
  5. 根据权利要求1所述的一种情绪刺激虚拟现实场景自动生成系统，其特征在于，所述素材库模块包含三维模型素材库、背景音乐素材库；
    所述三维模型素材库用于提供合适的三维模型；所述背景音乐素材库用于提供合适的背景音乐。
  6. 根据权利要求5所述的一种情绪刺激虚拟现实场景自动生成系统，其特征在于，所述素材库模块，在搭建素材库时需要标签体系模块确定三维场景的环境及背景，即整体三维模型、单体三维模型、背景音乐，并且都需要具有树形结构标签、情绪标签、属性标签；
    在三维模型素材库收集三维模型素材时，先建立树形结构标签对应的三维模型素材，然后对三维模型进行愉悦度、唤醒度、支配度三个维度1到9的评级，即借助情绪SAM量表评估实验，让使用者通过HMD充分观察后对素材进行VAD值评价，进行至少K名使用者实验后，得到三个维度方差小于3的三维模型为通过评级，放入三维模型素材库；
    在背景音乐素材库收集背景音乐素材时，先建立树形结构标签对应的背景音乐素材，然后对背景音乐进行愉悦度、唤醒度、支配度三个维度1到9的评级，即借助情绪SAM量表评估实验，让使用者收听背景音乐后对素材进行VAD值评价，进行至少L名使用者实验后，得到VAD值三个维度平均值，与预期值接近，则放入背景音乐素材库；所述预期值的设定参照三维情绪识别量表(VAD)并做从区间（-1,1）到（1,9）尺度转换。
  7. 根据权利要求6所述的一种情绪刺激虚拟现实场景自动生成系统，其特征在于，所述K＞50；所述L＞100。
  8. 一种情绪刺激虚拟现实场景自动生成方法，其特征在于，包括以下步骤：
    S1、参考ImageNet图片库的分类系统，建立包含树形结构标签、情绪标签、属性标签的一套完整标签体系，即搭建标签体系模块；
    S2、根据标签体系的树形结构标签，收集并筛选或自行搭建三维模型以及收集背景音乐，并通过文件夹树状分类；参考IAPS、CAPS为三维模型添加情绪标签，参考CADS为背景音乐添加情绪标签；并根据物体和背景音乐的客观属性添加必要的属性标签，与标签体系进行对应关系，搭建完善的三维模型素材库和背景音乐素材库，即搭建素材库模块；
    S3、用户通过人机交互模块，输入情绪，并设定参数；
    S4、虚拟现实场景自动生成模块根据人机交互模块输入的情绪和设定，结合素材库模块提供的素材及标签体系模块提供的标签，创建符合情绪的虚拟现实场景，并录制全景视频。
  9. 根据权利要求8所述的一种情绪刺激虚拟现实场景自动生成方法，其特征在于，还包括，用户选择单体模型在确定的三维模型中进行恰当位置摆放，并选择合适的背景音乐，完善虚拟现实场景。
  10. 根据权利要求8所述的一种情绪刺激虚拟现实场景自动生成方法，其特征在于，所述虚拟现实场景包含三种创建方法：
    第一种、用户通过筛选素材，虚拟现实场景自动生成模块根据点击的情绪标签和属性标签，从素材库中选择并在特定位置生成整体三维模型、单体三维模型、背景音乐、从而搭建一个具有对应情绪刺激效果的虚拟现实场景，其中，整体三维模型和背景音乐都只能同时存在一个，单体模型数量不限；
    第二种、根据用户在人机交互输入的文字，进行自然语言处理，提取用户对场景情绪和物体种类的需求，从素材库中选择并在特定位置生成背景模型、单体模型、背景音乐，从而搭建一个具有对应情绪刺激效果的场景，其中背景模型与背景音乐都只能同时存在一个，单体模型数量不限；
    第三种、运用关联规则算法，从已有的完整情绪场景中得到不同情绪与物体的关系权重网，再运用机器学习算法得出物体模型绝对位置不同物体模型自己的相对位置关系，在这基础上，根据用户输入的单一情绪要求，从关系权重网中选择权重较高的物体随机抽取若干类，并根据得出的位置关系生成模型，添加对应背景音乐，从而得到若干组符合情绪要求的场景；
    最后，沿选取的背景模型所附带的摄像机路径录制全景视频，得到虚拟现实情感刺激场景视频。
PCT/CN2019/112884 2019-01-22 2019-10-23 一种情绪刺激虚拟现实场景自动生成系统及方法 Ceased WO2020151273A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910055832.6A CN109887095A (zh) 2019-01-22 2019-01-22 一种情绪刺激虚拟现实场景自动生成系统及方法
CN201910055832.6 2019-01-22

Publications (1)

Publication Number Publication Date
WO2020151273A1 true WO2020151273A1 (zh) 2020-07-30

Family

ID=66926446

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/112884 Ceased WO2020151273A1 (zh) 2019-01-22 2019-10-23 一种情绪刺激虚拟现实场景自动生成系统及方法

Country Status (2)

Country Link
CN (1) CN109887095A (zh)
WO (1) WO2020151273A1 (zh)

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109887095A (zh) * 2019-01-22 2019-06-14 华南理工大学 一种情绪刺激虚拟现实场景自动生成系统及方法
CN110018738B (zh) * 2019-03-04 2021-09-21 华南理工大学 一种基于真实场景情感表达的情感转换系统
CN110648264B (zh) * 2019-09-30 2023-02-28 彭春姣 包含或挂接情绪调节成分的课件、调节情绪的方法和装置
CN111063037A (zh) * 2019-12-30 2020-04-24 北京中网易企秀科技有限公司 一种三维场景编辑方法及装置
JP7677979B2 (ja) * 2019-12-31 2025-05-15 ノボキュア ゲーエムベーハー 画像セグメンテーションのための方法、システム、および装置
CN113075996B (zh) * 2020-01-06 2024-05-17 京东方艺云(杭州)科技有限公司 一种用户情绪改善方法和系统
CN111680185A (zh) * 2020-05-29 2020-09-18 平安科技（深圳）有限公司 乐曲生成方法、装置、电子设备及存储介质
CN113011504B (zh) * 2021-03-23 2023-08-22 华南理工大学 基于视角权重和特征融合的虚拟现实场景情感识别方法
CN113284256B (zh) * 2021-05-25 2023-10-31 成都威爱新经济技术研究院有限公司 一种mr混合现实三维场景素材库生成方法及系统
CN115665461B (zh) * 2022-10-13 2024-03-22 聚好看科技股份有限公司 一种视频录制方法及虚拟现实设备
CN116211303B (zh) * 2022-12-06 2024-03-12 北京师范大学 一种场景化心理压力评测模型构建系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101887489A (zh) * 2010-05-24 2010-11-17 陈益强 一种使现实和虚拟世界中人物交互影响的方法
CN106991172A (zh) * 2017-04-05 2017-07-28 安徽建筑大学 一种多模态情感交互数据库的建立方法
CN107578807A (zh) * 2017-07-17 2018-01-12 华南理工大学 一种虚拟现实情感刺激系统的创建方法
WO2018022808A1 (en) * 2016-07-27 2018-02-01 Warner Bros. Entertainment Inc. Control of social robot based on prior character portrayal in fiction or performance
CN109887095A (zh) * 2019-01-22 2019-06-14 华南理工大学 一种情绪刺激虚拟现实场景自动生成系统及方法

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2002067194A2 (en) * 2001-02-20 2002-08-29 I & A Research Inc. System for modeling and simulating emotion states
CN101706842A (zh) * 2009-08-25 2010-05-12 浙江大学 中国情感图片系统的创建方法
CN106502425A (zh) * 2016-11-30 2017-03-15 宇龙计算机通信科技(深圳)有限公司 一种虚拟现实场景的构建方法及装置
CN108846887A (zh) * 2018-06-20 2018-11-20 首都师范大学 Vr视频的生成方法及装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101887489A (zh) * 2010-05-24 2010-11-17 陈益强 一种使现实和虚拟世界中人物交互影响的方法
WO2018022808A1 (en) * 2016-07-27 2018-02-01 Warner Bros. Entertainment Inc. Control of social robot based on prior character portrayal in fiction or performance
CN106991172A (zh) * 2017-04-05 2017-07-28 安徽建筑大学 一种多模态情感交互数据库的建立方法
CN107578807A (zh) * 2017-07-17 2018-01-12 华南理工大学 一种虚拟现实情感刺激系统的创建方法
CN109887095A (zh) * 2019-01-22 2019-06-14 华南理工大学 一种情绪刺激虚拟现实场景自动生成系统及方法

Also Published As

Publication number Publication date
CN109887095A (zh) 2019-06-14

Similar Documents

Publication Publication Date Title
WO2020151273A1 (zh) 一种情绪刺激虚拟现实场景自动生成系统及方法
Ding et al. Stimulus-driven and concept-driven analysis for image caption generation
Li et al. CAS (ME) 3: A third generation facial spontaneous micro-expression database with depth information and high ecological validity
Wang et al. Unlocking the emotional world of visual media: An overview of the science, research, and impact of understanding emotion
Happy et al. The Indian spontaneous expression database for emotion recognition
Dewan et al. A deep learning approach to detecting engagement of online learners
Soleymani et al. A multimodal database for affect recognition and implicit tagging
Zhao et al. Dfme: A new benchmark for dynamic facial micro-expression recognition
Wei et al. BNU-LSVED 2.0: Spontaneous multimodal student affect database with multi-dimensional labels
Alshammari et al. Robotics utilization in automatic vision-based assessment systems from artificial intelligence perspective: A systematic review
Ali et al. High-level concepts for affective understanding of images
Dawood et al. Affective computational model to extract natural affective states of students with Asperger syndrome (AS) in computer-based learning environment
CN116091667B (zh) 一种基于aigc技术的人物艺术形象生成系统
Stathopoulou Visual affect recognition
Ma et al. Multimodal art pose recognition and interaction with human intelligence enhancement
Bian et al. An academic emotion database and the baseline evaluation
Kasiran et al. Facial expression as an implicit customers' feedback and the challenges
Zhang et al. Film and TV character expression identification combined with deep learning and automatic generation of character animation
Mandia et al. Yolo-v5 based single step student affect state detection system
Sra et al. Deepspace: Mood-based image texture generation for virtual reality from music
Mandıra et al. Spatiotemporal and multimodal analysis of personality traits
CN118568290A (zh) 用于音视频匹配的模型训练方法、音视频匹配方法及装置
Utami et al. The Development of Facial Expressions Dataset for Teaching Context: Preliminary Research
Rothkrantz An affective distant learning model using avatars as user stand-in
Wu [Retracted] Multimodal Opera Performance Form Based on Human‐Computer Interaction Technology

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19911150

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 09.11.2021)

122 Ep: pct application non-entry in european phase

Ref document number: 19911150

Country of ref document: EP

Kind code of ref document: A1