WO2018090740A1 - 一种基于混合现实技术实现陪伴的方法及装置 - Google Patents

一种基于混合现实技术实现陪伴的方法及装置 Download PDF

Info

Publication number
WO2018090740A1
WO2018090740A1 PCT/CN2017/103968 CN2017103968W WO2018090740A1 WO 2018090740 A1 WO2018090740 A1 WO 2018090740A1 CN 2017103968 W CN2017103968 W CN 2017103968W WO 2018090740 A1 WO2018090740 A1 WO 2018090740A1
Authority
WO
WIPO (PCT)
Prior art keywords
character
data
behavior
user
instruction
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2017/103968
Other languages
English (en)
French (fr)
Inventor
刘丽
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
JRD Communication Shenzhen Ltd
Original Assignee
JRD Communication Shenzhen Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by JRD Communication Shenzhen Ltd filed Critical JRD Communication Shenzhen Ltd
Publication of WO2018090740A1 publication Critical patent/WO2018090740A1/zh
Priority to US16/282,334 priority Critical patent/US20190188903A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/16—Sound input; Sound output
    • G06F3/167—Audio in a user interface, e.g. using voice commands for navigating, audio feedback
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three-dimensional [3D] modelling for computer graphics
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03H—HOLOGRAPHIC PROCESSES OR APPARATUS
    • G03H1/00—Holographic processes or apparatus using light, infrared or ultraviolet waves for obtaining holograms or for obtaining an image from them; Details peculiar thereto
    • G03H1/02—Details of features involved during the holographic process; Replication of holograms without interference recording
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/174—Facial expression recognition
    • G06V40/176—Dynamic expression
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G06V40/23—Recognition of whole body movements, e.g. for sport training
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L17/00—Speaker identification or verification techniques
    • G10L17/26—Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/48—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use
    • G10L25/51—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination
    • G10L25/63—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination for estimating an emotional state
    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03H—HOLOGRAPHIC PROCESSES OR APPARATUS
    • G03H1/00—Holographic processes or apparatus using light, infrared or ultraviolet waves for obtaining holograms or for obtaining an image from them; Details peculiar thereto
    • G03H1/02—Details of features involved during the holographic process; Replication of holograms without interference recording
    • G03H2001/0204—Object characteristics

Definitions

  • the invention relates to the field of mixed reality technology, in particular to a method and a device for implementing companion based on a mixed reality technology.
  • the invention provides a method and a device for implementing companion based on a mixed reality technology, which can improve interaction efficiency and effect.
  • a technical solution adopted by the present invention is to provide a method for implementing companion based on a mixed reality technology, the method comprising:
  • the character is a virtualized object to the real person, and when called, simulates the action state of the real person;
  • the presentation of the character is updated with the behavioral data using a laser holographic projection technique.
  • another technical solution adopted by the present invention is to provide a method for implementing companion based on a mixed reality technology, the method comprising:
  • the character is a virtualized object to the real person, and when called, simulates the action state of the real person;
  • the presentation of the character is updated with the behavior data.
  • an apparatus for implementing companion based on a mixed reality technology including:
  • a communication circuit configured to receive a call instruction and an interactive instruction of a user summoning a character
  • a processor configured to: after receiving the summoning instruction, match a virtual model corresponding to the summoning character, and, after receiving the interactive instruction of the user, match a correspondence of the character corresponding to the interactive instruction Behavioral data;
  • a display for presenting the virtualized object and updating the presentation of the character with the behavior and reaction data.
  • the beneficial effects of the present invention are: by establishing a virtual model of a real person in advance and collecting daily behavior data of the user and the real person, when the user needs to be accompanied by a person, calling the virtual model by calling a command, and calling the daily routine
  • the behavior data allows the virtual model to interact with the user, realizing the interaction between the reality and the virtual, and effectively improving the interaction efficiency and effect.
  • FIG. 1 is a schematic flow chart of an embodiment of a method for implementing companion based on a mixed reality technology according to the present invention
  • FIG. 2 is a schematic diagram of a specific process of step S11 in FIG. 1;
  • FIG. 3 is a schematic flowchart of establishing a virtualized object in step S12 in FIG. 1;
  • step S13 in FIG. 1 is a schematic diagram of a specific process of step S13 in FIG. 1;
  • FIG. 5 is a schematic flowchart of the establishment of the database in step S14 of FIG. 1;
  • FIG. 6 is a schematic structural diagram of an embodiment of a device for implementing companion based on a mixed reality technology according to the present invention.
  • FIG. 7 is a schematic structural diagram of an embodiment of a system for implementing companion based on a hybrid reality technology according to the present invention.
  • FIG. 1 is a schematic flowchart of an embodiment of a method for implementing companion based on a mixed reality technology according to the present invention, including the following steps:
  • the smart wearable device When the user needs someone to accompany in daily life, the smart wearable device is put on, and the sound and/or behavior and/or expression is emitted, and the smart wearable device receives the sound and/or behavior and/or expression, and the The summoning instruction of the user's summoning character.
  • the smart wearable device may include, but is not limited to, a wearable virtual helmet.
  • the smart wearable device can suppress noise of the environment by using noise reduction measures.
  • step S11 may specifically include:
  • S111 Collect sound and/or behavior and/or expression data of the user.
  • the smart device collects the sound through a microphone such as a microphone, and at the same time, in order to obtain the call instruction more accurately, when collecting the user
  • a microphone such as a microphone
  • the behavior and/or expression data sent by the user can also be collected by a video collector such as a camera.
  • S112 Identify the sound and/or behavior and/or expression data to obtain the call instruction of the user.
  • the sound is recognized by a voice recognition technique to obtain the summoning command.
  • the voice content is identified by using a voice recognition technology, and key feature data such as a word or a word, such as a noun, is extracted, and based on the key feature data, the user includes the summoned object.
  • the summoning command while collecting the behavior and/or expression of the user while collecting the sound data, using image extraction techniques to extract key behaviors and/or expressions in the behavior and/or expression, such as The beating action, the crying expression, combined with the sound and the behavior and/or expression data, can more accurately obtain the summoning instruction.
  • the character is a virtualized object for a real person, and is a virtual model based on real human height, weight, measurements, bones, and facial features.
  • it can imitate the realistic human action state, and Presented in front of the user, the character is presented by the smart wearable device using a laser holographic projection technique.
  • the virtualized object corresponding to the person in the command is previously established by the server.
  • the step of establishing the virtualization object may include:
  • S121 Collect the virtualized object shape data.
  • parameters such as height, weight, measurements, bones, and the like of the real person are collected by the acquisition sensor, and the real face of the real person is collected by the image acquisition device.
  • S122 Input the shape data to perform simulation to generate the virtualized object.
  • the acquisition sensor and the image collection device After collecting the data, the acquisition sensor and the image collection device transmit the data to the server, and simultaneously upload the identity information of the real person.
  • the server establishes the 3D model of the real person based on parameters such as height, weight, measurements, and bones, and then generates a real face of the real person by using a face recognition technology, according to the 3D model and the The virtual appearance is established, and the server saves the identity information and the corresponding virtualized object.
  • the virtualized object can also be created by computer software.
  • the smart wearable device continues to collect and recognize the user's voice and/or behavior and/or expression data to obtain an interactive instruction of the user.
  • step S13 may specifically include:
  • S131 Acquire the sound and/or behavior and/or expression data of the user.
  • the wearable smart device continues to collect the sound and/or behavior and/or expression data sent by the user through the microphone and/or the video collector, and the step and the step S111 are performed.
  • the content is the same or similar and will not be described again.
  • S132 Identify the sound and/or behavior and/or expression data to obtain an interactive instruction of the user.
  • the sound is recognized by a voice recognition technique to obtain the summoning command.
  • the voice content is identified by using a voice recognition technology, and key feature data such as a word or a word, such as an action word, is extracted, and the interaction information of the user is obtained based on the key feature data.
  • the image extraction technique is used to extract key behaviors and/or expressions in the behavior and/or expression, such as limb movements.
  • the establishment of the database includes the following steps:
  • S141 Collect behavior and reaction data between the user and the virtualized object.
  • the interaction data between the user and the virtualized object is collected by a microphone and a video collector in daily life, including collecting behaviors performed by the user and reacting the virtualized object to the behavior.
  • the behavioral response includes both language and actions.
  • the microphone includes but is not limited to a microphone
  • the collector includes but is not limited to a camera, and the specific number and position of the collector are not limited, but the range of capture can include the range of activities of the user and the virtualized object as much as possible. .
  • the data is uploaded to the server.
  • the server uses the mining of big data to analyze the data: the behavior analysis of the user is statistically generated to generate instruction information, and the response of the virtualized object to the language and action of the behavior is calculated, and The relationship between the instruction information and the reaction is saved to form the database.
  • the collection behavior and the reaction data are continuously performed in daily life, so the instruction information and the behavior response data in the database are continuously updated and stored.
  • the behavior response data corresponding to the virtualized object After reading the behavior response data corresponding to the virtualized object, updating the presentation of the character with the data, and causing the character to make the corresponding behavioral response, the behavior including an action and a language.
  • the language can be sent through a device such as a speaker, such as an earpiece, and the presentation of the action is presented to the user by the wearable smart device using laser holographic projection technology.
  • FIG. 6 is a schematic structural diagram of an apparatus for implementing companion based on a mixed reality technology according to the present invention, including:
  • the receiving module 21 is configured to receive a call and interaction instruction of the user summoning the character, including:
  • the collecting unit 211 is configured to collect sound and/or behavior and/or expression data of the user.
  • the collecting unit 211 collects the data.
  • the acquisition unit 211 can include a microphone and/or a video collector. Specifically, after the user emits a sound, the microphone collects the sound data, or when collecting the sound data for more accurate instruction to the user, the video data may also be collected. The device collects the behavior and/or expression data. At the same time, in order to collect the sound data more clearly, the receiving unit 211 can use the noise reduction measure to suppress the noise of the environment as much as possible.
  • the microphone may be, but is not limited to, a microphone.
  • the video capture device may be, but not limited to, a camera, and the location of the video capture device installation is not limited, but the range of capture must include the upper body portion of the user.
  • the first identifying unit 212 is configured to obtain a voice command and/or a behavior command by using the voice recognition and/or image recognition technology after the collecting unit 211 first collects the voice and/or behavior and/or expression data of the user. And obtaining, according to the instruction, a call instruction of the user summoning person.
  • the second identifying unit 213 is configured to obtain the interactive instruction of the user by using voice recognition and/or image recognition technology, after the virtualized object is summoned, for the sound and/or behavior and/or expression data that is continuously collected. .
  • the processing module 24 is configured to: after receiving the summoning instruction, match the virtualization model corresponding to the calling instruction, and after receiving the interaction instruction of the user, match the corresponding to the interaction instruction The corresponding behavior data of the character.
  • a presentation module 25 configured to present the virtualized object through the wearable smart device by using a laser holographic projection technology after being matched to the virtualized model, and after matching the corresponding behavior data of the character
  • the laser holographic projection technique updates the presentation of the character with the behavioral data by the wearable smart device, and causes the character to make the corresponding behavioral response, wherein the behavioral response includes both a language and an action.
  • the device further includes: a model building module 24, configured to establish a virtual model of the character, including:
  • the first data collection unit 241 includes an acquisition sensor and an image collector for collecting parameters such as height, weight, measurements, bones, and real features of the character.
  • a model establishing unit 242 configured to establish a 3D model of the character based on the height, weight, measurements, bones, and the like in the server, and generate a real face of the character by using a face recognition technology, thereby establishing the Realistic human virtualization model.
  • the database module 25 is configured to establish the database, including:
  • the second data collecting unit 251 includes a microphone such as a microphone and a video collector such as a camera for collecting interaction data between the user and the virtualized object through a microphone and a video collector in daily life, including a collecting station. Describe the behavior of the user and the response of the virtualized object to the behavior.
  • the database unit 252 is configured to perform the analysis of the data by using the mining of big data in the server: analyzing the behavior of the user to generate instruction information, and counting the language of the virtualized object to the behavior Reacting with the action and saving the relationship between the instruction information and the reaction to form the database.
  • the second data collection unit 251 collects behaviors and reaction data continuously in daily life, so the instruction information and behavior response data in the database unit 251 are continuously updated and stored.
  • FIG. 7 is a schematic structural diagram of an embodiment of a system for implementing companion based on a hybrid reality technology according to the present invention.
  • the system includes a terminal device and a server, and the terminal device can perform the steps in the foregoing method.
  • the terminal device can perform the steps in the foregoing method.
  • the terminal includes a processor 31, a first sensor 32 coupled to the processor 31, a communication circuit 33, and a virtual reality presentation device 34.
  • the first sensor 32 collects information for the processor 31 to acquire a call instruction of the user summoning the character
  • the processor 31 calls the virtual reality rendering device to present a character in response to the instruction, and the character is a virtualized object to the real person, and simulates the action state of the real person when called;
  • the first sensor 32 continues to collect information for the processor 31 to receive an interactive command from the user to the character;
  • the processor 31 sends the interactive instruction to the server 36 through the communication circuit 33 to match the interactive instruction in the database of the server to obtain the corresponding behavior data of the character;
  • the second sensor 35 is included to collect language interaction data between the user and the virtualized person in daily life, and record behavior and reaction data between the user and the virtualized person in daily life, and the shooting center The true shape of the real person is passed to the server 36.
  • the first sensor, the second sensor may be a microphone, a video collector, and/or an image collector.
  • the server 36 uses the big data to analyze statistical language, behavior, and reaction data, generates a database, and generates a virtual model corresponding to the real person according to the real shape of the captured real person, and saves the database and the virtual model.
  • the virtualized object by pre-establishing a virtual model of the virtualized object and a database of language, behavior, and reaction of the user and the virtualized object in daily life, when the user issues an instruction to be accompanied by a person, the virtualized object is called out.
  • the virtual model updates the presentation of the virtualization model with the language, behavior, and response data in the database, allowing the user to interact with the virtual world, which can effectively improve the interaction effect and efficiency.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • General Engineering & Computer Science (AREA)
  • General Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Software Systems (AREA)
  • Computer Graphics (AREA)
  • Acoustics & Sound (AREA)
  • Computer Hardware Design (AREA)
  • Psychiatry (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Hospice & Palliative Care (AREA)
  • Geometry (AREA)
  • Child & Adolescent Psychology (AREA)
  • Computational Linguistics (AREA)
  • Signal Processing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Social Psychology (AREA)
  • User Interface Of Digital Computer (AREA)
  • Processing Or Creating Images (AREA)

Abstract

一种基于混合现实技术实现陪伴的方法及装置。其中，所述方法包括：接收用户召唤人物的召唤指令(S11)；响应所述指令进而呼唤出所述人物，所述人物是对现实人的虚拟化对象，当被呼唤后模拟现实人的动作神态(S12)；接收所述用户对所述人物的交互指令(S13)；在数据库中匹配所述交互指令进而得到所述人物的对应行为数据(S14)；以所述行为数据更新所述人物的呈现(S15)。通过上述方式，能够通过现实人和虚拟世界的交互，可有效提高交互的效率和效果。

Description

一种基于混合现实技术实现陪伴的方法及装置
【技术领域】
本发明涉及混合现实技术领域，特别是涉及一种基于混合现实技术实现陪伴的方法及装置。
【背景技术】
随着虚拟现实、增强现实技术的日趋成熟，用户既可以在用计算机构建的虚拟世界中有身临其境的感觉，也可以将虚拟内容叠加到现实世界中，但无论是虚拟现实技术还是增强现实技术，都无法满足用户和虚拟内容进行交互的要求，于是混合现实技术应运而生，它在虚拟世界、现实世界和用户之间搭建了一个交互反馈的回路，使用户可以和虚拟世界进行交互，提升用户体验的真实感。
【发明内容】
本发明提供一种基于混合现实技术实现陪伴的方法及装置，能够提高交互效率和效果。
为解决上述技术问题，本发明采用的一个技术方案是：提供一种基于混合现实技术实现陪伴的方法，该方法包括：
采集用户的声音和/或行为和/或表情数据，识别所述声音和/或行为和/或表情数据而得到所述用户召唤人物的召唤指令；
响应所述指令进而呼唤出所述人物，所述人物是对现实人的虚拟化对象，当被呼唤后模拟现实人的动作神态；
接收所述用户对所述人物的交互指令；
在数据库中匹配所述交互指令进而得到所述人物的对应行为数据；
采用激光全息投影技术以所述行为数据更新所述人物的呈现。
为解决上述技术问题，本发明采用的另一个技术方案是：提供一种基于混合现实技术实现陪伴的方法，该方法包括：
接收用户召唤人物的召唤指令；
响应所述指令进而呼唤出所述人物，所述人物是对现实人的虚拟化对象，当被呼唤后模拟现实人的动作神态；
接收所述用户对所述人物的交互指令；
在数据库中匹配所述交互指令进而得到所述人物的对应行为数据；
以所述行为数据更新所述人物的呈现。
为解决上述技术问题，本发明提供的又一种技术方案是：提供一种基于混合现实技术实现陪伴的装置，包括：
通信电路，用于接收用户召唤人物的召唤指令及交互指令；
处理器，用于当接收到所述召唤指令后，去匹配所述召唤人物对应的虚拟模型，以及当接收到所述用户的交互指令后，去匹配所述交互指令对应的所述人物的对应行为数据；
显示器，用于呈现所述虚拟化对象，并以所述行为、反应数据更新所述人物的呈现。
本发明的有益效果是：通过事先建立现实人的虚拟模型和收集用户与所述现实人的日常行为数据，当所述用户需要人陪伴时，通过召唤指令，呼出所述虚拟模型，并调用日常行为数据让所述虚拟模型和所述用户进行交互，实现了现实与虚拟的交互，可有效提高交互效率及效果。
【附图说明】
图1是本发明基于混合现实技术实现陪伴的方法一实施例的流程示意图；
图2是图1中步骤S11的具体流程示意图；
图3是图1中步骤S12中所述虚拟化对象建立的流程示意图；
图4是图1中步骤S13的具体流程示意图；
图5是图1中步骤S14中所述数据库建立的流程示意图；
图6是本发明基于混合现实技术实现陪伴的装置一实施例的结构示意图；
图7是本发明基于混合现实技术实现陪伴的系统一实施例的结构示意图。
【具体实施方式】
参阅图1，图1是本发明基于混合现实技术实现陪伴的方法一实施例的流程示意图，包括以下步骤：
S11：接收用户召唤人物的召唤指令。
当用户在日常生活中需要有人陪伴时，戴上智能可穿戴设备，发出声音和/或行为和/或表情，所述智能可穿戴设备接收所述声音和/或行为和/或表情，得出所述用户的召唤人物的召唤指令。其中，所述智能可穿戴设备可包括但不限于可戴式虚拟头盔。同时为了更加清楚地采集到所述声音，所述智能穿戴设备可以采用降噪措施抑制环境的声音。
参阅图2，步骤S11具体可包括：
S111：采集所述用户的声音和/或行为和/或表情数据。
当所述用户带穿上所述设备，第一次发出声音数据时，所述智能设备通过传声器如麦克风采集所述声音，同时为了更加准确地得到所述召唤指令，当在采集所述用户的声音数据时，还可以通过视频采集器如摄像头采集所述用户发出的行为和/或表情数据。
S112：识别所述声音和/或行为和/或表情数据进而得到所述用户的召唤指令。
当采集到所述声音后，通过语音识别技术识别出所述声音，得出所述召唤指令。具体地，当采集到声音后，运用语音识别技术对所述声音内容进行识别，提取出字或词等关键特征数据，如称呼名词，基于所述关键特征数据，得出所述用户包含召唤对象的召唤指令，同时如果在采集声音数据的同时，还采集到所述用户的行为和/或表情后，运用图像提取技术，提取所述行为和/或表情中的关键行为和/或表情，如招手动作、哭泣的表情，结合所述声音及所述行为和/或表情数据能够更加准确地得到所述的召唤指令。
S12：响应所述指令进而呼唤出所述人物，所述人物是对现实人的虚拟化对象，当被呼唤后模拟显示人的动作神态。
当所述智能可穿戴设备识别出所述召唤指令后，召唤出所述指令中对应的人物。其中，所述人物是对现实人的虚拟化对象，是一个基于现实人身高、体重、三围、骨骼以及面貌等参数建立的虚拟模型，当被呼唤出来后，能够模仿现实人的动作神态，并呈现在所述用户眼前，所述人物通过所述可智能穿戴设备采用激光全息投影技术呈现。
所述与指令中人物对应的虚拟化对象事先通过服务器进行建立。参阅图3，所述虚拟化对象建立的步骤可包括：
S121：采集所述虚拟化对象外形数据。
当需要建立现实人的虚拟化对象时，通过采集传感器采集所述现实人的身高、体重、三围、骨骼等参数，同时通过图像采集装置采集所述现实人的真实面貌 。
S122：输入所述外形数据进而模拟仿真生成所述虚拟化对象。
当采集到所述数据后，所述采集传感器及所述图像采集装置将所述数据传输带服务器，同时还上传所述现实人的身份信息。所述服务器基于所述身高、体重、三围以及骨骼等参数建立所述现实人的3D模型，再利用人脸识别技术仿真生成所述现实人的真实面貌，根据所述的3D模型和所述的真实面貌，建立所述虚拟化对象，同时所述服务器对所述身份信息及对应的虚拟化对象进行保存。
在其他实例中，所述虚拟化对象也可以通过计算机软件进行建立。
S13：接收所述用户对所述人物的交互指令。
当所述虚拟化对象被召唤后，所述可智能穿戴设备继续采集识别所述用户的声音和/或行为和/或表情数据，得到所述用户的交互指令。
参阅图4，步骤S13具体可包括：
S131：采集所述用户的声音和/或行为和/或表情数据。
当所述虚拟化对象被召唤出来后，所述可穿戴智能设备继续通过传声器和/或视频采集器采集所述用户发出的声音和/或行为和/或表情数据，此步骤与所述步骤S111内容相同或类似，不再赘述。
S132：识别所述声音和/或行为和/或表情数据进而得到所述用户的交互指令。
当采集到所述声音后，通过语音识别技术识别出所述声音，得出所述召唤指令。具体地，当采集到声音后，运用语音识别技术对所述声音内容进行识别，提取出字或词等关键特征数据，如动作词，基于所述关键特征数据，得出所述用户的交互信息，同时如果在采集声音数据的同时，还采集到所述用户的行为和/或表情后，运用图像提取技术，提取所述行为和/或表情中的关键行为和/或表情，如肢体动作、面部的喜怒哀乐，结合所述声音及所述行为和/或表情数据能够更加真实准备地得到所述用户想要传达的信息，得出所述交互指令。
S14：在数据库中匹配所述交互指令进而得到所述人物的对应行为数据。
当识别所述用户的交互指令后，在所述服务器的数据库中查找匹配相同或最相似的交互指令，并读取所述交互指令对应的所述虚拟化对象的行为数据。参阅图5，所述数据库的建立包括以下步骤：
S141：采集所述用户和所述虚拟化对象之间的行为、反应数据。
在日常生活中通过传声器及视频采集器采集所述用户和所述虚拟化对象的之间的互动数据，包括采集所述用户做出的行为以及所述虚拟化对象对所述行为作出的反应。所述的行为反应既包括语言也包括动作。所述传声器包括但不限于麦克风，所述采集器包括但不限于摄像头，所述采集器的具体数量和位置不作限定，但是捕获的范围尽量能够包括所述用户和所述虚拟化对象活动的范围。
S142：分析所述行为、反应数据进而得到所述的数据库
采集到所述用户和所述虚拟化对象之间的行为、反应数据后，将所述数据上传至所述服务器。所述服务器运用大数据的挖掘，对所述数据进行分析：将所述用户的行为分析统计生成指令信息，并统计出所述虚拟化对象对所述行为作出的语言和动作的反应，并将所述指令信息及所述反应之间的关系进行保存形成所述数据库。
所述采集行为、反应数据在日常生活中不断进行的，因此所述数据库中的指令信息、行为反应数据是在不断更新存储的。
S15：以所述行为数据更新所述人物的呈现。
当读取到对应所述虚拟化对象的行为反应数据后，以所述数据更新所述人物的呈现，让所述人物做出所述对应的行为反应，所述行为包括动作及语言。其中语言可以通过扬声器如听筒等设备发出，动作的呈现则采用激光全息投影技术通过所述可穿戴智能设备的呈现在所述用户眼前。
参阅图6，图6是本发明基于混合现实技术实现陪伴的装置一实施例的结构示意图，包括：
接收模块21，用于接收用户召唤人物的召唤和交互指令，包括：
采集单元211，用于采集所述用户的声音和/或行为和/或表情数据。
当所述用户需要有人陪伴时，穿上智能可穿戴设备后，发出声音和/或行为和/或表情数据后，采集单元211对所述数据进行采集。所述采集单元211可包括传声器和/或视频采集器。具体地，当所述用户发出声音后，所述传声器对所述声音数据进行采集，或当为了更加准确地的到所述用户的指令，在采集所述声音数据的同时，还可以通过视频采集器对所述行为和/或表情数据进行采集。同时为了更加清楚地采集到所述声音数据，接收单元211可以采用降噪措施尽量抑制环境的噪声。所述传声器可为但不限于麦克风，所述视频采集装置可为但不限于摄像头，且对于所述视频采集器安装的位置不作限定，但是捕获的范围必须包括所述用户的上身部位。
第一识别单元212，用于当采集单元211第一次采集到所述用户的声音和/或行为和/或表情数据后，运用声音识别和/或图像识别技术得到声音指令和/或行为指令，基于所述指令，进而得到所述用户召唤人物的召唤指令。
第二识别单元213，用于当召唤出所述虚拟化对象后，对继续采集得到的声音和/或行为和/或表情数据，运用声音识别和/或图像识别技术得到所述用户的交互指令。
处理模块24，用于当接收到召唤指令后，去匹配与所述召唤指令对应的所述虚拟化模型，以及当接收到所述用户的交互指令后，去匹配与所述交互指令对应的所述人物的对应行为数据。
呈现模块25，用于当匹配到所述虚拟化模型后，运用激光全息投影技术通过所述可穿戴智能设备呈现出所述虚拟化对象，以及当匹配到所述人物的对应行为数据后，运用激光全息投影技术通过所述可穿戴智能设备以所述行为数据更新所述人物的呈现，让所述人物做出所述对应的行为反应，其中所述的行为反应既包括语言也包括动作。
同时，所述装置还包括：模型建立模块24，用于建立所述人物的虚拟模型，包括：
第一数据收集单元241，包括采集传感器和图像采集器，用于采集所述人物的身高、体重、三围、骨骼等参数及真实面貌。
模型建立单元242，用于在服务器中基于所述身高、体重、三围、骨骼等参数，建立所述人物的3D模型，以及运用人脸识别技术，生成所述人物的真实面貌，从而建立所述现实人的虚拟化模型。
数据库模块25，用于建立所述的数据库，包括：
第二数据收集单元251，包括传声器如麦克风及视频采集器如摄像头，用于在日常生活中通过传声器及视频采集器收集所述用户和所述虚拟化对象的之间的互动数据，包括采集所述用户做出的行为以及所述虚拟化对象对所述行为作出的反应。
数据库单元252，用于在所述服务器中运用大数据的挖掘对所述数据进行分析：将所述用户的行为分析统计生成指令信息，并统计出所述虚拟化对象对所述行为作出的语言和动作的反应，并将所述指令信息及所述反应之间的关系进行保存形成所述数据库。
所述第二数据采集单元251收集行为、反应数据在日常生活中不断进行的，因此所述数据库单元251中的指令信息、行为反应数据是在不断更新存储的。
参阅图7，图7是本发明基于混合现实技术实现陪伴的系统一实施例的结构示意图。该系统包括终端设备以及服务器，终端设备可以执行上述方法中的步骤，相关内容请参见上述方法中的详细说明，在此不再赘述。
本实施例中，该终端包括：处理器31、与处理器31耦接的第一传感器32、通信电路33以及虚拟现实呈现装置34。
第一传感器32采集信息，以供处理器31获取用户召唤人物的召唤指令；
处理器31响应指令进而呼唤出人物，控制所述虚拟现实呈现装置呈现人物，所述人物是对现实人的虚拟化对象，当被呼唤后模拟现实人的动作神态；
第一传感器32继续采集信息，以供处理器31接收用户对人物的交互指令；
处理器31通过通信电路33将交互指令发给服务器36，以在服务器的数据库中匹配交互指令进而得到人物的对应行为数据；
进一步，包括第二传感器35，在日常生活中收集用户和虚拟化人物之间的语言互动数据，以及在日常生活中记录所述用户和所述虚拟化人物之间的行为、反应数据，拍摄所述现实人的真实外形，传给服务器36。
第一传感器、第二传感器可以是传声器、视频采集器和/或图像采集器。
服务器36运用大数据分析统计语言、行为、反应数据，生成数据库，并根据所拍摄的现实人的真实外形，生成所述现实人对应的虚拟模型，并对数据库和虚拟模型进行保存。
以上方案，通过事先建立虚拟化对象的虚拟模型及用户和所述虚拟化对象在日常生活中的语言、行为、反应的数据库，当所述用户发出指令需要有人陪伴时，呼出所述虚拟化对象的虚拟模型，并以所述数据库中的语言、行为、反应数据更新所述虚拟化模型的呈现，让用户和虚拟世界进行交互，能够有效提高交互效果和效率。
以上所述仅为本发明的实施方式，并非因此限制本发明的专利范围，凡是利用本发明说明书及附图内容所作的等效结构或等效流程变换，或直接或间接运用在其他相关的技术领域，均同理包括在本发明的专利保护范围内。

Claims (14)

  1. 一种基于混合现实技术实现陪伴的方法，其中，包括：
    采集用户的声音和/或行为和/或表情数据，识别所述声音和/或行为和/或表情数据而得到所述用户召唤人物的召唤指令；
    响应所述指令进而呼唤出所述人物，所述人物是对现实人的虚拟化对象，当被呼唤后模拟现实人的动作神态；
    接收所述用户对所述人物的交互指令；
    在数据库中匹配所述交互指令进而得到所述人物的对应行为数据；
    采用激光全息投影技术以所述行为数据更新所述人物的呈现。
  2. 如权利要求1所述的方法，其中，所述虚拟化对象采用以下方法获得：
    采集所述虚拟化对象外形数据；
    输入所述外形数据进而模拟仿真生成所述虚拟化对象。
  3. 如权利要求1所述的方法，其中，所述数据库采用以下方法获得：
    采集所述虚拟化对象行为、反应数据；
    分析所述行为、反应数据进而得到所述虚拟化对象对应行为反应的所述数据库。
  4. 如权利要求1所述的方法，其中，所述接收所述用户对所述人物的交互指令，包括：
    采集所述用户的声音和/或行为和/或表情数据；
    识别所述声音和/或行为和/或表情数据进而得到所述用户对所述人物的交互指令。
  5. 一种基于混合现实技术实现陪伴的方法，其中，包括：
    接收用户召唤人物的召唤指令；
    响应所述指令进而呼唤出所述人物，所述人物是对现实人的虚拟化对象，当被呼唤后模拟现实人的动作神态；
    接收所述用户对所述人物的交互指令；
    在数据库中匹配所述交互指令进而得到所述人物的对应行为数据；
    以所述行为数据更新所述人物的呈现。
  6. 如权利要求5所述的方法，其中，所述虚拟化对象采用以下方法获得：
    采集所述虚拟化对象外形数据；
    输入所述外形数据进而模拟仿真生成所述虚拟化对象。
  7. 如权利要求5所述的方法，其中，所述数据库采用以下方法获得：
    采集所述虚拟化对象行为、反应数据；
    分析所述行为、反应数据进而得到所述虚拟化对象对应行为反应的所述数据库。
  8. 如权利要求5所述的方法，其中，所述以所述行为数据更新所述人物的呈现具体采用激光全息投影技术以所述行为数据更新所述人物的呈现。
  9. 如权利要求5所述的方法，其中，所述接收所述用户的指令，包括：
    采集所述用户的声音和/或行为和/或表情数据；
    识别所述声音和/或行为和/或表情数据进而得到所述指令。
  10. 一种基于混合现实技术实现陪伴的装置，其中，包括：
    通信电路，用于接收用户召唤人物的召唤指令及交互指令；
    处理器，用于当接收到所述召唤指令后，去匹配所述召唤人物对应的虚拟模型，以及当接收到所述用户的交互指令后，去匹配所述交互指令对应的所述人物的对应行为数据；
    显示器，用于呈现所述虚拟化对象，并以所述行为、反应数据更新所述人物的呈现。
  11. 如权利要求10所述的装置，其中，所述处理器还用于生成所述人物的虚拟模型，
    所述装置还包括：第一采集设备，用于采集所述人物的外形数据，所述处理器根据所述外形数据进而模拟仿真生成所述虚拟化对象。
  12. 如权利要求10所述的装置，其中，所述处理器还用于建立所述的数据库，
    所述装置还包括：第二采集设备，用于采集所述虚拟化对象行为、反应数据，所述处理器分析所述行为、反应数据进而得到所述虚拟化对象对应行为反应的所述数据库。
  13. 如权利要求10所述的装置，其中，所述显示器具体采用激光全息投影技术以所述行为数据更新所述人物的呈现。
  14. 如权利要求10所述的装置，其中，所述通信电路采集所述用户的声音和/或行为和/或表情数据，并识别所述声音和/或行为和/或表情数据进而得到所述的召唤指令及交互指令。
PCT/CN2017/103968 2016-11-15 2017-09-28 一种基于混合现实技术实现陪伴的方法及装置 Ceased WO2018090740A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US16/282,334 US20190188903A1 (en) 2016-11-15 2019-02-22 Method and apparatus for providing virtual companion to a user

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201611036528.X 2016-11-15
CN201611036528.XA CN106775198A (zh) 2016-11-15 2016-11-15 一种基于混合现实技术实现陪伴的方法及装置

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/282,334 Continuation US20190188903A1 (en) 2016-11-15 2019-02-22 Method and apparatus for providing virtual companion to a user

Publications (1)

Publication Number Publication Date
WO2018090740A1 true WO2018090740A1 (zh) 2018-05-24

Family

ID=58971104

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/103968 Ceased WO2018090740A1 (zh) 2016-11-15 2017-09-28 一种基于混合现实技术实现陪伴的方法及装置

Country Status (3)

Country Link
US (1) US20190188903A1 (zh)
CN (1) CN106775198A (zh)
WO (1) WO2018090740A1 (zh)

Families Citing this family (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106775198A (zh) * 2016-11-15 2017-05-31 捷开通讯(深圳)有限公司 一种基于混合现实技术实现陪伴的方法及装置
CN107276883B (zh) * 2017-06-19 2020-09-25 李明 用于混合现实的终端装置、网络侧装置及信息交互方法
CN107562195A (zh) * 2017-08-17 2018-01-09 英华达（南京）科技有限公司 人机交互方法及系统
CN109426860A (zh) * 2017-08-23 2019-03-05 幻视互动（北京）科技有限公司 一种基于神经网络的mr混合现实信息处理方法及装置
CN107678626A (zh) * 2017-10-11 2018-02-09 广东小天才科技有限公司 一种互动交流方法、系统及终端设备
CN108021896B (zh) * 2017-12-08 2019-05-10 北京百度网讯科技有限公司 基于增强现实的拍摄方法、装置、设备及计算机可读介质
CN108614638B (zh) * 2018-04-23 2020-07-07 太平洋未来科技（深圳）有限公司 Ar成像方法和装置
CN109965466A (zh) * 2018-05-29 2019-07-05 北京心有灵犀科技有限公司 Ar虚拟角色智能珠宝
CN109692392A (zh) * 2019-01-30 2019-04-30 华东交通大学 一种基于云平台的音乐治疗方法及系统
CN109917996A (zh) * 2019-02-26 2019-06-21 北京达佳互联信息技术有限公司 互动内容展示方法、装置、电子设备及存储介质
CN111176435A (zh) * 2019-11-06 2020-05-19 广东小天才科技有限公司 一种基于用户行为的人机交互方法及音箱
WO2022036473A1 (zh) * 2020-08-17 2022-02-24 南京翱翔智能制造科技有限公司 一种基于动态三维重建的混合现实协同场景共享方法
CN112530218A (zh) * 2020-11-19 2021-03-19 深圳市木愚科技有限公司 多对一陪伴智能教学系统及教学方法
CN112508161A (zh) * 2020-11-26 2021-03-16 珠海格力电器股份有限公司 陪伴数字替身的控制方法、系统和存储介质
CN112749777A (zh) * 2021-01-14 2021-05-04 郑州捷安高科股份有限公司 情景交互仿真方法、装置、设备及存储介质
CN113724353A (zh) * 2021-08-31 2021-11-30 天津泰讯视动科技有限责任公司 虚拟演播图像生成方法、系统、终端及存储介质
CN117524417A (zh) * 2022-07-29 2024-02-06 武汉联影智融医疗科技有限公司 自闭症康复训练系统、方法、设备及介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104866101A (zh) * 2015-05-27 2015-08-26 世优（北京）科技有限公司 虚拟对象的实时互动控制方法及装置
CN105446953A (zh) * 2015-11-10 2016-03-30 深圳狗尾草智能科技有限公司 一种智能机器人与虚拟3d的交互系统及方法
CN105807933A (zh) * 2016-03-18 2016-07-27 北京光年无限科技有限公司 一种用于智能机器人的人机交互方法及装置
CN106775198A (zh) * 2016-11-15 2017-05-31 捷开通讯(深圳)有限公司 一种基于混合现实技术实现陪伴的方法及装置

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104866101A (zh) * 2015-05-27 2015-08-26 世优（北京）科技有限公司 虚拟对象的实时互动控制方法及装置
CN105446953A (zh) * 2015-11-10 2016-03-30 深圳狗尾草智能科技有限公司 一种智能机器人与虚拟3d的交互系统及方法
CN105807933A (zh) * 2016-03-18 2016-07-27 北京光年无限科技有限公司 一种用于智能机器人的人机交互方法及装置
CN106775198A (zh) * 2016-11-15 2017-05-31 捷开通讯(深圳)有限公司 一种基于混合现实技术实现陪伴的方法及装置

Also Published As

Publication number Publication date
US20190188903A1 (en) 2019-06-20
CN106775198A (zh) 2017-05-31

Similar Documents

Publication Publication Date Title
WO2018090740A1 (zh) 一种基于混合现实技术实现陪伴的方法及装置
WO2020149591A1 (en) Method and apparatus for detecting abnormality of caller
WO2018082626A1 (zh) 虚拟现实系统的实现方法及虚拟现实装置
WO2020213799A1 (ko) 스마트 휴대 기기를 이용한 스마트 기기와 로봇의 개인 맞춤형 서비스 운용 시스템 및 방법
WO2020171621A1 (en) Method of controlling display of avatar and electronic device therefor
WO2022196921A1 (ko) 인공지능 아바타에 기초한 인터랙션 서비스 방법 및 장치
WO2011081379A2 (en) Display device and control method thereof
WO2019156332A1 (ko) 증강현실용 인공지능 캐릭터의 제작 장치 및 이를 이용한 서비스 시스템
WO2014019317A1 (zh) 基于云的智能人脸识别检索方法
WO2019078507A1 (en) ELECTRONIC DEVICE AND METHOD FOR PROVIDING A STRESS INDEX CORRESPONDING TO THE ACTIVITY OF A USER
WO2017071453A1 (zh) 一种语音识别的方法及装置
WO2016107230A1 (zh) 一种3d场景中重现物体的系统和方法
WO2023224433A1 (en) Information generation method and device
WO2016155284A1 (zh) 一种终端的信息采集方法及其终端
WO2019132459A1 (ko) 사용자 정서적 행동 인식을 위한 멀티 모달 정보 결합 방법 및 그 장치
CN114420302A (zh) 一种企事业单位智能防疫控制系统
WO2022145655A1 (ko) 증강현실시스템
WO2019223547A1 (zh) 教学分析方法、服务器及计算机可读存储介质
WO2018164451A1 (ko) 오디오 신호를 이용하여 입력을 검출하기 위한 방법 및 그 전자 장치
Abbas et al. A hybrid alert system for deaf people using context-aware computing and image processing
WO2020218899A1 (ko) 다중 반응형영상 제작방법, 다중 반응형영상 메타데이터 생성방법, 인간 행동을 이해하기 위한 상호 작용 데이터 분석 방법 및 이를 이용한 프로그램
CN111739181A (zh) 考勤方法及装置、电子设备及存储介质
WO2017026834A1 (ko) 반응형 영상 생성방법 및 생성프로그램
WO2012034469A1 (zh) 基于手势的人机交互方法及系统、计算机存储介质
WO2023120754A1 (ko) 메타버스 기반의 가상화 영상 제공 시스템 및 그 방법

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17871530

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205 DATED 17/10/2019)

122 Ep: pct application non-entry in european phase

Ref document number: 17871530

Country of ref document: EP

Kind code of ref document: A1