WO2021102845A1

WO2021102845A1 - Remote family perception and virtual presentation method based on natural interaction

Info

Publication number: WO2021102845A1
Application number: PCT/CN2019/121733
Authority: WO
Inventors: 卢光辉; 蔡洪斌; 傅鹏
Original assignee: 电子科技大学
Priority date: 2019-11-28
Filing date: 2019-11-28
Publication date: 2021-06-03
Also published as: CN112578906A

Abstract

Disclosed is a remote family perception and virtual presentation method based on natural interaction, wherein same is aimed at addressing elderly peoples' mental health problems, alleviates the loneliness of elderly people, specifically empty-nesters, and solves the problem of service industries in the present market ignoring the mental health of elderly people and instead paying too much attention to their physical health. A product system constructed by means of the method comprises hardware and software, wherein a hardware structure comprises: an action behavior collection/presentation device, a sound collection/presentation device, and a data processing and analysis device; and a software domain structure comprises: an action behavior collection/presentation system, a sound collection/presentation system, a data processing and analysis system, a database management system, etc. According to the method of the present invention, action behavior data of a remote family member during their daily life is collected in real time, an activity situation is locally reproduced in real time and in a synchronous manner, and an activity state is presented in a three-dimensional manner, thereby realizing natural interaction and an immersive experience. In the present invention, visual images of the environment and members of a family are not collected, such that the family's privacy is effectively protected, which is conducive to promoting the use of the present invention.

Description

A remote home perception and virtual presentation method based on natural interaction

Technical field

The invention belongs to the technical field of home medical and health care, and in particular relates to a remote home perception and virtual presentation technology based on natural interaction.

Background technique

my country’s census results in recent years show that by 2020, the country’s elderly population will reach 248 million, and the elderly population will account for 17.17%, of which 30.67 million will reach 30.67 million, while the population over 60 is expected to be When it reaches 300 million in 2025, my country will become a super-elderly country, and due to my country’s one-child problem, most of these elderly families will become empty nest families. At present, when the medical, insurance and information services industries deal with the problem of social aging, most of them start with the physical health of elderly care, maintenance and medical care, but for the elderly, especially the mental health of the empty-nest elderly With very few products, how to solve the psychological problem of loneliness among the elderly has become an urgent social problem. With the research and development of holographic imaging technology, virtual reality and augmented reality technology, sensor technology, 5G technology and other related fields, the technology system based on natural interaction and experience will gradually mature, focusing on "natural immersive" remote home perception and virtual The presentation technology will gradually mature. The service platforms and products developed based on these technologies can effectively deal with the mental health problems of the elderly in an aging society, solve the loneliness of the elderly, especially the empty-nest elderly, in their daily lives, and always feel that their children are around. So I no longer feel lonely, and enjoy my old life happily.

The daily life scenes of remote homes are presented locally through technical means, and interactions such as voice calls can be conducted in real time. Such technologies and products mainly include two categories: video surveillance and video conferencing. Video surveillance can transmit the scenes of remote homes to the local site in real time through the camera for local viewing. Video conferences can transmit and display scenes from different locations in real time on different terminals, and all users can also interact through voice. However, these two types of technologies or products are not suitable for use in the family. There are two main problems. One is that the interactive mode of video and voice is very unnatural, and it is inconvenient to use in the family. Every time a parent talks with their children, it needs cumbersome The operation steps make people feel that their relatives are not around at all times, and it is impossible to create an atmosphere where the relatives are around. Although this traditional way of call and communication is quite mature and has existed for many years, it still cannot solve the problem of loneliness among the elderly. The second is to use video technology to transmit remote scenes in the form of images. This method is not suitable for occasions where family privacy is very important. At the same time, video conferences only collect images in a very narrow and relatively fixed space such as the meeting place. Unable to meet the requirements of immersion, but always remind the elderly: children are not around. Therefore, even if privacy issues are not considered, video conferencing still cannot solve the problem of loneliness among the elderly.

The present invention can solve the above-mentioned problems existing in existing products through behavior perception technology, three-dimensional virtual presentation technology, and the like.

Summary of the invention

The present invention aims to address the mental health of the elderly, adopt remote family perception and virtual presentation technology based on natural interaction, collect and transmit remote family environment and member activity information in real time, and present the daily activities of remote family members synchronously in real time by the local family. A natural and immersive family environment combining reality and virtuality where relatives are at home and never leave, so as to solve the loneliness of the elderly, especially the empty-nest elderly.

The present invention provides a remote home perception and virtual presentation method based on natural interaction. The method includes the following main steps:

Step 1. Collect and extract the movement behavior and sound data of family members in daily activities in the remote home's daily activity scenes (such as living room, kitchen, entrance hall, home garden, entrance, balcony and other relatively public places) in real time.

Step 1.1, the action collection equipment and the sound collection equipment are processed at any time in the connected working state. Once the activities or sound signals of the members in the daily activity scene of the family are captured, the data collection is started immediately, through binocular or multi-eye vision technology and audio technology , Collect family members' activities and behaviors in real time to form bone data and scene depth data, extract movement behavior characteristic data through analysis, and collect voice or other sound data at the same time.

Step 1.2, the use of motion behavior data and voice and other audio data does not require manual operations such as traditional dial-up or network connections. Family members can move and talk freely, and the collection equipment can sense data changes and collect data in real time to realize family members Natural interaction.

Step 1.3, the acquisition device adopts infrared or other imaging technology and audio capture technology to sense the activities, voices and other audio data of members in the daily activity scene of the remote family at any time.

Step 1.4, the action behavior collection only obtains the depth data or infrared image of the scene, and does not obtain the visible image data, so as to ensure the privacy of the family members.

Step 1.5, action collection equipment and sound collection equipment can be arranged in relatively public places such as living room, kitchen, entrance hall, home garden, entrance, balcony, etc. according to the requirements of different families to collect data in real time.

Step 2. The motion behavior collection device and the sound collection device transmit the motion behavior data and sound data to the data processing and analysis device. The device processes the data, extracts the characteristic data, and uses its data transmission module in the Internet or 4G in real time. Remote data transmission on /5G network.

Step 3: The data processing and analysis equipment of the local family receives remotely transmitted family member action data and voice data from the Internet or 4G/5G network in real time through its data receiving module.

Step 4. The data processing and analysis equipment identifies the family members to which the action and sound belong, performs three-dimensional modeling of the action data, generates a three-dimensional virtual image of the family member, and performs a three-dimensional virtual presentation at the corresponding location of the local family in real time.

Step 4.1: Collect the three-dimensional information of the family environment and family members (including data such as bones and textures) locally in advance, establish a three-dimensional model, and save the three-dimensional model in the database.

Step 4.1.1, collect environmental data of relatively public places such as living room, kitchen, entrance hall, entrance garden, entrance, balcony, etc., and establish a three-dimensional scene model of the family environment.

Step 4.1.2, collect three-dimensional information data of family members, especially data related to the image of family members, and only save it locally without remote transmission to ensure the privacy of family members.

Step 4.1.3, build a three-dimensional model library of family members based on the three-dimensional information of family members, and also build a small number of virtual guest models of non-family members.

Step 4.2: Once the data processing and analysis equipment receives the action and sound data of the remote family, it first recognizes the data, finds out the family members and the place to which the current action data and voice data belong, and generates the family through three-dimensional modeling Scenes and three-dimensional virtual images of members.

Step 4.2.1, the data processing and analysis equipment obtains the location of the family members from the received data, such as the living room, kitchen, hall, entrance garden, entrance, balcony and other relatively public places, and extracts the corresponding 3D scene model data from the database. Restore the remote family member activity scene.

In step 4.2.2, the data processing and analysis equipment identifies family members from the received data, and obtains the three-dimensional information data of the corresponding members from the three-dimensional model database to establish a three-dimensional virtual image.

Step 4.2.3: The data processing and analysis equipment analyzes the action data of the remote family members, and combines the three-dimensional information of the member to restore the activities of the family members.

In step 4.2.4, the data processing and analysis equipment identifies non-family members, and then selects the three-dimensional information of virtual guests from the three-dimensional model database to restore the activities of the non-family members.

Step 4.3: The data processing and analysis device transmits the generated remote home environment scenes, three-dimensional virtual images of family members and other data to the presentation device for virtual display, realizing an immersive experience.

In step 4.3.1, the presentation device can be a display screen such as a liquid crystal display, or a virtual reality or augmented reality device such as a virtual helmet, a ring screen, a dome screen, and glasses, or a virtual presentation device using holographic technology.

Step 5: The remote and local use the same manner as described above for data interaction, so that both the remote home and the local home appear as a complete family.

The present invention is a remote home perception and virtual presentation method based on natural interaction. The beneficial effect is to address the mental health problems of the elderly, solve the loneliness of the elderly, especially the empty-nest elderly, and solve the problem of excessive emphasis on various service industries in the current market The mental health of the elderly is neglected by the physical health of the elderly. The method invented has a clear structure and reasonable thinking, which can effectively provide technical guidance and solutions for the development of products and solutions for the mental health of the elderly.

Description of the drawings

Fig. 1 is a schematic diagram of an application scenario in an embodiment of the present invention.

Fig. 2 is a schematic diagram of a system configuration example in an embodiment of the present invention.

Detailed ways

In order to make the technical means, creative features, objectives and effects of the present invention easy to understand, the following further describes the present invention with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, not All examples. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work shall fall within the protection scope of the present invention.

Please refer to Figure 1 and Figure 2. The present invention provides a specific embodiment of a remote home perception and virtual presentation method based on natural interaction. The specific process of the entire embodiment includes:

101. Build the corresponding equipment in the demand scenario;

It should be noted that the corresponding equipment in the construction demand scene includes: data processing and analysis equipment, motion behavior presentation equipment, sound presentation equipment, and data transmission module. In this implementation, industry presentation equipment is exemplified but not limited to LCD screens, holographic projection equipment, holographic imaging equipment or unconstrained VR/AR equipment, etc., sound presentation equipment is exemplified but not limited to regional three-dimensional sound simulation equipment, etc., data transmission module Examples but not limited to microcomputers or data routers.

A further solution is:

The demand scene is set as a family house for the elderly. The corresponding equipment in the demand scene can be arranged in the living room, kitchen, hall, entrance garden, entrance, balcony and other places. Each public independent living activity area can be divided into a layout area. One or more collection and presentation equipment groups can be arranged in each relevant area. The relevant technical personnel should reasonably arrange the division of the deployment area, and make detailed plans in accordance with the actual situation and user needs. It should be noted that this embodiment emphasizes the privacy of the user, so it is not recommended to arrange the corresponding equipment in a living activity area with strong privacy such as a bedroom or a bathroom, but this does not represent such actions by related technicians. It does not belong to the substantive category summarized by the implementation of the present invention, and any divided area that belongs to the demand scenario is included in the essence and spirit of the embodiment of the present invention.

102. Build corresponding equipment in the target scene;

It should be noted that the response equipment in the target scene includes: data processing and analysis equipment, motion behavior collection equipment, sound collection equipment, and data transmission module. In this implementation, the action behavior collection device is exemplified but not limited to the character action collection device based on video images, the sound collection device is exemplified but not limited to the regional three-dimensional sound collection device or microphone array, etc., the data transmission module is exemplified but not limited to the microcomputer Or data routers, etc.

A further solution is:

The target scene is set as the family house of the children of the elderly. The action collection equipment and sound collection equipment can be arranged in the living room, kitchen, hall, entrance garden, entrance, balcony and other places. Each public independent living activity area can be divided into Arrangement area, each relevant area can be arranged one or more action collection equipment and sound collection equipment group, the relevant technical personnel should reasonably arrange the division of the layout area, according to the actual situation and user needs to make detailed plans. It should be noted that this embodiment emphasizes the privacy of the user, so it is not recommended to arrange the corresponding equipment in a living activity area with strong privacy such as a bedroom or a bathroom, but this does not represent such actions by related technicians. It does not belong to the substantive category summarized by the implementation of the present invention, and any divided areas that belong to the target scene are included in the essence and spirit of the present invention.

A further solution is:

The embodiment proposes a character action behavior collection device, supporting the character action behavior feature extraction technology, and extracts the action behavior features of the target scene character, especially the bone feature in real time. The data collected by the action behavior collection device comes from the action behavior feature of the target scene character , It does not perform direct video image collection actions for the target scene characters. It should be noted that, based on the consideration of protecting user privacy, the collection based on the action behavior of the characters in the target scene proposed in this embodiment does not limit the image acquisition device or the image acquisition method. Any image acquisition device for the target scene characters Or image collection methods, such as direct video collection action behaviors, etc., are all included in the scope of this embodiment or the content of the invention.

103. Deploy the service platform operating system based on this method;

It should be noted that the deployment service platform operating system includes: scene perception system, action behavior collection system, action behavior presentation system, sound collection system, sound presentation system, data processing and analysis system, and three-dimensional model database system. It needs to be further explained that the above system is only a deployment solution of the service platform operating system proposed in this embodiment, and those skilled in the art should clearly understand that any form of integration, modification and addition of the above system should be carried out. Substantial or innovative solutions are all variations of the solutions given in this embodiment; at the same time, the deployment solution of the service platform operating system given in this embodiment does not limit the solution itself, and those skilled in the art should understand it as Instructive program.

A further solution is:

The action behavior presentation system and the sound presentation system should be deployed in supporting the corresponding equipment in the demand scene of the service platform; further explanation is that the action behavior presentation system should be deployed in the action in the demand scene of the service platform Behavior presentation equipment, and the sound presentation system should be deployed in conjunction with the sound presentation equipment in the required scene of the service platform.

A further solution is:

The action behavior collection system and the sound collection system should be deployed in the corresponding equipment in the target scene of the service platform; a further explanation is that the action collection system should be deployed in the target scene of the service platform. Behavior collection equipment, the sound collection system should be equipped with the sound collection equipment deployed in the target scene of the service platform.

A further solution is:

The data processing and analysis system includes data analysis module, data receiving module, data transmission module and so on.

A further solution is:

The data processing module of the data processing and analysis system should be equipped with the data processing and analysis equipment deployed in the demand scenario of the service platform.

The data receiving module of the data processing and analysis system should be equipped with the data processing and analysis equipment deployed in the demand scenario of the service platform.

A further solution is:

The data sending module of the data processing and analysis system should be equipped with the data processing and analysis equipment deployed in the target scene of the service platform.

104. Perform character modeling for characters in the target scene;

It should be noted that those skilled in the art should perform character modeling work on the target person of the user in need according to actual user needs. This example is an example but not limited to: the user in need is an empty nest elderly in an empty nest family, and the user in need The target characters are relatives such as the children of the empty nest elderly. It needs to be further explained that the solution proposed in this example for character modeling of the target scene character is an example but not limited to a video-based rapid modeling technology method for a true three-dimensional virtual character or a true three-dimensional virtual character based on image features. Drive technology methods, etc.

105. Service platform operation;

It should be noted that this embodiment completes the equipment construction work of the service platform's belonging demand scene and the belonging target scene, completes the service platform's operating system deployment work, and completes the service platform's belonging target scene characters. After the character modeling work, you can enter the operational phase.

The operation process of the service platform is as follows:

The service platform collects and extracts data of actions and sounds of characters in the target scene;

The service platform transmits the action behavior and sound data from the target scene to the demand scene;

The service platform performs data restoration of the actions and sounds of the characters in the target scene in the demand scene.

A further solution is:

The operating system of the service platform can be managed by the user on the switch status and the running time. It should be further explained that the action behavior presentation system and the sound presentation system in the demand scene can be managed by the demanding user on the switch status and running time; The action behavior collection system and sound collection system in the target scene can be managed by the target user on the switch state and running time.

A further solution is:

In the data transmission system of the service platform, the transmission encryption and decryption method is adopted in this embodiment. It should be further explained that the action behavior collection system and the sound collection system in the target scene are matched with data after the data collection work is performed. The processing system, the data sending system, and the data transmission system encrypt the transmission data. The encryption method in this embodiment is an example but not limited to the triple DES algorithm, etc.; the data transmission system, data receiving system, and data processing system in the demand scenario, To decrypt the transmission data, the decryption method in this embodiment is an example but not limited to the triple DES algorithm.

So far, the specific process of this embodiment ends.

Those skilled in the art can clearly understand that, for the convenience of description and brief introduction, the systems, devices, and methods disclosed in the above embodiments can be implemented in other ways. For example, the systems, devices, and methods described above only Is schematic. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-limiting; the embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them. Although the present invention has been described in detail in accordance with the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or equivalently replace some of the technical features; and these modifications Or replacement does not cause the essence of the corresponding technical solution to deviate from the spirit and scope of the technical solution of the present invention. More specifically, within the scope of the disclosure, drawings, and claims of the present application, various modifications and improvements can be made to the component parts and/or the layout of the theme combination layout. For those skilled in the art, other applications will also be used. Is obvious.

Claims

A remote home perception and virtual presentation method based on natural interaction, which is characterized by:

Step 1. Collect and extract the movement behavior and sound data of family members in daily activities in the remote home's daily activity scenes (such as living room, kitchen, entrance hall, home garden, entrance, balcony and other relatively public places) in real time.

Step 2. The motion behavior collection device and the sound collection device transmit the motion behavior data and sound data to the data processing and analysis device. The device processes the data, extracts the characteristic data, and uses its data transmission module in the Internet or 4G in real time. Remote data transmission on /5G network.

Step 3: The data processing and analysis equipment of the local family receives remotely transmitted family member action data and voice data from the Internet or 4G/5G network in real time through its data receiving module.

Step 4. The data processing and analysis equipment identifies the family members to which the action and sound belong, performs three-dimensional modeling of the action data, generates a three-dimensional virtual image of the family member, and performs a three-dimensional virtual presentation at the corresponding location of the local family in real time.

Step 5: The remote and local use the same manner as described above for data interaction, so that both the remote home and the local home appear as a complete family.
A remote home perception and virtual presentation method based on natural interaction according to claim 1, characterized in that: in step 1, the daily activity scenes of remote homes (such as living room, kitchen, entrance hall, home garden, entrance, etc.) In relatively public places such as balconies, real-time collection and extraction of movement behavior and sound data of family members’ daily activities. The step 1 further includes:

Step 1.1, the action collection equipment and the sound collection equipment are processed at any time in the connected working state. Once the activities or sound signals of the members in the daily activity scene of the family are captured, the data collection is started immediately, through binocular or multi-eye vision technology and audio technology , Collect family members' activities and behaviors in real time to form bone data and scene depth data, extract movement behavior characteristic data through analysis, and collect voice or other sound data at the same time.

Step 1.2, the use of motion behavior data and voice and other audio data does not require manual operations such as traditional dial-up or network connections. Family members can move and talk freely, and the collection equipment can sense data changes and collect data in real time to realize family members Natural interaction.

Step 1.3, the acquisition device adopts infrared or other imaging technology and audio capture technology to sense the activities, voices and other audio data of members in the daily activity scene of the remote family at any time.

Step 1.4, the action behavior collection only obtains the depth data or infrared image of the scene, and does not obtain the visible image data, so as to ensure the privacy of the family members.

Step 1.5, action collection equipment and sound collection equipment can be arranged in relatively public places such as living room, kitchen, entrance hall, home garden, entrance, balcony, etc. according to the requirements of different families to collect data in real time.
The method of remote home perception and virtual presentation based on natural interaction according to claim 1, characterized in that: in step 4, the data processing and analysis equipment recognizes the action behavior and the family members to which the sound belongs, and performs three-dimensional analysis on the action behavior data. Modeling, generating a three-dimensional virtual image of family members, and real-time three-dimensional virtual presentation in the corresponding location of the local family. The step 4 further includes:

Step 4.1: Collect the three-dimensional information of the family environment and family members (including data such as bones and textures) locally in advance, build a three-dimensional model, and save the three-dimensional model in the database.

Step 4.1.1, collect environmental data of relatively public places such as living room, kitchen, entrance hall, entrance garden, entrance, balcony, etc., and establish a three-dimensional scene model of the family environment.

Step 4.1.2, collect the three-dimensional information data of the family members, especially the data related to the image of the family members, and only save it locally without remote transmission, so as to ensure the privacy of the family members.

Step 4.1.3, build a three-dimensional model library of family members based on the three-dimensional information of family members, and also build a small number of virtual guest models of non-family members.

Step 4.2: Once the data processing and analysis device receives the action and sound data of the remote family, it first recognizes the data, finds out the family members and places where the current action data and voice data belong, and generates the family through three-dimensional modeling Scenes and three-dimensional virtual images of members.

Step 4.2.1, the data processing and analysis equipment obtains the location of the family members from the received data, such as the living room, kitchen, hall, entrance garden, entrance, balcony and other relatively public places, and extracts the corresponding 3D scene model data from the database, Restore the remote family member activity scene.

Step 4.2.2, the data processing and analysis equipment identifies family members from the received data, and obtains the three-dimensional information data of the corresponding members from the three-dimensional model database to establish a three-dimensional virtual image.

Step 4.2.3: The data processing and analysis equipment analyzes the action data of the remote family members, and combines the three-dimensional information of the member to restore the activities of the family members.

In step 4.2.4, the data processing and analysis equipment identifies non-family members, and then selects the three-dimensional information of virtual guests from the three-dimensional model database to restore the activities of the non-family members.

Step 4.3: The data processing and analysis device transmits the generated remote home environment scenes, three-dimensional virtual images of family members and other data to the presentation device for virtual display, realizing an immersive experience.

Step 4.3.1, the presentation device can be a display screen such as a liquid crystal display, or a virtual reality or augmented reality device such as a virtual helmet, a ring screen, a dome screen, and glasses, or a virtual presentation device using holographic technology.
The remote home perception and virtual presentation method based on natural interaction according to claim 1, 2, 3, characterized in that:

The action behavior presentation system and the sound presentation system in the demand scene can be managed by the demand user on the switch state and the running time.

The action behavior collection system and the sound collection system in the target scene can be managed by the target user on the switch state and running time.