CN115291718A - Man-machine interaction system in smart home space and application method thereof - Google Patents

Abstract

The invention relates to a human-computer interaction system in a smart home space and an application method thereof. The system includes a multi-channel sensor network, an intelligent perception module and a human-computer interaction module deployed in the smart home space; the multi-channel sensor network is used for intelligent The home space is used for multi-modal information collection; the intelligent perception module is used to intelligently perceive the user according to the collected multi-modal information; the human-computer interaction module is used to realize the multi-modal communication between various intelligent terminal devices and users based on the user's intelligent perception results Human-computer interaction; among them, intelligent perception includes at least behavior perception and emotion perception. The invention uses multi-channel sensor network to collect multi-source heterogeneous data in the smart home environment, and uses multi-modal fusion analysis method to accurately perceive the user's natural behavior and emotional state from the multi-source heterogeneous data; on this basis, the smart home is realized Multi-modal human-computer interaction in space, thereby providing users with a more intelligent home environment.

Description

A human-computer interaction system and its application method in smart home space

technical field

The present invention relates to the technical field of smart home, in particular to a human-computer interaction system in a smart home space and an application method thereof.

Background technique

Smart home is to connect various household devices in the home through the Internet of Things technology to provide intelligent networking control of household devices. Compared with ordinary homes, smart homes not only have traditional living functions, but also have the characteristics of information sharing and automation of home equipment, which can create a high-quality living environment for families.

However, the form of the existing smart home is limited by its technical means and industrial structure. It can only be a combination of independent home smart products, lacking a systematic, global, and unified control of smart device interconnection solutions and key technologies. . This kind of smart home can only be called a "single-product smart" home, not an "overall smart" home. It lacks a deep understanding of users during human-computer interaction and cannot achieve true intelligent perception.

Therefore, the present invention is committed to breaking the existing island model of smart products, transforming from single "smart" to overall "smart", and constructing a complete set of user-centered intelligent, personalized and humanized smart home products.

Contents of the invention

The purpose of the present invention is to provide a human-computer interaction system and its application method in the smart home space, to solve the problem that the existing smart home lacks a deep understanding of the user during human-computer interaction, and cannot realize real intelligent perception, and provide users with Bring a smart, comfortable, warm and convenient personalized home experience that integrates home environment and home entities.

In a first aspect, the present invention provides a human-computer interaction system in a smart home space, the system comprising:

The multi-channel sensor network is used to collect multi-modal information on the smart home space;

The intelligent perception module is configured to perform intelligent perception on users in the smart home space according to the collected multimodal information;

The human-computer interaction module is configured to realize multimodal human-computer interaction between various intelligent terminal devices in the smart home space and the user based on the user's intelligent perception result;

Wherein, the intelligent perception includes at least behavior perception and emotion perception.

According to the human-computer interaction system under the smart home space provided by the present invention, the multimodal information includes image information, voice information, tactile perception information, smell information and environmental information; the multi-channel sensor network includes: visual channel, auditory channels, tactile channels, olfactory channels and environmental information channels;

The visual channel includes a visual sensor for collecting the image information using the visual sensor;

The auditory channel includes an auditory sensor for collecting the voice information by using the auditory sensor;

The tactile channel includes a tactile sensor for collecting the tactile perception information by using the tactile sensor;

The olfactory channel includes an olfactory sensor for collecting the odor information by using the olfactory sensor;

The environment information channel includes an environment detection sensor for collecting the environment information by using the environment detection sensor.

According to the human-computer interaction system under the smart home space provided by the present invention, the visual sensor is deployed on the roof of the smart home space, and at least includes the following one:

RGB camera matrix, 360-degree panoramic camera, infrared camera and depth camera;

The tactile sensor includes at least one of the following:

Capacitive floor sensors, pressure sensors arranged in smart sofas and/or smart beds;

The auditory sensor includes: a microphone array;

The environmental information sensor includes at least one of the following:

Temperature sensor, humidity sensor, carbon dioxide sensor and air quality sensor arranged in the smart bed;

Wherein, the capacitance value of each floor patch in the capacitive floor sensor is determined by the pressure it bears.

According to the human-computer interaction system under the smart home space provided by the present invention, the multi-channel sensor network further includes: a trigger device;

The triggering device is configured to trigger the visual channel, the auditory channel, the tactile channel, the olfactory channel and the environmental information channel at a certain frequency to perform synchronous collection of information.

According to the human-computer interaction system under the smart home space provided by the present invention, the intelligent perception module includes: a preprocessing unit, a multimodal feature extraction unit and a multimodal feature fusion analysis unit;

The preprocessing unit is configured to preprocess the image information, the voice information, the tactile perception information, the smell information and the environmental information;

The multimodal feature extraction unit is used to perform feature extraction on the preprocessed image information, voice information, tactile perception information, odor information and environmental information to obtain multimodal feature information;

The multimodal feature fusion analysis unit is configured to perform fusion analysis on the multimodal feature information to determine the user's intelligent perception result.

According to the human-computer interaction system under the smart home space provided by the present invention, the preprocessing of the image information includes at least one of the following:

performing data alignment on the image information; performing de-redundancy on the image information; denoising the image information;

Preprocessing the voice information includes at least one of the following:

performing background sound and human voice separation on the voice information; removing noise in the voice information; removing blank tones in the voice information;

Preprocessing the tactile perception information includes at least one of the following:

Correcting the obviously wrong data in the tactile perception information;

Completing missing data in the tactile perception information;

Preprocessing the odor information includes at least one of the following:

removing the identification failure information contained in the odor information;

removing information that obviously deviates from the user's behavior habits in the smell information;

Preprocessing the environmental information includes at least one of the following:

Complete missing data in the environmental information;

The interpolation method is used to supplement the data lost due to the collection frequency in the environmental information.

According to the human-computer interaction system under the smart home space provided by the present invention, the multimodal feature extraction unit includes: a posture feature extraction subunit, a face feature extraction subunit, a thermal imaging feature extraction subunit, and a depth feature extraction subunit. Unit, Semantic Feature Extraction Subunit, Pose Feature Extraction Subunit, User Sleep Quality Feature Extraction Subunit, User Motion Trajectory Feature Extraction Subunit, User Behavior Event Feature Extraction Subunit, Temperature Feature Extraction Subunit, Humidity Feature Extraction Subunit and air quality feature extraction subunit;

Wherein, the posture feature extraction subunit is used to use the preprocessed data of the RGB camera matrix, the 360-degree panoramic camera, the infrared camera in a dark environment and/or the image information collected by the depth camera as the first a target data, and extract gesture features from the first target data based on a gesture recognition algorithm;

The human face feature extraction subunit is used to use the preprocessed data of the image information collected by the RGB camera matrix and the 360-degree panoramic camera as the second target data, and extract from the second target data based on the face recognition algorithm. Extract facial features from the data;

The thermal imaging feature extraction subunit is used to use the preprocessed data of the image information collected by the infrared camera as the third target data, and extract the thermal imaging features of the human body and the thermal imaging of the object from the third target data feature;

The depth feature extraction subunit is used to use the preprocessed data of the image information collected by the depth camera as the fourth target data, and use a deep learning algorithm or a three-dimensional point cloud feature point extraction algorithm to extract the data from the fourth target Extract human body depth features, spatial depth features and object depth features from the data;

The semantic feature extraction subunit is configured to use the preprocessed data of the voice information collected by the microphone array as the fifth target data, and use a voice recognition algorithm and/or a voice emotion analysis algorithm to extract the data from the fifth target Extract semantic features from the data;

The pose feature extraction subunit is used to use the preprocessed data of the pressure value information collected by the pressure sensor arranged in the smart sofa and/or smart bed as the sixth target data, and analyze the sixth target data. The target data obtains the pose characteristics of the user on the smart sofa and/or smart bed;

The user movement trajectory feature extraction subunit is used to use the preprocessed data of the capacitance value information collected by the capacitive floor sensor as the seventh target data, and analyze the seventh target data to obtain the user movement trajectory characteristics; The characteristics of the user's motion trajectory include: position, gait, direction and speed;

The user behavior event feature extraction subunit is used to use the preprocessed data of the smell information collected by the olfactory sensor as the eighth target data, and analyze the eighth target data to obtain the user behavior event features;

The user sleep quality feature sub-extraction unit is used to use the preprocessed data of the carbon dioxide value information collected by the carbon dioxide sensor arranged in the smart bed as the ninth target data, and analyze the ninth target data to obtain the user sleep quality characteristics;

The temperature feature extraction subunit is configured to use the preprocessed data of temperature information collected by the temperature sensor as the tenth target data, and analyze the tenth target data to obtain a temperature feature;

The humidity feature extraction subunit is configured to use the preprocessed data of the humidity information collected by the humidity sensor as the eleventh target data, and analyze the eleventh target data to obtain a humidity feature;

The air quality feature extraction subunit is configured to use the preprocessed data of the air quality information collected by the air quality sensor as the twelfth target data, and analyze the twelfth target data to obtain the air quality feature.

According to the human-computer interaction system under the smart home space provided by the present invention, the multimodal feature fusion analysis unit includes: a data storage unit, an intelligent perception model construction unit and an intelligent perception execution unit;

The data storage unit is used to store each modal feature in the multi-modal feature in each modal feature database;

The intelligent perception model construction unit is used to train the intelligent perception model based on each modal feature using each modal feature database in the training phase; and the intelligent perception model based on each modal feature is model-fused to obtain the intelligent perception model ; Also used to update the intellisense model using the multi-modal feature in the application phase;

The intelligent perception executing unit is configured to input the multimodal features into the intelligent perception model, and obtain the user's intelligent perception result output by the intelligent perception model.

According to the human-computer interaction system in the smart home space provided by the present invention, the model fusion methods include but not limited to: Voting, Boosting, Bootstrap Aggregating and multi-layer fusion.

In a second aspect, the present invention provides an application method of a human-computer interaction system in a smart home space, the method comprising:

The multi-channel sensor network collects multi-modal information on the smart home space;

The intelligent perception module performs intelligent perception on users in the smart home space according to the collected multimodal information;

The human-computer interaction module realizes multimodal human-computer interaction between various intelligent terminal devices in the smart home space and the user based on the user's intelligent perception results;

Wherein, the intelligent perception includes at least behavior perception and emotion perception.

The present invention provides a human-computer interaction system and its application method in the smart home space, which utilizes the multi-channel sensor network deployed in the smart home space to collect multi-source heterogeneous data in the user smart home environment; based on the multi-source heterogeneous data And use the intelligent perception module to perceive the user's natural behavior and emotional state, and realize the deep understanding of the user; realize the multi-modal human interaction between various intelligent terminal devices and users in the smart home environment according to the perception results of the user's natural behavior and emotional state. Computer interaction, bringing users a personalized home experience that integrates home environment and home entities with wisdom, comfort, warmth and convenience.

Description of drawings

In order to more clearly illustrate the present invention or the technical solutions in the prior art, the accompanying drawings that need to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings in the following description are the For some embodiments of the present invention, those of ordinary skill in the art can also obtain other drawings based on these drawings on the premise of not paying creative efforts.

Fig. 1 is a schematic structural diagram of a human-computer interaction system under a smart home space provided by the present invention;

Fig. 2 is a flow chart of the application method of the human-computer interaction system in the smart home space provided by the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention , but not all examples. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

A human-computer interaction system and its application method in the smart home space provided by the present invention will be described below with reference to FIG. 1 and FIG. 2 .

First, at the level of scientific research and exploration, smart living is a comprehensive research direction with a high degree of cross-discipline, and a scientific research field highly related to human activities, involving human-computer interaction, artificial intelligence, computer vision, Internet of Things, user experience, psychology The comprehensive cross-integration of disciplines and fields, and its related theories and technological innovations play an important role in deeply understanding human behavior patterns and emotional communication methods, which is in line with the core appeal of artificial intelligence "people-oriented".

At present, the human-computer interaction methods of smart homes are generally based on a single smart product and its application scenarios, which lack universality and integrity. At the same time, it usually aims to improve the ease of learning, ease of use, and user stickiness of a single smart product, often ignoring the user's behavioral intentions and emotional experience, and failing to achieve the desired effect.

Based on this, the present invention provides a human-computer interaction system in a smart home space, which is a multi-modal human-computer interaction system, which is more intelligent, more accurate, and more natural in terms of human-computer interaction. As shown in Figure 1, the system includes: a multi-channel sensor network 110, an intelligent perception module 120 and a human-computer interaction module 130 deployed in a smart home space;

Wherein, the multi-channel sensor network 110 can be used to collect multi-modal information in the smart home space.

Specifically, the multi-channel sensor network 110 is composed of sensors deployed at various positions in the smart home space, because different types of sensors are used. Therefore, the multimodal information collected is actually multi-source heterogeneous data, that is, data from multiple sources with different data formats, including but not limited to video data, picture data, voice data, gait data, smell data, temperature data , light data, humidity data, etc.

It should be noted that for smart home users, it is difficult to deploy a complete and complete set of sensor networks in the home environment for standardized and large-scale data collection. Moreover, at present, the houses of smart home users are mostly single-level smart home systems, which mainly realize the simple linkage of single product functions or smart home systems, and cannot form a large-scale application of a full set of level systems, and the collected data is prone to inaccuracy or unavailability Case. Therefore, it is necessary to design a sensor deployment scheme in the multi-channel sensor network 110 through the smart home experimental research platform to achieve optimal control of smart home devices, thereby improving the usability and practicability of smart home devices in the smart home environment.

The intelligent perception module 120 can be used to perform intelligent perception on users in the smart home space according to the collected multimodal information; the intelligent perception includes at least behavior perception and emotion perception;

Specifically, the concept of intelligent perception is to obtain user data through various sensor networks, use big data, deep learning, computer vision and other technologies to imitate the cognitive mechanism of humans and animals, and complete the process of object feature extraction and intelligent reasoning. In the present invention, in addition to behavior perception and emotion perception, intelligent perception may also include event perception and environment perception. Event perception is mainly aimed at the event correlation between people and surrounding objects. Environmental perception mainly refers to the perception of the environment created by smart home space objects.

The intelligent perception module 120 desensitizes, cleans, marks, extracts features, and fuses the multi-modal information to obtain multi-modal feature information that characterizes the user's natural behavior and emotional state in the smart home environment. Then, with the support of multi-modal feature information, it can deeply understand the user's natural behavior and emotional state.

The human-computer interaction module 130 is configured to realize multimodal human-computer interaction between various intelligent terminal devices in the smart home space and the user based on the user's intelligent perception result;

It should be noted that in the prototype space of the smart home, the present invention uses technologies such as AIoT and Zigbee to realize the intercommunication and data sharing of different smart terminal devices.

It is understandable that with the official licensing of 5G technology and the arrival of the era of smart Internet of Things, AI+IoT (ie AIoT) has become the main track of major industry giants, and smart home has officially entered the 3.0 era. It is about to build a cloud Internet platform to realize real intelligent interaction, intelligent perception and complex decision-making. In the era of smart home 3.0, data is the core and interaction is the key. As a smart product in the era of 3.0, the human-computer interaction system in the smart home space of the present invention widely relies on user big data for in-depth learning and data mining, accurately portrays user portraits, and provides intelligent and personalized services. At the same time, it will also provide multi-modal human-computer interaction based on emotional computing, speech recognition, computer vision, knowledge graph, natural language processing and other methods, realizing the deep integration of "human-machine-object".

The present invention provides a human-computer interaction system in the smart home space, which uses the multi-channel sensor network 110 deployed in the smart home space to collect multi-source heterogeneous data in the user's smart home environment; Perception module 120, which perceives the user's natural behavior and emotional state, and realizes a deep understanding of the user; realizes multi-modal human-computer interaction between various smart terminal devices and users in the smart home environment based on the perception results of the user's natural behavior and emotional state , to bring users a smart, comfortable, warm and convenient personalized home experience that integrates home environment and home entities.

On the basis of the above embodiments, as an optional embodiment, the multimodal information includes image information, voice information, tactile perception information, odor information and environmental information; the multi-channel sensor network 110 includes: Visual channels, auditory channels, tactile channels, olfactory channels and environmental information channels;

The visual channel includes a visual sensor for collecting the image information using the visual sensor;

The auditory channel includes an auditory sensor for collecting the voice information by using the auditory sensor;

The tactile channel includes a tactile sensor for collecting the tactile perception information by using the tactile sensor;

The olfactory channel includes an olfactory sensor for collecting the odor information by using the olfactory sensor;

The environment information channel includes an environment detection sensor for collecting the environment information by using the environment detection sensor.

The present invention is oriented towards multi-modal natural human-computer interaction research for smart homes, and constructs a multi-channel sensing layer with vision, hearing, touch, and smell as the main modes and environmental information as the auxiliary mode. Build an enriched sensing space including RGB camera, high-speed camera, infrared camera, depth camera, microphone array, tactile sensor, gait sensing floor, olfactory sensor, environmental information sensor and other hardware at the infrastructure node of the smart home space. The collection of multi-source heterogeneous data provides the basis for realization.

On the basis of the above embodiments, as an optional embodiment, the visual sensor is deployed on the roof of the smart home space, at least including the following one:

RGB camera matrix, 360-degree panoramic camera, infrared camera and depth camera;

The tactile sensor includes at least one of the following:

Capacitive floor sensors, pressure sensors arranged in smart sofas and/or smart beds;

The auditory sensor includes: a microphone array;

The environmental information sensor includes at least one of the following:

Temperature sensor, humidity sensor, carbon dioxide sensor and air quality sensor arranged in the smart bed;

Wherein, the capacitance value of each floor patch in the capacitive floor sensor is determined by the pressure it bears.

Specifically, air quality sensors can be divided into multiple types in terms of functions, including air quality sensors that detect PM2.5, air quality sensors that detect PM10, air quality sensors that detect sulfur dioxide, air quality sensors that detect carbon monoxide, and other air quality sensors. ;

All the cameras arranged in the smart home space in the visual channel form a multi-camera matrix; the RGB camera matrix refers to a camera matrix composed of a single RGB camera with a field of view less than 360°, and the human body can be extracted from the captured image or video stream. Posture, action, facial data, spatial location information, etc. A 360-degree panoramic camera refers to a 360-degree RGB camera, and data such as human posture, movement, and face can also be extracted from images or video streams captured by it. Nighttime data, thermal imaging data, etc. can be extracted from images or video streams captured by infrared cameras; depth data of people, space, and objects can be extracted from images or video streams captured by depth cameras;

The auditory channel mainly consists of microphones, such as multi-directional microphone arrays; dialogue data can be extracted from multi-directional microphone arrays. In addition, due to the different distances and positions between a single microphone and the sound source, the waveform and spectrum information of the sound signal of the same sound are different in different microphones. Therefore, the spatial feature information and the like in the audio collected by the microphone array can be obtained through this distinction.

The tactile channel is mainly composed of self-developed smart homes; for example, the whole house capacitive floor, smart sofas and smart beds with pressure sensors; the user's position, gait, and direction can be analyzed from the capacitance value of each floor in the whole house capacitive floor , speed and other user trajectory information, which can realize the drawing of heat map of activity area, recording of activity information, multi-person identification and fall alarm. From the pressure information sensed by the smart sofa and smart bed, the user's position, sitting posture, sleep state, desktop activity, etc. can be analyzed.

The olfactory channel is mainly composed of olfactory sensors; indoor user activities can be identified from the smell information perceived by the olfactory sensors;

The environmental information channel is mainly composed of various environmental information sensors; the environmental data can be obtained from the environmental information sensed by the environmental information sensors, and then the indoor environmental status can be identified.

Examples of visual sensors, auditory sensors, and tactile sensors are given in the present invention, providing directions for the selection of sensors in the multi-channel sensor network 110 . In addition, the examples provided by the present invention do not limit the selection of visual sensors, auditory sensors, tactile sensors, and olfactory sensors in the multi-channel sensor network 110 .

On the basis of the foregoing embodiments, as an optional embodiment, the multi-channel sensor network 110 further includes: a trigger device;

The triggering device is configured to trigger the visual channel, the auditory channel, the tactile channel, the olfactory channel and the environmental information channel at a certain frequency to perform synchronous collection of information.

Here, the synchronous acquisition covers the synchronization of acquisition start and stop time and acquisition frequency.

In order to facilitate the fusion processing of multi-modal feature information, the present invention sets up an integrated control system of multi-channel sensors, that is, a trigger device, to realize synchronous collection of multi-source heterogeneous data.

On the basis of the above embodiments, as an optional embodiment, the intelligent perception module 120 includes: a preprocessing unit, a multimodal feature extraction unit and a multimodal feature fusion analysis unit;

The preprocessing unit is configured to preprocess the image information, the voice information, the tactile perception information, the smell information and the environmental information;

The multimodal feature extraction unit is used to perform feature extraction on the preprocessed image information, voice information, tactile perception information, odor information and environmental information to obtain multimodal feature information;

The multimodal feature fusion analysis unit is configured to perform fusion analysis on the multimodal feature information to determine the user's intelligent perception result.

In the present invention, the image information, the voice information, the tactile perception information, the smell information and the environmental information collected cannot be used directly, and a series of preprocessing is required before they can be put into use; exemplary It is necessary to desensitize the image information, the voice information, the tactile perception information, the smell information and the environmental information, so as not to leak the user’s personal information; it is also necessary to desensitize the image information, the voice information Information, the tactile perception information, the odor information and the environmental information are cleaned, normalized, etc., to remove messy and useless information.

After preprocessing, it is necessary to perform feature fusion analysis on multimodal information to achieve deep intelligent perception of users. Among them, there are two ways to perform feature fusion analysis on multimodal information:

The first one is to combine multi-channel data (that is, multi-modal information, each channel source represents a modality), and use interpolation methods (nearest neighbor interpolation, linear interpolation, multiple interpolation, etc.) for small-scale data and large-scale data Alignment; then input the merged multi-channel data into the intelligent perception model to obtain the user fusion analysis results. In this manner, the intellisense model is trained using sample data with time stamps and intellisense labels, where the sample data includes: data obtained by merging past multi-channel data.

The second is to extract the features of the multi-modal information separately to obtain the multi-modal feature information; input the multi-modal feature information into the intelligent perception model to obtain the user fusion analysis results.

In this way, the intelligent perception model is obtained by fusing the intelligent perception models in each mode; the intelligent perception model in each mode is trained by using the feature information data of the previous models with the intelligent perception label . Relatively speaking, the Intellisense effect of the second method is better than that of the first method, so the following mainly describes the second method.

The present invention integrates auditory, visual, tactile, olfactory modalities (main modal) and environmental information (auxiliary modal), constructs a coupling model of multi-component information and multi-modal interaction, and then intelligently perceives users in all aspects, providing intelligent Lay the foundation for natural human-computer interaction in the home scene.

On the basis of the foregoing embodiments, as an optional embodiment, preprocessing the image information includes at least one of the following:

performing data alignment on the image information; performing de-redundancy on the image information; denoising the image information;

Preprocessing the voice information includes at least one of the following:

performing background sound and human voice separation on the voice information; removing noise in the voice information; removing blank tones in the voice information;

Preprocessing the tactile perception information includes at least one of the following:

Correcting the obviously wrong data in the tactile perception information;

Completing missing data in the tactile perception information;

Preprocessing the odor information includes at least one of the following:

removing the identification failure information contained in the odor information;

removing information that obviously deviates from the user's behavior habits in the smell information;

Preprocessing the environmental information includes at least one of the following:

Complete missing data in the environmental information;

The interpolation method is used to supplement the data lost due to the collection frequency in the environmental information.

In the present invention, the tactile perception information is generally numerical information such as pressure value and capacitance value, so there will be missing and obviously wrong data during the collection process, so the corresponding preprocessing measures are incomplete and correct data and correction of obviously wrong data. The identification failure information does not identify the information of the smell, that is, the identification of the smell sensor fails. Odor information that deviates significantly from the user's behavior habits, for example, the user dislikes the smell of durian, and if the smell sensor recognizes the smell of durian, there is a high probability of misidentification. Therefore, the corresponding pretreatment measure is removal.

It can be understood that the present invention only exemplifies the preprocessing methods of image information, voice information, tactile perception information, odor information and environmental information, but in fact, image information, audio information, tactile perception information, odor information and environmental information There are countless means of preprocessing information, so it can be adaptively adjusted.

The present invention simplifies effective features in multivariate information through a preprocessing method, laying a foundation for subsequent feature extraction.

On the basis of the above embodiments, as an optional embodiment, the multi-modal feature extraction unit includes: gesture feature extraction subunit, face feature extraction subunit, thermal imaging feature extraction subunit, depth Feature extraction subunit, semantic feature extraction subunit, pose feature extraction subunit, user sleep quality feature extraction subunit, user motion trajectory feature extraction subunit, user behavior event feature extraction subunit, temperature feature extraction subunit, humidity feature extraction subunit and air quality feature extraction subunit;

Wherein, the posture feature extraction subunit is used to preprocess the image information collected by the RGB camera matrix, the 360-degree panoramic camera, the infrared camera in a dark environment, and/or the depth camera as the first target data, and extract gesture features from the first target data based on a gesture recognition algorithm;

The face feature extraction subunit is used to use the preprocessed data of the RGB camera matrix and the image information collected by the 360-degree panoramic camera as the second target data, and obtain the second target data based on the face recognition algorithm. Extract facial features from the target data;

The thermal imaging feature extraction subunit is used to use the preprocessed data of the image information collected by the infrared camera as the third target data, and extract the thermal imaging features of the human body and the thermal imaging of the object from the third target data feature;

The depth feature extraction subunit is used to use the preprocessed data of the image information collected by the depth camera as the fourth target data, and use a deep learning algorithm or a three-dimensional point cloud feature point extraction algorithm to extract the data from the fourth target Extract human body depth features, spatial depth features and object depth features from the data;

The visual channel uses a multi-camera array to collect image information, and the collected image information is stored in the form of time stamps and pictures/videos. After the picture/video data is collected, operations such as image recognition and key bone point recognition of the human body will be performed on the single-frame data to extract from the single-frame image. Features include but are not limited to: object information, spatial distribution information, etc.; Multi-frame key bone point information of the human body is used for feature extraction. The extracted features include but not limited to: body movements, movement acceleration and displacement, etc.

The semantic feature extraction subunit is configured to use the preprocessed data of the voice information collected by the microphone array as the fifth target data, and use a voice recognition algorithm and/or a voice emotion analysis algorithm to extract the data from the fifth target Extract semantic features from the data;

The auditory channel uses a microphone array to collect audio information at a certain sampling rate, and the audio information is stored in units of time stamps. Feature extraction of audio information, the method of feature extraction includes but not limited to Fourier transform, spectrogram, Nyquist sampling, extraction of zero-crossing rate, spectral center, spectral roll-off point, Mel frequency cepstral coefficient (MFCC )Wait.

The pose feature extraction subunit is used to use the preprocessed data of the pressure value information collected by the pressure sensor arranged in the smart sofa and/or smart bed as the sixth target data, and analyze the sixth target data. The target data obtains the pose characteristics of the user on the smart sofa and/or smart bed;

Furthermore, dimensional features such as the user's sleep health can also be derived from the user's sleep quality feature.

The user movement trajectory feature extraction subunit is used to use the preprocessed data of the capacitance value information collected by the capacitive floor sensor as the seventh target data, and analyze the seventh target data to obtain the user movement trajectory characteristics; The characteristics of the user's motion trajectory include: position, gait, direction and speed;

The user behavior event feature extraction subunit is configured to use the preprocessed data of the smell information collected by the olfactory sensor as the eighth target data, and analyze the eighth target data to obtain the user behavior event features.

Specifically, user behavior event features refer to the characteristics of human daily life behaviors that are happening in different spaces.

The user sleep quality feature sub-extraction unit is used to use the preprocessed data of the carbon dioxide value information collected by the carbon dioxide sensor arranged in the smart bed as the ninth target data, and analyze the ninth target data to obtain the user sleep quality characteristics;

The temperature feature extraction subunit is configured to use the preprocessed data of temperature information collected by the temperature sensor as the tenth target data, and analyze the tenth target data to obtain a temperature feature;

The humidity feature extraction subunit is configured to use the preprocessed data of the humidity information collected by the humidity sensor as the eleventh target data, and analyze the eleventh target data to obtain a humidity feature;

The air quality feature extraction subunit is configured to use the preprocessed data of the air quality information collected by the air quality sensor as the twelfth target data, and analyze the twelfth target data to obtain the air quality feature.

The invention provides the feature extraction mode of data collected by multiple channels, and lays the foundation for the analysis and learning of multi-modal feature data.

On the basis of the above embodiments, as an optional embodiment, the multi-modal feature fusion analysis unit includes: a data storage unit, an intelligent perception model construction unit, and an intelligent perception execution unit;

The data storage unit is used to store each modal feature in the multi-modal feature in each modal feature database;

Specifically, the intelligent perception module 120 extracts multimodal feature information representing the user's natural behavior and emotional state in the smart home environment from the multimodal information. Since the multi-modal information collected by the multi-channel sensor network 110 has the characteristics of multi-dimensionality, low value density, and fast generation speed, it is possible to build a multi-modal feature database in a short period of time, providing a basis for multi-modal human-computer interaction based on intelligent perception. data support.

The intelligent perception model construction unit is used to train the intelligent perception model based on each modal feature using each modal feature database in the training phase; and the intelligent perception model based on each modal feature is model-fused to obtain the intelligent perception model ; Also used to update the intellisense model using the multi-modal feature in the application phase;

The intelligent perception executing unit is configured to input the multimodal features into the intelligent perception model, and obtain the user's intelligent perception result output by the intelligent perception model.

In the present invention, the feature data of various channels are input into the multi-modal multi-target recognition algorithm model for smart home, and through the analysis and learning of multi-modal feature data, user behavior, user emotion, occurrence events, Environmental status, etc., are used to guide various smart terminal devices (such as home servers, smart home appliances, etc.) in the smart home environment to carry out multi-modal human-computer interaction with users.

On the basis of the foregoing embodiments, as an optional embodiment, the model fusion manner includes, but is not limited to: Voting, Boosting, Bootstrap Aggregating and multi-layer fusion.

The present invention provides the selection direction of the model fusion mode, and does not limit the selection range of the model fusion mode.

In a second aspect, the present invention provides an application method of a human-computer interaction system in a smart home space, as shown in FIG. 2 , the method includes:

S11. The multi-channel sensor network collects multi-modal information on the smart home space;

S12. The intelligent perception module performs intelligent perception on users in the smart home space according to the collected multimodal information;

S13. The human-computer interaction module implements multimodal human-computer interaction between various intelligent terminal devices in the smart home space and the user based on the user's intelligent perception result;

Wherein, the intelligent perception includes at least behavior perception and emotion perception.

The invention provides an application method of a human-computer interaction system in a smart home space, which uses a multi-channel sensor network deployed in a smart home space to collect multi-source heterogeneous data in a user's smart home environment; based on multi-source heterogeneous data and Use the intelligent perception module to perceive the user's natural behavior and emotional state, and realize a deep understanding of the user; realize the multi-modal human-machine interaction between various smart terminal devices and users in the smart home environment based on the perception results of the user's natural behavior and emotional state Interaction brings users a personalized home experience that integrates home environment and home entities with wisdom, comfort, warmth and convenience.

The device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in One place, or it can be distributed to multiple network elements. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. It can be understood and implemented by those skilled in the art without any creative efforts.

Through the above description of the implementations, those skilled in the art can clearly understand that each implementation can be implemented by means of software plus a necessary general-purpose hardware platform, and of course also by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the prior art can be embodied in the form of software products, and the computer software products can be stored in computer-readable storage media, such as ROM/RAM, magnetic discs, optical discs, etc., including several instructions to make a computer device (which may be a personal computer, server, or network device, etc.) execute the methods described in various embodiments or some parts of the embodiments.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still be Modifications are made to the technical solutions described in the foregoing embodiments, or equivalent replacements are made to some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims (10)

1. A man-machine interaction system in an intelligent home space is characterized by comprising a multi-channel sensor network, an intelligent sensing module and a man-machine interaction module, wherein the multi-channel sensor network, the intelligent sensing module and the man-machine interaction module are deployed in the intelligent home space;

the multi-channel sensor network is used for performing multi-mode information acquisition on the intelligent home space;

the intelligent sensing module is used for intelligently sensing the user in the intelligent home space according to the collected multi-mode information;

the man-machine interaction module is used for realizing multi-mode man-machine interaction between various intelligent terminal devices in the intelligent home space and the user based on the intelligent perception result of the user;

wherein the intelligent perception comprises at least behavioral perception and emotional perception.

2. The human-computer interaction system under the intelligent home space according to claim 1, wherein the multi-modal information comprises image information, voice information, tactile perception information, smell information and environment information; the multichannel sensor network includes: a visual channel, an auditory channel, a tactile channel, an olfactory channel, and an environmental information channel;

the visual channel comprises a visual sensor and is used for acquiring the image information by utilizing the visual sensor;

the auditory channel comprises an auditory sensor and is used for collecting the voice information by the auditory sensor;

the tactile channel comprises a tactile sensor and is used for collecting the tactile perception information by the tactile sensor;

the smell channel comprises a smell sensor and is used for collecting the smell information by utilizing the smell sensor;

the environment information channel comprises an environment detection sensor and is used for collecting the environment information by utilizing the environment detection sensor.

3. The human-computer interaction system under the smart home space of claim 2, wherein the visual sensor is deployed on a roof of the smart home space and comprises at least one of:

an RGB camera matrix, a 360-degree panoramic camera, an infrared camera, and a depth camera;

the tactile sensor comprises at least one of:

a capacitive floor sensor, a pressure sensor arranged in the smart sofa and/or the smart bed;

the auditory sensor includes: a microphone array;

the environmental information sensor includes at least one of:

the intelligent bed comprises a temperature sensor, a humidity sensor, a carbon dioxide sensor arranged in the intelligent bed and an air quality sensor;

wherein, the capacitance value of each floor patch in the capacitance floor sensor is determined by the pressure born by the floor patch.

4. The system according to claim 2, wherein the multichannel sensor network further comprises: a trigger device;

the trigger device is used for triggering the visual channel, the auditory channel, the tactile channel, the olfactory channel and the environmental information channel to synchronously acquire information at a certain frequency.

5. The human-computer interaction system under the intelligent home space according to claim 3, wherein the intelligent sensing module comprises: the system comprises a preprocessing unit, a multi-modal feature extraction unit and a multi-modal feature fusion analysis unit;

the preprocessing unit is used for preprocessing the image information, the voice information, the tactile perception information, the smell information and the environment information;

the multi-modal feature extraction unit is used for extracting features of the preprocessed image information, voice information, touch perception information, smell information and environment information to obtain multi-modal feature information;

and the multi-modal feature fusion analysis unit is used for performing fusion analysis on the multi-modal feature information and determining the intelligent perception result of the user.

6. The human-computer interaction system under the intelligent home space according to claim 5, wherein the image information is preprocessed, and the preprocessing at least comprises one of the following:

performing data alignment on the image information; performing redundancy removal on the image information; denoising the image information;

preprocessing the voice information, wherein the preprocessing at least comprises one of the following steps:

separating background sound and human voice of the voice information; removing noise in the voice information; removing empty white sound in the voice information;

preprocessing the haptic perception information, including at least one of:

correcting obvious error data in the tactile perception information;

completing missing data in the tactile perception information;

preprocessing the odor information, including at least one of:

removing identification failure information contained in the smell information;

removing information which obviously deviates from the behavior habit of the user from the smell information;

preprocessing the environment information, wherein the preprocessing at least comprises one of the following steps:

completing missing data in the environment information;

and completing the data lost due to the acquisition frequency in the environment information by using an interpolation method.

7. The human-computer interaction system under the smart home space according to claim 5, wherein the multi-modal feature extraction unit comprises: the system comprises a posture feature extraction subunit, a face feature extraction subunit, a thermal imaging feature extraction subunit, a depth feature extraction subunit, a semantic feature extraction subunit, a pose feature extraction subunit, a user sleep quality feature extraction subunit, a user motion trajectory feature extraction subunit, a user behavior event feature extraction subunit, a temperature feature extraction subunit, a humidity feature extraction subunit and an air quality feature extraction subunit;

the gesture feature extraction subunit is configured to use, as first target data, data obtained by preprocessing image information acquired by the RGB camera matrix, the 360-degree panoramic camera, the infrared camera in a dark environment, and/or the depth camera, and extract a gesture feature from the first target data based on a gesture recognition algorithm;

the face feature extraction subunit is configured to use data obtained by preprocessing the RGB camera matrix and the image information acquired by the 360-degree panoramic camera as second target data, and extract a face feature from the second target data based on a face recognition algorithm;

the thermal imaging feature extraction subunit is configured to use data obtained by preprocessing image information acquired by the infrared camera as third target data, and extract a human thermal imaging feature and an object thermal imaging feature from the third target data;

the depth feature extraction subunit is configured to use data obtained by preprocessing image information acquired by the depth camera as fourth target data, and extract a human depth feature, a spatial depth feature and an object depth feature from the fourth target data by using a deep learning algorithm or a three-dimensional point cloud feature point extraction algorithm;

the semantic feature extraction subunit is configured to use data obtained by preprocessing the voice information acquired by the microphone array as fifth target data, and extract a semantic feature from the fifth target data by using a voice recognition algorithm and/or a voice emotion analysis algorithm;

the pose feature extraction subunit is configured to use the preprocessed data of the pressure value information acquired by the pressure sensor arranged in the intelligent sofa and/or the intelligent bed as sixth target data, and analyze the sixth target data to obtain pose features of the user on the intelligent sofa and/or the intelligent bed;

the user motion track characteristic extraction subunit is used for taking the data obtained by preprocessing the capacitance value information acquired by the capacitive floor sensor as seventh target data and analyzing the seventh target data to obtain user motion track characteristics; the user motion trajectory features include: position, gait, direction and speed;

the user behavior event feature extraction subunit is used for taking the data obtained by preprocessing the smell information acquired by the smell sensor as eighth target data and analyzing the eighth target data to obtain user behavior event features;

the user sleep quality characteristic sub-extraction unit is used for taking data obtained by preprocessing carbon dioxide value information acquired by the carbon dioxide sensor arranged in the intelligent bed as ninth target data and analyzing the ninth target data to obtain user sleep quality characteristics;

the temperature characteristic extraction subunit is configured to use data obtained by preprocessing the temperature information acquired by the temperature sensor as tenth target data, and analyze the tenth target data to obtain a temperature characteristic;

the humidity characteristic extraction subunit is configured to use data obtained by preprocessing humidity information acquired by the humidity sensor as eleventh target data, and analyze the eleventh target data to obtain a humidity characteristic;

and the air quality characteristic extraction subunit is used for taking the data of the air quality information acquired by the air quality sensor after preprocessing as twelfth target data and analyzing the twelfth target data to obtain the air quality characteristics.

8. The human-computer interaction system under the smart home space according to claim 5, wherein the multi-modal feature fusion parsing unit comprises: the intelligent perception system comprises a data storage unit, an intelligent perception model construction unit and an intelligent perception execution unit;

the data storage unit is used for storing each modal feature in the multi-modal features into each modal feature database;

the intelligent perception model construction unit is used for training an intelligent perception model based on each modal characteristic by using each modal characteristic database in a training stage; carrying out model fusion on the intelligent perception models based on the modal characteristics to obtain intelligent perception models; also for updating the smart perception model with the multi-modal features during an application phase;

and the intelligent perception execution unit is used for inputting the multi-modal characteristics into the intelligent perception model to obtain a user intelligent perception result output by the intelligent perception model.

9. The human-computer interaction system under the intelligent home space according to claim 5, wherein the model fusion mode includes but is not limited to: voting, boosting, bootstrap aggregation and multilamellar fusion.

10. An application method of a man-machine interaction system in an intelligent home space is characterized by comprising the following steps:

the multi-channel sensor network carries out multi-mode information acquisition on the intelligent home space;

the intelligent sensing module intelligently senses the user in the intelligent home space according to the collected multi-mode information;

the man-machine interaction module realizes multi-mode man-machine interaction between various intelligent terminal devices in the intelligent home space and the user based on the intelligent perception result of the user;

wherein the intelligent perception comprises at least behavioral perception and emotional perception.

Images