WO2022041178A1

WO2022041178A1 - Brain wave-based information processing method and device, and instant messaging client

Info

Publication number: WO2022041178A1
Application number: PCT/CN2020/112408
Authority: WO
Inventors: 马宇尘
Original assignee: 上海量明科技发展有限公司
Priority date: 2020-08-29
Filing date: 2020-08-31
Publication date: 2022-03-03
Also published as: CN112217939A; CN112217939B

Abstract

A brain wave-based information processing method and device, and an instant messaging client, relating to the technical field of information processing. The brain wave-based information processing method comprises the following steps: obtaining operation information of triggering information recording by a user (S100); acquiring a brain wave signal of the user when the user records information (S200); identifying thinking information corresponding to the brain wave signal as associated information of the information (S300); and outputting the associated information and the recorded information together, or performing deduction on the associated information and the recorded information to obtain an information output that accords with the emotion of the user (S400). By using the method, a brain wave signal of a user can be acquired when the user records information, and thinking information corresponding to the brain wave signal is obtained as associated information for output, so that the convenience, intelligence, and interestingness of information processing are improved.

Description

Brainwave-based information processing method, device and instant messaging client

technical field

The present invention relates to the technical field of information processing.

Background technique

When using the auxiliary operation functions of terminal devices such as smartphones, tablets, personal computers (including but not limited to Voiceover, Talkback and other screen-reading software and other applications with similar functions), user interface elements and their functions will be extracted and processed through TTS (Text to Speech, text-to-speech technology) plays the selected text with voice to help users understand the content currently displayed on the screen of the mobile phone; in addition, users can click, slide and other operations on the touch screen to perform operations on the terminal. The operation brings a more comprehensive and richer user experience to the user.

The operating systems of existing terminals mostly stay at the level of user input and one-to-one machine response, and cannot think intelligently according to the human way of thinking, so as to provide users with other information related to the input in a pre-emptive manner. The input tools of instructions are mainly displayed on the computer, mainly keyboard and mouse and other voice input tools, and on portable interactive devices, mainly keyboard and voice, no matter what the specific input tools and input methods are, they are all human thoughts. However, none of the existing technologies can change the habit of people making a single expression and machines making a single feedback.

At present, humans have developed devices such as brain wave translators and iris recognizers, and applied them as input devices, of which there have been experimental applications in the field of robotics. With the development of brain wave detection technology becoming more and more perfect, we can apply brain wave detection technology to the field of intelligent terminal interaction. Through the brain wave sensing structure, the user's thoughts can be directly transmitted to the intelligent terminal, and the user can be processed through the intelligent terminal. information about thinking. How to provide users with a more intelligent and convenient communication method in combination with the above-mentioned existing technologies is an urgent problem to be solved.

technical problem

The purpose of the present invention is to overcome the deficiencies of the prior art and provide an information processing method, device and instant messaging client based on brain waves. By using the present invention, the user's brain wave signal can be collected when the user records information, and the thinking information corresponding to the brain wave signal can be obtained as the associated information output, which improves the convenience, intelligence and interest of information processing, especially in communication When interacting, the user's thinking information can be obtained as a message output, making the chat interaction intelligent, interesting and vivid.

technical solutions

To achieve the above-mentioned goals, the present invention provides the following technical solutions:

An information processing method based on brain waves, comprising the following steps:

Obtain the operation information of user-triggered information recording;

Collect the user's brain wave signal when the user records information;

Identify the thinking information corresponding to the brain wave signal as the associated information of the information;

The associated information and the recorded information are output together, or the associated information and the recorded information are deduced to obtain information output that conforms to the user's emotion.

Further, the information is a streaming media message used for communication interaction.

Further, there are multiple users, and the brainwave signals of two or more users are acquired for identification when the multiple users are chatting and interacting.

Further, the user's selection operation on the output information is collected, and the selected information is sent to the target client as a message.

Further, the associated information and the recording information are sent to the target client as a message, or information that conforms to the user's mood is sent to the target client as a message.

Further, the way of identifying the thinking information corresponding to the brain wave signal is,

Obtain the user's personal information storage unit based on the user's feature information, where the personal information is personal inherent information generated based on the user's brainwave test data, which includes the mapping relationship between the user's brainwave signal and target feature information;

In the personal information storage unit, obtain target feature information matching the aforementioned collected brain wave signals;

Acquire the text information, image information and/or virtual scene information corresponding to the target feature information as the corresponding thinking information.

Further, it also includes a group information storage unit for storing group information, the group information is statistical information obtained based on the brain wave test data of other subjects, and it includes the statistical information obtained by performing validity statistics on the subject's brain wave signals. The mapping relationship between brain wave signal and target feature information;

When the personal information storage part of the user cannot be obtained based on the user's characteristics, obtain the target characteristic information matching the aforementioned collected brain wave signal in the group information storage part;

Preferably, the detected person is an unspecified majority.

Preferably, the detected person is the majority of people whose matching degree with the user attribute reaches a preset matching degree threshold, and the attribute includes gender, age, occupation, education level, place of residence, place of study and/or hobby information.

The target feature information includes one or more of target characters, target keywords, target graphic elements, target actions/gestures, and target symbols.

Further, the user triggers the operation information of the information recording through the input method tool, and outputs the associated information through the input method tool.

The present invention also provides an information processing device based on brain waves, including the following structures:

an information collection module, used to obtain the operation information of the user-triggered information recording;

Brain wave collecting electrodes are used to collect the user's brain wave signal when the user records information;

a processor module, used to identify the thinking information corresponding to the brain wave signal as the associated information of the information;

The information output module outputs the associated information together with the recorded information, or deduces the associated information and the recorded information to obtain information output that conforms to the user's emotion.

The present invention also provides an instant messaging client for performing instant messaging interaction, which includes:

The voice message trigger module is used to collect the user's voice trigger operation;

The brain wave acquisition module is used to collect the user's brain wave signal when the user records the voice;

a processor module for identifying the thinking information corresponding to the brain wave signal as the associated information of the speech;

an output module, which outputs the associated information together with the voice, or deduces the associated information and the voice to obtain information output that conforms to the user's emotion;

The message sending module is used for sending the aforementioned output information to the target client according to the user's selection operation, or sending the aforementioned output information in combination with the voice message to the target client.

beneficial effect

Compared with the prior art, the present invention has the following advantages and positive effects due to the adoption of the above technical solutions, as an example, the user's brain wave signal can be collected when the user records information, and the thinking information corresponding to the brain wave signal can be obtained. As the output of related information, the convenience, intelligence and interest of information processing are improved, especially in the communication interaction, the user's thinking information can be obtained as the message output, making the chat interaction intelligent, interesting and vivid.

Description of drawings

FIG. 1 is a flowchart of an information processing method based on brain waves provided by an embodiment of the present invention.

FIG. 2 is a module structure diagram of an instant messaging client provided by an embodiment of the present invention.

FIG. 3 to FIG. 7 are diagrams illustrating operation examples of instant messaging interaction provided by an embodiment of the present invention.

8 to 10 are exemplary diagrams when a voice message including associated information is received according to an embodiment of the present invention.

FIG. 11 is an example diagram of outputting associated information through an input method tool during document editing according to an embodiment of the present invention.

Description of reference numbers:

Instant messaging client 100, voice message trigger module 110, brain wave acquisition module 120, processor module 130, output module 140, message sending module 150; user terminal 200, desktop 210, instant messaging tool icon 211, contact 220, microphone 230 ; the communication interface 300 .

Embodiments of the present invention

Type the paragraphs describing embodiments of the invention here. The brainwave-based information processing method, device and instant messaging client provided by the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the technical features or combinations of technical features described in the following embodiments should not be considered isolated, and they can be combined with each other to achieve better technical effects. In the drawings of the following embodiments, the same reference numerals appearing in the various drawings represent the same features or components, which may be used in different embodiments. Therefore, once an item is defined in one figure, it need not be discussed further in subsequent figures.

It should be noted that the structures, proportions, sizes, etc. shown in the accompanying drawings in this specification are only used to cooperate with the contents disclosed in the specification, so as to be understood and read by those who are familiar with the technology, and are not used to limit the invention. The limited conditions for implementation, any structural modification, change in proportional relationship or adjustment of size, shall fall within the scope of the technical content disclosed in the invention without affecting the efficacy and purpose of the invention. within the range. The scope of the preferred embodiments of the present invention includes additional implementations in which the functions may be performed out of the order described or discussed, including performing the functions in a substantially simultaneous manner or in the reverse order depending upon the functions involved, which should be Embodiments of the invention will be understood by those skilled in the art to which the embodiments of the invention pertain.

Techniques, methods, and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the authorized description. In all examples shown and discussed herein, any specific value should be construed as illustrative only and not as limiting. Accordingly, other examples of exemplary embodiments may have different values.

Example

Referring to FIG. 1 , an information processing method based on brain waves is disclosed, including the following steps:

S100: Acquire operation information for user-triggered information recording.

In this embodiment, the recorded information is preferably a streaming media message used for communication interaction. The streaming media information, by way of example and not limitation, may be voice information, animation information, lip language video, sign language video information, and the like.

Taking voice information as an example, when a user needs to send a voice message, the audio collection device can be activated to record the voice. Taking instant messaging tool (IM tool) WeChat as an example for illustration, the message at this time is an instant messaging message. After the user enters WeChat, he can trigger the voice recording button to start the audio collection device of the terminal where he is located. After the pickup is activated, the user's voice information can be recorded.

The terminal, by way of example and not limitation, may be various commonly used mobile terminals such as mobile phones, palmtop computers, and tablet computers, and various smart wearable electronic devices, such as smart glasses, smart watches, and the like. In this embodiment, a mobile phone is used as the mobile terminal, and the mobile phone has an audio collection structure, an image collection structure and a display structure.

S200, collect the brain wave signal of the user when the user records information.

By way of example and not limitation, for example, brain wave signals are collected from the frontal, central, and parietal regions of the user's brain.

Generally speaking, the brain wave collection device may be a smart wearable device in the form of a wearable helmet, headband, and the like. For the user's brainwave signal, the brainwave acquisition device sends the sensed brainwave signal to the terminal through a preset route. In this embodiment, the terminal is a terminal device that establishes a communication connection with the brainwave sensor.

Specifically, the EEG acquisition head-mounted device is provided with an EEG acquisition card.

As an example and not a limitation, the EEG acquisition card may include acquisition electrodes, reference electrodes, and the EEG acquisition chip with its own wireless transceiver, and the acquisition electrodes, the reference electrode, and the built-in wireless transceiver are all connected to the EEG acquisition chip. As an example and not a limitation, the EEG acquisition chip can be a product of EMOTIV company. The acquisition electrode collects the user's original brain wave value and transmits it to the EEG acquisition chip for processing. The EEG acquisition chip amplifies the original brain wave value. Obtain the EEG signal, which is the scalp potential at each electrode, and transmit the data packet composed of the EEG signal to the associated processor in real time through the built-in wireless transmitter for processing.

The brain wave signal may include one or more of the waveform of the brain wave, the number of the brain wave, and the intensity information of the brain wave.

S300, identifying the thinking information corresponding to the brain wave signal as the associated information of the recorded information.

After acquiring the brainwave signal, the terminal needs to identify the brainwave signal, and identify relevant feature information contained in the brainwave signal, such as brainwave type, frequency, potential, and the like.

Then, corresponding thinking information is obtained based on the feature information.

In this embodiment, the way to obtain the corresponding thinking information is preferably the following way:

The personal information storage unit may store the subject's ID (the user), gender, age, name, serial number, and other personal basic information, as well as the subject's brain wave test data. The test data may specifically include delta waves, theta waves, alpha waves, beta waves, and gamma waves in spontaneous brain waves, as well as external stimuli such as auditory, visual, and tactile brain waves in evoked brain waves, or due to the appearance of expectations. , attention, decision-making, pleasure and other special frequency components produced by internal psychological activities.

Further, it can also include a group information storage unit for storing group information, the group information is statistical information obtained based on the brain wave test data of other subjects, which includes the subject's brain wave signal obtained after validity statistics. The mapping relationship between the brain wave signal and the target feature information;

The detected persons may be an unspecified majority.

The target feature information may preferably include one or more of target characters, target keywords, target graphic elements, target actions/gestures, and target symbols.

S400: Output the associated information together with the recorded information, or perform deduction on the associated information and the recorded information to obtain information output that conforms to the user's emotion.

The associated information may specifically be text information, image information and/or virtual scene information. The image information may be still pictures or videos or animations.

As an example but not a limitation, for example, the voice message recorded by the user is "Let's go to Yangcheng Lake to eat big crabs together", and the user's brainwaves also detect the expectation and joy of food when he speaks. "Yangcheng Lake" and "Hairy Crab" obtain image data output of a person eating a delicious big crab.

In this embodiment, there may be multiple users, and the brain wave signals of two or more users may be acquired and identified respectively when multiple users are chatting and interacting. In this way, the associated information related to the interactive content can be obtained for the chat information of multiple users.

Further, after step S400, the following steps may also be included:

The user's selection operation on the output information is collected, and the selected information is sent to the target client as a message.

Alternatively, the associated information and the recorded information are sent to the target client as a message, or information that conforms to the user's mood is sent to the target client as a message.

In another implementation of this embodiment, the image information when the user records the voice can also be collected, the user's facial expression and/or action information can be obtained, and the emotional information corresponding to the facial expression and/or the action information can be identified as the first part of the voice message. 2. Related information.

As an example and not a limitation, according to the acquired emotional information (second associated information), searching for an emoticon icon corresponding to the emotional information is output together with the voice message.

In another implementation manner of this embodiment, the information recording is not through a communication tool, but may also be any other application capable of recording information. Preferably, the user can trigger the operation information of the information recording through the input method tool, and output the associated information through the input method tool at a later stage.

Referring to FIG. 2 , the present invention also provides an instant messaging client 100 for performing instant messaging interaction, which includes the following structure:

The voice message triggering module 110 is used for collecting user's voice triggering operation.

Take the instant messaging tool (IM tool) WeChat as an example. After the user enters WeChat, he can trigger the voice recording button to start the audio collection device of the terminal where he is located. After the pickup is activated, the user's voice information can be recorded.

The brain wave acquisition module 120 is used to collect the user's brain wave signal when the user records voice.

The brain wave acquisition module 120 can collect brain wave signals in the frontal, central and parietal regions of the user's brain.

Specifically, the brain wave acquisition module may be a smart wearable device in the form of a wearable helmet, headband, or the like. For the user's brainwave signal, the brainwave acquisition device sends the sensed brainwave signal to the terminal through a preset route. In this embodiment, the terminal is a terminal device that establishes a communication connection with the brainwave sensor.

In this embodiment, the EEG acquisition head-mounted device is provided with an EEG acquisition card.

As an example and not a limitation, the EEG acquisition card may include acquisition electrodes, reference electrodes, and the EEG acquisition chip with its own wireless transceiver, and the acquisition electrodes, the reference electrode, and the built-in wireless transceiver are all connected to the EEG acquisition chip. As an example and not a limitation, the EEG acquisition chip can be a product of EMOTIV company. The acquisition electrode collects the user's original brain wave value and transmits it to the EEG acquisition chip for processing. The EEG acquisition chip amplifies the original brain wave value. Obtain the EEG signal, which is the scalp potential at each electrode, and transmit the data packet composed of the EEG signal to the processor in real time through the built-in wireless transmitter for processing.

The processor module 130 is used for identifying the thinking information corresponding to the brain wave signal as the associated information of the speech.

After acquiring the brain wave signal, the processor needs to identify the brain wave signal, and identify relevant feature information contained in the brain wave signal, such as brain wave type, frequency, potential, and the like.

The detected persons may be an unspecified majority.

The output module 140 outputs the associated information together with the voice, or deduces the associated information and the voice to obtain information output that conforms to the user's emotion.

As an example but not a limitation, for example, the voice message recorded by the user is "Let's go to Yangcheng Lake to eat big crabs together", and the user's brainwaves also detect the expectation of food and the happy mood when speaking. "Yangcheng Lake" and "Hairy Crab" obtain image data output of a person eating a delicious big crab.

The message sending module 150 is configured to send the aforementioned output information to the target client according to the user's selection operation, or send the aforementioned output information in combination with a voice message to the target client.

In this way, the user and the target contact can not only interact based on the recorded information, but also obtain the brain wave data of one or more parties to interact, which improves the fun and intelligence of the interaction.

The implementation of this embodiment will be described in detail with reference to FIGS. 3 to 7 .

Referring to FIG. 3 , the user enters the instant messaging tool “Quick Message” through the user terminal 200 carried by the user. The user terminal 200 is preferably a mobile phone in this embodiment.

Referring to FIG. 4 , the desktop 210 of the user terminal 200 outputs a user interface to the user, on which all communication messages are displayed, and the communication messages display the contacts 220, the latest interactive messages, and a virtual microphone 230 (voice trigger control).

As an example, as shown in FIG. 4 , for example, when a user chats with a contact leo, the virtual microphone 230 corresponding to leo can be triggered, and then the voice message collection function can be directly started.

Referring to FIG. 5 , a message input box is displayed in the user interface, and the input box displays the voice being entered by the user, the text content corresponding to the voice, and related operation keys.

During the process of the user recording the voice, the user's brain wave signal is simultaneously collected and displayed in the message input box.

Referring to FIG. 6 , after the corresponding associated information “Forest Park” and “Xiaoduoduo” are obtained based on the brain wave signal, the associated information and voice are deduced to obtain image information that conforms to the user’s emotions, and the information is presented to the user. output.

The message input box after the voice is triggered can be displayed directly on the current user interface as shown in Figure 5, or it can be displayed after a separate voice message interface is generated for the contact leo. Referring to Figure 7, the voice message interface displays the following: Contact information, voice message input box, virtual microphone, and corresponding generated image information.

When the user is recording voice, he can perform sending and pausing operations by operating the virtual microphone 230 . As an example of a preferred manner, for example, pressing and sliding the microphone up is a send operation, and pressing and sliding the microphone to the right is a pause operation.

In this embodiment, the output image information may be independently sent to the target contact as a message based on the user's selection, or the output image information may be sent together with a voice message.

Referring to Fig. 8, the voice message and the output image information are sent together as two independent messages.

Alternatively, as shown in FIG. 9 , the image information is inserted into the voice message as streaming media to form a multimedia message and sent to the contact. For the image data, a play button can also be set in the message box. After triggering the play button, the user can play the image data through the floating window or based on the current area.

Alternatively, a floating window is set corresponding to the voice message, and the image data is directly displayed through the floating window when a message is output.

Alternatively, as shown in FIG. 10 , the image information may also be acquired and displayed at different positions of the text content of the voice. In this way, rich and colorful image information can be displayed based on the text content of the voice, making the message interesting and vivid.

8 to 10 also show the audio file of the voice message in the multimedia message output.

The text content is displayed in the message box of the multimedia message, and an audio file play button may be set corresponding to the message box, and triggering the play button can trigger the audio file to play.

The instant messaging client can also be set with other functional modules as required, and the specific functions can be found in the previous embodiments, which will not be repeated here.

In another embodiment of this embodiment, the information recording is not through a communication tool, and may also be any other application capable of recording information.

Referring to FIG. 11 , preferably, the user can trigger the operation information of the information recording through the input method tool, and output the associated information through the input method tool in the later stage.

As an example rather than a limitation, for example, when a user edits a document, he enters the text based on the input method tool "The day before yesterday, I went to the forest park with my family to test the drone, and I also brought my cute pet Xiaoduoduo. Xiaoduoduo and I are in the forest. I had a lot of fun in the park", the input method tool obtained the user's brainwave signal when recording the above text information, obtained the associated information "Forest Park" and "Xiaoduoduo" through the brainwave signal, and then analyzed the associated information. Deduce with voice to obtain image information that conforms to the user's emotions, and output it to the user in the currently edited document.

Another embodiment of the present invention also provides an information processing device based on brain waves, which includes the following structure:

The information processing device can be set independently, or embedded in other client terminals to implement corresponding functions, and can also set other functional modules as required. For details, refer to the previous embodiments, which will not be repeated here.

In the above description, although all components of various aspects of the present disclosure may be explained as being assembled or operatively connected as a circuit, the present disclosure is not intended to limit itself to these aspects. Rather, the various components may be selectively and operatively combined in any number within the intended scope of this disclosure. Each of these components may also itself be implemented in hardware, while the individual components may be combined in part or selectively collectively and implemented as a computer program having program modules for performing the functions of the hardware equivalent. Code or code segments to construct such programs can be readily derived by those skilled in the art. Such a computer program can be stored in a computer-readable medium, which can be executed to implement aspects of the present disclosure. The computer-readable media may include magnetic recording media, optical recording media, and carrier wave media.

Additionally, terms like "includes," "includes," and "has" should by default be construed as inclusive or open, rather than exclusive or closed, unless explicitly defined to the contrary. All technical, scientific or other terms have the meaning as understood by those skilled in the art unless they are defined to the contrary. Common terms found in dictionaries should not be interpreted too ideally or too practically in the context of related technical documents, unless this disclosure explicitly defines them as such.

While exemplary aspects of the present disclosure have been described for illustrative purposes, those skilled in the art will appreciate that the foregoing description is merely a description of preferred embodiments of the present invention and is not intended to limit The scope of the embodiments includes additional implementations in which the functions may be performed out of the order described or discussed. Any changes and modifications made by those of ordinary skill in the field of the present invention according to the above disclosure fall within the protection scope of the claims.

Claims

An information processing method based on brain waves, characterized in that it comprises the following steps:

Obtain the operation information of user-triggered information recording;

Collect the user's brain wave signal when the user records information;

Identify the thinking information corresponding to the brain wave signal as the associated information of the recorded information;

The associated information and the recorded information are output together, or the associated information and the recorded information are deduced to obtain information output that conforms to the user's emotion.
The method according to claim 1, wherein the recorded information is a streaming media message used for communication interaction.
The method according to claim 1 or 2, wherein there are multiple users, and the brainwave signals of more than two users are acquired for identification when the multiple users are chatting and interacting.
The method according to claim 1 or 2, characterized in that: collecting a user's selection operation on the output information, and sending the selected information as a message to the target client.
The method according to claim 1 or 2, characterized in that: the associated information and the recorded information are sent to the target client as a message, or the information conforming to the user's mood is sent to the target client as a message.
The method according to claim 1, wherein the method of identifying the thinking information corresponding to the brain wave signal is:

Obtain the user's personal information storage unit based on the user's feature information, where the personal information is personal inherent information generated based on the user's brainwave test data, which includes the mapping relationship between the user's brainwave signal and target feature information;

In the personal information storage unit, obtain target feature information matching the aforementioned collected brain wave signals;

Acquire the text information, image information and/or virtual scene information corresponding to the target feature information as the corresponding thinking information.
The method according to claim 6, further comprising a group information storage unit for storing group information, the group information is statistical information obtained based on the brain wave test data of other subjects, which includes The mapping relationship between the brain wave signal and the target feature information obtained after the validity statistics of the brain wave signal of the subject;

When the personal information storage part of the user cannot be obtained based on the user's characteristics, obtain the target characteristic information matching the aforementioned collected brain wave signal in the group information storage part;

Acquire the text information, image information and/or virtual scene information corresponding to the target feature information as the corresponding thinking information.
The method according to claim 7, wherein the detected persons are an unspecified majority.
The method according to claim 7, wherein the detected person is a majority of people whose matching degree with the user attribute reaches a preset matching degree threshold, and the attribute includes gender, age, occupation, education level, place of residence , place of study and/or interest information.
The method according to claim 6, wherein the target feature information includes one or more of target characters, target keywords, target graphic elements, target actions/gestures and target symbols.
The method according to claim 1, wherein the user triggers the operation information of the information recording through an input method tool, and outputs the associated information through the input method tool.
An information processing device based on brain waves is characterized by comprising the following structures:

an information collection module, used to obtain the operation information of the user-triggered information recording;

Brain wave collecting electrodes are used to collect the user's brain wave signal when the user records information;

a processor module, used to identify the thinking information corresponding to the brain wave signal as the associated information of the information;

The information output module outputs the associated information together with the recorded information, or deduces the associated information and the recorded information to obtain information output that conforms to the user's emotion.
An instant messaging client for instant messaging interaction, characterized by comprising:

The voice message trigger module is used to collect the user's voice trigger operation;

The brain wave acquisition module is used to collect the user's brain wave signal when the user records the voice;

a processor module for identifying the thinking information corresponding to the brain wave signal as the associated information of the speech;

an output module, which outputs the associated information together with the voice, or deduces the associated information and the voice to obtain information output that conforms to the user's emotion;

The message sending module is used for sending the aforementioned output information to the target client according to the user's selection operation, or sending the aforementioned output information in combination with the voice message to the target client.