CN116872222B - Psychological consultation robot, psychological consultation system, psychological consultation control method and psychological consultation storage medium - Google Patents
Psychological consultation robot, psychological consultation system, psychological consultation control method and psychological consultation storage medium Download PDFInfo
- Publication number
- CN116872222B CN116872222B CN202310743021.1A CN202310743021A CN116872222B CN 116872222 B CN116872222 B CN 116872222B CN 202310743021 A CN202310743021 A CN 202310743021A CN 116872222 B CN116872222 B CN 116872222B
- Authority
- CN
- China
- Prior art keywords
- psychological
- psychological consultation
- user
- electroencephalogram
- consultation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 69
- 238000003860 storage Methods 0.000 title claims abstract description 15
- 230000008451 emotion Effects 0.000 claims abstract description 58
- 210000004556 brain Anatomy 0.000 claims abstract description 39
- 230000006978 adaptation Effects 0.000 claims abstract description 6
- 238000009223 counseling Methods 0.000 claims description 30
- 239000013598 vector Substances 0.000 claims description 27
- 230000004927 fusion Effects 0.000 claims description 18
- 238000010586 diagram Methods 0.000 claims description 16
- 239000011159 matrix material Substances 0.000 claims description 16
- 230000004044 response Effects 0.000 claims description 15
- 230000001815 facial effect Effects 0.000 claims description 13
- 230000008909 emotion recognition Effects 0.000 claims description 12
- 238000003062 neural network model Methods 0.000 claims description 12
- 238000003825 pressing Methods 0.000 claims description 7
- 238000003491 array Methods 0.000 claims description 4
- 210000003141 lower extremity Anatomy 0.000 claims description 4
- 238000007781 pre-processing Methods 0.000 claims description 4
- 238000013473 artificial intelligence Methods 0.000 abstract description 5
- 230000006870 function Effects 0.000 description 24
- 230000000875 corresponding effect Effects 0.000 description 17
- 230000008569 process Effects 0.000 description 17
- 238000004458 analytical method Methods 0.000 description 13
- 238000012545 processing Methods 0.000 description 13
- 238000001914 filtration Methods 0.000 description 9
- 210000000349 chromosome Anatomy 0.000 description 7
- 230000035939 shock Effects 0.000 description 7
- 201000010099 disease Diseases 0.000 description 6
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 6
- 230000001276 controlling effect Effects 0.000 description 5
- 230000002068 genetic effect Effects 0.000 description 5
- 230000035772 mutation Effects 0.000 description 5
- 238000013527 convolutional neural network Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 238000003384 imaging method Methods 0.000 description 4
- 238000012880 independent component analysis Methods 0.000 description 4
- 238000010606 normalization Methods 0.000 description 4
- 238000000354 decomposition reaction Methods 0.000 description 3
- 230000002996 emotional effect Effects 0.000 description 3
- 230000003993 interaction Effects 0.000 description 3
- 230000033001 locomotion Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000006467 substitution reaction Methods 0.000 description 3
- 238000012935 Averaging Methods 0.000 description 2
- 206010034960 Photophobia Diseases 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 239000006185 dispersion Substances 0.000 description 2
- 230000004424 eye movement Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000003340 mental effect Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 241000167857 Bourreria Species 0.000 description 1
- 206010010904 Convulsion Diseases 0.000 description 1
- 241000282412 Homo Species 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000008602 contraction Effects 0.000 description 1
- 230000036461 convulsion Effects 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000005611 electricity Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000008921 facial expression Effects 0.000 description 1
- 238000007429 general method Methods 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 210000003205 muscle Anatomy 0.000 description 1
- 230000004118 muscle contraction Effects 0.000 description 1
- 230000036640 muscle relaxation Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 239000000047 product Substances 0.000 description 1
- 108090000623 proteins and genes Proteins 0.000 description 1
- 238000011946 reduction process Methods 0.000 description 1
- 230000000241 respiratory effect Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 230000002194 synthesizing effect Effects 0.000 description 1
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J11/00—Manipulators not otherwise provided for
- B25J11/0005—Manipulators having means for high-level communication with users, e.g. speech generator, face recognition means
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J11/00—Manipulators not otherwise provided for
- B25J11/008—Manipulators for service tasks
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J13/00—Controls for manipulators
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/16—Matrix or vector computation, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
- G06F18/2134—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods based on separation criteria, e.g. independent component analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/02—Reservations, e.g. for tickets, services or events
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Mathematical Physics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Business, Economics & Management (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Health & Medical Sciences (AREA)
- Software Systems (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Robotics (AREA)
- Mechanical Engineering (AREA)
- Tourism & Hospitality (AREA)
- Computational Mathematics (AREA)
- Databases & Information Systems (AREA)
- Pure & Applied Mathematics (AREA)
- Mathematical Optimization (AREA)
- Mathematical Analysis (AREA)
- Molecular Biology (AREA)
- Marketing (AREA)
- General Business, Economics & Management (AREA)
- Human Computer Interaction (AREA)
- Entrepreneurship & Innovation (AREA)
- Strategic Management (AREA)
- Quality & Reliability (AREA)
- Operations Research (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Development Economics (AREA)
- Economics (AREA)
- Human Resources & Organizations (AREA)
- Computational Linguistics (AREA)
Abstract
The invention discloses a psychological consultation robot, a psychological consultation system, a psychological consultation control method and a psychological consultation storage medium, wherein the psychological consultation control method comprises the following steps: acquiring voice information, face images and a plurality of brain electrical signals of a user, which characterize psychological consultation problems of the user, and generating a psychological image of the user; generating a user emotion mark according to the user psychological image; generating corresponding consultation questions according to the voice information, and selecting a plurality of psychological consultation answers matched with the psychological consultation questions from an answer corpus; according to the emotion marks, selecting the answer with the highest adaptation degree with the emotion marks from the plurality of answers, outputting the selected answer as psychological consultation advice, and controlling the loudspeaker equipment to play the psychological consultation advice. The invention realizes the automatic generation of psychological consultation advice, can provide timely, convenient and objective psychological consultation service for users, ensures the objectivity and quality of psychological consultation, and can meet a great deal of psychological consultation demands in modern society. The invention is applied to the field of artificial intelligence.
Description
Technical Field
The invention relates to the technical field of artificial intelligence, in particular to a psychological consultation robot, a psychological consultation system, a psychological consultation control method and a psychological consultation storage medium.
Background
Psychological consultation refers to the process of providing assistance to psychological problems of visitors by applying psychological methods. Psychological consultation refers to the process of providing assistance to psychological problems of visitors by applying psychological methods. Today, people are increasingly stressed by modern high-speed development of society, families and the like, and the psychology of more people is mostly in a sub-health state. Currently, psychological counseling services still have the following problems:
1. professional talents such as professional doctors and psychological consultants are rare, and a great deal of psychological consultation demands in modern society are difficult to meet, and the quality of psychological consultation services is uneven;
2. psychological consultants often have multiple functions and face multiple roles, so that the consultants often encounter situations that the identities are difficult to change and adapt when facing visitors, and the objectivity of the consultation results is difficult to maintain;
3. for visitors, firstly, due to the dirty name of psychological diseases, part of psychological disease patients are photophobia to be acknowledged as suffering from psychological diseases without psychological consultation, and the patients suffer from negative emotion and negative experience; secondly, some patients can be hidden from the actual situation during psychological consultation, so that psychological consultants cannot provide consultation services according to the actual situation of the patients; moreover, the traditional psychological consultation generally adopts reservation type, so that the patient needs to go to the appointed field in the reservation time period to carry out the psychological consultation, the way easily leads to the problems that the patient cannot carry out the psychological consultation due to untimely reservation, the doctor temporarily cancels the reservation and leads to the patient not to carry out the psychological consultation in time and the like, the time and the labor are wasted, the timeliness of psychological dispersion of visitors is difficult to be ensured, and convenient and fast psychological consultation service cannot be provided for the patient.
It follows that the above problems are in need of a solution.
Disclosure of Invention
The present invention aims to solve at least one of the technical problems existing in the prior art to a certain extent.
To this end, the present invention aims to provide a psychological counseling robot, a psychological counseling system, a psychological counseling control method and a psychological counseling storage medium.
In order to achieve the technical purpose, the technical scheme adopted by the embodiment of the invention comprises the following steps:
in a first aspect, an embodiment of the present invention provides a psychological consultation robot, where the robot is in a humanoid form, the robot includes a head, a trunk portion, and a lower limb portion, an ear portion, an eye portion, and a mouth portion are provided on the head, the trunk portion is provided with a hand, the ear portion is provided with a voice acquisition device, the eye portion is provided with a camera, the mouth portion is provided with a speaker device, a response button is provided on a surface of the hand, a psychological consultation operation device is installed inside the head, the psychological consultation operation device is respectively connected with the voice acquisition device, the camera, the response button, and the speaker device, and the psychological consultation operation device is also connected with an external electroencephalogram acquisition device.
In a second aspect, an embodiment of the present invention provides a psychological consulting system, including: the brain electrical signal acquisition device is used for acquiring a plurality of brain electrical signals of a user and transmitting the brain electrical signals to the psychological consultation robot.
In a third aspect, an embodiment of the present invention provides a method for controlling a psychological consulting system, including: s100, responding to a pressing instruction of a response button, acquiring voice information of a user through a voice acquisition device, acquiring facial images of the user through a camera device, and acquiring a plurality of electroencephalogram signals of the user; wherein, the voice information characterizes psychological consultation problems of users;
s200, generating a user psychological image according to the face image and a plurality of the brain electrical signals;
s300, carrying out emotion recognition on the user psychological image by using a first neural network model to obtain a user emotion mark;
s400, performing voice recognition on the voice information to generate corresponding psychological consultation questions, traversing a preset answer corpus, and selecting a plurality of psychological consultation answers matched with the psychological consultation questions from the answer corpus;
S500, selecting the answer with the highest adaptation degree with the user emotion mark from a plurality of psychological consultation answers according to the user emotion mark, outputting the selected psychological consultation answer as a psychological consultation suggestion, and controlling a loudspeaker device to play the psychological consultation suggestion.
In a fourth aspect, an embodiment of the present invention provides a storage medium in which a program executable by a processor is stored, which when executed by the processor is used to implement a control method of a psychological consulting system as described above.
The beneficial effects of the invention are as follows: the robot of the invention takes the posture of a person as the appearance, so that a patient can have more feeling and desire to speak, the patient carries out consultation or complaint on the robot, an operation device of the robot carries out operation according to the consultation or the complaint, and corresponding consultation advice is provided for a user, thereby realizing psychological consultation based on man-machine interaction; the invention realizes the automatic generation of psychological consultation advice, saves the daily workload of psychological consultants, provides convenient, timely and objective psychological consultation service for users, avoids the problems that the users cannot carry out psychological consultation due to untimely reservation and the patients cannot carry out psychological consultation due to temporary cancellation of reservation, and the like, improves the generation accuracy of the psychological consultation advice, ensures the psychological consultation advice to better meet the actual consultation demands of the users, ensures the objectivity and quality of the psychological consultation and meets a great deal of psychological consultation demands in the modern society.
Additional features and advantages of the application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application. The objectives and other advantages of the application will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following description is made with reference to the accompanying drawings of the embodiments of the present invention or the related technical solutions in the prior art, and it should be understood that the drawings in the following description are only for convenience and clarity of describing some embodiments in the technical solutions of the present invention, and other drawings may be obtained according to these drawings without the need of inventive labor for those skilled in the art.
FIG. 1 is a block diagram of a psychological consulting robot provided by the present invention;
FIG. 2 is a block diagram of an electroencephalogram signal acquisition device provided by the invention;
FIG. 3 is a block diagram of an earplug provided by the invention;
FIG. 4 is a flowchart of a control method of the psychological consulting system provided by the present invention;
FIG. 5 is a flow chart of generating a psychological image of a user provided by the present invention;
FIG. 6 is a flow chart of an imaging process provided by the present invention;
FIG. 7 is a flow chart of finding the best psychological consulting advice provided by the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.
The present application is further described below with reference to the drawings and specific examples. The described embodiments should not be construed as limitations on the present application, and all other embodiments, which may be made by those of ordinary skill in the art without the exercise of inventive faculty, are intended to be within the scope of the present application.
In the following description, reference is made to "some embodiments" which describe a subset of all possible embodiments, but it is to be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments and can be combined with one another without conflict.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of the present application only and is not intended to be limiting of the present application.
Psychological consultation refers to the process of providing assistance to psychological problems of visitors by applying psychological methods. Psychological consultation refers to the process of providing assistance to psychological problems of visitors by applying psychological methods. Today, people are increasingly stressed by modern high-speed development of society, families and the like, and the psychology of more people is mostly in a sub-health state. Currently, psychological counseling services still have the following problems:
1. professional talents such as professional doctors and psychological consultants are rare, and a great deal of psychological consultation demands in modern society are difficult to meet, and the quality of psychological consultation services is uneven;
2. psychological consultants often have multiple functions and face multiple roles, so that the consultants often encounter situations that the identities are difficult to change and adapt when facing visitors, and the objectivity of the consultation results is difficult to maintain;
3. for visitors, firstly, due to the dirty name of psychological diseases, part of psychological disease patients are photophobia to be acknowledged as suffering from psychological diseases without psychological consultation, and the patients suffer from negative emotion and negative experience; secondly, some patients can be hidden from the actual situation during psychological consultation, so that psychological consultants cannot provide consultation services according to the actual situation of the patients; moreover, the traditional psychological consultation generally adopts reservation type, so that the patient needs to go to the appointed field in the reservation time period to carry out the psychological consultation, the way easily leads to the problems that the patient cannot carry out the psychological consultation due to untimely reservation, the doctor temporarily cancels the reservation and leads to the patient not to carry out the psychological consultation in time and the like, the time and the labor are wasted, the timeliness of psychological dispersion of visitors is difficult to be ensured, and convenient and fast psychological consultation service cannot be provided for the patient.
Artificial intelligence (Artificial Intelligence, AI) is an emerging computer science technology in recent years, also known as smart machine, machine intelligence, which refers to machines manufactured by humans that can exhibit intelligence. If the artificial intelligence technology is applied to the psychological consultation field, a robot, a system and a method for psychological consultation are provided, and timely, convenient and objective psychological consultation services are provided for users, so that the defects in the current psychological consultation service industry can be overcome. In view of the above, the present application provides a psychological counseling robot, system, control method and storage medium, which overcome the drawbacks of the prior art.
A psychological counseling robot according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings.
Referring to fig. 1, the psychological counseling robot 100 provided by the present invention includes a head 110, a trunk part 120, and a lower limb part 130. Specifically:
the psychological consultation operation device is arranged in the head 110, and the head 110 is provided with the following parts:
the ear portion 111 is provided with a voice acquisition device.
It should be noted that the voice acquisition device is used for acquiring voice information of the user, and the voice information is used for characterizing psychological consultation problems of the user. In other words, the user can present his psychological consultation questions by means of voice. Alternatively, the voice acquisition device may be any device for collecting voice information.
The eye portion 112 is provided with a camera.
It should be noted that the camera is used for acquiring a facial image of a user. Alternatively, the eye site 112 of the present application may be provided with other image capturing devices other than cameras.
A mouth portion 113 is provided with a playback device.
It should be noted that the playing device plays a role of playing psychological consultation advice matched with the psychological consultation problem. Optionally, the playing device in the embodiment of the present invention is a speaker device. In other embodiments of the invention the playback device may be other devices for playing back sound.
The trunk portion 120 is provided with a hand 121. The surface of the hand 121 is provided with a response button 122.
The response button 122 is connected to the arithmetic device, and the response button 122 functions to transmit a pressing instruction to the arithmetic device when pressed by the user. And the operation device starts to perform the operation of the counseling advice of the psychological counseling question in response to the pressing instruction.
The lower limb portion 130 includes a foot portion 131, and the foot portion 131 is provided with casters. Casters are used for the mobile psychological counseling robot 100.
In the embodiment of the invention, the operation device is respectively connected with the voice acquisition equipment, the camera and the playing equipment. The first function of the computing device is to acquire face images and voice information of the user in response to a pressing instruction of the user. Meanwhile, the operation device is also connected with an external electroencephalogram signal acquisition device 200. The second function of the computing device is to acquire the electroencephalogram signal of the user. The third function of the operation device is to perform the operation of the counseling advice of the psychological counseling problem when the corresponding electroencephalogram signal, facial image and voice information are obtained, so as to obtain the counseling advice corresponding to the problem and control the playing equipment, so that the playing equipment plays the counseling advice to complete the interaction with the user.
Specifically, the arithmetic device of the present invention is provided with the following modules and units:
and a response acquisition module for acquiring brain electrical signals, facial images and voice information in response to a user's pressing instruction of the response button 122.
And the image generation module is used for generating a psychological image of the user according to the face image and the plurality of electroencephalogram signals.
And the emotion recognition module is used for carrying out emotion recognition on the user psychological image by utilizing the first neural network model to obtain a user emotion mark.
The suggestion processing module is used for carrying out voice recognition on voice information to generate corresponding psychological consultation questions, traversing a preset answer corpus, and selecting a plurality of psychological consultation answers matched with the psychological consultation questions from the answer corpus.
And the advice output module is used for selecting an optimal answer from a plurality of psychological consultation answers by using the emotion identification of the user, outputting the optimal answer as the psychological consultation advice, and controlling the loudspeaker equipment to play the psychological consultation advice.
It should be noted that, the optimal answer is the psychological consultation answer with the highest adaptation degree with the emotion mark of the user.
A psychological counseling system according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings.
Referring to fig. 2 and 3, the psychological consulting system provided by the present invention includes: the psychological counseling robot 100 and the electroencephalogram signal acquiring apparatus 200 described above. The acquisition device 200 is used for acquiring the electroencephalogram signals of the user and transmitting the electroencephalogram signals to the computing device positioned on the robot head 110.
In order to reduce the computation load of the acquisition device 200 and the computation device, the present invention sets the data transmission method of the acquisition device 200 and the computation device to be a bluetooth transmission method. Bluetooth (Bluetooth) is a wireless technology standard that enables short-range data exchange between fixed devices, mobile devices and the building personal area network. Bluetooth uses UHF radio waves in the ISM band of 2.4-2.4815 GHz.
Further, the number of the acquisition devices 200 of the embodiment of the present invention is preferably one. The acquisition device 200 includes an earplug 210, the earplug 210 being adapted to be inserted into an ear of a user. End 212 of earplug 210 is provided with a multi-turn flexible electrode array. The multiple turns of the flexible electrode array are equidistantly distributed and surround the end 212 of earplug 210. The flexible electrode array includes a plurality of punctate brain electrodes 211. The plurality of punctiform brain electrodes 211 are uniformly arranged, the distances between two adjacent punctiform brain electrodes 211 are the same, and the distances between two adjacent flexible electrode arrays are the same.
In the embodiment of the present invention, the user needs to wear the acquisition device 200 and insert the earplug 210 into the ear when performing psychological consultation. The flexible electrode array is used for collecting the electroencephalogram signals of the user. Each punctiform brain electrode 211 is used as an acquisition channel, and a plurality of acquisition channels are used for acquiring multi-channel brain electrical signals so as to ensure the accuracy and diversity of the brain electrical signals. A control module is installed in the earplug 210, and the control module is used for transmitting a plurality of brain electrical signals to an operation device of the robot.
It should be noted that each electroencephalogram signal carries its corresponding channel identifier.
Further, the end 212 of the earplug 210 is a sphere, the body 213 is a cylinder, and the sphere is disposed at one end of the cylinder. A three-dimensional coordinate system is constructed by taking the center of the body 213 of the earplug 210, which is close to one surface of the sphere, as an origin, and the punctiform brain electrodes 211 contained in each flexible electrode array correspond to coordinate positions under the three-dimensional coordinate system, and the coordinate positions are to be used as channel marks of the punctiform brain electrodes 211. When the control module acquires the electroencephalogram signals of the plurality of punctiform brain electrodes 211, the control module synchronously detects the channel identifications of the punctiform brain electrodes 211 with the electroencephalogram signals acquired, embeds the channel identifications into the electroencephalogram signals and transmits the channel identifications to the operation device of the robot.
Further, the acquisition device 200 also includes an arcuate ear hook 220, the arcuate ear hook 220 being configured to hang on the back of the pinna of the user.
Based on the psychological counseling robot and the psychological counseling system, the embodiment of the invention also provides a control method of the psychological counseling system, and the control method is executed by an operation device of the robot. A control method of a psychological counseling system according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings.
The method for controlling the psychological consultation system provided by the embodiment of the invention can be applied to the terminal, the server, software running in the terminal or the server and the like. The terminal may be, but is not limited to, a tablet computer, a notebook computer, a desktop computer, etc. The server may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, basic cloud computing services such as big data and artificial intelligent platforms.
Referring to fig. 4, the method mainly includes the steps of:
s100, responding to a pressing instruction of a response button, acquiring voice information and face images of a user, and acquiring brain electrical signals of the user.
In the step, the facial image and psychological consultation problem of the user are acquired through the psychological consultation robot in the system, the multichannel electroencephalogram signals of the user are detected through the electroencephalogram signal acquisition device in the system, and the multimodal data can be used for representing the emotion of the user.
S200, generating a user psychological image according to the facial image and the brain electrical signal.
In the related art, a user may intentionally hide his emotion when performing psychological consultation, and his facial expression may not accurately reflect the current true emotion of the user. In order to capture the true emotion of the user, the face image of the user and the multichannel electroencephalogram signals are fused into the image, one-dimensional information is reconstructed under the condition of keeping the time dependence of the data, and the multichannel electroencephalogram signals are upscaled into two-dimensional images, so that the advantages of the convolutional neural network in image processing are brought into play, the true emotion expression of the user is captured through the convolutional neural network in the follow-up process, and the implicit characteristic information in the data can be better discovered.
S300, carrying out emotion recognition on the user psychological image by using the first neural network model to obtain a user emotion identification.
The step combines the deep learning algorithm to identify and obtain the true emotion expression of the user so as to conveniently screen the optimal answer from a plurality of psychological consultation answers later, and the true emotion expression is used as the screening basis of the answer.
S400, performing voice recognition on the voice information to generate corresponding psychological consultation questions, traversing a preset answer corpus, and selecting a plurality of psychological consultation answers matched with the psychological consultation questions from the answer corpus.
It should be noted that the answer corpus is a pre-constructed corpus, and includes a plurality of psychological consultation suggestions related to psychological consultation questions.
S500, selecting an optimal answer from a plurality of psychological consultation answers according to the emotion identification of the user, outputting the optimal answer as a psychological consultation suggestion, and controlling a loudspeaker device to play the psychological consultation suggestion.
The method comprises the step of selecting an optimal answer by using a mode of searching the optimal solution by using a genetic algorithm.
With reference to fig. 5, in one embodiment of the present application, the implementation procedure of S200 will be further described and illustrated below. S200 may include, but is not limited to, the following steps.
S210, preprocessing the electroencephalogram signals and the facial images.
Further, the method for preprocessing the facial image mainly comprises the following steps:
s211, graying the face image, displaying the face image information by using the gray value as a variable, and generating a gray face image.
The gray value of the changed point is obtained by the following formula:
where Gray (x, y) represents the grayed-out face image, and R (x, y), G (x, y), and B (x, y) represent the red, green, and blue channels of the original face image, respectively.
S212, carrying out equalization processing and normalization processing on the gray face image;
s213, filtering noise of the normalized gray-scale face image through a Gaussian filter to obtain a preprocessed face image.
It should be noted that gaussian filtering (gaussian filtering) is a linear smoothing filtering, which is suitable for eliminating gaussian noise, and is widely used in a noise reduction process of image processing. In popular terms, gaussian filtering is a process of weighted averaging over the entire image, where the value of each pixel is obtained by weighted averaging itself and other pixel values in the neighborhood. The specific operation of gaussian filtering is to scan each pixel in the image with a template (or convolution, mask), and replace the value of the center pixel point of the template with the weighted average gray value of the pixels in the neighborhood determined by the template.
Further, the method for preprocessing the electroencephalogram signals mainly comprises the following steps:
s214, acquiring and detecting a plurality of electroencephalograms, wherein the electroencephalograms all carry channel identifiers, and distinguishing and classifying the plurality of electroencephalograms according to the channel identifiers when the channel identifiers are detected, so as to generate a plurality of single-channel electroencephalograms;
s215, performing artifact removal processing on each single-channel electroencephalogram signal to obtain a plurality of preprocessed electroencephalogram signals.
In the art, an electroencephalogram signal is a weak time-varying nonlinear non-stationary physiological electric signal and is extremely easy to interfere, so that the electroencephalogram signal can be influenced by a plurality of different artifacts in the acquisition process, and the artifacts are mainly divided into non-physiological artifacts and physiological artifacts. The non-physiological artifacts mainly comprise 50 Hz power frequency interference, and further comprise a series of interference caused by external sources, such as baseline drift, poor electrode contact, cable motion, mutual electromagnetic interference among devices, noise of electronic components of an amplifier and the like. Physiological artifacts are mainly caused by a series of human internal sources such as eye movements (e.g., eye movements, blinks), muscle movements (e.g., muscle contraction and relaxation, respiratory contraction), heart movements, etc. For non-physiological artifacts, filtering or stuffing wave mode is generally adopted for processing; for the physiological artifacts, the physiological artifacts can be introduced at any time, any frequency band and any electrode during the electroencephalogram acquisition, and the frequency band range is similar to the electroencephalogram containing effective information and can be simultaneously aliased in the electroencephalogram. Therefore, the generation of physiological artifacts cannot be avoided by means of filtering, environment control and the like, and proper electroencephalogram artifact removal algorithm is needed to be selected for processing.
Furthermore, the method for removing the artifacts of the electroencephalogram signals comprises the following steps:
s2151, performing ICA analysis on the single-channel brain electrical signals by using FastICA algorithm, and converting the single-channel brain electrical signals to generate a plurality of independent components.
The independent component analysis (Independent Components Analysis, ICA) method is a general method for realizing blind source separation. The basic principle of ICA is to solve the mixed signal and approximate the estimated component of the real source data as much as possible through the optimization algorithm on the premise that the non-Gaussian property and mutual statistical independence of the source signals are assumed and the acquired brain electrical data are linearly combined by the source signals.
S2152, calculating the artifact duty ratio of each independent component, regarding each independent component as an artifact component when the artifact duty ratio is larger than or equal to a preset threshold value, and screening all artifact components and non-artifact components;
s2153, performing artifact removal operation on the artifact components by using an empirical mode decomposition method, and performing signal reconstruction on the artifact components and the non-artifact components which remain after the artifact removal operation to obtain a reconstructed single-channel electroencephalogram signal.
It should be noted that the empirical mode decomposition (Empirical Mode Decomposition, EMD) is based on data driving, has good data adaptation and does not need to select a basis function, and can adaptively decompose a signal into a set of linear combinations of eigenmode functions (Intrinsic Mode Function, IMF) with physical significance according to signal characteristics.
S220, extracting features of the preprocessed face image and the preprocessed electroencephalogram signals to generate a one-dimensional face feature vector and a plurality of one-dimensional electroencephalogram feature vectors.
Optionally, the method for extracting the characteristics of the electroencephalogram signal includes any one of a time domain analysis method, a frequency domain analysis method, or a time-frequency analysis method. The time domain analysis method mainly extracts information of waveforms changing with time as characteristics; the frequency domain analysis method is to convert a time domain waveform into a frequency spectrum of a frequency domain by using methods such as Fourier transform; the time-frequency analysis method combines a time domain analysis method and a frequency domain analysis method, the electroencephalogram signal has time-varying and non-stationary characteristics, and the uncertainty is realized by simply adopting the time domain analysis method or the frequency domain analysis method. In order to further strengthen the characteristics, the time domain and the frequency domain are combined to extract the characteristics so as to improve the certainty of the information.
Preferably, the invention selects a time-frequency analysis method to extract the characteristics of the brain electrical signals, in particular to wavelet transformation.
S230, generating a user psychological image by using the face characteristic vector and the plurality of brain electricity characteristic vectors.
Referring to fig. 6, in some embodiments of the present invention, S230 is a step of an imaging process. In order to comprehensively capture the true emotion of the user, the invention takes the emotion information represented in the facial image as a basis and takes the emotion information represented in the electroencephalogram signal as a supplement to carry out imaging processing. The specific implementation process of S230 may include the following steps:
S231, dividing all the electroencephalogram feature vectors into corresponding channel intervals according to the channel identification of each electroencephalogram signal to form an electroencephalogram feature sequence of each channel interval, and further generating a first electroencephalogram feature sequence, a second electroencephalogram feature sequence and a third electroencephalogram feature sequence.
It should be noted that the number of the channel sections is three, that is, the channel sections include a first channel section, a second channel section and a third channel section, and all the electroencephalogram feature vectors of the present invention are divided into one of the three channel sections according to the channel identifications thereof, so as to form an electroencephalogram feature sequence corresponding to the section.
It should be noted that, the number of the electroencephalogram feature vectors included in each channel interval is the same, and the number is constant as n.
Optionally, the method for dividing the channel interval adopts a random method to divide three channel intervals according to all current channel identifications. In other embodiments of the present invention, the lane intervals may be partitioned according to a numbered ordering of lane identifications. For example, the number of channels is 30, and the brain electrical characteristic vector with the number of channels being 1 to 10 is divided into a first channel interval; the electroencephalogram feature vectors with the channel marks of 11 to 20 are divided into a second channel interval; the electroencephalogram feature vectors with channel identifications 21 to 30 are divided into third channel intervals.
S232, respectively fusing the facial feature vectors into each electroencephalogram feature sequence to obtain fused feature sequences.
Specifically, feature fusion is carried out on each electroencephalogram feature vector in the first electroencephalogram feature sequence and a face feature vector respectively, and a plurality of fused vectors are generated and form a first fusion feature sequence. Similarly, a second fused feature sequence and a third fused feature sequence are generated.
Optionally, the feature fusion mode includes a cascade fusion mode, a feature dimension splicing mode, and the like, and the feature fusion mode is shown in the following formula:
wherein: fv is a characteristic vector of a human face,for the ith brain electrical characteristic sequence, +.>For the ith fusion signature, i=1, 2,3, t denotes the number of channel identifications.
S233, generating a user psychological image according to the first fusion characteristic sequence, the second fusion characteristic sequence and the third fusion characteristic sequence.
Further, the generating process of the psychological image of the user comprises the following steps:
and S2331, generating a red channel diagram through a recursion diagram according to the first fusion characteristic sequence.
A Recurrence Plot (RP) reflects the Recurrence characteristics of the time series by qualitatively describing the distance between vectors in Gao Weixiang space, and reflects the similarity and Recurrence of the time series.
Specifically, the generation process of the red channel map may include the steps of:
first, for a given sequenceAnd (3) reconstructing a phase space:
wherein n=n- (a-1) b, a is the embedding dimension, b is the delay channel and is a constant value,the result of the phase space reconstruction is represented as a two-dimensional phase space trajectory.
Then, the euclidean norm of each vector in the phase space trajectory is calculated according to the euclidean theorem, as shown in the following equation:
，/>i.e. euclidean norms. Wherein: u, v=1, 2, …, N.
Finally, a recursive graph is constructed from the euclidean norms of each vector, as shown in the following equation, and a red channel map is generated from the constructed recursive graph:wherein: />For the Herveledy function, < >>Is a distance threshold.
And S2332, generating a green channel map through the relative position matrix map according to the second fusion characteristic sequence.
Specifically, the generation process of the green channel map may include the steps of:
first, a given sequence is to be determinedNormalization processing is carried out to obtain a standard normal distribution sequenceThe normalization process is as follows:
wherein,and->Respectively the sequences->Mean and standard deviation of>For the sequence->I data of (a) is provided.
Then, a relative position matrix is constructed, and the relative positions between the two channels are calculated to realize the sequence Conversion to a two-dimensional matrix is shown by the following formula:
the relative position matrix has a size ofAny two channels are correlated, and each row and each column in the matrix Q takes a certain channel as a reference point, and contains the information of the whole sequence. Each row of matrix Q shows a sequence with a different reference channel, and each column shows a mirror image of the former, with the information of the sequence being viewable by reverse vision. In practice, the relative position matrix can be seen as a method of data enhancement, which improves the generalization ability of the sequence by providing redundant features.
And finally, converting the matrix Q into a gray value matrix by a min-max normalization method, generating a final relative position matrix, and generating a green channel diagram by using the relative position matrix.
And S2333, generating a blue channel graph through a gram sum angle field matrix according to the third fusion characteristic sequence.
The one-dimensional signal can be converted into a two-dimensional image sequence by using a gram angle field (Gramian Angular Field, GAF). According to the encoded form of GAF, it can be divided into gladhand angular fields (Gramian Angular Summation Field, GASF) and gladhand angular fields (Gramian Angular Difference Field, GADF).
Specifically, the generation process of the blue channel map may include the steps of:
first, a third fused feature sequenceZoom to [0,1 ]]To obtain the scaled third fusion feature sequence +.>。
The scaled third fused feature sequence is then transformed into a polar coordinate system for mapping, with one result in polar coordinates and one inverse map unique for a given third fused feature sequence. The polar coordinate system is represented by the tailpiece angle and the radius.
Note that the cosine angle is expressed by the following formula:
wherein:and->Expression sequence->The feature vector of the i-th position in (a).Cosine angle representing polar coordinate representation, whose value range satisfies +.>And->。
Thereafter, a gram and angle field matrix is constructed from the cosine angles described above, as follows:
。
in the step, the correlation of different channels is represented by calculating the sum value of each residual chord angle, meanwhile, in the polar coordinates, the matrix G represents the channel correlation between each pair of points through superposition of nonlinear cosine functions, different pairs of points have different characteristics, after the field of the Gellam angles is reconstructed, the characteristics in the third fusion characteristic sequence are all enhanced by data, and further, the characteristic difference between different channels can be more highlighted.
And S2334, overlapping and recombining the red channel diagram, the green channel diagram and the blue channel diagram to obtain a psychological image of the user.
The embodiment of the invention ensures the diversity of the brain electrical signals through a plurality of channels, and when converting the mental images of the users, the application adopts a split-channel processing mode, and synthesizes the channel images of the three primary colors of red, green and blue through different channels, thereby synthesizing the mental images of the users so as to ensure the diversity of the brain electrical characteristics and the information characterized by the characteristics to the greatest extent.
In one embodiment of the present application, the implementation procedure of S300 will be further described and illustrated below. S300 may include, but is not limited to, the following steps.
S310, carrying out emotion recognition on the user psychological image by using the first neural network model, and outputting an initial recognition result.
In the embodiment of the invention, the first neural network model is a model obtained through training a preset face database. The preset face database comprises a plurality of sample face images, each sample face image carries an emotion mark, and the emotion mark comprises any one of happiness, frightening, endanger, melancholy, anger or frightening. The first neural network model learns the relevance of the sample face image characteristics and the emotion marks through the face database, and further realizes emotion recognition.
It should be noted that the first recognition result includes any one of happiness, shock, hate, depression, anger, or panic.
Optionally, the first neural network model is a model based on an Adaboost algorithm.
S320, performing emotion detection on the electroencephalogram signals by using the second neural network model to obtain a corrected recognition result.
In the embodiment of the invention, the second neural network model is a model obtained through training a preset electroencephalogram database. The preset electroencephalogram database comprises a plurality of sample electroencephalograms and emotion marks corresponding to the sample electroencephalograms, the emotion marks of the sample electroencephalograms are different from those of the face database, the six emotions are mapped, the fluctuation degrees of the emotions are mapped, and the fluctuation degrees of the emotions can be classified into strong degrees and normal degrees. That is, the emotional markers of the sample brain signals include twelve markers of intense and normal happiness, convulsion, hate, depression, anger, or panic. The second neural network model learns the relevance between the sample brain electrical signals and the emotion marks corresponding to the sample brain electrical signals through the brain electrical database, and further emotion detection is achieved.
It should be noted that the second recognition result includes any one of strong happiness, normal happiness, strong shock, normal shock, strong back-fire, normal back-fire, strong depression, normal depression, strong anger, normal anger, strong panic, or normal panic.
S330, fusing the initial recognition result and the corrected recognition result to obtain a final emotion recognition result, namely the emotion identification of the user.
It should be noted that the user emotion mark includes any one of strong happiness, normal happiness, strong shock, normal shock, strong back, normal back, strong depression, normal depression, strong anger, normal anger, strong panic, or normal panic.
In the step, expression parameters are calculated through a first recognition result, the internal emotion of a person can be subjected to surface mapping through the expression, the intensity of the internal emotion is calculated through a second recognition result, the calculation results are subjected to information fusion to gather the information, the real emotion of a final user is given out, and the real emotion is output as a user emotion mark.
In one embodiment of the present application, the implementation procedure of S400 will be further described and illustrated below. S400 may include, but is not limited to, the following steps.
S410, performing voice recognition on the voice information to generate corresponding text information, wherein the text information is the psychological consultation problem.
Optionally, the voice information is decomposed into a plurality of IMF components by an EMD method, correlation coefficients between each IMF component and the input voice information are calculated, and a certain number of effective IMF components are screened out of all IMF components by the correlation coefficients of each IMF component. And then, extracting the characteristics of the screened IMF, wherein the specific process is as follows: and carrying out short-time Fourier transform, mel triangular filter bank filtering, logarithmic transformation and first-order difference solving on the voice signal formed by the IMF components to obtain the difference characteristics of a plurality of effective IMF components. And finally, splicing all the effective IMF components and the differential characteristics thereof to obtain the Flank spectrogram characteristics. And finally, carrying out voice recognition according to the feature of the Flank spectrogram to generate corresponding text information.
S420, constructing an answer corpus in advance, wherein the answer corpus comprises sample data of various psychological counseling questions and counseling suggestions corresponding to the sample data, calculating the correlation between each sample data and the psychological counseling questions, and selecting the counseling suggestions corresponding to the sample data with the highest correlation as a plurality of matched psychological counseling answers.
Alternatively, the correlation may be calculated by pearson coefficients (Pearson correlation coefficient).
Based on the above embodiment, in order to select the most suitable psychological consulting answer from the plurality of answers obtained by traversing the corpus, the present invention adopts a genetic algorithm to select the optimal solution, i.e., the optimal psychological consulting answer, from the plurality of answers.
With reference to fig. 7, in one embodiment of the present application, the implementation procedure of S500 will be further described and illustrated below. S500 may include, but is not limited to, the following steps.
S510, coding: and coding a plurality of psychological consultation answers according to a preset coding mode.
S520, initializing a population: initializing a plurality of psychological consultation answers to generate a first generation population.
Specifically, the population initialization mainly comprises the following steps:
and carrying out semantic emotion recognition on each psychological consultation answer through an emotion recognition algorithm to obtain emotion recognition results corresponding to each psychological consultation answer. Wherein the emotion recognition result includes any one of strong happiness, normal happiness, strong shock, normal shock, strong hate, normal hate, strong depression, normal depression, strong anger, normal anger, strong panic, or normal panic. And initializing all psychological consultation answers so that each psychological consultation answer carries emotion identification and generates a first generation population. Each primary population comprises a plurality of individuals, namely chromosomes, and the individuals correspondingly represent a psychological consultation problem, and the optimal individuals are selected from all the individuals through selection, crossing and mutation.
S530, constructing a fitness function according to the emotion marks of the users.
The invention constructs fitness functions by similarity between the user's emotional identity and the emotional identity of each chromosome. Specifically, the user emotion marks and the emotion marks of each chromosome are converted into character strings, the similarity between the user emotion marks and the emotion marks of each chromosome in the character strings is calculated through hamming distances, and the fitness value is calculated. Namely, the hamming distance calculation formula is the fitness function.
It should be noted that, the fitness value of the population individuals is calculated through the fitness function, and the smaller the fitness value is, the higher the fitness of the individuals is represented, and the closer the individuals are to the requirement of the optimal solution. Finally, the individual with the smallest fitness value is the optimal solution. It should be understood that fitness values and fitness of genetic algorithms are terms of different meaning.
S540, calculating the fitness value of the individuals of the current population through the fitness function;
s550, judging whether the algorithm termination condition is satisfied. If yes, go to S560; if not, selecting, crossing and mutating individuals of the current population to generate a new population, and returning to S540;
It should be noted that, when the current fitness value is less than or equal to the preset threshold value, the algorithm is terminated. Alternatively, in other embodiments of the invention, termination criteria that pre-specify the number of evolution generations (gens) are employed to end the algorithm.
It should be noted that the application of genetic operators for selection, crossover and mutation to populations creates a new generation based on the better individuals in the current generation. Wherein: the selection (selection) operation is responsible for selecting the dominant individuals in the current population. A crossover operation, also known as a reconstruction operation, is the creation of offspring from selected individuals, typically by interchanging parts of their chromosomes by two selected individuals to create two new chromosomes representing the offspring. A mutation (mutation) operation may randomly vary one or more chromosome values (genes) for each newly created individual, with mutation typically occurring with a particular probability.
S560, taking the individual with the highest fitness in the current population as the optimal individual, decoding the optimal individual according to the rule of the preset coding mode, obtaining the answer with the highest fitness with the emotion mark of the user and outputting the answer.
It should be noted that the optimal individual is the individual with the highest adaptation degree with the emotion mark of the user.
The embodiment of the present invention also provides a computer-readable storage medium in which a program executable by a processor is stored, which when executed by the processor is used to perform a control method of a psychological consulting system as described above.
Similarly, the content in the above method embodiment is applicable to the present storage medium embodiment, and the specific functions of the present storage medium embodiment are the same as those of the above method embodiment, and the achieved beneficial effects are the same as those of the above method embodiment.
In summary, the invention has the following technical effects:
1. the robot provided by the invention takes the posture of the robot as the appearance, so that the patient has more feeling and desire to speak, the patient carries out consultation or complaint on the robot, the operation device of the robot carries out operation according to the consultation or complaint, and corresponding consultation advice is provided for the user, thereby realizing psychological consultation based on man-machine interaction.
2. According to the invention, according to the corpus and the true emotion expression of the user, the optimal psychological consultation advice is found by combining a genetic algorithm, so that the automatic generation of the psychological consultation advice is realized, the daily workload of a psychological consultation engineer is saved, convenient, timely and objective psychological consultation services are provided for the user, the problems that the psychological consultation cannot be carried out by the user due to untimely reservation, the psychological consultation cannot be carried out by the patient due to temporary cancellation of reservation by a doctor and the like are avoided, the generation accuracy of the psychological consultation advice is improved, the psychological consultation advice is more in accordance with the actual consultation requirement of the user, the objectivity and quality of the psychological consultation are ensured, and a large amount of psychological consultation requirements in the modern society are met.
3. The invention measures the true emotion expression of the user through the electroencephalogram and the facial image of the user, divides the electroencephalogram according to different channels of the electroencephalogram, fuses the divided electroencephalogram with the characteristics of the facial image to obtain three sequences for constructing a three-primary-color channel map, and respectively constructs the three-primary-color channel map according to the three sequences so as to realize imaging processing. In order to capture the true emotion of the user, the face image of the user and the multichannel electroencephalogram signals are fused into the image, one-dimensional information is reconstructed under the condition of keeping the time dependence of the data, and the multichannel electroencephalogram signals are upscaled into two-dimensional images, so that the advantages of the convolutional neural network in image processing are brought into play, the true emotion expression of the user is captured through the convolutional neural network in the follow-up process, the implicit characteristic information in the data can be better discovered, and the generation accuracy of psychological consultation suggestions is effectively improved.
In some alternative embodiments, the functions/acts noted in the block diagrams may occur out of the order noted in the operational illustrations. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved. Furthermore, the embodiments presented and described in the flowcharts of the present invention are provided by way of example in order to provide a more thorough understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed, and in which sub-operations described as part of a larger operation are performed independently.
Furthermore, while the invention is described in the context of functional modules, it should be appreciated that, unless otherwise indicated, one or more of the functions and/or features may be integrated in a single physical device and/or software module or may be implemented in separate physical devices or software modules. It will also be appreciated that a detailed discussion of the actual implementation of each module is not necessary to an understanding of the present invention. Rather, the actual implementation of the various functional modules in the apparatus disclosed herein will be apparent to those skilled in the art from consideration of their attributes, functions and internal relationships. Accordingly, one of ordinary skill in the art can implement the invention as set forth in the claims without undue experimentation. It is also to be understood that the specific concepts disclosed are merely illustrative and are not intended to be limiting upon the scope of the invention, which is to be defined in the appended claims and their full scope of equivalents.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in the form of a software product stored in a storage medium, including several programs for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
Logic and/or steps represented in the flowcharts or otherwise described herein, e.g., a ordered listing of executable programs for implementing logical functions, can be embodied in any computer-readable medium for use by or in connection with a program execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the programs from the program execution system, apparatus, or device and execute the programs. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the program execution system, apparatus, or device.
More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer readable medium may even be paper or other suitable medium on which the program is printed, as the program may be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.
It is to be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable program execution system. For example, if implemented in hardware, as in another embodiment, may be implemented using any one or combination of the following techniques, as is well known in the art: discrete logic circuits having logic gates for implementing logic functions on data signals, application specific integrated circuits having suitable combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like.
In the foregoing description of the present specification, reference has been made to the terms "one embodiment/example", "another embodiment/example", "certain embodiments/examples", and the like, means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
While embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that: many changes, modifications, substitutions and variations may be made to the embodiments without departing from the spirit and principles of the invention, the scope of which is defined by the claims and their equivalents.
While the preferred embodiment of the present invention has been described in detail, the present invention is not limited to the embodiments described above, and various equivalent modifications and substitutions can be made by those skilled in the art without departing from the spirit of the present invention, and these equivalent modifications and substitutions are intended to be included in the scope of the present invention as defined in the appended claims.
Claims (5)
1. A control method of a psychological counseling system, characterized in that it is applied to a psychological counseling system, the system comprising: the psychological consultation robot and the electroencephalogram signal acquisition device are characterized in that the robot is in a human shape, the robot comprises a head, a trunk part and a lower limb part, an ear part, an eye part and a mouth part are arranged on the head, the trunk part is provided with a hand part, the ear part is provided with voice acquisition equipment, the eye part is provided with a camera, the mouth part is provided with speaker equipment, a response button is arranged on the surface of the hand part, a psychological consultation operation device is arranged in the head, the psychological consultation operation device is respectively connected with the voice acquisition equipment, the camera, the response button and the speaker equipment, and the psychological consultation operation device is also connected with the electroencephalogram signal acquisition device; the electroencephalogram signal acquisition device is used for acquiring a plurality of electroencephalogram signals of a user and transmitting the electroencephalogram signals to the psychological consultation robot; the electroencephalogram signal acquisition device comprises an earplug, the earplug is used for being plugged into the ear of a user, a plurality of circles of flexible electrode arrays are distributed at the end part of the earplug, each flexible electrode array comprises a plurality of dot brain electrodes which are uniformly arranged, a control module is arranged in the earplug, each flexible electrode array is used for acquiring electroencephalogram signals of the user, and each control module is used for transmitting the acquired plurality of electroencephalogram signals to the psychological consultation operation device; each punctiform brain electrode is used as an acquisition channel, multichannel brain electrical signals are acquired through a plurality of acquisition channels, and each brain electrical signal carries a corresponding channel identifier;
The method is executed by the psychological consultation operation device, and comprises the following steps:
s100, responding to a pressing instruction of a response button, acquiring voice information of a user through a voice acquisition device, acquiring facial images of the user through a camera device, and acquiring a plurality of electroencephalogram signals of the user; wherein, the voice information characterizes psychological consultation problems of users;
s200, generating a user psychological image according to the face image and a plurality of the brain electrical signals;
s300, carrying out emotion recognition on the user psychological image by using a first neural network model to obtain a user emotion mark;
s400, performing voice recognition on the voice information to generate corresponding psychological consultation questions, traversing a preset answer corpus, and selecting a plurality of psychological consultation answers matched with the psychological consultation questions from the answer corpus;
s500, selecting an answer with highest adaptation degree with the user emotion mark from a plurality of psychological consultation answers according to the user emotion mark, outputting the selected psychological consultation answer as a psychological consultation suggestion, and controlling a loudspeaker device to play the psychological consultation suggestion;
wherein S200 specifically includes:
Preprocessing a plurality of the brain electrical signals and the face image;
extracting features of the preprocessed face image and the preprocessed electroencephalogram signals to generate a face feature vector and a plurality of electroencephalogram feature vectors;
generating a user psychological image by using the face feature vector and a plurality of the brain electrical feature vectors;
wherein the generating a user psychological image by using the face feature vector and the plurality of electroencephalogram feature vectors comprises:
dividing all the electroencephalogram feature vectors into corresponding channel intervals according to the channel identification of each electroencephalogram signal to form an electroencephalogram feature sequence of each channel interval, and further generating a first electroencephalogram feature sequence, a second electroencephalogram feature sequence and a third electroencephalogram feature sequence;
respectively fusing the face feature vectors into the first electroencephalogram feature sequence, the second electroencephalogram feature sequence and the third electroencephalogram feature sequence to obtain a first fused feature sequence, a second fused feature sequence and a third fused feature sequence;
generating a red channel map through a recursion map according to the first fusion feature sequence;
generating a green channel diagram through a relative position matrix diagram according to the second fusion characteristic sequence;
Generating a blue channel diagram through a gram sum angle field matrix according to the third fusion characteristic sequence;
and overlapping and recombining the red channel diagram, the green channel diagram and the blue channel diagram to obtain a user psychological image.
2. The control method of a psychological counseling system as set forth in claim 1, wherein said electroencephalogram signal acquisition means further comprises an arc-shaped ear hook for hanging on the back of the auricle of the user.
3. The control method of a psychological consulting system of claim 1, wherein said first neural network model is an Adaboost algorithm-based model.
4. The control method of psychological consulting system as set forth in claim 1, wherein S500 specifically includes:
coding a plurality of psychological consultation answers according to a preset coding mode, and initializing to generate a first generation population;
calculating the fitness value of individuals of the current population;
judging that an algorithm termination condition is met; if yes, entering the next step; if not, selecting, crossing and mutating individuals of the current population to generate a new population, and returning to the previous step;
and taking the individual with the highest fitness in the current population as an optimal individual, wherein the optimal individual is the individual with the highest fitness with the user emotion mark, decoding the optimal individual according to a rule of a preset coding mode, obtaining an answer with the highest fitness with the user emotion mark, and outputting the answer.
5. A computer-readable storage medium in which a processor-executable program is stored, wherein the processor-executable program, when executed by a processor, is for implementing a control method of a psychological consulting system as set forth in any one of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310743021.1A CN116872222B (en) | 2023-06-21 | 2023-06-21 | Psychological consultation robot, psychological consultation system, psychological consultation control method and psychological consultation storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310743021.1A CN116872222B (en) | 2023-06-21 | 2023-06-21 | Psychological consultation robot, psychological consultation system, psychological consultation control method and psychological consultation storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116872222A CN116872222A (en) | 2023-10-13 |
| CN116872222B true CN116872222B (en) | 2024-02-06 |
Family
ID=88268934
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310743021.1A Active CN116872222B (en) | 2023-06-21 | 2023-06-21 | Psychological consultation robot, psychological consultation system, psychological consultation control method and psychological consultation storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116872222B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105364931A (en) * | 2015-11-30 | 2016-03-02 | 北京光年无限科技有限公司 | Emotion information processing method and small intelligent robot |
| CN106855879A (en) * | 2016-12-14 | 2017-06-16 | 竹间智能科技（上海）有限公司 | The robot that artificial intelligence psychology is seeked advice from music |
| CN106933345A (en) * | 2017-01-18 | 2017-07-07 | 北京光年无限科技有限公司 | For the multi-modal exchange method and device of intelligent robot |
| CN109426653A (en) * | 2017-08-27 | 2019-03-05 | 南京乐朋电子科技有限公司 | Psychological consultation robot |
| CN115512702A (en) * | 2022-10-11 | 2022-12-23 | 徐州工程学院 | Accompanying robot and emotion recognition system |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7706866B2 (en) * | 2004-06-24 | 2010-04-27 | Cardiac Pacemakers, Inc. | Automatic orientation determination for ECG measurements using multiple electrodes |
| US20070219958A1 (en) * | 2006-03-20 | 2007-09-20 | Park Joseph C | Facilitating content generation via participant interactions |
-
2023
- 2023-06-21 CN CN202310743021.1A patent/CN116872222B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105364931A (en) * | 2015-11-30 | 2016-03-02 | 北京光年无限科技有限公司 | Emotion information processing method and small intelligent robot |
| CN106855879A (en) * | 2016-12-14 | 2017-06-16 | 竹间智能科技（上海）有限公司 | The robot that artificial intelligence psychology is seeked advice from music |
| CN106933345A (en) * | 2017-01-18 | 2017-07-07 | 北京光年无限科技有限公司 | For the multi-modal exchange method and device of intelligent robot |
| CN109426653A (en) * | 2017-08-27 | 2019-03-05 | 南京乐朋电子科技有限公司 | Psychological consultation robot |
| CN115512702A (en) * | 2022-10-11 | 2022-12-23 | 徐州工程学院 | Accompanying robot and emotion recognition system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116872222A (en) | 2023-10-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Liu et al. | Emotion recognition by deeply learned multi-channel textual and EEG features | |
| JP7337953B2 (en) | Speech recognition method and device, neural network training method and device, and computer program | |
| Alhussein et al. | EEG pathology detection based on deep learning | |
| CN111461176B (en) | Multi-mode fusion method, device, medium and equipment based on normalized mutual information | |
| CN109199414B (en) | Audio-visual evoked emotion recognition method and system based on electroencephalogram signals | |
| Hamedi et al. | EMG-based facial gesture recognition through versatile elliptic basis function neural network | |
| US10878818B2 (en) | Methods and apparatus for silent speech interface | |
| CN110059575A (en) | A kind of augmentative communication system based on the identification of surface myoelectric lip reading | |
| Bourbakis et al. | Extracting and associating meta-features for understanding people’s emotional behaviour: face and speech | |
| WO2015158017A1 (en) | Intelligent interaction and psychological comfort robot service system | |
| Chen et al. | Emotion recognition with audio, video, EEG, and EMG: a dataset and baseline approaches | |
| Hsu | Brain–computer interface connected to telemedicine and telecommunication in virtual reality applications | |
| WO2021203880A1 (en) | Speech enhancement method, neural network training method, and related device | |
| Hussain et al. | A radial base neural network approach for emotion recognition in human speech | |
| Tang et al. | A hybrid SAE and CNN classifier for motor imagery EEG classification | |
| Tang et al. | Motor imagery EEG decoding based on multi-scale hybrid networks and feature enhancement | |
| CN113749656B (en) | Emotion recognition method and device based on multidimensional physiological signals | |
| Altaheri et al. | Dynamic convolution with multilevel attention for EEG-based motor imagery decoding | |
| Mobin et al. | Voice conversion using convolutional neural networks | |
| Kim et al. | Diff-E: Diffusion-based learning for decoding imagined speech EEG | |
| CN116872222B (en) | Psychological consultation robot, psychological consultation system, psychological consultation control method and psychological consultation storage medium | |
| Wilroth et al. | Improving EEG-based decoding of the locus of auditory attention through domain adaptation | |
| Dang et al. | Motor imagery EEG recognition based on generative and discriminative adversarial learning framework and hybrid scale convolutional neural network | |
| Bang et al. | Interpretable convolutional neural networks for subject-independent motor imagery classification | |
| Tang et al. | Eye movement prediction based on adaptive BP neural network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |