JP2023000524A - Information management system - Google Patents

Abstract

To provide an information processing system capable of enlarging the usage range of a system that can recognize the heart rate of a subject.SOLUTION: In an information processing system 100, an information storage part 48 stores a word contained in a voice uttered by a subject, and information correlated with emotion that represents a stress state and interest/concern state of the subject determined at timing when the subject has uttered the aforementioned word. An emotion determination part 47 issues a notification to staff of a facility to urge a support for the subject in a case where the number of times of determination of the stress state within a given period is equal to or more than a given number of times, and in a case where the interest/concern state is determined, introduces another subject who has a history of uttering a voice similar to the word contained in the voice uttered by the subject when the aforementioned determination is made.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information management system. In particular, the present invention relates to a system that manages information about a subject (a person targeted by the system) and that manages information related to the subject's emotions.

Conventionally, as disclosed in Patent Document 1, the heartbeat of a person (subject) captured by a camera is recognized from the face image of the subject, and the balance of the activity of the sympathetic nervous system is determined from the period of heartbeat fluctuation. A system is known for making a judgment and determining a subject's stress state.

In Patent Document 1, heartbeat interval data (data relating to periodic heartbeat intervals) is created from acquired face image data of a subject, and heartbeat variability spectrogram data is obtained by performing frequency conversion on the heartbeat interval data. and discriminate between a subject's resting state and stress state based on the value of the heart rate variability frequency.

JP 2017-29318 A

The inventors of the present invention considered expanding the range of use of this type of system. The inventors of the present invention have also found that this type of system is capable of obtaining the heart rate and heart rate variability of the subject, and that there is a correlation between these heart rate and heart rate variability and the subject's emotions. Focusing on a certain point, we considered how to expand the range of use of the system using these.

The present invention has been made in view of the above points, and its object is to expand the range of use of a system capable of recognizing the heartbeat of a subject.

The solution of the present invention to achieve the above object is directed to an information management system for managing information related to a subject's emotions. This information management system is characterized by comprising photographing means, voice acquisition means, word extraction section, heart rate measurement section, emotion determination section, and information storage section. The photographing means is capable of photographing the target person. The voice acquisition means can acquire the voice uttered by the subject. A word extraction unit extracts words included in the voice from the voice acquired by the voice acquisition means. The heartbeat measuring unit measures the heartbeat of the subject based on the image information of the subject photographed by the photographing means. The emotion determination unit detects at least a stress state among various emotions of the subject from information on at least one of heart rate variability and heart rate calculated based on information on the heart rate of the subject measured by the heart rate measurement unit. and interest state. The information storage unit stores information that associates the word extracted by the word extraction unit with the subject's emotion determined by the emotion determination unit at the timing when the subject utters the word.

According to this specific matter, at least one of heart rate variability and heart rate information calculated based on the heart rate information of the subject measured by the heart rate measuring unit indicates that the current emotion of the subject is at least the state of stress and interest. It is determined which of the states it is in. Then, when the target person utters a voice at the timing when the target person's emotion such as stress state or interest state is determined, the words included in the voice and the target person's emotion are associated. Information is stored in the information storage unit. As a result, the relationship between the words included in the voice uttered by the subject and the emotion of the subject can be grasped. For this reason, it is possible to grasp the situation in which the subject feels stressed based on the word, the interest and the matter of concern, etc., and it is possible to effectively provide support to the subject according to that, and the system can be improved. It is possible to expand the range of use.

At least the photographing means and the sound acquisition means are mounted on an information management device, the information management device includes a moving means capable of autonomous movement, and the heartbeat measurement unit is adapted to perform photographing by the photographing means. heartbeat of the subject based on the image information of the subject's face that has been obtained, and when the subject's face is photographed by the photographing means, the information management device operates the moving means. It is designed to turn around to the front of the subject person by.

Since the information management device moves around in front of the subject in this manner, the subject's face can be easily photographed by the photographing means. Therefore, the heartbeat of the subject can be measured with high accuracy based on the image information of the subject's face.

Further, output means for externally outputting the information of the subject is provided, and the emotion determination unit determines that the subject is in a stress state when the heart rate variability is equal to or less than a predetermined threshold, and determines that the subject is in a stress state. When the determination of the stress state is made more than a predetermined number of times during the predetermined period, the output means outputs information indicating that the subject is in the stress state.

By externally outputting information indicating that the subject is in a stressed state, it is possible to notify a person other than the subject (for example, a system administrator, etc.) that the subject is in a stressed state. Therefore, it is possible to make people other than the target person recognize that the target person who is in a stressed state needs support, and it is possible to encourage support for the target person.

The emotion determination unit determines that the subject is in a state of interest when the heart rate variability exceeds a predetermined threshold and the heart rate is equal to or greater than a predetermined threshold, and in this state of interest, When a certain thing is determined, the subject is provided with Web search information based on the word extracted by the word extraction unit or a word similar to the word at the timing when the determination is made. It's becoming

The Web search information referred to here may be information on the result of a Web search using the word or a word similar to the word, or information for assisting selection of a search word.

In this way, when the target person is in a state of interest, by providing the target person with Web search information using the word or words similar to the word, the target person can easily obtain information that interests the target person. can do.

Further, a plurality of subjects are targeted, and the words extracted by the word extraction unit are associated with the subject's emotions determined by the emotion determination unit at the timing when the subject utters the words. Information is individually stored in the information storage unit according to each of the plurality of subjects, and the emotion determining unit determines that the specific subject is in a state of interest. In this case, other subjects who have a history of uttering the words extracted by the word extraction unit at the timing of the determination, or words similar to the words, are introduced to the specific subjects. ing.

According to this, it is possible to deepen exchanges between a plurality of subjects who have common interests. As a result, it is possible to further expand the range of use of the system.

In the present invention, a word contained in a voice uttered by a subject and an emotion of the subject determined at the timing when the subject uttered the word (stress state or interest state) are combined. It is designed to store associated information. Therefore, it is possible to understand the relationship between the words contained in the voice uttered by the target person and the target person's emotions, and based on the words, the situation where the target person feels stress, the interest, etc. It is possible to effectively support the target person accordingly. As a result, it is possible to expand the range of use of the system.

1 is a block diagram showing the configuration of an information management device provided with an information management system according to an embodiment; FIG. FIG. 4 is a flow chart diagram showing a part of procedures of information management operation and support operation by the information management system according to the embodiment; FIG. 7 is a flow chart diagram showing another part of the procedure of the information management operation and the support operation by the information management system according to the embodiment;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings. This embodiment acquires the biological information (heart rate, etc.) of an elderly person or the like in a facility such as a nursing care facility, a nursing home, or a serviced elderly housing, and acquires the biological information of the subject, and the information related to the emotion of the subject. A case will be described in which the present invention is applied to an information management system used to manage information to be processed. Further, in this embodiment, an example will be described in which the information management system according to the present invention is provided in an information management device (personal mobility) capable of moving within a facility. The information management system according to the present invention may be constructed by an information management device and a server, which will be described later. Also, in this embodiment, an example will be described in which the information management system manages information related to the emotions of each of a plurality of subjects.

- Configuration of Information Management Device -
FIG. 1 is a block diagram showing the configuration of an information management device 1 having an information management system 100 according to this embodiment. As shown in FIG. 1, the information management device 1 includes a surrounding information acquisition device 2 for acquiring information on a subject and its surrounding environment, a communication device 3 having a communication function, and operations of the information management device 1. It has an electronic control unit (ECU) 4 for control, a movement actuator 51 controlled by the ECU 4 , and wheels 52 driven by the movement actuator 51 . The information management device 1 also includes a GPS device 61 for detecting the current position of the information management device 1, a display 62 and a speaker 63 for providing various information to the target person and the like, and and a microphone 64 for detecting voice and surrounding sounds of the subject. These configurations will be specifically described below.

The ambient information acquisition device 2 acquires a three-dimensional distance image by scanning a camera that captures a subject and its surroundings to generate image data, and by scanning a laser to measure the distance to objects around the subject. Lidar (Light Detection and Ranging). A CCD (Charge Coupled Device) camera, an infrared camera, or the like can be used as the camera. The ambient information acquisition device 2 outputs image data acquired by a camera or Lidar to the ECU 4 . This ambient information acquisition device 2 corresponds to the photographing means referred to in the present invention.

The communication device 3 includes a transmitter 31 and a receiver 32, and transmits and receives information to and from infrastructure such as a relay base station of a mobile phone network existing around the information management device 1, for example. is. The communication device 3 is set so as to be able to transmit and receive information within a radius of about 100 m, for example. For example, it is possible to transmit and receive information to and from a server installed at the facility (a server that manages biometric information of each of a plurality of subjects). Further, when a wireless IC tag, which is a small transmitter that transmits predetermined information, is present in the vicinity, the communication device 3 receives information transmitted from the wireless IC tag. The communication device 3 and the ECU 4 are connected by a signal line so that they can exchange information with each other. That is, the transmission information generated by the ECU 4 is transmitted from the ECU 4 to the transmitter 31 . For example, the subject's biometric information received from the ECU 4 is transmitted to the transmitter 31, and the information is transmitted from the transmitter 31 to a facility server or the like. On the other hand, various information received by the receiver 32 is transmitted from the receiver 32 to the ECU 4 . For example, the receiver 32 receives information about a subject whose biological information is to be obtained from a facility server, and the information is transmitted from the receiver 32 to the ECU 4 .

The movement actuator 51 includes an electric motor, a steering mechanism, etc., and transmits to the wheels 52 a driving force generated by the operation of the electric motor and a steering force generated by the operation of the steering mechanism, thereby rotating and steering the wheels 52 . is to be performed. As the wheels 52 rotate and steer, the information management apparatus 1 can move (autonomously move). Therefore, the moving actuator 51 and the wheels 52 constitute the moving means of the present invention.

The GPS device 61 receives GPS signals transmitted from artificial satellites and measures the current position of the information management device 1 . Information on the current position of the information management device 1 positioned by the GPS device 61 is transmitted from the GPS device 61 to the ECU 4 .

The display 62 and speaker 63 are means for providing information to the subject. The display 62 and the ECU 4 are connected by a signal line, and when a control signal is output from the ECU 4 to the display 62, the display 62 displays various information according to the control signal from the ECU 4.

The speaker 63 and the ECU 4 are connected by a signal line, and when the ECU 4 outputs a control signal to the speaker 63, the speaker 63 emits a sound according to the control signal from the ECU 4. The sound emitted from the speaker 63 is not particularly limited. face toward the surrounding information acquisition device 2).

The microphone 64 detects the voice uttered by the target person and the sounds around the target person (such as the voice of a person other than the target person). Information on sounds (including voice) detected by the microphone 64 is transmitted from the microphone 64 to the ECU 4 . As will be described later, this sound information is used as information for grasping situations in which the subject feels stress, interests, and matters of interest (by extracting the words contained in the voice, However, it is used as information to understand the situations in which the subject feels stress, interests, and matters of interest). This microphone 64 corresponds to the voice acquisition means of the present invention.

The ECU 4 physically includes a microprocessor that performs calculations, a ROM that stores programs and the like for causing the microprocessor to execute various processes, a RAM that stores various data such as calculation results, and a battery that stores the stored contents. is configured by a backup RAM or the like in which is held. With this physical configuration, the ECU 4 includes a surrounding environment recognition unit 41, a subject identification unit 42, an autonomous movement control unit 43, a self position recognition unit 44, a biometric information acquisition unit 45, a word extraction unit 46, an emotion determination unit. 47 and an information storage unit 48 are realized.

The surrounding environment recognizing unit 41 performs image processing for recognizing persons including the target person and surrounding objects by processing images acquired by the camera or lidar of the surrounding information acquisition device 2 . In this image processing, processing for extracting and recognizing a person and surrounding objects is performed by edge extraction, pattern recognition processing, or the like from an image acquired by a camera or lidar.

The subject identification unit 42 is a functional unit for identifying a person (subject) whose biometric information is to be acquired. Specifically, the information management device 1 acquires biometric information of each of a plurality of subjects in a facility such as a nursing care facility. Then, the biometric information acquisition operation is performed at preset times or at preset time intervals for each subject. The subject identification unit 42 identifies a subject whose biometric information needs to be acquired at the present time (for example, identifies the subject based on the information received from the server), and performs a search operation of the subject and Perform the authorized action of the target person.

The search motion of the target person and the recognition motion of the target person are performed by determining whether or not the shape and pattern of the clothes worn by the photographed person match those of the target person learned in advance. be able to. Further, when the target person is wearing a wireless IC tag that transmits radio waves indicating that the target person is a target person, the target person identification unit 42 receives the radio waves transmitted by the wireless IC tag via the communication device 3. By doing so, it is possible to search and identify the target person. In addition, when the face image can be captured by a camera or lidar, the image data of the subject's face annotated in advance (labeled image data; teacher for recognizing the subject's face by deep learning) data), it is possible to recognize whether or not the person whose face image is taken is the target person (the target person whose biometric information is to be acquired this time). In addition, if the target person is speaking, it is possible to recognize that the target person is using existing speech recognition technology (speech recognition technology that uses deep learning using a speech model or language model, etc.). .

The autonomous movement control unit 43 transmits a movement control signal to the movement actuator 51 in order to cause the information management device 1 to move autonomously. The movement control signal includes a rotational speed control signal sent to the electric motor and a steering control signal sent to the steering mechanism. The electric motor and steering mechanism receive these signals to rotate and steer the wheels 52 . Specifically, when the position of the target person is recognized by the information from the surrounding environment recognition unit 41 and the information from the target person identification unit 42, by receiving the position information of the target person, The autonomous movement control unit 43 moves the information management device 1 to a position where the surrounding information acquisition device 2 can photograph the target person's face. That is, the information management apparatus 1 is moved by transmitting a movement control signal to the movement actuator 51 so as to move to the front side of the subject (turn around to the front side of the subject). This operation is performed while recognizing the self-position (the position of the information management device 1) by the self-position recognition unit 44. FIG.

The self-position recognition unit 44 receives information on the current position of the information management device 1 positioned by the GPS device 61 . This information is transmitted from the self-position recognition unit 44 to the autonomous movement control unit 43, and the autonomous movement control unit 43 sends a movement control signal to the movement actuator 51 so as to move the information management device 1 while recognizing the self-position. will be sent.

The biometric information acquiring unit 45 acquires the biometric information of the target person from the face image when the target person's face image is acquired by the camera or Lidar of the ambient information acquisition device 2 . Specifically, biometric information such as the subject's complexion, pulse (pulse for determining heartbeat), blood pressure, etc. is acquired from the face image.

As a technology to acquire the pulse of the subject from the image of the subject's face, we use the characteristic of hemoglobin contained in blood that absorbs green light to capture the brightness change on the face surface, which is thought to be caused by blood flow. to detect the pulse. In other words, after obtaining the average value of each of the color components (red, green, and blue) of the face area for each captured frame from a moving image of a face taken with a camera, etc., noise common to the three color components is removed, and the green Extract the luminance waveform from the component. Then, the pulse is calculated from the peak value of this luminance waveform. In addition, as a technology to acquire a subject's blood pressure from an image of the subject's face, a machine learning algorithm that detects slight changes in blood flow from a video of the subject's face captured by a camera or the like and obtains the blood pressure from this change in blood flow. to calculate the blood pressure from the blood flow data of the face. The biological information acquisition unit 45 obtains the pulse of the subject based on the image information of the subject's face captured by the camera or lidar of the surrounding information acquisition device 2, and acquires the heartbeat of the subject from the pulse information. is possible (in general, the heartbeat corresponds to the pulse). Therefore, the functional unit for acquiring the heartbeat in the biological information acquiring unit 45 is the heartbeat measuring unit (heartbeat measuring unit for measuring the heartbeat of the subject based on the image information of the subject photographed by the photographing means) according to the present invention. part).

The word extraction unit 46 extracts words included in the voice uttered by the subject acquired by the microphone 64 . Specifically, the voice uttered by the subject (for example, the voice uttered while talking with another person) is converted into text, and the text information is processed to extract words by natural language processing. there is For example, when the subject is conversing with another person, as mentioned above, only the voice uttered by the subject is extracted by voice recognition technology, and the words are extracted by converting the voice information into text and natural language processing. will be extracted.

The emotion determination unit 47 calculates heart rate variability and heart rate based on information on the subject's heartbeat measured by the biometric information acquisition unit (heartbeat measurement unit) 45, and uses these information to determine various emotions of the subject. At least one of the stress state and the interest state is determined. Specifically, heartbeat interval data (data relating to periodic heartbeat intervals) is created from image information of the subject's face captured by the camera of the ambient information acquisition device 2 or Lidar, and the frequency of the heartbeat interval data is generated. Heart rate variability spectrogram data (frequency spectrum calculation result data) is created by conversion. Based on the heart rate variability spectrogram data, the value of the heart rate variability frequency is calculated, and the subject's emotional state is determined based on the value of the heart rate variability frequency. Then, when the calculated heart rate variability is equal to or less than a predetermined threshold value (for example, 0.15 Hz), the emotion determination unit 47 determines that the subject is in a stressed state (stressed state). In addition, when the calculated heart rate variability exceeds a predetermined threshold value (for example, 0.15 Hz) and the heart rate is equal to or greater than a predetermined threshold value (for example, 80 beats/min), the emotion determination unit 47 It is determined that the person is in a state of interest (a state in which he or she is thinking about a matter of interest or concern). These thresholds can be set arbitrarily, but are set empirically or experimentally.

The information storage unit 48 stores various types of biometric information acquired by the biometric information acquisition unit 45 . As described above, the information management device 1 acquires biometric information of each of a plurality of subjects. Therefore, the information storage unit 48 stores each subject and each subject's biometric information acquired by the biometric information acquisition unit 45 in association with each other.

In addition, in this embodiment, the information storage unit 48 stores the words extracted by the word extraction unit 46 and the subject's emotion determined by the emotion determination unit 47 at the timing when the subject utters the word. is also stored. Since the information management device 1 acquires the biometric information of each of a plurality of subjects, it also stores information relating the words and the emotions corresponding to the words for each of the subjects. will be These pieces of information may be transmitted to the server and managed by the server.

Specific examples of words that may cause a subject to feel stressed include "disease," "accident," and "disaster." In other words, when a subject utters these words, there is a high possibility that the subject will be in a state of stress as an emotion. Therefore, these words and stress states are stored in association with each other. On the other hand, "travel", "cooking", "sports" and the like are examples of words that may cause interest in the subject's emotion. In other words, when the target person utters these words, there is a high possibility that the target person will be in a state of interest. Therefore, these words and interest states are stored in association with each other.

－Information management operation and support operation－
Next, the information management operation and support operation by the information management system (the information management system provided in the information management apparatus 1) 100 configured as described above will be described. 2 and 3 are flow charts showing the procedures of this information management operation and support operation. This flow chart is repeatedly executed each time the ECU 4 of the information management device 1 moving within the facility issues a biometric information acquisition request.

First, in step ST1, a subject whose biological information (biological information such as heartbeat) is to be obtained this time is specified. This operation is performed by the subject identification unit 42 described above. For example, the acquisition time of past biometric information for each subject is acquired, and the subject with the oldest acquisition time of biometric information (the longest elapsed time since acquiring biometric information in the past) among each subject is identified as a subject whose biometric information needs to be obtained. In addition, schedule data of biometric information acquisition times and acquisition intervals for each subject is stored in advance in the RAM of the ECU 4, and the subject whose biometric information needs to be acquired is specified according to this schedule data. good too. Alternatively, a biometric information acquisition instruction signal may be received from the server, and the target person whose biometric information needs to be acquired may be specified according to the information of the target person included in the signal.

In step ST2, the target person's search motion and the target person's authentication motion specified in step ST1 are performed. That is, while recognizing the position of the target person by the information from the surrounding environment recognition unit 41 and the information from the target person identification unit 42, the target person is searched (searched while the information management device 1 is running) and certified (face image certification by photographing, etc.) is performed.

In step ST3, as an operation when the subject's face cannot be photographed, the movement actuator 51 rotates and steers the wheels 52 to photograph the subject's face by the camera or lidar of the surrounding information acquisition device 2. An operation of turning the information management apparatus 1 in front of the person is executed. At this time, assuming that the target person may be conversing with another person, the turnaround position of the information management apparatus 1 is set to a position where it is easy to acquire the voice uttered by the target person. For example, when it is determined that there is another person on the right side of the subject based on the image captured by the camera or Lidar of the surrounding information acquisition device 2, the information management device 1 is moved to the left side of the subject. executed.

In step ST4, the subject's face is photographed by the camera or the Lidar in a state in which the photographing of the subject's face (face recognition) by the camera or the Lidar is enabled. Then, the heartbeat of the subject is measured based on the face image data obtained by photographing the face of the subject (operation of the heartbeat measuring function unit in the biometric information acquisition unit 45). Also, the voice uttered by the subject is detected by the microphone 64 . Accompanying the detection of this voice, words included in the voice uttered by the subject are extracted by converting the voice into text and natural language processing of the text information (word extraction operation by the word extraction unit 46). The words extracted here may be all words contained in the voice uttered by the subject, or words that are judged to be effective in determining the subject's emotion (e.g., It may be only words that are considered to be related to emotions).

After that, the process proceeds to step ST5, and it is determined whether or not the heart rate variability calculated as described above is equal to or less than a predetermined threshold value.

If the heart rate variability is equal to or less than the predetermined threshold and the determination in step ST5 is YES, the process proceeds to step ST6. Add 1 to At this time, when a plurality of words are extracted by the word extraction unit 46, each word is assigned a stress determination counter and 1 is added.

In step ST7, the current value of the stress determination counter is stored in the information storage section 48. FIG. This current stress determination counter value may be stored in a separate database (for example, a database within the server).

After that, the process proceeds to step ST8, and it is determined whether or not the value of the stress determination counter accumulated during a predetermined period (for example, the past seven days) is equal to or greater than a predetermined threshold. The value of this period is not limited to this. At this time, when a plurality of words are extracted by the word extraction unit 46 as described above, it is determined whether or not the stress determination counter assigned to each word is equal to or greater than a predetermined threshold value. be done.

If the value of the stress determination counter accumulated during the predetermined period is still less than the threshold value and the determination in step ST8 is NO, the process is returned as is, and the same information management operation as described above is performed for other subjects. will be

On the other hand, if the value of the stress determination counter accumulated during the predetermined period is equal to or greater than the threshold value and the determination in step ST8 is YES, the process proceeds to step ST9, and the subject is told to the staff of the facility that the subject is in a stressed state. to be notified. As a means for this notification, the information that the subject is in a stressed state is transmitted from the ECU 4 to the transmitter 31, and the information is sent from the transmitter 31 to the terminal carried by the facility staff. It is transmitted together with information that identifies the person (for example, name, etc.). This allows facility staff to identify the presence of a stressed subject, identify the subject, and recognize the need for support for the subject. This makes it possible to prompt the staff of the facility to support the subject. Therefore, the transmitter 31 corresponds to the output means (output means for externally outputting the information of the subject) according to the present invention.

After such notification, the process moves to step ST10, and after resetting the stress determination counter in the subject included in the information of this notification, it is returned, and the same information management operation and Support action is taken.

On the other hand, if the heart rate variability exceeds the predetermined threshold in the determination of step ST5 and the determination is NO, the process proceeds to step ST11 (FIG. 3), and the heart rate calculated as described above is equal to or greater than the predetermined threshold. Determine whether or not there is

If the heart rate is equal to or greater than the predetermined threshold value and the determination in step ST11 is YES, the process proceeds to step ST12, and the words extracted in step ST4 are subjected to interest determination that is stored in advance in the RAM of the ECU 4. A flag is set and the process moves to step ST13.

On the other hand, if the heart rate is less than the predetermined threshold and the determination in step ST11 is NO, the process returns without setting the interest determination flag.

In step ST13, the current interest determination flag is stored in the information storage section 48. FIG. This current interest determination flag may be stored in a separate database (for example, a database within the server).

After that, the process moves to step ST14, and classifies the category of words (words included in the voice uttered by the subject) extracted by the word extraction unit 46 when the determination in step ST11 is YES. In other words, the field to which the words included in the voice uttered by the subject when the subject's emotion is determined to be in the state of interest is determined.

After that, the process moves to step ST15, and based on the category classification and its attribute information determined in step ST14, the cosine similarity between a plurality of subjects (between subjects) and between Web search information big data is calculated. . This cosine similarity is a similarity calculation method used when comparing documents in a vector space model. It uses a trigonometric function (vector If the cosine similarity of two words is close to 1, they are similar, and if it is close to 0, they are dissimilar.

After the cosine similarity is calculated in this way, the process moves to step ST16, and subjects whose cosine similarity is equal to or higher than a predetermined value (other subjects and , a subject whose cosine similarity is equal to or greater than a predetermined value) exists.

Then, if there is a subject whose cosine similarity is equal to or greater than the predetermined value, and YES is determined in step ST16, the process proceeds to step ST17, and the subject whose biometric information is acquired this time (specified person in the present invention) ), an operation of introducing a target person whose cosine similarity is equal to or higher than a predetermined value (a target person who has a history of uttering a voice whose cosine similarity is equal to or higher than a predetermined value). As for this introduction operation, the terminal carried by the subject whose biometric information is acquired this time, the information of this introducer (the subject whose cosine similarity is equal to or higher than a predetermined value) and the emotion of this introducer are sent to the terminal. Information such as words included in the voice uttered when in the state is notified. For example, if the emotion was in the state of interest when the subject uttered the word "travel", the history of the emotion being in the state of interest when the subject uttered words such as "trip" and "hot springs" in the past You will be referred to other subjects who have

In addition to the operation (introduction operation) of step ST17, in step ST18, Web search information based on words extracted from the voice uttered by the subject whose biometric information is acquired this time or words similar to the words is searched. offer. This information is also provided by notifying the terminal carried by the subject.

If there is no target person whose cosine similarity is equal to or greater than the predetermined value and NO is determined in step ST16, only the web search information is provided in step ST18 without performing the introduction operation in step ST17. will be done.

After that, the process moves to step ST19, and it is determined whether or not the target person has performed a reaction operation in response to the notification of step ST17 or step ST18 (notification of introducer or provision of web search information). For example, when a notification is sent to a terminal carried by the target person, it is determined whether or not a response operation has been performed in response to the notification. For example, when the introducer is notified, the operation indicating acceptance of the introduction (pressing the acceptance button displayed on the screen of the terminal, etc.), or when providing web search information, the information It is determined whether or not a search operation (such as pressing a search button displayed on the screen of the terminal) has been performed.

When the target person reacts to the notification operation and the determination in step ST19 is YES, the process is returned as is, and the same information management operation and support operation are performed for other target persons.

On the other hand, if the target person does not respond to the notification operation for a predetermined period of time and the determination in step ST19 is NO, the process moves to step ST20, resets the interest determination flag, and returns to another target person. Similar information management and support operations will be performed.

The above operation is repeated in order for each subject.

- Effects of Embodiment -
As described above, in the present embodiment, a word included in a voice uttered by a subject and an emotion of the subject determined at the timing when the subject uttered the word (stress state, interest, etc.) state) is stored in the information storage unit 48 . Therefore, it is possible to understand the relationship between the words contained in the voice uttered by the target person and the target person's emotions, and based on the words, the situation where the target person feels stress, the interest, etc. It is possible to effectively support the target person accordingly. As a result, it is possible to expand the range of use of the system.

Further, in the present embodiment, the information management device 1 is arranged to go around the front of the subject when the subject's face is photographed by the camera or lidar of the surrounding information acquisition device 2 . Therefore, the face of the subject can be easily photographed, and the heartbeat of the subject based on the image information of the face of the subject can be measured with high accuracy.

In addition, in this embodiment, when the subject is determined to be in a stressed state more than a predetermined number of times during a predetermined period of time, the staff is notified to that effect. Therefore, it is possible to make the staff aware that the subject who is in a stressed state needs support, and to prompt the subject to support.

Further, in the present embodiment, when the subject is in a state of interest, the subject receives Web search information based on the word extracted by the word extraction unit 46 or a word similar to the word at the timing when the determination is made. I am trying to provide it to For this reason, the subject can easily obtain the information that interests him/her.

Furthermore, in this embodiment, as the cosine similarity between each subject, if there is a subject whose cosine similarity is a predetermined value or more, the cosine similarity is a predetermined value or more. The action of introducing the target person is implemented. For this reason, it is possible to deepen exchanges between a plurality of subjects who have common interests with each other. As a result, it is possible to further expand the range of use of the system.

-Other embodiments-
It should be noted that the present invention is not limited to the above-described embodiments, and all modifications and applications within the scope of the claims and their equivalents are possible.

For example, in the above-described embodiment, the information management system 100 according to the present invention is provided in the information management apparatus 1. The information management system 100 may be constructed by the information management apparatus 1 and the server.

Further, in the above-described embodiment, the case where only the stress state and the interest state can be determined as the emotions of the subject has been described as an example. The present invention is not limited to this, and when other emotions (emotions other than the stress state and the interest state) can be determined from the biometric information, the word extracted by the word extraction unit 46 and the subject person using the word Information associated with other emotions (emotions other than the stress state and the interest state) determined at the timing of the utterance may be stored.

Further, in the above embodiment, the information management device 1 acquires the biometric information of each of a plurality of subjects, associates each of the subjects with the acquired biometric information of each of the subjects, and stores them. was The present invention is not limited to this, and may be configured as a personal information management device for acquiring biometric information of only one specific subject (for example, one registered subject). .

INDUSTRIAL APPLICABILITY The present invention is applicable to an information management system that acquires biometric information from an image of a subject's face and manages information related to the subject's emotion.

1 Information management device 2 Surrounding information acquisition device (shooting means)
31 transmitter (output means)
45 biological information acquisition unit (heart rate measurement unit)
46 word extraction unit 47 emotion determination unit 48 information storage unit 51 movement actuator (moving means)
52 wheels (means of transportation)
64 microphone (voice acquisition means)
100 Information Management System

Claims (5)

An information management system for managing information related to a subject's emotions,
a photographing means capable of photographing the subject;
a voice acquisition means capable of acquiring a voice uttered by the subject;
a word extraction unit for extracting words included in the voice from the voice acquired by the voice acquisition means;
a heartbeat measurement unit that measures the heartbeat of the subject based on the image information of the subject captured by the imaging means;
Based on information on at least one of heart rate variability and heart rate calculated based on the heart rate information of the subject measured by the heart rate measuring unit, at least the stress state and the state of interest among the various emotions of the subject. an emotion determination unit that determines which
an information storage unit that stores information that associates the word extracted by the word extraction unit with the subject's emotion determined by the emotion determination unit at the timing when the subject utters the word;
An information management system comprising: In the information management system according to claim 1,
at least the photographing means and the sound acquisition means are mounted on an information management device,
The information management device comprises a moving means capable of autonomous movement,
The heartbeat measuring unit is configured to measure the heartbeat of the subject based on image information of the subject's face photographed by the photographing means,
1. An information management system, wherein said information management device moves around in front of said subject by operating said moving means when said subject's face is photographed by said photographing means. In the information management system according to claim 1 or 2,
An output means for externally outputting the information of the subject,
The emotion determination unit determines that the subject is in a stressed state when the heart rate variability is equal to or less than a predetermined threshold, and when the stressed state is determined a predetermined number of times or more during a predetermined period of time. (3) An information management system, wherein the output means externally outputs information indicating that the target person is in a stressed state. In the information management system according to claim 1, 2 or 3,
The emotion determination unit determines that the subject is in a state of interest when the heart rate variability exceeds a predetermined threshold and the heart rate is equal to or greater than a predetermined threshold, and determines that the subject is in the state of interest. When the determination is made, the subject is provided with Web search information based on the word extracted by the word extraction unit at the timing of the determination or a word similar to the word information management system. In the information management system according to any one of claims 1 to 4,
Information relating the words extracted by the word extracting unit to the emotion of the target person determined by the emotion determination unit at the timing when the target person utters the word, for a plurality of target people It is designed to be individually stored in the information storage unit according to each of the plurality of subjects,
When the emotion determination unit determines that a specific target person is in a state of interest, the word extracted by the word extraction unit at the timing when the determination is performed, or the word is similar to the word An information management system characterized by introducing other subjects having a history of uttering words to the specific subject.

Images