JP2016115057A - Biological information processing system, server system, biological information processing apparatus, biological information processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a biological information processing system, a server system, a biological information processing apparatus, a biological information processing method, and a program capable of knowing a user's mental state corresponding to life log information. A biological information processing system 100 includes a biological information acquisition unit 110 that acquires biological information measured by a biological sensor 420, and a processing unit 130 that performs estimation processing of a user's mental state information based on the biological information. And a storage unit 150 for storing life log information. And the process part 130 provides the mental state information estimated based on biometric information to life log information as an index. The storage unit 150 stores life log information to which mental state information is added as an index. [Selection] Figure 1

Description

  The present invention relates to a biological information processing system, a server system, a biological information processing apparatus, a biological information processing method, a program, and the like.

  In recent years, life log research has been underway. The life log is a record (Log) obtained by converting human actions (Life) into digital data. Specific examples of the life log include image data, video data, audio data, sentence data, and position data.

  For example, as an invention related to an apparatus for acquiring a life log, there are conventional techniques described in Patent Document 1 and Patent Document 2. In the invention of Patent Document 1, an information processing apparatus that automatically performs shooting according to a user's psychological state is disclosed. In the invention of Patent Document 1, image data obtained by photographing is a life log.

  Further, in the invention of Patent Document 2, the user's movement route information, exercise state information, life situation information, moving environment information and the like are detected and stored, and the user's uniqueness is based on the stored various information. The correct status.

JP2012-120206A JP 2010-279638 A

  Until now, life logs were mostly used as customer information for sales promotion purposes, and were rarely used for personal life improvement purposes. In addition, even when the life log is used for the purpose of improving the life of an individual, most of them are recorded just like a diary.

  However, in recent years, it has been required to use a life log for the purpose of improving an individual's life. For example, there is a need for a mechanism that can confirm later what the user feels when a certain life log is acquired.

  On the contrary, the invention of Patent Document 1 described above is to take a picture when the user enters a certain psychological situation, and therefore can only view life logs (images) acquired under a specific psychological situation. That is, the life log cannot be continuously acquired and the mental state corresponding to an arbitrary life log cannot be confirmed later.

  Moreover, although invention of patent document 2 provides biometric information as an index to a life log, the biometric information provided at this time is a physical quantity itself such as a heart rate. For example, when the heart rate exceeds a given threshold, it is determined that the user is “pounding” and the life log is indexed. However, in reality, even if the user's heart rate is increased, the heart rate may be increased due to, for example, exercise, or the heart rate may be increased due to a fear state. In the invention of Patent Document 2, these cannot be distinguished. In other words, in the invention of Patent Document 2, since the mental state is simply determined only by the level or change of the value of the biological information, the mental states that can be estimated are limited.

  Ideally, it would be desirable to be able to check the life log and later confirm information that better represents the actual mental state of the user.

  According to some aspects of the present invention, a biological information processing system, a server system, a biological information processing apparatus, a biological information processing method, a program, and the like that can know the mental state of a user corresponding to life log information are provided. can do.

  According to one aspect of the present invention, a biological information acquisition unit that acquires biological information measured by a biological sensor, a processing unit that performs estimation processing of a user's mental state information based on the biological information, and life log information A storage unit that stores the mental state information estimated based on the biological information as an index to the life log information, and the storage unit includes the mental state information Is related to a biological information processing system that stores the life log information assigned as the index.

  In one aspect of the present invention, mental state information estimated based on biological information is assigned to life log information as an index. This makes it possible to know the mental state of the user corresponding to the life log information.

  Moreover, in one aspect of the present invention, a display processing unit that performs display processing of the time-series change information of the mental state information based on the life log information may be included.

  Thereby, after acquiring life log information, it becomes possible to read how the mental state of the user has changed while checking the life log information.

  In the aspect of the invention, the display processing unit may perform the display process of the time-series change information of the mental state information in association with the time-series change information of the life log information.

  Thereby, for example, it is possible to confirm the mental state of the user when acquiring the life log information while confirming the life log information acquired in time series.

  In the aspect of the invention, the life log information includes position information, and the display processing unit is configured to change the time series change of the mental state information corresponding to the position indicated by the time series change information of the position information. You may perform the said display process of information.

  Thereby, it becomes possible to confirm the mental state of the user corresponding to each position of the user.

  Moreover, in one aspect of the present invention, the processing unit identifies the activity state information of the user based on the biological information, and the display processing unit is associated with the time-series change information of the activity state information, The display process of the time series change information of the mental state information may be performed.

  This makes it possible to check the user's mental state information corresponding to the activity state information in chronological order.

  Moreover, in one aspect of the present invention, a display processing unit that performs display processing of time-series change information of the mental state information based on the life log information, the life log information includes image information, video information, Content information that is at least one of sound information and text information, and the display processing unit acquires first content information to Nth content information as the life log information in time series order, and the first content The time-series change information of the mental state information may be displayed in association with each content information of information to the Nth content information.

  Thereby, it becomes possible to confirm the mental state information of the user corresponding to each content information.

  In the aspect of the invention, the time series change information of the mental state information may be information representing a time series change of a value obtained by quantifying the mental state.

  Accordingly, it is possible to check not only the mental state of the user at a certain moment but also the change of the mental state of the user during a certain period.

  In one aspect of the present invention, the processing unit is estimated for the life log information acquired at a timing corresponding to an acquisition timing of the biological information used for estimating the mental state information. The mental state information may be associated as the index.

  Thereby, it becomes possible to confirm the mental state information at the acquisition timing of the biological information.

  In the aspect of the invention, the processing unit performs an event detection process based on the life log information, and is acquired at a timing corresponding to a timing at which the life log information at which the event is detected is acquired. The mental state information may be estimated based on the biological information, and the estimated mental state information may be added to the life log information where the event is detected as the index.

  This makes it possible to check mental state information at the event occurrence timing.

  Moreover, in 1 aspect of this invention, it further includes the body movement information acquisition part which acquires the body movement information of the said user, The said process part is based on the said biological information and the said body movement information, and the said mental state information May be estimated.

  This makes it possible to estimate the mental state information more accurately.

  In one aspect of the present invention, the processing unit estimates the mental state information including at least one of the tension state, the excited state, and the relaxed state of the user based on the biological information. Also good.

  This makes it possible to display at least one piece of information among the tension state, the excited state, and the relaxed state of the user in association with the life log information.

  In the aspect of the invention, the processing unit may estimate the height of the user's motivation as the mental state information.

  This makes it possible to display the life log information in association with the height of the user's motivation.

  In one aspect of the present invention, the processing unit performs user evaluation on the user's action result based on at least one information of the user's sleep state, calorie consumption, physical strength, physical condition, and drinking state. It may be estimated as mental state information.

  Thereby, it is possible to display the life log information in association with the user evaluation for the user's action result.

  In the aspect of the invention, the mental state information may be information related to at least one of the user's tension state, excitement state, relaxed state, motivation height, and user evaluation with respect to an action result.

  This makes it possible to associate mental state information that better represents the state of the user's heart than biological information with life log information.

  In the aspect of the invention, the life log information may be at least one of image information, video information, sound information, text information, and position information.

  This makes it possible to display information on the display unit that allows the user to easily remember the situation at the time.

  Another aspect of the present invention relates to a server system including a biological information processing system.

  Another aspect of the present invention relates to a biological information processing apparatus including a biological information processing system.

  In another aspect of the present invention, the biological information measured by the biological sensor is acquired, the mental state information of the user is estimated based on the biological information, and is estimated based on the biological information. The present invention relates to a biological information processing method in which the mental state information is assigned to life log information as an index, and the life log information in which the mental state information is assigned as the index is stored.

  According to another aspect of the present invention, the mental state information estimated based on the biological information measured by the biological sensor is a processing unit that performs reception processing of life log information given as an index, and the received life The present invention relates to a program that causes a computer to function as a storage unit that stores log information and a display processing unit that performs processing for displaying time-series change information of the mental state information on a display unit based on the life log information.

The system configuration example of this embodiment. 2A and 2B are configuration examples of the server system, the measurement device, and the terminal device of the present embodiment. Explanatory drawing of the flowchart explaining the flow of a process of this embodiment. Explanatory drawing of the specific example of biometric information, mental state information, activity state information, and life log information. Explanatory drawing of the estimation process of mental state information. Explanatory drawing of the example of a display of life log information and mental state information. Explanatory drawing of the 1st example which displays mental state information on a map. Explanatory drawing of the 2nd example which displays mental state information on a map. Explanatory drawing of the 3rd example which displays mental state information on a map. Explanatory drawing of the 4th example which displays mental state information on a map. FIG. 12 is another explanatory diagram of mental state information estimation processing. Explanatory drawing of the specific example of the quantified mental state information. Explanatory drawing of ease of influence on exercise intensity and motivation. 14A and 14B show specific configuration examples of the measurement apparatus. A specific configuration example of a measurement apparatus. The structural example of the measuring device and terminal device which perform communication connection by a network.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. Outline In recent years, research on a life log, which is a record (Log) obtained by converting human actions (Life) into digital data, has been promoted. Specific examples of the life log include image data, video data, audio data, sentence data, and position data. There is also a demand for a mechanism that allows the user to confirm later how the user feels when a certain life log is acquired.

  Also, the mental state that the user wants to confirm later by looking at the life log is not simply observation data such as the rise and fall of the pulse rate, but the user's motivation at that time, what the user thinks, etc. For this reason, simply associating biometric information with a life log cannot provide information requested by a user.

  Therefore, in the present embodiment, the mental state information estimated based on the biological information is added to the life log information as an index.

2. System Configuration Example Specifically, FIG. 1 shows a configuration example of the biological information processing system 100 of the present embodiment. The biological information processing system 100 includes a biological information acquisition unit 110, a processing unit 130, a storage unit 150, a display processing unit 170, and a body motion information acquisition unit 190. The biological information processing system 100 is not limited to the configuration in FIG. 1, and various modifications such as omitting some of these components or adding other components are possible.

  Next, processing performed in each unit will be described. The body motion information acquisition unit 190 will be described later.

  The biological information acquisition unit 110 acquires biological information measured by the biological sensor 420. The biometric information acquisition unit 110 may be a communication unit (I / F unit, antenna unit, etc.) that transmits and receives data when connected to the biosensor 420 by wireless or wired communication.

  The biological sensor 420 is a pulse wave sensor, for example. The pulse wave sensor is a sensor for detecting a pulse wave sensor signal. For example, a photoelectric sensor can be considered. In addition, when using a photoelectric sensor as a pulse wave sensor, you may use the sensor comprised so that the signal component of external lights, such as sunlight, may be cut. This can be realized by, for example, a configuration in which a plurality of photodiodes are provided and difference information is obtained by feedback processing using these signals.

  The pulse wave sensor is not limited to the photoelectric sensor, and may be a sensor using ultrasonic waves. In this case, the pulse wave sensor has two piezoelectric elements, transmits one ultrasonic wave by exciting one of the piezoelectric elements, and converts the ultrasonic wave reflected by the blood flow of the living body into the other piezoelectric element. Receive by the element. Since the frequency change occurs in the transmitted ultrasound and the received ultrasound due to the Doppler effect of blood flow, a signal corresponding to the blood flow can be obtained in this case as well, and pulsation information can be estimated. . Moreover, you may use another sensor as a pulse wave sensor.

  Next, the processing unit 130 performs estimation processing of the user's mental state information based on the biological information. And the process part 130 provides the mental state information estimated based on biometric information to life log information as an index. Specific mental state information estimation processing and index assignment will be described later. The function of the processing unit 130 can be realized by hardware such as various processors (CPU, GPU, etc.), ASIC (gate array, etc.), a program, and the like.

  The life log information is acquired by the life log information acquisition unit 480. For example, when the life log information to be acquired is image information or video information, the life log information acquisition unit 480 is an imaging unit (camera). When the life log information to be acquired is sound information, the life log information acquisition unit 480 is a sound acquisition unit such as a microphone. Furthermore, when the life log information to be acquired is text information, the life log information acquisition unit 480 is a character input unit. When the life log information to be acquired is position information, the life log information acquisition unit 480 is a position information acquisition unit such as a GPS. However, the specific configuration of the life log information acquisition unit 480 is not limited to these.

  And the memory | storage part 150 memorize | stores the life log information to which mental state information was provided as an index. The storage unit 150 serves as a work area for each unit, and its function can be realized by a memory such as a RAM, an HDD (hard disk drive), or the like.

  Further, the display processing unit 170 performs processing for displaying life log information, biological information, mental state information, and the like on the display unit 430. More specifically, the display processing unit 170 generates a display image to be displayed on the display unit 430, for example, or performs timing control when displaying the display image on the display unit 430. The function of the display processing unit 170 can be realized by hardware such as various processors (CPU, GPU, etc.), ASIC (gate array, etc.), a program, and the like.

  The display unit 430 can be realized by, for example, a liquid crystal display, an organic EL display, electronic paper, or the like.

  This makes it possible to know the mental state of the user corresponding to the life log information.

  In addition, the function of the biological information processing system 100 according to the present embodiment may be realized by a server system 600 as illustrated in FIG.

  In this case, for example, the measurement device 400 includes the biological sensor 420, and the terminal device 200 includes the display unit 430 and the life log information acquisition unit 480. The server system 600 implements the functions of the biological information acquisition unit 110, the processing unit 130, the storage unit 150, and the display processing unit 170. The server system 600 includes a plurality of server devices such as a processing server and a data server. However, the specific configurations of the server system 600, the measuring device 400, and the terminal device 200 are not limited to this, and various components such as omitting some of these components or adding other components. Variations are possible. In addition, some functions of the server system 600 may be realized by the terminal device 200, the measurement device 400, and the like being distributed.

  In the example of FIG. 2A, the measurement device 400, the terminal device (smart device) 200, and the server system 600 are connected for communication by a network 900 including at least one of wired and wireless. The network 900 may include various networks with different communication standards. For example, the measuring device 400 and the terminal device 200 are connected by Bluetooth (registered trademark), a wireless LAN, or the like, and the terminal device 200 and the server system 600 are connected by the Internet. However, this embodiment is not limited to this.

  Then, the biological information acquisition unit 110 of the server system 600 acquires biological information from the biological sensor 420 of the measuring device 400, and the processing unit 130 of the server system 600 acquires the life log information from the life log information acquisition unit 480 of the terminal device 200. To get. Further, the processing unit 130 of the server system 600 performs mental state information estimation processing based on the biological information, and assigns the mental state information as an index to the acquired life log information. Thereafter, the display processing unit 170 of the server system 600 generates a display image, transmits the display image to the display unit 430 of the terminal device 200, and controls the display timing of the display image. The display unit 430 of the terminal device 200 displays a display image based on an instruction from the display processing unit 170.

  2B shows a detailed configuration example of the terminal device 200 when the biological information processing system 100 of the present embodiment is realized by the configuration shown in FIG. The terminal device 200 includes a processing unit 230, a storage unit 250, a display processing unit 270, a display unit 430, and a life log information acquisition unit 480. The display unit 430 may be provided outside the terminal device 200 or may be included in the terminal device 200.

  In this case, the processing unit 230 of the terminal device 200 includes a reception processing unit 231, a transmission processing unit 232, and a control unit 233. The reception processing unit 231 performs reception processing such as a display image and a display instruction transmitted from the display processing unit 270 of the server system 600. In other words, the processing unit 230 performs reception processing of life log information in which the mental state information estimated based on the biological information measured by the biological sensor 420 is added as an index. The transmission processing unit 232 controls transmission processing of life log information, biological information, and mental state information to the information storage medium 251 and the like. Then, the control unit 233 controls each unit such as the reception processing unit 231 and the transmission processing unit 232.

  Further, the display processing unit 270 performs processing for displaying a display image on the display unit 430 based on the display instruction transmitted from the display processing unit 170 of the server system 600. In other words, the display processing unit 270 performs processing for displaying the time series change information of the mental state information on the display unit based on the life log information.

  The storage unit 250 stores the received life log information. The information storage medium 151 is, for example, an SD card or HDD, and stores life log information, biological information, mental state information, and the like.

  Furthermore, as another modified example, a biological information processing apparatus including the above-described biological information processing system 100, the biological sensor 420, the display unit 430, and the life log information acquisition unit 480 can be considered. In this case, the biological information processing apparatus is a wearable apparatus, for example, and performs all the processing of this embodiment with one apparatus.

3. Details of Processing Next, the method of the present embodiment will be described. First, the flow of processing according to this embodiment will be described with reference to FIG. 3, and then the details of each processing will be described.

  First, the biological information acquisition unit 110 acquires the biological information of the user wearing the biological sensor 420 from the biological sensor 420 (S101). The biological information is, for example, a pulse, blood pressure, blood glucose level, etc. as shown in FIG.

  Next, the processing unit 130 performs estimation processing of the user's mental state information based on the acquired biological information (S102). For example, the processing unit 130 estimates mental state information including information on at least one of a user's tension state, excitement state, and relaxation state based on the biological information.

  Here, the mental state information is information relating to at least one of user evaluation on the user's tension state, excitement state, relaxed state, motivation height and action result, as shown in FIG. 4, for example.

  As a result, as will be described later, information (mental state information) that better represents the state of the user's heart than the biological information can be displayed on the display unit 430 in association with the life log information. Details of the mental state information estimation process will be described later.

  Next, the life log information acquisition unit 480 acquires life log information (S103). The process of step S103 may be performed in parallel regardless of the order of the processes of step S101 and step S102, or the process of step S103 may be performed first.

  Here, the life log information is at least one information of image information, video information, sound information, text information, and position information, as shown in FIG. 4, for example.

  As a result, as will be described later, information that allows the user to easily remember the situation at that time can be displayed on the display unit 430.

  Next, the processing unit 130 assigns mental state information as an index to the life log information (S104). Then, the storage unit 150 stores life log information and mental state information (S105). Thereafter, the display processing unit 170 performs display processing for displaying the life log information associated with the mental state information on the display unit 430 (S106), and ends the processing.

  Next, each process of the present embodiment will be specifically described. In the present embodiment, mental state information is estimated based on physical quantities obtained from biological information (vital information). For example, FIG. 4 shows an example of mental state information that can be estimated from the pulse. As shown in FIG. 4, a tension state, an excited state, a relaxed state, or the like can be estimated from the pulse. Since the pulse is closely related to the state of the sympathetic nerve, the state of mental excitement can be estimated in this way by the transition of the pulse level. From such a state of excitement, it can be presumed that the present state of the heart, what the person is currently interested in or seeing, what is highly motivated, and the like.

  In this embodiment, activity state information is estimated based on biological information. Specifically, the activity state information includes, for example, calorie consumption, physical strength, physical condition, sleep state, drinking state, etc. as shown in FIG.

  Specifically, at the time of sleep, whether the sleep is deep or shallow can be known from the height of the fluctuation average value of the pulse. In addition, calorie consumption can be estimated from the time integral value of the pulse. It is more accurate if you know the amount of body movement by body movement. In addition, since the basal metabolism can be understood from the base pulse rate, such as the average pulse rate of the day, as a result of behaviors such as whether the person's physical strength, physical condition, drinking and drinking caused a hangover, You can also see if your body is well adapted. Based on these activity state information, it can be estimated whether the person is currently satisfied and highly motivated, or because the body is tired and has lost motivation. FIG. 4 shows an example of a pulse, but similarly, the mental state can be estimated from other biological information. Since the biological information has some relation to the state of the sympathetic nerve, the state of the heart is reflected. Accordingly, it is possible to comprehensively determine the mental state estimated from the individual biological information, and to more accurately estimate the present mental state. A more detailed mental state estimation process will be described later.

  Next, FIG. 5 shows a time-series processing example in which the estimated mental state is indexed (indexed) into life log information (life log data). This example is an example of processing every Δt time. The life log data indicates information, files, and the like used for describing a blog, such as images, videos, and messages. In this example, for example, life log data LD1 to life log data LD8 are acquired in chronological order, and are recorded at any time when shooting and comments are written.

  At the same time, a plurality of pieces of biological information (biological information 1, biological information 2, etc.) are acquired from the biological sensor 420, and the physical quantity of the biological sensor 420 is recorded as shown in the graph of FIG. Analyze these physical quantities for each Δt period, analyze the physical quantities in the period of Δt (average processing, frequency analysis processing, comparison processing with the previous period, etc.), compare and examine biological information, and estimate the mental state To do. For example, a mental state is estimated such that the body is slightly tired but motivated and very satisfied and highly motivated. This state is indexed by inserting and adding it to normal life log data every Δt period. In this example, for example, mental state information is assigned to life log data LD3 as index IDX1, mental state information is assigned to life log data LD4 as index IDX2, and mental state information is assigned to life log data LD6 as index IDX3. Has been granted.

  Further, the mental state information may be language information as described above, may be a numerical value of the degree, or may be classified by a code such as motivation level. Furthermore, the period Δt may be adjusted, for example, by shortening it in a scene where the emotional change is considered to be large, or by lengthening it during a time zone during which it is considered that there is little significant change while sleeping. . Alternatively, the mental state estimation process may be performed using a pulse as a trigger when the pulse is greater than or equal to a certain threshold.

  Next, FIG. 6 shows a display example of life log information to which mental state information is assigned as an index. FIG. 6 shows the height of motivation where the horizontal axis is the time axis and the vertical axis is one of the mental state information. When displaying an image on the time axis when the image was recorded, laying out based on the height of the motivation at the time of recording the image, what happened when the motivation was high or what was happening when it was low You can check at a glance. In addition to images, it is also effective to display comments.

  In this way, the display processing unit 170 performs the display process of the time series change information of the mental state information based on the life log information.

  Thereby, after acquiring life log information, it becomes possible to read how the mental state of the user has changed while checking the life log information.

  Here, the time series change information of the mental state information is information representing a time series change of a value obtained by quantifying the mental state. For example, in the example of FIG. 6, the information represents a change in which the state of high motivation is changed to a state of low motivation and transitions to a state of high motivation again.

  Accordingly, it is possible to check not only the mental state of the user at a certain moment but also the change of the mental state of the user during a certain period.

  Strictly speaking, the display processing unit 170 performs time-series change information display processing of mental state information in association with time-series change information of life log information as shown in FIG. That is, for the first timing life log information, the first mental state information is displayed as an index, and for the second timing life log information, the second mental state information is used as an index. Give and display.

  Thereby, for example, it is possible to confirm the mental state of the user when acquiring the life log information while confirming the life log information acquired in time series.

  Further, the display processing unit 170 acquires the first content information to the Nth content information as life log information in chronological order, and associates them with the content information of the first content information to the Nth content information, Displays time-series change information of mental state information.

  In the example of FIG. 5 described above, N = 8, the first content information is LD1, the second content information is LD2, and similarly, the eighth content information is LD8. In the example of FIG. 5, the time-series change information of the mental state information is included only in the third content information LD3, the fourth content information LD4, the sixth content information LD6, and the eighth content information LD8. However, the time series change information of the mental state information may be associated with each content information of the first content information to the Nth content information.

  Thereby, it becomes possible to confirm the mental state information of the user corresponding to each content information.

  Further, as described above, the processing unit 130 specifies the activity state information of the user based on the biological information. Then, the display processing unit 170 performs display processing of the time series change information of the mental state information in association with the time series change information of the activity state information.

  For example, there is a first timing to a third timing, and the first mental state information is associated with the first activity state information acquired at the first timing and displayed on the display unit 430. Then, the second mental state information is displayed in association with the second activity state information acquired at the second timing, and the third activity state information acquired at the third timing is displayed as the third activity state information. Display related mental state information.

  This makes it possible to check the user's mental state information corresponding to the activity state information in chronological order.

  Next, an example in which the recorded life log information is displayed superimposed on a two-dimensional map and the physical state that can be determined based on the mental state information or the biological information is displayed is shown in FIGS. .

  First, in the example of FIG. 7, a place where the motivation is higher than a given threshold value is displayed on the map by an arrow (filled mountain). The height of the arrow indicates the height of motivation. Then, at the point where the arrow indicating motivation is displayed, the date and time and the event that occurred at that time (seeing fireworks, going to drink with someone, etc.) are also displayed. As a result, it is possible to inform the viewer in an easy-to-understand manner that there were fun things that increased motivation, what happened when they were fulfilling, what they were doing, and so on.

  Further, as shown in FIG. 8, the height of motivation may be indicated on the map by an arrow, and an image acquired at that position may be associated with the arrow and displayed. However, it is not necessarily limited to displaying the level of motivation with arrows. For example, a place that has been tense, a place that has been surprised, or a place that has been relieved may be mapped and displayed on a map. Then, if there is an image or comment at that time, it is displayed, or the user edits the comment while viewing the display as shown in FIG. 7 or FIG. It is also possible to show the degree of tension or surprise by the size of the mark of a surprised person, a picture of a person who sweats a lot, or a picture of a surprised expression. If this is a commuting / commuting route display, it will be a material for identifying danger points and calling attention when commuting or going to school. Furthermore, if linked with a map information service or the like, it can also be a specific danger avoidance map. Even in factories and facilities, if dangerous places and cameras are installed, it is possible to actually capture dangerous acts and improve safety.

  Furthermore, FIG. 9 shows an example in which a running route is displayed on a map, and a change in motivation and a change in calorie consumption in the route are simultaneously displayed as a line graph. The running direction is from the front of the drawing to the back, and in the graph, the front is the start point and the back is the end point. The position of the graph is changed according to the shape of the running path for easy viewing. This shows that the motivation was high immediately after the start of running, but the motivation gradually decreased. In addition, the calorie consumption is maintained at a constant level, and it can be seen that there was a certain exercise effect.

  In the example of FIG. 9, a line graph of motivation changes and calorie consumption is collectively displayed on the left side of the running route diagram. However, the present embodiment is not limited to this, and as shown in the example of FIG. 10, a line graph of change in motivation is displayed on the left side of the diagram of the running route, and a line of calorie consumption is displayed on the right side of the diagram of the running route. Variations such as displaying a graph are possible.

  In this way, by displaying the life log on the time axis and two-dimensional map with the axis and scale of the quantified mental state displayed, how the person really felt when the event occurred , You can see how you thought, what kind of mental state you were. As a result, the reliability of the information is improved, information and behavior that cannot be understood by simple life log display can be analyzed and interpreted, and the utilization of the life log can be improved.

  As described above, the display processing unit 170 performs display processing of the time series change information of the mental state information corresponding to the position indicated by the time series change information of the position information.

  Thereby, it becomes possible to confirm the mental state of the user corresponding to each position of the user.

  In addition, the processing unit 130 may associate the estimated mental state information as an index with respect to the life log information acquired at a timing corresponding to the acquisition timing of the biological information used for the estimation of the mental state information. Good.

  Thereby, it becomes possible to confirm the mental state information at the acquisition timing of the biological information.

  Further, the processing unit 130 performs an event detection process based on the life log information, and based on the biological information acquired at a timing corresponding to the timing at which the life log information at which the event is detected is acquired, the mental state The information may be estimated, and the estimated mental state information may be added as an index to the life log information in which the event is detected.

  This makes it possible to check mental state information at the event occurrence timing.

4). Modified Example In order to estimate a mental state, in addition to biological information, if body motion information can be used simultaneously, the estimation accuracy can be further increased.

  Therefore, the biological information processing system 100 according to the present embodiment may further include a body motion information acquisition unit 190 that acquires body motion information of the user as illustrated in FIG. Then, the processing unit 130 estimates mental state information based on the biological information and the body motion information.

  The body motion information acquisition unit 190 acquires body motion information from, for example, a body motion sensor. As the body motion sensor, a motion sensor (acceleration sensor), a pressure sensor (contact pressure sensor), or the like is used. Further, as the body motion sensor, a plurality of types of sensors may be provided, or a configuration including one type of sensor may be used. The motion sensor is, for example, an acceleration sensor. The acceleration sensor is composed of an element whose resistance value is increased or decreased by an external force, for example, and detects triaxial acceleration information. The pressure sensor is, for example, a contact pressure sensor. The contact pressure sensor may be one that directly contacts the subject and measures the contact pressure, or may indirectly measure the contact pressure using a cuff structure or the like. That is, a piezoelectric element may be used, or an atmospheric pressure sensor or the like may be used.

  A specific example is shown in FIG. In the example of FIG. 11, the biological information and the activity state information are analyzed for Δt, the mental state is estimated from these comprehensive judgments, and added to the life log information.

  More specifically, the life log data LD1 to the life log data LD8 are acquired in time series as life log information. At the same time, the biological information acquisition unit 110 acquires the biological information 1 and the biological information 2, and the body movement information acquisition unit 190 acquires the body movement information 1 and the body movement information 2. Then, for each Δt period, the user's mental state information is estimated based on the biological information 1, the biological information 2, the body motion information 1, and the body motion information 2. In this example, mental state information IDX1 is assigned as an index to the life log information LD3. Similarly, mental state information IDX2 is assigned as an index to life log information LD5, and mental state information IDX3 is assigned as an index to life log information LD7.

  This makes it possible to estimate the mental state information more accurately.

5. Mental state estimation processing Next, mental state information estimation processing will be described in detail. In particular, among mental state information, a specific method for estimating the level of motivation and introspective mental states such as user evaluation of behavior from mental states such as tension, relaxation, and arousal. explain. Note that user evaluation of behavior is information that indicates, for example, the degree to which a user feels that he / she was good when he / she took an action or regrets that he / she should not do that action. is there. This user evaluation may be consciously felt by the user himself or unconsciously.

  First, mental states such as a tension state, a relaxed state, and an arousal state are estimated from biological information such as a heart rate (pulse), blood pressure, and body temperature. Here, a method of estimating from a heartbeat (pulse) will be described. Heartbeat (heart beat) represents the activity state of the heart. Since the heart is affected by the sympathetic nerve, the activity state of the sympathetic nerve can be estimated from the state of the heartbeat.

  Frequency analysis is a common method for analyzing the effects of sympathetic nerves and parasympathetic nerves in the heartbeat. The time change of the peak-to-peak distance (RR interval) in the heart rate change graph is subjected to frequency analysis (FFT), and is divided into a low frequency component LF and a high frequency component HF for analysis.

  The influence of the sympathetic nerve and the parasympathetic nerve can be quantified from the low frequency component LF and the high frequency component HF. This quantified value corresponds to the degree of mental stress, and it is possible to estimate a mental state such as being relaxed, nervous, or feeling good from the magnitude of the numerical value.

  Now, the quantified mental state such as a tension state does not always coincide with a mental state such as a feeling of watching TV (high motivation) or good (user evaluation). Even if they are numerically excited or relaxed, it is often impossible to judge with only one mental index that they are always in a comfortable state. Therefore, it is considered necessary to make a comprehensive judgment in consideration of other mental states based on the situation of the user at that time.

  Next, a method for estimating a mental state such as a motivation level or a user evaluation of a behavior result from a plurality of quantified mental states will be described.

  First, a plurality of mental states are digitized from a heart rate (HR), which is one of biological information, by frequency analysis of the heart rate. Here, an example is shown in which three indicators A: Relax-Excitement / Tension, B: Sleep-Awake, C: Fatigue-Energy (Physical Strength) shown in FIG. 12 are used. Specifically, A is the ratio of HF and LF, B is the HR at rest (basal HR), and C is the maximum value of HR (when the latest maximum value is low, the body is tired and feelings I guess I'm tired too).

  Since the indices of A, B, and C indicate opposite states, for example, in the case of A, the neutral point of relaxation and excitement is set to 0, and the maximum value when relaxing is set to -5. The maximum value when the user is in a numerical value such as 5. The same applies to B and C.

  Next, not only a plurality of mental states but also the movement state of the person is added to the estimation of the mental state. Even in the same mental state, when the exercise intensity increases, the influence of the mental state on the heartbeat decreases. Therefore, the influence of exercise intensity is expressed by a function M (I) as shown in FIG. 13 and used for estimation of a mental state. The horizontal axis in FIG. 13 is the exercise intensity I (Mets, etc.), and the vertical axis is the exercise state M (I), which is a function of I. In a state where the body does not move much, such as sleeping sideways, sitting, or raising the upper body, M (I) is lower than in other exercise states. M (I) becomes maximum when walking, and M (I) becomes smaller than when walking when exercise intensity such as running or sports is obtained. In the example of FIG. 13, the range of M (I) is set to −5 to 5 in accordance with the example of FIG.

  The mental state X is quantified from the following equation from the mental state and the movement state thus obtained. Note that α, β, and γ are variables and can be adjusted as appropriate.

In the example of FIG. 12, A = 3 and B = C = 2. Here, α, β, and γ are all set to 1, and the user is in a sitting state. Since M (I) = 0 in the sitting state, X = 7. That is, the height of motivation can be specified as 7.

  When motivation is high, there are many cases of excitement, but when the half body is sleeping or not feeling well, there is a possibility that it is just excited by some stimulus, in such a case Is not motivated enough. In addition, when exercising, because the pulse rises by the exercise itself, it is thought that the influence of motivation is reduced, so it is necessary to discount the influence of exercise. Thus, the more exact amount of motivation can be estimated by considering a plurality of mental states and exercises.

  As described above, the mental state can be quantified from the mental state due to the heartbeat and the movement state due to the movement. Similarly, in the case of blood pressure and body temperature other than the heartbeat, A, B, and C are obtained by estimation based on other biological information. In the case of a plurality of pieces of biological information, for example, A, B, and C are obtained from the heartbeat, A ′, B ′, and C ′ are obtained from the blood pressure, and the mental state is obtained by linear combination thereof.

  In summary, the processing unit 130 of the present embodiment estimates the height of the user's motivation as mental state information. The height of motivation may be estimated directly from biological information, or may be estimated from a tension state or the like.

  This makes it possible to display the life log information in association with the height of the user's motivation.

  Moreover, the process part 130 estimates user evaluation with respect to a user's action result as mental state information based on at least 1 information of a user's sleep state, calorie consumption, physical strength, physical condition, and a drinking state.

  Thereby, it is possible to display the life log information in association with the user evaluation for the action result.

6). Configuration Example of Measurement Device FIGS. 14A, 14B, and 15 are external views of the measurement device 400 (biological information detection device) of this embodiment. 14A is a view of the measuring apparatus 400 as viewed from the front direction side, FIG. 14B is a view as viewed from the upper direction side, and FIG. 15 is a view as viewed from the side direction side.

  As shown in FIGS. 14A to 15, the measuring apparatus 400 of the present embodiment includes a band unit 510, a case unit 530, and a sensor unit (biological sensor) 420. Case portion 530 is attached to band portion 510. The sensor unit 420 is provided in the case unit 530. Note that the measuring apparatus 400 of the present embodiment is not limited to the configuration shown in FIGS. 14A to 15 and the like, and some of the components are omitted, replaced with other components, or other components are replaced. Various modifications such as addition are possible.

  The band unit 510 is for wrapping around the user's wrist and mounting the measuring device 400. The band part 510 has a hole part 512 and a buckle part 514. The buckle part 514 includes a band insertion part 515 and a protrusion part 516. The user inserts one end side of the band part 510 into the band insertion part 515 of the buckle part 514, and inserts the projection part 516 of the buckle part 514 into the hole part 512 of the band part 510, whereby the measuring device 400 is placed on the wrist. Can be installed. In this case, the magnitude of the pressure on the sensor unit 420 (the pressure on the wrist surface) is adjusted according to which hole portion 512 the projection 516 is inserted into.

  The case part 530 corresponds to the main body part of the measuring apparatus 400. Various components such as the sensor unit 420, the biological information acquisition unit 110, the processing unit 130, the storage unit 150, and the display processing unit 170 are provided inside the case unit 530. That is, the case portion 530 is a housing that houses these components.

  The case portion 530 is provided with a light emission window portion 532. The light emission window part 532 is formed of a translucent member. The case portion 530 is provided with a light emitting portion (LED) mounted on a flexible substrate, and light from the light emitting portion is irradiated to the outside of the case portion 530 through the light emitting window portion 532.

  As shown in FIG. 15, the case portion 530 is provided with a terminal portion 531. When the measuring apparatus 400 is attached to a cradle (not shown), the terminal part of the cradle and the terminal part 531 of the case part 530 are electrically connected. Thereby, the secondary battery (battery) provided in the case part 530 can be charged.

  The sensor unit 420 detects measurement biological information such as a pulse wave of the subject. For example, the sensor unit 420 includes a light receiving unit and a light emitting unit. The sensor unit 420 includes a convex portion 552 that is formed of a translucent member and that makes contact with the skin surface of the subject to apply pressure. In this state where the convex portion 552 presses the skin surface, the light emitting portion emits light, and the light receiving portion receives the light reflected by the subject (blood vessel), and the light reception result is detected. The signal is output to the biological information acquisition unit 110 as a signal. Note that the measurement biological information to be detected by the measurement apparatus 400 of the present embodiment is not limited to the pulse wave (pulse rate), and the measurement apparatus 400 may measure biological information other than the pulse wave (for example, oxygen saturation in blood) Device for detecting temperature, body temperature, heart rate, etc.).

  FIG. 16 is an explanatory diagram regarding the mounting of the measuring apparatus 400 and the communication with the information processing apparatus 200.

  As shown in FIG. 16, the user who is the subject wears the measuring device 400 on the wrist 410 like a watch. As shown in FIG. 15, a sensor unit 420 is provided on the surface of the case unit 530 on the subject side. Accordingly, when the measuring device 400 is attached, the convex portion 552 of the sensor unit 420 comes into contact with the skin surface of the wrist 410 to apply pressure, and in this state, the light emitting unit of the sensor unit 420 emits light, and the light receiving unit By receiving the reflected light, measured biological information such as a pulse wave is detected.

  The measuring apparatus 400 and the information processing apparatus 200 are connected by communication so that data can be exchanged. The information processing device 200 is a mobile terminal device such as a smartphone, a mobile phone, or a future phone. Alternatively, the information processing apparatus 200 may be an information processing terminal such as a tablet computer. As a communication connection between the measurement apparatus 400 and the information processing apparatus 200, for example, proximity wireless communication such as Bluetooth can be employed. In this way, the measurement device 400 and the terminal device 200 are connected by communication, and various information (measurement biological information, activity state information) such as the pulse rate and calorie consumption is displayed on the display unit 430 (LCD, etc.) of the terminal device 200. Can be displayed. That is, various information obtained based on the detection signal of the sensor unit 420 can be displayed. Note that the calculation processing of information such as the pulse rate and calorie consumption may be executed by the measuring apparatus 400, or at least a part thereof may be executed by the terminal apparatus 200.

  Note that the biological information processing system, the server system, the biological information processing apparatus, and the like of this embodiment may realize part or most of the processing by a program. In this case, a biological information processing system, a server system, a biological information processing apparatus, and the like according to the present embodiment are realized by a processor such as a CPU executing a program. Specifically, a program stored in a non-temporary information storage device is read, and a processor such as a CPU executes the read program. Here, an information storage device (device readable by a computer) stores programs, data, and the like, and functions as an optical disk (DVD, CD, etc.), HDD (hard disk drive), or memory (card type). It can be realized by memory, ROM, etc. A processor such as a CPU performs various processes according to the present embodiment based on a program (data) stored in the information storage device. That is, in the information storage device, a program for causing a computer (an apparatus including an operation unit, a processing unit, a storage unit, and an output unit) to function as each unit of the present embodiment (a program for causing a computer to execute the processing of each unit). Is memorized.

  In addition, the biological information processing system, the server system, the biological information processing apparatus, and the like according to this embodiment may include a processor and a memory. The processor here may be, for example, a CPU (Central Processing Unit). However, the processor is not limited to the CPU, and various processors such as a GPU (Graphics Processing Unit) or a DSP (Digital Signal Processor) can be used. The processor may be a hardware circuit based on ASIC (Application Specific Integrated Circuit). The memory stores instructions that can be read by a computer, and the instructions are executed by the processor, thereby realizing each unit such as the biological information processing system, the server system, and the biological information processing apparatus according to the present embodiment. Will be. The memory here may be a semiconductor memory such as an SRAM (Static Random Access Memory) or a DRAM (Dynamic Random Access Memory), or may be a register or a hard disk. Further, the instruction here may be an instruction of an instruction set constituting the program, or an instruction for instructing an operation to the hardware circuit of the processor.

  Although the present embodiment has been described in detail as described above, it will be easily understood by those skilled in the art that many modifications can be made without departing from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included in the scope of the present invention. For example, a term described at least once together with a different term having a broader meaning or the same meaning in the specification or the drawings can be replaced with the different term in any part of the specification or the drawings. In addition, the configurations and operations of the biological information processing system, the server system, and the biological information processing apparatus are not limited to those described in the present embodiment, and various modifications can be made.

100 biological information processing system, 110 biological information acquisition unit, 130 processing unit,
150 storage unit, 170 display processing unit, 190 body movement information acquisition unit,
200 terminal device (information processing device, smart device, portable terminal device), 230 processing unit,
231 reception processing unit, 232 transmission processing unit, 233 control unit, 250 storage unit,
251 Information storage medium, 270 display processing unit, 400 measuring device, 410 wrist,
420 sensor unit (biological sensor), 430 display unit, 480 life log information acquisition unit, 510 band unit, 512 hole unit, 514 buckle unit, 515 band insertion unit,
516 Protruding part, 530 Case part, 531 Terminal part, 532 Light emitting window part, 552 Convex part, 600 Server system, 900 Network

Claims (19)

A biological information acquisition unit that acquires biological information measured by the biological sensor;
Based on the biological information, a processing unit that performs a process of estimating the mental state information of the user,
A storage unit for storing life log information;
Including
The processor is
The mental state information estimated based on the biological information is given to the life log information as an index,
The storage unit
The biological information processing system storing the life log information to which the mental state information is assigned as the index. In claim 1,
A biological information processing system comprising: a display processing unit that performs display processing of time-series change information of the mental state information based on the life log information. In claim 2,
The display processing unit
A biological information processing system that performs the display process of the time-series change information of the mental state information in association with the time-series change information of the life log information. In claim 3,
The life log information is
Including location information,
The display processing unit
A biological information processing system that performs the display process of the time-series change information of the mental state information corresponding to a position indicated by time-series change information of the position information. In claim 2,
The processor is
Based on the biometric information, identify the activity status information of the user,
The display processing unit
A biological information processing system that performs the display process of the time-series change information of the mental state information in association with the time-series change information of the activity state information. In claim 1,
Based on the life log information, including a display processing unit that performs display processing of time series change information of the mental state information,
The life log information is
Content information that is at least one of image information, video information, sound information, and text information,
The display processing unit
The first content information to the Nth content information are acquired as the life log information in chronological order,
The biological information processing system, wherein the time-series change information of the mental state information is displayed in association with each content information of the first content information to the Nth content information. In any one of Claims 2 thru | or 6.
The time series change information of the mental state information is:
A biological information processing system, which is information representing a time-series change in a value obtained by quantifying a mental state. In any one of Claims 1 thru | or 7,
The processor is
Associating the estimated mental state information as the index to the life log information acquired at a timing corresponding to the acquisition timing of the biological information used for the estimation of the mental state information. Living body information processing system. In any one of Claims 1 thru | or 7,
The processor is
An event detection process is performed based on the life log information,
Based on the biological information acquired at a timing corresponding to the timing at which the life log information at which the event is detected is acquired, the mental state information is estimated,
The biological information processing system, wherein the estimated mental state information is added as the index to the life log information in which the event is detected. In any one of Claims 1 thru | or 9,
A body motion information acquisition unit that acquires the body motion information of the user;
The processor is
A biological information processing system that estimates the mental state information based on the biological information and the body motion information. In any one of Claims 1 thru | or 10.
The processor is
The biological information processing system, which estimates the mental state information including at least one of the tension state, the excited state, and the relaxed state of the user based on the biological information. In any one of Claims 1 thru | or 11,
The processor is
A biological information processing system for estimating the height of motivation of the user as the mental state information. In any one of Claims 1 to 12,
The processor is
Biometric information characterized by estimating a user evaluation for the user's behavior result as the mental state information based on at least one of the user's sleep state, calories burned, physical strength, physical condition, and drinking state. Processing system. In any one of Claims 1 thru | or 13.
The mental state information is
A biological information processing system, which is information related to at least one of a user's evaluation of the user's tension state, excitement state, relaxed state, motivation height and action result. In any one of Claims 1 thru | or 14.
The life log information is
A biological information processing system, which is at least one of image information, video information, sound information, text information, and position information.   A server system comprising the biological information processing system according to claim 1.   A biological information processing apparatus comprising the biological information processing system according to claim 1. Obtain biological information measured by a biological sensor,
Based on the biological information, perform a process of estimating the mental state information of the user,
The mental state information estimated based on the biological information is given to life log information as an index,
The biological information processing method, wherein the life log information to which the mental state information is assigned as the index is stored. A processing unit that performs reception processing of life log information in which mental state information estimated based on biological information measured by a biological sensor is given as an index;
A storage unit for storing the received life log information;
Based on the life log information, as a display processing unit that performs processing for displaying time-series change information of the mental state information on a display unit,
A program characterized by operating a computer.