JP2019180666A - Pulse wave detection terminal program, pulse wave analysis server program and pulse wave detection analysis terminal program - Google Patents

Abstract

To perform mental analysis while using a smartphone as an information terminal.SOLUTION: A program is provided for making a computer of a smartphone 10 perform the following processing: pass determination processing for performing pass determination when all of measurement precision, heart rate, standard deviation and color calculation are present in a predetermined range; content/time/finger-tip information association data transmission processing for transmitting content/time/finger-tip information association data associating a finger-tip video brightness signal obtained just before that or RR interval data with content information showing a location place of the information obtained by a personal digital assistant or a pulse wave detection time to a server, whenever the personal digital assistant downloads information or every predetermined time; and screen information reception display processing for receiving the screen information from the server and displaying it on the screen.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a pulse wave detection terminal program, a pulse wave analysis server program, and a pulse wave detection analysis terminal program.

  Modern people always carry their smartphones and spend a lot of time operating their smartphones. A smartphone is a type of mobile phone equipped with an OS (operating system) for advanced mobile information terminals. Modern smartphones are equipped with high-resolution cameras.

  The inventor (hereinafter simply referred to as the inventor) described in the present specification uses a smartphone camera to photograph a fingertip to detect a pulse wave, and what kind of mental state it is (hereinafter referred to as a mental state). A device for measuring the surface) is proposed. The high frequency component HF of the RR interval represents parasympathetic nerve activity, and the low frequency component LF is defined as an index representing sympathetic nerve activity. The high frequency component HF is an integral value of a power spectrum of 0.15 to 0.4 Hz obtained by a filter that passes the band of 0.15 to 0.4 Hz after Fourier transform, and the low frequency component LF is 0.04. It is described that it is an integral value of a power spectrum obtained by a filter that passes a band of ˜0.15 Hz. (See Patent Document 1). In Patent Document 2, electrocardiographic data of a measurement subject is measured by an electrocardiogram monitor, and based on the measured electrocardiographic data, a high-frequency component HF, a low-frequency component LF, and a CVRR (electrocardiogram RR) (Interval variation coefficient) is calculated.

JP 2014-140587 JP 2011-120618 A

  Time spent using a smartphone (for example, browsing news, sending messages on LINE (registered trademark), listening to music, watching movies, etc.) is usually dedicated to collecting information for that purpose. The modern society is full of information, and most of the information can be obtained through a smartphone, which is one type of portable information terminal. Since it is difficult to understand what the emotions that arise from such information are, it is necessary to find a way to objectively understand their emotions. In addition, it is important in a society where diversity is positively recognized to know the reaction of others who have contacted the same information, whether the information is of interest to others, and what emotions they have about the information. Therefore, sharing the same excitement is important for community bond formation.

  Smartphones are also characterized by responsiveness that allows them to search for information related to each other one after another. Therefore, while paying attention to the same information, the mental aspect of the person using the smartphone (hereinafter simply referred to as the operator) is objectively evaluated, and the content of the information presented by the smartphone and the spirit of the smartphone operator If it is possible to collect overwhelming big data by linking them to each other, it is necessary to select information, time constraints for evaluation of information, difficulty in looking at yourself objectively, and access to the same information. The difficulty of knowing the various emotions of others can be solved.

  Regularly knowing one's mental aspects is also desirable for health management, but it is difficult and forgetful to make daily measurements using a device dedicated to measuring mental aspects. become. Smartphone operations are frequently performed in modern times, so if you can analyze the mental aspects at the same time, you can easily solve the problem of knowing your mental aspects every day.

  In order to solve the above-described problems, the present invention causes a smartphone computer to execute a pulse wave detection terminal program, a pulse wave analysis server program, and a pulse wave while using a smartphone (generally, a portable information terminal) as usual. A detection analysis terminal program is provided.

  The pulse wave detection terminal program of the present invention is a computer of a portable information terminal incorporating a camera, and R (red), which is captured by the camera at a predetermined time during which the operator is using information obtained by the portable information terminal. R, G, B component acquisition processing for acquiring G (green) and B (blue) components, video luminance signal acquisition processing for acquiring a video luminance signal obtained by adding the R, G, B components, and the video luminance signal A peak interval detection process for detecting a peak interval included in, a measurement accuracy determination process for determining a ratio (measurement accuracy) of normal peaks calculated by (number of normal peak intervals) / (total number of peak intervals), ( Heart rate calculation processing for obtaining the number of occurrences (heart rate) per minute of the peak interval calculated by (total peak interval number) / (total peak interval elapsed time: unit is minute), and standard of amplitude of the video luminance signal Deviation (vibration A standard deviation detection process for calculating (standard deviation), a color calculation process for calculating (the R component-the G component) and (the R component-the B component) (color calculation), the measurement accuracy, and the heart rate When the number, the standard deviation, and the color calculation are all within predetermined ranges, the video luminance signal is detected from the fingertip, and the peak interval is RR interval data. The pass determination process for determining pass, and each time the portable information terminal downloads information or every predetermined time, the fingertip video luminance signal or the RR interval data obtained immediately before and the portable information terminal are obtained. Content / time / fingertip information associated with content / time / fingertip information associated with content information indicating the location of information or pulse wave detection time A program for executing data transmission processing and screen information reception display processing for receiving screen information from the server and displaying the screen.

  The pulse wave analysis server program of the present invention provides a computer of a server connected to a portable information terminal, the fingertip video luminance signal or RR interval data transmitted from the portable information terminal, and the location of information obtained by the portable information terminal Content / time / fingertip information associated data receiving processing for receiving content / time / fingertip information associated data associated with content information indicating pulse wave detection time, and selective when receiving the fingertip video luminance signal RR interval data detection processing for detecting the RR interval data from the fingertip video luminance signal, resampling processing for converting the RR interval data into RR interval time-series data having a constant sampling period, and the RR interval time-series data Fast Fourier transform processing for fast Fourier transform and the frequency spectrum obtained by the fast Fourier transform. Psychosomatic analysis that separates the high-frequency component and low-frequency component of the signal and analyzes the mental surface based on the high-frequency component and the low-frequency component or based on the high-frequency component, the low-frequency component, and the heart rate Processing, screen creation processing for creating screen information for displaying the screen in association with the content information or the pulse wave detection time, and screen information transmission processing for transmitting the screen information to the portable information terminal. , Is a program for executing.

  The pulse wave detection / analysis terminal program according to the present invention provides a computer of a portable information terminal with a built-in camera, and the information obtained by the portable information terminal is captured by the camera every predetermined time while the operator is using R (red). , G (green), B (blue) component acquisition R, G, B component acquisition processing, video luminance signal acquisition processing to acquire the video luminance signal obtained by adding the R, G, B components, and the video luminance A peak interval detection process for detecting a peak interval included in the signal, a measurement accuracy determination process for determining a ratio (measurement accuracy) of normal peaks calculated by (number of normal peak intervals) / (total number of peak intervals), A heart rate calculation process for obtaining the number of occurrences (heart rate) of the peak interval in one minute calculated by (total peak interval number) / (total peak interval elapsed time: unit is minutes), and amplitude of the video luminance signal standard deviation Standard deviation detection processing for calculating (amplitude standard deviation), color calculation processing for calculating (color calculation) of (the R component-the G component) and (the R component-the B component), the measurement accuracy, and the When the heart rate, the standard deviation, and the color calculation are all within a predetermined range, the pass determination process for determining that the peak interval is RR interval data with good quality, and the portable information terminal receives information. Content / time / fingertip information associating process for associating the RR interval data with the content information indicating the location of the information obtained by the portable information terminal or the pulse wave detection time for each download or at a predetermined time; Resampling processing for converting RR interval data into RR interval time-series data having a constant sampling period, and fast Fourier transform for fast Fourier transform of the RR interval time-series data The high frequency component and the low frequency component of the frequency spectrum subjected to the conversion process and the fast Fourier transform are separated, and based on the high frequency component and the low frequency component, or the high frequency component, the low frequency component, and the heart rate A mental plane analysis process for analyzing the mental plane based on the screen, a screen creation process for creating screen information for displaying the mental plane in association with the content information or the pulse wave detection time, and the screen information. This is a program for executing screen display processing for displaying a screen on a display.

  According to the present invention, the smartphone operator causes the smartphone or / and the server computer to perform pulse wave detection analysis while contacting the Internet information, so that how the heart is moved by the Internet information is being operated. It is possible to notice how the heart was moved at the pulse wave detection time. In addition, it is possible to know how others have accepted the same information that the smartphone operator has contacted, and to compare and contrast with others to know their own characteristics. Furthermore, the server can analyze big data that associates the operator's feelings with the information downloaded by the smartphone, and can select information that would be useful in view of the mental aspects of the operator and send it to the smartphone. . In addition, you can manage your mind without forgetting while operating your smartphone.

It is a figure which shows a pulse wave detection analysis system. It is a drawing substitute photograph of the state which is detecting the pulse wave using a smart phone. It is an example of the waveform of a fingertip video luminance signal when a fingertip is imaged with a camera. It is a figure which shows typically the peak space | interval detection process which detects the peak space | interval contained in a video luminance signal. It is a figure explaining the parameter | index of a color determination. It is a top screen when a pulse wave detection terminal program is started in the computer of a smart phone. It is a screen display of “Change in feeling”. It is a screen display showing “trend analysis” analyzed by the server in “feeling change”. It is a screen display of “Your feelings NEWS” and “Everybody's feelings NEWS”. This is the screen that the server displays on the smartphone when the operator attributes have not yet been collected. It is a screen display in "stress analysis". It is a screen display of “feeling log” and “life rhythm”. It is a screen display of “stress radar chart” and “detailed data”. It is a screen display of “Monthly Report” and “Life Rhythm Comparison”. It is a screen display of “stress radar chart comparison” and “autonomic nerve age”. It is a screen display of “detailed data” and “recommended column”. It is a figure which shows each of "neutral", "concentration", "stress", "fatigue", and "relaxation". It is a flowchart of a 1st pulse wave detection terminal program. It is a flowchart of a pulse wave analysis server program. It is a flowchart of the 2nd pulse wave detection terminal program. It is a flowchart of a pulse wave detection analysis terminal program.

“First Embodiment”
The pulse wave detection terminal program according to the first embodiment captures information obtained by the portable information terminal on the computer of the portable information terminal incorporating the camera at a predetermined time during which the operator is using R (red). , G (green), B (blue) components acquisition R, G, B component acquisition processing, R, G, B component addition video luminance signal acquisition processing, and video luminance signal Peak interval detection processing to detect the included peak interval, measurement accuracy determination processing to determine the ratio of normal peaks (measurement accuracy) calculated by (number of normal peak intervals) / (total number of peak intervals), The number of occurrences (heart rate) per minute of the peak interval calculated by (number of peak intervals) / (total peak interval elapsed time: unit), and the standard deviation (amplitude) of the video luminance signal amplitude Standard deviation) Standard deviation detection processing to calculate, color calculation processing to calculate (color calculation) of (R component-G component) and (R component-B component), all of measurement accuracy, heart rate, standard deviation and color calculation A pass determination process for determining whether or not the video luminance signal is a good-quality fingertip video luminance signal for detecting a video luminance signal from the fingertip when each is in a predetermined range and the peak interval is high-quality RR interval data; Each time the information is downloaded or every predetermined time, the fingertip video luminance signal or RR interval data obtained immediately before is associated with the content information indicating the location of the information obtained by the portable information terminal or the pulse wave detection time Content / time / fingertip information association data transmission processing for transmitting data / time / fingertip information association data to the server, and screen information reception for receiving screen information from the server and displaying it on the screen Is a program for executing the communication display processing.

“Second Embodiment”
The pulse wave analysis server program of the second embodiment is a computer of a server connected to a portable information terminal, where the fingertip video luminance signal or RR interval data transmitted from the portable information terminal and the location of information obtained by the portable information terminal Content / time / fingertip information associated data receiving process for receiving content / time / fingertip information associated data associated with a pulse wave detection time and a fingertip video luminance signal. RR interval data detection processing for detecting RR interval data from a fingertip video luminance signal, re-sampling processing for converting RR interval data into RR interval time-series data having a constant sampling period, and fast Fourier transform of the RR interval time-series data Fast Fourier transform processing and high-frequency and low-frequency components of the frequency spectrum that has been fast Fourier transformed And analyzing the mental aspect based on the high frequency component, the low frequency component, and the heart rate, or based on the high frequency component, the low frequency component, and the heart rate. A psychological analysis process, a screen creation process for creating screen information for displaying the screen in association with the content information or pulse wave detection time, and a screen information transmission process for transmitting the screen information to the portable information terminal. This is a program for execution.

“Third Embodiment”
The pulse wave detection / analysis terminal program of the present invention is a computer of a portable information terminal with a built-in camera, in which R (red), which is captured by the camera at a predetermined time while the operator is using information obtained by the portable information terminal. Included in the video luminance signal, R, G, B component acquisition processing for acquiring G (green) and B (blue) components, video luminance signal acquisition processing for acquiring a video luminance signal obtained by adding R, G, B components Peak interval detection processing for detecting the peak interval to be detected, measurement accuracy determination processing for determining the ratio of normal peaks (measurement accuracy) calculated by (number of normal peak intervals) / (total number of peak intervals), and (all peaks (Number of intervals) / (total peak interval elapsed time: unit is minutes) heart rate calculation processing for obtaining the number of occurrences (heart rate) per minute of peak interval, and standard deviation of amplitude of video luminance signal (amplitude standard) Deviation) Standard deviation detection processing, color calculation processing for (R component-G component) and (R component-B component) calculation (color calculation), measurement accuracy, heart rate, standard deviation, and color calculation are all A pass determination process for determining that the peak interval is RR interval data with good quality when in each predetermined range, and every time the portable information terminal downloads information or every predetermined time, the RR interval data and the portable information Content / time / fingertip information linking processing for linking content information indicating the location of information obtained by the terminal or pulse wave detection time, and resampling processing for converting RR interval data to RR interval time-series data of a fixed sampling period A fast Fourier transform process for performing a fast Fourier transform of time series data of RR intervals, a high frequency component and a low frequency component of a frequency spectrum subjected to the fast Fourier transform, And analyzing the mental aspect based on the high frequency component and the low frequency component, or analyzing the mental aspect based on the high frequency component, the low frequency component and the heart rate, and the mental aspect as content information or pulse wave detection time This is a program for executing screen creation processing for creating screen information for screen display in association with the screen and screen display processing for displaying screen information on the display.

  The pulse wave detection terminal program of the first embodiment and the pulse wave analysis server program of the second embodiment are related in a complementary manner with their respective special technical features. In other words, the pulse wave detection terminal program according to the first embodiment is a content-link that associates a fingertip video luminance signal or RR interval data with content information indicating a location of information obtained by the portable information terminal or a pulse wave detection time. Content / time / fingertip information associated data transmission processing for transmitting time / fingertip information associated data to the server. On the other hand, the pulse wave analysis server program of the second embodiment uses the fingertip video luminance signal or the RR interval data transmitted from the portable information terminal, the content information indicating the location of the information obtained by the portable information terminal, or the pulse wave detection time. Content / time / fingertip information associated data reception processing for receiving associated content / time / fingertip information associated data. In this way, each special technical feature is complementarily related to each other, and a pulse wave detection analysis system is configured by executing the program. Further, the pulse wave detection analysis terminal program of the third embodiment shares special technical features with each of the pulse wave detection terminal program of the first embodiment and the pulse wave analysis server program of the second embodiment, By combining these special technical features and using a stand-alone configuration, processing similar to that performed by combining both programs is executed.

"Outline of the first embodiment"
The pulse wave detection terminal program according to the first embodiment executes the pulse wave detection process “while” using the camera function of the computer of the portable information terminal incorporating the camera. Since the concept of “while” pulse wave detection has never existed before, this will be explained first.

("While" pulse wave detection of this embodiment and "concentrated" pulse wave detection of the prior art)
In the pulse wave detection / analysis system, the operator of the smartphone “detects” “pulse wave” while “using” the smartphone as a portable information terminal. The word “while”, “pulse wave” and “detection” are connected to the word source of “while” pulse wave detection, which is named by the inventor. While using as a portable information terminal, you can see the screen display surface on the front side of the smartphone, operate the screen display surface that also functions as a touch input panel, browse the news, send a message on LINE (registered trademark) , In a state where consciousness is concentrated on listening to music, watching movies, etc. In other words, the pulse wave detection in such a state can be said to be “while” pulse wave detection.

  In addition to the LINE (registered trademark) mentioned above, the terms Facebook (registered trademark), Twitter (registered trademark), Instagram (registered trademark), COCOLOLO (registered trademark) are described in the specification and drawings. It is stated that they are registered trademarks, and it is not stated each time that they are registered trademarks.

  On the other hand, the conventional pulse wave detection technology, including the invention described in Patent Document 1, has performed pulse wave detection using a tool dedicated to detecting pulse waves. Conventionally, a smartphone functions as a tool dedicated to detecting pulse waves. Therefore, the operator of the smartphone consciously “concentrates” and concentrates only on “detection” of “pulse wave”. The pulse wave detection in such a state can be said to be a “concentrated” pulse wave detection.

  Comparing “while” pulse wave detection and “concentrated” pulse wave detection, the difference between them is as follows. In “while” pulse wave detection, attention is not focused on photographing the fingertip with the camera, and the time during which the position of the camera and the fingertip is kept in a predetermined relationship is irregular. On the other hand, in “concentrated” pulse wave detection, the time during which the position between the camera and the fingertip is maintained in a predetermined relationship is consciously managed, and the position between the camera and the fingertip is maintained in a predetermined relationship. In order to improve the detection accuracy of the pulse wave for a long time, for example, 2 minutes or more, even if it is lengthened, no big problem occurs.

  In `` while '' pulse wave detection, when an operator who is concentrating on exchanging information while looking at the screen of the smartphone unconsciously measures how long the fingertip is resting on the camera, The length varies, and is longer than 1 minute and shorter than 10 seconds.

  On the other hand, in the “concentrated” pulse wave detection, the operator concentrates his / her consciousness only on the pulse wave detection, and the operator can hold his / her fingertip on the camera for about 30 seconds. Also, in “concentrated” pulse wave detection, it is checked after detecting whether or not “concentrated” pulse wave detection has been performed properly, and a warning from the smartphone indicates that pulse wave detection has not been performed properly. Once found, there is no problem even if the “concentrated” pulse wave is detected again.

  What the operator is aware of when performing “while” pulse wave detection is to utilize a smartphone as a portable information terminal. Therefore, as in “concentrated” pulse wave detection, awareness is not concentrated in pulse wave detection. If the smartphone issues an excessive warning that the pulse wave detection has not been performed properly to the operator, the concentration of information acquisition and transmission while viewing the screen of the operator's smartphone is interrupted. In this case, since the pulse wave detection is not “while”, the smartphone cannot issue an excessive warning that the concentration is interrupted in the “while” pulse wave detection.

  In “while” pulse wave detection, the operator visually recognizes the screen display surface on the front side of the smartphone and operates the touch input panel to contact various information (contents). In many cases, the content changes every moment. Under such circumstances, even if it becomes clear after the fact that the pulse wave detection was not performed properly due to the occurrence of a warning from the smartphone, it will return to the original content again and detect the pulse wave detection again. However, the impression is different from the first contact with the content, and for this reason, the pulse wave detection is not scheduled for re-detection.

  Explain where the technical difficulties to achieve "while" pulse wave detection is and how the inventor has overcome this technical difficulty in contrast to conventional "concentrated" pulse wave detection do.

(Purpose of “While” pulse wave detection)
The purpose of "while" pulse wave detection is as follows.
First purpose The operator unconsciously how the mental aspect of the person who operates the smartphone (hereinafter referred to as the operator) changes depending on the content of the information (referred to as content) obtained from the smartphone. If the mental wave analysis can be performed “while” the pulse wave is detected, the correspondence between the contents and the mental side becomes clear. Therefore, the operator can select the content according to the mental state at that time, manage his / her mental aspect, and maintain a healthy mental state.

Second purpose Every day for health management, it is desirable for health management to perform “concentrated” pulse wave detection at a fixed time, and it is desirable for health management, but many people have such complicated habits. I don't get used to it. However, modern people are familiar with the habit of operating their smartphones many times a day. Therefore, when searching for various contents on a smartphone and enjoying it, if the operator can perform pulse wave detection unconsciously and perform a mental analysis, the operator can perform regular operations without burdening the user. Health management.

(Characteristics of pulse wave detected by smartphone)
Before describing the pulse wave detection “while”, the characteristics of the pulse wave detected by the smartphone will be described. A normal smartphone has a camera. Therefore, the smartphone captures the fingertip with the camera, obtains a luminance signal (referred to as a fingertip video luminance signal), and detects the pulse wave by processing the fingertip video luminance signal. The fingertip video luminance signal obtained by photographing the fingertip changes in accordance with the blood flow amount flowing through the fingertip blood vessel. The blood flow varies depending on the electrical signal sent to the heart muscle. Therefore, the fingertip video luminance signal contains a component corresponding to the electrical signal (pulse wave) of the heart muscle obtained by the electrocardiograph, so that the pulse wave can be detected by processing the fingertip video luminance signal. It will be.

  Since the electrocardiograph described in Patent Document 2 directly detects an electrical signal that controls the function of the heart in which the mental aspect is reflected, information on the mental aspect can be obtained with high accuracy. The method for obtaining the high frequency component HF and the low frequency component LF used for the analysis of the mental aspect is described in paragraph "0043" of Patent Document 2 and FIG. In summary, the operation of detecting RR interval data and plotting each RR interval as the amplitude at the time position of the backward R wave (the backward peak of the fingertip video signal detected by the camera corresponds) is sequentially repeated. Thus, RR interval data having unequal time intervals is created. Thereafter, a resampling process is performed to convert the RR interval data having unequal time intervals into time series data having RR intervals having equal time intervals. This conversion is a well-known interpolation method. Thereafter, the frequency spectrum of the RR interval time series data is obtained by using a well-known fast Fourier transform (FFT), and a low frequency component LF which is an integral value of a power spectrum component of 0.04 to 0.15 Hz. And a high frequency component HF that is an integral value of the power spectrum component of 0.15 to 0.40 Hz is obtained.

  On the other hand, the pulse wave component detected from the waveform of the fingertip video luminance signal by the camera of the smartphone is very weak, and (S / N; signal-to-noise ratio) is not good. The reason is that the camera does not capture the electrical signals that directly control the heart like an electrocardiograph, but captures the minute changes in the blood flow that flows through the blood vessels of the fingertips as a result of the movement of the heart. is there. In addition, what is photographed by the camera is an image noise component due to the reflection of an image other than the blood flow, an image noise component due to a blurring change of the fingertip and the lens of the camera, and an image due to ambient stray light reflected on the camera The original fingertip video luminance signal corresponding to the blood flow is contaminated by noise components and the like. As countermeasures for removing this noise contamination, the invention described in Patent Document 1 performs “advanced logic processing”, “functionalization processing of luminance data”, “Savitzky-Golay filter (Sabitsuki / Golay filter) processing”, etc. The noise component was removed. These noise removal processes had a remarkable noise removal effect for each process.

(Difficult to combine “While” pulse wave detection with conventional technology)
In the conventional “concentrated” pulse wave detection, the operator knew when to detect the pulse wave. Therefore, it was possible to detect a high-quality pulse wave over a certain period of time. However, the timing of performing “while” pulse wave detection is when the operator is concentrating on the screen display surface of the smartphone, and the operator is not aware of the timing in most cases. is there. On the other hand, at any time, it is difficult to keep the finger and the camera in the correct position at all times and keep it from moving. Therefore, it is self-evident that the pulse wave detection cannot be performed properly while using the conventional technology alone.

("While" pulse wave detection method)
* First “while” pulse wave detection method It is said that the heart rate of a healthy person is 50 to 100 times / minute. As described above, the high frequency component HF is an integral value of the power spectrum of 0.15 to 0.4 Hz, and the low frequency component LF is an integral value of the power spectrum of 0.04 to 0.15 Hz. If the low frequency component LF and the high frequency component HF are to be accurately detected, a fingertip video luminance signal that is continuous for at least 25 seconds or more is required to detect a low frequency of 0.04 Hz. That is, it is necessary to acquire a good fingertip video luminance with little continuous noise for at least 25 seconds.

  Therefore, in the first “while” pulse wave detection method, the camera of the smartphone captures the fingertip video signal at least every 25 seconds, preferably every 30 seconds with some margin. In the following description, 30 seconds is used as a reference time for pulse wave detection “while”. After that, the quality of the continuous fingertip video luminance signal for 30 seconds is confirmed, and only the high-quality fingertip video luminance signal is used as target data for pulse wave analysis. The number of RR interval data at this time is 25 when the heart rate is 50 times / minute, and 50 when the heart rate is 100 times / minute. These 25 to 50 time-unequally spaced RR interval data are resampled and converted to time-equalized RR interval time-series data, fast Fourier transformed, and then the high frequency component HF and the low frequency component LF will be separated.

  The first “while” pulse wave detection method is different from the conventional “concentrated” pulse wave detection method in which a pulse wave is detected by an operator's intention at a specific time. , B component is acquired by the smartphone computer every predetermined time of 30 seconds. Then, according to the four criteria, whether or not the continuously acquired video luminance signal, R, G, B component is a fingertip video luminance signal, R, G, B component with less noise available for pulse wave analysis. The computer automatically determines, and those that pass the determination are automatically extracted as fingertip video luminance signals and time-unequally spaced RR interval data (time-unequally spaced RR interval data). The content / time / fingertip information associated data in which the fingertip video luminance signal or the time-series RR interval data is associated with the content information indicating the location of the information obtained by the portable information terminal or the pulse wave detection time is transmitted to the server. Data that does not pass the judgment is discarded.

  The following four criteria (1) to (4) are used as criteria for determination. (1) The ratio (measurement accuracy) of normal peaks calculated by (number of normal peak intervals) / (total number of peak intervals). Here, the normal peak interval is, for example, an RR interval corresponding to a heart rate of 50 to 100 times / minute of a healthy person, 1.2 seconds to 0.6 seconds. The total peak interval is all peak intervals and includes a peak in a range wider than 1.2 seconds to 0.6 seconds. The reason why the normal peak interval occurs in all peak intervals is that even a healthy person may rarely suddenly generate a short RR interval or a long RR interval, and when the fingertip is not properly hitting the camera. This is because image noise leaks into the fingertip video luminance signal, the S / N is deteriorated, and the position of the peak point is affected by the noise. In many experiments by the inventor, the probability that the peak interval falls within the range of 1.2 seconds to 0.6 seconds is 94.2% in the case of the fingertip video luminance signal, and the video luminance of the orange memo pad as a comparative example In the case of the signal, it was 48%, in the case of the video luminance signal of the book with a red cover, it was 54%. Therefore, an appropriate numerical value of 70 to 90% can be adopted as the measurement accuracy acceptance standard.

  (2) Number of occurrences (heart rate) of peak interval calculated in (total peak interval number) / (total peak interval elapsed time: unit is minute) per minute. It is desirable to pass if it is 50 to 100 times / minute (heart rate acceptance criteria) which is the range of the heart rate of a healthy person. In the inventor's numerous experiments, 87 for the fingertip video luminance signal (heart rate), 113 for the video luminance signal of the orange memo pad that is the comparative example (heart rate), 113 In the case of the video luminance signal of the book, (heart rate) was 148. Therefore, 50 to 100 times / min was adopted as a heart rate pass criterion.

  (3) The standard deviation (amplitude standard deviation) of the amplitude of the video luminance signal. The amplitude standard deviation was determined by the inventors' experiment. In many experiments by the inventor, the range is 0.15 to 0.17 for the fingertip video luminance signal, and the range is 0.0034 to 0.0036 for the orange luminance memo pad video luminance signal as a comparative example. In the case of the video luminance signal of the book with a red cover, it was in the range of 0.04 to 0.0006. Therefore, an appropriate numerical value of 0.1 to 0.2 can be adopted as the standard deviation acceptance criterion.

  (4) Calculation (color calculation) of (R component-G component) and (R component-B component). Although the details will be described later with reference to FIG. 5, the inventor is in the dark (flash) outdoors (with flash), dark place (with flash), outdoors (without flash), and when the fingertip is accurately placed on the camera lens. None), when the image is taken with the fingertip halfway through the camera, when the image is taken away from the camera, when the image is placed while the terminal is placed on the desk, This standard was determined from the relationship between the luminance of each component of R (red), G (green), and B (blue) and R / G, R / B, RG, and RB. The inventor refers to FIG. 5 and, as a color calculation pass criterion, each of (R component-G component) and (R component-B component) is 20% (51) or more of the maximum luminance value (255). Adopted.

  As described above, it is determined whether or not (measurement accuracy acceptance criteria), (heart rate acceptance criteria), (standard deviation acceptance criteria), and (color calculation acceptance criteria) are satisfied every predetermined time of 30 seconds. Then, in the case of the pass determination, the process proceeds to the next processing step. When it is not the pass determination, the same process of 30 seconds is repeated again.

* Second “while” pulse wave detection method In the first “while” pulse wave detection method, (measurement accuracy), (heart rate), (amplitude standard deviation), (color) every predetermined time of 30 seconds. As a result, the result of the determination is obtained every predetermined time of 30 seconds. In such a method, if a failure determination is made at a predetermined time of 30 seconds, the determination after 30 seconds is again waited, and the timing for starting the pulse wave analysis is delayed. Therefore, regarding the acceptance criteria that can be sequentially judged among the above four acceptance criteria, when it is found that the criteria are not satisfied, the data accumulated so far is reset, and the reset time is determined for the next 30 seconds. By setting the start time of the predetermined time, the determination result can be obtained quickly. Furthermore, an abnormal peak interval elimination process can be added to obtain the determination result earlier.

(Abnormal peak interval elimination processing)
Each time a peak interval is detected from the video luminance signal, first, when an abnormal peak interval is detected, the whole process is reset as a failure determination, and pulse wave detection is performed again from the beginning. The abnormal peak interval refers to a peak interval greatly deviating from a peak interval of 1.2 seconds to 0.6 seconds corresponding to 50 to 100 times / minute, which is regarded as a heart rate of a healthy person. When it is eliminated by this abnormal peak interval elimination process, (measurement accuracy), (heart rate), and (color calculation) often do not pass, so reset the whole process and restart the pulse wave detection from the beginning. There is no big problem.

(Measurement accuracy)
If (the number of normal peak intervals from the start of the predetermined time to the time) / (the number of peak intervals from the start of the predetermined time to the time) is calculated (measurement accuracy pass standard), it is determined as a pass determination. Determination can be made every time a peak is detected from the time when the second peak is detected from the video luminance signal after the start of a predetermined time of 30 seconds. The (measurement accuracy acceptance criteria) in the second method can be variously determined, for example, the same as the first “while” pulse wave detection method (measurement accuracy acceptance criteria). In this case, since (measuring accuracy pass standard) is constant, if the normal peak interval is not at the beginning of the predetermined time when the (total number of peak intervals) is small, it is easy to make a failure determination, and the next 30 It can be said that it is efficient because the start of a predetermined time in seconds starts. On the other hand, those that continue to have normal peak intervals later cannot be saved. In consideration of this point, for example, at the beginning of a predetermined time, the (measurement accuracy acceptance standard) is relaxed and severe for a while, and at the end of the predetermined time, the same as in the first “while” pulse wave detection method ( Measurement accuracy acceptance criteria). If a situation that does not satisfy (measurement accuracy acceptance criteria) occurs within a predetermined time, the process returns to the initial process of pulse wave detection again at that time.

(Heart rate)
Every time the peak interval is detected from the video luminance signal, (the number of peak intervals from the start of the predetermined time to the current point) / (the total peak interval elapsed time from the start of the predetermined time to the current point: the unit is minutes) The heart rate at that time can be calculated. Similarly to the pass determination of (measurement accuracy), if there is no normal peak interval at the beginning of the predetermined time, it is easy to make a fail determination, and it can be said that it is efficient because the start of the next predetermined time of 30 seconds starts earlier. . On the other hand, those that continue to have normal peak intervals later cannot be saved. For example, at the beginning of a predetermined time, the (heart rate acceptance criteria) is relaxed and strict for a while, and the same (heart rate acceptance criteria) as in the first “while” pulse wave detection method at the end of the prescribed time. good. If a situation that does not satisfy the (heart rate acceptance criteria) occurs within a predetermined time, the process returns to the first process of pulse wave detection again at that time.

(Amplitude standard deviation)
The (amplitude standard deviation) of the video luminance signal is a well-known formula.
{(1 / n) Σ ₁ ⁿ (video luminance signal amplitude n−video luminance signal amplitude average value) ² } ^1/2 . The video luminance signal amplitude n is the amplitude of the nth video luminance signal from the start of a predetermined time output from the CMOS for each high-frequency clock signal generated by the smartphone hardware. The video luminance signal amplitude average value is an average value of the amplitudes of the first to nth video luminance signals. As a rule of thumb, the amplitude of the video luminance signal is such that the peaks and valleys of the amplitude alternate and the envelope slowly fluctuates. Therefore, (amplitude standard deviation) changes every time when n is small, and settles to a substantially constant value as n becomes large. Therefore, when all the other three criteria pass (standard deviation acceptance criteria) It is sufficient to determine whether or not to clear.

(Color calculation)
This is a calculation (color calculation) of (R component-G component) and (R component-B component). For example, it is desirable to pass if it is 20% or more of the maximum value of each luminance (color calculation pass criteria). (R component-G component) and (R component-B component) of one screen are detected at short intervals of two digits or more of the peak interval. However, changes in (R component-G component) and (R component-B component) in accordance with changes in blood flow are not significant. Therefore, when detecting (R component-G component) and (R component-B component), for example, it is desirable to obtain an average value of (color calculation) every time a peak interval is detected and to remove noise. When a situation occurs in which the average value of (color calculation) does not satisfy (color calculation pass criterion) every time the peak interval is detected, the process returns to the first process of pulse wave detection again.

  As described above, if any of (abnormal peak interval elimination processing), (measurement accuracy), (heart rate), or (color calculation) does not meet the acceptance criteria, at that time, again If the process returns to the first process of pulse wave detection, the process is continued for a predetermined time of 30 seconds even when there is a high probability of finally failing as in the first “while” pulse wave detection method. Without waiting for the elapse of a predetermined time, it is possible to return to the initial processing of pulse wave detection again. Whether or not (standard deviation acceptance criteria) is satisfied when (abnormal peak interval elimination processing) (measurement accuracy), (heart rate), and (color calculation) are all passed after a predetermined time of 30 seconds has passed In the case of a pass determination, the process proceeds to the next processing step. If it is not a pass determination, the same process for 30 seconds is repeated and executed again.

(Content / time / fingertip information linking process)
As described above, the RR interval data obtained by the first “while” pulse wave detection method or the second “while” pulse wave detection method, or the fingertip video luminance from which the RR interval data is based The signal is used in the following processing. The RR interval data that has obtained the failure determination or the fingertip video luminance signal that is the basis of the RR interval data is cleared and discarded at the time of the failure determination. Content information or pulse wave detection indicating the location of the fingertip video luminance signal or RR interval data obtained immediately before and the information obtained by the portable information terminal every time the portable information terminal downloads information or each time of pulse wave detection Content / time / fingertip information associated data transmission processing for transmitting content / time / fingertip information associated data associated with time to the server is executed.

  The following processing in the subsequent server is performed. Since the RR interval data that has obtained a pass determination is simply an information sequence of RR intervals, the operation of plotting each RR interval as the amplitude at the time position of the backward R wave (the backward peak of the fingertip video signal) is sequentially repeated. Thus, RR interval data having unequal time intervals is created. After that, a resampling process is performed to convert the RR interval data having time unequal intervals into RR interval time series data having time equal intervals (hereinafter simply referred to as RR interval time series data). Then, a fast Fourier transform process for performing fast Fourier transform on the time series data of RR intervals is executed, and a mental surface analysis is performed based on a high frequency component and a low frequency component of a frequency spectrum or based on the heart rate. Execute the process. Then, a screen creation process for creating screen information for displaying the screen in association with the content information or the pulse wave detection time is executed. After the processing in the server is completed, the smartphone executes screen information reception display processing for receiving screen information from the server and displaying the screen.

“Outline of Second Embodiment”
The pulse wave analysis server program of the second embodiment is complementary to the pulse wave detection terminal program of the first embodiment with special technical features. In other words, the pulse wave detection terminal program according to the first embodiment is a content-link that associates a fingertip video luminance signal or RR interval data with content information indicating a location of information obtained by the portable information terminal or a pulse wave detection time. On the other hand, the pulse wave analysis server program of the second embodiment transmits the time / fingertip information association data to the server. Content / time / fingertip information associated data associated with content information indicating the location of the user or pulse wave detection time.

  When the server receives the fingertip video luminance signal from the smartphone, the server selectively receives the RR interval data obtained through the RR interval data detection process for detecting the RR interval data from the fingertip video luminance signal, or the RR interval received from the smartphone. Resampling processing for converting data into RR interval time-series data having a fixed sampling period, fast Fourier transform processing for performing fast Fourier transform of RR interval time-series data, and high-frequency components and low-frequency components of the frequency spectrum subjected to fast Fourier transform The psychosomatic analysis processing for separating the components and analyzing the mental aspect based on the high-frequency component and the low-frequency component, or based on the high-frequency component, the low-frequency component, and the heart rate; Screen creation processing for creating screen information to display the screen in association with the detection time, and screen information And the screen information transmission processing to be sent to the band information terminal, to run. The heart rate can be easily obtained from the RR interval data or the fingertip video luminance signal.

  What the operator wants to obtain is, for example, a mental aspect immediately before the operator touches the specific content (information) and a mental change after touching the specific content. Therefore, immediately after the portable information terminal downloads the specific content, the content / time in which the fingertip video luminance signal or RR interval data obtained immediately before is associated with the content information indicating the location of the information obtained by the portable information terminal Since the server receives the fingertip information association data, it is possible to analyze the mental aspect immediately before the operator touches the specific content in the server. As the content information, a URL, an article title, and various keywords in the article (various keywords such as a keyword expressing joy, a keyword expressing sadness, a keyword expressing anger, a keyword expressing fear, and the like) can be used. In order to detect various keywords, it is efficient that a server whose processing speed is higher than that of the smartphone is extracted from the content of the information source searched using the associated URL.

  Then, when the operator downloads another content (that is, immediately after the use of the current information is finished), the fingertip video luminance signal or RR interval data obtained immediately before and the other information obtained by the portable information terminal are again obtained. The content / time / fingertip information associated data indicating the location of the content of the content is transmitted to the server, thereby analyzing the mental aspect immediately after the operator has contacted the specific content on the server. be able to. Further, the server can acquire the contents of the content by analyzing the content information indicating the location of the information. By associating the pulse wave detection time with the content information, the server can acquire the time of contact with the specific content and the time away from the specific content. The server determines in more detail what part of the content affected the operator's mental aspect from the content information and the pulse wave detection time, and the succession of the influence of the content on the operator's mental aspect. Change can be analyzed.

  Further, the server can analyze only how the mental aspect of the operator changes in the morning, noon, and evening, paying attention only to the pulse wave detection time, for example. Furthermore, when the position information obtained from the GPS built in the smartphone is linked to the content / time / fingertip information linked data and sent to the server, the server determines whether the mental position and the operator's stay position or movement The relationship with speed can be analyzed.

  In addition, since similar information from many smartphone operators is accumulated on the server, big data on the mental aspects of many operators in contact with the same content is accumulated. Therefore, the server can also notify a specific operator to the specific operator what kind of mental change has occurred in a large number of others in contact with the same content. Furthermore, the server can advise the download of the most appropriate content for a specific operator from the huge amount of content and mental data related to a large number of operators. The server can provide various other services, and individual embodiments will be described later.

  The server executes screen creation processing for creating image information to be displayed on the smartphone display so that the operator can understand the analysis results of those mental aspects at a glance, and performs screen information transmission processing for sending the screen information to the smartphone. Execute.

"Example of the first embodiment and the second embodiment"
Examples of the first embodiment and the second embodiment will be described below with reference to the drawings. In the first embodiment and the second embodiment, special technical features act complementarily to function as a pulse wave analysis system.

  Hereinafter, the pulse wave detection analysis system will be described with reference to the drawings.

"Pulse wave detection analysis system 1"
FIG. 1 is a diagram showing a pulse wave detection analysis system 1. The pulse wave detection / analysis system 1 includes a smartphone 10, a server 20, and the Internet 30. The smartphone 10 is one smartphone connected to the Internet 30 as with other smartphones (not shown). Other smartphones can also function in the same manner as the smartphone 10 by installing the pulse wave detection terminal program of this embodiment.
In the pulse wave detection / analysis system 1, the smartphone 10 is caused to function as a pulse wave detector by executing a “pulse wave detection / analysis terminal program” which is an application program developed by the inventors.

  A smart phone is a kind of portable information terminal having both functions of a mobile phone and a computer, and is a common name of a portable information terminal with a telephone function. The heart of the smartphone is an OS (Operation System). The OS controls a computer as hardware and various peripheral terminals, and provides a basic function for executing a pulse wave detection analysis terminal program as an application program. As typical examples of the OS, (Android) Android (registered trademark) and (iOS) IOS (registered trademark) are widely used. The “pulse wave analysis terminal program” is an application program that can be executed on such an OS.

  It is well known that smartphones incorporate various peripheral devices as well as telephone devices. Examples of peripheral devices known to be built-in include a camera, a flash for enabling photographing even in a dark place, an acceleration sensor, a GPS (Global Positioning System), an Internet connection device, etc., and a smartphone 10 also incorporates these.

  A camera arranged in the smartphone 10 as in a normal smartphone has a CMOS image sensor (Seamos image sensor) as an image sensor. The CMOS image sensor outputs a video signal corresponding to an object to be photographed by dividing it into three primary colors of R (red), G (green), and B (blue).

  FIG. 2 is a drawing-substituting photograph in a state in which a pulse wave is detected using the smartphone 10.

  FIG. 2A shows the smartphone 10 taken from the opposite side of the screen display surface. In FIG. 2A, only the operator's hand portion is displayed, and the operator who is viewing the screen display surface (the surface of the smartphone 10) on which various information is displayed is not displayed. The lens of the camera (hidden by the fingertip in FIG. 2A) is disposed on the back surface of the smartphone 10. FIG. 2B is a diagram viewed from the viewpoint of the operator who is viewing the screen display surface arranged on the surface of the smartphone 10. The operator operates the smartphone 10 while detecting the pulse wave “while” as described above. In “While” pulse wave detection, instead of turning on the flash according to the intention of the operator as in “Concentrated” pulse wave detection, automatic turning on / off is adopted based on the purpose of “While” pulse wave detection. ing. “Dark place (without flash)” in the first row of “abnormal measurement and judgment” in the lower part of FIG. 5 and “Dark place (with flash)” in the second line of “normal measurement and judgment” in the upper part of FIG. In the “dark place (no flash)”, all of R, G, and B are 30 or less. Therefore, when all of R, G, and B are 30 or less (12% or less of the maximum value 255), the flash is automatically turned on, and otherwise the flash is automatically turned off. It should be noted that the timing for automatically turning on / off the flash is not made every time the R, G, B signals are detected, but is frequently turned on / off due to noise that is made every time a pass determination is made. That is, when all of the average values of R, G, and B are less than a predetermined value (30 or less in the embodiment), the flash is turned on, and when it is larger than the predetermined value (greater than 30 in the embodiment), Turn off the flash. Each average value is cleared to prepare for the calculation of each average value of R, G, B for the next 30 seconds.

(Outline of processing in the smartphone 10)
In the pulse wave detection analysis system 1, the computer of the smart phone 10, which is a portable information terminal incorporating a camera, is caused to execute the following processing at predetermined times during which the operator is using the information obtained by the smart phone 10. The reason why the predetermined time is selected as 30 seconds which is 25 seconds or more is to cope with 0.04 Hz which is the lowest frequency of the low frequency component LF as described above. If it is 25 seconds or more, it is not necessary to limit to 30 seconds, but “30” is adopted in order to match the purpose of pulse wave detection.

(TS1) R, G, B component acquisition processing for acquiring R (red), G (green), B (blue) components imaged by the camera.
(TS2) Video luminance signal acquisition processing for acquiring a video luminance signal obtained by adding R, G, and B components.
(TS3) Peak interval detection processing for detecting a peak interval included in the video luminance signal.
(TS4) Measurement accuracy determination processing for determining the ratio (measurement accuracy) of normal peaks calculated by (number of normal peak intervals) / (total number of peak intervals).
(TS5) Heart rate calculation processing for obtaining the number of occurrences (heart rate) per minute of the peak interval calculated by (total peak interval number) / (total peak interval elapsed time: unit is minute).
(TS6) Standard deviation detection processing for calculating the standard deviation (amplitude standard deviation) of the amplitude of the video luminance signal.
(TS7) Color calculation processing for calculating (color calculation) of (R component-G component) and (R component-B component).
(TS8) RR interval data that is a high-quality fingertip video luminance signal for detecting a video luminance signal from the fingertip when all of the measurement accuracy, heart rate, standard deviation, and color calculation are within the predetermined ranges, and the peak interval is good. A pass determination process for determining pass.
(TS9) Every time the smartphone 10 downloads information or every predetermined time, content information or pulse wave detection time indicating the location of the fingertip video luminance signal or RR interval data obtained immediately before and the information obtained by the smartphone 10 Content / time / fingertip information associated data transmission processing for transmitting to the server 20 the content / time / fingertip information associated data associated with.
(TS10) Screen information reception display processing for receiving screen information from the server 20 and displaying the screen information.

The following (TS1) to (TS3) will be described.
(TS1) R, G, B component acquisition processing for acquiring R (red), G (green), B (blue) components imaged by the camera.
(TS2) Video luminance signal acquisition processing for acquiring a video luminance signal obtained by adding R, G, and B components. (TS3) Peak interval detection processing for detecting a peak interval included in the video luminance signal.

  FIG. 3 is an example of a waveform of a fingertip video luminance signal when the fingertip is photographed with a camera.

  The above processes (TS1) to (TS3) will be described with reference to FIGS. The vertical axis in FIG. 3 represents the luminance obtained from all the sensors corresponding to R (red), G (blue), and B (blue). The larger the luminance (the larger the fingertip video luminance signal), the larger the value on the vertical axis, and the smaller the luminance (the smaller the fingertip video luminance signal), the smaller the value on the vertical axis. The horizontal axis in FIG. 3 is the elapsed time from the origin, and the signal processing in the microprocessor of the smartphone 10 is all time-discrete, so the numerical value on the horizontal axis is the number of sampling numbers (the number of signals sampled from the origin. A number indicating whether or not there is). As described above, since a fingertip luminance signal having a time of 25 seconds or more is necessary for accurate analysis, in the embodiment, the fingertip luminance signal is acquired for 30 seconds, and the total number of samplings corresponds to 30 seconds. The number is set.

  As shown in FIG. 3, the fingertip video luminance signal is a signal having a certain range of luminance and repeating a luminance peak approximately periodically. Note that the sampling period shown on the horizontal axis in FIG. 3 is determined by the sampling clock generated by the hardware (crystal oscillator or the like) of the smartphone 10. In the pulse wave detection analysis terminal program, the pulse wave detection terminal program executes the processes (TS1) to (TS3) as an interrupt process by a hardware interrupt.

  FIG. 4 is a diagram schematically illustrating peak interval detection processing for detecting a peak interval included in the video luminance signal of (TS3). FIG. 4A is a partially enlarged view of FIG. Each broken line time t1 to time t14 in the vertical axis direction shown in FIG. 4A is a peak position, and each of the intervals between adjacent peak positions is a peak interval T1 to a peak interval T13. For the peak point, a signal obtained by differentiating the video luminance signal (hereinafter referred to as a differential video luminance signal) is generated, and a point where the differential video luminance signal rises to zero (0) and crosses is obtained.

  FIG. 4B is a diagram obtained from FIG. The operation of plotting each peak interval as an amplitude at the time position of the rear peak is sequentially repeated to create time-series data of peak intervals with unequal time intervals. The amplitude at time t2 is (t2-t1), the amplitude at time t3 is (t3-t2)... The amplitude at time t14 is (t14-t13). FIG. 4B is referred to as time interval data with peak intervals that are unequal in time. Since the amplitude at time tn is (tn−t (n−1)), both time and amplitude can be expressed simply by arranging the intervals T1, T2,... T14. Such peak interval time-series data is simply referred to as peak interval data. When the video luminance signal is a fingertip video luminance signal, FIG. 4B is referred to as RR interval data.

The following (TS4) to (TS7) will be described.
(TS4) Measurement accuracy determination processing for determining the ratio (measurement accuracy) of normal peaks calculated by (number of normal peak intervals) / (total number of peak intervals).
(TS5) Heart rate calculation processing for obtaining the number of occurrences (heart rate) per minute of the peak interval calculated by (total peak interval number) / (total peak interval elapsed time: unit is minute).
(TS6) Standard deviation detection processing for calculating the standard deviation (amplitude standard deviation) of the amplitude of the video luminance signal.
(TS7) Color calculation processing for calculating (color calculation) of (R component-G component) and (R component-B component).

  Unlike “concentrated” pulse wave detection, “while” pulse wave detection is the first time after confirming whether the detected signal is a fingertip video luminance signal and whether there is no noise contamination. The processing of (TS4) to (TS7) described above, which has a great significance in the pulse wave analysis target, is a fingertip video signal with less noise contamination in the detected video luminance signal, and the detected peak interval signal is of good quality. A criterion for determining whether or not the data is RR interval data is obtained.

  When noise contamination exists in the video luminance signal, a clear peak point may not be recognized in the video luminance signal, and conversely, a peak point may be detected although it is not originally a peak point. For example, if the fingertip moves and image noise occurs, a moving object other than the fingertip is reflected, the brightness changes due to stray light, the fingertip is not correctly placed on the camera, or the blood is too dark If the flow change is not successful, etc. The processes (TS4) to (TS7) are determination processes for assuring that the video luminance signal imaged by the camera is a fingertip luminance signal obtained by imaging the fingertip and is good quality RR interval data.

  (TS4) In the measurement accuracy determination process for determining the ratio (measurement accuracy) of normal peaks calculated by (number of normal peak intervals) / (total number of peak intervals), the normal peak interval is a peak interval within a predetermined range. Say. As described above, since a healthy person has a heart rate of about 50 to 100 times per minute, the peak interval is between 1.2 seconds and 0.6 seconds. The ratio of normal peaks calculated by (number of normal peak intervals) / (total number of peak intervals) (measurement accuracy) was 85% or more as an acceptance criterion.

  In the heart rate calculation process for determining the number of occurrences (heart rate) per minute of the peak interval calculated by (TS5) (total peak interval number) / (total peak interval elapsed time: unit is minutes), the heart rate of a healthy person Corresponding to the number range of 50 to 100 times / minute, (heart rate) was 50 to 100 times / minute, and in the case of 30 seconds, 25 to 50 times were used as acceptance criteria.

  (TS6) In the standard deviation detection process for calculating the standard deviation (amplitude standard deviation) of the amplitude of the video luminance signal, the (amplitude standard deviation) is set to 0.1 to 0.2 as a pass criterion.

  (TS7) In color calculation processing for calculating (color calculation) of (R component-G component) and (R component-B component), each of (R component-G component) and (R component-B component) is The case where the luminance value was 20% or more of the maximum value of the luminance (color calculation acceptance criteria) was determined as the acceptance criteria.

  FIG. 5 is a diagram for explaining an index for color determination.

  The shooting conditions for “normal measurement and judgment” in the upper part of FIG. 5 are taken when the fingertip is accurately applied to the camera lens under the three conditions of outdoor (with flash), dark place (with flash), and outdoor (without flash). It is. The right side of the upper row in FIG. 5 shows the luminance of each component of R (red), G (green), and B (blue) of the fingertip video luminance signal, R / G, R / B, and R− under each shooting condition. It is a figure of the table | surface which shows G and RB.

  The shooting conditions of “abnormal measurement and determination” in the lower part of FIG. 5 are as follows: when the fingertip is accurately applied to the lens of the camera but the image is taken in a dark place (without flash), the image of the fingertip is applied to the camera half When the image is taken away from the camera, when the terminal is placed on the desk and the image is taken, the image is taken under four conditions. The lower right side of FIG. 5 shows the luminance of each component of R (red), G (green), and B (blue) of the fingertip video luminance signal, R / G, R / B, and R− in each shooting condition. It is a figure of the table | surface which shows G and RB.

  When the upper right side of FIG. 5 is compared with the lower right side, normal determination (determination that (measurement accuracy), (heart rate), and (amplitude standard deviation) meet the acceptance criteria) and abnormality determination ( (Measurement accuracy), (Heart rate), (Amplitude standard deviation) In the case of determination that at least one of the three acceptance criteria does not meet the acceptance criteria, the magnitudes of (RG) and (RB) There is a big difference. That is, the sizes of (RG) and (RB) in normal determination are 10 times or more the sizes of (RG) and (RB) in abnormality determination.

  Therefore, if the sizes of (RG) and (RB) are 51 or more (20% of the maximum value 255), it is determined that a fingertip video luminance signal not contaminated with noise is detected. Adding judgment criteria makes the judgment more accurate.

The following (TS8) to (TS9) will be described.
(TS8) RR interval data that is a high-quality fingertip video luminance signal for detecting a video luminance signal from the fingertip when all of the measurement accuracy, heart rate, standard deviation, and color calculation are within the predetermined ranges, and the peak interval is good. A pass determination process for determining pass.
(TS9) Every time the smartphone 10 downloads information or every predetermined time, content information or pulse wave detection time indicating the location of the fingertip video luminance signal or RR interval data obtained immediately before and the information obtained by the smartphone 10 Content / time / fingertip information associated data transmission processing for transmitting to the server 20 the content / time / fingertip information associated data associated with.

  In (TS8), (Measurement accuracy) meets the acceptance criteria for measurement accuracy, (Heart rate) meets the acceptance criteria for heart rate, (Amplitude standard deviation) meets the acceptance criteria for amplitude standard deviation, ) Is a good fingertip video luminance signal for detecting a video luminance signal from the fingertip when the color calculation pass criterion is satisfied, and a pass determination is made that the peak interval is RR interval data with good quality. In other words, the fingertip video luminance signal that has passed this determination, or information about the mental aspect obtained by analyzing the RR interval data in the server 20 is highly accurate. In addition, when the process of automatically turning on / off the flash is executed, the following process is executed after the process of (TS8). When all the average values of R, G, and B are less than or equal to a predetermined value (30 or less in the embodiment), the flash is turned on, and when larger than the predetermined value (greater than 30 in the embodiment), the flash is turned on. Turn off the light. And each average value is cleared and it prepares for the calculation of each average value of R, G, B for the next 30 seconds.

In (TS9), every time the smartphone 10 downloads information or every predetermined time, content information or a pulse indicating the location of the fingertip video luminance signal or RR interval data obtained immediately before and the information obtained by the smartphone 10 The content / time / fingertip information association data associated with the wave detection time is transmitted to the server 20.
Thus, naturally the following combinations are included, and other combinations are possible.
(1) Each time the smartphone 10 downloads information, the fingertip video luminance signal or RR interval data obtained immediately before is linked to the content information indicating the location of the information obtained by the smartphone 10 or the pulse wave pulse wave detection time. The attached content / time / fingertip information association data is transmitted to the server 20.
(2) Each time the smartphone 10 downloads information, the fingertip video luminance signal or RR interval data obtained immediately before is linked to the content information indicating the location of the information obtained by the smartphone 10 and the pulse wave pulse wave detection time. The attached content / time / fingertip information association data is transmitted to the server 20.
(3) Content that links the fingertip video luminance signal or RR interval data obtained immediately before the content information indicating the location of the information obtained by the smartphone 10 and the pulse wave pulse wave detection time for each predetermined time The time / fingertip information association data is transmitted to the server 20.
(4) The content / time / fingertip information associated data in which the fingertip video luminance signal or RR interval data obtained immediately before is associated with the pulse wave pulse wave detection time is transmitted to the server 20 every predetermined time.

  Here, the fingertip video luminance signal or the RR interval data means that either the fingertip video luminance signal or the RR interval data may be transmitted to the server 20. When transmitting the fingertip video luminance signal, the amount of data to be transmitted is large, but the server 20 can perform the RR interval detection process with higher accuracy than in the smartphone 10. When transmitting RR interval data, the amount of data to be transmitted is smaller. The content information indicating the location of the content is, for example, URL: Uniform Resource Locator (Uniform Resource Locator).

  In (TS10), the screen information reception display process for receiving the screen information from the server 20 and displaying the screen is executed by the computer. The content of the screen display displayed on the display is the screen information created by the server 20. Since this is a response, the processing in the server 20 will be described.

  FIG. 6 is a top screen when the pulse wave detection terminal program is started on the computer of the smartphone 10. The heart symbol to the right of COCOLOLO NEWS in the top row in FIG. 6 (nicknamed “Tokimeki-chan”) blinks when all of (Measurement accuracy), (Heart rate), and (Standard deviation) are in the pass judgment state. To do. The operator does not need to be particularly aware of the blinking of “Tokimeki-chan”, and can recognize that the pulse wave detection has been performed correctly within a range that can be visually recognized.

(Outline of processing in the server 20)
The server 20 of the pulse wave detection analysis system 1 having the smartphone 10 that is a portable information terminal causes the computer of the server 20 to execute the following processing.
(SS1) Content / time / fingertip information associated data in which the fingertip video luminance signal or RR interval data transmitted from the smartphone 10 is associated with the content information indicating the location of the information obtained by the smartphone 10 or the pulse wave detection time. Received content / time / fingertip information associated data reception processing.
(SS2) RR interval data detection processing for detecting RR interval data from a selective fingertip video luminance signal when a fingertip video luminance signal is transmitted from the smartphone 10.
(SS3) Resampling processing for converting RR interval data into RR interval time-series data having a constant sampling period.
(SS4) Fast Fourier transform processing for fast Fourier transform of RR interval time series data.
(SS5) The high frequency component and the low frequency component of the frequency spectrum subjected to the fast Fourier transform are separated, and based on the high frequency component and the low frequency component, or based on the high frequency component, the low frequency component, and the heart rate, The mental analysis process to analyze.
(SS6) Screen creation processing for creating screen information for displaying the screen in association with the content information or the pulse wave detection time.
(SS7) Screen information transmission processing for transmitting screen information to the smartphone 10.

(SS1) to (SS2) will be described.
(SS1) Content / time / fingertip information associated data in which the fingertip video luminance signal or RR interval data transmitted from the smartphone 10 is associated with the content information indicating the location of the information obtained by the smartphone 10 or the pulse wave detection time. Received content / time / fingertip information associated data reception processing.
(SS2) RR interval data detection processing for selectively detecting RR interval data from the fingertip video luminance signal when receiving the fingertip video luminance signal.

  In (SS1), the computer of the server 20 receives the content / time / fingertip information association data, receives either the fingertip video luminance signal or the RR interval data, or both from the smartphone 10, and the smartphone The content information indicating the location of the information obtained by 10 or the pulse wave detection time is received. If the RR interval data is received, the process can be immediately transferred to (SS3). When the fingertip video luminance signal is received (SS2), RR interval data detection processing for detecting RR interval data from the fingertip video luminance signal can be selectively performed.

  The computer of the server 20 has a higher calculation speed and higher processing capability than the computer of the smartphone 10, so in the RR interval data detection process for detecting the RR interval data from the fingertip video luminance signal, the higher order than in the smartphone 10. The accuracy of the RR interval data can be improved by performing differential filtering. When only the fingertip video luminance signal is received, the server 20 selectively executes the RR interval data detection process in (SS2), and then moves the process to (SS3). When both the fingertip video luminance signal and the RR interval data are received, the server 20 moves the processing to (SS3) after executing the RR interval data detection process in the server 20, or uses the received RR interval data. It is possible to select whether to execute the process of (SS3).

(SS3) to (SS5) will be described. .
(SS3) Resampling processing for converting RR interval data into RR interval time-series data having a constant sampling period.
(SS4) Fast Fourier transform processing for fast Fourier transform of RR interval time series data.
(SS5) The high frequency component and the low frequency component of the frequency spectrum subjected to the fast Fourier transform are separated, and based on the high frequency component and the low frequency component, or based on the high frequency component, the low frequency component, and the heart rate, The mental analysis process to analyze.

  The RR interval data is time-series data with time unequal intervals as shown in FIG. In the fast Fourier transform in (SS4), if the RR interval data is not time-series data with equal time intervals, the accuracy of the fast Fourier transform is deteriorated. Also, if the number of samplings is small, spurious noise that does not exist is generated. Therefore, in (SS3), a known resampling process for converting the RR interval data into RR interval time-series data having a constant sampling period is performed. The resampling process is a high-order interpolation filtering process, and is a process for creating new data at a position corresponding to the resampling cycle. It is well known that the resampling period coincides with the period of the fast Fourier transform executed in (SS4), and the resampling period is shorter than the RR interval to increase the accuracy of the fast Fourier transform. Prior to the fast Fourier transform executed in (SS4), a known window function filtering process (for example, a filtering process using a Hanning window or a Hamming window) may be executed in order to reduce the occurrence of a known edge effect. The reason why the edge effect occurs is that the number of RR interval time-series data for 30 seconds is small, but the reason for setting it to 30 seconds is to match the pulse wave detection “while” as described above.

  The window function is a discrete function having the same time as the length of the RR interval time series data, that is, the same number of samples as the sampling number of the RR interval time series data. The well-known Hanning window is a Cosin function in which the first and last sampling values of time-series data are 0, and become the maximum in the middle between the first and last sampling points. The Hamming window is also a Cosin function similar to the Hanning window.

  (SS5) The high frequency component and the low frequency component of the frequency spectrum subjected to the fast Fourier transform are separated, and based on the high frequency component and the low frequency component, or based on the high frequency component, the low frequency component, and the heart rate, Regarding the mental analysis processing to be analyzed, the high frequency component HF of the RR interval represents parasympathetic nerve activity, and the low frequency component LF is defined as an index representing sympathetic nerve activity. The high frequency component HF is an integral value of a power spectrum of 0.15 to 0.4 Hz obtained by a filter that passes the band of 0.15 to 0.4 Hz after Fourier transform, and the low frequency component LF is 0.04. It is well known that it is an integral value of a power spectrum obtained by a filter that passes a band of ˜0.15 Hz. Usually, the logarithmic value of the high frequency component HF is denoted as LnHF to represent the parasympathetic nerve activity index, and the logarithmic value of the low frequency component LF is denoted as LnLF to represent the sympathetic nerve activity index. LnHF and LnLF are objective evaluations of the mental aspect of the operator regardless of whether or not the operator of the smartphone 10 is aware. The heart rate can be easily obtained from the RR interval data.

(SS6) to (SS7) will be described.
(SS6) Screen creation processing for creating screen information for displaying the screen in association with the content information or the pulse wave detection time.
(SS7) Screen information transmission processing for transmitting screen information to the smartphone 10.

“Various contents of (SS6) and (SS7)”
In (SS6), the contents of the screen information to be created are transmitted to the smartphone 10 in (SS7). Since it is embodied on the display of the smartphone 10, the contents of the processing executed by the computer of the server 20 will be described with reference to the screen displayed on the smartphone 10.

  The second row from the top of the top page of FIG. 6 is content classification, and the current screen displays the article title “Darkness of work style reform”, which is a related article of “main”.

(Change in feeling)
FIG. 7 is a screen that displays changes in feeling. (Change in feeling) is to display a mental change on the screen according to the information touched or in time series.

An example of displaying “change in feeling” on the screen will be described below. As shown in FIG. 7A, the operator's “feel” just before opening by tapping “Environment and action from childhood creates“ sports brain ”! ...” is displayed on the screen.
This display is displayed as a result of processing according to the following order.
* Sequence 1 “When the computer of the smartphone 10 downloads the content / time / fingertip information associated data transmission process of (TS9) (1) Each time the smartphone 10 downloads information, the fingertip video luminance signal or RR obtained immediately before The content / time / fingertip information associated data in which the interval data and the content information indicating the location of the information obtained by the smartphone 10 or the pulse wave pulse wave detection time are associated are transmitted to the server 20.
Here, the mental aspect at the pulse wave detection time 11:07:24, which is analyzed based on the fingertip video luminance signal or RR interval data obtained immediately before, is “the environment and behavior from childhood make a“ sports brain ” It is in the state when it has not yet contacted!
* Order 2 “The computer of the server 20 finishes the processing from the content / time / fingertip information tied data reception processing of (SS1) to the screen information transmission processing of (SS7).”
* Order 3 “The computer of the smartphone 10 performs the screen information reception display process of (TS10).” As a result, the screen shown in FIG.
FIG. 7B is a screen display of “change in feeling” displayed on the smartphone 10 after reading “Environment and action from childhood make“ sports brain ”!”.
When the next content is tapped, the server 20 determines that “after reading” and displays the screen of FIG. 7B corresponding to the mental aspect at the closest pulse wave detection time 11:12:36 before that time. it's shown. As described above, the timing at which the screen of FIG. 7B is displayed can be appropriately set by the smartphone 10 for the server 20. Not only “After reading” can be set, but also “every predetermined time” can be set. The server 20 performs the following processing from the order 4 onward while switching the screen display from FIG. 7 (A) to FIG. 7 (B).
* Sequence 4 “Fingertip video luminance signal or RR interval data obtained immediately before and information obtained by the smartphone 10 at a predetermined time in the content / time / fingertip information linking data transmission processing of the computer of the smartphone 10 (TS9) The content / time / fingertip information associated data in which the content information indicating the location and the pulse wave / pulse wave detection time are associated is transmitted to the server 20. "
Here, the “predetermined time” can be set in the smartphone 10. In this case, the time at which the smartphone 10 moves to the next content after downloading information is set as the “predetermined time”. ing. Since the pulse wave is detected every 30 seconds, the pulse wave is detected several times during this period. When all of the plurality of “while” pulse wave detections have passed, the screen corresponding to the fingertip video luminance signal or RR interval data obtained immediately before is shown in FIG. If the last “while” pulse wave detection before moving to this content is a failure determination, the content / time / fingertip information associated data is transmitted to the server 20 based on the previous previous pass determination result. FIG. 7B shows a screen corresponding to the time.
* Order 5 “The computer of the server 20 executes the screen information transmission processing of (SS7).”
* Order 6 “The computer of the smartphone 10 executes the screen information reception display process of (TS10).” As a result, the screen shown in FIG.
That is, the pulse wave analysis server program causes the computer of the server 20 to display the screen on the smartphone 10 by comparing the mental aspect immediately before contacting the content downloaded by the smartphone 10 with the mental aspect after contacting the content. It is also a program. Moreover, it is also a program for analyzing the tendency between the content downloaded by the smartphone 10 and the mental aspect and executing a process of creating screen information for displaying this tendency on the smartphone 10.
Even if the screen display as shown in FIG. 7B is not displayed, the smartphone 10 transmits the content / time / fingertip information associated data to the server 20 every time a pass determination is obtained. ) To (SS6) are performed to store the big data.

  As described above, the computer of the smartphone 10 uses the pulse wave as a background process of the process of displaying the content related to “the environment and behavior from childhood create a“ sports brain ”!”, Which is a process of the normal smartphone 10. The detection program is running. The face mark “Piripi” at time 11:07:24 has been changed to the face mark “perfect” at time 11:12:36, and below that, “It has become perfect!” Is displayed. Tap ▲ in the right corner and ▲ in the left corner to display the results of other patterns. (Cases such as batch ⇒ leisurely, leisure ⇒ tingling, etc. are displayed, and all patterns are displayed.)

  Here, each of “batchi”, “piri-piri”, and “piri-piri” is obtained as follows. Based on the big data of the autonomic nerve accumulated by the inventor, the distribution of the autonomic nerve balance Ln (LF / HF) which is the logarithm of (low frequency component LF / high frequency component HF) has already been created as shown in FIG. Yes. Considering the distribution of this autonomic balance Ln (LF / HF), the standard deviation of the distribution is defined as “batch” or “neutral”, and the parasympathetic dominant (Ln (LF / HF) exceeds the standard deviation. ) Is low)) is defined as "relaxing" or "relaxing", and the sympathetic dominant (Ln (LF / HF) is high) exceeding the standard deviation is defined as "spiring" or "stress" ( Whether the standard range is ± 1σ is variable based on the distribution characteristics.) Therefore, since the result of the calculation of the autonomic nerve balance Ln (LF / HF) at the pulse wave detection time 11:07:24 was in the range of “spirit”, it corresponds to “spirit” shown in FIG. A face mark is displayed. In addition, since the result of the calculation of the autonomic balance Ln (LF / HF) at the pulse wave detection time 11:12:36 was in the range of “batchiness”, it corresponds to “batchiness” shown in FIG. A face mark is displayed.

  FIG. 8A is a screen display showing “trend analysis” analyzed by the server 20 in “change in feeling”.

  As shown in the upper part of the screen display surface of the smartphone 10 in FIG. 8A, when the server 20 completes the “trend analysis”, “Your trend analysis result has arrived” is displayed. The message “Your trend analysis results have arrived” in FIG. When “view” on the right side is tapped, screen information is sent to the smartphone 10 so that the server 20 displays the screen shown in FIG. During this time, the server 20 performs the following processing.

  While the server 20 analyzes the mental aspect of the operator of the specific smartphone 10 at the present time, the server 20 analyzes the tendency of the operator from the big data regarding the operator. The analysis is performed as follows, for example. Based on the content information when the mental state of “piripiri” is found, the server 20 searches the past mental aspect of the operator of the specific smartphone 10, and from the “keyword” included in the content of the content, Classify types. Among them, the type of content changed from “keyword” to “batch” included in the content of the content whose operator has changed to “batch” is classified.

  When the server 20 completes the analysis of such big data, the upper “view” display as shown in FIG. The above-described analysis is always performed, and the database of the operator of the specific smartphone 10 is continuously updated based on the latest content.

  FIG. 8B is a screen display showing “trend analysis” analyzed by the server 20 in “change in feeling”.

  As shown in FIG. 8B, the server 20 displays a message “Click here for your trend analysis result!” And displays the message as follows. “You are ... POINT1“ When you're on the edge, you tend to read animal news. ”POINT2“ When you read lifestyle news, it ’s easy to do it! ” That is, the server 20 analyzes and notifies the tendency between the read article and the mental aspect at that time. The server 20 stores feelings before reading the news and the big data of feelings after reading the news. As a result, it is possible to find out what kind of news is being read at what kind of feeling. After reading what kind of news, etc., you can see the tendency of feeling change. By text mining for news etc., you can see how the feelings are influenced by the words that appear. Information about what kind of feeling you feel when you come into contact with it. In cooperation with external data such as weather forecasts, you can see the tendency to read sports articles on sunny days, and read political articles on rainy days. In addition, the number of steps can be collected using an acceleration sensor, and the relationship between the amount of activity and the feeling can be analyzed. These data can be used for preparing advice from the server 20, recommending news articles, and the like.

  FIG. 9 is a screen display of “your feeling NEWS” and “everybody feeling NEWS” that the server 20 displays on the smartphone 10. The screen shown in FIG. 9A is a screen displayed when tapping “Your feelings NEWS”. A list of articles that have recently changed from "Piripi" to "Batchari" is displayed.

  The screen shown in FIG. 9B is a screen displayed when tapping “everyone's feelings NEWS”.

“Minna no Kimochi NEWS” is for recognizing one's own mind (the operator of the smartphone 10) and the mind of many others.
That is, the pulse wave analysis server program causes the computer of the server 20 to execute a process of creating screen information for displaying the mental aspect of the server in comparison with the average mental aspect of others connected to the server. It is also a program for.

  The change from “Piripi” to “Leisurely” shows the most mental tendency among the operators in contact with “Environment and action from childhood create“ sports brain ”!”. Moreover, the tendency of the mental aspect with the most majority when reading the same article as the operator of the smartphone 10 is shown in the list.

  As for the operator of the smartphone 10 who has seen the screen display shown in FIG. 9 (A) and FIG. 9 (B), many of the others who have contacted the same article change their mentality from “Piripi” to “Leisurely”. On the other hand, I know that there is a tendency to change from “piripi” to “perfect”.

  FIG. 10A is a screen that the server 20 displays on the smartphone 10 when the server 20 has not yet collected the attributes of the operator of the smartphone 10. As shown in FIG. 10A, a message prompting the user to input “Please tell me your gender and date of birth!” Is displayed. When the operator of the smartphone 10 inputs gender and date of birth and taps OK, the smartphone 10 sends information regarding the sex and date of birth to the server 20. When the operator of the smartphone 10 taps Cancel, the message “Tell me your gender and date of birth!” Disappears.

  FIG. 10B is a screen displayed when the upper right gear icon is tapped. The server 20 causes the smartphone 10 to display a message “News and feelings that are better for you when you enter all information!”. When the gender, date of birth, height, weight, prefecture are entered and “Change” is tapped, the smartphone 10 sends these pieces of information to the server 20.

(Stress analysis process)
The stress analysis process uses the autonomic nerve balance Ln (LF / HF) described above, and the mental aspect is “batch” or “neutral”, “slowly” or “relaxed”, “spirit” or “stress”. It is not classified into classifications, but is a classification from another point of view in which the mental aspect is classified into five as follows in addition to the heart rate.

  The contents of the five classifications are “batch” or “neutral”, “concentration”, “spirit” or “stress”, “fatigue”, “relaxing” or “relaxing”. The meanings of “batchi” and “neutral” are synonymous. Then, the terms “neutral”, “stress”, and “relax” are used.

  FIG. 17 is a diagram illustrating each of “neutral”, “concentration”, “stress”, “fatigue”, and “relaxation”.

  With reference to FIG. 17, the meanings of the terms “neutral”, “concentration”, “stress”, “fatigue”, and “relax” will be described. The meanings of the terms shown in FIG. 17 are the distribution of total power (parasympathetic nerve activity index LnHF + sympathetic nerve activity index LnLF) and autonomic nerve balance Ln (LF / HF) from the big data of the autonomic nerve accumulated by the inventor. Based on the distribution of.

"fatigue"
Within the standard deviation of the distribution of total power (parasympathetic activity index LnHF + sympathetic activity index LnLF) is defined as “normal”, and when the total power (LnHF + LnLF) is low beyond the standard deviation, it is defined as “fatigue” (See FIG. 17A). In this case, any value of the autonomic nerve balance Ln (LF / HF) is determined as “fatigue”. Whether the standard range is set to ± 1σ is variable based on the distribution characteristics.

"neutral"
When the total power is higher than a certain level (usually in a state of high vitality), the distribution of the autonomic balance Ln (LF / HF) is taken into account and the standard deviation of the distribution is defined as “neutral” (See FIGS. 17A and 17B). Whether the standard range is set to ± 1σ is variable based on the distribution characteristics.

"relax"
The parasympathetic dominance exceeding the standard deviation, that is, the case where the autonomic nerve balance Ln (LF / HF) is low is defined as “relaxation” (see FIGS. 17A and 17B). Whether the standard range is set to ± 1σ is variable based on the distribution characteristics.

  The case where the sympathetic nerve predominates beyond the standard deviation, that is, the autonomic nerve balance Ln (LF / HF) is high is defined as “stress” (see FIG. 17C). Whether the standard range is set to ± 1σ is variable based on the distribution characteristics.

"stress"
In the case where the balance Ln (LF / HF) of the autonomic nerve is high and the sympathetic nerve is dominant, the distribution of the heart rate is taken into consideration, and the case where the heart rate is high due to the sympathetic nerve is defined as “stress” (FIG. 17 ( C), see FIG. (The threshold of whether the heart rate is high or low is variable from the distribution characteristics.)

"Concentration"
In the case where the balance of autonomic nerves Ln (LF / HF) is high and the sympathetic nerve is dominant, the state of sympathetic nerve is dominant and the heart rate is low is defined as “concentration” in consideration of the distribution of the heart rate (FIG. 17 ( C), see FIG. (The threshold of whether the heart rate is high or low is variable from the distribution characteristics.)

  FIG. 11A is a top menu screen when “stress analysis” is tapped. Each item can be selected by tapping "Kimochi Log", "Life Rhythm", "Stress Radar Chart", "Detailed Data", and "Monthly Report".

The “feeling log” is a screen display representing the mental state of the day.
“Life rhythm” is a screen display showing a life rhythm such as where is the time zone in which the person is most concentrated in the day and where is the time during which fatigue is accumulated.
The “stress radar chart” is a screen display that shows the tendency of one's stress by comparing the tendency of feeling for each day of the week and season, comparison with the same gender age as oneself.
“Detailed data” is a screen display in which detailed indexes of indexes such as heart rate, stress level, total power, and number of steps are graphed.
The “Monthly Report” is a screen display that describes relaxed contents (healing music, advice, etc.) tailored to the feeling by commenting on the tendency of the feeling of the month every month.

FIG. 11B is a screen display that the server 20 displays on the smartphone 10 before billing for stress analysis.
FIG. 11C is a screen display displayed on the smartphone 10 by the server 20 for ticket application (billing) for stress analysis.

  FIG. 12A shows a screen display of “feeling log”. The change of feeling is represented by a face mark. That is, after applying for a ticket from the smartphone 10 and tapping “feeling log” on the screen display of FIG. 11A, the server 20 displays the date and the face mark corresponding to the mental face in the order of the pulse wave detection time. Create screen information to display the screen side by side. Then, it is sent to the smartphone 10. As a result, the screen display of FIG. That is, the pulse wave analysis server program causes the computer of the server 20 to execute a process of creating screen information for displaying the screen by arranging the date and time and the face mark corresponding to the mental face in the order of the pulse wave detection times. It is also a program.

  FIG. 12B is a screen display of “life rhythm”. The vertical axis is time, and the horizontal axis is mental. The mental aspects are arranged in the order of “stress”, “concentration”, “neutral”, “relax”, and “fatigue”. A darker portion indicates that the corresponding mental tendency on the horizontal axis is strong.

  FIG. 12C shows another screen display of “life rhythm”. A column “Recommended for you” is provided to display content suitable for the current mental aspect of the operator.

FIG. 13A is a screen display of a “stress radar chart”. Each of “neutral”, “concentration”, “stress”, “fatigue”, and “relax” associated with the pulse wave detection time is displayed as a percentage. The percentage of the screen display is calculated as follows. For example, FIG. 13A shows a case where the total number of times measured in one day is 30 and the following results are obtained.
・ Neutral count: 3 times, the ratio is 3/30 = 10%
-Number of times of stress: In the case of 19, the ratio is 19/30 = 63%
-Number of times of fatigue: In case of 3 times, the ratio is 3/30 = 10%
・ Number of times of relaxation: In the case of 2 times, the ratio is 2/30 = 7%
-Number of times of concentration: In the case of 3 times, the ratio is 3/30 = 10%
That is, the pulse wave analysis server program creates screen information for displaying the mental states of “neutral”, “concentration”, “stress”, “fatigue”, and “relax” on the computer of the server 20 in%. It is also a program for executing

  FIG. 13B is a screen display of “detailed data”. “Heart rate”, “Stress”, “Total power” are displayed.

  FIG. 14A is a screen display when “Monthly Report” is tapped. "Life rhythm comparison", "Stress radar chart comparison", "Autonomic nerve age", "Detailed data", "Recommended column" items can be selected by tapping.

  FIG. 14B is a screen display of “life rhythm comparison”. This is a display for comparing “self” with “the whole (the whole smartphone connected via the Internet 30)”.

  FIG. 14C shows another screen display of “life rhythm comparison”. This is a display for comparing "self" with "whole". “Myself” is my own mental aspect, and “entire” is the average mental aspect of others connected to the server 20. In other words, the server 20 executes a process of displaying a comparison between its own mental aspect and the average mental aspect of others connected to the server.

  FIG. 15A is a screen display of “stress radar chart comparison”. This is a display for comparing “self” with “average of women in their 30s”. Data of any two or more days before, one day before, or today can be selected.

  FIG. 15B is another screen display of “Stress radar chart comparison”. This is a display for comparing “self” with “average of women in their 30s”. Data for any number of days in a week from Monday to Sunday can be selected.

  FIG. 15C is a screen display of “autonomous nerve age”. “Real age”, “autonomous nerve age” and “autonomous nerve age of near real age (people who are close to the operator and real age)” are displayed.

  FIG. 16A is a screen display of “detailed data”. You can select past detailed data by day, week, month, or year. In FIG. 16A, “heart rate”, “stress”, and “total power” are displayed.

  FIG. 16B is a screen display of “recommended column”. Some of the content is tailored to you, so you can tap the displayed content to display it on the screen.

  FIG. 18 is a flowchart of the pulse wave detection terminal program.

  FIG. 19 is a flowchart of the pulse wave analysis server program.

(Second pulse wave detection terminal program)
A second pulse wave detection program that is a modification of the first embodiment will be described. The second pulse wave detection program executes (abnormal peak interval elimination processing), (measurement accuracy), (heart rate), and (color calculation) determination processing each time a peak interval of a video luminance signal is detected. It is. If any one of the determination processes does not pass, all the processes are reset at that time, the process returns to the beginning, and the processes for each predetermined period are started again. As a result, the failure determination can be made without waiting for 30 seconds, so that the pulse wave detection for the next 30 seconds can be promptly performed. Other processes are not different from the pulse wave detection terminal program.

  The second pulse wave detection terminal program has a hardware clock interruption process synchronized with a sampling period for acquiring one screen of the video luminance signal, a peak detection interruption every time a peak of the video luminance signal is detected, and a pass processing routine. Yes.

  FIG. 20 is a flowchart of the second pulse wave detection terminal program.

In the second pulse wave detection terminal program, the computer of the smartphone 10 is configured such that R, G, B component acquisition processing, video luminance signal acquisition processing, and peak interval detection processing are synchronized with the sampling cycle of video luminance signal one screen acquisition. Executes wear clock interrupt processing.
(TS21) R, G, and B component acquisition processing for acquiring R (red), G (green), and B (blue) components captured by the camera is executed.
(TS22) A video luminance signal acquisition process for acquiring a video luminance signal obtained by adding the R, G, and B components is executed.
(TS23) Peak detection processing for detecting a peak included in the video luminance signal is executed.
(TS24) It is determined whether or not a peak is detected and whether or not 30 seconds have elapsed. If 30 seconds have elapsed, the process proceeds to a pass processing routine. If any of the pass determination results of (abnormal peak interval elimination process), (measurement accuracy), (heart rate), and (color calculation) fails, all processes are reset and the process returns to the beginning. The time elapses only when all three pass as a result of the determination processing of (abnormal peak interval elimination processing), (measurement accuracy), (heart rate), and (color calculation). Each time a peak is detected, the process executes a peak detection interrupt process.

In the peak detection interrupt, the computer of the smartphone 10 executes the following process.
(TS25) A peak interval that is an interval from the previous peak is detected.
(TS26) It is determined whether or not the peak interval is an abnormal peak interval for a predetermined number of times or more.
Here, how to set “abnormal peak interval” and “predetermined number” which is a positive integer of 1 or more becomes a problem. Since the value of the heart rate of a healthy person is 50 to 100 times / minute (1.2 seconds to 0.6 seconds / peak interval), for example, with k as a positive number of 1 or less, “ The “abnormal peak interval” is defined as follows.
1.2 seconds × (1 + k) <(abnormal peak interval) <0.6 seconds × (1-k)
Here, as k approaches 1, the (abnormal peak interval) expressed by the above equation becomes farther from the normal peak interval, which is the peak interval of a healthy person, and the effect of eliminating the abnormal peak interval is diminished. On the other hand, the closer k is to 0, the more likely the healthy person's peak interval is determined to be an abnormal peak interval. With regard to the “predetermined number of times”, the normal peak interval and the abnormal peak interval cannot be separated unless the number of times becomes smaller as k approaches 1 and the number of times becomes larger as k approaches 0. How to set the “abnormal peak interval” and “predetermined number of times” depends on the environment such as the performance of the camera and the ambient brightness, and can be appropriately set in consideration of these. For example, the setting may be such that k = 0.3, the abnormal peak interval is expanded by about 30% from the peak interval of the healthy person, and the predetermined number of times is one. If a predetermined number of abnormal peak intervals are detected (YES), all the processes are reset at that time and the process returns to the beginning. Otherwise (NO), the process proceeds to (TS27).

(TS27) It is determined whether (number of normal peak intervals) / (total number of peak intervals), that is, (measurement accuracy) is an acceptable standard. For example, if (measurement accuracy) is less than 85%, all the processes are reset at that time and the process returns to the beginning. Otherwise, the process proceeds to (TS28).
(TS28) (number of peak intervals so far) / (elapsed time until that point: unit is minutes), that is, (heart rate) is calculated. For example, if (heart rate) is less than 50 times / minute and more than 100 times / minute (when converted to 30 seconds, other than the range of 25 times / 30 seconds to 50 times / 30 seconds) The process is reset and the process returns to the beginning. Otherwise, the process proceeds to (TS29).
(TS29) If the result of the calculation (color calculation) of (R component-G component) and (R component-B component) is less than 20% of the maximum luminance (255), all the processes are reset and processed Will return to the beginning.

In the acceptance processing routine, the computer of the smartphone 10 executes the following processing.
(TS30) It is determined whether or not the video luminance signal satisfies the standard deviation acceptance criterion. If the standard deviation acceptance criterion is not cleared, all the processes are reset at that time, and the process returns to the beginning. Otherwise, the process proceeds to (TS31).
(TS31) Content information or pulse wave detection indicating the location of the fingertip video luminance signal or RR interval data obtained immediately before and the information obtained by the smartphone every time the smartphone downloads information or every predetermined pulse wave detection time Content / time / fingertip information associated data transmission processing for transmitting content / time / fingertip information associated data associated with time to the server is executed.

"Pulse wave detection analysis terminal program of the third embodiment"
The pulse wave analysis program of the third embodiment is a pulse wave detection analysis terminal program for causing the smartphone 10 to function as a stand-alone pulse wave detection analysis terminal.

  The pulse wave detection / analysis terminal program causes a computer of a smartphone incorporating a camera to execute the following processing at predetermined time intervals while the operator is using information obtained by the smartphone. As for every predetermined time, 30 seconds is a standard.

The following processing corresponds to the pulse wave detection terminal program of the first embodiment.
R, G, B component acquisition processing for acquiring R (red), G (green), B (blue) components imaged by the camera, and a video luminance signal for acquiring a video luminance signal obtained by adding the R, G, B components Acquisition processing, peak interval detection processing for detecting the peak interval included in the video luminance signal, and determining the ratio of normal peaks (measurement accuracy) calculated by (number of normal peak intervals) / (total number of peak intervals) A measurement accuracy determination process, a heart rate calculation process for determining the number of occurrences (heart rate) in one minute of the peak interval, a standard deviation detection process for calculating a standard deviation (amplitude standard deviation) of the amplitude of the video luminance signal, (R The peak interval when the color calculation processing for calculating (component-G component) and (R component-B component) (color calculation), and the measurement accuracy, heart rate, standard deviation, and color calculation are all within the predetermined ranges. Is RR interval data with good quality The pass determination process for determining pass and the RR interval data and the content information indicating the location of the information obtained by the smartphone or the pulse wave detection time are linked every time the smartphone downloads information or every predetermined time Content / time / fingertip information linking processing.

The following processing corresponds to the pulse wave analysis server program of the second embodiment.
Resampling processing for converting RR interval data into RR interval time-series data having a fixed sampling period, fast Fourier transform processing for performing fast Fourier transform of RR interval time-series data, and high-frequency components and low frequency of the frequency spectrum subjected to fast Fourier transform A psychological analysis process that separates frequency components and analyzes a mental aspect based on a high frequency component and a low frequency component, or based on a high frequency component, a low frequency component, and a heart rate; And screen creation processing for creating screen information for screen display in association with the wave detection time.

FIG. 21 shows a flowchart of the pulse wave detection analysis terminal program.
(KS1) R, G, B component acquisition processing for acquiring R (red), G (green), and B (blue) components imaged by the camera (KS2) Acquisition of a video luminance signal obtained by adding R, G, and B components (KS3) Peak interval detection processing for detecting the peak interval included in the video luminance signal (KS4) (normal peak interval number) / (normal peak interval number) Measurement accuracy determination processing for determining (measurement accuracy) (KS5) Heart rate for obtaining the number of occurrences (heart rate) per minute of the peak interval calculated by (total peak interval number) / (total peak interval elapsed time: unit is minute) Number calculation processing (KS6) Standard deviation detection processing for calculating standard deviation (amplitude standard deviation) of video luminance signal (KS7) (R component-G component) and (R component-B component) calculation (color calculation) Color calculation (KS8) A pass determination process for determining that the measurement accuracy, the heart rate, the standard deviation, and the color calculation are all in a predetermined range and that the peak interval is RR interval data with good quality (KS9) Content / time / fingertip information associating process (KS10) for associating RR interval data with content information indicating the location of information obtained by a smartphone or pulse wave detection time every time information is downloaded or every predetermined pulse wave detection time ) Resampling process for converting RR interval data into RR interval time-series data with a fixed sampling period (KS11) Fast Fourier transform process for performing fast Fourier transform of RR interval time-series data (KS12) High frequency of frequency spectrum subjected to fast Fourier transform The component and the low frequency component are separated, and based on the high frequency component and the low frequency component, Or, psychological analysis processing that analyzes the mental plane based on the high-frequency component, the low-frequency component, and the heart rate (KS13) Create screen information for displaying the mental plane in association with the content information or the pulse wave detection time. Screen creation processing (KS14) Screen display processing for displaying screen information on the display

  Examples that combine all or part of the components described in the examples of the first embodiment, the examples of the second embodiment, the examples of the third embodiment, and other examples. It can be implemented and is included in the embodiments of the present application. In addition, the Example using the component which has the same effect | action even if the shape of the component represented by each Example mentioned above differs is also contained in embodiment of this application. It goes without saying that the present invention includes not only these embodiments but also the scope of the same technical idea.

1 Pulse Wave Detection Analysis System 10 Smartphone 20 Server 30 Internet

Claims (8)

In the computer of the portable information terminal with a built-in camera,
Every predetermined time when the operator is using the information obtained by the portable information terminal,
R, G, B component acquisition processing for acquiring R (red), G (green), B (blue) components imaged by the camera;
Video luminance signal acquisition processing for acquiring a video luminance signal obtained by adding the R, G, and B components;
A peak interval detection process for detecting a peak interval included in the video luminance signal;
A measurement accuracy determination process for determining a ratio (measurement accuracy) of normal peaks calculated by (number of normal peak intervals) / (total number of peak intervals);
A heart rate calculation process for obtaining the number of occurrences (heart rate) in one minute of the peak interval calculated by (total peak interval number) / (total peak interval elapsed time: unit is minutes);
A standard deviation detection process for calculating a standard deviation of the amplitude of the video luminance signal (amplitude standard deviation);
Color calculation processing for calculating (color calculation) of (the R component-the G component) and (the R component-the B component);
A fingertip video luminance signal for detecting the video luminance signal from the fingertip when all of the measurement accuracy, the heart rate, the standard deviation, and the color calculation are in respective predetermined ranges, and the peak interval is improved. A pass determination process for performing pass determination as being RR interval data;
Each time the portable information terminal downloads information or at a predetermined time, content information or a pulse indicating the location of the fingertip video luminance signal or the RR interval data obtained immediately before and the information obtained by the portable information terminal Content / time / fingertip information associated data transmission processing for transmitting content / time / fingertip information associated data associated with wave detection time to the server;
Screen information reception display processing for receiving screen information from the server and displaying the screen,
Pulse wave detection terminal program. To the server computer connected to the portable information terminal,
Content / time / fingertip information associated data in which fingertip video luminance signal or RR interval data transmitted from the portable information terminal is associated with content information indicating the location of information obtained by the portable information terminal or pulse wave detection time Content / time / fingertip information associated data reception processing,
RR interval data detection processing for selectively detecting the RR interval data from the fingertip video luminance signal when receiving the fingertip video luminance signal;
Resampling processing for converting the RR interval data into RR interval time-series data having a constant sampling period;
A fast Fourier transform process for performing a fast Fourier transform of the RR interval time-series data;
The high frequency component and the low frequency component of the frequency spectrum subjected to the fast Fourier transform are separated, and based on the high frequency component and the low frequency component, or based on the high frequency component, the low frequency component, and the heart rate. A mental analysis process to analyze the surface,
Screen creation processing for creating screen information for displaying the screen in association with the content information or the pulse wave detection time;
Screen information transmission processing for transmitting the screen information to the portable information terminal,
Pulse wave analysis server program. In the computer of the portable information terminal with a built-in camera,
Every predetermined time when the operator is using the information obtained by the portable information terminal,
R, G, B component acquisition processing for acquiring R (red), G (green), B (blue) components imaged by the camera;
Video luminance signal acquisition processing for acquiring a video luminance signal obtained by adding the R, G, and B components;
A peak interval detection process for detecting a peak interval included in the video luminance signal;
A measurement accuracy determination process for determining a ratio (measurement accuracy) of normal peaks calculated by (number of normal peak intervals) / (total number of peak intervals);
A heart rate calculation process for obtaining the number of occurrences (heart rate) in one minute of the peak interval calculated by (total peak interval number) / (total peak interval elapsed time: unit is minutes);
A standard deviation detection process for calculating a standard deviation of the amplitude of the video luminance signal (amplitude standard deviation);
Color calculation processing for calculating (color calculation) of (the R component-the G component) and (the R component-the B component);
A pass determination process for performing a pass determination that the measurement accuracy, the heart rate, the standard deviation, and the color calculation are RR interval data with good quality when the peak interval is in each predetermined range;
Content / time / linkage of the RR interval data and the content information indicating the location of the information obtained by the portable information terminal or the pulse wave detection time every time the portable information terminal downloads information or at a predetermined time Fingertip information linking process;
Resampling processing for converting the RR interval data into RR interval time-series data having a constant sampling period;
A fast Fourier transform process for performing a fast Fourier transform of the RR interval time-series data;
Separating the high frequency component and the low frequency component of the fast Fourier transformed frequency spectrum, based on the high frequency component and the low frequency component, or based on the high frequency component, the low frequency component, and the heart rate, Or a mental analysis process for analyzing a mental aspect based on the high frequency component, the low frequency component, and the heart rate;
Screen creation processing for creating screen information for displaying the screen in association with the content information or the pulse wave detection time;
Screen display processing for displaying the screen information on a display,
Pulse wave detection analysis terminal program. Each determination process of the measurement accuracy, the heart rate, and the color calculation is executed every time the peak interval of the video luminance signal is detected,
If any one of the determination processes does not pass, all the processes are reset at that time, the process is returned to the beginning, and the process for each predetermined period is started again.
The pulse wave detection terminal program according to claim 1. In the screen creation process,
In order to execute processing for creating screen information for displaying the screen by arranging the date and time and the face mark corresponding to the mental side in the order of the pulse wave detection time,
The pulse wave analysis server program according to claim 2. In the screen creation process,
Analyzing the trend between the read article and the mental aspect at that time, and executing the process of creating screen information for displaying the trend on the screen,
The pulse wave analysis server program according to claim 2. In the screen creation process,
In order to execute the process of creating screen information for displaying the mental states of “neutral”, “concentration”, “stress”, “fatigue”, and “relax” in%,
The pulse wave analysis server program according to claim 2. In the screen creation process,
In order to execute a process of creating screen information for displaying a comparison between the mental aspect of oneself and the average mental aspect of another person connected to the server,
The pulse wave analysis server program according to claim 2.