TW201923694A - Program and information processing system - Google Patents

Abstract

A program that is characterized by making a computer function as, a measurement means that measures an expression value for a user included in a photographic image, a display means that displays the photographic image and the expression value measured by the measurement means, an instruction means that specifies the expression value measured by the measurement means and gives instructions to the user for the photography of the face of the user by an imaging device, an extraction means that analyzes a facial image for the user captured by the imaging device and extracts the positions of a plurality of feature points in the facial image, a calculation means that uses the positions extracted by the extraction means for the plurality of feature points to calculate a stress index value, and an output means that outputs the stress index value calculated by the calculation means. The program is also characterized in that, from a plurality of feature points obtained from a facial image of the user when the user is making the expression of the designated expression value, the calculation means calculates, as the stress index value, a value that expresses facial distortion.

Description

Program and information processing system

The invention relates to a program and an information processing system.

Although there are differences in the degree of psychological or spiritual stress, it has a significant impact on the health of modern people. Accumulation of stress can lead to increased emotional ups and downs, fatigue, loss of appetite, fever, insomnia and other physical disorders, and thus induce lifestyle habits such as obesity and diabetes. Ignorance can cause mental illness such as depression. Danger. Therefore, in recent years, the importance of mental health has been promoted. It is not necessary for everyone to manage their own stress, but also for enterprises to grasp the stress status of their employees and manage them appropriately.

As a method for measuring a person's stress state, for example, Patent Document 1 discloses a measurement device that analyzes a change in heartbeat number and respiratory information caused by stress. In addition, methods for measuring the state of stress (depression) based on the appearance state of stress-related genes and changes in blood specific amino acid concentrations have also been proposed (Patent Documents 2 and 3).

Cortisol is a hormone secreted by the pararenal cortex, and its secretion will increase under excessive stress, so it is also called stress hormone. At present, the stress state is measured based on the cortisol concentration in body fluids such as blood, urine, and saliva, and it has been reported that the cortisol concentration in saliva can reflect the stress state excellently (Non-Patent Documents 1 and 2). In addition, there are devices and the like that can reduce the burden on a subject when collecting a sample to measure the concentration of cortisol in a body fluid (Patent Document 4).

However, according to the conventional methods described above, it is necessary to purchase a device for pressure measurement, or an invasive treatment such as blood collection must be performed in order to obtain a measurement sample during actual measurement. For these reasons, the existing methods have not been widely used for daily pressure. management.

It is often said in the streets that facial expressions change or faces are distorted when stressed. Indeed, if you continue to be in a stressful state of tension, you will experience facial expression muscle tension and stiffness, and expression stiffness. However, to date, there has not been any example of quantitatively correlating changes in the face (expression, etc.) with stress.
＜ Prior art literature ＞
＜ Patent Literature ＞

Patent Document 1: (Japan) JP 2010-234000 Patent Document 2: (Japan) JP 2013-110969 Patent Document 3: (Japan) Patent No. 5856364 Patent Document 4: (Japan) JP 2012 －251857 <Non-patent literature>

Non-Patent Document 1: Ross F. Vining, et al., Ann. Clin. Biochem, Vol. 20, PP. 329-335 (1983)
Non-Patent Document 2: J. Gonzalez-Cabrera, et al., STRESS: The International Journal on the Biology of Stress, Vol. 17, Issue 2, pp. 149-156 (2014)

<Problems to be Solved by the Invention>

Under the above-mentioned technical situation and background, the present invention aims to provide a program that can easily and non-invasively evaluate the stress state of a subject.
＜ Means for solving problems ＞

The inventors of the present invention have repeatedly and persistently studied the relationship between changes in the subject's face (expression, etc.) and stress while verifying the effects of stress on a person's mood and body. As a result, it was found that the stress of the subject also affected the left-to-right distortion of the face, and the behavior of aligning his own face with the camera. And, for the first time, it was found that by using parameters obtained by processing a subject's face image taken under certain conditions, an index with good reproducibility can be obtained, which can be used to quantitatively evaluate the state of stress, thereby completing the present invention .

That is, the present invention provides a program, which is characterized in that a computer functions as the following units: a measurement unit that measures the expression value of a user included in a photographed image; a display unit that displays the photographed image and the expression measured by the measurement unit Value; an instruction unit that specifies an expression value measured by the measurement unit and instructs the user to photograph the user's face with an imaging device; an extraction unit that analyzes the facial image of the user photographed by the imaging device and extracts the facial image The position of the plurality of characteristic points in the calculation unit; using the position of the plurality of characteristic points extracted by the extraction unit to calculate the value indicating the pressure index; and the output unit outputting the value indicating the pressure index calculated by the calculation unit. The calculation unit calculates a value representing a degree of distortion of the face based on the plurality of feature points obtained from the facial image when the user makes an expression with a specified expression value, and uses the value as a value indicating a pressure index.
＜ Effect of the invention ＞

The program of the present invention does not require a special device for measurement, and can evaluate the subject's pressure in a completely non-invasive method without taking a body fluid sample such as blood.

[First Embodiment]
The pressure evaluation method of the present invention can be roughly divided into the following steps (1) to (4).
(1) the process of giving instructions to the subject;
(2) the process of photographing the subject's face with an imaging device;
(3) a process of processing the captured image to obtain a parameter reflecting the pressure; and
(4) A process of calculating a pressure index value using parameters obtained through image processing.

According to the first aspect of the method of the present invention, in the step (1), the subject is instructed to face the face to the imaging device and make a maximum smile. According to this instruction, the subject is placed in front of the imaging device, and his face is directly pointed at the imaging device to make the expression that the subject thinks is the largest smile (hereinafter, referred to as the "largest smile").

The instructions in step (1) are not limited to pointing the face directly to the camera and making the biggest smile, but may also include instructions to prompt the input to indicate that the camera is ready for preparation, and remind to avoid moving the face until the shooting in step (2) is completed. Instructions, etc.

Then, in step (2), the subject's face is captured by the imaging device. A memory device is preferably connected to the imaging device to memorize the captured facial image in the memory device.

Next, in step (3), the captured facial image is processed. This image processing method will be described below with reference to FIG. 1.

(3-1) First, in the facial image, the positions of the outer corner (a1) and the mouth corner (b1) of the one side, the outer corner (a2), and the mouth corner (b2) of the other side as the feature points are determined.

Here, in FIG. 1, for convenience of explanation, the angle of the eyes other than the right side of the subject is (a1), the corner of the mouth is (b1), the angle of the eyes other than the left is (a2), and the angle of the mouth is (b2). However, this setting is not fixed and can be reversed.

(3-2) Next, draw a straight line (a1-b1) passing through the outer corner (a1) and mouth corner (b1) and a straight line passing through the outer corner (a2) and mouth corner (b2) ( a2-b2).

(3-3) Furthermore, the angle (θ1) formed by the straight line (a1-b1) drawn in (3-2) and the reference line, and the angle formed by the straight line (a2-b2) and the reference line ( θ2), and the values of "θ1" and "θ2" are used as parameters.

The reference line in this first aspect is a horizontal line in the image (straight line A-A in FIG. 1).

In addition, "an angle formed by a straight line (a1-b1) and a reference line" and "an angle formed by a straight line (a2-b2) and a reference line" in this specification mean the straight line (a1-b1) or the straight line (a2 -b2) Among the two angles (complementary to each other) formed when crossing the reference line, "an angle of 90 ° or less".

Finally, in step (4), a pressure index value is calculated based on the parameters θ1 and θ2 obtained in step (3). The pressure index value in the first aspect is an angle difference (θ1-θ2) between θ1 and θ2.

The angle difference (θ1-θ2) calculated as the pressure index value according to the method of the first aspect increases when under pressure, and its increase and decrease follow the change in the concentration of cortisol in saliva as a stress hormone, that is, confirm The pressure index value (θ1-θ2) reflects the stress state of the subject.

The pressure index value obtained in this first form is a value calculated by using the horizontal line (A) of the image as a reference line and based on the angle with respect to the reference line. Therefore, this value reflects the maximum smile The value of the left-right twist and the inclination of the entire face. The reason for choosing this value as a pressure indicator is that according to observation results, when the subject is under pressure, the facial expression made by the subject who thinks that he or she has the biggest smile will be distorted left and right, and sometimes the subject has difficulty Carry out front-facing camera operation.

A second aspect of the method of the present invention includes the same steps (1) to (4) as the first aspect.

However, in step (3-1) of step (3), in addition to determining the positions of the eye corner (a1) and mouth corner (b1) other than one side, and the eye corner (a2) and mouth corner (b2) other than the side, It also includes processing to determine the positions of the right eye center point (c1) and the left eye center point (c2).

In step (3-2) of step (3), in addition to drawing a straight line (a1-b1) passing through the corner of the eye (a1) and the corner of the mouth (b1) and passing through the corner of the eye (a2) other than the other side In addition to the straight line (a2-b2) of the corner of the mouth (b2), a straight line drawn through the right eye center point (c1) and the left eye center point (c2) (hereinafter, referred to as the "eye center line", (in the figure) Processing of straight lines A'-A ')).

In this second aspect, the "eye center line" is used as a reference line. Therefore, in step (3-3) of step (3), measure the angle formed by the straight line (a1-b1) and the reference line (straight line A'-A 'in FIG. 1) (θ1' in FIG. 1), and The angle (θ2 'in FIG. 1) formed by the straight line (a2-b2) and the reference line (the straight line A'-A' in FIG. 1), and the values of "θ1 '" and "θ2'" are used as parameters.

In step (4) of the second aspect, the pressure index value is calculated based on the parameters θ1 'and θ2' obtained in the above (3-3). The pressure index value in the second aspect is the angle difference (θ1'-θ2 ') between θ1' and θ2 '.

The angle difference (θ1'-θ2 ') calculated as a pressure index value according to the method of this second aspect increases when under pressure, and its increase and decrease follow the change in the concentration of cortisol in saliva as a stress hormone. That is, it was confirmed that the pressure index value (θ1'-θ2 ') reflects the stress state of the subject.

The pressure index value obtained in this form is a value calculated based on the angle with respect to the center line of the eye (the straight line A'-A 'in FIG. 1). Therefore, it is possible to compensate for the influence caused by the overall tilt of the face, and only Reflects the value of left and right distortion of the face.

The third aspect of the method of the present invention includes steps (1) to (4) similarly to the first and second aspects described above. However, the parameters obtained when obtaining the "maximum smile" and the parameters obtained when "serious expression" are used to calculate the pressure index value. details as follows.

Steps (1) and (2) are the same as the first form and the second form, and include (1A): instruct the subject to point his face to the camera and make the biggest smile, and (2A): take the biggest smile . Also included are (1B): Instruct the subject to make a serious expression, and (2B): Take a serious expression. Here, the "serious expression" refers to a state in which the subject is completely relaxed and the facial muscles are not tense (a state in which the face is expressionless).

In step (3), image processing is performed on the maximum smile image captured in the above (2A) and the serious expression image captured in the above (2B). The image processing of the largest smile is described with reference to FIG. 1, and the image processing of the serious expression is described with reference to FIG. 2.

(3A): In the image with the largest smile (Fig. 1), draw a straight line (a1-b1) passing through the corner of the eye (a1) and the corner of the mouth (b1) and the corner of the eye (a2) ) And the straight line (a2-b2) of the mouth corner (b2), and measure the angle (θ1) formed by the straight line (a1-b1) and the reference line (AA in Figure 1), and the straight line (a2-b2) and the reference line (Figure The angle (θ2) formed by AA) of 1 is used as a parameter.

(3B): In the "serious expression" image (Figure 2), draw a straight line (a1 "-b1") that passes through the corner of the eye (a1 ") and the corner of the mouth (b1") outside one side, and through the other Straight line (a2 "-b2") of the eye corner (a2 ") and mouth corner (b2") outside the side, and measure the angle (θ1 ") formed by the straight line (a1" -b1 ") and the reference line (AA in Figure 2) , And the angle (θ2 ") formed by the straight line (a2" -b2 ") and the reference line (AA in Fig. 2) as parameters.

In addition, in FIG. 2, for convenience of explanation, the outer corner of the right side of the subject's face is (a1 ”), the corner of the mouth is (b1”), the outer corner of the left side is (a2 ”), and the corner of the mouth is (b2”). Left and right can be reversed.

In step (4), the maximum smile parameter (θ1, θ2) obtained in (3A) and the serious expression parameters (θ1 "and θ2") obtained in (3B) are used to calculate the pressure index value. That is, the angle difference (θ1-θ2) between θ1 and θ2 in the largest smile image and the angle difference (θ1 "-θ2") between θ1 "and θ2" in the serious expression image are calculated, and then the angle difference is calculated The difference between (θ1-θ2) and the angle difference (θ1 "-θ2") ((θ1-θ2)-(θ1 "-θ2")) is used as the pressure index value in the third aspect.

Here, the pressure index value ((θ1-θ2)-(θ1 "-θ2")) in the method of this aspect is the difference between the index value at the "serious expression" and the index value at the "maximum smile", so To compensate for numerical changes caused by face tilt. Therefore, as the "reference line" in this third aspect, the horizontal line (A-A) of the image may be used in the same manner as in the first aspect, or the eye center line (A'-A ') may be used in the same manner as in the second aspect.

((Θ1-θ2)-(θ1 "-θ2")), which is a pressure index value calculated according to the method of this third aspect, increases when under pressure, and its increase and decrease follow cortisol, which is a stress hormone, in saliva. The concentration in the sample fluctuates, that is, it is confirmed that the pressure index value ((θ1-θ2)-(θ1 "-θ2")) reflects the stress state of the subject.

In the method of the third aspect, the difference between the index value at the time of "maximum smile" and the index value at the time of "serious expression" is used as the pressure index. Therefore, it is possible to compensate for the influence caused by the overall tilt of the face in the same manner as in the second mode. , And only reflects the stress-induced stiffness of the expression muscles.

According to the method of the present invention, in the first, second, and third aspects, steps (1) and (2) can be repeated multiple times to obtain a plurality of images, and each image can be obtained from step (3). The average value of the parameters obtained is used in the calculation in step (4), thereby reducing measurement errors.

The number of repetitions is not particularly limited. For example, it may be performed 2 to 10 times, preferably 3 to 9 times, more preferably 4 to 8 times, even more preferably 5 to 7 times.

The invention also provides a system and a computer program for implementing the above-mentioned pressure evaluation method, and a memory medium storing the computer program.

The system of the present invention includes an instruction unit for a subject, an imaging unit that captures the subject's face, an image processing unit that processes the captured facial image, and a calculation unit that calculates parameters obtained through image processing .

The instruction may be a file (instruction manual, etc.) attached to the imaging device, and the instruction unit for the subject may be an image display device connected to the imaging device and displaying the instruction, or a voice output device that issues instructions by voice.

The imaging unit is preferably an imaging device such as a digital camera. In addition, it is preferable to provide a display capable of displaying an image acquired from the camera in real time so that the subject can look at the display and adjust the position of his face to the center of the camera. Here, in order to prevent the subject from intentionally changing the inclination or expression of his face after seeing the display, it is preferable to cover the part of the display including the subject's eyes, nose, and mouth. For masking, a method such as pasting a masking member on a display screen may be adopted, and a function of displaying a masked image on the display may be added.

The system of the present invention is preferably provided with a memory unit to memorize captured facial image data.

The image processing unit includes a unit that can identify positions of feature points (left and right outer eye corners, left and right mouth corners, and eye center points are arbitrary) in the acquired facial image, and measure the predetermined angle based on these positions.

The calculation unit includes a unit that calculates a pressure index value by using an angle obtained in image processing as a parameter and performing a predetermined calculation.

The system of the present invention is preferably provided with a display device to display the pressure index value obtained by the calculation unit.

In addition, a unit that can memorize the pressure index value of each subject in the memory device, and compare the newly obtained pressure index value with the past pressure index value, and remind the subject to pay attention to the change.

The system of the present invention may be a personal computer (PC), a tablet or a smart phone provided with a display unit, a camera unit, a memory unit, and a calculation unit, and the stress evaluation of the present invention may be implemented by downloading from the PC, tablet or smart phone. The method program (application program) can constitute the system of the present invention.

The computer program of the present invention is characterized by including:
(1) the step of giving instructions to the subject;
(2) a step of processing the subject's face image captured by the imaging device and outputting parameters reflecting the subject's pressure, and
(3) the step of calculating the pressure index value using the parameters output after image processing,
Moreover, in this step (2), the positions of the left and right outer eye corners, the left and right mouth corners in the facial image, and the positions of the left and right eye center points are arbitrarily determined, and the eye corners (a1) and the mouth corners (a1) outside the side b1) straight line (a1-b1) and the straight line (a2-b2) passing through the corner of the eye (a2) and mouth corner (b2) on the other side, and measure the angle (θ1) between the straight line (a1-b1) and the reference line ), And the angle (θ2) formed by the straight line (a2-b2) and the reference line, and outputting the values of "θ1" and "θ2" as parameters;
In this step (3), the parameters (θ1 and θ2) are used to calculate the pressure index values.

[Example]
The present invention will be described in more detail with specific examples below, but the present invention is not limited to the following examples.

There are various proposals for additional methods of stress for verifying the causes of psychological and physical changes. Especially for space pilot candidates, a method of spending a certain period of time in a closed environment that is cut off from the outside is used. . For details, refer to, for example, Inoue N, et al., Aviation, Space, and Environmental Medicine, Vol. 84, No. 8, pp. 867-871 (2013), Shimiya Minan, etc., "Psychology and Physiology in a Closed Environment" Stress ", Space Utiliz. Res., 24 (2008), etc.

When testing the method of the present invention, a stress test for collective living in an environment isolated from the outside was also performed. Specifically, subjects (23 persons) were allowed to live collectively in an environment isolated from the outside, while controlling diet, sleep time, and issues. One day before entering the environment isolated from the outside (L-1), two weeks in the environment isolated from the outside (C1 ~ C14), one day after leaving the environment isolated from the outside (R + 1) and 7 days After that (R + 7), the method of the present invention was used to measure the pressure index value according to the facial image at a certain time every day, and the cortisol concentration in the saliva at this time was also measured.

(Example 1)
FIG. 3 is a graph depicting pressure index values (θ1-θ2) measured in accordance with the first aspect of the method of the present invention (see FIG. 1) and changes in cortisol concentration in saliva measured simultaneously.

In this form of image processing, as shown in FIG. 1, a straight line (a1-b1) passing through the corner of the eye (a1) and the corner of the mouth (b1) is drawn, and the corner of the eye (a2) is passed through the other side The straight line (a2-b2) from the corner of the mouth (b2), and based on the angle (θ1) formed by the straight line (a1-b1) and the horizontal line (AA) of the image, and the straight line (a2-b2) and the horizontal line of the image (AA ), And calculate the pressure index value (θ1-θ2).

It can be seen from the data in Figure 3 that the pressure index value (L-1) increased one day before entering the environment isolated from the outside, and the pressure eased (C2-C7) in the first week after entering the room, and the pressure increased after one week (C8 -14), the pressure is relieved (R + 1) after leaving the environment isolated from the outside, and the pressure is further reduced (R + 7) after 7 days. These trends are consistent with changes in cortisol concentrations.

(Example 2)
FIG. 4 is a graph depicting pressure index values (θ1′-θ2 ′) measured in accordance with a second aspect of the method of the present invention (see FIG. 1) and cortisol concentrations in saliva measured simultaneously.

In this form of image processing, as shown in FIG. 1, a straight line (a1-b1) passing through the corner of the eye (a1) and the corner of the mouth (b1) is drawn, and the corner of the eye (a2) is passed through the other side A straight line (a2-b2) with the corner of the mouth (b2), and draw a straight line ("eye center line": A'-A 'of Fig. 1) passing through the eye center points (c1 and c2), according to the line (a1-b1 ) The angle (θ1 ') formed by the eye center line (A'-A') and the angle (θ2 ') formed by the straight line (a2-b2) and the eye center line (A'-A'), calculate the pressure index value (θ1'-θ2 ').

It can be seen from the data in Figure 4 that the value of the pressure index (L-1) is lower in the first day before entering the environment isolated from the outside, and will increase after entering (C2-7), and further increase in the second half of the week (C8-14 ). However, the pressure eased after leaving the environment isolated from the outside (R + 1 and R + 7). These trends are in good agreement with those of cortisol concentrations.

(Example 3)
Fig. 5 depicts the pressure index values ((θ1-θ2)-(θ1 "-θ2")) measured according to the third aspect of the method of the present invention (see Figs. 1 and 2), and the salivary cortex measured simultaneously Graph of alcohol concentration.

In the image processing in this form, the image with the largest smile and the image with the serious expression are taken seven times each, and the acquired images are subjected to the same image processing as in Example 1. Use the average value of the angles obtained from the 7 images with the largest smiles as parameters (θ1 and θ2), and use the average value of the angles obtained from the 7 images with serious expressions as the parameters (θ1 "and θ2"), Based on these parameters, the pressure index value ((θ1-θ2)-(θ1 "-θ2")) is calculated.

It can be seen from the data in Figure 5 that the pressure index value of the first day (L-1) before entering the environment isolated from the outside is negative, and after entering, it is positive, which is in phase with the first half of the period (C2-7) Ratio, the index value in the second half of the week (C8-14) increases. After leaving the environment isolated from the outside, the pressure eased (R + 1), and after 7 days the pressure decreased further (R + 7). These trends are in good agreement with those of cortisol concentrations.
[Second Embodiment]
In the second embodiment, the stress evaluation method of the first embodiment is used to diagnose the stress of the subject by using a pressure diagnosis application as an example of the program. Here, regarding the same contents as those in the first embodiment, the description thereof is appropriately omitted.

＜ System configuration ＞
6A and 6B are configuration diagrams of an example of an information processing system according to this embodiment. The information processing system of this embodiment may be constituted by a single information processing device 1 as shown in FIG. 6A, or may be a client terminal 2 or a servo of one or more subjects connected to the network 5 as shown in FIG. 6B. The device 3 and the client terminal 4 of the inspection operator are configured.

The information processing device 1 of FIG. 6A is implemented by, for example, a computer that can execute the pressure diagnosis application program according to this embodiment. The information processing device 1 is a computer such as a PC, a smartphone, or a tablet.

The client terminals 2 and 4 of the information processing system of FIG. 6B are implemented by a computer such as a PC, a smartphone, or a tablet computer that can execute the pressure diagnosis application program of this embodiment. The server device 3 is used to manage or control the stress diagnostic application and the like operated by the subject and the test implementer through the client terminals 2 and 4.

As described above, the information processing system of this embodiment can be implemented by the single information processing device 1 shown in FIG. 6A, or by the client-servo type system shown in FIG. 6B. In addition, the information processing system of FIG. 6A and FIG. 6B is only an example, and there is no doubt that there are various system configuration examples depending on the use and purpose. For example, the server device 3 of FIG. 6B may be composed of a plurality of computers.

＜ Hardware Configuration ＞
＜＜ Information processing device, client terminal of subject ＞＞
The information processing device 1 and the client terminal 2 of the subject can be implemented by a computer with a hardware structure as shown in FIGS. 7A and 7B, for example. 7A and 7B are diagrams showing a hardware configuration of an example of a computer.

The computer of FIG. 7A includes an input device 501, a display device 502, an external I / F 503, a RAM 504, a ROM 505, a CPU 506, a communication I / F 507, an HDD 508, a photographing device 509, and the like, and each part is connected to each other through a bus B. Here, the input device 501 and the display device 502 may be in a built-in form, and may be used in a connected form if necessary.

The input device 501 is a touch panel, operation keys or buttons, a keyboard or a mouse, and the like for helping a subject to input various signals. The display device 502 is a display screen such as a liquid crystal or an organic EL for displaying a screen. The communication I / F 507 is an interface for connecting to a network 5 such as a local area network (LAN) or the Internet. The computer of FIG. 7A can communicate via the network 5 using the communication I / F507.

HDD508 is an example of a non-volatile memory device for storing programs and the like. The programs stored in the HDD508 include basic software OS, stress diagnostic applications, and so on. Here, the HDD 508 may be replaced with a drive device (for example, a solid state drive: SSD) or a memory device such as a memory card using a flash memory as a storage medium. The external I / F 503 is an interface for an external device such as the recording medium 503a. The computer of FIG. 7A can read and / or write to the recording medium 503a using the external I / F 503.

The recording medium 503a is a flexible magnetic disk, a CD, a DVD, an SD memory card, a USB memory, or the like. The ROM 505 is an example of a non-volatile semiconductor memory (memory device) that can hold programs and the like even when the power is turned off. The ROM 505 stores various programs such as a BIOS program executed at the time of startup, OS settings, and network settings. The RAM 504 is an example of a volatile semiconductor memory (memory device) for temporarily holding a program or the like. The CPU 506 is a calculation device that reads a program in a memory device such as the ROM 505 or the HDD 508 onto the RAM 504 and executes processing. The imaging device 509 performs photography using a camera.

The information processing device 1 and the client terminal 2 of this embodiment can implement various processes described later by executing a pressure diagnosis application program in the hardware configuration described above. Here, the computer of FIG. 7A has a configuration of an imaging device 509 (built-in type), and, for example, as shown in FIG. 7B, a configuration in which the imaging device 509 is used by connection via an external I / F 503 may be used. The computer of FIG. 7B is different from the computer of FIG. 7A in that the photographing device 509 is a peripheral device.

＜＜ Inspection client terminal and server device ＞＞
The server device 3 and the client terminal 4 of the inspector may be implemented by a computer with a hardware structure as shown in FIG. 8, for example. FIG. 8 is a hardware configuration diagram showing an example of a computer. Here, descriptions of the same parts as those in FIGS. 7A and 7B are omitted.

The computer in FIG. 8 includes an input device 601, a display device 602, an external I / F 603, a RAM 604, a ROM 605, a CPU 606, a communication I / F 607, an HDD 608, and the like, and each part is connected to each other through a bus B. The computer configuration of FIG. 8 lacks an imaging device 509 compared to the computer of FIGS. 7A and 7B. The computer in FIG. 8 communicates via the network 5 using the communication I / F607.

＜ Software composition ＞
The software configuration of the information processing device 1 shown in FIG. 6A will be described below. FIG. 9 is a functional block diagram of an example of an information processing apparatus according to this embodiment. The information processing apparatus 1 includes an operation accepting section 10, an image input section 12, a smile value measuring section 14, a feature point extracting section 16, an instructing section 18, a content generating section 20, a pressure index calculating section 22, a content display section 24, The structure of the pressure index storage unit 26.

The operation receiving unit 10 receives an operation from the subject. For example, the operation accepting unit 10 receives a pressure diagnosis of a subject and starts an operation. After receiving the operation from the subject, for example, the start of the pressure diagnosis, the operation receiving unit 10 notifies the smile value measurement unit 14 and the instruction unit 18 of the start of the pressure diagnosis, and starts the pressure diagnosis process.

The image input unit 12 acquires an image (input image) captured by the imaging device 509. The image input unit 12 supplies the input image to the smile value measurement unit 14, the feature point extraction unit 16, and the content generation unit 20. The content generation unit 20 generates a content image described below based on the input image provided by the image input unit 12. The content display unit 24 causes the display device 502 to display a content image generated by the content generation unit 20.

The smile value measurement unit 14 measures a smile value of a face image included in an input image provided by the image input unit 12. Here, a technique for measuring a smile value based on a facial image belongs to a known technique, and description thereof is omitted. The smile value measured by the smile value measurement unit 14 (hereinafter, referred to as "measured smile value") is supplied to the content generation unit 20 and is displayed on the content image as described below. In addition, the smile value measurement unit 14 supplies the measured smile value to the feature point extraction unit 16 and the instruction unit 18.

The instruction unit 18 supplies the target value of the smile value (hereinafter, referred to as the “target smile value”) to the content generation unit 20 after the start of the stress diagnosis process. The target smile value provided to the content generation unit 20 is displayed on the content image as described below. Therefore, the subject can adjust the smile while looking at the content image displayed on the display device 502 so that the measured smile value becomes the target smile value.

The feature point extraction section 16 obtains a measured smile value provided by the smile value measurement section 14 and a target smile value provided by the instruction section 18. The feature point extraction unit 16 extracts left and right outer corners, mouth corners, and eye center points based on the subject's facial image included in the input image when the measured smile value reaches the target smile value as the feature point of the first embodiment. And other locations.

For example, in the case of the first aspect using the pressure evaluation method according to the first embodiment, the eye corners (a1) and mouth corners (b1) outside one side and the eye corners (a2) and mouth corners (b2) outside the other side are determined. position. In the case of the second aspect using the pressure evaluation method of the first embodiment, the positions of the corners of the eyes (a1) and mouth corners (b1) and the center of the eye (c1) outside one side and the corners of the eyes outside the other side ( a2) with the corners of the mouth (b2) and the center of the eye (c2).

The pressure index calculation unit 22 calculates a value (pressure index value) as a pressure index based on the position of the feature points extracted by the feature point extraction unit 16 according to the pressure evaluation method of the first embodiment. For example, in the case of using the first aspect of the pressure evaluation method of the first embodiment, the pressure index calculation unit 22 draws a straight line (a1-b1) passing through the corner of the eye (a1) and the corner of the mouth (b1) other than one side, and Pass the straight line (a2-b2) between the corner of the eye (a2) and the corner of the mouth (b2) on the other side.

Then, the pressure index calculation unit 22 measures an angle (θ1) formed by the drawn straight line (a1-b1) and the reference line, and an angle (θ2) formed by the straight line (a2-b2) and the reference line. The pressure index calculation unit 22 uses the measured values of the angles “θ1” and “θ2” as parameters. The pressure index calculation unit 22 calculates the angle difference (θ1-θ2) between the obtained parameters θ1 and θ2, and uses this as the pressure index value.

In the case of using the second aspect of the pressure evaluation method of the first embodiment, the pressure index calculation unit 22 draws a straight line (a1-b1) passing through the corner of the eye (a1) and the corner of the mouth (b1) other than one side, and the other A straight line (a2-b2) of the corner of the eye (a2) and the corner of the mouth (b2) other than one side, and the eye center line passing through the eye center point (c1) and the eye center point (c2).

Then, the pressure index calculation unit 22 measures the angle (θ1 ') formed by the drawn straight line (a1-b1) and the eye center line as a reference line, and the straight line (a2-b2) and the eye center line as a reference line Angle (θ2 '). The pressure index calculation unit 22 uses the measured values of the angles "θ1 '" and "θ2'" as parameters. The pressure index calculation unit 22 calculates the angle difference (θ1'-θ2 ') between the obtained parameters θ1' and θ2 ', and uses this as the pressure index value.

In the case of using the third aspect of the stress evaluation method according to the first embodiment, the above-mentioned first aspect or second aspect is used to calculate parameters when the subject makes the biggest smile and when the subject makes a serious expression. Parameters, and calculate the difference of these parameters as the pressure index value. The pressure index storage unit 26 uses the pressure index value calculated by the pressure index calculation unit 22 as the resume data, and memorizes it in association with date data and the like.

Here, the information processing device 1 in FIG. 9 illustrates that the feature point extraction unit 16 extracts feature points of the first embodiment from the subject face image included in the input image when the measured smile value reaches the target smile value. The situation of the location, however, does not necessarily have to be processed immediately. For example, the feature point extraction unit 16 may first save the subject's face image when the measured smile value reaches the target smile value as a track image, and then process it by the pressure index calculation unit 22.

＜ Processing ＞
<< Overall Processing >>
The information processing apparatus 1 according to this embodiment performs processing in the order shown in FIG. 10, for example. FIG. 10 is a flowchart showing an example of the overall processing of the information processing apparatus according to the present embodiment. After the subject executes the pressure diagnosis application, the information processing device 1 accepts the operation of starting the pressure diagnosis, proceeds to step S10, and aligns the face image of the subject displayed in the content image, that is, performs the following Facial registration.

Proceeding to step S12, the information processing device 1 obtains the subject's face image when the measured smile value reaches the target smile value, that is, it performs the following data acquisition processing. Proceeding to step S14, the information processing device 1 calculates a pressure index value based on the facial image of the subject's target smile value obtained in step S12, and performs data analysis processing. Proceeding to step S16, the information processing device 1 outputs the traverse data of the pressure index value calculated in step S14 as a report, that is, performs report output processing.

＜ S10: Face registration processing ＞＞
FIG. 11 is a flowchart of an example of a face alignment process. In step S20, the image (input image) captured by the imaging device 509 is input to the image input unit 12 of the information processing device 1 in frame units. The image input unit 12 supplies the input image to the content generation unit 20 and causes the stress diagnosis application screen 1000 to display a content image as shown in FIG. 12. FIG. 12 is an image diagram showing an example of a pressure diagnosis application program screen during face alignment processing. A frame 1002 as a target position of a face image included in the input image is displayed on the pressure diagnosis application screen 1000.

Proceeding to step S22, the smile value measuring section 14 performs face image analysis on the input image provided by the image input section 12, and detects a face image included in the input image. The processes of steps S20 to S24 are repeated until a facial image can be detected. After the face image is detected, the smile value measurement unit 14 proceeds to step S26, and performs area (overall position) detection on the detected face image.

Proceeding to step S28, the smile value measuring unit 14 determines whether the entire position of the detected facial image is inside the frame 1002. The processing of steps S20 to S28 is repeated until it can be determined that the entire position of the detected facial image is inside the frame 1002. If it is determined that the entire position of the detected facial image is inside the frame 1002, the content generation unit 20 proceeds to step S30 and displays an "OK indicator" to convey the position of the facial image to the subject is included in the frame 1002. Inside (the position of the facial image is correct).

Proceeding to step S32, the operation accepting unit 10 of the information processing apparatus 1 displays the "OK indicator", and after receiving the instruction to start the acquisition from the subject, ends the face alignment processing of FIG. The data acquisition processing shown in step S12 in FIG. 10.

＜ S12: Data Acquisition Processing ＞＞
FIG. 13 is a flowchart of an example of data acquisition processing. In step S40, the image (input image) captured by the photographing device 509 is input to the image input unit 12 of the information processing device 1 on a frame-by-frame basis. The image input unit 12 supplies the input image to the content generation unit 20 and causes the stress diagnostic application screen 1000 to display a content image display as shown in FIG. 14. FIG. 14 is an image diagram showing an example of a pressure diagnosis application screen during data acquisition processing. The target smile value 1004 and the measured smile value 1006 are displayed on the pressure diagnosis application screen 1000. The "smile value" is shown as "smile score" in FIG.

The pressure diagnosis application screen 1000 on the left is an example of a target smile value 1004 that instructs the subject to make the largest smile. The pressure diagnosis application screen 1000 on the right is an example of a target smile value 1004 that instructs the subject to make the smallest smile (serious expression). The measured smile value 1006 is an instant smile value measured by the smile value measurement section 14.

Proceeding to step S42, the smile value measuring section 14 performs face image analysis on the input image provided by the image input section 12, and detects a face image included in the input image. The processes of steps S40 to S44 are repeated until a face image can be detected. After the face image is detected, the smile value measurement unit 14 proceeds to step S46 and measures the smile value of the detected face image.

The processing of steps S40 to S48 is repeated until the measured smile value 1006 reaches the target smile value 1004. After the measured smile value 1006 reaches the target smile value 1004, the feature point extraction unit 16 proceeds to step S50, and saves the subject's face image and smile value when the measured smile value 1006 reaches the target smile value 1004 as a track image. Repeat the steps while switching the target smile value 1004 displayed on the stress diagnostic application screen 1000 to the target smile value 1004 that instructs the subject to make the largest smile, or the target smile value 1004 that instructs the subject to make the smallest smile. Processing from S40 to S52.

Accordingly, in step S50, the subject's face image when the measured smile value 1006 becomes the maximum smile or the minimum smile can be stored as a track image. After the input of the input image is completed, the data acquisition process of FIG. 13 is ended, and the data analysis process shown in step S14 of FIG. 10 is started.

Here, as described in the first embodiment, in order to prevent the subject from consciously changing his or her expression after seeing the display, the pressure diagnosis application screen 1000 of FIG. 14 may be set as the pressure diagnosis application of FIG. Picture 1000. 15 is an image diagram showing an example of a pressure diagnosis application screen provided with a mask on the subject's face. The pressure diagnosis application screen 1000 of FIG. 15 is an example in which a mask 1008 is added to cover a part of the facial image including the subject's eyes, nose, and mouth displayed on the display device 502.

＜ S14: Data analysis processing ＞＞
FIG. 16 is a flowchart of an example of data analysis processing. In step S60, the feature point extracting unit 16 of the information processing device 1 reads the track image stored in the data acquisition process in step S12. Proceeding to step S62, the feature point extraction unit 16 performs face image analysis on the crawler image read in step S60, and detects a facial image included in the crawler image. The processes of steps S60 to S64 are repeated until a face image can be detected.

After the face image is detected, the feature point extraction unit 16 proceeds to step S66, and extracts the positions of the feature points (landmark data) such as the left and right outer eye corners, mouth corners, and eye center points from the detected face image. Step S66 is a process of extracting a feature point position from the subject's face image at the maximum smile or the minimum smile. By applying AI (machine learning) to the feature point position extraction process from the facial image in step S66, extraction can be performed with higher accuracy.

Proceeding to step S68, the pressure index calculation unit 22 calculates a value (pressure index value) as a pressure index based on the position of the feature points extracted by the feature point extraction unit 16 according to the pressure evaluation method of the first embodiment. The process proceeds to step S70, and the pressure index calculation unit 22 uses the calculated pressure index value as the track data and stores it in the pressure index storage unit 26. After the shoe image has been read, the data analysis process in FIG. 16 ends.

The image of the data analysis processing in FIG. 16 may be displayed on the pressure diagnosis application screen 1000 shown in FIGS. 17A and 17B, for example. FIG. 17A is an image diagram of an example of a pressure diagnosis application screen displaying an image of data analysis processing at the time of the largest smile. FIG. 17B is an image diagram of an example of a pressure diagnosis application screen displaying an image of data analysis processing at the minimum smile.

In the pressure diagnosis application screen 1000 of FIGS. 17A and 17B, the positions of the eye corners, mouth corners, and eye center points other than one side and the positions of the eye corners, mouth corners, and eye center points other than the side described in the first embodiment are described. Is displayed as a feature point. In addition, in the pressure diagnosis application screen 1000 of FIGS. 17A and 17B, a straight line passing through an eye corner and a mouth corner other than one side, a straight line passing through an eye corner and a mouth corner other than the other side, a horizontal line in the image, and passing through both eyes are displayed. Eye center line at the center point.

In addition, the pressure diagnosis application screen 1000 of FIGS. 17A and 17B is provided with a switch button 1010 for a resume image at the maximum smile and a resume image at the minimum smile, and a resume map for the maximum or minimum smile In the case where there are a plurality of images, there are switching buttons 1012 for the crawler image.

＜＜ S16 ： Report output processing ＞＞
FIG. 18 is a flowchart of an example of report output processing. In step S80, the content generation unit 20 of the information processing apparatus 1 reads the pedigree data of the pressure index value stored in step S14 from the pressure index storage unit 26.

The process proceeds to step S82, and the content generation unit 20 compares the current (today) pressure index value with the past pressure index value. Proceeding to step S84, the content generating unit 20 generates a report chart based on the comparison result in step S82. The processes of steps S80 to S86 are repeated until all the track data of the subject is read.

After reading all of the subject's performance data, the content display unit 24 proceeds to step S88 and displays an overall report 1020 including a report chart generated by the content generation unit 20 as shown in FIG. 19. FIG. 19 is an image diagram showing an example of a pressure diagnosis application screen at the time of an overall report. The overall report 1020 in FIG. 19 shows the transition of the pressure index value in time series. In addition, the overall report 1020 in FIG. 19 shows the value of the stress index as the degree of distortion of the smile. As shown in the overall report 1020 of FIG. 19, by graphing the transition of the pressure index value, the subject can easily understand the change in the pressure.

In addition, the pressure diagnosis application according to this embodiment may also be provided with a function of performing pressure diagnosis based on a questionnaire, and for example, a questionnaire screen 1030 shown in FIG. 20 may be used to perform a questionnaire survey for the subject on the pressure diagnosis.

In this case, for example, as shown in FIG. 21, the content display unit 24 may display the stress diagnosis result 1040 of the facial image analysis and the stress diagnosis result 1042 of the questionnaire.

The present invention is not limited to the embodiments specifically disclosed above, and various modifications or changes can be made without departing from the scope of the patent application. For example, although the information processing device 1 has been described as an example in the above embodiment, there is no doubt that the functional blocks shown in FIG. 9 may be distributed to the client terminal 2, the server device 3, and the client shown in FIG. 6B. Configuration of the terminal 4.

In addition, in the information processing system shown in FIG. 6B, the client terminal 2 of the subject and the client terminal 4 of the inspection implementer may display the same overall report 1020, and may also display subjects or inspection implementers with different contents. Used overall report 1020. In addition, the term "stress indicator" is used in this embodiment, and it is also possible to use a term that is synonymous or antonymous to stress.

It should be noted that the smile described as an example of the expression in this embodiment can also be applied to expressions other than a smile. A software or service for outputting expression analysis of various emotion levels based on facial expressions is a well-known technology. In addition, in the present embodiment, the “face twist degree” as an example of the “face twist degree” can be applied to the “twist degree” other than the “face twist degree”. For example, it is also possible to capture the "distortion" of the mouth corner or outer eye corner at any time in the up, down, left, and right directions with the virtual center of the face as the center.

As mentioned above, although this invention was demonstrated based on embodiment, this invention is not limited to the said embodiment, Various deformation | transformation can be performed within the description range of the patent application range. This application is based on a basic patent application No. 2017-196505 filed with the Japan Patent Office on October 10, 2017, and refers to the entire contents of the basic patent application No. 2017-196505.
[Industrial availability]

According to the stress evaluation method of the present invention, a PC, a tablet computer, or a smartphone can be used to easily and non-invasively measure the pressure index value. Therefore, it can be used for both self-stress management and the stress state of the company to employees Proper management.

1‧‧‧ Information Processing Device

2. 4‧‧‧ client terminal

3‧‧‧ server device

5‧‧‧ internet

10‧‧‧Operation Acceptance Department

12‧‧‧Image Input Department

14‧‧‧Smile value measurement department

16‧‧‧ Feature point extraction section

18‧‧‧ Instruction

20‧‧‧Content Generation Department

22‧‧‧Pressure indicator calculation department

24‧‧‧Content Display Department

26‧‧‧Pressure indicator memory

501, 601‧‧‧ input devices

502, 602‧‧‧ display device

503, 603‧‧‧ External I / F

503a, 603a‧‧‧Recording media

504, 604‧‧‧RAM

505, 605‧‧‧ROM

506, 606‧‧‧CPU

507, 607‧‧‧communication I / F

508, 608‧‧‧HDD

509‧‧‧Photographic device

1000‧‧‧ Pressure Diagnostic Application Screen

1002‧‧‧box

1004‧‧‧Target smile value

1006‧‧‧Measure smile value

1008‧‧‧Mask

1010, 1012‧‧‧ switch button

1020‧‧‧Overall Report

1030‧‧‧ Questionnaire screen

1040‧‧‧ Stress diagnosis results based on facial image analysis

1042‧‧‧Result of stress diagnosis based on questionnaire

B‧‧‧ Bus

FIG. 1 is a schematic diagram illustrating a method of extracting feature points in a facial image captured at the maximum smile in the method of the present invention, and calculating a value indicating a pressure index based on the feature points.

FIG. 2 is a schematic diagram illustrating a method of extracting feature points in a facial image captured during a serious expression in the method of the present invention, and calculating a value indicating a pressure index based on the feature points.

FIG. 3 is a graph showing a correlation between a pressure index value of a subject obtained in a first aspect of the present invention (Example 1) and a salivary cortisol concentration of the same subject.

FIG. 4 is a graph showing a correlation between a pressure index value of a subject and a salivary cortisol concentration of the same subject obtained in a second aspect (Example 2) of the present invention.

FIG. 5 is a graph showing a correlation between a pressure index value of a subject obtained in a third aspect of the present invention (Example 3) and a salivary cortisol concentration of the same subject.

FIG. 6A is a configuration diagram of an example of an information processing system according to this embodiment.

FIG. 6B is a configuration diagram of an example of an information processing system according to this embodiment.

FIG. 7A is a hardware configuration diagram of an example of a computer.

FIG. 7B is a hardware configuration diagram of an example of a computer.

FIG. 8 is a hardware configuration diagram of an example of a computer.

FIG. 9 is a functional block diagram of an example of an information processing apparatus according to this embodiment.

FIG. 10 is a flowchart showing an example of the overall processing of the information processing apparatus according to the present embodiment.

FIG. 11 is a flowchart of an example of a face alignment process.

FIG. 12 is an image diagram showing an example of a pressure diagnosis application program screen during face alignment processing.

FIG. 13 is a flowchart of an example of data acquisition processing.

FIG. 14 is an image diagram showing an example of a pressure diagnosis application screen during data acquisition processing.

15 is an image diagram showing an example of a pressure diagnosis application screen provided with a mask on the subject's face.

FIG. 16 is a flowchart of an example of data analysis processing.

FIG. 17A is an image diagram showing an example of a pressure diagnosis application screen of a data analysis process image at the time of maximum smile.

FIG. 17B is an image diagram showing an example of a pressure diagnosis application screen of the image analysis processing image at the minimum smile.

FIG. 18 is a flowchart of an example of report output processing.

FIG. 19 is an image diagram showing an example of a pressure diagnosis application screen at the time of an overall report.

FIG. 20 is an image diagram of an example of a questionnaire screen.

21 is an image diagram showing an example of a screen of a pressure diagnosis result based on a facial image analysis and a questionnaire-based pressure diagnosis result.

Claims (6)

A program characterized in that a computer functions as the following units: a measurement unit that measures a user's expression value included in a photographed image; a display unit that displays the photographed image and the expression value measured by the measurement unit; an instruction unit, Specify the expression value measured by the measurement unit and instruct the user to use the camera to capture the user's face; an extraction unit that parses the face image of the user captured by the camera and extracts the feature points of the facial image A position; a calculation unit that uses the positions of the plurality of feature points extracted by the extraction unit to calculate a value representing a pressure index; and an output unit that outputs a value that represents the pressure index calculated by the calculation unit, and the calculation unit, based on The plurality of feature points obtained in the facial image when the user makes an expression of a specified expression value, calculates a value indicating the degree of distortion of the face, and uses this as a value indicating a pressure index. The program according to claim 1 is characterized in that the display unit sets a mask on a part of the user's face included in the photographed image for display. The program according to claim 1 or 2 is characterized in that the instruction unit specifies two expression values measured by the measurement unit and instructs the user to photograph the user's face using a camera device, and the calculation unit is based on the specified The plurality of feature points respectively obtained in the face image when the expressions of the two expression values are expressed are used to calculate the difference between the values representing the degree of distortion of the face, and use this as the value representing the pressure index. According to the formula of claim 1 or 2, characterized in that the calculation unit draws out the outer corner (a1) and mouth corner (b1) of the right side of the facial image extracted by the extraction unit as the plurality of feature points. ) And a straight line (a2-b2) passing through the outer corner of the eye (a2) and the corner of the mouth (b2) on the left, based on the angle (θ1) formed by the straight line (a1-b1) and the reference line An angle (θ2) formed by the straight line (a2-b2) and the reference line is used to calculate a value representing the degree of distortion of the face when the user makes an expression with a specified expression value, and use this as a value indicating a pressure index. The formula according to claim 4, characterized in that the reference line is a horizontal line of the facial image, or a right eye center point (c1) in the facial image extracted by the extraction unit as the plurality of feature points. ) Eye center line with left eye center point (c2). An information processing system composed of a server device and a client device through a network connection, comprising: a measurement unit that measures a user's expression value included in a photographed image; a display unit that displays the photographed image and the An expression value measured by the measurement unit; an instruction unit that specifies the expression value measured by the measurement unit and instructs the user to photograph the user's face using an imaging device; an extraction unit that analyzes the facial image of the user photographed by the imaging device and extracts the The position of the plurality of feature points in the facial image; the calculation unit uses the positions of the plurality of feature points extracted by the extraction unit to calculate a value indicating the pressure index; and the output unit outputs the indicated pressure calculated by the calculation unit The value of the index, and the calculation unit calculates a value representing the degree of distortion of the face based on the plurality of feature points obtained from the facial image when the user makes an expression of a specified expression value, and uses the value as the expression pressure The value of the indicator.