JP2007200126A - Feeling informing reporting device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a feeling information reporting device capable of reporting feeling information on a person to be taken care of without disturbing the work of a carer who performs other work. <P>SOLUTION: A video module program 1200 includes a face detection section 1208 which extracts a face area from a moving picture stored on a hard disk 1110 and an optical flow calculation section 1400 which calculates information for specifying movements of inner parts of the detected and extracted face on the image on the basis of an optical flow. A sound effect addition module program 1400 decides the feeling state of the person to be examined from the moving image captured and stored on the hard disk 110 and adds a sound effect corresponding to a speech signal inputted from an audio reproducing device 200. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a configuration of an emotion information notification device for recording a facial expression image of a subject over a long period of time and for performing a search after the fact.

  Modern life requires us to play a variety of roles in occupation, in the home, and in the local community.

  On the other hand, in many countries, the proportion of the elderly population is increasing, and the proportion of people who are no longer able to do their own life alone is also increasing.

  In Japan, such elderly people often live with their families and require the family to pay attention for a considerable amount of time every day. This means that caregivers who are required to play various other roles as described above will continue to keep an eye on the cared person, and the caregiver's great mental and physical It becomes a burden.

  Eventually, it is desirable to develop a system that reduces the burden on such caregivers and takes over some of the attention directed to the caregivers.

  On the other hand, in recent years, a method for grasping the movement of an object around the imaging device from an image taken with the imaging device has been proposed. In this method, the movement of an object being captured is grasped by measuring an optical flow between frames of a moving image.

  Such an “optical flow” is obtained by using a feature matching method (pattern matching), a spatiotemporal gradient method, or the like for each frame of an image (see, for example, Patent Document 1 and Non-Patent Document 1).

In addition to motion detection in a photographed image, attempts have been reported to associate human facial expressions using an optical flow (see, for example, Non-Patent Document 2).
JP 2004-355082 A Barrron, JL, Fleet, DJ, Beauchemin, SS: Performance of optical flow techniques.The International Journal of Computer Vision, 12 (1994) pp.43-77 Lyons, MJ, Funk, M., Kuwabara, K .: Segment and Browse: A Strategy for Supporting Human Monitoring of Facial Expression Behavior, INTERACT 2005 (LNCS 3585), pp. 1120-1123 (2005).

  However, a method for effectively notifying the caregiver who is performing other work of the psychological state of the cared person when a change in the face image or the like is detected is not always clear.

  The present invention has been made to solve such problems, and its purpose is to notify emotional information of a cared person without interfering with the work of a carer who is doing other work. It is to provide an emotion information notification device capable of doing this.

  According to one aspect of the present invention, there is provided an emotion information notification device, a detection means for obtaining observation information for a subject, an emotion state calculation means for determining the emotion state of the subject according to the observation information, Sound effect processing means for adding a predetermined sound effect to an externally input sound signal and outputting the sound signal according to the determination result.

  Preferably, the detecting means includes a photographing means for acquiring moving image data for a target image area including the face image of the subject, and the emotional state calculating means includes a storage device for storing the moving image data, and a storage device. Face detection means for detecting a face in the stored target image area, motion detection means for detecting a motion vector for each unit area in the detection area corresponding to the detected face, and time of the detected motion vector Judgment processing means for judging whether the subject's emotional state is at least an excited state based on the change.

  Preferably, the motion detection means includes optical flow calculation means for calculating an optical flow for each point on the lattice point in the face image data corresponding to the detection area at a plurality of different speed levels. Based on the corresponding optical flow, as information related to the motion vector, first information corresponding to the magnitude of the movement of the subject's head and second information corresponding to the magnitude of the movement of the facial expression of the subject. An extraction means is further provided, and the determination processing means performs a determination process according to the accumulation of the first and second information.

(Outline of the present invention)
Facial expressions provide important information about human mental state. As described below, the present invention determines the emotional state of the subject (caregiver) from the image information, and notifies the caregiver of the determined emotion information by the acoustic effect superimposed on the audio information. It is. However, the object to be detected is not limited to the facial expression as long as it is a clue to determine the emotional state of the subject (caregiver). For example, voice information and skin resistance of the subject (caregiver) Biometric information such as The subject may be an elderly care recipient as described above, or may be a person who needs care such as a newborn.

  That is, the emotion information notification device of the present invention includes a computer such as a personal computer having an image input interface, software executed on the computer, an imaging device such as a camera, or a microphone for inputting information such as sound. And an acoustic effect processing device for adding an acoustic effect corresponding to the determined emotion information to the input voice signal and outputting the voice signal input from the external playback device. Is. More typically, a face portion is extracted from the image of a person's face, and the movement inside the face is detected, and the emotional state of the subject (care recipient) is determined from the movement information.

  The computer is not limited to the general-purpose computer as described above, but may be a built-in computer incorporated in a dedicated device.

[Hardware configuration]
Hereinafter, an emotion information notification device according to an embodiment of the present invention will be described.

FIG. 1 is a schematic block diagram showing the configuration of the emotion information notification device 100 of the present invention.
Referring to FIG. 1, this emotion information notification device 100 includes a computer 110 such as a personal computer, and a display 108 connected to the computer 110 as a display device for displaying information to a user (caregiver) 2. The audio signal input from the audio reproduction apparatus 200 is connected to the computer 120 and the speaker 120 for reproducing the audio signal after the computer 110 has added the acoustic effect, and an instruction or information from the user 2 is received. A keyboard 106 as an input device for inputting and a camera 102 for capturing an image including the face of the subject 4 are included. The input device may further include a mouse, a tablet, and a pen. In addition, as a sensor for acquiring information that is a clue to determine the emotional state of the subject 4, a microphone or a change in skin resistance of the subject 4 is detected to detect electrical skin reflex (Galvanic Skin Response (GSR)). An electrode or the like that is in contact with the skin surface of the subject 4 may be provided.

  In the system of this embodiment, for example, a video camera including a CCD (solid-state imaging device) is used as the camera 102 to capture image data as digital data.

  Although not shown, it is assumed that the computer 110 is provided with a device for taking in data from an external recording medium such as a CD-ROM (Compact Disc Read-Only Memory) drive.

  Further, referring to FIG. 1, a computer 110 includes a CPU (Central Processing Unit) 1104, a ROM (Read Only Memory) 1106, a RAM (Random Access Memory) 1108, and a hard disk 1110, which are connected to each bus BS. And an interface for transferring data such as receiving an image from the camera 102, receiving an audio signal from the audio playback device 200, outputting an audio signal with an added sound effect, or receiving a signal from the input device 106 Part 1102. When the sensors as described above are provided, the interface unit 1102 receives input signals from these sensors.

  As already described, the main part of the part that functions as the emotion information notification device 100 is realized by computer hardware and software executed by the CPU 1104. Generally, such software is stored and distributed in a storage medium such as a CD-ROM, read from the storage medium by a CD-ROM drive or the like, and temporarily stored in the hard disk 1110. Alternatively, when the device is connected to the network, it is temporarily copied from the server on the network to the hard disk 1110. Then, it is further read from the hard disk 1110 to the RAM 1108 and executed by the CPU 1104. In the case of being connected to a network, for example, the program may be directly loaded into the RAM 1108 without being stored in the hard disk 1110 and executed.

  The computer hardware itself shown in FIG. 1 and its operating principle are general. Therefore, the most essential part of the present invention is software stored in a storage medium such as a CD-ROM or a hard disk 1110.

  As a recent general trend, various program modules are prepared as part of a computer operating system, and an application program generally calls a module in a predetermined arrangement to advance processing when necessary. is there. In such a case, the software itself for realizing the emotion information notification device does not include such a module, and the emotion information notification device is realized only when the computer cooperates with the operating system. However, as long as a general platform is used, it is not necessary to distribute software including such modules. The software itself that does not include these modules and the recording medium that records the software (and the software distributes on the network). Data signal) can be considered to constitute the embodiment.

Software configuration
FIG. 2 is a functional block diagram for explaining the configuration of software operating on the emotion information notification apparatus 100 shown in FIG.

  In the following, it is assumed that the information used as a clue to determine the emotional state of the subject (cared person) is information such as a facial expression detected from the captured image.

  In FIG. 2, the program operating on the computer 110 is an audio signal input from the audio playback device 200 by judging the emotional state of the subject from the video module program 1200 and a moving image captured and stored in the hard disk 1110. And an acoustic effect addition module program 1400 for adding an acoustic effect. Note that the function of the sound effect addition module program 1400 may be realized by hardware.

  In the present invention, by adopting such a configuration, information related to the emotion of the subject is superimposed on the user 2 on the music or the like from the user's favorite audio playback device 200 that the user 2 is listening to while working. Notification can be made. For this reason, when a predetermined change is not detected in the subject's emotion, the user 2 can continue working while listening to desired music or the like.

  As described below, in the emotion information notification apparatus 100, a moving image is processed in real time, a face area is separated, and a facial expression is recorded. An optical flow method is used to interpret the movement of the face.

  The video module program 1200 includes an image capture unit 1202 that acquires a moving image including a target person and surrounding environment from the camera 102, and an image data recording processing unit that stores the captured moving image in the hard disk 1110. 1204. The image capture unit 1202 converts the digital moving image data input from the camera 102 into an image format suitable for storage as necessary.

  The video module program 1200 further identifies a face detection unit 1208 that extracts a face area from a moving image stored in the hard disk 1110, and a motion on the extracted face itself or an image inside the face based on the optical flow. And an optical flow calculation unit 1206 for calculating information (for example, distribution of motion vectors in the image). The optical flow calculation unit 1206 includes moving image data of a face image together with calculated motion attribute data. Store in the hard disk 1110.

  That is, the moving image data of the face image is stored in the hard disk 1110 while being temporally associated with the following two attribute data obtained as described in detail later.

1) Index of rigid head movement (size of movement: index 1)
2) Index of movement inside the face caused by facial expressions and utterances (size of movement: index 2)
On the other hand, the sound effect addition module program 1400 is based on the feature extraction unit 1402 that extracts feature of index change from the index information obtained from the face image stored in the hard disk 1110, and the feature extracted by the feature extraction unit 1402. Then, the emotion model calculation unit 1404 that calculates the change in emotion and the audio signal from the audio playback device 200 are received via the audio buffer 130 in the interface unit 1102, and, as will be described later, And an acoustic effect processing unit 1406 that adds an acoustic effect corresponding to the received audio signal according to the corresponding information and outputs the resultant to the speaker 120.

(Video module program 1200)
Moving image data input from the camera 102 is thinned from, for example, 640 × 480 pixels to 320 × 240 pixels in order to reduce the amount of calculation for optical flow processing and the data capacity for storage in the hard disk 1110. It is burned.

  A face detection unit 1208 determines whether or not a face exists in the field of view of the camera 102 using a well-known face detection algorithm, and also determines where it exists. Although not particularly limited, this face detection algorithm registers a rectangular area having an aspect ratio fixed to the face so as to be positioned at the center of the head image. For example, a face rotating from a plane up to about 30 ° can be accurately detected, and similarly, a face inclined at about 30 ° in the image plane can also be accurately detected. The face image is scaled and the face image corresponding to each frame is stored with a time stamp, for example, at a rate of up to 30 frames / second.

  In the optical flow calculation unit 1206, the optical flow region of the image detects motion between various video frames. The detected motion vector is stored in the hard disk 1110 in association with the moving image data of the face image together with the average of the motion vectors over the entire internal area of the face.

  Optical flow measures the shift in image area due to motion in a visual scene. For a fixed camera and approximately constant illumination, the field of optical flow will be local. The method used for calculating the optical flow is not particularly limited, but a method known as block matching or block correlation described in Non-Patent Document 1 can be used.

  By using block correlation techniques, a region of an image frame is matched to a region of exactly the same size in the following frame. Such matching is determined by minimizing the size of the distance between images (for example, the absolute sum or the sum of squares of the luminance difference of each pixel). For example, in an optical flow field with a frame rate of 15 fps, the displacement of the feature amount due to the motion inside the face is about several pixels for each time step.

  In order to measure motion as velocity, for example, the optical flow field between frames separated by 1, 3, 5, 7 and 10 video frames is calculated. The optical flow value is further averaged between corresponding frames for each area corresponding to the internal part of the face detected by the face detection module.

  FIG. 3 is a diagram illustrating an example of an optical flow field detected for a face image.

  In the case of a movement that stiffens one side of the face, the motion vector is indicated by a white arrow at each of the lattice points (referred to as a field) on the face image.

  It can be seen that a motion vector appears in accordance with the movement of each part of the face when there is movement such as during the stiffness of the face or during the relaxation of the facial expression.

  Regarding the average of the optical flow values calculated by the optical flow calculation unit 1206, five velocity levels that are the magnitudes of the motion vectors obtained by the averaging process at the intervals of the above-described 1, 3, 5, 7, and 10 video frames. The distribution of operating speed is required. For this reason, it may correspond to the movement due to such a large movement and the whole face (head) moving rigidly, or it corresponds to a more subtle movement corresponding to facial expressions and conversation. It can be detected. In other words, when the entire face is moving, a peak appears in any of the five motion vectors, whereas if it is a movement inside the face, the speed level at which the peak exists depends on the video frame interval. The feature extraction unit 1402 can interpret the movement of the face such that non-existing speed levels coexist.

  Although not particularly limited, the feature extraction unit 1402 calculates feature amounts corresponding to the index 1 and the index 2 according to the following equations.

  Here, the subscript X is X = {R, NR}, and when it is R, it means a feature amount corresponding to the rigid motion of the index 1, and when it is NR, it is inside the face of the index 2. This indicates that the feature amount corresponds to the movement.

  The feature amount is updated at a time interval T (for example, 0.5 seconds). n is the number of times index 1 or index 2 exceeds a predetermined threshold value during period [t-T, t], and δ is an integration coefficient (for example, 0.05).

  As another index (index 3), information indicating whether a face is detected in the image is used. The feature extraction unit 1402 changes the index 3 so as to attenuate to a predetermined level or less when a face image is not detected from the screen for a predetermined period (for example, 15 seconds).

  The emotion model calculation unit 1404 calculates the emotion state according to the following formula, for example, when the index 3 is equal to or higher than a predetermined level.

  Accordingly, although this is not particularly limited, for example, when it is determined as “excited state” based on the calculation result of the emotion model calculation unit 1404, the acoustic effect processing unit 1406 adds the audio signal received from the audio reproduction device 200 to the audio signal. A predetermined acoustic effect is added and output to the speaker 120.

  When skin electrical reflex is measured alone, for example, when “skin conductivity” is larger than a predetermined value, it can be determined as “excited state”.

  Furthermore, for example, when observing both a face image and skin electrical reflection, for example, the calculation result of the emotional state as described above indicates “excited state” or “skin conductivity” is lower than a predetermined value. When it is large, it can be determined as an “excited state”.

(Sound effect addition module program 1400)
FIG. 4 is a diagram illustrating an example of an acoustic effect added in the acoustic effect addition module program 1400.

Hereinafter, the sound effect is referred to by the following code name.
That is, “N” is “no sound effect”, “LP” is “low-pass filter processing”, “SB” is “two-beat backward skip”, and “SF” is “two-beat forward skip”. “HP” indicates “high-pass filter processing”, “dub” indicates “dub effect (plays echo part while delaying 8 beats by 8 beats)”, and “FS” indicates “12 “Filter sweep processing from 100 to 10,000 Hz in the beat section”, “RW” “plays backward by one beat and then plays forward”, “dub HP” “dub processing + high-pass filter processing” ”,“ Dub LP ”is“ dub processing + low-pass filter processing ”,“ T ”is“ 1 beat period, tempo is changed ± 10% ”,“ FG ”is“ intermittent flange effect ” , “Vol” is “12 beat periods, amplitude “± 10% change” and “FG HP” indicate “intermittent flange effect + high-pass filter processing”, respectively.

  Here, the acoustic effect added by the acoustic effect addition module program 1400 is preferably such that the impression that the acoustic effect gives to the user 2 is reminiscent of the “emotional state” of the subject 4. When such association works, the user 2 can intuitively sense the “emotional state” of the subject 4.

FIG. 5 is a diagram showing the results of a sensory experiment for the acoustic effect shown in FIG.
In this sensory experiment, 10 subjects listened to 45 music samples (music clips) for 20 seconds each. The music samples are three songs of different genres, bossa nova, classical (solo piano) and electronic music.

  The subjects listened in random order to the original music clip followed by a version with 14 sound effects and evaluated them according to three criteria. First, the activity level impression received from music was rated as “1” when feeling relaxed, “2” when feeling normal, and “3” when feeling excited. . Further, regarding whether or not such an acoustic effect can be recognized, “1” is “when the effect cannot be recognized”, “2” when it is simply felt, and “2” when it is recognized that it can be sensed. “3”, “4” when clearly felt, and “5” when very noticeable. Also, the degree of discomfort of music was evaluated as “1” when it was very fun, “2” when it was fun, “3” when neutral, “4” when it was uncomfortable, and “5” when it was very uncomfortable. . The evaluation was averaged for each item.

  From the results of FIG. 5, for example, “LP” and “FS” are suitable for expressing the “relaxed state”, and “dub HP” and “RW” are suitable for expressing the “excited state”. It turns out that it is suitable.

  As described above, according to the present invention, it is possible to notify emotion information of a cared person without interfering with the work of the caregiver who is doing other work.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

It is a schematic block diagram which shows the structure of the emotion information notification apparatus 100 of this invention. 3 is a functional block diagram for explaining a configuration of software operating on the emotion information notification device 100. FIG. It is a figure which shows the example of the field of the optical flow detected about the face image. It is a figure which shows the example of the acoustic effect added in the acoustic effect addition module program 1400. FIG. It is a figure which shows the result of the sensory experiment with respect to the acoustic effect shown in FIG.

Explanation of symbols

  2 users, 4 subjects, 100 emotion information notification device, 102 camera, 106 keyboard, 108 display, 110 computer, 200 audio playback device, 1102 interface unit, 1104 CPU, 1106 ROM, 1108 RAM, 1110 hard disk, 1200 video module program, 1202 Capture unit, 1204 Image data recording processing unit, 1206 Optical flow calculation unit, 1208 Face detection unit, 1400 Sound effect addition module program, 1402 Feature extraction unit, 1404 Emotion model calculation unit, 1406 Sound effect processing unit

Claims (3)

Detection means for obtaining observation information on the subject;
An emotional state calculating means for determining the emotional state of the subject according to the observation information;
An emotion information notification apparatus comprising: an acoustic effect processing unit that adds a predetermined acoustic effect to an externally input audio signal according to the determination result of the emotional state and outputs the result. The detection means includes imaging means for acquiring moving image data about a target image area including a face image of a subject,
The emotional state calculating means includes
A storage device for storing the moving image data;
Face detecting means for detecting a face in the target image area stored in the storage device;
Motion detection means for detecting a motion vector for each unit region in the detection region corresponding to the detected face;
The emotion information notification apparatus according to claim 1, further comprising: a determination processing unit that determines whether the emotion state of the subject is at least an excitement state based on the detected temporal change of the motion vector. The motion detection means includes optical flow calculation means for calculating an optical flow for each point on a lattice point in the face image data corresponding to the detection area at a plurality of different speed levels,
Based on the optical flow corresponding to the plurality of different speed levels, as information related to the motion vector, first information corresponding to the magnitude of the movement of the subject's head, and the movement of the facial expression of the subject A calculation unit for extracting second information corresponding to the size;
The emotion information notification apparatus according to claim 1, wherein the determination processing unit performs determination processing according to accumulation of the first and second information.