JP2007200127A - Expression image recording and retrieving device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an expression image recording and retrieving device capable of easily retrieving feelings of a human recorded for a long period of time. <P>SOLUTION: A video module program 1200 includes a face detection section 1208 which extracts a face area from a moving picture stored on a hard disk 1110 and an optical flow calculation section 1400 which calculates information for specifying movements of inner parts of the detected and extracted face on the image on the basis of an optical flow. An optical flow calculation section 1206 stores the moving picture of a face image together with calculated movement property data. Further, information on skin electric reflection is also stored. A retrieving module program 1400, on the other hand, includes a retrieval processing section 1404 which retrieves data on the basis of input information from a user 2 and a display control section 1402 which performs processing for displaying a face image corresponding to a specified frame in the moving picture of the face image of the retrieved person 4 to be examined on a display section 108. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a configuration of a facial expression image recording / retrieval device for recording a facial expression image of a subject over a long period of time and retrieving it afterwards.

  In many countries, the proportion of the elderly population is increasing, and the proportion of people who have a lifestyle that makes it possible for them to do their own things is also increasing.

  In Japan, such elderly people often live with their families and require the family to pay attention for a considerable amount of time every day. This means that the caregiver will continue to be in a state of being able to keep an eye on the care recipient, which will be a great mental and physical burden on the caregiver.

  Eventually, it is desirable to develop a system that reduces the burden on such caregivers and takes over some of the attention directed to the caregivers.

  However, unless the caregiver's condition, for example, the facial expression is recorded over a long period of time and the data of the correlation between the change in the caregiver's facial expression and the attention that should be accompanied with it is accumulated, Development of such a system is difficult.

  On the other hand, in recent years, a method for grasping the movement of an object around the imaging device from an image taken with the imaging device has been proposed. In this method, the movement of an object being captured is grasped by measuring an optical flow between frames of a moving image.

  Such an “optical flow” is obtained by using a feature matching method (pattern matching), a spatiotemporal gradient method, or the like for each frame of an image (see, for example, Patent Document 1 and Non-Patent Document 1).

In addition to motion detection in a photographed image, attempts have been reported to relate to human facial expressions using optical flow (see, for example, Non-Patent Document 2).
JP 2004-355082 A Barrron, JL, Fleet, DJ, Beauchemin, SS: Performance of optical flow techniques.The International Journal of Computer Vision, 12 (1994) pp.43-77 Lyons, MJ, Funk, M., Kuwabara, K .: Segment and Browse: A Strategy for Supporting Human Monitoring of Facial Expression Behavior, INTERACT 2005 (LNCS 3585), pp. 1120-1123 (2005).

  However, a method for observing a change in face image more objectively in connection with the psychological state of the subject has not always been clear.

  Moreover, if the system as described above is developed, there is a problem of how to easily search for human facial expressions recorded over a long period of time. In other words, in order to proceed with further development, it is necessary to develop a method for helping to evaluate the emotional state of the person receiving care as described above.

  The present invention has been made to solve such problems, and an object thereof is to provide a facial expression image recording / retrieval device capable of easily retrieving a human facial expression recorded over a long period of time. It is to be.

  According to one aspect of the present invention, a facial expression image recording / retrieval device, which is a photographing unit that acquires moving image data for a target image area including a face image of a subject, and a measuring unit that measures the skin electrical reflection of the subject A motion vector for each unit region in a storage device for storing moving image data, face detection means for detecting a face in the target image region, and a detection region corresponding to the detected face, Motion detection means for associating the face vector data corresponding to the motion vector and the detection area and identification information for specifying the face image data on the time axis and storing them in the storage device, and information relating to the face image data and the motion vector And display control means for displaying a list of information corresponding to the skin electroreflection, the display control means corresponding to the identification information specified by the image specifying information, The information corresponding to the information and electrodermal reflections associated with the data and the motion vector, and updates the list.

Preferably, an input means for receiving image specifying information from the user is further provided.
Preferably, the motion detection means includes optical flow calculation means for calculating the optical flow for each point on the grid point in the face image data corresponding to the detection area at a plurality of different speed levels, and the facial expression image record search device includes: Calculating means for extracting information corresponding to the motion of the subject's head and information corresponding to the movement of the facial expression of the subject as information related to the motion vector based on optical flows corresponding to a plurality of different speed levels; Further prepare.

  Preferably, the calculation means further calculates an importance index that monotonously increases in accordance with at least one of the magnitude of the subject's head movement, the magnitude of the subject's facial expression, and the magnitude of the skin electrical reflection, and performs display control. The means displays the importance index in a list display.

(Outline of the present invention)
Facial expressions provide important information about human mental state. As described below, the present invention is a human who judges human emotions from facial expressions by combining facial expression data with contextual information in terms of judging the emotions of subjects (caregivers). It is a system that reinforces the ability of

  That is, the facial expression image recording / retrieval apparatus of the present invention is for inputting a search information, a computer such as a personal computer or a workstation having an image input interface, software executed on the computer, an imaging apparatus such as a camera, and the like. And a display device for displaying a search result. The face portion is extracted from the image of the face of the person, and the movement of the face is detected. The change in skin resistance is also detected, and the movement information, the change in skin resistance and the input annotation information are stored in association with each other, and the corresponding face image is obtained using the movement information and the annotation information as a clue. It is possible to search.

  The search information may be input each time from the input device as described above in order to proceed with the search interactively with the user. The search information itself is predetermined and is not necessarily input each time. It does not have to be information to be made. Alternatively, if the facial expression image recording / retrieval device outputs a retrieval result in response to an inquiry from another system, the facial expression image recording / retrieval device itself is not provided with an input device that accepts retrieval information from the user. Also good.

  The computer is not limited to the general-purpose computer as described above, but may be a built-in computer incorporated in a dedicated device.

[Hardware configuration]
Hereinafter, a facial expression image recording / retrieval device according to an embodiment of the present invention will be described.

FIG. 1 is a schematic block diagram showing the configuration of a facial expression image recording / retrieval apparatus 100 according to the present invention.
Referring to FIG. 1, a facial expression image recording / retrieval device 100 includes a computer 110 such as a personal computer, a display 108 as a display device connected to the computer 110 and displaying information to the user 2, and the computer. 110, a keyboard 106 as an input device for inputting instructions and information from the user 2, a camera 102 for capturing an image including the face of the subject 4, and a change in skin resistance of the subject 4 are detected. And an electrode 104 in contact with the skin surface of the subject 4 in order to detect Galvanic Skin Response (GSR). The input device may further include a mouse, a tablet, and a pen.

  In the system of this embodiment, for example, a video camera including a CCD (solid-state imaging device) is used as the camera 102 to capture image data as digital data.

  Although not shown, it is assumed that the computer 110 is provided with a device for taking in data from an external recording medium such as a CD-ROM (Compact Disc Read-Only Memory) drive.

  Further, referring to FIG. 1, a computer 110 includes a CPU (Central Processing Unit) 1104, a ROM (Read Only Memory) 1106, a RAM (Random Access Memory) 1108, and a hard disk 1110 connected to each bus BS. And an interface unit 1102 for transferring data such as capturing an image from the camera 102, measuring the skin resistance of the subject 4 with the electrode 104, or receiving a signal from the input device 106.

  As described above, the main part of the part that functions as the facial expression image recording / retrieval device 100 is realized by computer hardware and software executed by the CPU 1104. Generally, such software is stored and distributed in a storage medium such as a CD-ROM, read from the storage medium by a CD-ROM drive or the like, and temporarily stored in the hard disk 1110. Alternatively, when the device is connected to the network, it is temporarily copied from the server on the network to the hard disk 1110. Then, it is further read from the hard disk 1110 to the RAM 1108 and executed by the CPU 1104. In the case of being connected to a network, for example, the program may be directly loaded into the RAM 1108 without being stored in the hard disk 1110 and executed.

  The computer hardware itself shown in FIG. 1 and its operating principle are general. Therefore, the most essential part of the present invention is software stored in a storage medium such as a CD-ROM or a hard disk 1110.

  As a recent general trend, various program modules are prepared as part of a computer operating system, and an application program generally calls a module in a predetermined arrangement to advance processing when necessary. is there. In such a case, the software itself for realizing the facial expression image recording / retrieval apparatus does not include such a module, and the facial expression image recording / retrieval apparatus is implemented only in cooperation with the operating system on the computer. However, as long as a general platform is used, it is not necessary to distribute software including such modules. The software itself that does not include these modules and the recording medium that records the software (and the software distributes on the network). Data signal) can be considered to constitute the embodiment.

Software configuration
FIG. 2 is a functional block diagram for explaining the configuration of software that operates on the facial expression image recording / retrieval apparatus 100 shown in FIG.

  In FIG. 2, the programs operating on the computer 110 include a video module program 1200 and a search module program 1400 for searching for a desired face image from moving images captured and stored in the hard disk 1110. The search module program 1400 can be a browser-based program.

  As will be described below, the facial expression image recording / retrieval apparatus 100 functions as a support system for facilitating long-term monitoring of a human mental state via facial expressions. The moving image is processed in real time to separate the facial region and record the facial expression. An optical flow method is used to interpret the movement of the face.

  The video module program 1200 includes an image capture unit 1202 that acquires a moving image including a target person and surrounding environment from the camera 102, and an image data recording processing unit that stores the captured moving image in the hard disk 1110. 1204. The image capture unit 1202 converts the digital moving image data input from the camera 102 into an image format suitable for storage as necessary.

  The video module program 1200 further identifies a face detection unit 1208 that extracts a face area from a moving image stored in the hard disk 1110, and a motion on the extracted face itself or an image inside the face based on the optical flow. And an optical flow calculation unit 1206 for calculating information (for example, distribution of motion vectors in the image). The optical flow calculation unit 1206 includes moving image data of a face image together with calculated motion attribute data. Store in hard disk 1110.

  Further, the GSR data recording processing unit 1304 stores GSR data corresponding to the skin resistance measured by the interface unit 1102 via the electrode 104 in the hard disk 1110 together with the moving image data. The GSR data is filtered in the GSR data recording processing unit 1304 in order to remove the slowly changing background signal. This change in the GSR data indicates the timing at which the subject 4 has changed psychological state such as feeling stress, being interested in something, or being excited about something.

  That is, the moving image data of the face image is stored in the hard disk 1110 while being temporally correlated with the following three attribute data obtained as described in detail later.

1) Index of rigid head movement (direction and size of movement)
2) Index of movement inside the face due to facial expression or speech 3) GSR data On the other hand, the search module program 1400 obtains facial image data in the hard disk 1110 based on input information from the user (for example, caregiver) 2. A search processing unit 1404 for searching, and a display control unit 1402 for performing a process of causing the display unit 108 to display a face image corresponding to a specific frame among the moving images of the face image of the searched subject (for example, a cared person) 4. Including.

  As will be described later, annotations (comment information) indicating the mental state of the subject 4 input from the input unit 106 to the display control unit 1402 are stored in the hard disk 1110 in association with the moving image of the face image. It can also be made. Here, the “mental state” is information representing the emotional state of the subject 4 such as “smiling state”, “depressed state”, “angry state”, “excited state”, and the like. It is.

(Video module program 1200)
Moving image data input from the camera 102 is thinned from, for example, 640 × 480 pixels to 320 × 240 pixels in order to reduce the amount of calculation for optical flow processing and the data capacity for storage in the hard disk 1110. It is burned.

  A face detection unit 1208 determines whether or not a face exists in the field of view of the camera 102 using a well-known face detection algorithm, and also determines where it exists. Although not particularly limited, this face detection algorithm registers a rectangular area having an aspect ratio fixed to the face so as to be positioned at the center of the head image. For example, a face rotating from a plane up to about 30 ° can be accurately detected, and similarly, a face inclined at about 30 ° in the image plane can also be accurately detected. The face image is scaled and the face image corresponding to each frame is stored with a time stamp at a rate of up to 30 frames / second. By normalizing the size and position of the face and separating the face from complex backgrounds, it is possible to more easily browse face data. Although the time stamp is used here, the time stamp is not necessarily limited as long as it is identification information that can identify the frame corresponding to the face image in the moving image in the time series.

  In the optical flow calculation unit 1206, the optical flow region of the image detects motion between various video frames. The detected motion vector is stored in the hard disk 1110 in association with the moving image data of the face image together with the average of the motion vectors over the entire internal area of the face.

  Optical flow measures the shift in image area due to motion in a visual scene. For a fixed camera and approximately constant illumination, the field of optical flow will be local. The method used for calculating the optical flow is not particularly limited, but a method known as block matching or block correlation described in Non-Patent Document 1 can be used.

  By using block correlation techniques, a region of an image frame is matched to a region of exactly the same size in the following frame. Such matching is determined by minimizing the size of the distance between images (for example, the absolute sum or the sum of squares of the luminance difference of each pixel). For example, in an optical flow field with a frame rate of 15 fps, the displacement of the feature amount due to the motion inside the face is about several pixels for each time step.

  In order to measure motion as velocity, for example, the optical flow field between frames separated by 1, 3, 5, 7 and 10 video frames is calculated. The optical flow value is further averaged between corresponding frames for each area corresponding to the internal part of the face detected by the face detection module.

  FIG. 3 is a diagram illustrating an example of an optical flow field detected for a face image.

  In the case of a movement that stiffens one side of the face, the motion vector is indicated by a white arrow at each of the lattice points (referred to as a field) on the face image.

  It can be seen that a motion vector appears in accordance with the movement of each part of the face when there is movement such as during the stiffness of the face or during the relaxation of the facial expression.

(Search module program 1400)
FIG. 4 is a view showing a screen shot of the browser display of the search module program 1400.

  The display components of this module include a display 200 normalized for the position and size of the face image at a selected time, a clock display 202 for displaying hours, minutes and seconds, and a specific day A calendar 204 for selecting an image, a time-resolution zoom control display 206 for selecting a timeline area, a timeline 208 for scrolling time by an operation from a mouse, and an expression or utterance An area 210 for displaying a movement index element (corresponding to the magnitude of an optical flow at a predetermined speed: non-rigid motion: index 1) and a rigid head movement index element (motion magnitude) : Rigid motion: area 212 for displaying index 2), area 214 for displaying skin conductivity (index 3), and importance index obtained from these three indices are displayed. An area 216 for displaying a GSR waveform, and a display area 220 for adding and displaying user comments (comments). Here, the “importance index” is an index that is defined by a function that monotonically increases for at least one of index 1, index 2, and index 3, and is calculated by the CPU 1104. Further, here, the meaning of “a function that increases monotonously for at least one” does not have to be a function for all the indices of index 1, index 2, and index 3, for example, index 1, index 2, and the like. When the importance index is defined as a function that depends on index 2 and index 3 of index 3, it is a function that monotonically increases when attention is paid to either index 2 or index 3. It is. For example, when the linear combination of the index 1, the index 2, and the index 3 is defined as the importance index, when this value exceeds a predetermined threshold, the search module program 1400 is important for the subject 4. It can be determined that there has been a change. The user 2 can also search for a face image corresponding to a time when the importance index is greater than or equal to a certain threshold value.

  The user selects a date, then zooms in on the time range and selects a timeline range. By scrolling the timeline, face images at each time point are displayed. At this time, in accordance with the scrolling of the timeline, the display contents of the areas 210 to 220 displayed in a list are also switched to the display of the data at the corresponding time. For example, if the importance index is color-coded according to a predetermined stage for its value, and the other indices 1 to 3 are also individually color-coded and displayed, 2 can easily determine whether or not a change in the psychological state of the subject 4 has occurred at the time of the current display while moving the timeline.

  Regarding the average of the optical flow values calculated by the optical flow calculation unit 1206, five velocity levels that are the magnitudes of the motion vectors obtained by the averaging process at the intervals of the above-described 1, 3, 5, 7, and 10 video frames. The distribution of operating speed is required. For this reason, it may correspond to the movement due to such a large movement and the whole face (head) moving rigidly, or it corresponds to a more subtle movement corresponding to facial expressions and conversation. It can be detected. In other words, when the entire face is moving, a peak appears in any of the five motion vectors, whereas if it is a movement inside the face, the speed level at which the peak exists depends on the video frame interval. The CPU 1104 can interpret the movement of the face so that a speed level that does not exist is mixed.

  Further, the user 2 can add a simple text annotation to the displayed face image data from the input device 106, and the annotation data is also stored in the hard disk 1110 in association with the face image data. The

  As described above, according to the present invention, it is possible to easily search for human facial expressions recorded over a long period of time.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

It is a schematic block diagram which shows the structure of the facial expression image record search apparatus 100 of this invention. 4 is a functional block diagram for explaining a configuration of software that operates on the facial expression image recording / retrieval apparatus 100. FIG. It is a figure which shows the example of the field of the optical flow detected about the face image. It is a figure which shows the screen shot of the browser display of the search module program 1400. FIG.

Explanation of symbols

  2 users, 4 subjects, 100 facial expression image record retrieval device, 102 camera, 106 keyboard, 108 display, 110 computer, 1102 interface unit, 1104 CPU, 1106 ROM, 1108 RAM, 1110 hard disk, 1200 video module program, 1202 capture unit, 1204 Image data recording processing unit, 1206 Optical flow calculation unit, 1208 Face detection unit, 1400 Search module program, 1402 Display control unit, 1404 Search processing unit

Claims (4)

Photographing means for acquiring moving image data about a target image area including a face image of a subject;
Measuring means for measuring the skin electrical reflex of the subject;
A storage device for storing the moving image data;
Face detection means for detecting a face in the target image region;
Identification for detecting a motion vector for each unit region in the detection region corresponding to the detected face, and specifying the motion vector, the face image data corresponding to the detection region, and the face image data on the time axis Motion detection means for associating information and storing the information in the storage device;
Display control means for displaying a list of information relating to the face image data, the motion vector, and information corresponding to the electrodermal reflection,
The display control means updates the list display to the corresponding face image data, information related to the motion vector, and information corresponding to the skin electrical reflection according to the identification information specified by the image specifying information. A facial expression image record retrieval device.   The facial expression image recording / retrieval apparatus according to claim 1, further comprising input means for receiving the image specifying information from a user. The motion detection means includes optical flow calculation means for calculating an optical flow for each point on a lattice point in the face image data corresponding to the detection area at a plurality of different speed levels,
Based on the optical flow corresponding to the plurality of different speed levels, as information related to the motion vector, information corresponding to the motion of the subject's head and information corresponding to the motion of the subject's facial expression are extracted. The facial expression image recording / retrieval apparatus according to claim 1, further comprising an arithmetic unit that performs the operation. The calculation means further calculates an importance index that monotonously increases according to at least one of the magnitude of the movement of the subject's head, the magnitude of the movement of the facial expression of the subject, and the magnitude of the electrical skin reflex,
4. The facial expression image recording / retrieval device according to claim 3, wherein the display control means displays the importance index in the list display.