JP2006075348A - Eye contact analyzing apparatus, eye contact analyzing method and eye contact analyzing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an eye contact analyzing apparatus and its method which enable detection at a low cost and with high precision that an eye contact (eye contact) which is important as the information expressing the speaker's condition during conversation is occurred. <P>SOLUTION: The eye contact analyzing apparatus is equipped with an eyeball movement measuring part 210 for detecting the movement of the eyeballs of one representative (corresponding to a subject) of participants participating in the conversation, a visual field image pickup part 220 for picking up the image of the region corresponding to the visual field of the subject, a face region detecting part 310 for detecting the face region of the participant other than the subject from the image picked up at the visual field image pickup part 220, and an eye contact judging part 320 for judging whether or not the eye contact is occurred by judging whether the visual line is directed to the participant from the movement of the eyeballs detected by the eyeball movement detecting part 210 and the face region. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an apparatus for analyzing line of sight during communication such as dialogue, and more particularly to a mutual vision analysis apparatus for analyzing mutual vision called eye contact.

  It is thought that people who actually meet and talk with people can convey their thoughts rather than telephone calls. This is because you can talk with your eyes. “Looking at your eyes” with your partner is a sign that you have acknowledged each other and is an important means of communication. Therefore, analyzing the mutual vision during dialogue is important in communication research and support device development.

  As a method of detecting mutual vision, by detecting the eye movements of all the people who are actually interacting and calculating the direction of the gaze of the participants, mutual vision is detected depending on whether the mutual eyes intersect And a method of determining that mutual observation has been performed when light emitted from the head of another person is received using a laser light source and a laser receiver installed on the head.

  As a method of actually measuring eye movement, measurement is performed using a dedicated goggle-shaped eye movement measuring instrument, or an eye area is extracted from a human face image using image processing, and eye movement is performed from the eye area image. There is a method of measuring.

  For example, Patent Document 1 relates to a visual information analysis apparatus, and in order to detect an eyeball position, infrared rays are applied to the eyeball, and the reflected light is detected to measure the eyeball position data.

JP-A-60-126140

  However, in the above-described method for detecting mutual vision, from the viewpoint of accuracy of mutual vision detection, it is desirable to measure the line of sight of a person who is interacting using a dedicated goggle eye movement measuring instrument. Therefore, it is necessary to prepare eye movement measuring instruments for a plurality of persons, which is problematic in terms of cost. In addition, wearing a goggle-shaped measuring instrument hinders the participants' natural dialogue.

  On the other hand, the method for measuring eye movement from a human face image does not require a special device such as an eye movement measuring instrument, and thus can be realized at a lower cost than the method using an eye movement measuring instrument. It is easy to be affected by the lighting conditions, and there is a problem in the accuracy of measuring eye movement, and the accuracy of detecting mutual vision is not high. In addition, in the method of measuring eye movements from a person's face image, the person moves outside the imaging range of the camera unless the camera is appropriately moved to capture the image as the person moves. It is difficult to obtain data.

  The method of detecting mutual vision using a laser light source and laser receiver installed on the head is superior to the case of actually measuring eye movements in that data can be obtained stably. Yes. However, in the mutual vision actually generated during the dialogue, there are cases where only the eyes are moved without moving the head, and in this case, the direction of the head and the direction of the line of sight are deviated, so that it is not determined as mutual vision. On the other hand, there is a problem that even if the eyes are not aligned, if there is only the head direction, it will be judged as mutual vision.

  SUMMARY OF THE INVENTION The present invention solves the above-mentioned problems, and a mutual vision analysis apparatus and method capable of accurately and inexpensively detecting that important mutual vision (eye contact) has occurred as information representing the state of a speaker during dialogue. I will provide a.

  In order to solve the above problems, the mutual vision analyzer of the present invention is an eye movement detection means for detecting the movement of the eyeball of one representative participant (participating subject) participating in the dialogue, and the field of view of the participant representative. Visual field imaging means for imaging an area corresponding to the above, a detection means for detecting a person area (preferably a face area) of a participant other than the participant representative from an image captured by the visual field imaging means, and an eye movement detection means And a mutual vision determination unit that determines whether or not the line of sight is directed to the participant based on the movement of the eyeball and the face area detected by the detection means and determines whether or not mutual vision has occurred.

  Whether the participant's eye movement is measured by measuring the eyeball movement of one subject representative of the conversation participant, detecting the face area of other participants from the image captured by the subject's visual field imaging means, and whether the line of sight is directed at the participant Therefore, unlike the conventional method, it is not necessary for all of the participants to wear a goggle-like eye movement measuring device and interact with each other, and an inexpensive mutual vision analyzer can be provided. Furthermore, in order to determine mutual vision by capturing images within the visual field range of the subject, a visual field image in the direction of the line of sight is captured even when only the eyes are moved without moving the head, which was a conventional problem. Therefore, it is possible to determine the mutual vision with high accuracy.

  In addition, the image of the field of view imaging means is also viewed from the viewpoint of the method of measuring the eye movement from the face image of the person and the problem that the camera has to be moved as the person moves. Since the other party that always performs mutual vision is always included in this case, the fact that mutual vision has been performed can be reliably detected.

  Preferably, the mutual vision determination means determines whether or not the subject's line of sight is directed toward the person area from the movement of the eyeball. For example, the mutual vision determination means compares the position of the subject's line of sight on the person area and the distance to the face of the person area is larger than a certain value, and there was mutual vision when the distance is smaller than the certain value You may make it determine with a thing. Alternatively, the mutual vision determination means compares the position of the subject's gaze direction on the person area and the distance to the eyes of the person area is greater than a certain value, and determines that there was mutual vision when the distance is smaller than the certain value. You may make it do.

  More preferably, the mutual vision determination unit calculates the gaze direction of the person (participant) from the face and eye position information included in the person area, and determines whether or not the gaze of the person is directed toward the subject. To do. In this case, the mutual vision determination means may compare the line-of-sight direction calculated from the person area with the line-of-sight direction of the subject, and determine that there was mutual vision when the difference is smaller than a certain value. .

  The mutual vision analysis method according to the present invention includes an eye movement detection step for detecting the movement of the eyeball of the subject, a visual field imaging step for imaging a region corresponding to the visual field of the subject, and an image captured by the visual field imaging means. A detection step for detecting a person region representing a person other than the above, a mutual determination for determining whether or not mutual vision has occurred based on the movement of the eyeball detected by the eye movement detection step and the person region detected by the detection step A visual determination step.

  The mutual vision analysis program according to the present invention includes an eye movement detection step for detecting the movement of the eyeball of the subject, a visual field imaging step for imaging a region corresponding to the visual field of the subject, and an image captured by the visual field imaging means. A detection step for detecting a person region representing a person other than the above, a mutual determination for determining whether or not mutual vision has occurred based on the movement of the eyeball detected by the eye movement detection step and the person region detected by the detection step A visual determination step.

  According to the present invention, since the mutual vision is determined using the eye movement of the subject and the image from the visual field imaging means of the subject, it is possible to provide a high-accuracy mutual vision analyzer that is low in cost. .

  A mutual vision analyzer according to an embodiment of the present invention will be described with reference to the drawings. The analysis apparatus according to the present embodiment is an analysis apparatus that can detect that important mutual vision has occurred as information representing the state of a speaker during a conversation. Mutual vision is a state in which both eyes agree with each other or can be regarded as coincident, and dialogue is not only a one-on-one situation, but also a meeting or a one-to-many meeting involving multiple people. Including situations where presentations are made. Furthermore, the dialogue includes not only actual face-to-face dialogue but also dialogue at a remote place using a video conference system or the like. Further, the device is synonymous with a concept including a system including a plurality of devices.

  Next, the mutual vision analyzer according to the present embodiment will be described. FIG. 1 shows an external shape of goggles preferably used in the mutual vision analyzer according to the present embodiment, and FIG. 2 shows a configuration example of a data processing device for processing data from the goggles.

  The goggles 10 are worn by representatives of participants who participate in the dialogue (hereinafter referred to as subjects), and the subjects interact with at least one dialogue participant. The goggles 10 are attached with a field-of-view camera 20 that captures an area corresponding to the visual field of the subject and an eye movement measurement device (sensor) 30 for measuring the eye movement of the subject. The field-of-view camera 20 is preferably attached to the approximate center of the goggles 10, uses an image sensor such as a CCD having an angle of view equivalent to the field of view of the subject, and has an autofocus function.

  The eye movement measuring device 30 is, for example, a method using corneal reflection light that irradiates the subject's cornea with infrared rays and detects the movement of the eyeball based on the reflected light, a method that detects movement of a contact lens mounted on the subject, EOG method that detects the movement of the eyeball from the usage of muscles near the eyeball of the eyeball, or the scleral reflection method that utilizes the difference in reflectance between the white and black eyes of the cornea as disclosed in Patent Document 1 Can do.

  Further, the goggles 10 can transmit the image information captured by the visual field camera 20 and the eye movement information measured by the eye movement measuring device 30 to the data processing device 40 via a communication means (not shown). Communication between the goggles 10 and the data processing device 40 is preferably performed by radio using radio waves so as not to limit the behavior of the subject.

  As shown in FIG. 2, the data processing device 40 receives information from the field camera 20 and the eye movement measuring device 30 of the goggles 10 or transmits a control signal to the field camera 20 and the eye movement measuring device 30. A communication unit 50 including a possible wireless communication unit, a data memory 60 for temporarily storing data from the goggles 10 and arithmetic processing data, a program memory 70 for storing a program for mutual analysis, and the result of mutual analysis According to a program stored in a program memory 70, a storage device 100 such as a hard disk having a large-capacity storage space, an input interface 90 that receives an instruction from a user via an input device such as a mouse or a keyboard, and the like. A central processing unit (CPU) 110 for controlling That.

  The mutual vision analyzer according to the present embodiment is configured by two devices or systems, that is, a goggle 10 and a data processing device 40 worn by a subject, and information sent from the visual field camera 20 and the eye movement measuring device 30 of the goggle 10. Based on the above, the data processing device 40 analyzes the mutual view between the dialogue participants.

  FIG. 3 is a block diagram illustrating a functional configuration of the mutual vision analyzer according to the present embodiment. As shown in the figure, an eye movement measurement unit 210 and a visual field imaging unit 220 are included on the goggles 10 side, and a face region detection unit 310, a mutual vision determination unit 320, and a mutual vision are included on the data processing device 40 side. An information display unit 330 is included.

  The eye movement measurement unit 210 outputs the eye movement information of the subject measured by the eye movement measurement device 30 to the mutual vision determination unit 320. The visual field imaging unit 220 outputs image information captured by the visual field camera 20 to the face area detection unit 310. The face area detection unit 310 receives the image information from the visual field imaging unit 210 and detects the face area of the dialog participant from the received image information. As a method for detecting a face area, two black areas that are likely to be eyes and two black areas that are likely to be nasal cavities are detected from the skin color areas in the image, and these black areas are spaced and positioned at a certain ratio. A method of recognizing a skin color area included in the relationship as a face can be used. Of course, the face area may be detected using other image processing techniques.

  The face area detection unit 310 outputs information about the detected face area image, face position, and face size to the mutual vision determination unit 320. At this time, if there are a plurality of regions that are likely to be face regions, information on all the regions that are likely to be face regions is output to the mutual vision determination unit 320. If the face area cannot be detected by the face area detection unit 310, information indicating that the face area cannot be detected is output to the mutual vision determination unit 320.

  The mutual vision determination unit 320 determines which point of the visual field camera image the subject's line of sight indicates based on the positional relationship between the eye movement of the subject obtained by performing calibration in advance and the visual field camera image by the visual field camera. Based on this, it is determined whether or not the subject's line of sight faces the participant's face area.

  FIG. 4 is a diagram for explaining calibration of the eyeball movement (gaze direction) of the subject and the visual field camera image. When the subject first wears the goggles 10, the test subject is at a certain distance from the field camera 20 and has a certain size, for example, faces a calibration display panel, for example, a signboard, and calibrates within the field of view of the field camera 20. Ensure that the panel is properly imaged. Let the visual field camera image at this time be S, and let the subject visually check nine calibration points of the visual field camera image S (A1 to A8 are equidistant points on the boundary of the image S, A9 is the center point of the image S). The eye movement of the subject when looking at each calibration point is measured by the eye movement measuring unit 210, and the measured value is registered as an initial set value. Thereby, it can be determined from the eye movement of the subject which position the subject is viewing on the visual field camera image.

  The mutual vision determination unit 320 determines whether the participant is facing the direction of the subject wearing the line-of-sight camera from the information on the face region when there is a face region to which the subject is directed by the subject, When facing the direction of the subject, it is determined that the mutual view has been performed.

  As a method for determining whether or not the face area is directed toward the subject from the face area to which the line of sight is directed, two black areas that are considered to be eyes included in the face area and two that are considered to be the nasal cavity For example, a method of determining which direction is directed based on the positional relationship between the center of gravity of the four regions and the center of gravity of the entire skin color region can be used.

  When the binocular vision determination unit 320 determines that the binocular vision has been viewed, the binocular vision information display unit 330 is a time when the binocular vision is determined, an image captured by the visual field camera 20 when the binocular vision is detected, and the like. Are displayed on the display 80. Alternatively, the information may be read from the storage device 100 storing the mutual vision information and displayed on the display 80.

  Next, the flow of the mutual analysis process will be described with reference to the flowchart of FIG. This operation follows the mutual vision analysis program included in the program memory 70. First, the eye movement of the subject wearing the goggles 10 is calibrated (step S101). The calibration of the eye movement includes an initial setting as shown in FIG. Thereby, the variation for every test subject resulting from wearing of goggles is eliminated, and deterioration in the accuracy of mutual vision determination is prevented.

  Next, a dialogue is started with the participants including the subject, the eye movement information of the subject is output from the eye movement measurement unit 210 to the mutual vision determination unit 320, and the field camera image is displayed from the field imaging unit 220 as a face. The data is output to the area detection unit 310 (step S102).

  The face area detection unit 310 detects the face area of the participant participating in the dialogue from the view camera image (step S103). When the face area is detected, the face area detection unit 310 outputs information on the position of the second eye included in the face to the mutual determination unit 320 in addition to the detected position and size of the face area.

  The mutual vision determination unit 320 determines whether or not the subject's line of sight is directed to the face region based on information from the face region detection unit (step S104). For example, as shown in FIG. 6, when the face area F of the participant is detected in the visual field camera image S, the gaze position of the subject in the visual field camera image S is compared with the face area F. The line-of-sight position in the visual field camera image is uniquely determined from the eye movement of the subject as described above. For example, when the subject's line-of-sight position P1 is within the face region F, it is determined that the subject's line of sight is directed toward the participant's face. On the other hand, when the subject's line-of-sight position P2 is away from the face region F by a certain distance or more, it is determined that the subject's line of sight is not directed toward the participant's face. This criterion is an example. In addition, the distance between the subject's line-of-sight position and the eye area included in the face area F is compared. It may be determined that it is directed to the face.

  If it is determined that the subject's line of sight is directed toward the participant's face, the participant's line-of-sight direction is then calculated based on the face region image (step S105). For example, as shown in FIG. 7, the centroid of the four areas including the two black areas E indicating the eyes included in the image of the face area F and the two black areas N that are considered to be the nasal cavity, and the centroid of the entire face area The participant's line-of-sight direction Q is determined based on the positional relationship.

  Next, the mutual vision determination unit 320 determines whether the participant is facing the subject based on the calculated gaze direction Q (step S106). Since the gaze direction of the subject can be easily calculated from the relationship with the gaze position of the visual field camera image, for example, the mutual vision determination unit 320 compares the gaze direction of the participant with the gaze direction of the subject, and the difference between the two is determined. If it is within a certain value, it is determined that the participant's line of sight is directed toward the subject.

  When the mutual vision determination unit 320 determines that there is mutual vision between the subject and the participant, the generation time is recorded in the storage device 100 (step S107). The visual field camera image can also be recorded in the storage device 100 in association with the time.

  When an instruction to end the dialogue is given from the subject or participant through the input interface 90, the mutual vision analysis operation ends (step S108). Information regarding the determination result of the mutual vision determination unit 320 is displayed on the display by the mutual vision information display unit 330 in real time, or once stored in the storage device 100 and then displayed on the display.

  Next, a second embodiment of the present invention will be described. FIG. 8 is a preferred flowchart for performing a mutual analysis when a subject interacts with a plurality of participants. In the second embodiment, in order to identify a plurality of dialogue participants, face images of each participant are stored in the storage device 100 in advance so that the participants can be identified. When a face area is detected in the field-of-view camera image as in the first embodiment (step S203), the mutual vision determination unit 320 identifies the participant X from the detected face area (step S204). ). The identification is performed by matching the face information of each participant stored in the storage device 100, for example, and the participant having the highest similarity is identified as the participant X. When a plurality of face areas are detected, each participant is identified.

  The subsequent steps are performed in the same manner as in the first embodiment, and when it is determined that there has been mutual vision with a plurality of participants, information such as the time of mutual vision and the storage position of the visual field camera image is identified. It is recorded in association with the file of the participant who has been recorded (step S208).

  FIG. 9 is a display example of the mutual vision determination result. As shown to Fig.9 (a), the time when the mutual vision generate | occur | produced between the participant 1,2, ... n and the test subject who are participating in the dialogue is displayed on the display. Further, when the storage position of the field camera image related thereto is displayed and the storage position is clicked with a mouse or the like, the field camera image can be read from the storage device 100 and displayed as a moving image on the display.

  In addition, as shown in FIG. 9B, a dialogue is performed and a subject is viewed for a certain period (in the example shown, from 2 to 3 pm on August 26). The occurrence frequency may be presented. For example, as a frequency calculation method, the number of times the subject has seen the participant's face is counted, or the time during which the subject has seen the participant's face is measured, and the ratio is calculated.

  As described above, according to the mutual vision analysis apparatus of the present embodiment, only one representative (subject) participating in the conversation wears goggles, so that mutual vision with other participants can be observed. Since the presence / absence of the occurrence can be analyzed, the analyzer can be reduced in cost. Furthermore, since the area corresponding to the visual field of the subject is imaged by the visual field camera, it is not necessary to move the camera according to the movement of the person as in the past, and the participant can be easily projected within the imaging range of the visual field camera. it can.

  In the above embodiment, the face area of the participant is detected from the image obtained by the visual field camera. However, the present invention is not limited to this, and a person area including the face area may be detected. In this case, the mutual vision may be determined using information on the entire person area of the participant. Furthermore, the eye area narrower than the face area may be detected with high accuracy to determine mutual vision.

  The preferred embodiments of the present invention have been described in detail above. However, the present invention is not limited to the specific embodiments, and various modifications can be made within the scope of the gist of the present invention described in the claims. Deformation / change is possible.

  The mutual vision analysis apparatus, analysis method, and analysis program according to the present invention can be used for communication research and support during dialogue.

It is a perspective view which shows the external shape of the goggles used for the mutual vision analyzer which concerns on a present Example. It is a figure which shows the structural example of the data processor used for the mutual vision analyzer which concerns on a present Example. It is a block diagram which shows the functional structure of the mutual vision analyzer which concerns on a present Example. It is a figure explaining calibration with a test subject's eyes | visual_axis direction and a visual field camera image. It is a flowchart which shows the flow of a process of a mutual vision analysis. It is a figure explaining whether a test subject's eyes | visual_axis is orient | assigned to the participant's face. It is a figure explaining whether a participant's eyes | visual_axis is orient | assigned to the test subject. It is a flowchart which shows the flow of a process of the mutual vision analysis of 2nd Example. It is an example of a display of a mutual vision determination result.

Explanation of symbols

10: Goggles 20: Field camera 30: Eye movement measuring device 40: Data processing device 50: Communication unit 60: Data memory 70: Program memory 80: Display 90: Input interface 100: Storage device 110: Central processing unit 210: Field imaging Unit 220: Eye movement measurement unit 310: Face region detection unit 320: Interview determination unit 330: Interview information display unit

Claims (14)

Eye movement detection means for detecting the movement of the eyeball of the subject;
Visual field imaging means for imaging a region corresponding to the visual field of the subject;
Detecting means for detecting a person region representing a person other than the subject from the image captured by the visual field imaging means;
Mutual vision determination means for determining whether or not mutual vision has occurred based on the movement of the eyeball detected by the eye movement detection means and the person region detected by the detection means;
A mutual vision analyzer. The mutual vision analysis device according to claim 1, wherein the mutual vision determination unit determines whether or not the subject's line of sight is directed toward the person region from the movement of the eyeball. The mutual vision analyzer according to claim 2, wherein the person area is a face area or an eye area. The mutual vision determination means compares the position of the subject's line of sight on the person area and the distance to the face of the person area is greater than a certain value, and if there is a mutual vision when the distance is smaller than the certain value, The mutual vision analyzer according to any one of claims 1 to 3, wherein the determination is performed. The mutual vision determination means compares the position of the subject's gaze direction on the person area with the distance to the eyes of the person area, and determines that there was mutual vision when the distance is smaller than the certain value. The mutual vision analyzer according to claim 2. The mutual vision according to claim 1, wherein the mutual vision determination means calculates the gaze direction of the person from the position information of the eyes included in the person region, and determines whether or not the gaze of the person is directed toward the subject. Analysis equipment. The mutual vision analysis device according to claim 6, wherein the mutual vision determination unit calculates a gaze direction of the person from position information of eyes and nasal cavity included in the person region. The mutual vision determination means compares the gaze direction of the person and the gaze direction of the subject, and determines that there was mutual vision when the difference is smaller than a certain value. The mutual vision analyzer described. The mutual vision analysis apparatus according to claim 1, further comprising a display unit that displays a determination result by the mutual vision determination unit. A mutual vision analysis system including the mutual vision analysis apparatus according to claim 1. An eye movement detection step for detecting the movement of the eyeball of the subject;
A visual field imaging step of imaging a region corresponding to the visual field of the subject;
A detection step of detecting a person region representing a person other than the subject from the image captured by the visual field imaging means;
A mutual vision determination step of determining whether or not mutual vision has occurred based on the movement of the eyeball detected by the eyeball movement detection step and the person region detected by the detection step;
A mutual vision analysis method. The mutual vision analysis device according to claim 11, wherein the mutual vision determination step determines whether or not the gaze direction calculated from the person region is directed toward the subject. An eye movement detection step for detecting the movement of the eyeball of the subject;
A visual field imaging step of imaging a region corresponding to the visual field of the subject;
A detection step of detecting a person region representing a person other than the subject from the image captured by the visual field imaging means;
A mutual vision determination step of determining whether or not mutual vision has occurred based on the movement of the eyeball detected by the eyeball movement detection step and the person region detected by the detection step;
A mutual vision analysis program. The mutual vision analysis program according to claim 13, wherein the mutual vision determination step determines whether or not the gaze direction calculated from the person region is directed toward the subject.

Images