JP2009106325A - Communication induction system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a system for inducing communication with a user using an anthropomorphic medium, e.g. a stuffed toy. <P>SOLUTION: The communication induction system 10 includes a stuffed toy 14 disposed in front of the user 12. A visual line server 18 predicts the direction of a user's visual line from an image of the face of the user 12 shot by a camera 16. A built-in computer in the stuffed toy 14 predicts or identifies a communication state between them according to conditions of speaking and the user's visual line. The motion (speaking and/or movement) of the stuffed toy is controlled so as to promote the communication with the user according to the communication state. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a communication inducing system, and in particular, for example, an anthropomorphic medium such as a robot is arranged at a position where a user with mild brain disorder such as a dementia patient can visually recognize, and communication from the user is induced by the anthropomorphic medium. It relates to a new communication induction system.

It is known from Patent Document 1 that the functional recovery can be achieved by giving a right brain stimulus to a dementia patient or the like.
JP 2005-160806 A [A63B 24/00 23/035]

  However, such functional recovery training is a condition that at least communication with other people is possible. Therefore, it is necessary to restore communication ability in such patients first. In that case, it would be advantageous to have a system that acts to elicit communication from the patient.

  Therefore, the main object of the present invention is to provide a novel communication induction system.

  Another object of the present invention is to provide a communication inducing system capable of actively drawing out communication from a user.

  The present invention employs the following configuration in order to solve the above problems. Note that reference numerals in parentheses, supplementary explanations, and the like indicate correspondence with embodiments to be described later in order to help understanding of the present invention, and do not limit the present invention.

  1st invention is a communication induction system which induces communication from a user by an anthropomorphic medium arranged at a position where the user can visually recognize, a line-of-sight determination means for determining the state of the user's line of sight, and a state of speech from the user Utterance determination means for determining the gaze state, the gaze state determination result by the gaze position determination means, the storage means for storing the utterance state determination result by the utterance determination means, and the user according to the gaze state determination result and the utterance state determination result stored in the storage means A communication inducing system comprising: a communication state specifying unit that specifies a communication state with an anthropomorphic medium; and a control unit that controls voice and operation of the anthropomorphic medium according to the communication state specified by the communication state specifying unit.

In the first invention, an anthropomorphic medium (14: reference numerals exemplifying corresponding portions in the embodiment; the same applies hereinafter) is disposed in front of the user (12) at a position where the user's line of sight can reach. The line-of-sight determination means (16, 18, 32, S1b, S3, S21b), for example, processes the user's face image captured by the camera (16) with the line-of-sight server (18), so that the user's line of sight (12A) The direction or position is estimated or detected in real time, and the positional relationship of the direction of the user's line of sight to the position of the anthropomorphic medium is determined. For example, it is determined whether the direction of the line of sight is the position of the anthropomorphic medium, the position adjacent to the position of the anthropomorphic medium, or the distance from the position of the anthropomorphic medium. The utterance determination means (32, S1a, S21a) determines the presence / absence of the user's utterance, its state, etc. by calculating the voice input power from the microphone (50), for example. The communication state specifying means refers to, for example, the interpretation table (36A) according to the gaze state determination result and the speech state determination result stored in the storage means (36C), and the communication between the user and the anthropomorphic medium at that time Estimate or identify the state of communication, such as whether the user is speaking while looking at the anthropomorphic medium or whether the user is speaking while looking at the same direction as the anthropomorphic medium. Then, the control means (32, 38, 46, S11, S27) refers to, for example, the reaction table (36B) according to the communication state specified by the communication state specifying means, and operates the anthropomorphic medium (utterance and / or Or control).

  According to the first aspect, since the anthropomorphic medium can be controlled according to the determination results of both the user's line-of-sight state and speech state, the communication state between the user and the anthropomorphic medium at that time can be controlled. And optimal communication triggering action can be performed.

  According to a second aspect of the present invention, the utterance state determination means includes utterance target estimation means for estimating what the user's utterance target is based on the line of sight state that the user uttered, and the communication state identification means includes the utterance target estimation means The communication induction system according to claim 1, wherein one of a plurality of communication states is specified based on the determination result and the gaze state determination result.

  In the second aspect of the invention, it is possible to respond finely according to the situation such as whether the user has spoken to an anthropomorphic medium (stuffed animal) or to another subject. For example, if the user utters without looking at the anthropomorphic medium, the reaction to the utterance (reaction) is suspended as “possibility of utterance other than the anthropomorphic medium”, or the user is looking at the anthropomorphic medium If the user speaks, the user can react (react) to the utterance with voice and action as “the utterance is to myself”.

  In the third invention, the line-of-sight determination means repeatedly determines the state of the user's line of sight, the utterance determination means repeatedly determines the state of the utterance from the user, and the utterance target estimation means has at least the previous line of sight stored in the storage means. The communication induction system according to claim 2, wherein the communication inducing system estimates whether the user's utterance is directed to an anthropomorphic medium according to the state determination result and the utterance state determination result and the current gaze state determination result and the utterance state determination result. is there.

  In the third invention, in the utterance target estimation, for example, referring to the utterance target table (36D), based on the previous gaze state determination result, the utterance state determination result, the current gaze state determination result, and the utterance state determination result. Thus, it is estimated whether the user has spoken to an anthropomorphic medium (stuffed toy) or to another object. Therefore, the utterance target can be reliably estimated.

  The fourth invention is the communication induction system according to claim 3, wherein the utterance target estimation means further estimates the utterance pair in consideration of the length of the time interval between the previous time and the current time.

  In the fourth invention, in the utterance target estimation, in addition to the previous gaze state determination result, the utterance state determination result, the current gaze state determination result, and the utterance state determination result, based on the length of the time interval between the previous time and the current time Then, it is estimated whether the user has spoken to an anthropomorphic medium (stuffed animal) or to another object. Therefore, the utterance target can be estimated more accurately.

  In the fifth invention, the line-of-sight determination means repeatedly determines the state of the user's line of sight, the utterance determination means repeatedly determines the state of the utterance from the user, and the communication identification means determines the previous line-of-sight state stored in the storage means. The communication induction system according to claim 1, wherein the communication state between the user and the anthropomorphic medium is specified according to the result and the speech state determination result and the current gaze state determination result and the speech state determination result.

  In the fifth invention, since the previous determination result and the current determination result stored in the storage means are used, a temporal change in the communication state can be further detected, and a finer response can be made.

  According to the present invention, since the optimal communication inducing action can be performed on the anthropomorphic medium according to the user's line-of-sight state and speech state, communication from the user can be actively drawn out.

  The above object, other objects, features, and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

  The communication inducing system 10 of one embodiment of the present invention shown in FIG. 1 at least works on this user 12 to actively elicit communication from a subject with mild brain damage, such as a patient with dementia, or the user 12. One stuffed toy 14 is included. This stuffed toy 14 is an anthropomorphic medium. An anthropomorphic medium is a medium that can speak like a human or can operate like a human. Typically, stuffed animals, humanoids, communication robots, etc. are also sufficient as this anthropomorphic medium. Can function. However, two or more stuffed animals may be used.

  In the system 10 of this embodiment, the camera 16 that mainly captures the face of the user 12 is used to detect the direction and position of the line of sight of the user 12 in real time, and by detecting the presence or absence of the utterance of the user 12, etc. Communication with the stuffed animal 14 from the user 12 is induced by controlling the utterance and operation of the stuffed animal 14 in accordance with the direction and position of the line of sight (the state of the line of sight) and the presence or absence of the utterance (state of utterance).

  The line-of-sight direction or position of the user 12 is detected or determined in real time as the line-of-sight server 18 processes face image data or signals from the camera 16 as will be described in detail later.

  FIG. 2 is an illustrative view illustrating a state in which the embodiment of FIG. 1 is looked down on, and FIG. 3 is an illustrative view showing the state from the side. As can be seen from these figures, the patient or subject or user 12 sits on a chair in one of the spaces 10A such as a room, and the stuffed toy 14 is disposed in the other of the spaces 10A in front of the patient. ing.

  The camera 16 is installed so that the front face of the user 12 can be photographed from one corner of the space 10A.

  The angle at which the line of sight 12A of the user 12 is deviated from the vertical line vline parallel to the optical axis of the camera 16 is the turning angle α, and the line of sight 12A is deviated from the horizontal line hline parallel to the optical axis of the camera 16. Is calculated in real time by the line-of-sight server 18 of FIG. 1 as the elevation angle β. Each stuffed toy 14 identifies the direction or position of the line of sight 12A from the detected angles α and β, and different operations and / or utterances depending on the direction or position of the line of sight 12A and the state of the utterance of the user 12. By trying, the user 12 tries to induce communication.

  FIG. 4 shows the stuffed animal 14. The stuffed toy 14 includes a head 20 and a body 22 that supports the head 20. Left and right left arms 24 </ b> L and right arms 24 </ b> R are provided on the upper portion of the body 22, a mouth 26 is disposed on the front surface of the head 20, and an eyeball 28 is provided above the mouth 26. An ear 30 is attached to the upper side surface of the head 20.

  The torso 20 is supported by the torso 22 so as to be able to turn and rise, and the eyeball 28 is also held movably. The mouth 26 has a built-in speaker 48 (FIG. 5), and the ear 30 has a built-in microphone 50 (FIG. 5). If the microphones 50 are incorporated in both ears 30, respectively, the microphones 50 function as stereo microphones, whereby the position of the sound input to the stereo microphones can be specified as necessary.

  FIG. 5 is a block diagram showing the electrical configuration of the stuffed toy 14. As shown in FIG. 5, the stuffed toy 14 according to the embodiment includes a computer 32, and the computer 32 is an example of a communication path. It is coupled to the line-of-sight server 18 shown in FIG. Therefore, the computer 32 can receive data indicating the line-of-sight direction and position of the user 12 identified or detected by the line-of-sight server 18 through the bus 34 every moment. However, the communication path may be the bus 34, another type of communication path, wireless, or wired.

  A memory 36 is coupled to the computer 32 via a bus 34. A ROM or RAM (not shown) is incorporated in the memory 36. The ROM mainly stores a program expressed in the flowcharts (FIGS. 20 and 23) described later, and specifies a communication state or The interpretation table 36A as shown in FIG. 21 and FIG. 24 for interpretation, and the operation of the stuffed toy 14 and the speech (speech) corresponding to the state of the user's speech and the state of the line of sight are set correspondingly. A reaction table 36B as shown in FIGS. 25-26 is set in advance. The reaction table 36B is set by associating the operation of the stuffed toy 14 with the speech. The RAM, for example, a determination for sequentially storing the determination result of the utterance state such as the presence / absence of the utterance of the user 12 and the utterance target and the gaze state determination result such as whether or not the sight line of the user 12 is directed toward the stuffed animal 14. It can be used as a temporary storage memory for the result storage unit 36C, an utterance flag (not shown) in which 1 or 0 is set according to the utterance state of the user 12, and a working memory. The ROM further stores an utterance target table 36D shown in FIGS. 27 and 28 in advance. This utterance target estimation table 36 </ b> D is used to determine or estimate who the utterance made by the user 12 is targeted, that is, whether the utterance is directed to the stuffed animal 14.

  The motor control board 38 is composed of, for example, a DSP (Digital Signal Processor), and controls each axis motor of each arm and head of the stuffed toy 14 shown in FIG. That is, the motor control board 38 receives the control data from the computer 32 and controls the three angles for controlling the X, Y and Z axes so that the right arm 24R (FIG. 4) can be moved back and forth and left and right. The rotation angle of the motor 40R (collectively shown as “right arm motor” in FIG. 5) 40R is adjusted. Further, the motor control board 38 adjusts the rotation angle of three motors 40L of the left arm 24L (collectively shown as “left arm motor” in FIG. 5) 40L. The motor control board 38 also adjusts the rotation angle of three motors 42 (collectively shown as “head motors” in FIG. 5) that control the turning angle and the elevation angle of the head 20. The motor control board 38 also controls an eyeball motor 44 that moves the eyeball 28.

  The motors described above are stepping motors or pulse motors for simplifying the control, but may be direct current motors.

  The speaker 48 is provided with the synthesized voice data from the computer 32 via the voice input / output board 46, and accordingly, the speaker 48 outputs voice or voice according to the data. Then, the voice input from the microphone 50 is taken into the computer 32 via the voice input / output board 46.

  Similarly, the sensor input / output board 52 is also configured by a DSP, and takes in signals from each sensor and camera and gives them to the computer 32. However, since there is not much relation in the embodiment, the details of the sensors and the like will be described here. Description is omitted.

  In the embodiment shown in FIG. 1, the front of the user 12, that is, the periphery of the stuffed toy 14, is partitioned as shown in FIG. Whether the line of sight of the user 12 is directed to the stuffed toy 14 or whether the stuffed toy 14 and the user 12 are within a range where the stuffed toy 14 and the user 12 can jointly watch, but are directed to an object other than the stuffed toy, such as another anthropomorphic medium or person, This is because the stuffed toy 14 determines the utterances and actions that the stuffed toy 14 performs on the user 12 depending on whether it is outside the range where the joint gaze is possible, that is, outside the range.

  However, such sections may be defined more finely, and the utterances and actions performed by the stuffed toy 14 with respect to the user 12 may be determined for each fine section.

  In this embodiment, the stuffed animal 14 is described as having a control circuit as shown in FIG. 5 that autonomously controls its own operation and speech, but one or more computers for controlling the stuffed animal 14 are also described. May be provided separately from the stuffed toy 14.

  In such an embodiment, first, a method for estimating the line of sight of the user 12 will be described. In this embodiment, as will be described below, a unique method using one camera (monocular camera) is used to estimate or detect the direction of the line of sight of the user 12. However, as a method for detecting the line of sight 12A (FIGS. 2 and 3) of the user 12, it is needless to say that a conventional general method using two or more cameras may be employed. That is, in the present invention, it is necessary to estimate and detect the line of sight of the user 12, but the specific method has no significant meaning, and any known method may be used.

As shown in FIG. 1, in front of the user 12, for example, a CCD (Charge Coupled Device)
Alternatively, a camera 16 including a solid-state imaging device such as a CMOS (Complementary Metal-Oxide Semiconductor) sensor is installed, a face image signal from the camera 16 is taken into the line-of-sight server 18, and the line-of-sight server 18 performs image processing. The angles α and β of the line of sight 12A are estimated.

  As shown in FIG. 7, an image captured by the camera 16 is displayed as a moving image in real time on a captured image display area 56 of a display 54 (not shown in FIG. 1) provided attached to the line-of-sight server 18. Although not particularly limited, for example, a line segment extending in the line of sight from the eyebrows may be displayed on the captured image display area 56 as an index indicating the line of sight.

  The line-of-sight server 18 is a general computer and does not have a particularly unusual hardware configuration, so the hardware itself will not be described, but estimation of the line-of-sight direction and line-of-sight position is realized by software described below.

  The line-of-sight server 18 holds a plurality of observed textures in different frames for the same feature point in order to ensure the stability of the feature point tracking process. In the initial calibration process, the relative relationship between the face feature point and the eyeball center is obtained from the relationship between these feature points and the iris center. In the gaze estimation process, the eyeball center position is estimated from the feature point group obtained in the current frame based on the relationship obtained in the calibration process, and the gaze direction is determined from the position and the iris center position.

  As a premise of the operation of the gaze direction estimation process, first, for example, a face detection process is executed using a six-divided rectangular filter.

  The line-of-sight server 18 is not particularly limited. For example, when processing a video image obtained by continuously capturing a face, the screen is scanned with a rectangular filter having a horizontal width of the face and a vertical size of about a half thereof. The rectangle is divided into, for example, 3 × 2, and the average brightness of each divided region is calculated, and when the relative brightness relationship is satisfied, the center of the rectangle is set as a candidate for the eyebrows.

  When consecutive pixels become the eyebrow candidate, only the center candidate of the frame surrounding it is left as the eyebrow candidate. By performing template matching or the like by comparing the remaining eyebrow candidates with the standard pattern, the false eyebrow candidates are discarded from the eyebrow candidates obtained by the above-described procedure, and the true eyebrow space is extracted. This will be described in more detail below.

FIG. 8 is a conceptual diagram for explaining a filter for detecting an eyebrow candidate region. FIG. 8A shows the above described 3 × 2 rectangular filter (hereinafter referred to as “6-divided rectangular filter”). "
Called).

  The six-divided rectangular filter is a filter that extracts facial features such as (1) nose muscles are brighter than both eye regions and (2) eye regions are darker than the cheeks, and obtains the position between the eyebrows. For example, a rectangular frame of horizontal i pixels and vertical j pixels (i, j: natural number) is provided centering on one point (x, y). Then, as shown in FIG. 8A, this rectangular frame is divided into three equal parts horizontally and two equal parts vertically, and is divided into six blocks S1 to S6.

  When such a 6-divided rectangular filter is applied to both eye regions and cheeks of a face image, the result is as shown in FIG.

  However, although the 6-divided filter in FIG. 8 is an equally divided rectangular area to be written, this filter may be modified as shown in FIG.

  Considering that the nose muscle portion is usually narrower than the eye region, it is more desirable that the width w2 of the blocks S2 and S5 is narrower than the width w1 of the blocks S1, S3, S4 and S6. Preferably, the width w2 can be half of the width w1. FIG. 9 shows the configuration of a six-divided rectangular filter in such a case. Further, the vertical width h1 of the blocks S1, S2 and S3 and the vertical width h2 of the blocks S4, S5 and S6 are not necessarily the same.

  In the six-divided rectangular filter shown in FIG. 9, the average value “bar Si” (with a superscript “−”) of the pixel luminance is obtained for each block Si (1 ≦ i ≦ 6).

  Assuming that one eye and eyebrows exist in the block S1 and another eye and eyebrows exist in the block S3, the following relational expressions (1) and (2) hold.

  Therefore, a point satisfying these relationships is extracted as an eyebrow candidate (face candidate).

For the process of calculating the sum of pixels in a rectangular frame, a known document (P. Viola and M. Jones, “Rapid Object Detection using a Boosted Cascade of Simple Features,” Proc. Of IEEE
Conf. CVPR, 1, pp.511-518, 2001), it is possible to incorporate a high-speed calculation method using an integral image. By using an integral image, it can be executed at high speed regardless of the size of the filter. By applying this method to a multi-resolution image, face candidates can be extracted even when the size of the face on the image changes.

  For the eyebrow candidate (face candidate) obtained in this way, the true eyebrow position (true face region) can be specified by template matching with the standard pattern of both eyes.

Note that a face area can be determined by applying verification processing using a face model by a support vector machine (SVM) to the obtained face candidates. In order to avoid a reduction in recognition rate due to differences in hairstyles, presence or absence of wrinkles, and changes in facial expressions, for example, as shown in FIG. 10, modeling by SVM can be performed using an image region centered between the eyebrows. For the determination of the true face area by SVM, see: S. Kawato, N. Tetsutaniand K. Hosaka: “Scale-adaptive face detection and tracking in real time with ssr fi1ters and support vector machine”, IEICE. Trans. on Info. and Sys., E88-D, 12, pp. 2857-2863 (2005). Real-time face detection is possible by combining high-speed candidate extraction with a six-divided rectangular filter and processing by SVM.

  Subsequently, the position of the eyes, nose, and iris center is recognized by a known document, for example, “Kawado, Utsumi, Abe:“ Gaze estimation from a monocular camera based on four reference points and three calibration images ”. Understanding Symposium (MIRU2005), pp. 1337-1342 (2005) ”or“ Shinjiro Kawato, Shinji Tetsuya: Detection of nose position and real-time tracking: IEICE Technical Report IE2002-263, pp. 25-29 (2003) ”.

  As for the positions of both eyes, since the pattern of the eyebrows is searched for by detecting the face area in the previous section, the positions of the eyes can be roughly estimated by searching again for the dark areas on both sides of the eyebrows. However, it is necessary to extract the iris center more accurately in order to estimate the gaze direction. Here, for the peripheral region of the eye obtained above, iris edge candidates are extracted by Laplacian, and the Hough transform of the circle is applied to detect the projection position of the iris and the center of the iris.

  The nose position is extracted by utilizing the fact that the nose tip is a convex curved surface, so that it can be easily observed as a bright spot with respect to the surroundings, and the nose presence range can be limited from the positions of both eyes. In addition, the orientation of the approximate face can be estimated using the positions of both eyes and nose.

  FIG. 11 is a diagram illustrating an example of a face detection result. In the detected face, the iris center, nose tip and mouth are also detected. For example, nose tips, left and right eye corners and eyes, both ends of the mouth, and the center of the nasal cavity can be used as the feature points.

  In the gaze estimation, the gaze direction is given as a three-dimensional straight line connecting the eyeball center and the iris center.

FIG. 12 is a conceptual diagram illustrating a model for determining the line-of-sight direction. If the eyeball radius on the image is r and the distance between the center of the eyeball and the iris center on the image is d, the angle θ formed by the line-of-sight direction and the camera optical axis is expressed by the following equation (3).

  In order to estimate the line-of-sight direction using Equation (3), the eyeball radius on the image and the projection positions of the eyeball center and iris center are required. Here, as described above, the projection position of the iris center can be obtained by the method using the Hough transform. The eyeball diameter r on the image may be an anatomical model (standard human eyeball diameter) or may be obtained by calibration separately.

  FIG. 13 is a conceptual diagram illustrating the relationship between the iris center, the eyeball center, and the projection point after the user transitions from the state illustrated in FIG. 12 to a state in which the user gazes at the camera.

  In general, the projection position at the center of the eyeball cannot be observed directly from the image. However, considering the case where the user 12 gazes at the camera 16, as shown in FIG. 13, the three points of the camera, the iris center, and the eyeball center are aligned on a straight line. You can see that it is projected.

  Therefore, in the gaze estimation in this embodiment, the user captures an image frame sequence in which the posture of the face is changed while gazing at the camera, and extracts and tracks the iris position and the facial feature point from these image sequences. Thus, the relative geometric relationship between the eyeball center and the face feature point is estimated.

  As will be described in detail later, in the estimation of the line-of-sight direction in this embodiment, the estimation process of the relative relationship between the eyeball center and the face feature point and the projection position estimation of the eyeball center are performed.

  As an initial setting for estimating the line-of-sight direction, the line-of-sight server 18 executes calibration represented by the flowchart shown in FIG.

  First, as an image sequence for calibration, the user captures an image frame sequence in which the posture of the face is changed while gazing at the camera (step S102). FIG. 15 shows four image frames taken in the calibration in this way.

  Here, more generally, it is assumed that N (N ≧ 2) image rows are obtained. Assume that each image frame is a frame I1,.

  Next, face detection processing is performed on each obtained image frame sequence by the method described above (step S104), and then eye and nose detection processing is performed (step S106).

  Further, the line-of-sight server 18 extracts and tracks feature points (step S108). In addition to the above-mentioned method, the feature point extraction method is, for example, J: Shi and C. Tomasi: “Good features to track”, Proc. CVPR94, pp. 593-600 (1994). It is also possible to use the method proposed in.

  Here, it is assumed that feature points pj (j = 1,..., M) of M (M ≧ 4) points can be detected and tracked in each image frame Ii (i = 1,..., N). The two-dimensional observation position of the feature point pj in the image frame Ii is expressed as xj (i) (bold) = [xj (i), yj (i)] t (i = 1,..., N, j = 1,..., M) And the two-dimensional observation positions of the iris centers of both eyes are xr (i) (bold) = [xr (i), yr (i)] t, xl (i) (bold) = [xl (i), yl ( i)] t (i = 1,..., N). Here, the matrix W is defined as follows.

  By the factorization method, a matrix W (measurement matrix) in which two-dimensional observation positions in each frame of feature points are vertically arranged can be decomposed as follows.

  Here, the matrix M (referred to as “photographing posture matrix”) includes only information regarding the posture of the camera, and the matrix S (referred to as “relative positional relationship matrix”) includes only information regarding the shape of the observation object. Therefore, the relative relationship between the three-dimensional position between the face feature point and the eyeball center is obtained as a matrix S (step S110). That is, assuming orthographic projection, each element of the matrix M is a unit vector that represents the posture of the camera in the image frame, and each of them is 1 and under the constraint that they are orthogonal to each other, It is known that the matrix W can be uniquely decomposed into a product of the matrix M and the matrix S by singular value decomposition. In addition, about the point which decomposes | disassembles such a measurement matrix W into the matrix showing the information of the motion of a camera and the shape information of a target object by factorization, literature: Kade, Paulman, Morita: factorization Restoration of object shape and camera motion by the method ", disclosed in IEICE Transactions D-II, J76-D-II, 8, pp. 1497-1505 (1993).

  FIG. 16 is a flowchart of real-time gaze direction estimation processing.

  Next, a procedure for estimating the line-of-sight direction using the results obtained above will be described.

  First, when an image frame is acquired from the camera 16 (step S200), face detection and eye / nose detection are performed in the same manner as in calibration (step S202), and feature points in the acquired image frame are extracted. (Step S204).

  Assume that an image frame Ik is obtained. Here, m points pj (j = j1,..., Jm) among feature points other than the center of the eyeball are observed at xj (k) (bold) = [xj (k), yj (k)] t, respectively. Suppose. At this time, for the observed feature points, by performing template matching using a template near the feature points as described above, the feature points identified during calibration and the feature points observed in the current image frame And the feature points in the current image frame are specified (step S206).

  As described above, the template for specifying the feature point is not limited to the template at the time of calibration. For example, the template having a predetermined size in the vicinity of the detected feature point for the predetermined number of recent image frames is used. A predetermined number of images in the region may be held, and the feature points having the highest degree of matching may be specified as a result of matching the predetermined number of templates.

  Two-dimensional observation position xj (k) (bold) = [xj (k), yj (k)] t of face feature point pj and three-dimensional position sj (bold) = [Xj, Yj, Zj] obtained from calibration If the attention is paid to the observed m feature points among the M feature points during t (j = 1,..., M), the following relationship is obtained.

  However, the matrix P (k) is a 2 × 3 matrix. The matrix S (k) of the second term on the right side is a partial matrix consisting of only elements corresponding to the observed feature points in the matrix S. As described above, it is assumed that the camera and the face are sufficiently separated from each other and an orthogonal projection is assumed. Here, if four or more feature points are observed, the matrix P (k) can be calculated as follows (step S208).

  The projection positions xr (i) (bold) and xl (i) (bold) at the center of the eyeball in the image frame Ik can be calculated as follows using the matrix P (k) (step S210).

  Therefore, by using the iris center projection position extracted as the feature point in the image frame Ik and the eyeball center projection position, the line of sight can be estimated (step S212).

  By decomposing the matrix P by QR decomposition, the face posture R can be calculated as follows.

  However, r1 and r2 are 1 × 3 vectors, respectively. Such detection of face posture R is described in literature: L.L. Quan: “Self-calibration of an affine camera from multiple views”, Int’l Journal of Computer Vision, 19, pp. 93-105 (1996).

  If it is determined that the tracking has been completed by an instruction from the user or the like (step S214), the process is terminated, and if the termination is not instructed, the process returns to step S202.

  In order to confirm the effectiveness of the gaze direction estimation apparatus described above, the results of experiments using real images will be described below.

  The camera was an Elmo PTC-400C, and was installed at a position of about 150 cm from the subject.

  First, the center of the eyeball and the facial feature point were calibrated using an image sequence of 50 frames. Examples of calibration image frame sequences and extracted feature points are as shown in FIG.

  The time required for capturing the calibration image frame sequence was about 3 seconds. (+ Mark is the extracted iris center (eyeball center)), x mark is the tracked facial feature point).

  Next, gaze estimation was performed using the face model (matrix S) obtained by calibration. Here, the subject changed the position and orientation of the face while gazing at the upper right, upper and lower left directions.

  17 to 19 show the line-of-sight estimation results. FIG. 17 shows a state of right upper gaze, FIG. 18 shows a state of upper gaze, and FIG. 19 shows a state of lower left gaze. Here, the gaze direction is an average value of the gaze directions calculated for both eyes. From the results, it was possible to estimate the gaze direction regardless of changes in the face position and orientation.

  As described above, in the gaze direction estimation method of this embodiment, the gaze direction is estimated by detecting and tracking the face feature points based on the observation of the monocular camera. In other words, as a calibration, the relationship between the eyeball center and the face feature point is modeled by using the iris position and the face feature point obtained from the image sequence in which only the face direction is different with the line of sight facing the camera direction (matrix). S is specified), and then the angles α and β in the line-of-sight direction are determined from the relationship between the eyeball center position and the iris position in the input image estimated based on the relationship.

  The flowchart shown in FIG. 20 is executed to control the speech and operation of the stuffed toy 14. The flowchart shown in FIG. 20 is the same as the flowchart shown in FIG. It shall be executed with a period of seconds.

  In the first step S1 in FIG. 20, step S1a for detecting the utterance of the user 12 and step S1b for determining the line of sight of the user 12 are processed in parallel.

  In step S1a, the computer 32 (FIG. 5) of the stuffed toy 14 calculates the power of voice input from the microphone 50 (FIG. 5) to determine whether the user 12 has spoken, that is, whether or not the user 12 has spoken. And the result (presence / absence of utterance) is stored in the determination result storage unit 36C of the memory 36 as data indicating the state of the utterance.

  However, in order to detect whether the user 12 has spoken, there are other methods such as a method using a super-directional microphone, a method of calculating a sound source and a vector, a method of detecting a harmonic component in the power spectrum of speech, etc. is there. In the present invention, any method may be used as a method for detecting the presence or absence of the user's utterance.

  In step S1b, the data of the line-of-sight angles α and β estimated by the line-of-sight server 18 as described above is provided to the computer 32 through the bus 34 (FIG. 5) of the stuffed animal 14. The computer 32 calculates the spatial coordinates (x, y, z) in the space 10A of the line of sight 12A (FIGS. 2 and 3) from the angle data as an absolute position.

  On the other hand, each of the stuffed toys 14 is fixedly arranged as shown in FIG. Accordingly, both the coordinates of the “range” shown in FIG. 6 and the coordinates of the position where the stuffed animal 14 exists in the space 10A where the stuffed animal 14 exists are already calculated, for example, the memory 36 (FIG. 5). Is remembered.

  In the next step S3, the computer 32 compares the spatial coordinates of the line of sight previously calculated in step S1b with the coordinates stored in advance in the memory 36, and the direction of the line of sight of the user 12 at that time, That is, whether the relative position of the line of sight is directed to the stuffed toy 14 itself, or to something other than the stuffed toy 14 but within the range (FIG. 6), such as anthropomorphic media or humans, or It is determined whether it is directed outside the range shown in FIG. The determination result of the gaze state determined as described above is stored in the storage unit 36C of the memory 36 as gaze state data.

  In subsequent step S5, the computer 32 determines whether the utterance of the user 12 at that time is directed to the stuffed toy 14 itself based on the above-described line-of-sight state of the user 12 at the timing when the user 12 uttered, or a different object within the range. Estimate whether it was aimed at or not. Not knowing is presumed as such when the user's line of sight is directed outside the range of FIG. 6 or when the line of sight is unstable and unstable. The utterance target estimated in the utterance target estimation step is also recorded in the storage unit 36C as the utterance state data.

  Here, the utterance target estimation operation in step S5 will be described more specifically. In the utterance target estimation, the utterance target table 36 </ b> D (FIG. 5) shown in FIGS. 27 and 28 is referred to and it is estimated whether the target uttered by the user 12 is himself, that is, the stuffed toy 14. In the utterance target table 36D, in addition to the determination results of the previous utterance, the previous sight line, the current utterance, and the current sight line (accumulated in the determination result storage unit 36C illustrated in FIG. 5), the previous utterance / gaze and the current utterance. / The length of the time interval from the line of sight is used as an estimation factor. For this time interval, for example, 1 second or longer is registered as “long” and less than 1 second is registered as “short”. However, in cases 1-4 and 9-14, each determination result and this time interval are estimated as elements, but in cases 5-8, 15-16, and 19-22, the determination result of speech and line of sight is simply Only the utterance target is estimated. In cases 17-18 and 23-28, since there is no utterance in the previous time and this time, this utterance target step is not relevant.

  For example, as shown in cases 1 and 2 in FIG. 27, the previous utterance determination result was “◯” and the previous gaze state determination result was “x”, but the current utterance determination result is “x”. When the line-of-sight state determination result is “◯”, it is highly possible that the previous utterance is yourself (stuffed toy 14) in the case of “short” (case 1) due to the short time interval. In the case of “long” (case 2), it is presumed that the (previous utterance) was probably not the subject but merely the subject of the previous gaze.

  For example, as shown in cases 3 and 4 in FIG. 27, the previous utterance determination result was “X” and the previous gaze state determination result was “O”, but the current utterance determination result is “O”. When the gaze state determination result is “×”, it is estimated that there is a high possibility that the current utterance is self (stuffed toy 14) in the case of “short” (case 3) due to the short time interval. However, in the case of “long” (case 4), it is presumed that it was probably the subject of gaze this time, not himself.

  On the other hand, in cases 5 and 6, the line of sight was not directed at me (“×”) both in the previous time and this time, so it is estimated that the utterance target is not myself regardless of the length of the time interval. is doing. Similarly, the cases 7 and 8, the cases 19 and 20, and the cases 21 and 22 that use only the determination results of the previous time and the current time also produce the same estimation result regardless of the time interval.

  By using such a table 36D, it is possible to roughly accurately estimate the utterance target of state data in which the utterance and the presence or absence of the line of sight do not match.

  Subsequently, in step S7, the computer 32 determines whether or not the user's line-of-sight state determined in step S3 (whether the line of sight of the user 12 is directed toward the stuffed toy 14 itself or other stuffed toy but not within the range). Whether the user's utterance is directed (directed toward the stuffed toy 14 or to a different object within the range, estimated in step S5) The communication state between the user 12 and the stuffed toy 14 at that time is estimated or specified.

  Specifically, as shown in FIG. 21, when the utterance target is “self”, “within other range”, “out of range”, or “no utterance”, the line-of-sight state is “self”, “self When it is either “inside of range” or “unknown”, it is estimated what the communication state of the two is. However, it is determined as “unknown” when the line-of-sight direction is “out of range” as shown in FIG. For example, when the utterance target and the line-of-sight state are both “self”, it can be estimated that the user 12 is speaking with the stuffed animal 14 in his / her eyes. Since all communication states that can be estimated or specified by the interpretation table 36A are described in detail in FIG. 21, refer to FIG. 21 for details.

  In step S9, the computer 32 determines a stuffed behavior (speech and / or action) effective for inducing communication from the user to the stuffed animal according to the communication state specified in step S7. The behavior (speech and / or movement) of the stuffed animal is specifically shown in FIG. 22. Basically, in the communication state in which the user is speaking with respect to the stuffed animal, the computer 32 has the stuffed animal 14. Set an action that responds to the user by voice (utterance). However, when the user is not speaking or talking to the stuffed animal, the behavior of the stuffed animal is not set to reply by voice. Further, in a communication state in which the user looks at the stuffed animal, the computer 32 determines an action such that the stuffed animal 14 expresses a reaction to the user. Then, in such a communication state in which the user does not look at the stuffed toy but the user's line of sight is “within range”, the computer 32 acts as the action of the stuffed toy 14 such as jointly gazing at the user's line of sight. Set. However, when the user's line of sight is “out of range”, no reaction operation is set for the stuffed animal.

  Specifically, FIG. 22 shows a table of communication states and stuffed behaviors, but these are merely examples, and it goes without saying that they can be changed as appropriate. However, it should be understood that the communication state number in FIG. 22 corresponds to the communication state number in FIG.

  In step S <b> 11, the computer 32 outputs necessary voice data and motor control data to the voice input / output board 46 and the motor control board 38 so that the stuffed animal 14 actually produces the behavior of the stuffed animal determined in step S <b> 9. However, the timing when the stuffed toy 14 speaks is after the user 12 has finished speaking, and therefore, the “speaking flag” described above is referred to. That is, since the utterance flag is “1” when the user is uttering, the stuffed animal 14 is uttered after each becomes “0”. However, the operation | movement of the stuffed toy 14 may be performed during a user's speech, and may be performed after a user's speech is complete | finished.

  In this manner, the stuffed toy computer 32 estimates the communication state with respect to the user's stuffed animal based on the determination result of the user's speech state and the determination result of the user's line-of-sight state. Control stuffed behavior, ie speech and movement, to further enhance or induce communication.

  FIG. 23 is a flowchart showing the operation of another embodiment of the present invention. In the previous example, only the current gaze state and the current utterance state were referred to identify the communication state, but the previous gaze state and the last utterance state were also considered and the communication state was determined. In this respect, this embodiment differs from the previous embodiment.

  In the first step S21 in FIG. 23, step S21a for detecting the utterance of the user 12 and step S21b for determining the line of sight of the user 12 are processed in parallel.

  In step S21a, as in step S1a of FIG. 20, the presence / absence of the utterance of the user 12 is detected, and the result (presence / absence of utterance) is stored in the determination result storage unit 36C of the memory 36 as data indicating the utterance state. . Again, the method for detecting whether the user 12 has spoken may be any method.

  In step S21b, the data of the line-of-sight angles α and β estimated by the line-of-sight server 18 as described above are provided to the computer 32 through the bus 34 (FIG. 5) of the stuffed toy 14. The computer 32 calculates the spatial coordinates (x, y, z) in the space 10A of the line of sight 12A (FIGS. 2 and 3) from the angle data as an absolute position.

  On the other hand, each of the stuffed toys 14 is fixedly arranged as shown in FIG. Accordingly, both the coordinates of the “range” shown in FIG. 6 and the coordinates of the position where the stuffed animal 14 exists in the space 10A where the stuffed animal 14 exists are already calculated, for example, the memory 36 (FIG. 5). Is remembered.

  Therefore, in step S1b, the computer 32 compares the previously calculated spatial coordinates of the line of sight with the coordinates stored in advance in the memory 36, and the direction of the line of sight of the user 12, that is, the relative line of sight. It is determined whether the position is directed toward the stuffed toy 14 itself or whether a certain point is directed toward other than the stuffed toy 14. The determination result of the gaze state determined as described above is stored in the storage unit 36C of the memory 36 as gaze state data.

  In subsequent step S23, the computer 32 determines the user 12 based on the determination result of the previous speech state and the determination result of the previous gaze state, the determination result of the current speech state and the determination result of the current gaze state. Estimate or specify the communication state between the doll and the stuffed toy 14. For example, if the user 12 was talking to the stuffed animal 14 last time, but this time talking to another subject, the communication between the stuffed animal and the user is not completely interrupted. It is necessary to pay close attention to the subject and the user who are talking to ".

  A specific example of the communication state estimated or specified based on the previous determination result and the current determination result is shown in FIG.

  For example, as shown in case 1 of FIG. 24, the previous utterance determination result was “×” and the previous gaze state determination result was “◯”, but both the current utterance determination result and the gaze state determination result are “ ", The communication state is interpreted as" the user talked to the stuffed animal while keeping an eye on the stuffed animal ". However, the utterance determination result “x” here means that the user did not utter at that time. If the line-of-sight state determination result is “◯”, it means that the user's line of sight is directed toward the stuffed animal at that time.

  In Case 2, the previous utterance determination result was “◯” and the previous gaze state determination result was “x”, but both the current utterance determination result and the gaze state determination result are “O”. In this case, the communication state is interpreted as “the user looks at the stuffed animal while speaking”. However, if the utterance determination result is “○”, it means that the user spoke at that time, and if the sight line determination result is “×”, then the user's line of sight is directed toward the stuffed animal. It means that it was not done.

  As in Case 3, when the previous utterance determination result is “×” and the previous gaze state determination result is “×”, both the current utterance determination result and the gaze state determination result are “O”. The communication state is interpreted as “the user talked to the stuffed animal at the same time.”

  The state shown in Case 4 is a case where the previous utterance determination result and the gaze state determination result are both “O”, and the current utterance determination result and the gaze state determination result are both “O”. The communication state can be interpreted as “the user is talking to the stuffed toy while watching the stuffed toy (status preservation)”.

  In the following, detailed description of each individual communication state is omitted, so refer to FIG. 24 as necessary.

  In step S25, the computer 32 determines a stuffed behavior (speech and / or action) effective for inducing communication from the user to the stuffed animal according to the communication state specified in step S23. The stuffed behavior (speech and / or movement) is specifically shown in FIGS. 25-26. Basically, the communication behavior is determined in response to the temporal change of the utterance state and the gaze state. For example, if the utterance line of sight was X in the previous time, but this time also becomes ◯, it is estimated that the communication state 3 is “the user is talking to the stuffed animal at the same time”. The action performed by the stuffed toy in response to such a state change causes the action shown in the communication state 3 of FIG. 25 to be executed. Specifically, the voice is “a little surprised reply” and the action is “shake the neck” as if he noticed the intention to communicate with the user. By such an action of the stuffed toy 14, the user's willingness to communicate with the stuffed toy can be continued.

  Note that FIG. 25-26 shows a table of communication states and stuffed behaviors, but these are merely examples, and it goes without saying that they can be changed as appropriate.

  In step S27, the computer 32 outputs necessary voice data and motor control data to the voice input / output board 46 and the motor control board 38 so that the stuffed animal 14 actually produces the behavior of the stuffed animal determined in step S25. However, as in the previous embodiment, it is necessary to consider that the actual utterance timing is set to a timing that does not disturb the utterance of the user 12.

  In this way, the stuffed toy computer 32 estimates the communication state for the user's stuffed animal based on the determination result of the user's speech state and the determination result of the user's gaze state for the previous time and this time, and from the communication state The behavior of the stuffed animal, i.e., speech and movement, is controlled so as to further enhance or induce the communication of the user with the stuffed animal.

  However, it is necessary to use both the previous determination result and the current determination result because only the determination result of the current utterance and the determination result of the current gaze can be used to specify or estimate the utterance target and the gaze state. There is no. In this case, each determination means does not need to repeatedly determine the utterance state and the line-of-sight state at regular intervals, and it may be determined as needed.

It is an illustration figure which shows the concept of the communication induction system of one Example of this invention. It is an illustration figure which shows the planar positional relationship of a user and a stuffed toy in FIG. 1 Example, and a user's gaze angle. FIG. 3 is an illustrative view showing a side positional relationship between a user and a stuffed toy and a user's line-of-sight angle in the embodiment in FIG. 1; It is an illustration figure which shows an example of the stuffed toy used in FIG. 1 Example. It is a block diagram which shows an example of the control circuit of the stuffed toy in FIG. 1 Example. It is an illustration figure which shows an example of the range which determines the state of a user's eyes | visual_axis in FIG. 1 Example. It is an illustration figure which shows an example of the user's face image currently displayed on the display of the gaze server in FIG. 1 Example. FIG. 8 is a conceptual diagram for explaining a filter for detecting an eyebrow candidate region. FIG. 9 is a conceptual diagram showing another configuration of the 6-divided rectangular filter. FIG. 10 is an illustrative view for explaining modeling by SVM using an image area centered on the eyebrows. FIG. 11 is an illustrative view showing an example of a face detection result. FIG. 12 is a conceptual diagram illustrating a model for determining the line-of-sight direction. FIG. 13 is a conceptual diagram showing the relationship between the iris center, the eyeball center, and the projection point after the user has shifted to a state of gazing at the camera. FIG. 14 is a flowchart showing an initial setting processing operation by the line-of-sight server. FIG. 15 is an illustrative view showing four image frames taken in the calibration. FIG. 16 is a flowchart showing the processing operation of the real-time gaze detection executed by the gaze server. FIG. 17 is an illustrative view showing a gaze estimation result in a state of gaze at the upper right. FIG. 18 is an illustrative view showing a line-of-sight estimation result in an upward gaze state. FIG. 19 is a diagram illustrating a line-of-sight estimation result in a state of lower left direction gaze. FIG. 20 is a flowchart showing the operation of the stuffed computer in the embodiment of FIG. FIG. 21 is a table showing an example of a user communication state estimation or interpretation table for the stuffed toy in FIG. 1 embodiment. FIG. 22 is a table showing an example of a reaction table that defines the behavior of the stuffed animal according to the communication state in the FIG. 1 embodiment. FIG. 23 is a flowchart showing the operation of the stuffed toy computer in another embodiment. FIG. 24 is a table showing an example of a user communication state estimation or interpretation table for the stuffed toy in the embodiment of FIG. FIG. 25 is a table showing an example of a reaction table that defines the behavior of the stuffed animal according to the communication state in the embodiment of FIG. FIG. 26 is a table showing a continuation of FIG. FIG. 27 is a table showing an example of the utterance target table in the embodiment of FIG. FIG. 28 is a table showing a continuation of FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Communication induction system 14 ... Stuffed toy 16 ... Camera 18 ... Line-of-sight server 32 ... Computer 36 ... Memory 50 ... Microphone

Claims (5)

A communication inducing system for inducing communication from the user by an anthropomorphic medium arranged at a position visible to the user,
Line-of-sight determining means for determining a state of the line of sight of the user,
Utterance judging means for judging the state of utterance from the user,
A storage unit for storing a gaze state determination result by the gaze position determination unit and a utterance state determination result by the utterance determination unit;
Communication state specifying means for specifying a communication state between the user and the anthropomorphic medium according to a gaze state determination result and an utterance state determination result stored in the storage means; and according to the communication state specified by the communication state specifying means A communication inducing system comprising control means for controlling voice and operation of the anthropomorphic medium.   The utterance state determination means includes utterance target estimation means for estimating what the user's utterance target is based on the line-of-sight state that the user has spoken, and the communication state identification means includes the utterance target estimation means The communication induction system according to claim 1, wherein one of a plurality of communication states is specified based on the determination result and the gaze state determination result. The line-of-sight determination means repeatedly determines the state of the user's line of sight, the utterance determination means repeatedly determines the state of utterance from the user,
The utterance target estimation unit is configured to determine whether the user's utterance is the anthropomorphic medium according to at least the previous gaze state determination result and the utterance state determination result and the current gaze state determination result and the utterance state determination result stored in the storage unit. The communication inducing system according to claim 2, wherein the communication inducing system estimates whether or not the target is directed to.   The communication induction system according to claim 3, wherein the utterance target estimation unit further estimates an utterance pair in consideration of the length of the time interval between the previous time and the current time. The line-of-sight determination means repeatedly determines the state of the user's line of sight, the utterance determination means repeatedly determines the state of utterance from the user,
The communication specifying means is a communication state between the user and the anthropomorphic medium according to a previous gaze state determination result and speech state determination result and a current gaze state determination result and speech state determination result stored in the storage unit. The communication inducing system according to claim 1, wherein: