JP2005322077A6 - Head posture estimation apparatus, head posture estimation method, and head posture estimation processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a head posture estimation device and a head posture estimation method which can be easily attached and detached and can accurately estimate the posture of a user's head without giving a third party a sense of incongruity. And a head posture estimation processing program.
A sensor unit 10 including an infrared camera 1, an infrared illuminator 2, a wide-angle camera 3, a microphone 4, and a communication unit 5 is mounted on a user's chest, and the infrared camera 1 includes a user's head and torso. After capturing an infrared image, the data processing unit 7 extracts a human region including the user's head and torso from the captured infrared image, and then displays a boundary line representing the boundary between the head and the torso in the human region. At the same time, the nose tip point having the maximum distance from the middle point of the boundary line is extracted, and the head posture is estimated based on the position of the nose tip point.
[Selection] Figure 1

Description

  The present invention relates to a head posture estimation apparatus, a head posture estimation method, and a head posture estimation processing program that estimate the posture of a human head.

  In recent years, various sensors can be attached to humans by downsizing computers, cameras, sensors, and the like. Therefore, it is considered that when the user wears a computer or the like, it is possible to provide support in daily life such as presenting information to the user wherever the user is. Such research is called “wearable computing”, and various researches have been conducted.

  In addition, research aimed at recording and analyzing user behavior by using wearable devices has been conducted. In such research, it is important to automatically tag the obtained information. In addition, it is necessary to record the behavior of a person and detect a target of interest. As a method for recording human behavior, various methods such as voice information, video information, body movement information such as position and gesture, and physiological information such as heartbeat, sweating and breathing can be considered. In order to obtain such information, a microphone, a camera, a gyro sensor, a GPS (global positioning system), a skin sensor, or the like is used.

Among the above information, video information, especially video from the user's viewpoint, includes information on the user's attention target and user's head movement in addition to the surrounding situation, so record and analyze behavior It is considered important to do so. Therefore, in many studies, a head mounted camera is used to obtain a user's visual field image (see Non-Patent Document 1).
Yasuyuki Kado et al., “Recording Cooperative Interaction by Multiple Sensors”, Interaction 2003, 2003, p. 255-p. 262

  However, it is troublesome for the user to attach and detach the head mounted camera, and the user gets tired by attaching the head mounted camera to the head. It is difficult to wear a head-mounted camera on a daily basis because there are various problems such as visually discomforting humans. In addition, since it is difficult to fix the head-mounted camera so that it is accurately aligned with the user's line-of-sight direction, the head movement causes a shift in the line-of-sight direction with the head-mounted camera, and the user's head posture is accurately estimated. It is difficult to do.

  An object of the present invention is to provide a head posture estimation device, a head that can be easily attached and detached and can accurately estimate the posture of a user's head without giving a third-party a sense of discomfort in the field of view. To provide a posture estimation method and a head posture estimation processing program.

  A head posture estimation apparatus according to the present invention is mounted on a human chest and observes the head of the human, and estimates the posture of the head from information observed by the head observation means. And a head posture estimation means.

  In the head posture estimation apparatus according to the present invention, the human head is observed by the head observation means attached to the human chest, and the head posture is estimated from the observed information. It is possible to estimate the posture of the head without attaching observation means to the user's head like a camera, and it is easy to attach and detach and use without giving a third party a sense of discomfort. The head posture can be accurately estimated.

  The head posture estimation apparatus is fixed integrally with the head observation unit, and is based on a front photographing unit that captures an image in front of the human and a head posture estimated by the head posture estimation unit. It is preferable that the camera further includes a line-of-sight image extracting unit that extracts an image in the line-of-sight direction of the human from the image photographed by the front photographing unit.

  In this case, since the image in the human gaze direction is extracted from the image photographed by the front photographing means based on the estimated head posture, the image in the user's gaze direction, that is, what the user is paying attention to. It can be detected accurately. Further, since the front photographing means is fixed integrally with the head observing means, the positional relationship between the front photographing means and the head observing means does not change, and the user's attention is detected more accurately. be able to.

  The head observing means is mounted on a human chest and includes a head photographing means for photographing an image including the human head and torso, and the head posture estimating means is photographed by the head photographing means. A region extracting means for extracting a human region including a human head and torso from the image obtained, and determining a boundary line representing a boundary between the head and the torso in the human region; From the region divided as the head by the boundary line, the tip point extraction means for extracting the tip point having the maximum distance from the midpoint of the boundary line, and the head posture is estimated based on the position of the tip point It is preferable that the estimation means to include.

  In this case, an image including the human head and torso is photographed by the head photographing means, and a person region including the human head and torso is extracted from the photographed image, and the head in the extracted person region A boundary line representing the boundary between the body and the torso is determined, the midpoint of the boundary line is determined, and the tip point with the maximum distance from the midpoint of the boundary line is the head of the person area by the boundary line Since the posture of the head is estimated based on the position of the extracted tip point extracted from the segmented region, the posture of the user's head can be accurately estimated. In addition, since the head imaging means is mounted on the human chest and does not need to be mounted on the user's head like a head-mounted camera, it can be easily attached and detached without giving a sense of incongruity to a third party. And can be used.

  The head posture estimation apparatus further includes infrared illumination means that illuminates the head and torso of a person, which is attached to a human chest and is photographed by the head photographing means, using infrared rays, and the head photographing The means includes infrared photographing means for photographing an infrared image including a human head and torso illuminated by infrared light by the infrared illuminating means, and the region extracting means is configured to extract a human image from the infrared image photographed by the infrared photographing means. It is preferable to extract a person region including the head and torso of the person.

  In this case, since the person area including the human head and torso can be extracted from the infrared image obtained by photographing the head and torso of the person illuminated using infrared rays, the person area and other background areas Can be clearly distinguished, and the person region can be extracted with high accuracy.

  It is preferable that the estimation means estimates a horizontal angle and a vertical angle of the head centering on the midpoint of the boundary line from the position of the tip point. In this case, since the horizontal angle and the vertical angle of the head centering on the center point of the boundary line are estimated from the position of the tip point, the posture of the user's head can be estimated accurately.

  The head posture estimation device is fixed integrally with the head photographing means, and includes a front photographing means for photographing an image in the front direction of a person wearing the head photographing means, and a head estimated by the estimating means. It is preferable to further include an image extracting unit that extracts an image in the line of sight of a human wearing the head photographing unit from the image photographed by the front photographing unit.

  In this case, since an image in the direction of the line of sight of the person wearing the head imaging unit is extracted from an image captured by the frontal imaging unit based on the estimated head posture, an image in the direction of the user's line of sight, that is, the user Can be accurately detected. In addition, since the front photographing unit is fixed integrally with the head photographing unit, the positional relationship between the front photographing unit and the head photographing unit does not change, and the user's attention can be detected more accurately. Can do.

  The head posture estimation apparatus further includes a calculation unit that calculates a posture change amount of the head photographing unit and the front photographing unit with respect to a human based on the boundary line, and the image extraction unit is calculated by the calculation unit. The head photographing means is mounted from the image photographed by the front photographing means based on the posture change amount of the head photographing means and the front photographing means and the posture of the head estimated by the estimating means. It is preferable to extract an image in the human gaze direction.

  In this case, the posture change amount of the head photographing unit and the front photographing unit with respect to the human being calculated based on the boundary line, and the head estimated by considering the calculated posture variation amount of the head photographing unit and the front photographing unit Since the image in the human gaze direction is extracted based on the posture of the head, the image in the human gaze direction based on the posture of the head corrected for the relative posture change between the head photographing means and the front photographing means and the human being. Can be extracted, and an image in the human gaze direction can be extracted with higher accuracy.

  The head posture estimation method according to the present invention uses a computer functioning as an area extraction unit, a boundary line determination unit, a tip point extraction unit, and an estimation unit, and shoots an image including a human head and a torso. A head posture estimation method for estimating the posture of a human head wearing a means on a chest, wherein the region extraction means includes a human head and a torso from an image photographed by the head photographing means Extracting a person area; and wherein the boundary line determining means determines a boundary line representing a boundary between a head and a torso in the person area, determining a midpoint of the boundary line, and the tip A point extracting unit extracting a tip point having a maximum distance from a middle point of the boundary line from an area of the person region divided as the head by the boundary line; and the estimating unit, Tip position The is intended to include a step of estimating the posture of the head based.

  The head posture estimation processing program according to the present invention is a head posture estimation process for estimating the posture of a human head having a head photographing means for photographing an image including the human head and torso on the chest. A region extracting means for extracting a person region including a human head and a torso from an image photographed by the head photographing means; and a boundary representing a boundary between the head and the torso in the person region A boundary line determining means for determining a middle point of the boundary line, and a distance from the middle point of the boundary line to a maximum from a region of the person region divided as a head by the boundary line The computer functions as tip point extraction means for extracting the tip point, and estimation means for estimating the posture of the head based on the position of the tip point.

  According to the present invention, the human head is observed by the head observing means attached to the human chest, and the posture of the head is estimated from the observed information. The posture of the head can be estimated without wearing an observation means on the head, and the posture of the user's head can be easily attached / detached and used without giving a sense of discomfort to a third party. Can be estimated accurately.

  Hereinafter, a head posture estimation apparatus according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram illustrating a configuration of a head posture estimation apparatus according to an embodiment of the present invention, and FIG. 2 illustrates a state where a sensor unit of the head posture estimation apparatus illustrated in FIG. 1 is attached to a person. FIG.

  The head posture estimation apparatus shown in FIG. 1 includes an infrared camera 1, an infrared illuminator 2, a wide-angle camera 3, a microphone 4, communication units 5 and 6, a data processing unit 7, and a data recording unit 8. The infrared camera 1, infrared illuminator 2, wide-angle camera 3, microphone 4, and communication unit 5 are integrally configured as a sensor unit 10 and are attached to the chest of the user (human) as shown in FIG. 2. As a method for mounting the sensor unit 10, various methods such as attachment to clothes using a clip or the like, or hanging from a neck using a pendant method can be used. In addition, the infrared camera 1 places the user's head (at least near the apex of the nose and the torso) and the torso (at least near the boundary with the head, for example, both shoulders). Although it is preferable to attach it near the center position of the bust line of the human body so as to be surely photographed, if it is a position where the infrared camera 1 can photograph the user's head and torso, it is attached to other positions around the chest You may make it do.

  The infrared camera 1 is composed of a CCD camera or the like having sufficient sensitivity in the infrared region, and is mounted upward on the chest of the user. The infrared illuminator 2 is composed of an infrared LED or the like, is configured integrally with the infrared camera 1, and the illumination intensity is adjusted so as to illuminate only a near-distance area using near infrared rays. Illuminate only the torso. The infrared camera 1 is provided with an infrared filter that transmits only near-infrared rays out of infrared rays. As the infrared filter, for example, a plastic IR path manufactured by Edmond that does not transmit visible light but transmits near-infrared light. A filter can be used. The infrared camera 1 captures an infrared image including the user's head and torso illuminated by infrared rays from the infrared illuminator 2 and outputs infrared image data to the communication unit 5.

  The wide-angle camera 3 is fixed integrally with the infrared camera 1 in a state in which the shooting direction of the infrared camera 1 and the shooting direction of the wide-angle camera 3 are adjusted to form a predetermined angle, and faces the user's chest toward the front of the user. It is attached to. The wide-angle camera 3 is composed of a CMOS camera or the like having a higher resolution and a wider angle than a normal camera. For example, a PL-A662 manufactured by Pixellink with a pixel number of 1280 × 1024 can be used. The wide-angle camera 3 captures a color image (visible light image) in the front direction of the user and outputs front image data to the communication unit 5.

  The microphone 4 collects a user's utterance or ambient sound and outputs voice data to the communication unit 5. When not recording sound, the microphone 4 may be omitted, and a speaker or the like may be integrally configured to transmit sound or the like to the user.

  The communication unit 5 transmits infrared image data, front image data, and audio data to the communication unit 6 by a wireless method. The transmission method of each data is not particularly limited to the above example, and each data may be transmitted by a wired method or the like. When the data processing unit 7 and the data recording unit 8 are downsized and configured integrally with the sensor unit 10, the communication units 5 and 6 are omitted and each data is directly transmitted from the infrared camera 1 or the like to the data processing unit 7. You may make it do.

  The communication unit 6, the data processing unit 7, and the data recording unit 8 are configured by a computer or the like that includes a ROM (Read Only Memory), a CPU (Central Processing Unit), a RAM (Random Access Memory), a hard disk drive, a wireless communication device, and the like. And installed at a predetermined position away from the user.

  The communication unit 6 receives infrared image data, front image data, and audio data transmitted from the communication unit 5 by a wireless method and outputs them to the data processing unit 7. The data processing unit 7 calculates the user's head posture data from the infrared image data, extracts user viewpoint image data in the user's line-of-sight direction from the front image data based on the head posture data, The user viewpoint image data and audio data are associated with the observation time and recorded in the data recording unit 8 as a database.

  FIG. 3 is a block diagram showing a functional configuration of the data processing unit 7 shown in FIG. In FIG. 3, the communication units 5 and 6 are not shown for ease of explanation. As shown in FIG. 3, the data processing unit 7 executes a head posture estimation processing program stored in a ROM or the like by a CPU or the like, thereby calculating a binarizing unit 11, a person area extracting unit 12, a boundary line calculation. Functions as a unit 13, a nose tip extraction unit 14, a head rotation calculation unit 15, a change amount calculation unit 16, a gaze direction calculation unit 17, and a gaze image extraction unit 18. The configuration of the data processing unit 7 is not particularly limited to the above example, and may be configured by dedicated hardware or the like.

  The binarization unit 11 acquires infrared image data from the infrared camera 1 via the communication units 5 and 6, binarizes the acquired infrared image data, and outputs the infrared image data to the person region extraction unit 12. The person area extracting unit 12 detects a boundary line between the person area including the user's head and torso and other background areas using the binarized infrared image data, and divides the person area and the background area. To do.

  The boundary line calculation unit 13 determines the boundary line between the head and the torso based on the shape of the boundary line between the person area and the background area, and determines the midpoint of the boundary line between the head and the body. Determine as the center of rotation of the head. The tip of the nose tip extraction unit 14 is the tip where the distance from the midpoint of the boundary line between the head and the torso is the maximum from the region divided as the head by the boundary between the head and the torso in the human region. A point is extracted as a nose tip point. The head rotation calculation unit 15 calculates a horizontal angle and a vertical angle of the head around the midpoint of the boundary line between the head and the torso from the position of the nose tip, and sends the head posture to the data recording unit 8 Record as data.

  The change amount calculation unit 16 calculates a horizontal correction angle and a vertical correction angle, which are posture change amounts of the infrared camera 1 and the wide-angle camera 3 with respect to the user, based on the inclination of the boundary line between the head and the torso and the position of the midpoint. To do. The line-of-sight direction calculation unit 17 includes the horizontal and vertical angles of the infrared camera 1 and the wide-angle camera 3 calculated by the horizontal and vertical angles of the head calculated by the head rotation calculation unit 15 and the change amount calculation unit 16. From this, the user's line-of-sight direction is calculated. The line-of-sight image extraction unit 18 acquires front image data from the wide-angle camera 3 via the communication units 5 and 6, extracts an image in a predetermined range centered on the user's line-of-sight direction from the acquired front image data, and records the data. The unit 8 records it as user viewpoint image data.

  In the present embodiment, the infrared camera 1 corresponds to an example of a head observing means, and a binarization unit 11, a person region extraction unit 12, a boundary line calculation unit 13, a nose tip extraction unit 14, and a head rotation calculation unit 15 Corresponds to an example of a head posture estimation unit, the wide-angle camera 3 corresponds to an example of a front photographing unit, and the line-of-sight direction calculation unit 17 and the line-of-sight image extraction unit 18 correspond to an example of a line-of-sight image extraction unit. In addition, the infrared camera 1 corresponds to an example of a head imaging unit and an infrared imaging unit, the binarization unit 11 and the person area extraction unit 12 correspond to an example of an area extraction unit, and the boundary line calculation unit 13 determines a boundary line. The nose tip point extraction unit 14 corresponds to an example of a tip point extraction unit, and the head rotation calculation unit 15 corresponds to an example of an estimation unit. The infrared illuminator 2 corresponds to an example of an infrared illumination unit, the wide-angle camera 3 corresponds to an example of a front photographing unit, the line-of-sight direction calculation unit 17 and the line-of-sight image extraction unit 18 correspond to an example of an image extraction unit, and the amount of change The calculation unit 16 corresponds to an example of a calculation unit.

  Next, the head posture estimation process by the data processing unit 7 shown in FIG. 3 will be described. FIG. 4 is a flowchart for explaining head posture estimation processing by the data processing unit 7 shown in FIG.

  First, the binarization unit 11 acquires infrared image data output from the infrared camera 1 via the communication units 5 and 6 (step S11), and binarizes the acquired infrared image data (step S12). ). Here, the coordinate system and infrared image of the head model used for the head posture estimation process will be described below.

  FIG. 5 is a schematic diagram showing a coordinate system of the head model. As shown in FIG. 5, the X0 axis and the Z0 axis are defined along the horizontal plane, the Y0 axis is defined in the vertical direction, the rotation center RC of the user's head as the origin is defined in front of the neck, and after rotation Is represented by X, Y, and Z. At this time, since the user's line of sight does not depend on the Z-axis rotation, the rotation angles α and β around the Y-axis and the X-axis become the horizontal angle and the vertical angle of the user's head.

  FIG. 6 is a diagram schematically showing an infrared image and a visible light image obtained by photographing the user's head and torso. As shown in FIG. 6, when a camera is mounted on the user's chest and the user's head and torso are photographed, the image shown in FIG. 6A is photographed as an infrared image, and the visible light image is depicted in FIG. An image shown in (b) of FIG.

  In other words, in the visible light image shown in (b), the ceiling air conditioner and illumination as a background are clearly photographed, and conversely, a part of the user's head is not clearly photographed by illumination. Even if such an image is binarized, it is difficult to separate the person area and the background area. On the other hand, in the infrared image shown in (a), since the infrared illuminator 2 uses infrared rays to illuminate only a region close to the distance, only the user's head and torso are clearly captured, and the other is the background. The part of is taken dark. Therefore, the person area and the background area can be accurately divided by binarizing the infrared image.

  Next, the person area extraction unit 12 detects a boundary line between the person area and the background area from the binarized infrared image data, and divides it into the person area and the background area (step S13), and the boundary line calculation unit 13 extracts the head region from the change in the inclination of the boundary between the person region and the background region, and detects the boundary between the head and the torso using only the boundary of the torso (step S14). Although not limited to this, for example, as shown in FIG. 7 to be described later, a head region is extracted by detecting points P4 and P5 of a portion where the change in the inclination of the boundary line is large as a body-head boundary. The head is extracted by detecting the points P4 and P5, and the boundary line between the head and the torso is detected using the points P4 and P5. Here, when the inclination of the boundary line at the position x is θ (x), the maximum point and the minimum point of dθ (x) / dx are P4 and P5, respectively.

  Next, the boundary line calculation unit 13 detects the midpoint of the boundary line between the head and the torso as the rotation center of the user's head (step S15), and the nose tip point extraction unit 14 detects the person in the person area. From the region divided as the head by the boundary line between the region and the background region, the nose tip point having the maximum distance from the middle point of the boundary line is extracted (step S16).

  FIG. 7 is a schematic diagram for explaining a method of determining the boundary line between the head and the torso and the nose tip. As shown in FIG. 7, the person area extraction unit 12 detects boundary lines L1 to L3 between the person area and the background area, and the boundary line calculation unit 13 determines the point P4 from the change in the inclination of the boundary lines L1 to L3. A boundary line BL between the head and the trunk is calculated using the point P5. Next, the boundary line calculation unit 13 extracts the midpoint MP between the points P4 and P5, and the nose tip point extraction unit 14 selects the head region between the points P4 and P5, that is, the head boundary line L2. A point NP having the maximum distance from the midpoint MP in the inner region is determined as the nose tip point.

  FIG. 8 is a schematic diagram illustrating a detection example of a person region, a boundary line between a head and a torso, and a nose tip point. In each upper image in FIG. 8, the hatched portion indicates the background region, the white portion without the oblique line indicates the person region PR, and in each lower image, the straight line indicates the boundary line BL between the head and the torso. The x mark indicates the nose tip NP. Thus, the person region PR, the boundary line BL between the head and the torso, and the nose tip NP were accurately estimated from the infrared image according to the movement of the user's head.

Next, the head rotation calculation unit 15 calculates the horizontal angle and the vertical angle of the head centering on the midpoint of the boundary line between the person area and the background area from the position of the nose tip (step S17). FIG. 9 is a schematic diagram showing the relationship between the nose tip point and the midpoint of the boundary line between the person area and the background area. As shown in FIG. 9, the nose tip point on the infrared image and the midpoint of the boundary line between the person area and the background area, that is, the head rotation center are respectively represented by P _n (= [u _n , v _n ]), P _c. (= [U _c , v _c ] = [0, 0]). Here, the u-axis coincides with the boundary line BL between the head and the trunk, and the v-axis is an axis perpendicular to the u-axis.

At this time, the horizontal angle α and the vertical angle β of the user's head are calculated by the following equations since the head rotation center P _c is located in front of the user's neck.

Here, P _n ⁽⁰⁾ (= [u _n ⁽⁰⁾ , v _n ⁽⁰⁾ ]) is the value of P _n when α = 0 and β = 0. This is set when the user makes an initialization setting (such as pressing a button) of the apparatus facing the front. Further, k is a constant determined by the size of the user's head and the internal parameters of the infrared camera 1, and is obtained as follows.

FIG. 10 is a schematic diagram when the user is viewed from the side. FIG. 10A shows a state where the user is facing the front, and FIG. 10B shows a state where the user is scolding. As shown in FIG. 10A, when the user is facing the front, the vertical distance from the infrared camera 1 to the head rotation center P _c is d ₀ , the nose tip point P _n and the head rotation center P _c . , D is the horizontal distance, h _{0 is} the horizontal distance, θ ₀ is the angle with respect to the horizontal line of the straight line connecting the nose tip point P _n and the head rotation center P _c , as shown in FIG. When the horizontal distance between the nose tip point P _n and the head rotation center P _c is h ₁ , and the angle with respect to the horizontal line of the straight line connecting the nose tip point P _n and the head rotation center P _c is θ ₁ , the following equation is established.

  Here, when the focal length of the infrared camera 1 is f, the following equation is established.

  Substituting equation (5) into equation (4),

Thus, k in the equation (2) is k = D (f / d ₀ ). Here, the distance D and the vertical distance d0 are measured in advance during the initialization. The focal length f is determined when designing the optical system.

  Next, the change amount calculation unit 16 uses the horizontal correction angle Δα and the vertical direction that are the amount of change in posture of the infrared camera 1 and the wide-angle camera 3 with respect to the user based on the inclination of the boundary line between the head and the torso and the position of the midpoint. A correction angle Δβ is calculated (step S18). The horizontal correction angle Δα is the inclination of the boundary line BL between the head and torso shown in FIG. 9, and the vertical correction angle Δβ is calculated by the following equation.

Here, P _c ⁽⁰⁾ (= [u _c ⁽⁰⁾ , v _c ⁽⁰⁾ ]) is the value of P _c when Δα = 0 and Δβ = 0.

  Next, the line-of-sight direction calculation unit 17 calculates the horizontal angle α and the vertical angle β of the head calculated by the head rotation calculation unit 15 and the horizontal direction of the infrared camera 1 and the wide-angle camera 3 calculated by the change amount calculation unit 16. From the correction angle Δα and the vertical correction angle Δβ, the user's line-of-sight directions xα, yβ on the front image of the wide-angle camera 3 are calculated using the following equations (step S19).

xα = f · tan (α + Δα), yβ = f · tan (β + Δβ) (8)
Finally, the line-of-sight image extraction unit 18 extracts an image in a predetermined range centered on the user's line-of-sight direction from the front image captured by the wide-angle camera 3 (step S20), and then repeats the processing after step S11 as appropriate. In the present embodiment, the line-of-sight direction does not directly mean the direction in which the eyes are facing, and the head posture, that is, the face direction, is usually the direction in which the eyes are facing, that is, the line-of-sight direction. Since they are almost the same, the gaze direction is estimated from the calculation of the posture of the head and the rotation of the head.

  Next, the estimation accuracy of the nose tip in the head posture estimation process will be described with a specific example. FIG. 11 is a diagram showing the estimation results of the nose tip by the head posture estimation apparatus shown in FIG. 1. The broken line in the figure shows the true value measured manually by the experimenter, and the solid line shows the head shown in FIG. The nose tip point estimation result by the head posture estimation device is shown. It can be seen that the head posture estimation apparatus shown in FIG. 11 can extract the nose tip with high accuracy.

  FIG. 12 is a diagram illustrating an example of the locus of the user's line-of-sight direction estimated in the room by the head posture estimation apparatus illustrated in FIG. A broken line VT shown in FIG. 12 is a trajectory in the user's line-of-sight direction, and it can be seen from the trajectory VT that the user has looked at three objects, that is, a clock T1, a calendar T2, and a chair T3 in order. Further, as a result of extracting the user's attention object from the stationary time of the head movement, an image including the clock T1, the calendar T2, and the chair T3 shown in FIG. 12 is extracted, and the user's attention object can be automatically extracted. It was.

  FIG. 13 is a diagram illustrating an example of user action history data recorded by the head posture estimation apparatus illustrated in FIG. 1. The upper and middle sections of FIG. 13 show the user's head posture data, and the lower section shows the audio data. The user viewpoint image data as described above was recorded in the data recording unit 8 together with these data. By using these pieces of information, it is possible to detect a user's head gesture (whipping, swinging, etc.), detecting the time the user is speaking, and the like.

  As described above, in the present embodiment, the infrared camera 1 captures an infrared image including the user's head and torso, and a person region including the user's head and torso is extracted from the captured infrared image. In the extracted person area, the boundary line representing the boundary between the head and the torso is determined, the midpoint of the boundary line is determined, and the nose point that has the maximum distance from the midpoint of the boundary line is the person Since the posture of the head is estimated based on the position of the extracted nose point extracted from the region divided as the head by the boundary line in the region, it is possible to accurately estimate the posture of the user's head it can.

  In addition, since the image in the user's line-of-sight direction is cut out from the front image captured by the wide-angle camera 3 based on the estimated head posture, the image in the user's line-of-sight direction, that is, the image that the user is paying attention to is accurate. Can be detected.

  Further, since the sensor unit 10 such as the infrared camera 1 and the wide-angle camera 3 is mounted on the chest of the user and does not need to be mounted on the user's head like a head-mounted camera, it is visually uncomfortable for a third party. It can be easily attached and detached without giving an image, and can be suitably used for recording daily activities.

  In the present embodiment, the nose tip point is detected using an infrared camera. However, the present invention is not particularly limited to this example as long as a feature point whose position changes according to the movement of the head can be detected. Various modifications, such as detecting the tip of the chin using a camera, are possible. In this embodiment, one data processing unit and one data recording unit are used for each sensor unit. However, the present invention is not particularly limited to this example, and processing for a plurality of sensor units (a plurality of humans) is performed. Various changes such as a single data processing unit are possible. In this embodiment, the posture of the head is estimated using the captured image. However, information used for estimating the head posture is not particularly limited to this example. The head shape may be acquired by measuring the distance from the head to the head, and the head posture may be estimated by pattern matching or the like.

  In addition, since the head posture estimation apparatus of the present invention can be suitably used for recording daily activities, it is suitable for uses such as recording of human experiences, summarization of human behavior history, analysis of human interactions, etc. In these cases, other data such as biological data such as pulse and body temperature may be recorded together.

It is a block diagram which shows the structure of the head posture estimation apparatus by one embodiment of this invention. It is the schematic which shows the mounting state to the person of the sensor part of the head posture estimation apparatus shown in FIG. It is a block which shows the functional structure of the data processing part shown in FIG. It is a flowchart for demonstrating the head posture estimation process by the data processing part shown in FIG. It is a schematic diagram which shows the coordinate system of a head model. It is a figure which shows typically the infrared image and visible light image which image | photographed the user's head and trunk | drum. It is a schematic diagram for demonstrating the method of determining the boundary line and nose tip point of a head and a trunk | drum. It is a schematic diagram which shows the example of a person area | region, the boundary line of a head and a trunk | drum, and a nose tip point. It is a schematic diagram which shows the relationship between a nose tip point and the midpoint of the boundary line of a person area | region and a background area | region. It is a schematic diagram when a user is seen from the side. It is a figure which shows the estimation result of the nose tip point by the head posture estimation apparatus shown in FIG. It is a figure which shows an example of the locus | trajectory of a user's gaze direction estimated in the room | chamber interior by the head posture estimation apparatus shown in FIG. It is a figure which shows an example of the user's action history data recorded by the head posture estimation apparatus shown in FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Infrared camera 2 Infrared illuminator 3 Wide angle camera 4 Microphone 5,6 Communication part 7 Data processing part 8 Data recording part 11 Binary part 12 Person area extraction part 13 Boundary line calculation part 14 Nose tip point extraction part 15 Head rotation calculation Unit 16 change amount calculation unit 17 gaze direction calculation unit 18 gaze image extraction unit

Claims (9)

A head observation means mounted on a human chest and observing the human head;
A head posture estimation apparatus comprising: head posture estimation means for estimating a head posture from information observed by the head observation means. A front photographing means fixed integrally with the head observing means and photographing a front image of the human;
The image processing apparatus further comprises: a line-of-sight image extracting unit that extracts an image in the human line-of-sight direction from an image captured by the front image capturing unit based on the head posture estimated by the head posture estimation unit. The head posture estimation apparatus according to claim 1. The head observing means includes a head photographing means that is mounted on a human chest and photographs an image including the human head and torso.
The head posture estimation means includes
Area extracting means for extracting a person area including a human head and torso from an image photographed by the head photographing means;
Determining a boundary line representing a boundary between the head and the torso in the person region, and a boundary line determining means for determining a midpoint of the boundary line;
Tip point extraction means for extracting a tip point having a maximum distance from a midpoint of the boundary line from an area of the person region divided as the head by the boundary line;
The head posture estimation apparatus according to claim 1, further comprising: estimation means for estimating a head posture based on the position of the tip point. Infrared illumination means for illuminating the human head and torso, which is mounted on a human chest and photographed by the head photographing means, using infrared rays,
The head photographing means includes an infrared photographing means for photographing an infrared image including a human head and a torso illuminated by infrared rays by the infrared illuminating means,
4. The head posture estimation apparatus according to claim 3, wherein the region extracting unit extracts a human region including a human head and a torso from an infrared image captured by the infrared imaging unit.   5. The head posture estimation apparatus according to claim 3, wherein the estimation unit estimates a horizontal angle and a vertical angle of a head centering on a midpoint of the boundary line from a position of the tip point. . A front imaging unit that is fixed integrally with the head imaging unit and that captures an image in the front direction of the person wearing the head imaging unit;
An image extracting means for extracting an image in the direction of the line of sight of a human wearing the head photographing means from an image photographed by the front photographing means based on the posture of the head estimated by the estimating means; The head posture estimation apparatus according to any one of claims 3 to 5. A calculation means for calculating a posture change amount of the head photographing means and the front photographing means with respect to a human based on the boundary line;
The image extracting means is photographed by the front photographing means based on the posture change amount of the head photographing means and the front photographing means calculated by the calculating means and the posture of the head estimated by the estimating means. The head posture estimation apparatus according to claim 6, wherein an image in a line of sight of a human wearing the head photographing means is extracted from the captured image. Using a computer functioning as an area extraction means, boundary line determination means, tip point extraction means, and estimation means, a head photographing means for photographing an image including a human head and torso is mounted on a chest. A head posture estimation method for estimating posture,
The region extracting means extracting a human region including a human head and torso from an image photographed by the head photographing means;
The boundary line determining means determines a boundary line representing a boundary between a head and a torso in the person area, and determines a midpoint of the boundary line;
The tip point extracting means extracts a tip point having a maximum distance from the middle point of the boundary line from an area of the person region divided as a head by the boundary line;
And a step of estimating the head posture based on the position of the tip point. A head posture estimation processing program for estimating the posture of a human head equipped with a head photographing means for photographing an image including a human head and a torso on a chest,
Area extracting means for extracting a person area including a human head and torso from an image photographed by the head photographing means;
Determining a boundary line representing a boundary between the head and the torso in the person region, and a boundary line determining means for determining a midpoint of the boundary line;
Tip point extraction means for extracting a tip point having a maximum distance from a midpoint of the boundary line from an area of the person region divided as the head by the boundary line;
A head posture estimation processing program for causing a computer to function as estimation means for estimating a head posture based on the position of the tip point.

Images