JP5111934B2 - Monitoring device - Google Patents

Description

  The present invention relates to a monitoring apparatus, and more particularly to a monitoring apparatus that detects a monitoring target person such as a person from a video image such as a monitoring image and monitors the behavior of the monitoring target person.

  Conventional monitoring devices generally detect a person or a face of a person based on a monitoring image, display the image of the person or the face or a flow line of the person in real time on the screen, and The behavior of the person was monitored by reproducing the trajectory information.

In other monitoring devices, a person's action is monitored by simultaneously displaying a current flow line locus and a past flow line locus on a divided screen (see, for example, Patent Document 1).
JP 2003-101999 A

  However, in the conventional monitoring device, the image of the person and the movement trajectory around the person are displayed in real time as information, but what the person is looking at, that is, what is in the person's field of view, There was a problem of not knowing what the subject was. In addition, when a conventional monitoring device is used, the person's behavior is monitored based on the information of the person's flow line trajectory, but since the person's attention target is unknown, the person's behavior is determined based on the person's visual field information. There was a problem of being unable to monitor.

  The present invention has been made to solve the conventional problems, and can display an image corresponding to the field of view of a person to be monitored such as a person on the screen, and the behavior of the person based on the field of view information of the person to be monitored It is an object of the present invention to provide a monitoring device that enables monitoring of the above.

  The monitoring apparatus of the present invention includes an image input unit that inputs an image of a monitoring target person in a monitoring region, a spatial position calculation unit that calculates the spatial position of the monitoring target person from the image input from the image input unit, and an image input A line-of-sight direction detection unit that detects the line-of-sight direction of the person to be monitored from the image input from the unit, a line-of-sight image storage unit that stores image data to which a shooting position and a shooting direction are added, and a spatial position and a line-of-sight direction A line-of-sight image extraction unit that extracts a line-of-sight image at a spatial position as a line-of-sight image from the line-of-sight image storage unit, and an image corresponding to the viewing angle of the monitoring subject among the line-of-sight images based on the spatial position and the line-of-sight direction A visual field image output unit that outputs the visual field image, and a display unit that displays the visual field image output by the visual field image output unit.

  According to this configuration, based on the spatial position and the line-of-sight direction of the monitoring target person detected from the monitoring image, the visual field image corresponding to the viewing angle of the monitoring target person can be displayed on the screen. You can determine what the subject is.

  The monitoring apparatus according to the present invention is configured to track a spatial position of a monitoring target person, detect a flow line locus of the monitoring target person, estimate a traveling direction of the monitoring target person from the flow line locus, and And a line-of-sight behavior determination unit that determines whether or not the monitoring target person is a suspicious person based on the information on the direction and the traveling direction.

  According to this configuration, it is possible to determine whether or not the monitoring target person is a suspicious person based on the traveling direction and the line-of-sight direction of the monitoring target detected from the monitoring image. can do.

  In the monitoring device of the present invention, the line-of-sight behavior determination unit calculates a relative angle of the line-of-sight direction with respect to the traveling direction, and based on the number of times the relative angle exceeds a predetermined threshold angle, whether or not the monitoring target person is a suspicious person Determine whether.

  According to this configuration, based on the relative angle of the line-of-sight direction with respect to the traveling direction of the monitoring target person, it is possible to detect the suspicious behavior of the monitoring target person and determine whether the monitoring target person is a suspicious person, A suspicious person can be found from the monitoring image.

  The monitoring device of the present invention includes an attention time measuring unit that measures, as an attention time, a time during which a predetermined object is included in the visual field of the monitoring subject based on a spatial position and a line-of-sight direction, and an attention time having a predetermined threshold time. And an attention behavior determination unit that outputs information on the target object when it exceeds.

  According to this configuration, the attention target of the monitoring target person can be specified by measuring the attention time during which the predetermined target is included in the visual field of the monitoring target person.

  In the monitoring device of the present invention, the attention behavior determination unit determines that the monitoring target person is a suspicious person when the attention time exceeds a threshold time.

According to this configuration, it is possible to determine whether or not the monitoring target person is a suspicious person by detecting the suspicious behavior of the monitoring target person based on the attention time, and to detect the suspicious person from the monitoring image. be able to.
In the monitoring apparatus of the present invention, the display unit simultaneously displays the image of the monitoring subject and the visual field image.

  According to this configuration, the visual field image can be displayed on the screen together with the image of the monitoring subject, and the image of the monitoring subject determined to be a suspicious person and the visual field image can be monitored simultaneously.

  The present invention provides a monitoring device that can display an image corresponding to the field of view of a person to be monitored, such as a person, on the screen, and can monitor the behavior of the person to be monitored based on the field of view information of the person to be monitored. It is something that can be done.

(First embodiment)
Hereinafter, a monitoring device according to an embodiment of the present invention will be described with reference to the drawings.
FIG. 1 shows a monitoring apparatus 100 according to the first embodiment of the present invention.

  In FIG. 1, a monitoring device 100 includes a camera unit 1, an image processing unit 2, a spatial position calculation unit 3, a gaze direction detection unit 4, a gaze direction estimation unit 5, a gaze image extraction unit 6, a gaze image storage unit 7, and a visual field image output. A unit 8 and a display unit 9 are provided.

  The camera unit 1 outputs a monitoring image such as a video image of a monitoring target person (for example, a person) in the monitoring area. The image processing unit 2 performs image processing such as processing for converting a monitoring image input by the camera unit 1 into a digital image. The camera unit 1 and the image processing unit 2 function as an image input unit that inputs an image of the person to be monitored in the monitoring area.

  The spatial position calculation unit 3 receives the image data processed by the image processing unit 2 and calculates the spatial position of the monitoring target person from the input monitoring image. For example, the spatial position calculation unit 3 detects the face of the person to be monitored from the monitoring image, specifies the position of the face in the two-dimensional coordinates, and uses the camera calibration information measured in advance to obtain the three-dimensional coordinates. The spatial position of the monitoring subject is calculated by calculating the position of the face part at.

  The line-of-sight direction detection unit 4 detects the face part of the monitoring target person from the image data input from the image processing part 2, and based on the detected image information of the face part, the line of sight of the monitoring target person in the three-dimensional coordinates Detect direction. The gaze estimation unit 5 uses the gaze direction at the spatial position as the gaze information of the monitoring subject based on the monitoring subject's spatial position calculated by the spatial position calculation unit 3 and the gaze direction detected by the gaze direction detection unit 4. presume. The line-of-sight information is information indicating where the monitoring target person is looking from which direction, and can also be information on the line-of-sight vector from the viewpoint. Here, the combination of the spatial position and the line-of-sight direction of the monitoring subject is used as the line-of-sight information.

  The line-of-sight image extraction unit 6 extracts a line-of-sight image corresponding to the line-of-sight information of the monitoring subject estimated by the line-of-sight estimation unit 5 from the line-of-sight image storage unit 7. The line-of-sight image storage unit 7 stores a peripheral image of the monitoring area.

The visual field image output unit 8 selects an image corresponding to the visual field angle of the monitoring target person from the visual line images extracted by the visual line image extraction unit 6 based on the visual line information of the monitoring target person, that is, the spatial position and the visual line direction. Output as field image. The display unit 9 displays the visual field image output by the visual field image output unit 8.
Next, operation | movement of the monitoring apparatus 100 concerning this Embodiment is demonstrated using figures.

  The camera unit 1 includes a monitoring camera that outputs an image of a monitoring area in real time, and outputs an image signal of the monitoring image captured in real time. The image signal output by the camera unit 1 is transferred to the image processing unit 2 by an image input device such as a capture board. The image processing unit 2 converts the input image signal into a digital signal and generates a digital image. For example, as illustrated in FIG. 2, the image processing unit 2 generates a digital image 20 (for example, an RGB color image) including the monitoring subject 21.

  The spatial position calculation unit 3 detects the face portion of the person to be monitored from the image data input from the image processing unit 2 and specifies the position of the face portion in two-dimensional coordinates. For example, as illustrated in FIG. 3, the spatial position detection unit 3 detects the face 22 of the person 21 to be monitored in the digital image 20 and specifies the position of the center of gravity of the face 22 as the face position P1. In this case, the face portion position P1 is represented by two-dimensional coordinates (x1, y1). Then, the spatial position detection unit 3 calculates the face position in the three-dimensional coordinates using the calibration information of the camera unit 1. For example, as illustrated in FIG. 4, the spatial position detection unit 3 includes calibration information such as the camera position C1 (X1, Y1, Z1) of the camera unit 1, the angle of view of the camera unit 1, and the focal length of the camera unit 1. , The face position P2 in the three-dimensional coordinates (X, Y, Z) as shown in FIG. 4 from the face position P1 (x1, y1) in the two-dimensional coordinates (x, y) as shown in FIG. (X2, Y2, Z2) is calculated as the spatial position of the monitoring subject 21.

  The gaze direction detection unit 4 detects the front direction of the face in the digital image as the gaze direction from the image data input from the image processing unit 2, and based on the gaze direction detected in the two-dimensional coordinates, the three-dimensional coordinates The direction of the line of sight is detected. For example, as illustrated in FIG. 5, the line-of-sight direction detection unit 4 detects the angle of the line-of-sight direction D <b> 1 in the digital image 20. In this case, the line-of-sight direction detection unit 4 detects the angle of the line-of-sight direction D <b> 1 from the template that approximates the face part 22 by matching the face part template at various angles with the face part 22 in the digital image 20. Then, the line-of-sight direction detection unit 4 detects the line-of-sight direction D2 in the three-dimensional coordinates based on the line-of-sight direction D1 on the two-dimensional digital image 20, as shown in FIG.

  The gaze estimation unit 5 receives the monitoring subject's spatial position from the spatial position calculation unit 3, and receives the monitoring subject's gaze direction from the gaze direction detection unit 4. And the gaze estimation part 5 estimates the gaze direction in a spatial position as gaze information of a monitoring subject. For example, as illustrated in FIG. 6, the line-of-sight estimation unit 5 estimates the line-of-sight direction D2 at the face position P2 (X2, Y2, Z2) in the three-dimensional coordinates as the line-of-sight information of the monitoring subject 21. Note that the line-of-sight information is a combination of a spatial position and a line-of-sight direction as described above.

  The line-of-sight image storage unit 7 stores a peripheral image of the monitoring area in advance. Here, a method of creating a peripheral image stored in the line-of-sight image storage unit 7 will be described. The peripheral image stored in the line-of-sight image storage unit 7 is created by adding the camera position and camera direction data to the image data of the peripheral image captured by moving the camera within the monitoring area. For example, as shown in FIG. 7, the peripheral image of the monitoring area includes information on the camera position C2 (X3, Y3, Z3) of the camera 30 and the camera direction θ in the image data of the peripheral image of the monitoring area captured by the camera 30. The peripheral image created by adding is stored in the line-of-sight image storage unit 7. Further, when an important object important as a monitoring target is in the monitoring region, a peripheral image of the important object is created by photographing the important object and its surrounding image in detail. For example, as shown in FIG. 7, a detailed peripheral image of the important object is created by finely changing the camera positions C2, C3, C4 and the camera direction θ. Such a peripheral image creation method is described in, for example, Japanese Patent No. 3433882.

  The line-of-sight image extraction unit 6 receives the line-of-sight information of the monitoring subject estimated by the line-of-sight estimation unit 5 and stores a line-of-sight image corresponding to the line-of-sight information of the monitoring subject and a peripheral image of the monitoring area. Extract from 7. In this case, the line-of-sight image extraction unit 6 refers to the camera position and camera direction information added to the peripheral image stored in the line-of-sight image storage unit 7 and is closest to the spatial position and line-of-sight direction of the monitoring subject. A peripheral image captured at the camera position and the camera direction is extracted from the line-of-sight image storage unit 7 as a line-of-sight image. For example, as shown in FIG. 8A, the line-of-sight image extraction unit 6 extracts an image in the line-of-sight direction D3 at the camera position C5 closest to the spatial position P3 of the monitoring subject as a line-of-sight image. In this case, the line-of-sight image extracting unit 6 extracts a line-of-sight image including the visual field of the monitoring target person by extracting an image from behind the spatial position P3 of the monitoring target person as the line-of-sight image.

  The visual field image output unit 8 receives the visual line image extracted by the visual line image extraction unit 6, and outputs an image corresponding to the viewing angle of the monitoring target person as the visual field image. Here, the value of the standard viewing angle of the monitoring subject is stored in advance, and the viewing image output unit 8 looks at the image within the range of the viewing angle of the monitoring subject based on the standard viewing angle. Extract from image. For example, as shown in FIG. 8A, the visual field image output unit 8 outputs an image corresponding to the viewing angle of the monitoring subject at the spatial position P3 of the monitoring subject as a visual field image. In this case, as shown in FIG. 8B, the visual field image included in the line-of-sight image is output by the visual field image output unit 8. The display unit 9 displays the visual field image output by the visual field image output unit 8. The display unit 9 can simultaneously display the monitoring image and the field-of-view image of the monitoring subject. For example, as shown in FIG. 9, the display unit 9 displays the real-time monitoring image 90 and the field-of-view image 91 of the monitoring subject 21 on the display screen 40 at the same time. The display unit 9 may display the line-of-sight direction D4 of the monitoring subject 21 on the monitoring image 90.

  As described above, according to the present embodiment, the visual field image corresponding to the visual field angle of the monitoring target person is displayed on the screen based on the spatial position and the line-of-sight direction of the monitoring target person detected from the monitoring image. It is possible to determine what the monitoring target person is interested in.

  Moreover, according to this Embodiment, a visual field image can be displayed on a screen with a monitoring subject's image, and the monitoring subject's image and visual field image judged to be a suspicious person can be monitored simultaneously. .

(Second Embodiment)
FIG. 10 shows a monitoring apparatus 200 according to the second embodiment of the present invention.
10 is different from the monitoring apparatus 100 according to the first embodiment shown in FIG. 1 in that the monitoring apparatus 200 includes a flow line locus detection unit 10 and a line-of-sight behavior determination unit 11. However, the same components as those of the monitoring device 100 according to the first embodiment are denoted by the same reference numerals, and have the same functions and operations, and thus the description thereof is omitted.

The flow line trajectory detection unit 10 inputs the monitoring subject's spatial position calculated by the spatial position calculation unit 3, tracks the monitoring subject's spatial position, and detects the monitoring subject's flow line trajectory in the monitoring region. To do. The line-of-sight behavior determination unit 11 estimates the traveling direction of the monitoring target person from the flow line trajectory detected by the flow line trajectory detection unit 10, and based on the information about the visual line direction and the traveling direction of the monitoring target person, Determine if you are a suspicious person.
Next, operation | movement of the monitoring apparatus 200 concerning this Embodiment is demonstrated using figures.

  The flow line trajectory detection unit 10 inputs the spatial position of the monitoring target calculated by the spatial position calculation unit 3, and the temporal change of the spatial position of the monitoring target continuously detected by the spatial position calculation unit 3 The flow line trajectory of the monitoring subject is detected. For example, as shown in FIG. 11, the flow line locus detection unit 10 detects the flow line locus 50 of the monitoring subject in the XY plane by continuously plotting the spatial position of the monitoring subject that changes with time. To do.

  Information on the flow line locus detected by the flow line locus detection unit 10 is input to the line-of-sight estimation unit 5. The line-of-sight estimation unit 5 monitors the line-of-sight direction at the spatial position of the monitoring target on the flow line locus based on the flow line locus input from the flow line locus detection unit 10 and the line-of-sight direction input from the line-of-sight locus detection unit 4. Estimated as the gaze information of the subject. For example, as illustrated in FIG. 11, the line-of-sight estimation unit 5 estimates line-of-sight directions D5 to D8 at the spatial positions P5 to P8 of the monitoring target on the flow path 50 as the monitoring target person's line-of-sight information.

  The line-of-sight information of the monitoring subject on the flow line trajectory is input to the line-of-sight behavior determination unit 11. The line-of-sight behavior determination unit 11 estimates the traveling direction of the monitoring target person based on the flow line trajectory of the monitoring target person, and whether the monitoring target person is a suspicious person based on the information about the line-of-sight direction and the traveling direction of the monitoring target person. Determine whether or not. For example, as illustrated in FIG. 12, the line-of-sight behavior determination unit 11 estimates tangents of the flow line locus 50 as the traveling directions L5 to L8 of the monitoring target person. Then, the line-of-sight behavior determination unit 11 calculates the relative angles α5 to α8 of the line-of-sight directions D5 to D8 with respect to the traveling directions L5 to L8, and monitors based on the number of times the relative angles α5 to α8 exceed a predetermined threshold angle. It is determined whether the person is a suspicious person. Specifically, the line-of-sight behavior determination unit 11 determines that the monitoring target person is a suspicious person when the number of times the relative angle exceeds a predetermined threshold angle exceeds the predetermined number. When it is determined that the person to be monitored is a suspicious person, the line-of-sight behavior determination unit 11 outputs a warning signal to the display unit 9, and the display unit 9 displays a warning to notify the monitoring staff and the like of the suspicious person discovery. .

  As described above, the line-of-sight behavior determination unit 11 compares the number of times that the relative angle exceeds the threshold angle with the predetermined reference number. The measurement time of this excess number may be set appropriately. For example, the number of times of excess in the time from when the monitoring subject appears in the monitoring image to when the monitoring subject leaves is measured. Further, the number of times of excess per unit time may be measured.

  Moreover, the reference | standard frequency | count for determining whether the eyes | visual_axis behavior determination part 11 is a suspicious person may be set appropriately. In general, the monitoring target person other than the suspicious person tends to travel mainly in the line-of-sight direction, and the relative angle of the line-of-sight direction with respect to the traveling direction tends to be within a predetermined angle. On the other hand, a suspicious person tends to look around and frequently look in a direction different from the traveling direction, and has a strong tendency that the relative angle of the line-of-sight direction with respect to the traveling direction exceeds a predetermined angle. Therefore, the reference number for determining whether or not the monitoring target person is a suspicious person is set so that the suspicious person can be identified in consideration of this frequency. This reference number is stored in advance and referred to by the line-of-sight behavior determination unit 11. The reference number may be set to a value obtained by subtracting the variation from the standard value in consideration of the standard value and variation of the excess number by the suspicious person.

  As described above, according to the present embodiment, it is possible to determine whether or not the monitoring target person is a suspicious person based on the traveling direction and the line-of-sight direction of the monitoring target person detected from the monitoring image. The suspicious person can be found from the monitoring image.

  Moreover, according to this Embodiment, based on the relative angle of the gaze direction with respect to the advancing direction of a monitoring subject, the suspicious behavior of a monitoring subject is detected and it is judged whether a monitoring subject is a suspicious person. And suspicious persons can be found from the monitoring image.

(Third embodiment)
A monitoring apparatus 300 according to the third embodiment of the present invention is shown in FIG.
In FIG. 13, the monitoring device 300 is different from the monitoring device 100 according to the first embodiment shown in FIG. 1 in that it includes an attention time measurement unit 12 and an attention behavior determination unit 13. However, the same components as those of the monitoring apparatus 300 according to the first embodiment are denoted by the same reference numerals, and the same functions and operations are performed.

The attention time measurement unit 12 measures the time during which a predetermined target is included in the visual field of the monitoring subject as the attention time based on the spatial position and the line-of-sight direction of the monitoring subject. The attention behavior determination unit 13 outputs information on an object included in the field of view of the monitoring subject when the attention time exceeds a predetermined threshold time. The attention behavior determination unit 13 determines that the monitoring target person is a suspicious person when the attention time measured by the attention time measurement unit 12 exceeds a predetermined threshold time.
Next, operation | movement of the monitoring apparatus 300 concerning this Embodiment is demonstrated using figures.

  As described above, the attention time measuring unit 12 measures, as the attention time, a time during which a predetermined object is included in the visual field of the monitoring subject based on the spatial position and the line-of-sight direction of the monitoring subject. As a specific operation, a visual field image is input from the visual field image output unit 8 to the attention time measuring unit 12. And the time when a predetermined target object is included in a visual field image is measured as attention time.

  For example, in FIG. 14, the time during which the object 92 is included in the visual field image 91 is measured based on the visual field image 91. In this case, the inter-frame difference of the visual field image 91 is sequentially calculated. Then, the time during which the difference is within a predetermined range is measured as the attention time. The attention time measured by the attention time measurement unit 12 is input to the attention behavior determination unit 13.

  In the above measurement processing, a visual field image obtained based on the spatial position and the line-of-sight direction of the monitoring subject is used, and the time during which a predetermined target exists in the visual field image is measured. In this way, the attention time measuring unit 12 measures the attention time based on the spatial position and the line-of-sight direction of the monitoring subject.

  When the attention time measured by the attention time measurement unit 12 exceeds a predetermined threshold time, the attention behavior determination unit 13 outputs information on the target object included in the visual field of the monitoring target person. This determination process is based on standard suspicious person behavior. Generally, a suspicious person tends to continue to look at a specific object of interest. Therefore, the predetermined threshold time is set according to the attention time that can identify the suspicious person in consideration of such a tendency. This threshold time is stored in advance and referred to by the attention behavior determination unit 13. The threshold time is preferably set to be smaller than the standard value by taking the standard value and variation of the suspicious person's attention time into consideration.

  Then, when the attention time exceeds the threshold time, information on the object is output. For example, the attention behavior determination unit 13 outputs information on the object 92 shown in FIG. 14 when the attention time exceeds a predetermined threshold time. Here, the information on the target object 92 includes, for example, an image of the target object 92, a field-of-view image 91 including the target object 92, a coordinate position of the target object 92, an image of a monitoring subject who pays attention to the target object 92, and a target object. 92 attention times and the like.

  The attention behavior determination unit 13 determines that the monitoring target person is a suspicious person when the attention time measured by the attention time measurement unit 12 exceeds the above threshold time. If it is determined that the person to be monitored is a suspicious person, the attention behavior determining unit 13 outputs a warning signal to the display unit 9, and the display unit 9 performs warning display to detect a suspicious person as a monitor. To inform.

  As described above, according to the present embodiment, the target object of the monitoring target person can be specified by measuring the attention time during which the predetermined target object is included in the field of view of the monitoring target person.

  Further, according to the present embodiment, it is possible to determine whether or not the monitoring target person is a suspicious person by detecting the suspicious behavior of the monitoring target person based on the attention time. Can be found.

(Other embodiments)
In the first to third embodiments, the spatial position calculation unit 3 calculates the position of the face part in the three-dimensional coordinates using the camera calibration information. Techniques may be applied.

  In the first to third embodiments, the gaze direction detection unit 4 detects the front direction of the face as the gaze direction, but various methods for detecting the gaze direction of the monitoring subject may be applied. . For example, the line-of-sight direction may be detected from the face direction and the eyeball direction. Further, the line-of-sight direction may be detected from the relative positions of the eyes and nose of the face.

  In the first to third embodiments, the line-of-sight image storage unit 7 stores the peripheral image in advance, and the line-of-sight image extraction unit 6 extracts the line-of-sight image from the line-of-sight image storage unit 7. May extract an image closest to the image in the line-of-sight direction in the spatial position of the monitoring subject as a line-of-sight image from real-time images captured by a plurality of cameras.

  Further, in the second embodiment, the line-of-sight behavior determination unit 11 estimates the tangent of the flow line trajectory as the traveling direction of the monitoring target person, but the body part of the monitoring target person from the position of both shoulders of the monitoring target person May be estimated as the traveling direction.

  In the third embodiment, the attention time measuring unit 12 is based on the difference between the visual field image signals in the line-of-sight direction in the spatial position of the monitoring target person based on the spatial position and the line-of-sight direction of the monitoring target person. However, the time during which the line of sight of the monitoring subject is within a predetermined range around the object may be measured as the attention time. In this way, it is possible to determine whether the monitoring subject is paying attention to the object directly from the spatial position and the gaze direction without using a visual field image based on the spatial position and the gaze direction of the monitoring subject, The attention time can be measured. The attention time measuring unit 12 previously inputs the coordinate position of the important object important as the monitoring target, and measures the time when the coordinate position of the important object is included in the field of view of the monitoring target as the attention time. Also good.

  In the third embodiment, the attention behavior determination unit 13 determines that the monitoring target person is a suspicious person when the attention time exceeds a predetermined threshold time, but the monitoring target person is a customer or the like. In this case, when the attention time exceeds a predetermined threshold time, the target object of the customer or the like may be determined as the product of interest, and the customer or the like may be analyzed.

  The monitoring device according to the present invention can display an image corresponding to the field of view of a person to be monitored, such as a person, and enables monitoring of the behavior of the person to be monitored based on the field of view information of the person to be monitored. It has an effect and is useful as a monitoring device for monitoring the behavior of a person or the like.

The block diagram of the monitoring apparatus concerning the 1st Embodiment of this invention The figure which showed an example of the digital image which imaged the monitoring subject The figure which showed an example of the face part of the monitoring subject detected in the two-dimensional coordinate The figure which showed an example of the face part of the monitoring subject detected in three-dimensional coordinates The figure which showed an example of the gaze direction of the monitoring subject detected in the two-dimensional coordinate The figure which showed an example of the gaze direction of the monitoring subject detected in three-dimensional coordinates The figure explaining an example of the creation method of the surrounding image stored in a gaze image storage part The figure explaining an example of extraction of a look image and generation of a visual field image The figure which showed an example of the screen which a display part displays The block diagram of the monitoring apparatus concerning the 2nd Embodiment of this invention The figure which showed an example of the flow line locus and line-of-sight direction of the monitoring subject The figure which showed an example of the relative angle of the gaze direction with respect to the advancing direction of the monitoring subject The block diagram of the monitoring apparatus concerning the 3rd Embodiment of this invention Diagram explaining attention time measurement by attention time measurement unit

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Camera part 2 Image processing part 3 Spatial position calculation part 4 Gaze direction detection part 5 Gaze estimation part 6 Gaze image extraction part 7 Gaze image memory | storage part 8 Visual field image output part 9 Display part 10 Flow line locus | trajectory detection part 11 Gaze behavior determination part 12 Attention Time Measurement Unit 13 Attention Behavior Determination Unit 100, 200, 300 Monitoring Device

Claims (6)

From the input image, the spatial position calculating unit for calculating a spatial position of the monitored person,
From the entering force image, and the line-of-sight direction detecting unit for detecting a gaze direction of the person to be monitored,
A line-of-sight image storage unit for storing image data to which a shooting position and a shooting direction are added;
Wherein among the image data stored in the line-of-sight image storage unit, line of sight to extract images and the photographing position closer to the spatial position and the shooting direction closer to the viewing direction is added, from the line-of-sight image storage unit as the sight image An image extraction unit ;
Monitoring device, characterized in that it comprises a. From among the line-of-sight images, based on the spatial position and the line-of-sight direction, a visual field image output unit that outputs, as the visual field image, an image corresponding to the viewing angle of the monitoring subject among the visual line images;
A display unit for displaying the visual field image output by the visual field image output unit;
The monitoring apparatus according to claim 1, further comprising: A flow line trajectory detection unit that tracks the spatial position of the monitoring target person and detects a flow line trajectory of the monitoring target person;
A line-of-sight behavior determination unit that estimates a traveling direction of the monitoring target person from the flow line trajectory and determines whether the monitoring target person is a suspicious person based on information on the line-of-sight direction and the traveling direction. monitoring device according to claim 1 or 2, characterized in that it comprises. The line-of-sight behavior determination unit calculates a relative angle of the line-of-sight direction with respect to the traveling direction, and determines whether the monitoring target person is a suspicious person based on the number of times the relative angle exceeds a predetermined threshold angle. The monitoring apparatus according to claim 3 , wherein the determination is performed. An attention time measurement unit that measures, as an attention time, a time during which a predetermined object is included in the visual field of the monitoring subject based on the spatial position and the line-of-sight direction;
Monitoring device according to any one of claims 1 to 4 wherein the target time is characterized by comprising a target behavior determination unit for outputting information of the object if it exceeds between predetermined threshold time. An image input step for inputting an image of the person to be monitored;
A spatial position calculating step for calculating a spatial position of the monitoring subject from the input image;
A line-of-sight direction detecting step for detecting a line-of-sight direction of the monitoring subject from the input image;
A line-of-sight image storage step for storing image data to which a shooting position and a shooting direction are added;
A line-of-sight image extraction step of extracting, from the line-of-sight image storage unit, an image added with a photographing position close to the spatial position and a photographing direction close to the line-of-sight direction among the stored image data;
The monitoring method characterized by including.

Images