JP5127531B2 - Image monitoring device - Google Patents

Description

  The present invention relates to an image monitoring apparatus, and more particularly, to an image monitoring apparatus that detects a suspicious person based on an image obtained by photographing a face of a store visitor or a passerby.

  In recent years, a monitoring device that detects a suspicious person based on a monitoring image obtained by capturing a monitoring area has been developed. An area corresponding to a person is extracted from the monitoring image, image information in the extraction area is examined, and the area is extracted. There is a proposal of a technique for determining whether or not a photographed person is a suspicious person. For example, a suspicious person determination device that determines a suspicious person based on the result of examining the orientation of the face from the extracted region has been proposed (see Patent Document 1). The suspicious person determination device focuses on the fact that a suspicious person tends to gaze at a specific direction such as a doorway or a window, and is set in advance by specifying the direction of the person's face in the photographed image. The person is determined to be a suspicious person when the time of facing the direction is long.

JP 2006-109014 A

  In the suspicious person determination device described in Patent Document 1, the direction of an entrance / exit, a window, or the like that a suspicious person will gaze at is different for each monitoring area. Therefore, every time a surveillance camera that captures the surveillance area is installed, the positional relationship between the entrance and exit windows and the surveillance camera is examined, and the direction of the entrance and exit is set on the image acquired by the surveillance camera. Is required.

  Accordingly, an object of the present invention is to provide an image monitoring apparatus capable of detecting a suspicious person from an image obtained by photographing a monitoring area without requiring such complicated work.

The present invention for solving this problem provides an image monitoring apparatus for detecting a suspicious person from a monitoring image obtained by photographing a person in a monitoring area. The image monitoring apparatus includes an imaging unit that sequentially acquires monitoring images, a person area detecting unit that detects a person area from the monitoring image, an arrangement information storage unit that stores a feature point arrangement of a face facing the front in advance, and a person Face orientation detection means for extracting facial feature points from a region and detecting the amount of feature point position deviation with respect to a face facing forward as compared with the face feature point arrangement stored in the arrangement information storage means And a suspicious behavior determination means for determining a suspicious behavior for a person region if the amount of deviation of the feature point position is equal to or greater than a predetermined amount corresponding to the state of turning away from the face, and a corresponding human region in a plurality of sequentially acquired monitoring images And a suspicious person determination means for determining a suspicious person based on the number of times the person area tracked by the tracking means is determined to be suspicious behavior.
As a psychological tendency peculiar to the suspicious person, there is a tendency to avoid image recording means such as a surveillance camera so that the face is not recorded. Therefore, in the present invention, by using the psychological tendency peculiar to the suspicious person, a person who is consciously or unconsciously turning away from the imaging means such as a monitoring camera is detected as a suspicious person in the monitoring area. Thereby, it becomes possible to detect a suspicious person from the captured image of the monitoring area without requiring a complicated operation of setting a specific gaze target in advance for each monitoring area.

  Further, in the image monitoring apparatus according to the present invention, the face orientation detecting unit further detects the orientation of the face on the monitoring image, and the tracking unit detects on the monitoring image from the person area detected for each of the plurality of monitoring images. The suspicious behavior determination means detects the person's direction of travel corresponding to the person area, and the face direction of the person area of any one of the monitoring images substantially matches the direction of travel. It is preferable not to determine that the person area is suspicious.

  Further, in the image monitoring apparatus according to the present invention, the face direction detecting means further detects the face direction on the monitoring image, and the suspicious behavior determining means is the person area of any one of the plurality of monitoring images. In this case, if there is another person area on the extension of the face direction of the person, it is preferable that the person area is not determined to be suspicious.

  Further, in the image monitoring apparatus according to the present invention, the arrangement information storage means stores in advance a three-dimensional shape model of a human face as the feature point arrangement of the face facing forward, and the face orientation detection means is extracted from the person area. The direction of the face of the three-dimensional shape model when the position of the feature point of the detected face and the position of the corresponding feature point in the three-dimensional shape model most closely match is detected as a deviation amount of the feature point position with respect to the face facing forward. It is preferable.

  The image monitoring apparatus according to the present invention has an effect that a suspicious person can be detected from an image obtained by photographing a monitoring area without requiring a complicated operation when the monitoring camera is installed.

Hereinafter, embodiments of an image monitoring apparatus according to the present invention will be described with reference to the drawings.
In the image monitoring apparatus to which the present invention is applied, when a suspicious person finds a surveillance camera, it uses a tendency to turn away from the surveillance camera in order to avoid taking a face with a high trail property. To do. For this purpose, the image monitoring apparatus extracts a region corresponding to a person's face from a monitoring image obtained by capturing the monitoring region with a monitoring camera, and determines the face orientation. Then, the image monitoring apparatus determines that the person corresponding to the face is a suspicious person when the state in which the direction of the face does not face the monitoring camera continues.

  FIG. 1 shows a schematic configuration of an image monitoring apparatus 10 to which the present invention is applied. As illustrated in FIG. 1, the image monitoring apparatus 10 includes an imaging unit 100, an image processing unit 200, and an output unit 300. Hereinafter, each part of the image monitoring apparatus 10 will be described in detail.

  The imaging unit 100 is a monitoring camera that captures a predetermined monitoring area, and is, for example, a photoelectric conversion element (for example, a CCD sensor, a C-MOS, or the like) that is arranged two-dimensionally and outputs an electrical signal corresponding to the amount of received light. And an imaging optical system for forming an image of the monitoring region on the photoelectric conversion element.

The imaging unit 100 is installed so as to capture the direction of the flow line at a position along the flow line of the monitoring area so that the face of a person passing through or staying in the monitoring area can be captured.
For example, when the image monitoring apparatus 10 monitors a visitor of a store such as a convenience store or a supermarket, the imaging unit 100 sets the shooting direction to the upper or ceiling of the wall in front of the entrance so that the entrance is included in the shooting range. It is attached with the side facing. When the image monitoring apparatus 10 monitors a passerby indoors in an office building, the imaging unit 100 is attached, for example, above the ceiling or wall of the hallway with the shooting direction slightly downward.

  The imaging unit 100 acquires one frame of a monitoring image obtained by capturing a monitoring area at a predetermined time interval (for example, 100 msec). The imaging unit 100 is connected to the image processing unit 200 and passes the acquired monitoring image to the image processing unit 200.

The image processing unit 200 is configured by a so-called computer, for example. Then, the image processing unit 200 detects a suspicious person based on the monitoring image received from the imaging unit 100. For this purpose, the image processing unit 200 includes a suspicious person detection unit 210 and a storage unit 220. Furthermore, the suspicious person detection unit 210 includes a person area detection unit 211, a face direction detection unit 212, a tracking unit 213, a suspicious behavior determination unit 214, and a suspicious person determination unit 215.
Among these, each means of the suspicious person detection unit 210 is a functional module implemented by a microprocessor, a memory, a peripheral circuit thereof, and software operating on the microprocessor. Alternatively, these means may be integrated by firmware. Moreover, you may comprise some or all of these means with an independent electronic circuit, firmware, a microprocessor, etc. Hereinafter, each unit of the suspicious person detection unit 210 will be described in detail.

The person area detection unit 211 extracts a person area, which is an area in which a person is shown, from each monitoring image received from the imaging unit 100. For this purpose, the person area detection unit 211 calculates, for example, one or more feature amounts from a partial area on the monitoring image, and the partial area when the feature amount satisfies a predetermined condition that is considered to correspond to a person. Are extracted as a person area. Specifically, the person area detection unit 211 performs edge pixel extraction using a Sobel filter or the like so that the inclination direction of the luminance change can be known. The feature amount can be, for example, the direction distribution of the edge pixels obtained in the partial region, or the luminance distribution of pixels near the edge. In addition, the predetermined condition described above is satisfied by determining the range of the feature value such as the direction distribution of the edge pixel corresponding to the person and the luminance distribution near the edge from a plurality of images obtained by photographing the person. It can be determined in advance.
Alternatively, the person area detection unit 211 performs template matching with a monitoring image using a predetermined template, and extracts an area where the degree of matching obtained as a result of the template matching is equal to or greater than a predetermined value as a person area. You may make it do. In addition, the person area detection unit 211 performs a difference between the monitoring image and a reference image captured in a state where there is no moving object such as a person in the monitoring area, and connects pixels whose difference value is equal to or greater than a predetermined value. Extracted difference area. Then, when the difference area has a size and shape considered to correspond to a person, the person area detection unit 211 may extract the difference area as a person area. Various methods are already known as the person region extraction processing, and the description thereof is omitted here.

  When the person area detecting unit 211 extracts the person area, the person area detecting unit 211 creates a person image by cutting out the area from the monitoring image. In addition, the person area detecting unit 211 creates person area information indicating the position and size of the center of gravity of the person area on the original monitoring image. The person area detection unit 211 assigns a person identifier to be used for identification from other person areas to the person image and related person area information, and sets them as one person information. The person area detection unit 211 stores the obtained person information in the storage unit 220 so that it can be used by other units of the suspicious person detection unit 210. In addition, when there are a plurality of person areas in one monitoring image, the person area detection unit 211 extracts all the person areas, creates person information for each, and stores the person information in the storage unit 220. At this time, the person identifier is set to a different value for each person information.

The face orientation detection means 212 detects the face orientation based on the front orientation of the person shown in the person image.
FIG. 2 shows an orthonormal coordinate system for defining the face orientation of a person. In this orthonormal coordinate system (X, Y, Z), the origin O is set to the approximate center of the head of the person 2 whose face orientation is to be estimated. The Z axis is a straight line connecting the head of the person 2 and the imaging unit 100. The X axis passes through the origin O and is set to a vertical straight line orthogonal to the Z axis. Similarly, the Y axis passes through the origin O and is set to a horizontal straight line orthogonal to the Z axis.
In this coordinate system, the face direction of the person 2 is a state in which the person 2 is directly facing the imaging unit 100 (that is, the person 2 is facing the front of the face to the imaging unit 100). It is represented by a set (ψ, θ) of a yaw angle ψ and a pitch angle θ. The yaw angle ψ represents the rotation angle in the left-right direction with respect to the face direction of the person 2 in the directly-facing state (that is, the rotation angle in the YZ plane with respect to the Z axis). The pitch angle θ represents a vertical rotation angle (that is, a rotation angle in the XZ plane with respect to the Z axis) with respect to the face direction of the person 2 in the face-to-face state. In the following, the yaw angle ψ and the pitch angle θ are expressed in radians, and the rightward direction or the downward direction is positive, respectively.

  The face orientation detection unit 212 reads out person information from the storage unit 220 and extracts a region corresponding to the head from a person image included in the person information. The face orientation detection unit 212 creates an edge image obtained by extracting an edge from a person image, for example. Then, the face direction detection unit 212 can detect an elliptical edge distribution that approximates the contour shape of the head from the edge image, and extract a region surrounded by the edge distribution as a head region. In this case, the face direction detection unit 212 can detect an elliptical edge distribution using, for example, a generalized Hough transform. Alternatively, the face orientation detection unit 212 performs template matching on a person image or an edge image of the person image using a head template prepared in advance, and the degree of matching obtained as a result of the template matching is a predetermined value. The above region can be extracted as a head region. Furthermore, the face direction detection unit 212 may set a region considered to correspond to the head of the person area extracted by the person region detection unit 211 as a head region (for example, an average value from the upper end of the person region). A region corresponding to the size and shape of the head is cut out to be a head region).

  When the head region is extracted, the face direction detection unit 212 extracts face feature points that are characteristic portions of the face from the head region. Then, the face orientation detection unit 212 calculates the type of the extracted face feature point and position information on the head region (for example, a two-dimensional coordinate value having the origin at the upper left corner of the head region). For example, the face orientation detection unit 212 extracts facial feature points such as both eye corners, both eye region centers, nose tips, mouth points, and mouth corner points. The face direction detecting means 212 can use various known methods for extracting face feature points. For example, the face direction detection unit 212 performs edge extraction processing on the head region and extracts edge pixels having a large luminance difference from surrounding pixels. Note that such edge pixels in the head region may be extracted from the edge image of the person image. Then, the face orientation detection unit 212 checks whether or not the feature amount obtained based on the position and pattern of the edge pixel satisfies a predetermined condition for a characteristic part of the face such as eyes, nose, and mouth. By specifying the position of each part, each facial feature point can be extracted. In addition, instead of performing edge extraction processing to extract edge pixels, the face direction detection means 212 performs Gabor conversion processing or wavelet conversion processing to extract pixels having large local changes in different spatial frequency bands. May be. Further, the face direction detection unit 212 may extract face feature points by performing template matching between a template corresponding to each part of the face and a head region to specify the position of each part of the face.

When the face feature points are extracted, the face direction detection unit 212 checks the two-dimensional distribution of each face feature point and detects the face direction. That is, in the present embodiment, the face direction can be said to be the amount of deviation of the feature points from the face facing front (facing state).
The relationship between the face orientation and the face feature points will be described with reference to FIG. FIG. 3A shows a face image 310 of the target person when the target person is facing the imaging unit 100 (that is, when the target person is in a directly-facing state). On the other hand, FIG. 3B shows a face image 330 of the target person when the target person is facing away from the imaging unit 100. As illustrated in FIG. 3A, when the target person faces the front of the face with respect to the imaging unit 100, the nose tip 311 that is the apex of the nose exists on the midline 320 of the face. In addition, left and right eye corners 312 and 313, left and right mouth corner points 314 and 315, and the like can be observed.
On the other hand, as shown in FIG. 3B, when the target person faces away from the imaging unit 100, the nose tip 331 protrudes from the eyes and mouth, so that the yaw angle ψ increases. , Away from the midline 340. Further, the left and right eye corners 332 and 333 and the left and right mouth corner points 334 and 335 are hidden and become invisible as the target person rotates the face. For example, when the target person rotates his face a little more to the right than shown in FIG. 3B, the right eye corner 333 or the right mouth corner point 335 is not reflected on the face image 330, and the face feature points cannot be detected. It becomes possible. As described above, the orientation of the face of the target person with respect to the imaging unit 100 can be examined from the position of the nose tip with respect to the midline of the face and the type of face feature points that cannot be detected.

  Therefore, the face direction detection unit 212 obtains, for example, a straight line connecting the midpoint and the mouth point of both eyes as the midline of the face. Then, the face direction detection unit 212 determines whether the nose tip that is the apex of the nose is on the left or right side of the midline and the distance from the midline. Further, the face orientation detection means 212 preliminarily checks the relationship between the direction of the median line and the distance from the median line to the nose apex as the information on the facial feature point arrangement, and the orientation of the face including at least the front face. Is stored in the storage unit 220 in advance. The storage unit 220 functions as an arrangement information storage unit by storing facial feature point arrangement information. Then, the face direction detection unit 212 refers to the look-up table and determines the direction of the calculated midline and the face direction corresponding to the distance from the midline to the nose tip.

The face orientation detection unit 212 may detect the face orientation based on the positional relationship between the extracted face feature points and the corresponding face feature points in the three-dimensional shape model representing the three-dimensional shape of the human face. . In this case, a standard three-dimensional shape model imitating a human head (for example, a wire frame model or a surface model) is prepared in advance and stored in the storage unit 220. Further, the storage unit 220 also stores the positions of 3D face feature points such as both eye corners, nose tips, and mouth points as information on the arrangement of the face feature points of the three-dimensional shape model.
When the face feature point is extracted from the head region of the person image, the face direction detecting unit 212 adjusts the face direction according to a predetermined rotation amount, translation amount, or enlargement / reduction ratio for the three-dimensional shape model, Virtually projecting onto a plane parallel to the image plane of the imaging optical system of the imaging unit 100, the position of the 3D face feature point on that plane is obtained. Then, the face orientation detection unit 212 obtains the total sum of positional deviation amounts between each face feature point extracted from the head region of the person image and the corresponding feature point among the projected 3D face feature points. The face orientation detection unit 212 changes the rotation amount, the translation amount, or the enlargement / reduction ratio, repeats the above procedure, and obtains the face orientation of the three-dimensional shape model when the sum of the positional deviation amounts is minimized. Then, the face direction detecting means 212 can obtain the set (ψ, θ) of the yaw angle ψ and the pitch angle θ from the face direction of the three-dimensional shape model. In this case as well, the face orientation is calculated from the positional relationship of the feature points with reference to the front orientation, that is, it can be said that the amount of deviation of the feature points from the face facing the front. The storage unit 220 functions as a feature point arrangement information storage unit by storing a three-dimensional shape model and a 3D face feature point.
The face direction detection unit 212 projects each face feature point extracted from the head region of the person image into the three-dimensional space, and then the amount of positional deviation from the 3D face feature point on the three-dimensional shape model is minimized. The three-dimensional shape model may be subjected to processing such as rotation and enlargement / reduction to determine the face orientation of the three-dimensional shape model.

  When there are face feature points that cannot be extracted from the head region of the person image, the face direction detection unit 212 may detect the face direction based on the type of the face feature points that could not be extracted. For example, when the face feature point corresponding to the right eye corner cannot be extracted, the face direction detection means 212 determines that the person is greatly turning away from the right direction, and the value of the yaw angle ψ Is set to a value larger than an angle, for example, π / 4, at which it is considered that face feature points cannot be extracted. Similarly, when the face feature point corresponding to the left eye corner cannot be extracted, the face direction detection unit 212 determines that the person is greatly facing away from the left and determines the value of the yaw angle ψ as, for example, an absolute value. The value is set to a negative value larger than π / 4.

Furthermore, because the person in the person image is wearing sunglasses or a mask, the eyes and mouth are hidden and face feature points may not be extracted. Further, even when the person is face down, the eyes and mouth are not shown on the image, and face feature points cannot be extracted. However, when the face is turned down, the ratio of the top of the head to the head region increases. Therefore, when face feature points cannot be extracted from the head region at all, or when feature points below the nose cannot be extracted, the face direction detection means 212 calculates the ratio of skin color pixels in the head region. The face orientation detection unit 212 sets larger values for the pitch angle θ and the yaw angle ψ as the ratio decreases. For example, when the ratio becomes a value (for example, 20%) corresponding to a case where a person in the person area is considered to be face down, the face direction detection unit 212 sets the pitch angle θ to π / 4. If the ratio of the skin color pixels in the head region is smaller than the value, the face direction detection means 212 makes the value of the pitch angle θ larger than π / 4. On the contrary, if the ratio is larger than the value, the face direction detecting means 212 makes the value of the pitch angle θ smaller than π / 4. The signs of the yaw angle ψ and the pitch angle θ are determined based on the barycentric positions of all skin color pixels in the head region. Since the skin color pixel is considered to correspond to the face surface region, for example, if the center of gravity of the skin color pixel is below the center of gravity of the entire head region, the pitch angle θ is set to a positive value. If the center of the flesh color pixel is on the right side of the center of gravity of the entire head region, the yaw angle ψ is set to a positive value.
When the face orientation detection means 212 obtains the values of the yaw angle ψ and the pitch angle θ representing the face orientation, the face orientation detection means 212 associates them with the corresponding person information as face orientation information and stores them in the storage unit 220.

  The tracking unit 213 detects, for each person information, the traveling direction of the person shown in the person image included in the person information. As will be described later, even if the face is turned away from the imaging unit, the behavior of the person is not suspicious as long as the orientation of the face is in the traveling direction. Therefore, in order to determine whether or not the person is facing the traveling direction, the tracking unit 213 obtains the traveling direction of the person. For this purpose, the tracking unit 213 examines the position change on the monitoring image of the person area considered to be due to the same person over a plurality of monitoring images continuously acquired at predetermined time intervals.

  With reference to FIG. 4, detection of a person's traveling direction will be described. The tracking means 213 is extracted from the center of gravity position 401 of the person area 400 extracted from the latest monitoring image and the monitoring image one frame before which is considered to be the same person as the person shown in the person area. The difference in the center of gravity position 411 of the person area 410 is obtained as the traveling direction vector 420, and the temporal movement of the corresponding person area is tracked. Then, the tracking unit 213 sets the direction of the traveling direction vector 420 as the traveling direction of the person. Further, the tracking unit 213 sets a value obtained by dividing the length of the traveling direction vector 420 (that is, the distance between the center of gravity positions) by the difference between the acquisition timings of the two monitoring images as the moving speed of the person.

  When there are a plurality of person regions extracted from the latest monitoring image or the monitoring image one frame before or several frames before, the tracking unit 213 uses a variety of known tracking processes, A person area corresponding to the same person in the monitoring image can be tracked and specified. For example, the tracking unit 213 extracts, from among the person regions extracted from the latest monitoring image, the position of the center of gravity of the person region of interest and all the extracted from the monitoring image acquired one frame before. The distance from the center of gravity of the person area is obtained. The tracking unit 213 can set the person area having the shortest distance between the centers of gravity as the person area of the same person as the person shown in the person-of-interest area. Alternatively, when the traveling direction and the moving speed are obtained for the person area of interest extracted from the monitoring image acquired one frame before, the tracking unit 213 determines that in the latest monitoring image from the traveling direction and the moving speed. The position of the person may be estimated. In this case, the tracking unit 213 selects the person area that is the closest to the estimated position from the person areas extracted from the latest monitoring image, and is the person who is the same person as the person shown in the person area of interest in the monitoring image one frame before. Can be an area.

When there is a person area that is considered to be the same person in the monitoring image acquired one frame before, the tracking means 213 indicates that the person areas belong to the same person. The person identifier assigned to the target person area is set to the same value as the person identifier assigned to the corresponding person area in the monitoring image acquired one frame before.
In addition, the tracking unit 213 stores, in the storage unit 220, the normalized traveling direction vector obtained by normalizing the obtained traveling direction vector length to 1 and the moving speed as traveling direction information in association with the corresponding person information.

  The suspicious behavior determination unit 214 determines whether or not a person who is captured in a focused person area (hereinafter referred to as a focused person) is taking suspicious behavior among the person areas extracted from the latest monitoring image. If it is determined that suspicious behavior is taken, the number of suspicious behaviors N of the person of interest is incremented by one. The number of suspicious behaviors N is a degree indicating that a predetermined person is performing suspicious behaviors over a plurality of monitoring images sequentially set with an initial value of 0, and corresponds to the same person over a plurality of monitoring images. Calculated for each person information.

Accordingly, first, the suspicious behavior determination unit 214 calculates a camera distance C _avoid that represents the degree to which the viewer is facing away from the imaging unit 100 as an index indicating that the viewer is taking suspicious behavior. The camera distance C _avoid is calculated by the following equation, for example.
C _avoid = α | ψ | + β | θ | (1)
Here, α and β are weighting factors for the yaw angle ψ and the pitch angle θ, respectively. The weighting factors α and β can be optimized according to the installation state of the imaging unit 100. For example, when the imaging unit 100 is installed on the ceiling of the passage and the direction of the passage is photographed from above, the yaw angle ψ obtained with respect to the direction of the face of the passerby coming from the back side of the passage toward the imaging unit 100 (Represents the orientation of the face in the left-right direction) does not depend on the distance between the passerby and the imaging unit 100. On the other hand, the pitch angle θ (representing the vertical direction of the face) obtained for the face direction of the passer-by is small even if the passer-by is always in the same direction. It is thought that it grows as it becomes. Therefore, in such a case, it is considered that the yaw angle ψ accurately represents whether or not the passerby is facing away from the imaging unit 100. Therefore, the weighting factor α is set larger than the weighting factor β (for example, α = 1, β = 0.7) so that the specific gravity of the yaw angle ψ occupying the camera telephotometry is larger than the specific gravity of the pitch angle θ. . Note that the weighting factors α and β may be set to the same value (for example, α = β = 1).
Further, the camera telemetry C _avoid may be calculated using another formula instead of using the above formula (1). For example, the camera distance C _avoid may be the sum of the square value of the yaw angle ψ and the square value of the pitch angle θ.

  Next, the suspicious behavior determination unit 214 is prohibited from determining that the behavior of the subject is suspicious behavior, that is, sets the value of the normal behavior index E indicating that the behavior of the subject is normal behavior. This normal behavior index E has a value of “0” in principle, and even if the viewer is facing away from the imaging unit 100, it is determined that the behavior is not suspicious, that is, the behavior is normal behavior. It is “1” when it is determined that there is. As an example, the suspicious behavior determination unit 214 determines whether or not the person of interest is facing the traveling direction. If the viewer is facing the direction of travel, the viewer is not considered to be intentionally turning away from the imaging unit 100, and thus the behavior of the viewer does not correspond to the suspicious behavior, that is, the suspicious behavior is determined. Is prohibited. Therefore, the suspicious behavior determination unit 214 calculates a unit direction vector representing the face orientation of the viewer on the monitoring image from the yaw angle ψ and the pitch angle θ of the face orientation information corresponding to the viewer. Then, the suspicious behavior determination unit 214 calculates the inner product of the unit direction vector and the normalized progress direction vector stored in the storage unit 220 in association with the personal information of the viewer. If the inner product value is equal to or greater than a predetermined value close to 1 (for example, 0.98, the angle formed by the bi-directional vectors is less than about 10 degrees), the suspicious behavior determination unit 214 determines that the person of interest is facing the traveling direction. The normal action index E is set to 1. The suspicious behavior determination unit 214 determines whether or not the shooting direction of the imaging unit 100 and the shooting distance such as the focal length are known in order to check whether the face direction and the traveling direction of the person of interest match. A three-dimensional unit direction vector obtained by projecting the normalized traveling direction vector onto the orthonormal coordinate system (X, Y, Z) of FIG. 2 by estimating the distance from the imaging unit 100 to the subject of interest using the imaging conditions. You may ask for. Similarly, the suspicious behavior determination unit 214 obtains a unit direction vector on the orthonormal coordinate system (X, Y, Z) from the face orientation information. Then, the suspicious behavior determination unit 214 may calculate an inner product value of the three-dimensional unit direction vector obtained from the traveling direction vector and the three-dimensional unit direction vector obtained from the face direction information.

  Further, the suspicious behavior determination unit 214 may use, as a condition for prohibiting determination of the behavior of the viewer as suspicious behavior, whether or not the viewer is facing another person. When the person of interest is talking to another person, the person of interest is not considered to be intentionally turning away from the imaging unit 100, and thus the action of the person of interest does not correspond to the suspicious behavior. Suppose that the behavior of the viewer is normal behavior. In this case, for example, when the extension destination of the unit direction vector obtained from the face orientation information of the viewer overlaps with another person area, the suspicious behavior determination unit 214 faces the other person. Therefore, the normal action index E is set to 1. Note that the suspicious behavior determination unit 214 determines that the target person is the other person only when the extension of the unit direction vector obtained from the face direction information of the target overlaps the head area of the other person areas. The normal action index E may be set to 1. Furthermore, when any head region overlaps with the extension destination of the unit direction vector obtained from the face orientation information of the viewer, the suspicious behavior determination means 214 determines the face orientation of the other person corresponding to the head region and the viewer's face. The normal action index E may be set to 1 when the face direction is substantially opposed. Specifically, the suspicious behavior determination unit 214 obtains a unit direction vector of the face direction of the other person and obtains an inner product value with the unit direction vector of the face direction of the viewer. When the inner product value is close to −1 (for example, −0.98 or less), the suspicious behavior determination unit 214 sets the normal behavior index E to 1.

When the normal action index E is 0, the suspicious behavior determination unit 214 determines whether or not the camera distance C _avoid is equal to or greater than a predetermined threshold Th1. When the camera distance C _avoid is equal to or greater than the threshold value Th1, the suspicious behavior determination unit 214 identifies the number of suspicious behaviors N at the time of the previous determination of the viewer based on the person identifier, reads out the suspicious behavior from the storage unit 220, Add 1 to the number of times N. Then, the suspicious behavior determination unit 214 stores the suspicious behavior count N of the viewer in the storage unit 220 in association with the personal information of the viewer. Note that the threshold value Th1 is a value for determining that the face orientation deviates a predetermined amount or more from a state in which the face orientation is facing the imaging unit 100, and is facing away from the imaging unit 100. It is a criterion for judging this. This threshold value Th1 can be set to a value corresponding to the face orientation when, for example, either the corner of the eyes of the subject or the corner points of the mouth are no longer displayed on the monitoring image. Alternatively, the boundary value obtained using a discriminant analysis or the like by photographing a person with the camera deliberately facing away from the camera in advance and obtaining a camera distance C _avoid in each state. May be the threshold Th1.
On the other hand, when the normal action index E is 1 or when the camera distance C _avoid is less than the predetermined threshold Th1, the suspicious action determination unit 214 does not add the number of suspicious actions N read from the storage unit 220, and pays attention as it is. The information is stored in the storage unit 220 in association with the person information of the person.

The suspicious person determination means 215 determines whether or not the person of interest is a suspicious person based on the number of suspicious actions N associated with the person of interest. For example, the suspicious person determination unit 215 first detects a person area corresponding to the person of interest in a plurality of temporally continuous monitoring images (hereinafter referred to as frame-in), and then determines the person area corresponding to the person of interest. If the viewer continues to take suspicious behavior until no longer being detected (hereinafter referred to as “frame out”) or during the time corresponding to 90% or more of the time until the latest monitoring image acquisition, A suspicious person. Specifically, when the number of suspicious actions N reaches 90% or more of the total number of monitoring images acquired from frame-in to frame-out or the latest monitoring image acquisition time, the suspicious person determination means 215 Identify the person of interest as a suspicious person. However, when the suspicious person determination unit 215 determines whether or not the person is a suspicious person based on the period from the frame-in to the acquisition of the latest monitoring image, the suspicious person determination unit 215 determines whether or not the person is a suspicious person after the period exceeds a predetermined time. It is preferable to carry out. This predetermined time is the minimum time required to determine that a person who passes or stays in the monitoring area is a suspicious person, and may be set according to the size of the monitoring area to be imaged. In the present embodiment, for example, it is set to 1 minute.
The suspicious person determination unit 215 outputs the person information regarding the person of interest identified as the suspicious person to the output unit 300.

The storage unit 220 includes a semiconductor memory such as a ROM and a RAM, or a magnetic recording medium and its access device or an optical recording medium and its access device. The storage unit 220 stores in advance a computer program and various parameters for controlling the image monitoring apparatus 10. The storage unit 220 stores the person information (that is, the person identifier, the person image, the face direction information, the traveling direction information, the number of suspicious actions, etc.) for each frame during the operation of the image monitoring apparatus 10. Furthermore, the storage unit 220 may store a face image of the person cut out from the person image (that is, an image including the head region and its surroundings) in association with the person identifier. In particular, the storage unit 220 is the face image facing the most front of the series of person images corresponding to the person determined to be a suspicious person by the suspicious person determination unit 215 (for example, the face image with the smallest camera distance C _avoid ). ) May be stored in association with the person identifier. Alternatively, the storage unit 220 may store a face image in which all face feature points have been detected in association with the person identifier. It should be noted that processing such as face image segmentation and association with a person identifier is performed by a control unit (not shown) that operates on the microprocessor constituting the image processing unit 200 and controls the entire image monitoring apparatus 10.
Furthermore, the storage unit 220 may store various conditions regarding the imaging unit 100 (for example, installation height, focal length, and shooting direction) in order to accurately determine the traveling direction of the person.

The output unit 300 is connected to the image processing unit 200 and includes a display device such as a liquid crystal display or an organic EL display, a speaker, or a vibration device. When the suspicious person detection unit 210 of the image processing unit 200 detects a suspicious person, the output unit 300 notifies the monitoring person by displaying a message to that effect or issuing an alarm. Further, the output unit 300 may display the person information related to the suspicious person or the face image associated with the person information.
Note that the output unit 300 may be a monitoring device that is connected to the image processing unit 200 via a public communication line or a local area network and is arranged separately from the image processing unit 200.

The operation of the suspicious person detection process of the image monitoring apparatus 10 to which the present invention is applied will be described below with reference to the flowchart shown in FIG. The operation flow described below is controlled by the control unit (not shown).
First, the image monitoring apparatus 10 acquires a monitoring image obtained by capturing a monitoring area using the imaging unit 100, and sends the monitoring image to the suspicious person detection unit 210 of the image processing unit 200 (step S100). Next, the person area detection unit 211 of the suspicious person detection unit 210 detects a person area from the acquired monitoring image (step S110). Then, the person area detection unit 211 creates person information (including a person image, person area information, and a person identifier) from the extracted person area. Next, the image processing unit 200 determines whether one or more person areas have been detected (step S120). If no person area is detected, the image monitoring apparatus 10 ends the suspicious person detection process.

  On the other hand, when one or more person areas are detected and person information is created in step S120, the suspicious person detection unit 210 sets the person information to be noticed from the created person information (step S130). ). Then, the face orientation detection unit 212 of the suspicious person detection unit 210 detects the orientation of the face of the viewer corresponding to the target person information based on the distribution of the facial feature amount extracted from the person image included in the target person information. To do. The face orientation detection unit 212 stores the detected orientation of the face of the subject in the storage unit 220 in association with the subject person information as face orientation information (step S140). In addition, the tracking unit 213 of the suspicious person detection unit 210 identifies a person area detected from the monitoring image acquired one frame before and that is considered to be the same person as the person of interest, and that one frame before The person information obtained for the monitoring image is associated with the person information of interest. Then, the tracking unit 213 detects the traveling direction of the viewer based on the difference between the position of the person area of the person in the latest monitoring image and the position of the person area of the same person as the person of interest one frame before ( Step S150). The tracking unit 213 stores the traveling direction information indicating the traveling direction in the storage unit 220 in association with the person-of-interest information.

Next, the suspicious behavior determination unit 214 of the suspicious person detection unit 210 calculates the camera distance C _avoid for the person of _interest based on the face orientation information (step S160). Further, the suspicious behavior determination unit 214 sets the value of the normal behavior index E based on the degree of coincidence between the orientation of the face of the viewer and the travel direction calculated from the face orientation information of the viewer, the traveling direction information, and the like ( Step S170).
Thereafter, the suspicious behavior determination unit 214 determines whether or not the person of interest is taking suspicious behavior (step S180). Specifically, the normal action index E does not indicate that the action of the viewer is normal action (that is, E = 0 in the present embodiment), and the camera distance C _avoid corresponds to turning away. If it is equal to or greater than the threshold value Th1, the suspicious behavior determination unit 214 determines that the person of interest is taking suspicious behavior. Then, the suspicious behavior determination unit 214 reads the suspicious behavior count N associated with the person identifier corresponding to the subject of interest from the storage unit 220 and updates it by adding one (step S190). On the other hand, in step S180, if the normal action index E indicates normal action (that is, E = 1 in the present embodiment), or if the camera distance C _avoidance is less than the threshold value Th1, the suspicious action determination unit 214 The suspicious action count N is updated without being added, and the process proceeds to step S200.

  Next, the suspicious person determination unit 215 of the suspicious person detection unit 210 compares the number of suspicious actions N of the person of interest with a predetermined number (step S200). The predetermined number of times is, for example, 90% of the total number of frames from when the subject framed in until the present time. If the number of suspicious behaviors N of the viewer is equal to or greater than the predetermined number of times, the suspicious person determination unit 215 identifies the viewer as the suspicious person and notifies the suspicious person that the suspicious person has been detected via the output unit 300 (step S21). S210). On the other hand, if the number of suspicious actions N of the viewer is less than the predetermined number in step S200, the suspicious person determination unit 215 determines that the viewer is not a suspicious person at the present time, and proceeds to step S220.

Thereafter, the suspicious person detection unit 210 determines whether or not personal information that has not been noticed remains among the personal information created from the latest monitoring image (step S220). If the person information that has not been noticed remains, the suspicious person detection unit 210 returns the control to step S130, sets the person information to be noticed from the person information that has not yet been noticed, and steps S140 to S220. Repeat the process.
On the other hand, in step S220, when there is no person information that is not focused on, the image monitoring apparatus 10 ends the suspicious person detection process.
The image monitoring apparatus 10 repeats the above suspicious person detection process at regular time intervals, for example, every time a monitoring image is acquired.

  As described above, the image monitoring apparatus 10 to which the present invention is applied detects the orientation of the face of the subject appearing in the monitoring image from the monitoring image obtained by capturing the monitoring area with the imaging unit 100, and the imaging unit When the face is turned away from 100 for a long time, the target person is identified as a suspicious person. Thus, since the image monitoring apparatus 10 can identify a suspicious person without setting a specific direction of a gaze target for determining a suspicious person in advance, when the image monitoring apparatus 10 is installed, It is not necessary to perform complicated settings depending on the installation environment. Furthermore, the image monitoring apparatus 10 checks whether the behavior of the subject person is a normal behavior based on the degree of coincidence between the direction of the subject's face and the traveling direction, and if the behavior is a normal behavior, Even if a person turns his face away from the imaging unit 100, by prohibiting the determination of suspicious behavior, it is possible to prevent a person who is performing normal behavior from being erroneously determined as a suspicious person. Therefore, the image monitoring apparatus 10 can specify a suspicious person with high accuracy.

  The preferred embodiments of the present invention have been described above, but the present invention is not limited to these embodiments. For example, a stereo camera may be used as the imaging unit 100, or a plurality of cameras that capture the same monitoring area at different angles may be used. In this case, the three-dimensional information of the monitoring area can be reconstructed from a plurality of monitoring images simultaneously captured by the stereo camera or the plurality of cameras using a known technique. For this reason, the tracking unit 213 can also determine the traveling direction of the person detected in the monitoring area three-dimensionally using a known technique, so that the traveling direction can be examined more accurately. The suspicious behavior determination unit 214 can also calculate the degree of coincidence between the direction of the person's face and the traveling direction more accurately when calculating the normal behavior index E. Furthermore, the suspicious behavior determination means 214 can accurately determine whether or not the person of interest is facing another person.

Further, in the above-described embodiment, the suspicious behavior determination unit 214 is configured to increment the number of suspicious behaviors by one when the camera distance C _avoid is _equal to or greater than the predetermined threshold Th1 as an index of suspicious behavior. However, instead, the suspicious behavior determination unit 214 may accumulate the camera distance C _avoid itself when the normal behavior index E is 0 (when normal behavior is not recognized). The suspicious person determination unit 215 determines that the corresponding person is a suspicious person when the accumulated value exceeds a predetermined threshold. Alternatively, the suspicious behavior determination unit 214 sets the value of the normal behavior index E for the viewer first, calculates the camera distance C _avoid only when the behavior of the viewer is not recognized as normal behavior, In this case, the camera distance C _avoid may be set to zero.
Further, the suspicious behavior determination unit 214 may store the duration of the suspicious behavior instead of storing the number of suspicious behaviors for each person information. In this case, the suspicious person determination unit 215 determines that the target person is a suspicious person when the duration of the suspicious action exceeds a predetermined time (for example, one minute).

Further, the suspicious behavior determination unit 214 extracts a face image of the person from the person image corresponding to the person when it is determined that the person of interest has taken the suspicious action, and stores it in association with the person together with the camera distance C _avoidance. When the suspicious person determination unit 215 determines that the person is a suspicious person, the face image stored in the storage unit 220 in association with the person is associated with the smallest camera distance C _avoidance. Detection means for detecting the detected face image may be further provided. When the camera distance C _avoid is small, it means that the person is facing the image capturing unit 100, and the face closest to the image capturing unit 100 in the frame indicating the minimum value. This is because it can be expected to be an image. The face image detected by the detection means may be used for tracking a suspicious person, automatically detecting a suspicious person by comparison with a monitoring image input from the image capturing unit 100, and detecting a person coming again.

Further, the suspicious person determining means 215 extracts the face image or the person image, the color histogram in the face image, the position of the face feature point, and the like from the person information corresponding to the target person determined to be the suspicious person. The feature amount may be registered as suspicious person feature information for collation processing and stored in the storage unit 220. The suspicious person characteristic information may be used for tracking a suspicious person, automatically detecting a suspicious person by comparison with a monitoring image input from the imaging unit 10, detecting a revisiting person concerned, and the like.
Further, when the image monitoring apparatus to which the present invention is applied determines whether or not one person who stays at a certain point such as an ATM operator is a suspicious person, the above tracking means may be omitted. In this case, if the subject turns away from the imaging unit, the imaging unit is placed in the vicinity of a place where the subject should pay attention for work, such as an ATM operation screen, so that it can be determined that the behavior is suspicious. It is preferable to install.
As described above, various modifications can be made within the scope of the present invention according to the embodiment to be implemented.

It is a schematic block diagram of the image monitoring apparatus to which this invention is applied. It is a figure which shows the coordinate system which prescribes | regulates the direction of a target person's face. (A) is a schematic diagram of a face image of the target person when the target person is facing the imaging unit, and (b) is a target when the target person is facing away from the imaging unit It is the schematic of a person's face image. It is a figure which represents typically the advancing direction of a subject person. It is a flowchart which shows the operation | movement regarding a suspicious person detection process of the image monitoring apparatus to which this invention is applied.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Image monitoring apparatus 100 Image pick-up part 200 Image processing part 210 Suspicious person detection part 211 Person area detection means 212 Face direction detection means 213 Tracking means 214 Suspicious action determination means 215 Suspicious person determination means 220 Storage part 300 Output part

Claims (4)

An image monitoring apparatus for detecting a suspicious person from a monitoring image obtained by photographing a person in a monitoring area,
An imaging unit for sequentially acquiring the monitoring images;
Person area detecting means for detecting a person area from the monitoring image;
Arrangement information storage means for storing the feature point arrangement of the face facing the front in advance;
A feature point of a face is extracted from the person area, and a feature point position shift amount with respect to a front-facing face is detected by comparison with the front-facing feature point arrangement stored in the arrangement information storage unit. Face orientation detection means;
Suspicious behavior determination means for determining the person area as suspicious behavior if the amount of shift of the feature point position is equal to or greater than a predetermined amount corresponding to a state of facing away from the face
Tracking means for tracking a corresponding person area in the plurality of sequentially acquired monitoring images;
A suspicious person determination means for determining a suspicious person based on the number of times the person area tracked by the tracking means is determined to be the suspicious behavior;
An image monitoring apparatus comprising: The face orientation detection means further detects the face orientation on the monitoring image,
The tracking means detects a traveling direction of a person corresponding to the person area on the monitoring image from the person area detected for each of the plurality of monitoring images;
The suspicious behavior determination means does not determine that the person area is suspicious when the face direction substantially matches the traveling direction in the person area of any one of the plurality of monitoring images. Item 8. The image monitoring apparatus according to Item 1. The face orientation detection means further detects the face orientation on the monitoring image,
The suspicious behavior determination means may be configured to perform suspicious behavior for a person area in the person area of any one of the plurality of monitoring images when there is another person area on the extension of the face direction of the person. The image monitoring apparatus according to claim 1, wherein the image monitoring apparatus is not determined to be The arrangement information storage means stores in advance a three-dimensional shape model of a human face as the feature point arrangement of the face facing the front,
The face orientation detection means is configured to detect the orientation of the face of the three-dimensional shape model when the position of the feature point of the face extracted from the person region and the position of the corresponding feature point in the three-dimensional shape model most closely match each other. Is detected as a shift amount of the feature point position with respect to the face facing forward.
The image monitoring apparatus according to claim 1.

Images