JP6562437B1 - Monitoring device and monitoring method - Google Patents

Abstract

To provide a monitoring apparatus capable of more accurately detecting the “line-of-sight direction” of a person to be monitored by means of a simpler measuring means than before. An imaging unit, a monitoring target specifying unit that can recognize and specify a monitoring target person, a skeleton extracting unit that extracts a skeleton of the monitoring target person, and a monitoring target area within each feature point A feature point position specifying unit 125 that specifies a three-dimensional position in the image, and a line-of-sight direction detection unit 126 that detects the line-of-sight direction of the person to be monitored. And a point corresponding to the position of the nose from the midpoint of the base connecting the points corresponding to the positions of both ears of the triangle formed by connecting the three gaze direction detection feature points corresponding to the position of the nose It is assumed that the monitoring device 100 detects the direction toward the apex of the triangle as the line-of-sight direction of the monitoring target person in the three-dimensional space. [Selection] Figure 7

Description

  The present invention relates to a monitoring device and a monitoring method.

  Conventionally, a monitoring device using a monitoring camera has been used for the purpose of crime prevention and disaster prevention, or for the purpose of nursing or nursing care. This type of monitoring device is generally configured to include a monitoring camera and an information processing device. Then, for example, the monitoring target is monitored using an image taken by a monitoring camera installed in a predetermined monitoring target area such as a store, a commercial facility, or a hospital or a nursing facility.

  With such a monitoring device, the “special motion” consisting of the movement of the head, torso, and limbs of the “monitoring person” in the image is detected from the image data, and the response to the “monitoring person” Whether or not is necessary is determined mechanically (see Patent Document 1).

  In such a monitoring operation by the monitoring device, it is necessary to deal with the monitoring target in particular by detecting the “sight line direction” of the person to be monitored among the movements of the body parts constituting the “specific motion”. Often useful in determining the degree.

  As a means for detecting the “line-of-sight direction” of the “person to be monitored” in the image by the monitoring device, for example, “matching” the “face template of various angles” and the face of the person to be monitored in the monitoring image By doing so, the gaze direction detecting means (see Patent Document 2) for detecting the gaze direction from the template that approximates the face, or “the whole head of the person” and “the position of the eyes and nose” of the person to be monitored A gaze direction detecting means (Patent Document 3) and the like are proposed in which the direction of the face is obtained from these “positional relationships” and the gaze direction is determined from this.

Japanese Patent No. 5899506 JP 2008-288707 A JP 2018-148402 A

  However, it is necessary for the gaze direction detection means of Patent Document 2 to prepare a large amount of basic data (“face templates of various angles”) in advance.

  On the other hand, the gaze direction detecting means of Patent Document 3 includes the three-dimensional position information of the entire outline of “the whole head of a person” and the three-dimensional information of “the position of eyes and nose” in order to accurately detect the gaze direction. It is necessary to grasp both, and it is essential to install expensive distance measuring means such as a 3D camera and a high-precision distance sensor.

  It is an object of the present invention to provide a monitoring apparatus that can accurately detect the “line-of-sight direction” of a person to be monitored with a minimum amount of information processing by a simpler measuring means than before.

  The present invention solves the above problems by the following means.

  (1) An imaging unit that captures a monitoring target area, a monitoring target identification unit that can recognize and identify a monitoring target person in a monitoring image captured by the imaging unit, and both ears and nose of the monitoring target person A skeleton extraction unit that extracts the skeleton of the person to be monitored, which is composed of skeleton lines that connect a plurality of feature points including three gaze direction detection feature points corresponding to the position of the skeleton, and the skeleton extraction unit A feature point position identifying unit that identifies a three-dimensional position of the feature point extracted in the monitoring target area, and a gaze direction detecting unit that detects a gaze direction of the monitoring target person. The direction detection unit is a point corresponding to the position of the nose from the midpoint of the base connecting the points corresponding to the positions of both ears of the triangle formed by connecting the characteristic points for detecting the gaze direction. The direction toward the apex is set to 3 of the monitored person. Detecting a gaze direction in the original space, the monitoring device.

  (2) A specific motion detection unit and a suspicious motion determination unit are further provided, wherein the specific motion detection unit detects the specific motion of the person to be monitored from the amount of change in the line-of-sight direction, and the suspicious motion determination unit Compares the specific action detected by the specific action detector with a reference suspicious action registered in advance, and determines and outputs the suspicious degree of the specific action of the person to be monitored, (1) The monitoring device described in 1.

  (3) A specific motion detection unit, a suspicious motion determination unit, and a torso direction detection unit are further provided, and the feature point further includes two locations corresponding to the positions of the right shoulder and the left shoulder of the person to be monitored. A fuselage direction detection feature point is included, and the fuselage direction detection unit is based on an angle difference from the line-of-sight direction among two directions orthogonal to a straight line connecting the two fuselage direction detection feature points. A small direction is detected as a trunk direction in the three-dimensional space of the person to be monitored, and the specific motion detection unit is a fluctuation amount of the gaze direction or a fluctuation amount of a difference between the gaze direction and the trunk direction. The suspicious operation determination unit compares the specific operation detected by the specific operation detection unit with a pre-registered reference suspicious operation to identify the person to be monitored. Determine and output the suspicious degree of movement, (1) The placement of the monitoring device.

  (4) Coordinate setting for setting, based on the monitoring image, coordinates that can be specified by associating a position corresponding to the floor or ground in the monitoring image with an actual dimension in the three-dimensional space constituting the monitoring target area Any one of (1) to (3), wherein the feature point position specifying unit specifies the position of the feature point in the three-dimensional space based on the coordinates set by the coordinate setting unit. The monitoring device described in 1.

  (5) A monitoring image display device that displays the monitoring image is further provided, and the monitoring image display device displays a graphic indicating the line-of-sight direction of the monitoring target person in the monitoring image. ).

  (6) A monitoring imaging step in which the imaging unit images the monitoring target region, a monitoring target specifying step in which the monitoring target specifying unit recognizes and specifies a monitoring target person in the monitoring image captured by the imaging unit, and a skeleton The extraction unit includes a skeleton line connecting a plurality of feature points including three gaze direction detection feature points corresponding to the positions of the nose, right ear, and left ear of the person to be monitored. A skeleton extraction step of a person to be monitored for extracting a skeleton, and a feature point position specifying step in which a feature point position specifying unit specifies a three-dimensional position in the monitoring target region of the feature points extracted by the skeleton extraction unit; The line-of-sight direction detection unit corresponds to the position of the nose from the midpoint of the base formed by connecting points corresponding to the positions of both ears of the triangle formed by the three characteristic points for line-of-sight detection. Direction toward the apex of a certain triangle The detected as line-of-sight direction in the three-dimensional space monitored person, comprising a line-of-sight direction sensing step, the monitoring method.

  (7) The specific action detecting step for detecting the specific action of the person to be monitored from the amount of change in the line-of-sight direction is compared with the detected specific action and a reference suspicious action registered in advance. The monitoring method according to (6), further comprising: a suspicious action determination step of determining a suspicious degree of the specific action of the person to be monitored.

  (8) The feature points include two trunk direction detection feature points corresponding to the positions of the right shoulder and the left shoulder of the person to be monitored, and the two trunk direction detection feature points. A body direction detecting step of detecting a direction having a smaller angle difference from the line-of-sight direction as a body direction in the three-dimensional space of the person to be monitored, A specific action detecting step for detecting a specific action of the person to be monitored from a fluctuation quantity or a difference quantity of difference between the line-of-sight direction and the body direction, the detected specific action, and a pre-registered reference (6) The monitoring method according to (6), further including a suspicious operation determination step of comparing a suspicious operation and determining a suspicious degree of the specific operation of the person to be monitored.

  (9) Based on the monitoring image, the coordinate setting unit can specify the coordinates that can be specified by associating the position corresponding to the floor surface or the ground in the monitoring image with the actual dimension in the three-dimensional space constituting the monitoring target area. The monitoring method according to any one of (6) to (8), further including a coordinate setting step for setting.

  According to the present invention, it is possible to provide a monitoring device that can accurately detect the “line-of-sight direction” of a person to be monitored with a minimum amount of information processing by a simpler measuring means than before.

It is a block diagram which shows the structure of the monitoring apparatus of this invention. It is a flowchart which shows the flow of the monitoring method of this invention. It is a figure which shows the state in which the monitoring person in a monitoring image is recognized by the monitoring object specific | specification part with which the monitoring apparatus of this invention is provided. It is a figure which shows the state in which the skeleton is extracted from the monitoring subject person of FIG. 3 by the skeleton extraction part with which the monitoring apparatus of this invention is provided. It is a figure which shows the state in which the feature point of said frame | skeleton is overlaid on the coordinate containing three-dimensional information (depth information) by the feature point position specific | specification part with which the monitoring apparatus of this invention is provided. Basic positional relationship between three gaze direction detection feature points corresponding to the nose, right ear, and left ear positions, and two trunk direction detection feature points corresponding to the right shoulder and left shoulder positions It is drawing used for description. It is drawing explaining the basic positional relationship between the gaze direction detection feature point, the body direction detection feature point, the gaze direction, and the body direction. It is drawing explaining the aspect of the positional relationship of the gaze direction detection feature point, the fuselage direction detection feature point, the gaze direction, and the fuselage direction. It is a figure which shows an example of the attachment aspect of the imaging | photography part which comprises the monitoring apparatus of this invention. It is a figure which shows an example of the background image used in order that a coordinate setting part with which the monitoring apparatus of this invention is provided performs a coordinate setting. It is a figure which shows an example of the coordinate which said coordinate setting part set.

  The best mode for carrying out the present invention will be described below with reference to the drawings.

<Monitoring device>
FIG. 1 is a block diagram showing the configuration of the monitoring apparatus 100 of the present invention. In the following description, an example of a specific configuration is shown and described. However, these can be implemented with appropriate modifications within the technical scope of the present invention.

[overall structure]
The monitoring device 100 includes an imaging unit 110 and an arithmetic processing unit 120. Moreover, it is preferable to further include a monitoring image display device 130 for displaying a monitoring image. The monitoring device 100 can also be configured by separately arranging the photographing unit 110, the arithmetic processing unit 120, and the monitoring image display device 130 at positions separated from each other. Further, the monitoring device 100 integrates the photographing unit 110 and the arithmetic processing unit 120, or only a part of the photographing unit 110 and the arithmetic processing unit 120 as one monitoring device, and these devices and the monitoring image display device 130 may be connected by wire or wirelessly.

[Shooting Department]
The imaging unit 110 is a so-called surveillance camera. And this imaging | photography part 110 performs the monitoring imaging | photography step (S12) which image | photographs the three-dimensional space which comprises the monitoring object area | region.

  Note that the monitoring camera constituting the photographing unit 110 is a monocular camera that photographs a three-dimensional space constituting a monitoring target area as a two-dimensional image, for example, by providing the monitoring apparatus 100 with a coordinate setting unit 121 described later. Can be configured. In this case, it is not necessary to introduce an expensive 3D camera, and the introduction cost of the monitoring device 100 can be significantly reduced.

  For example, as shown in FIG. 9, the installation method of the monitoring camera that constitutes the imaging unit 110 may be attached to the end position of the indoor ceiling portion that becomes the monitoring target area. However, the location where the surveillance or the camera is attached can be changed as appropriate according to the size of the surveillance area, the angle of view of the camera lens, and the like.

Further, the photographing unit 110 can be configured to include an imaging element 111, a photographing lens 112, and an image processing unit 113. The image sensor 111 captures an image formed by the photographing lens 112 and sends it to the image processing unit 113. FIG. 9 schematically shows the optical axis (optical axis of image light) O of the photographing lens 112 and the photographing range (angle θ ₀ range). Note that the actual shooting range in each of the vertical and horizontal directions differs depending on the aspect ratio, so here the vertical shooting range on the shooting screen as shown in FIG. It was shown as a range of θ ₀ (vertical angle of view). The image processing unit 113 processes the data output from the image sensor 111 to form image data, and transmits the image data to the arithmetic processing unit 120.

[Operation processing unit]
The arithmetic processing unit 120 performs arithmetic processing necessary for monitoring the image data transmitted from the photographing unit 110. The arithmetic processing unit 120 can be configured using, for example, a personal computer, a tablet terminal, a smartphone, or the like. Alternatively, the arithmetic processing unit 120 can be configured as a dedicated device specialized for the monitoring operation. In any of these configurations, the arithmetic processing unit 120 includes hardware such as a CPU, a memory, and a communication unit. And the arithmetic processing part 120 which consists of such a structure can specifically perform the various operation | movement demonstrated below and the monitoring method by running a computer program (monitoring program).

  The arithmetic processing unit 120 is connected to the photographing unit 110 so that image data can be received from the photographing unit 110. This connection can be a wired connection using a dedicated communication cable or a connection by a wired LAN. Further, the connection is not limited to a wired connection, and may be a connection using various wireless communications such as a wireless LAN, a short-range wireless communication, and a mobile phone line. Note that the arithmetic processing unit 120 may be disposed in a remote place away from the image capturing unit 110 without being disposed in the vicinity of the image capturing unit 110.

  The arithmetic processing unit 120 includes at least a monitoring target specifying unit 123, a skeleton extracting unit 124, a feature point position specifying unit 125, and a gaze direction detecting unit 126. With this minimum configuration, it is possible to output information related to the line-of-sight direction of the person to be monitored as useful information for any monitoring device.

  The arithmetic processing unit 120 preferably further includes a coordinate setting unit 121, an input unit 122, a body direction detection unit 127, a specific motion detection unit 128, and a suspicious motion determination unit 129 as necessary. .

(Coordinate setting part)
The coordinate setting unit 121 is a coordinate that sets a identifiable coordinate by associating a position corresponding to the floor or ground in the monitoring image, which is a captured image captured by the imaging unit 110, with an actual dimension in the monitoring target region three-dimensional space. A setting step (S11) is performed. In the following, an embodiment in which the coordinates are set without requiring measurement of the distance to the monitoring object will be described as a specific example of the preferred embodiment. However, the photographing unit 110 includes a 3D camera, various distance sensors, and the like. Thus, it is possible to set the coordinates by obtaining the depth information in the monitoring target area, and in this case, the independent coordinate setting unit 121 is not necessarily an essential component.

(Input section)
The input unit 122 assists the function of the coordinate setting unit 121. The input unit 122 receives an external input regarding the distance (height h) from the imaging unit 110 to the floor or the ground. Further, the input unit 122 receives an input of the direction of the optical axis of the image light imaged by the imaging unit 110 (angle α with respect to the vertical direction (Z direction)). The input unit 122 may accept a value input via a general-purpose input device such as a keyboard provided in the arithmetic processing unit 120, or input a value input via a storage medium in which a setting value is stored. You may receive and the value input via a network etc. may be received.

(Monitoring target identification part)
During the monitoring operation, the monitoring target specifying unit 123 recognizes an arbitrary “person (monitoring target person)” in the monitoring image and performs a monitoring target specifying step (S13) for specifying this as a monitoring target person. Further, the monitoring target specifying unit 123 has a function of specifying a plurality of monitoring target persons individually in parallel at the same time.

(Skeleton extractor)
The skeleton extraction unit 124 performs a skeleton extraction step (S14) for extracting the skeleton of the monitoring target person composed of skeleton lines connecting a plurality of feature points (see FIG. 4).

(Feature point location)
The feature point position specifying unit 125 performs a feature point position specifying step (S15) for specifying the “position in the monitoring target region three-dimensional space” of the feature points constituting the skeleton extracted by the skeleton extracting unit 124 (see FIG. 5). ). The “position in the monitoring target area in the three-dimensional space” is a position that each feature point actually occupies in the three-dimensional space constituting the actual monitoring target area. This position can be specified based on the “three-dimensional position information” that can be obtained from the position that each feature point occupies on the coordinates.

(Gaze direction detector)
The line-of-sight direction detection unit 126 is based on the triangular “three-dimensional position information” formed by connecting “three gaze direction detection feature points” corresponding to the positions of both ears and nose of the “monitoring person”. Then, a gaze direction detection step (S16) for detecting the gaze direction of the monitoring target is performed (see FIGS. 6 to 8). Specifically, the direction from the midpoint of the base connecting the points corresponding to the positions of both ears in the above triangle to the apex, which is the point corresponding to the position of the nose, is set in the three-dimensional space of the monitored person. Detected as the line of sight direction.

(Body direction detector)
The torso direction detection unit 127 calculates the torso of the monitoring target from the “three-dimensional position information” of the straight line connecting “two torso direction detection feature points” corresponding to the positions of the shoulders of the “monitoring person”. A trunk direction detection step (S17) for detecting the direction is performed (see FIGS. 6 to 8). Specifically, a direction having a smaller angle difference from the line-of-sight direction among the two directions orthogonal to the straight line is detected as the line-of-sight direction in the three-dimensional space of the person to be monitored.

(Specific motion detector)
The specific motion detection unit 128 is based on the variation amount of the “line-of-sight direction” detected by the line-of-sight direction detection unit 126 or the variation amount of the difference between the “line-of-sight direction” and the “body direction” detected by the body direction detection unit 127. A specific operation detecting step (S18) for detecting the “specific operation” of the “monitoring person” is performed. The “specific motion” is, for example, an operation in which the “line-of-sight direction” or the difference between the “line-of-sight direction” and the “body direction” fluctuates periodically within a specific short time. And the “body direction” vector can be quantified and grasped.

(Suspicious motion determination unit)
The suspicious action determination unit 129 inputs a numerical value related to the specific action of the monitoring target person detected by the specific action detection part 128 (for example, a numerical value related to the specific action determined as described above), and is registered in advance. The suspicious degree of the specific action of the person to be monitored is determined by comparing with the numerical value related to the reference suspicious action. Then, the suspicious degree of the operation of the monitoring target person is determined and output to the monitoring image display device 130 or the like.

[Monitoring image display device]
The monitoring image display device 130 can be visually recognized by adding monitoring information acquired by the arithmetic processing unit 120 to the monitoring image captured by the imaging unit 110, specifically, an arrow indicating the line-of-sight direction of the person to be monitored. Any known various image display devices can be used as long as they can display. The monitoring image display device 130 may be a stationary monitor that is stationary in the administrator's work area, or may be a portable terminal monitor that can be carried by the administrator.

[Monitoring method]
FIG. 2 is a flowchart showing the flow of the monitoring method of the present invention that can be executed using the monitoring device 100.

(Coordinate setting step)
In the coordinate setting step (S11), the coordinate setting unit 121 performs a coordinate setting process. In the coordinate setting process performed in this coordinate setting step (S11), preferably, the coordinate setting unit 121 determines a position corresponding to the floor surface or the ground in the monitoring image that is a captured image captured by the imaging unit 110 as a monitoring target. This is a process for setting coordinates that can be specified in association with actual dimensions in the area three-dimensional space, that is, three-dimensional coordinates that also have depth information about the monitoring target area.

  This coordinate setting step (S11) is a preliminary preparation for monitoring the monitoring area, and the actual monitoring as the actual operation is started by the subsequent steps. In other words, the coordinate setting step (S11) may be performed at least once after the installation of the photographing unit 110, prior to the start of the monitoring operation. For example, if the above coordinates are appropriately set when the monitoring device 100 is installed, the monitoring system is operated as long as there is no particular change in the shooting conditions of the monitoring image, such as a change in the arrangement of the shooting unit 110. It is possible to eliminate the need for setting coordinates again.

  The coordinates set by the coordinate setting unit 121 are specified when a certain arbitrary position is specified in the monitoring image photographed by the photographing unit 110 and the position is on the floor surface or the ground surface. Are coordinates that can specify which position corresponds to the space of the actual monitoring target area. That is, the position on the set coordinates is set in association with the actual dimension in the three-dimensional space constituting the monitoring target area.

  FIG. 10 is a diagram illustrating an example of a background image captured when the coordinate setting unit 121 performs coordinate setting. In the example illustrated in FIG. 10, the floor surface 501, the wall surface 502, and the display shelf 503 are included in the captured monitoring image. 10 and 11, the plane parallel to the floor is defined as the XY plane, and the left-right direction in FIG. 3 of the XY plane is defined as the X direction, and the depth direction orthogonal to the left-right direction is defined as the Y direction. The vertical direction perpendicular to the floor is defined as the Z direction.

  When the monitoring image captured by the imaging unit 110 is two-dimensional image information, even if a position in the monitoring image is selected (specified), the selected (specified) position (two-dimensional position) alone is It is impossible to specify what position (three-dimensional position) is in the actual three-dimensional space. However, on the floor surface or the ground, the coordinates associated with the “actual dimensions in the monitoring target area three-dimensional space” are set, and the position to be selected (specified) in the monitoring image is determined on the floor surface or the ground. If it is limited, it becomes possible to specify the position of the floor or the ground in the monitoring target area three-dimensional space that is selected (specified) in the monitoring image. Therefore, the coordinate setting unit 121 sets coordinates corresponding to the floor surface or the ground.

The coordinate setting unit 121 includes a distance h from the image capturing unit 110 to the floor or the ground, the direction (angle α) of the optical axis O of the image light captured by the image capturing unit 110, and an image of the monitoring image captured by the image capturing unit 110. Coordinates are set using the angle (θ ₀ ).

  As shown in FIG. 9, the actual center position PO of the floor surface or the ground existing at the center position, that is, the optical axis O of the photographing lens 112 intersects the floor surface or the ground in the monitoring image captured by the imaging unit 110. The center position PO can be specified geometrically. Specifically, an angle α (an angle formed between the optical axis O and the vertical direction G) that can specify the height h from the floor surface or the ground at the position where the photographing unit 110 is installed and the direction of the optical axis O. Can be obtained, the distance LO from the position directly below the photographing unit 110 to the position PO can be obtained.

Since the vertical angle of view θ ₀ is a known value that is uniquely determined from the size of the image sensor 111 and the specifications of the photographing lens 112, the distance L1 from the center position PO to the closest point P1 of the photographing range, The distance L2 from the center position PO to the farthest point P2 in the photographing range can also be obtained geometrically. Similarly, the position in the image and the actual position (actual dimension, actual distance) in the monitoring target area three-dimensional space can be associated with any of these intermediate positions. Similarly to the depth direction (Y direction in the figure) described above, the position in the image and the actual position in the monitoring target area three-dimensional space (actual dimensions, actual dimensions) in the left-right direction (X direction in the figure) are also described. Distance).

  As described above, the coordinate setting unit 121 can set the coordinates associated with the “position in the monitoring target region three-dimensional space” in the monitoring image.

  FIG. 11 is a diagram illustrating an example of coordinates set by the coordinate setting unit 121. In FIG. 11, for the sake of explanation, grids that are equally spaced in the actual dimensions in the monitoring target area three-dimensional space are superimposed on the monitoring image of FIG. 10 in the Y direction and the X direction. In addition, dividing the area with the grid in this way is an example, and continuous coordinates may be set without dividing the grid. Note that the coordinates set by the coordinate setting unit 121 are set on the assumption that the floor surface 501 (or the ground) is infinitely spread. To explain this, the walls 502, the display shelves 503, etc. The grid was overlaid on the display.

  Here, in the captured monitoring image, even if the actual size in the monitoring target region three-dimensional space is the same, the position far from the near position appears smaller. Therefore, in the set coordinates, the grids that are equally spaced on the actual dimensions are set so that the far side becomes smaller. As described above, the coordinates set by the coordinate setting unit 121 are associated with the actual position (actual dimension, actual distance) in the monitoring target region three-dimensional space.

  As described above, the coordinates set by the coordinate setting unit 121 are associated with the actual position (actual dimension, actual distance) in the monitoring target region three-dimensional space. This also means that each grid or each point on the coordinates to be set includes distance information from the imaging unit 110. Then, it is possible to acquire the three-dimensional information related to the size and the three-dimensional shape of the monitoring target by grasping in which grid the monitoring target in the predetermined area is located.

  In the monitoring apparatus 100 of the present invention, the height h from the floor surface or the ground at the position where the photographing unit 110 is installed and the angle α that can specify the direction of the optical axis O (the optical axis O and the vertical direction). Can be obtained, the distance LO from the position directly below the photographing unit 110 to the position PO can be obtained. On the other hand, the imaging unit 110 configuring the monitoring device 100 does not have a configuration corresponding to a distance measuring unit that performs distance measurement by irradiating infrared rays or the like as an essential configuration. As described above, the “height h” that is the initial information necessary for geometrically specifying the position of the monitoring target region three-dimensional space is not a value automatically obtained by the monitoring device 100. The height at which the imaging unit 110 is installed varies depending on the site where monitoring is performed. In the monitoring apparatus 100, the input unit 122 receives an input of the height h, and acquires an appropriate height h.

  Also, it is necessary to obtain a correct value for the angle α that can specify the direction of the optical axis O. The photographing unit 110 described here is configured so that the direction of the optical axis O can be set to an arbitrary direction, but the direction of the optical axis O does not change after the setting. Therefore, in the monitoring apparatus 100 of the present embodiment, the appropriate angle α is acquired when the input unit 122 receives input for the angle α. Note that the angle α is adjusted at the time of installation work of the photographing unit 110, and therefore input is performed after the installation work is completed. At that time, for example, if a scale is provided in the orientation adjustment mechanism of the photographing unit 110 so that a final adjustment value can be obtained, an appropriate value can be obtained by inputting a value corresponding to the scale to the input unit 122. It is possible to input a simple value. Note that in the case of the photographing unit 110 having a simple configuration in which the direction of the optical axis O cannot be changed, the angle α may be used as the angle α, so that the input of the angle α by the input unit 122 can be omitted.

  In the case where the photographing unit 110 has a configuration corresponding to a distance measuring unit that performs distance measurement by irradiating infrared rays or the like, the distance measuring unit firstly takes from the photographing unit to actual points corresponding to the background of the monitoring image. And the coordinates associated with the actual position (actual dimension, actual distance) in the monitoring target area three-dimensional space can be set based on the actually measured value. According to a preferred embodiment of the present invention, the measurement of the distance from the imaging unit to the monitoring target, which is essential for obtaining initial information for coordinate setting in the monitoring system disclosed in Patent Document 2, is performed. Although not necessary, the coordinates may be set by a method as disclosed in the above document.

(Monitoring shooting step)
In the monitoring imaging step (S12), the imaging unit 110 performs monitoring imaging. Here, the monitoring shooting is performed by continuously shooting still images at a predetermined interval, and performing a monitoring operation described later as a sequence of images to be shot, but by making the shooting interval very short, It can also be understood that a surveillance operation is performed as a moving image shooting.

(Monitoring target identification step)
In the monitoring target specifying step (S13), processing for recognizing and specifying the monitoring target person in the monitoring image acquired in the monitoring photographing step (S12) is performed. More specifically, the monitoring target specifying unit 123 determines whether or not the monitoring target in the monitoring image captured by the imaging unit 110 is recognized and specified. When the person to be monitored is detected and identified (S13, Yes), the process proceeds to the skeleton extraction step (S14). When the person to be monitored is not identified (S13, No), the process proceeds to the monitoring photographing step (S12). Return and continue surveillance shooting.

  Specifically, the person to be monitored can be detected by any one of various conventionally known methods or a combination thereof. For example, “person” in the monitoring area can be recognized by the background difference. This background difference is a known technique, and a technique for recognizing a person to be monitored by taking a difference between image data acquired by a monitoring camera and a background image of a monitoring area acquired in advance. It is.

  In addition, for the identification of the detected person to be monitored, the combination of the present invention with an image recognition technique using deep learning, which has been attracting attention in recent years in the field of image recognition, has been effective. By combining with such an image recognition technique, it is possible to automatically detect and specify “people” and objects in the captured surveillance image with a very high recognition accuracy rate. It is also possible to classify and recognize a large number of persons to be monitored for each category and type, and to detect and identify them in parallel.

For example, the image recognition technique using deep running is disclosed below.
"Deep learning and image recognition, operations research"
(Http://www.orsj.o.jp/archive2/or60-4/or60_4_198.pdf))

(Skeleton extraction step)
In the skeleton extraction step (S14), as shown in FIG. 4, the skeleton extraction unit 124 has a plurality of feature points and a plurality of features of the monitoring target person H detected and specified in the monitoring target specifying step (S13). A skeleton of each person to be monitored, which is composed of skeleton lines connecting points, is extracted.

In this specification, the “skeleton” of a monitoring target person is a linear figure formed by connecting a plurality of feature points of the monitoring target person and connecting them. FIG. 5 is a diagram illustrating a state in which a skeleton is extracted from the monitoring target person H by the skeleton extraction unit 124. In FIG. 5, the positions corresponding to the top of the person H to be monitored, the left hand H ₂ , the tips of the other limbs and the main joints are recognized as feature points (h ₁ ,..., H _n ). The “skeleton” of the monitoring target person H formed by the plurality of feature points and the line segments connecting them is recognized as a figure in the monitoring image that is a two-dimensional captured image.

  The number of feature points that divide the person to be monitored from which the skeleton is extracted in the skeleton extraction step (S14) is subdivided into parts that are units of motion analysis. It may be designed as appropriate in consideration of whether it is necessary to distinguish and distinguish between the differences. However, as a minimum configuration for enjoying the advantageous effects of the monitoring device and the monitoring method of the present invention, the plurality of feature points are at least the feature points (h1, h2, h3) shown in FIG. It is indispensable that it is configured to include “three gaze direction detection feature points (h1, h2, h3) corresponding to the positions of the nose, right ear, and left ear of the person to be monitored. In order to enjoy a further advantageous effect of the invention, a plurality of the above feature points correspond to the feature points (h4, h5) shown in FIG. 6, that is, two positions corresponding to the positions of the right shoulder and the left shoulder of the person to be monitored. It is preferable that the fuselage direction detection feature points (h4, h5) are included.

Specifically, the skeleton of the person to be monitored can be extracted by any one of various conventionally known methods or a combination thereof. As an example, by using a technique called “OpenPose” disclosed in the following document, a skeleton of a monitoring target person formed by connecting arbitrary feature points from a two-dimensional captured image can be extracted. .
"Zhe Cao et al. Realtime Multi-Person 2D Human Pose Estimating using Part Affinity Fields, CVPR 2017"

(Feature point location step)
In the feature point position specifying step (S15), the three-dimensional configuration of the monitoring target region of each feature point constituting the monitoring target person's skeleton position information extracted as a two-dimensional figure in the skeleton extraction step (S14) Specify a position in space.

  FIG. 5 shows the feature points (h1,...) Of each skeleton extracted in the skeleton extraction step (S14) for the person H detected as the monitoring target person in the monitoring target specifying step (S13) by the feature point position specifying unit 125. h2... h11) are superimposed on coordinates including three-dimensional information (depth information) preset in the coordinate setting step (S11).

  As shown in FIG. 5, by the operation of the feature point position specifying unit 125, the position of the feature point (h10, h11) in the vicinity of the leg tip end of the skeleton of the person H to be monitored is converted into three-dimensional information (depth information). It is possible to specify that the standing position of the person H to be monitored is in the hatched part of the grid depending on which position on which the coordinates are occupied (in which grid).

  If the standing position of the person H can be specified on coordinates including three-dimensional information (depth information), for example, the size and shape of the person H in the monitoring image can be used to determine the person H's position in the monitoring target area three-dimensional space. The actual size and solid shape can be calculated and grasped. That is, three-dimensional data relating to the position and motion of the person H can be acquired from two-dimensional image data (position information of feature points superimposed on coordinates in the monitoring image).

  Based on the above principle, in the feature point position specifying step (S15), “positions in the monitoring target region in the three-dimensional space” are specified for the feature points constituting the skeleton of the monitoring target person. If the “position in the monitoring target area in the three-dimensional space” can be specified, the three-dimensional information related to the size or three-dimensional shape of the monitoring target person can be acquired as described above.

  For example, by using the above-described “OpenPose”, a plurality of feature points (a plurality of feature points composed of h1 to h11) are concatenated from the two-dimensional captured image of “person H” to be a monitoring target person. It is possible to extract a skeleton. Then, from the position on the coordinates of each feature point, the “position in the monitoring target region three-dimensional space” of each feature point can be specified.

(Gaze direction detection step)
In the line-of-sight direction detection step (S16), these three points are formed from the position information of the three line-of-sight direction detection feature points (h1, h2, h3) acquired in the feature point position specifying step (S15). The position information of the triangle that is the point (h1) corresponding to the position of the nose from the midpoint of the base connecting the points (h2, h3) corresponding to the positions of both ears of the triangle is acquired. The direction toward the apex is detected as the line-of-sight direction E in the three-dimensional space of the person to be monitored (see FIG. 7).

  As described above, if “OpenPose” or the like is used for the monitoring target person, three gaze direction detection feature points (h1, h2, h3) are first easily extracted as two-dimensional information in the monitoring image. be able to. Based on the three-dimensional information that can be calculated from the two-dimensional information according to the above principle, it is possible to accurately detect the line-of-sight direction in the three-dimensional coordinates. That is, the line-of-sight direction of the person can be accurately detected from only the position information of the three points that have a substantially common positional relationship among all the persons. In addition, even if one of the above three points in the monitoring image is in a blind spot in the image and is not directly visible, the position of the remaining one point is estimated with a certain degree of accuracy from the two visible points. By using this estimated value, the gaze direction of the monitoring target person can be detected with sufficiently high accuracy regardless of the posture of the monitoring image target person and the face direction in the monitoring image. it can.

  The “sight line direction” of the person to be monitored detected in this way is displayed on the above-described monitoring image display device as necessary, for example, as easy-to-understand graphic information such as an arrow indicating the direction. Is preferred. The same applies to the “body direction”.

(Body direction detection step)
In the trunk direction detection step (S17), the two points are formed from the positional information of the two trunk line-of-sight direction detection feature points (h4, h5) acquired in the feature point position specifying step (S15). Further, the position information of the straight line is acquired, and a direction having a smaller angle difference from the visual line direction E obtained in the visual line direction detection step (S16) is selected from the two directions orthogonal to the straight line in the three-dimensional space of the monitoring target person. It is detected as the body direction B at (see FIG. 7).

  If “OpenPose” or the like is used for the person to be monitored in the same manner as described above, the two body direction detection feature points (h4, h5) can be easily extracted. The body direction of the person can be accurately detected from only the position information of the straight line connecting the shoulders of the person to be monitored.

(Specific motion detection step)
In the specific operation detection step (S18), the specific operation of the monitoring target person in the monitoring target region is detected from the variation amount of the visual direction E detected in the visual direction detection step (S16). In the specific motion detection step (S18), the specific motion of the person to be monitored is similarly detected from the amount of change in the difference between the “line of sight” and the “body direction” detected by the body direction detector 127.

FIG. 8 is a diagram illustrating a state in which the specific motion detection unit 128 recognizes the “specific motion” of the person H to be monitored as the difference (angle θ ₁ ) between the line-of-sight direction E and the body direction B or the amount of variation thereof. is there. Here, human H identified as monitored person, the body direction B that has towards the line of sight only in different directions the angle theta ₁ minute, and is recognized by a particular operation detection unit 128. Further, from the angle theta ₁ of the variation per unit time, it is also possible to detect suspicious swinging operation and the like.

(Suspicious action judgment step)
In the suspicious operation determination step (S19), the suspicious operation determination unit (129) compares the “specific operation” of the monitoring target detected in the specific operation detection step (S18) with the reference suspicious operation registered in advance. The suspicious degree of the specific operation of the monitoring target person is determined. In this suspicious operation determination step (S19), when it is determined that the suspicious degree of the operation of the monitoring target person is high (abnormal operation is performed) (S19, Yes), a warning is sent to the monitoring person. After performing the warning step (S20), the monitoring is terminated. On the other hand, when it is determined that the suspicious degree of the motion of the person to be monitored is low (no abnormal motion is performed) (S19, No), the process returns to the monitoring photographing step (S12).

  The manner in which the warning is transmitted in the suspicious operation determination step (S19) is not limited to a specific mode, and may be a simple process such as generating a warning sound or displaying a warning, You may further report to a security company.

  In the suspicious motion determination step (S19), the “specific motion” of the person to be monitored is detected from the variation in the “line of sight” or the variation in the difference between the “line of sight” and the “body direction”. It is determined whether or not the “specific action” is an “action” having a high suspicious degree. As an example, if a monitored person stays at a certain position for a predetermined time or more and detects an action that repeats the swing motion for a predetermined number of times within a short time, such a “motion” is suspicious. Judged as a high degree of movement. In that case, the article on the coordinates in the line-of-sight direction can be identified by image recognition, and the determination criterion can be changed depending on the type of the article.

  Further, in the suspicious motion determination step (S19), a more advanced suspicious motion determination is performed by combining the gaze direction detected by the gaze direction detection unit 126 and the movements of both hands that can be extracted by the function of the skeleton extraction unit 124. Can be done. For example, when the hand moves violently on the extension line of the sight line direction E, or when the hand moves violently at a position deviated from the extension line of the sight line direction, the “specific action” to be detected is determined according to various monitoring targets. By appropriately setting, appropriate monitoring can be efficiently performed based on mechanical judgment.

DESCRIPTION OF SYMBOLS 100 Monitoring apparatus 110 Image pick-up part 111 Image pick-up element 112 Shooting lens 120 Calculation processing part 121 Coordinate setting part 221 Condition setting part 122 Input part 123 Monitoring object specific | specification part 124 Skeletal extraction part 125 Feature point position specific part 126 Eye-gaze direction detection part 127 Body direction Detection unit 128 Specific motion detection unit 129 Suspicious motion determination unit 130 Monitoring image display device 501 Floor surface 502 Wall surface 503 Display shelf S11 Coordinate setting step S12 Monitoring imaging step S13 Monitoring target specifying step S14 Skeletal extraction step S15 Feature point position specifying step S16 Line of sight Direction detection step S17 Body direction detection step S18 Specific motion detection step S19 Suspicious motion determination step S20 Warning step

Claims (5)

An imaging unit for imaging the monitored area;
A monitoring target specifying unit capable of recognizing and specifying a monitoring target person in a monitoring image captured by the shooting unit;
Three gaze direction detection feature points corresponding to the positions of both ears and nose of the monitoring target person, and two torso direction detection characteristics corresponding to the right shoulder and left shoulder positions of the monitoring target person a skeleton extraction unit for extracting the skeleton of the monitored person consists of skeleton line for connecting a plurality of feature points comprising point,
A feature point position specifying unit for specifying a three-dimensional position in the monitoring target area of the feature point extracted by the skeleton extracting unit;
A line-of-sight direction detector that detects the line-of-sight direction of the person to be monitored;
A fuselage direction detection unit;
A specific motion detector,
A suspicious action determination unit,
The line-of-sight direction detection unit is a point corresponding to the position of the nose from a midpoint of a base formed by connecting points corresponding to positions of both ears of a triangle formed by connecting the characteristic points for detecting the line-of-sight direction. Detecting the direction toward the apex of the triangle as the line-of-sight direction in the three-dimensional space of the person to be monitored ;
The torso direction detection unit is configured to select a direction having a smaller angle difference from the line-of-sight direction among two directions orthogonal to a straight line connecting the two torso direction detection feature points. Detected as the body direction in the
The specific motion detection unit detects a specific motion from the amount of change in the line-of-sight direction or the amount of change in the difference between the line-of-sight direction and the body direction,
The suspicious operation determination unit compares the specific operation detected by the specific operation detection unit with a reference suspicious operation registered in advance, and determines the suspicious degree of the specific operation of the person to be monitored. Output,
Monitoring device. A coordinate setting unit configured to set, based on the monitoring image, coordinates that can be specified by associating a position corresponding to the floor surface or the ground in the monitoring image with an actual dimension in a three-dimensional space constituting the monitoring target area; Prepared,
The feature point position specifying unit specifies the position of the feature point in the three-dimensional space based on the coordinates set by the coordinate setting unit.
The monitoring apparatus according to claim 1. A monitoring image display device for displaying the monitoring image;
The monitoring image display device displays the figure indicating the viewing direction of the monitored person in said monitored image A monitoring device as claimed in claim 1 or 2. A monitoring shooting step in which the imaging unit images the monitoring target area; and
A monitoring target specifying step in which a monitoring target specifying unit recognizes and specifies a monitoring target person in a monitoring image taken by the shooting unit;
The skeleton extraction unit has three gaze direction detection feature points corresponding to the positions of the nose, right ear, and left ear of the person to be monitored, and 2 corresponding to the positions of the right shoulder and left shoulder of the person to be monitored. A skeleton extraction step of a monitoring target person that extracts a skeleton of the monitoring target person composed of a skeleton line that connects a plurality of feature points including feature points for detecting the body direction of the place ;
A feature point position specifying unit for specifying a three-dimensional position in the monitoring target region of the feature point extracted by the skeleton extracting unit;
The line-of-sight detection unit is a point corresponding to the position of the nose from the midpoint of the base connecting the points corresponding to the positions of both ears of the triangle formed by the three feature points for detecting the line-of-sight direction. A line-of-sight direction detecting step of detecting a direction toward the apex of the triangle as a line-of-sight direction in the three-dimensional space of the person to be monitored;
Of two directions orthogonal to the straight line connecting the two body direction detection feature points, a direction having a smaller angle difference from the line-of-sight direction is detected as the body direction in the three-dimensional space of the person to be monitored. A torso direction detecting step,
A specific action detecting step of detecting a specific action of the person to be monitored from a fluctuation amount of the gaze direction or a fluctuation amount of a difference between the gaze direction and the trunk direction;
A suspicious action determination step for comparing the detected specific action with a reference suspicious action registered in advance to determine a suspicious degree of the specific action of the person to be monitored;
Monitoring method, comprising. The coordinate setting unit sets coordinates that can be specified by associating a position corresponding to the floor surface or the ground in the monitoring image with an actual dimension in the three-dimensional space constituting the monitoring target area based on the monitoring image. A coordinate setting step;
The monitoring method according to claim 4 .

Images