JP2016119625A - Monitoring system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a monitoring system that can grasp a situation in which monitoring camera should not be allowed to track an object to be tracked.SOLUTION: A monitoring system 1 comprises: a memorizing portion 44 that memorizes position information and photographable ranges of monitoring cameras 2; a monitoring camera tracking portion 43b that detects and tracks an object to be tracked on the basis of an image of each monitoring camera 2; a deviation determining portion 43c that determines whether or not an object to be tracked deviates from photographable ranges in all the monitoring cameras 2; an allowable time setting portion 43d that extracts the monitoring cameras 2 other than the monitoring cameras 2 which determine the deviations of the object on the basis of the position information, calculates predicted time of movement of the object to be tracked from the monitoring cameras 2 determining the deviations of the object to the other monitoring cameras 2, and sets allowable time on the basis of the longest predicted time of movement of the predicted time of movement; and a tracking-impossible determining portion 43e that determines that tracking of the object to be tracked is impossible when none of the monitoring cameras 2 detect the object to be tracked, and outputs the fact that the object cannot be tracked.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a monitoring system for monitoring a moving object such as a person or a vehicle.

  As a system for monitoring a moving body such as a person or a vehicle, for example, a system disclosed in Patent Document 1 below is known. In this patent document 1, in order to efficiently perform tracking of a moving body over the entire system and comprehensive management of information distributed to a plurality of video cameras, an internal camera that tracks people in a distributed manner for each of the plurality of video cameras. An inter-camera tracking means for tracking a moving body in cooperation with a plurality of in-camera tracking means, comprising a tracking means, and a main tracking element for tracking and a sub-tracking element for supplying a feature quantity of the moving object to the main tracking element; In addition, a plurality of video camera units distributed in the monitoring area are operated in cooperation with a moving body searching unit for searching for a moving body over a plurality of in-camera tracking units.

JP 2004-72628 A

  The system disclosed in Patent Document 1 described above has a configuration in which a plurality of video cameras are distributed in the monitoring area, so that many video cameras are required to cover the entire monitoring area. This not only increases the cost for constructing the system, but it may be difficult to increase the number of video cameras so that the blind spot area is eliminated depending on the situation of the monitoring area. In this case, since the video camera cannot be installed so as to cover the entire monitoring area, the blind spot area of the video camera cannot be monitored by the system in the monitoring area. According to Patent Document 1, even if the tracking target moves to the blind spot area, the tracking target can be tracked by moving again to the imaging range of the video camera. However, the tracking target does not always move again to the imaging range of the video camera. In addition, even if the tracking target is moved again to the shooting range of the video camera, there is a possibility that an illegal act is performed in the blind spot area of the video camera. In order to cope with such a situation, it is necessary to take other measures other than the system as in Patent Document 1. However, Patent Document 1 does not consider the problem due to the presence of the blind spot area of the video camera. For this reason, monitoring of the tracking target should not be entrusted to a plurality of video cameras, and other appropriate timing should be taken. Could not be grasped easily.

  Accordingly, the present invention has been made in view of the above problems, and provides a monitoring system capable of grasping a situation in which the tracking of a tracking target should not be entrusted to a video camera in a monitoring area at an appropriate timing. It is intended.

In order to achieve the above object, a monitoring system according to the present invention is a monitoring system including a plurality of monitoring cameras,
A storage unit that stores position information and a shootable range regarding the plurality of monitoring cameras;
A monitoring camera tracking unit that detects and tracks a predetermined tracking target from an image captured for each monitoring camera;
A deviation determination unit that determines whether or not the tracking target has deviated from the photographing range in all of the plurality of monitoring cameras based on a tracking result of the monitoring camera tracking unit;
When the deviation is determined, other monitoring cameras other than the monitoring camera that finally detected the tracking target are extracted from the position information, and finally from the monitoring camera that detected the tracking target to each of the other monitoring cameras. The expected travel time or the expected travel distance required for the tracking target to move is calculated, and the allowable time from the longest predicted travel time of the calculated predicted travel time or the longest predicted travel distance of the calculated predicted travel distance An allowable time setting section for setting
Even if the elapsed time from the determination of the deviation exceeds the allowable time, when the tracking target is not detected in any of the plurality of monitoring cameras, the tracking target is determined to be untraceable, An untraceable determination unit that outputs the effect,
It is provided with.

In addition, when the monitoring system according to the present invention outputs that it is determined that the tracking is impossible, the prediction system estimates the prediction area where the tracking target is present from the position of the monitoring camera that finally detected the tracking target. An area estimation unit;
A flying device control unit that controls a flying device having an imaging unit that captures an image from above, so that the estimated prediction area can be captured;
It is good also as a structure provided with.

Furthermore, when the monitoring system according to the present invention outputs that it is determined that the tracking is impossible, the predicted area where the tracking target exists is determined from the position of the monitoring camera that finally detected the tracking target. A prediction area estimation unit that estimates and excludes the shootable range of the other surveillance camera that did not detect the tracking target;
A display control unit for displaying the estimated prediction area on the display unit;
It is good also as a structure provided with.

Moreover, in the monitoring system according to the present invention, the flying device is a mobile flying device,
The flying device control unit may control the movement of the mobile flying device to a target position that can be photographed including the prediction area when an output indicating that the tracking is impossible is made.

In the monitoring system according to the present invention, the prediction area estimation unit repeatedly executes estimation of the prediction area,
The flying device control unit may reset the target position according to the execution result.

Moreover, in the monitoring system according to the present invention, the flight device is a plurality of moored flight devices,
The flying device control unit may select and control the shooting of a moored flying device that can capture an image including the prediction area when an output indicating that the tracking is impossible is performed.

  Furthermore, the monitoring system according to the present invention continues the estimation of the prediction area and the mooring capable of photographing the prediction area when none of the plurality of moored flight devices can photograph the prediction area. It may be selected when a type flight device is found.

  According to the monitoring system of the present invention, the position information regarding the plurality of monitoring cameras and the shootable range are stored in the storage unit, and the monitoring camera tracking unit extracts a predetermined tracking target from the image captured for each monitoring camera. The monitoring camera tracking unit detects and tracks, and the deviation determination unit determines whether or not the tracking target has deviated from the photographing possible range in all the monitoring cameras. Then, when the monitoring camera tracking unit determines a deviation from the shootable range, the allowable time setting unit extracts other monitoring cameras other than the monitoring camera that finally detected the tracking target from the position information, and finally selects the tracking target. Calculate the expected travel time or predicted travel distance required for the tracking target to move from the detected surveillance camera to each of the other surveillance cameras, and calculate the longest expected travel time or the calculated expected travel distance of the calculated predicted travel times. The allowable time is set from the longest expected travel distance. The untrackable determination unit is unable to track when the tracking target is not detected in any of the plurality of monitoring cameras even if the elapsed time from the time when the deviation from the shootable range is determined exceeds the allowable time. And output that fact. With such a configuration, it is possible to grasp a situation in which tracking of the tracking target should not be entrusted to the plurality of monitoring cameras at an appropriate timing without the monitoring person constantly confirming the images of the plurality of monitoring cameras.

  Furthermore, according to the monitoring system of the present invention, when the prediction area estimation unit outputs that the tracking target is determined to be untrackable from the untrackable determination unit, the prediction area estimation unit finally determines the position of the monitoring camera that detected the tracking target. Estimate the prediction area where the tracking target exists. Then, the flying device control unit controls the flying device having an imaging unit that captures an image from the sky so that it can be captured including the prediction area estimated by the prediction area estimation unit. With this configuration, when tracking of the tracking target should not be entrusted to a plurality of monitoring cameras, the flying device is controlled toward the prediction area estimated from the position of the monitoring camera that finally detected the tracking target, and the tracking target exists. The flying device can capture the predicted area with high possibility from the sky.

  Further, according to the monitoring system of the present invention, when the prediction area estimation unit outputs that the tracking target is determined to be untrackable from the untrackable determination unit, the prediction area estimation unit finally determines the position of the monitoring camera that detected the tracking target. The prediction area in which the tracking target exists is estimated by excluding the shootable range of other surveillance cameras that did not detect the tracking target, and is displayed on the display unit. With such a configuration, it is possible to display the predicted area where the tracking target estimated by omitting unnecessary information where the tracking target cannot exist can be displayed and notify the monitoring staff.

  Further, according to the monitoring system of the present invention, when the flying device is a mobile flying device, the flying device control unit outputs the mobile flying device when the non-tracking determination unit determines that the tracking is impossible. Move to the prediction area. With this configuration, when the tracking of the tracking target should not be entrusted to a plurality of monitoring cameras, the mobile flying device can be quickly controlled to move toward the prediction area.

  Further, according to the monitoring system of the present invention, the prediction area estimation unit repeatedly executes estimation of the prediction area, and the flying device control unit moves according to the prediction area estimation result executed by the prediction area estimation unit. Reset the target position for movement control of the flying device. With this configuration, it is possible to control the movement of the mobile flying device so that the predicted area that changes with time can be reliably imaged.

  Further, according to the monitoring system of the present invention, when the flying device is a plurality of moored flying devices, the flying device control unit outputs the prediction area when it is determined that the tracking is impossible from the tracking impossible determination unit. Select a mooring-type flying device that can shoot and control shooting. With such a configuration, it is possible to select the mooring type flying device capable of shooting the predicted area and perform shooting reliably.

  Further, according to the monitoring system of the present invention, when there is no flight device that can shoot the prediction area among the plurality of moored flight devices, the prediction area estimation unit continues to estimate the prediction area, When a moored flying device capable of photographing an area is found, the moored flying device is selected. With such a configuration, the moored flying device can photograph the prediction area from the sky when an object that can be photographed according to the prediction area that changes with time is found.

It is a schematic block diagram which shows the whole structure of the monitoring system which concerns on this invention. It is a figure which shows an example of the monitoring area | region which the monitoring system which concerns on this invention monitors. It is a figure which shows an example of the feature information used in order to calculate the similarity of a tracking target from the image of a monitoring camera, and the image of a flying device in the monitoring system which concerns on this invention. (A)-(d) It is explanatory drawing regarding the estimation process of a prediction area. It is an operation example of a surveillance camera controlled by a control signal. It is an operation | movement flowchart (1) of the tracking function by the monitoring system which concerns on this invention. It is an operation | movement flowchart (2) of the tracking function by the monitoring system which concerns on this invention. It is a flowchart of the allowable time setting process in FIG. FIG. 7 is a flowchart of an untraceable determination process in FIG. 6. It is a flowchart of the prediction area estimation process in FIG. It is a flowchart of the flying device control process in FIG. It is a flowchart of the object candidate detection process in FIG. It is a flowchart of the cooperation process in FIG.

  Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to FIGS. 1 to 13 of the accompanying drawings.

  As shown in FIG. 1, a monitoring system 1 according to the present invention is constructed by including a monitoring camera 2, a flying device 3, and a center device 4 as a tracking processing device.

(About surveillance camera configuration)
The surveillance camera 2 captures a predetermined imaging range from the ground. In this example, the altitude that the surveillance camera 2 captures from the ground is the most important information (the most important information among a plurality of types of feature information described later) for specifying the tracking target (for example, a suspicious person, a suspicious vehicle, etc.) For example, if the tracking target is a person, it is a face image, and if it is a vehicle, an automobile registration number mark (number plate) image) can be recognized at a relatively low altitude from the ground. As a result, the tracking target is photographed from the side, and a side image in which the tracking target can be easily grasped from the photographed image, that is, an image photographed from a direction substantially perpendicular to the direction of gravity (substantially horizontal direction) is mainly obtained. be able to.

  A plurality of surveillance cameras 2 are fixedly installed at predetermined locations in the surveillance area. FIG. 2 shows an example of installation of the surveillance camera 2 and shows a part of the surveillance area in the event venue. In the example of FIG. 2, surveillance cameras 2 (2a, 2b, 2c, 2d, 2e) with fixed shooting ranges E1, E2, E3, E4, and E5 head to the four locations around the gate of the Coliseum that will be the event venue. A surveillance camera 2 (2f), which is a pan / tilt / zoom camera having a photographic range E7 in which the photographic range E6 is variable, is installed at one location in the vicinity of the parking lot. Moreover, the monitoring camera 2a and the monitoring camera 2c are installed so that a part of mutual imaging | photography range E1, E3 may overlap.

  As shown in FIG. 1, the monitoring camera 2 includes a camera unit 21, a control unit 22, and a communication unit 23.

  The camera unit 21 forms a main body that captures a predetermined shooting range, and includes an imaging unit 21a, a direction control unit 21b, and an image data generation unit 21c.

  The imaging unit 21a includes an imaging element such as a CCD element or a C-MOS element, an optical component, and the like. The imaging unit 21a captures a predetermined imaging range in the monitoring area at predetermined time intervals, and sequentially outputs the captured images to the image data generation unit 21c.

  Note that when the monitoring camera 2 is configured as a pan / tilt zoom camera, the imaging unit 21a has a zoom position controlled by the control of the control unit 22 based on a shooting instruction of a control signal from the center device 4. Information on the focal length based on the zoom position is output to the image data generation unit 21c.

  The direction control unit 21b is necessary when the monitoring camera 2 is configured as a pan / tilt / zoom camera. The direction control unit 21b is controlled in the horizontal direction and the vertical direction under the control of the control unit 22 based on the shooting instruction of the control signal from the center device 4. The angle is controlled. The angle information in the horizontal direction and the vertical direction is a preset displacement angle from the shooting direction of the monitoring camera 2, and is output to the image data generation unit 21c.

  The image data generation unit 21c generates image data in which the focal length at the time of shooting and the angle information (shooting direction) in the horizontal direction and the vertical direction are added to the image taken by the imaging unit 21a as necessary. When the monitoring camera 2 is configured as a pan / tilt / zoom camera, a shooting direction obtained by correcting the previously set shooting direction of the monitoring camera 2 with the above-described displacement angle is added to the image data.

  The control unit 22 controls transmission of the image data generated by the image data generation unit 21c of the camera unit 21 to the center device 4 via the communication unit 23 at predetermined frame intervals at predetermined intervals. When the control unit 22 receives a control signal from the center device 4 via the communication unit 23, the control unit 22 controls the imaging unit 21a and the direction control unit 21b of the camera unit 21 in accordance with a shooting instruction of the control signal.

  The communication unit 23 is a communication interface for performing wired or wireless communication. Under the control of the control unit 22, the communication unit 23 communicates with the center device 4 via the communication line by wired or wireless, and transmits and receives data and signals. I do. Specifically, the image data generated by the image data generation unit 21c is transmitted to the center device 4 at predetermined intervals with a predetermined frame period. In addition, a control signal is received from the center device 4 for controlling the imaging unit 21a and the direction control unit 21b.

  As shown in FIG. 2, the monitoring camera 2 is not limited to a camera that is fixedly installed at a predetermined location in the monitoring area. The surveillance camera 2 may be capable of photographing at a relatively low altitude from the ground, and can be photographed at a low altitude, such as an in-vehicle camera mounted on various vehicles, a small mobile flying robot having an imaging unit, a drone, and the like. The monitoring area may be moved as described above. Further, when the monitoring camera 2 can move within the monitoring area, a configuration in which at least one monitoring camera 2 is provided may be used. When the surveillance camera 2 moves in the surveillance area, the position information of the surveillance camera 2 at the time of shooting changes with the movement. Therefore, the surveillance camera 2 is provided with a GPS positioning unit, and is monitored on the earth. Position information (latitude, longitude, height) indicating the three-dimensional position (current position) of the camera 2 is output to the image data generation unit 21c. Further, when the monitoring camera 2 moves in the monitoring area, the shooting direction of the monitoring camera 2 at the time of shooting also changes due to the posture change caused by the movement, and thus has the same configuration as the posture detection unit 35 described later, and an image data generation unit 21c corrects the shooting direction based on the posture information obtained thereby. Then, the image data generation unit 21c transmits to the center device 4 image data to which position information at the time of shooting by the monitoring camera 2 and a shooting direction of the monitoring camera 2 at the time of shooting are added.

(About the configuration of the flying device)
The flying device 3 is a mobile flying device capable of moving control according to an instruction of a control signal from the center device 4, that is, a mobile airship, a small mobile flying robot, a drone, an artificial satellite, or the like. Alternatively, it may be a moored flying device (for example, a moored airship) in which the main body is connected to a ground mooring device via a cable in a state where power supply and communication are possible. Furthermore, you may comprise by these combination.

  When the flying device 3 is a moored flying device, it is preferable to provide the number corresponding to the size of the monitoring area so as to include the blind spot area that cannot be captured by the monitoring camera 2.

  As shown in FIG. 1, the flying device 3 includes a camera unit 31, a propulsion unit 32, a rudder 33, a positioning unit 34, an attitude detection unit 35, a control unit 36, and a wireless communication unit 37.

  The camera unit 31 is mounted on the main body of the flying device 3 and shoots in a wider shooting range than the surveillance camera 2 from above. In this example, the altitude captured by the camera unit 31 of the flying device 3 is relatively high as compared with the surveillance camera 2, and among the plurality of types of feature information described below for specifying the tracking target. Although it is an altitude at which it is difficult to recognize the most important information, which is regarded as the most important, it is an altitude at which the tracking target can be photographed from above. As a result, it is possible to obtain mainly an upper image from which the surrounding situation on the horizontal plane to be tracked can be easily grasped from the photographed image, that is, an image photographed from a substantially vertical direction that is substantially the same as the direction of gravity.

  The camera unit 31 includes an imaging unit 31a, a direction control unit 31b, and an image data generation unit 31c.

  The imaging unit 31a includes, for example, an imaging element such as a CCD element or a C-MOS element, an optical component, and the like. The imaging unit 31a captures a predetermined imaging range from the sky every predetermined time and images the captured image. The data is sequentially output to the data generation unit 31c.

  When the camera unit 31 is configured as a pan / tilt zoom camera, the zoom position of the imaging unit 31a is controlled by the control unit 36 based on a shooting instruction of a control signal from the center device 4. Information on the focal length based on the zoom position is output to the image data generation unit 31c.

  The direction control unit 31b is necessary when the camera unit 31 is configured as a pan / tilt / zoom camera. The direction control unit 31b is controlled in the horizontal and vertical directions under the control of the control unit 36 based on the shooting instruction of the control signal from the center device 4. The angle is controlled. The angle information in the horizontal direction and the vertical direction is a displacement angle from the photographing direction corresponding to a preset direction of the camera unit 31, and is output to the image data generation unit 31c.

  The image data generation unit 31c generates image data in which the focal length at the time of shooting and the angle information (shooting direction) in the horizontal and vertical directions are added to the image taken by the imaging unit 31a as necessary. Further, image data is generated by adding the shooting direction corrected from the position information obtained by the positioning unit 34 described later and the posture information obtained by the posture detecting unit 35 described later as needed. To do.

  The propulsion unit 32 applies propulsive force to the flying device 3 by transmitting the engine power to the propeller and rotating it, and is controlled by the control unit 36 according to the flight instruction of the control signal from the center device 4.

  The rudder 33 is a moving blade used for maneuvering the flying device 3 and is controlled by the control unit 36 in accordance with a flight instruction of a control signal from the center device 4.

  The positioning unit 34 includes a GPS (Global Positioning System), and outputs position information (latitude, longitude, height) indicating the three-dimensional position (current position) of the flying device 3 on the earth to the control unit 36. This position information is also output to the image data generation unit 31c.

  The attitude detection unit 35 includes, for example, a gyrocompass and a servo accelerometer, detects the attitude of the flying device 3 by measuring the azimuth angle, pitch angle, and roll angle of the flying device 3, and controls the attitude information obtained by this detection. To the unit 36. This attitude information is also output to the image data generation unit 31c.

  The control unit 36 controls the wireless communication unit 37 so that the image data generated by the image data generation unit 31c of the camera unit 31 is transmitted to the center device 4 at predetermined intervals at predetermined frame periods. The control unit 36 includes a flight control unit 36a and a camera control unit 36b. When the flight control unit 36a receives a control signal from the center device 4 via the wireless communication unit 37, the flight control unit 36a controls the propulsion unit 32 and the rudder 33 according to the flight instruction of the control signal. When the control signal from the center device 4 is received via the wireless communication unit 37, the camera control unit 36b controls the imaging unit 31a and the direction control unit 31b of the camera unit 31 in accordance with the shooting instruction of the control signal.

  The wireless communication unit 37 performs wireless communication with the center device 4 and transmits / receives data and signals under the control of the control unit 36. Specifically, the image data generated by the image data generation unit 31c is transmitted to the center device 4 at predetermined time intervals in a predetermined frame period. In addition, in order to control the imaging unit 31a, the direction control unit 31b, the propulsion unit 32, and the rudder 33, a control signal (imaging instruction, flight instruction) is received from the center device 4.

(About the configuration of the center device)
The center device 4 as a tracking processing device includes a communication unit 41, a wireless communication unit 42, a control unit 43, a storage unit 44, an operation unit 45, and a display unit 46.

  The communication unit 41 is a communication interface for performing wired or wireless communication, and communicates with a plurality of monitoring cameras 2 via a communication line under the control of the control unit 43. Specifically, image data transmitted from the plurality of monitoring cameras 2 at predetermined frame intervals is received. In addition, a control signal for instructing photographing is transmitted to the monitoring camera 2 to be controlled by controlling the imaging unit 21a and the direction control unit 21b.

  The wireless communication unit 42 performs wireless communication with the flying device 3 under the control of the control unit 43. Specifically, the image data transmitted from the flying device 3 at a predetermined frame period and every predetermined time is received. Further, a control signal for instructing shooting by controlling the imaging unit 31a and the direction control unit 31b of the flying device 3, and a control signal for instructing flight by controlling the propulsion unit 32 and the rudder 33 of the flying device 3 are necessary. Send according to.

  The control unit 43 includes a target feature extraction unit 43a, a monitoring camera tracking unit 43b, a deviation determination unit 43c, an allowable time setting unit 43d, an untrackable determination unit 43e, and a prediction function for executing a tracking target tracking function to be described later. An area estimation unit 43f, a flying device control unit 43g, a target candidate detection unit 43h, a cooperation processing unit 43i, and a display control unit 43j are provided.

  The target feature extraction unit 43a extracts a plurality of types of feature information for specifying a tracking target, and stores the extracted feature information in the storage unit 44 as tracking target feature information (tracking target feature).

  Here, when the tracking target is, for example, a person, the plurality of types of feature information include, as shown in FIG. Consists of color, height, and figure.

  As a method for extracting the feature information, a known method may be used. For example, the feature information may be extracted as follows.

  The face image is extracted by extracting a person area from a specified image using a classifier for detecting a person and extracting a face area using a classifier for detecting a face in the extracted person area. . Then, this face area is extracted as feature information of the face image.

  In the extraction of age, a general reference face image for each age is stored in advance in the storage unit 44 in association with the age of the reference face image, and these reference face images and face images are extracted. The degree of similarity is calculated by comparing the extracted face image. Then, the age stored in association with the reference face image having the highest similarity is extracted as age characteristic information.

  In the gender extraction, a general reference face image for each gender is stored in advance in the storage unit 44 in association with the gender of the reference face image, and these reference face images and face images are extracted. The degree of similarity is calculated by comparing the extracted face image. Then, the gender stored in association with the reference face image having the highest similarity is extracted as gender feature information.

  In the hair color extraction, the head region is extracted by a discriminator for detecting the head in the human region extracted in the face image extraction. A region excluding the face region extracted at the time of face image extraction from the head region is defined as a hair region, a color information histogram in the hair region is created, and this is extracted as hair color feature information.

  In the extraction of the upper body clothes color and the lower body clothes color, an area corresponding to the upper body clothes and the lower body clothes in the height direction (for example, the ratio from the ground to the lower body clothes: upper body clothes: Histogram of color information is created from the head as 4: 3: 1), and this is extracted as feature information of upper body clothes color and feature information of lower body clothes color.

  Extraction by type of possession is performed by determining the presence and type of possession by a classifier for detecting the type of possession (for example, bag, backpack, umbrella, etc.) Extract as

  In the possession color extraction, a histogram of the color information of the possession area extracted at the time of extraction for each possession type is created, and this is extracted as characteristic information of the possession color.

  The height is extracted by referring to shooting parameters (monitoring camera information) such as the shooting position, focal length, and depression angle of the monitoring camera 2 included in the image data, and the position and size of the person region extracted when extracting the face image. The height is calculated from this, and this is extracted as the feature information of the height.

  In the extraction of the body type, the ratio of the length in the height direction and the length in the width direction of the person region extracted when extracting the face image is calculated, and this is extracted as the feature information of the body type.

  In addition, as a method of inputting the feature information of these tracking targets, when a monitor operates the operation unit 45 as necessary, a portion corresponding to a person is specified from the image of the monitoring camera 2 stored and stored in the storage unit 44. Then, the target feature extraction unit 43a extracts feature information from the designated image by the above-described method and stores it in the storage unit 44 as a tracking target feature. In addition, when the tracking target is determined in advance regardless of the designation operation of the monitor, the target feature of the tracking target is stored in the storage unit 44 in advance.

  The monitoring camera tracking unit 43b performs a later-described monitoring camera tracking process surrounded by a broken line in FIG. 6, and the tracking target feature extracted by the target feature extraction unit 43a from images sequentially captured for each of the plurality of monitoring cameras 2. Is used to detect a predetermined tracking target and to monitor the detected tracking target.

  The surveillance camera tracking process is a process executed on all the images of a plurality of surveillance cameras 2, and a person area in the image is extracted from an image photographed by the surveillance camera 2 by a discriminator. Each type of feature information is extracted in the same manner as the feature extraction method performed by the target feature extraction unit 43a described above, and tracking is performed by comparing this feature information with the tracking target feature extracted by the target feature extraction unit 43a. Object detection is performed as object detection processing.

  Specifically, detection of the tracking target using the feature information given in the example of FIG. 3 will be described. First, these multiple types of feature information are weighted in consideration of reliability in detection of the tracking target for each feature information. The coefficient is set. For example, the face image that can be collated with the highest accuracy is set as the most important feature information, the weighting coefficient of this face image is set to be the highest, and the weighting coefficient for height and body type that is difficult to collate with high accuracy is set to be low. It can be.

  Then, for each type of feature information, the feature information extracted by the monitoring camera tracking unit 43b is compared with the tracking target feature extracted by the target feature extraction unit 43a, and the similarity is calculated.

  Next, the above-described weighting coefficient is multiplied to the similarity calculated for each of the plurality of types of feature information. Then, the total similarity is obtained by multiplying the similarity for each type of feature information by the weighting coefficient and adding them together. Next, the tracking target detection threshold (for example, 90% similarity) that can be regarded as a tracking target stored in advance in the storage unit 44 is compared with the overall similarity, and it is determined that the overall similarity exceeds the tracking target detection threshold. When detected, it is detected as a tracking target.

  In the monitoring camera tracking process, when the tracking object is detected by the above-described object detection process, a tracking flag indicating whether the monitoring camera 2 is tracking the tracking object is turned ON. The tracking flag is stored in the storage unit 44 in association with each monitoring camera 2. Note that the image at this time is displayed on the display unit 46 under the control of the display control unit 43j.

  Further, in the monitoring camera tracking process, as the target tracking process, a process of tracking the tracking target detected in the above-described target detection process is performed on the image data from the monitoring camera 2 transmitted every predetermined time in a predetermined frame period. . For this tracking process, a known method may be used. For example, a change target in the image is extracted by taking a difference between frames of the current image and the previous frame, and the tracking target detected by the target detection process is detected. The change area corresponding to the position of is specified. Then, when new image data is acquired next, the difference region is similarly extracted to extract the change region, and the extracted change region is identified as the change region corresponding to the previous tracking target, and the change region It is determined whether the object can be regarded as the same object in terms of position, size, shape, and the like. By repeating these processes each time new image data is acquired, it is possible to track the tracking target in the image of the monitoring camera 2. Also in this target tracking process, the tracking target may be tracked by further using the feature information as performed in the above-described target detection process.

  Then, by the target tracking process described above, it is determined whether or not the tracking target has moved out of the shootable range of the monitoring camera 2, and the tracking target moves out of the shootable range of the monitoring camera 2 and deviates from the shootable range. If it is determined that the tracking flag of the monitoring camera 2 is turned off.

  Further, in the monitoring camera tracking process, when an input of a control signal for loosely changing the tracking target detection threshold for detecting the tracking target is received from the cooperation processing unit 43i described later, the tracking target detection threshold is determined according to the control signal. Is loosened (for example, 70% similarity).

  The departure determination unit 43c monitors the tracking flag controlled by the monitoring camera tracking process described above with reference to the storage unit 44. The departure determination unit 43c determines whether or not the tracking target has deviated from the shootable range in all of the plurality of monitoring cameras 2 by determining whether or not all the tracking flags of each monitoring camera 2 are turned off. ing. In addition, the departure determination unit 43c identifies the monitoring camera 2 that has finally turned OFF among the monitoring cameras 2 whose tracking flag is ON as the monitoring camera that has finally detected the tracking target.

  When the deviation determination unit 43c determines a deviation from the shootable range, the allowable time setting unit 43d stores other monitoring cameras 2 within a predetermined range (for example, 100 m) of the monitoring camera 2 that finally detected the tracking target. 44 is extracted from the position information of the monitoring camera information stored in 44. Then, the allowable time setting unit 43d calculates an estimated moving time required for the tracking target to move from the monitoring camera 2 that finally detected the tracking target to the other monitoring camera 2, and out of the calculated predicted moving time. Set the allowable time by adding the grace time to the longest expected travel time. For example, when the tracking target is a person, the expected movement time is calculated by using the general movement speed of the person and the distance from the monitoring camera 2 that finally detected the tracking target to the other monitoring camera 2. It can be calculated from Further, the grace time is a time determined in accordance with the monitoring policy.

  It should be noted that the predetermined range when extracting the other monitoring cameras 2 may be a predetermined range that is a circle drawn with a predetermined distance from the position of the monitoring camera 2 that finally detected the tracking target. In addition, when the tracking target is a vehicle, for example, an area in which the vehicle can pass (such as a road or a parking lot) is specified based on monitoring space information to be described later to be a predetermined range, and the predetermined range is included in the shootable range. The camera 2 may be extracted. Furthermore, all the monitoring cameras 2 stored in the monitoring camera information may be extracted.

  The tracking impossibility determination unit 43e determines whether or not tracking of the tracking target by the monitoring camera tracking unit 43b is disabled. More specifically, the untrackable determination unit 43e determines which monitoring time the elapsed time from when the departure determination unit 43c determines the departure from the shootable range passes the allowable time set by the allowable time setting unit 43d. If the camera 2 does not detect the tracking target, the tracking target cannot be tracked, that is, the tracking target should not be entrusted to a plurality of monitoring cameras 2. Is output.

  The prediction area estimation unit 43f can shoot the monitoring camera 2 that did not detect the tracking target from the position of the monitoring camera 2 that last detected the tracking target when the tracking impossible determination unit 43e outputs the tracking impossible determination signal. Prediction area estimation processing for estimating a prediction area where a tracking target exists by excluding the range is executed.

  More specifically, in the prediction area estimation process, first, the movement range of the tracking target is calculated. Specifically, it is assumed that the position at which the surveillance camera 2 last detected the tracking target is a position P indicated by a cross in FIG. In this case, with reference to the position P indicated by the x mark, when the movement range in which the tracking target can move to the maximum is calculated from a value obtained by multiplying the movement speed of the tracking target and the elapsed time, the hatched portion in FIG. E0. Next, the monitoring camera 2 that does not detect the tracking target within the calculated movement range E0 is extracted from the position information of the monitoring camera information stored in the storage unit 44. In the example of FIG. 4B, five surveillance cameras 2a, 2b, 2c, 2d, and 2e indicated by camera icons are extracted. Next, the imaging ranges E1, E2, E3, E4, and E5 of the extracted surveillance cameras 2a, 2b, 2c, 2d, and 2e are read out from the imaging ranges of the monitoring camera information stored in the storage unit 44. In the example of FIG. 4B, shaded portions drawn from the five surveillance cameras 2a, 2b, 2c, 2d, and 2e indicated by camera icons are taken by the surveillance cameras 2a, 2b, 2c, 2d, and 2e. The range is E1, E2, E3, E4, E5. Then, in consideration of the moving direction of the tracking target (the direction of the arrow m in FIG. 4C), the moving range E0 of the tracking target in FIG. 4A and the monitoring cameras 2a, 2b, 2c in FIG. , 2d, 2e and the immovable area E10 of the tracking target is specified from the immovable area where the tracking target cannot move, which will be described later. In the example of FIG. 4C, the hatched portion is the immovable area E10 to be tracked. Then, the movement range E0 of the tracking target of FIG. 4A and the non-movable area E10 of the tracking target of FIG. 4C are overlapped to estimate a non-overlapping portion as the prediction area E11. In the example of FIG. 4D, the shaded area is the prediction area E11.

  Note that the object detection process by the monitoring camera tracking unit 43b described above is always performed while the tracking function is ON. The prediction area estimation unit 43f continuously executes the above-described prediction area estimation process until the monitoring target is detected by the monitoring camera tracking unit 43b. For this reason, no tracking camera 2 has detected the tracking target when the prediction area estimation process is being executed. Therefore, extracting the surveillance camera 2 within the movement range and excluding the immovable area E10 including the imaging range from the movement range E0 as the prediction area E11 allows the surveillance camera 2 that has not detected the tracking target. The prediction area where the tracking target exists is estimated by excluding the imageable range.

  When the flying device 3 is a mobile flying device, the flying device control unit 43g can capture an image including a prediction area estimated by the prediction area estimation unit 43f when the untrackable determination unit 43e outputs a non-trackable determination signal. A flight instruction control signal for controlling movement of the mobile flight device to a position is output.

  Note that the target position is set to a predetermined height at the center of gravity of a region having a margin of a predetermined width outside the circumscribed rectangular region of the prediction area estimated by the prediction area estimation unit 43f.

  Further, when the prediction area estimation unit 43f repeatedly executes the estimation of the prediction area, the flying device control unit 43g resets the target position according to the result of the execution, and the flying device reaches the reset target position. 3 outputs a flight instruction control signal for controlling the movement of 3.

  Further, when the flying device 3 is a plurality of moored flight devices, the flying device control unit 43g captures the prediction area estimated by the prediction area estimating unit 43f when the untrackable determination unit 43e outputs the untrackable determination signal. A possible moored flying device is selected, and a control signal for controlling the flight state of the moored flying device is output so that the selected moored flying device images the prediction area.

  Even if the flying device control unit 43g determines that there is no moored flying device capable of photographing the estimated prediction area, if the flying device control unit 43g detects a moored flying device capable of photographing in the estimated prediction area after that, the detection is performed. The moored flying device is selected, and a control signal for controlling the flight state of the moored flying device is output so that the selected moored flying device images the predicted area.

  The target candidate detection unit 43h executes target candidate detection processing for detecting a tracking target candidate from an image captured by the camera unit 31 of the flying device 3 when the untrackable determination unit 43e outputs a nontrackable determination signal. To do.

  In the target candidate detection process, feature search is performed from the image of the image data acquired from the camera unit 31 of the flying device 3. In the feature search, a tracking target candidate is searched by calculating similarity using feature information based on a determination criterion that is looser than the determination criterion of the object detection processing of the monitoring camera 2 described above.

  The feature search is a criterion for using a part of the plurality of types of feature information, which is less than the feature information used in the criterion for the object detection process of the monitoring camera 2. Specifically, as indicated by a cross in FIG. 3, when the tracking target is a person, the facial image, age, gender, height, and body feature information used in the criteria for the target detection process of the monitoring camera 2 are used. First, feature information limited to the hair color indicated by a circle, the upper body clothes color, the lower body clothes color, the possession type, and the possession product color is used. Then, by using the same method as the target detection process described above, the total similarity is obtained by multiplying the similarity with the tracking target feature calculated for each feature information by multiplying the weighting coefficient and adding up the total similarity. When it is determined that a candidate detection threshold value that can be regarded as a tracking target candidate stored in advance in the storage unit 44 (for example, a similarity of 50% or the like) is exceeded, it is detected as a tracking target candidate. Note that the candidate detection threshold is set lower than the tracking target threshold because it is difficult to specify the tracking target in the image taken from the flying device 3 taken from above.

  The cooperation processing unit 43i executes the cooperation processing of FIG. 13 to be described later, and specifies the monitoring camera 2 installed in the image of the flying device 3 when the target candidate detection unit 43h detects the tracking target candidate. Then, the shootable range of the specified surveillance camera 2 is read from the surveillance camera information stored in the storage unit 44. Then, for all the tracking target candidates, all the monitoring cameras 2 whose tracking target candidate positions are within the shooting possible range of the monitoring camera 2 are identified, and a control signal for shooting instruction is transmitted to the identified monitoring camera 2 41, or a control signal for loosely setting the tracking target detection threshold by the monitoring camera tracking unit 43b for the image captured by the specified monitoring camera 2 is output to the monitoring camera tracking unit 43b.

  FIG. 5 shows an example of the operation of the monitoring camera 2 controlled by the control signal described above when the tracking target is linked from the flying device 3 to the monitoring camera 2. When there is a tracking target candidate at the position indicated by a cross in FIG. 5, the monitoring camera 2f capable of shooting the tracking target candidate is specified as a control target, and a control signal for shooting instructions is sent to the monitoring camera 2f of the control target. Is output. This control signal is transmitted to the monitoring camera 2f via the communication unit 41. When the monitoring camera 2f receives the shooting instruction control signal, the monitoring camera 2f changes from the current shooting range E6 to the shooting range E6 ′ where the tracking target candidate can be shot. Therefore, the zoom position of the imaging unit 21a and the horizontal and vertical angles by the direction control unit 21b are controlled in accordance with the shooting instruction of the control signal.

  The display control unit 43j controls the display on the display unit 46. Specifically, the prediction area estimated by the prediction area estimation unit 43f is displayed on the display unit 46 when a tracking function for a tracking target described later is executed. The prediction area can be displayed superimposed on the image captured by the camera unit 31 of the flying device 3 (for example, the prediction area is displayed in a semi-transparent manner superimposed on the image from the flying device 3). In the normal monitoring state, the display control unit 43j displays the latest image included in the image data transmitted from the monitoring camera 2 or the flying device 3 every predetermined time on the display unit 46, and can be monitored by the monitor. It is like that.

  The storage unit 44 includes, as various information necessary for executing a tracking function of a tracking target described later, a tracking target feature, a tracking flag, an elapsed time, tracking information, monitoring camera information, flying device information, monitoring space information, an image. Data is stored.

  The various types of information will be described. The tracking target feature is information indicating the tracking target feature information extracted by the target feature extraction unit 43a. For example, when the subject to be tracked is a person, facial images, age, sex, hair color, upper body clothes color, lower body clothes color, possession type, possession color, height, body shape, movement direction, and the like are the target features.

  The tracking flag indicates whether or not the tracking target is being tracked by the monitoring camera 2, and is stored for each monitoring camera 2. The tracking flag is ON if tracking is in progress, and is OFF if tracking is not in progress.

  The elapsed time is information indicating the time that has elapsed since the tracking flags of the plurality of surveillance cameras 2 are all turned off.

  The tracking information is information for tracking the tracking target in the same monitoring camera 2, and is information such as a position, a size, and a shape of the tracking target in the image.

  The monitoring camera information is information related to each monitoring camera 2, and includes position information (shooting position: latitude, longitude, height), shooting direction (horizontal angle and vertical angle) of each monitoring camera 2, and shooting currently being shot. Information such as a range, a focal length, and a shootable range that can be shot by pan / tilt zoom is included.

  In the movable monitoring camera 2, the position information, the shooting direction, the shooting range, the focal length, and the shooting range change, so that the control unit 43 sequentially updates the information based on the information included in the received image data. . Specifically, the position information, shooting direction, and focal length are updated as received information, and the shooting range and shooting range are the focal length, shooting direction, and position information included in the image data, and pre-stored pan / tilt zoom It is calculated and updated based on the changeable range.

  In addition, information related to the position information of the monitoring camera 2 that is fixedly installed at a predetermined location in the monitoring area can be stored in advance in the storage unit 44 as a part of the monitoring camera information. Among such surveillance cameras 2, the surveillance camera 2 capable of pan / tilt / zoom changes the shooting direction, shooting range, focal length, and shooting range, so that the above and the like are based on the information included in the received image data. Similarly, the controller 43 sequentially updates. Note that for the surveillance camera 2 that cannot be pan / tilt zoomed, the shooting range and the shooting range are the same, so they are stored as the same information.

  The flying device information is information related to the flying device 3, and includes position information of the flying device 3 (flight position or shooting position: latitude, longitude, height), shooting direction of the camera unit 31 (horizontal angle and vertical angle), current Information such as a photographing range, a focal length, a photographing possible range that can be photographed by pan / tilt zoom is included, and information that dynamically changes includes information included in image data received from the flying device 3 Based on the above, the controller 43 sequentially updates the same as described above.

  The monitoring space information is information related to the monitoring space, for example, three-dimensional information in which the shape such as terrain, buildings, roads and the like is represented in a predetermined coordinate system, the monitoring area to be monitored, and a person such as a building or a wall. This is information such as a non-movable area that cannot exist.

  The image data is data transmitted from the respective monitoring cameras 2 and the flying devices 3 at predetermined frame intervals every predetermined time, and is sequentially stored in the storage unit 44 for each monitoring device 2 and each flying device 3.

  In addition to the above-described information, the storage unit 44 also stores weighting coefficients and detection threshold values (tracking target detection threshold values and candidate detection threshold values) used for calculating similarity.

  The operation unit 45 includes, for example, a mouse, a keyboard, and soft keys on the display screen of the display unit 46. The operation unit 45 executes or terminates a tracking function for a tracking target, which will be described later, performs an operation for storing characteristic information of the tracking target in the storage unit 44 in advance, specifies a tracking target from image data stored and saved in the storage unit 44, and monitors each It is operated by a monitor when giving instructions to the camera 2 and the flying device 3 and deleting image data.

  The display unit 46 is composed of, for example, a liquid crystal display. The display unit 46 displays the image data acquired from the plurality of monitoring cameras 2 and the flying device 3, displays the prediction area, and predicts the image captured by the camera unit 31 of the flying device 3 under the control of the display control unit 43 j. (For example, the prediction area is displayed in a translucent manner on the image from the flying device 3).

(About the tracking function)
Next, as an operation of the monitoring system 1 configured as described above, a tracking function when tracking and monitoring a tracking target moving in the monitoring area will be described with reference to the flowcharts of FIGS. Here, it is assumed that the plurality of surveillance cameras 2 includes at least a pan / tilt / zoom camera, and the flying device 3 is a mobile flying device.

  First, the main flow of the tracking function will be described with reference to FIGS. This flowchart is started when the monitoring person operates the operation unit 45 of the center apparatus 4 to give an instruction to start processing of the tracking function and the tracking function is turned on (ST1). The tracking function ends when the monitoring person performs an end operation to turn off the tracking function at the operation unit 45. Following ST1, the target feature extraction unit 43a executes target feature extraction processing (ST2).

  In the target feature extraction process, a feature for specifying a tracking target is extracted as a tracking target feature. When the target feature extraction process is completed, the monitoring camera tracking unit 43b performs the monitoring camera tracking process (ST3 to ST11) on the images of the image data transmitted from all of the plurality of monitoring cameras 2 in the monitoring area.

  In the monitoring camera tracking process, it is determined whether or not an input of a control signal for loosely setting the tracking target detection threshold is received from the cooperation processing unit 43i (ST3). When the input of the control signal is determined, the tracking target detection threshold is set loosely (ST4). When this control signal input determination is completed, the routine proceeds to target detection processing (ST5). In the monitoring camera tracking process, when the tracking target detection threshold is relaxed, a process for returning to the original tracking target detection threshold is performed when the tracking function is turned off.

  In the target detection process, a tracking target is detected by using a plurality of types of tracking target features extracted in ST2 for images of image data transmitted from a plurality of monitoring cameras 2.

  When the target detection process ends, it is determined whether or not a tracking target is detected (ST6). If it is determined that the tracking target is detected, the tracking flag is turned ON (ST7), and the display control unit 43j displays the image of the monitoring camera 2 at that time on the display unit 46 of the center device 4 (ST8). Then, a target tracking process for tracking the tracking target detected by the target detection process is executed (ST9).

  In the target tracking process, the change area is extracted by the inter-frame difference from the monitoring camera 2 transmitted every predetermined time in a predetermined frame period, and the position and size of the change area corresponding to the tracking target detected in the target detection process of ST5. Now, the tracking target is tracked by associating the change areas of the same moving object across the frames in consideration of the shape.

  By the monitoring camera tracking processing (ST3 to ST11) described above, the tracking processing is performed for the image data sequentially acquired for each monitoring camera 2, and the tracking flag is controlled to be ON or OFF according to the result of the tracking processing. As a result, each monitoring camera 2 monitors whether or not the tracking target is being tracked.

  On the other hand, when the target feature extraction processing ends, processing parallel to the monitoring camera tracking processing described above is started. First, the departure determination unit 43c executes departure determination processing for determining whether all the tracking flags of the plurality of monitoring cameras 2 are OFF (ST12). Specifically, by determining that all tracking flags are OFF, it is determined whether or not all the monitoring cameras 2 are not tracking the tracking target. If it is determined in the departure determination process that all tracking flags are OFF, the measurement of elapsed time is started (ST13), and the process proceeds to the monitoring camera tracking disable determination process (ST14 to ST17).

  In the monitoring camera tracking impossibility determination process, first, the allowable time setting unit 43d executes an allowable time setting process for setting an allowable time for determining whether or not tracking by the monitoring camera tracking process (ST3 to ST11) is impossible ( ST14).

  When the permissible time is set, the tracking impossibility determination unit 43e compares the permissible time with the elapsed time started in ST13 to determine whether or not tracking by the monitoring camera tracking process is impossible. Untraceable determination processing is executed (ST15).

  When the tracking impossibility determination process ends, it is determined whether or not it is determined in the tracking impossibility determination process that tracking of the tracking target by the monitoring camera 2 is impossible (ST16). If it is determined that it is not determined that tracking of the tracking target by the monitoring camera 2 is impossible, it is determined whether or not the tracking flag of the monitoring camera 2 is ON (ST17). If it is determined that the tracking flag is ON, the elapsed time is cleared (ST18), and the process returns to the determination of whether all the tracking flags in ST12 are OFF. In this state, any one of the plurality of monitoring cameras 2 is retracking the tracking target. If it is determined that the tracking flag is not ON, the process returns to the tracking inability determination process in ST15. On the other hand, if it is determined in ST16 that the monitoring target cannot be tracked by the monitoring camera 2, the process proceeds to the prediction area estimation process of FIG. 7 (ST19).

  As described above, the monitoring camera tracking impossibility determination process (ST14 to ST17) is performed until all the tracking cameras 2 are not tracking the tracking target (ST12-Yes) until the tracking target is retracked (ST17). -Yes), the process is repeated, and it is determined whether or not the monitoring camera 2 cannot track the tracking target.

  In the prediction area estimation process, the prediction area estimation unit 43f estimates the prediction area. When the prediction area estimation process ends, the display control unit 43j displays the estimated prediction area on the display unit 46 of the center device 4 (ST20). When the display of the estimated prediction area is completed, the process proceeds to the flying device handling process (ST21 to ST26).

  In the flying device handling process, first, the flying device control unit 43g executes a flying device control process as a control for moving the flying device 3 to a target position where the flying device 3 can photograph the predicted area estimated in ST19 ( ST21).

  Then, it is determined whether or not the flying device 3 has arrived at the target position in the flying device handling process (ST22). If it is determined that the flying device 3 has not arrived at the target position, it is determined whether or not the tracking flag of the surveillance camera 2 is ON (ST23). If it is determined that the tracking flag of the monitoring camera 2 is ON, since at least one monitoring camera 2 of the plurality of monitoring cameras 2 is tracking the tracking target, it is determined whether all the tracking flags of ST12 are OFF. Return. If it is determined that the tracking flag of the surveillance camera 2 is not ON, the process returns to the prediction area estimation process of ST19, and the prediction area estimation process by the prediction area estimation unit 43f is continuously executed. Furthermore, if it determines with the flying device 3 having arrived at the target position, it will transfer to object candidate detection processing.

  In the target candidate detection process, the target candidate detection unit 43h detects a tracking target candidate with respect to an image photographed by the flying device 3, using a plurality of types of limited feature information as determination criteria.

  Next, it is determined whether or not a tracking target candidate has been detected by the target candidate detection process described above (ST25). If it is determined that a tracking target candidate has been detected, the process proceeds to linkage processing for linking the detected candidate with a plurality of monitoring cameras 2 and the monitoring camera tracking unit 43b (ST26). If it is determined that the tracking target candidate is not detected, it is determined whether or not the tracking flag of the monitoring camera 2 is ON (ST23). If it is determined that the tracking flag of the monitoring camera 2 is ON, since at least one monitoring camera 2 of the plurality of monitoring cameras 2 is tracking the tracking target, it is determined whether all the tracking flags of ST12 are OFF. Return to. If it is determined that the tracking flag of the monitoring camera 2 is not ON, the process returns to the prediction area estimation process of ST19, and the prediction area estimation process by the prediction area estimation unit 43f is continuously executed, depending on the result of this estimation process. The flight device handling process is executed again.

  In the cooperation process, the cooperation processing unit 43i outputs a control signal for cooperation to the monitoring camera 2 and the monitoring camera tracking unit 43b (ST26).

  When the cooperation process ends, it is determined whether or not the tracking flag of the monitoring camera 2 is ON (ST23). When it is determined that the tracking flag of the monitoring camera 2 is ON, since at least one monitoring camera 2 of the plurality of monitoring cameras 2 is tracking the tracking target, whether all the tracking flags of ST12 are OFF or not. Return to discrimination. If it is determined that the tracking flag of the monitoring camera 2 is not ON, the process returns to the prediction area estimation process of ST19, and the prediction area estimation process by the prediction area estimation unit 43f is continuously executed, depending on the result of this estimation process. The flight device handling process is executed again.

  The above-described prediction area estimation process and flying device handling process are performed until it is determined that tracking by all the monitoring cameras 2 is impossible (ST16-Yes) and until the tracking target is re-tracked by the monitoring camera 2 (ST23-Yes). The processing is repeated and the handling by the flying device 3 is continued.

  In the tracking function described above, after the predicted area is displayed in ST20, the next process of shifting to the flight device control process of ST21 displays on the display unit 46 whether or not to execute the flight device handling process. When an operator performs a permission operation on the operation unit 45, the process may proceed to ST21.

  Next, a processing flow not described in detail in the main flow of FIGS. 6 and 7 will be described.

  First, the allowable time setting process executed by the allowable time setting unit 43d in ST14 of FIG. 6 will be described with reference to FIG. The allowable time setting unit 43d extracts the monitoring camera 2 within a predetermined range from the periphery of the monitoring camera 2 that finally detected the tracking target, that is, the extracted position of the monitoring camera 2 (ST31). Subsequently, an expected movement time is calculated for each extracted surveillance camera 2 (ST32). Next, it is determined whether or not the estimated travel times of all extracted surveillance cameras 2 have been calculated (ST33). If it is determined that the predicted travel times of all the extracted surveillance cameras have been calculated, the longest predicted travel time is selected from the calculated predicted travel times (ST34). Then, an allowable time is set by adding a grace time for giving a margin to the selected longest expected travel time (ST35), and the process is terminated. In the case where there is no allowance for the allowable time, the selected longest expected movement time itself is the allowable time.

  With reference to FIG. 9, the untraceability determination process executed by untrackability determination unit 43e in ST15 of FIG. 6 will be described. The untraceable determination unit 43e determines whether or not the elapsed time is equal to or longer than the allowable time (ST41). If it is determined that the elapsed time is equal to or longer than the allowable time, it is determined that the tracking target cannot be tracked by the monitoring camera 2 and tracking is impossible (ST42), and the process ends. If it is determined that the elapsed time is not longer than the allowable time, the process ends without determining that the tracking is impossible.

  With reference to FIG. 10, the prediction area estimation process which the prediction area estimation part 43f performs by ST19 of FIG. 7 is demonstrated. In the prediction area estimation process, a moving range in which the tracking target can move to the maximum is calculated based on the detection position of the monitoring camera 2 that finally detected the tracking target (ST51). Next, the surveillance camera 2 within the calculated movement range is extracted (ST52). And the imaging | photography range of each surveillance camera 2 extracted is read from the surveillance camera information memorize | stored in the memory | storage part 44 (ST53). Subsequently, an immovable region where the tracking target cannot move is specified from the moving range of the tracking target and the imaging range of each monitoring camera 2 (ST54). Then, the identified non-movable area and the movement range to be tracked are overlapped to estimate a non-overlapping area as a prediction area (ST55), and the process ends.

  With reference to FIG. 11, the flying device control process executed by the flying device control unit 43g in ST21 of FIG. 7 will be described. The flying device control unit 43g determines a target position for moving the flying device 3 (ST61). Next, in order to control the movement of the flying device 3 to the target position, a control signal based on a flight instruction is transmitted from the center device 4 to the flying device 3 (ST62). Then, the center device 4 acquires position information from the flying device 3 that is movement-controlled according to the flight instruction of the control signal (ST63), and determines whether or not the flying device 3 has arrived at the target position (ST64). If the flying device 3 arrives at the target position, it is determined that the flying device 3 has arrived at the target position (ST65), and the process ends. Note that the determination of arrival at ST64 of the flying device is executed after a predetermined time (for example, 3 minutes) from the movement instruction at ST62.

  With reference to FIG. 12, the target candidate detection process which the target candidate detection part 43h performs by ST24 of FIG. 7 is demonstrated. The target candidate detection unit 43h acquires image data from the flying device 3 (ST71), and uses a plurality of types of feature information limited by performing a feature search from the acquired image as a criterion for tracking target candidates. It is detected (ST72). Then, it is determined whether or not a tracking target candidate is detected in this feature search (ST73). If a candidate is detected, it is determined that the target candidate is detected (ST74). If no candidate is detected, the process ends without determining ST74.

  With reference to FIG. 13, the cooperation process which the cooperation process part 43i performs by ST26 of FIG. 7 is demonstrated. The cooperation processing unit 43i specifies the monitoring camera 2 in the image of the image data acquired from the flying device 3 from the position information of the monitoring camera information stored in the storage unit 44 (ST81). And the imaging | photography possible range of the specified monitoring camera 2 is read from the monitoring camera information memorize | stored in the memory | storage part 44 (ST82). Next, it is determined whether or not the position of the tracking target candidate (target candidate position) is within the image capturing possible range of the specified monitoring camera 2 (ST83). If it is determined that the target candidate position is within the shootable range of the specified monitoring camera 2, the monitoring camera 2 is specified as the monitoring camera 2 to be controlled (ST84). Then, it is determined whether or not all specified surveillance cameras 2 have been processed (ST85). If it is determined that all specified surveillance cameras 2 have been processed, it is determined whether or not all candidates for tracking have been processed (ST86). When it is determined that all the tracking target candidates have been processed, a control signal is output to the control target (the monitoring camera 2 and the monitoring camera tracking unit 43b) (ST87), and the process is terminated.

  If it is determined that all the specified surveillance cameras 2 have not been processed, or if it has been determined that all the tracking target candidates have not been processed, is the target candidate position in ST83 within the shootable range? Return to the determination of NO.

  By the way, although the flying device handling process in the tracking function described above has been described on the assumption that the flying device 3 is a mobile flying device, when the flying device 3 is a moored flying device, the flying device control process of FIG. Instead, the following processing is executed. When the untrackable determination unit 43e outputs the untrackable determination signal, a moored flying device capable of photographing the prediction area estimated by the flying device control unit 43g is selected. Then, a control signal is output to the moored flight device so that the selected moored flight device images the predicted area. Accordingly, the flight control unit 36a controls the propulsion unit 32 and the rudder 33 according to the control signal received from the center device 4 so that the moored flight device can capture images including the prediction area, and the camera control unit 36b The focal length of the camera unit 31 and the horizontal and vertical angles are controlled. When the moored flight device is controlled by this control signal, the process proceeds to the target candidate detection process of ST24.

  If there is no mooring-type flying device capable of photographing the prediction area among the plurality of mooring-type flying devices, the process proceeds to determination of whether the tracking flag in ST23 is ON. If it is determined that the tracking flag is ON, at least one monitoring camera 2 of the plurality of monitoring cameras 2 is tracking the tracking target, and the process returns to the determination of whether all the tracking flags in ST12 are OFF. If it determines with a tracking flag not being ON, it will return to the prediction area estimation process of ST19, and the prediction area estimation process by the prediction area estimation part 43f will be performed continuously. When a moored flying device capable of photographing the estimated predicted area is found, the moored flying device is selected, and a control signal is output to the moored flying device so that the predicted area can be photographed. The focal length and horizontal and vertical angles of the propulsion unit 32, the rudder 33, and the camera unit 31 are controlled.

  Further, as another aspect of the allowable time setting process in the tracking function described above, the following may be performed. If it demonstrates using FIG. 8, it replaces with the process of ST32, and the estimated moving distance required for a tracking object to move from the monitoring camera 2 which finally detected the tracking object to the other monitoring camera 2 extracted in ST31 is respectively shown. calculate. Then, instead of the process of ST34, the longest predicted movement distance is selected from the calculated predicted movement distances. Then, instead of the process of ST35, an expected travel time is calculated from the selected predicted travel distance, and an allowable time obtained by adding a grace time to the predicted travel time is set. The calculation of the estimated travel time and the addition of the grace time can be performed in the same manner as the method described in the description of the allowable time setting unit 43d.

  As described above, the monitoring system 1 according to the above-described embodiment has the following effects.

  The center device 4 stores the position information and the shootable range regarding the plurality of monitoring cameras 2 in the storage unit 44, and selects a predetermined tracking target (for example, a person, a vehicle, etc.) from an image captured for each monitoring camera 2. The surveillance camera tracking unit 43b detects and tracks. That is, normally, tracking is performed by detecting a tracking target based on image data from a plurality of surveillance cameras 2 photographed from the ground within a predetermined photographing range. Then, the departure determination unit 43c determines whether or not the tracking target has deviated from the shootable range in all the monitoring cameras 2. When the deviation determination unit 43c determines a deviation from the shootable range, the allowable time setting unit 43d extracts, from the position information, other monitoring cameras 2 installed around the monitoring camera 2 that has determined the deviation. An estimated travel time required for the tracking target to move from the determined monitoring camera 2 to another monitoring camera 2 is calculated, and an allowable time is set from the longest predicted travel time among the calculated predicted travel times. The untrackable determination unit 43e is untrackable when no tracking target is detected in any of the plurality of monitoring cameras 2 even if the elapsed time from the time when the deviation from the shootable range is determined exceeds the allowable time. And output that fact.

  That is, when the tracking target cannot be detected by all the monitoring cameras 2, another monitoring camera 2 within a predetermined range (for example, within 100 m) is extracted from the monitoring camera 2, and the tracking target is extracted by the extracted monitoring camera 2. Calculate the estimated travel time of the tracking target required until it can be detected, and detect the tracking target with any surveillance camera 2 even if the calculated expected travel time exceeds the allowable time including the longest expected travel time When it is not possible, a message to that effect is output. With this configuration, the monitoring person can grasp the situation where it is difficult to find the tracking target again only with the monitoring camera 2 at an appropriate timing without constantly checking the images of the plurality of monitoring cameras 2. it can. Then, if the output when the tracking target becomes difficult to be tracked by all the monitoring cameras 2 is used as, for example, a notification signal for display output or audio output, it is possible to notify the monitoring person.

  The prediction area estimation unit 42e of the center device 4 finally detects the tracking target (for example, a suspicious person or a suspicious vehicle) when the tracking target determination unit 43e outputs that the tracking target is determined to be untrackable. The prediction area where the tracking target exists is estimated from the position of the monitoring camera 2 by excluding the imaging range of the monitoring camera 2 that did not detect the tracking target. Then, the prediction area where the tracking target estimated by excluding the imaging range of the monitoring camera 2 that did not detect the tracking target is present is displayed on the display unit 46. With such a configuration, it is possible to display a prediction area where a tracking target may exist without extra information and notify the monitoring staff.

  The flying device control unit 43g of the center device 4 controls the flying device 3 having the imaging unit 31a that captures an image from the sky so that it can be captured including the prediction area estimated by the prediction area estimation unit 42e. With such a configuration, when the tracking target cannot be tracked by a plurality of monitoring cameras 2, a prediction area is estimated from the position of the monitoring camera 2 that finally detected the tracking target, and the flying device 3 is directed toward the prediction area. The imaging area 31a of the flying device 3 can take an image of the predicted area where the tracking target is highly likely to be controlled from above.

  When the flying device 3 is a mobile flying device, the flying device control unit 43g of the center device 4 outputs the mobile flying device to the prediction area when an output indicating that the untrackable determination unit 43e determines that the tracking is impossible is made. Move control. With this configuration, when the tracking target becomes difficult to be tracked by all the monitoring cameras 2, the mobile flight device can be quickly controlled to move toward the prediction area.

  The prediction area estimation unit 43f of the center device 4 repeatedly performs prediction area estimation. Then, the flying device control unit 43g resets the target position for controlling the movement of the mobile flying device according to the prediction area estimation result executed by the prediction area estimating unit 43f, toward the reset target position. Control mobile flight equipment. With this configuration, the mobile flying device is controlled to the target position that is reset according to the result of estimation of the prediction area that is repeatedly executed, so that the mobile flying device can be reliably controlled toward the target position. .

  When the flying device 3 is a plurality of moored type flying devices, the flying device control unit 43g of the center device 4 can photograph the prediction area when the tracking failure determination unit 43e outputs that it is determined that tracking is impossible. The mooring-type flying device is selected to control shooting. With such a configuration, it is possible to select the mooring type flying device capable of shooting the predicted area and perform shooting reliably.

  When there is no flying device 3 that can shoot a prediction area among a plurality of moored flight devices, the prediction area estimation unit 43f continues to estimate the prediction area and can shoot the prediction area. When found, select the moored flying device. With this configuration, if there is no one that can capture the prediction area among the plurality of moored flight devices, the estimation of the prediction area is continued, so the prediction area where the tracking target can exist is estimated, and this prediction area is When a mooring-type flying device that can be photographed is found, the mooring-type flying device can photograph the predicted area from the sky.

  When it is determined that the tracking target tracking unit 43b cannot track the tracking target, the target candidate detection unit 43h of the center device 4 detects the tracking target candidate from the image captured by the flying device 3. When the cooperation processing unit 43i determines that the tracking target candidate detected by the target candidate detection unit 43h has entered the shootable range, it outputs a control signal to the monitoring camera 2 or the monitoring camera tracking unit 43b. For example, when the surveillance camera 2 is configured by a pan / tilt / zoom camera, the focal length and the horizontal and vertical angles of the surveillance camera 2 are controlled by the control signal so that the tracking target candidate can be photographed. With this configuration, it is possible to track the tracking target continuously by linking the monitoring camera 2 and the flying device 3 together.

  When detecting a tracking target that moves within a monitoring area using images from a plurality of monitoring cameras 2, a plurality of types of feature information including the most important feature information that can be collated with the highest accuracy is used as a criterion. The tracking target is detected. On the other hand, when detecting a tracking target candidate in a region including a prediction area using an image from the flying device 3, a part of feature information of a plurality of types of feature information not including the most important feature information is obtained. A candidate for tracking is detected as a criterion. With this configuration, the image from the monitoring camera 2 detects the tracking target with high accuracy, while the image from the flying device 3 does not use feature information that does not greatly contribute to the improvement in detection accuracy, and the image from the monitoring camera 2 In addition, by detecting the tracking target candidate with low accuracy, the processing load required to detect the tracking target candidate using the image from the flying device 3 can be reduced.

  As described above, according to the monitoring system according to the present embodiment, the monitoring camera 2 and the flying device 3 are linked to efficiently detect a tracking target (for example, a person, a vehicle, etc.) moving in the monitoring area. Track and monitor.

  The best mode of the monitoring system according to the present invention has been described above, but the present invention is not limited by the description and drawings according to this mode. That is, it is a matter of course that all other forms, examples, operation techniques, and the like made by those skilled in the art based on this form are included in the scope of the present invention.

DESCRIPTION OF SYMBOLS 1 Monitoring system 2 (2a-2f) Monitoring camera 3 Flight apparatus 4 Center apparatus (tracking processing apparatus)
DESCRIPTION OF SYMBOLS 21 Camera part 21a Imaging part 21b Direction control part 21c Image data generation part 22 Control part 23 Communication part 31 Camera part 31a Imaging part 31b Direction control part 31c Image data generation part 32 Propulsion part 33 Directional steering 34 Positioning part 35 Attitude detection part 36 Control Unit 36a flight control unit 36b camera control unit 37 wireless communication unit 41 communication unit 42 wireless communication unit 43 control unit 43a target feature extraction unit 43b surveillance camera tracking unit 43c deviation determination unit 43d allowable time setting unit 43e untrackable determination unit 43f prediction area Estimating unit 43g Flight device control unit 43h Target candidate detection unit 43i Cooperation processing unit 43j Display control unit 44 Storage unit 45 Operation unit 46 Display unit E0 Movement range of tracking target E1 to E6 Shooting range E6 'Shooting range after control E7 Shooting possible Range E10 Unmovable area to be tracked E11 Preliminary Measurement area m Direction of movement of tracking target

Claims (7)

A surveillance system comprising a plurality of surveillance cameras,
A storage unit that stores position information and a shootable range regarding the plurality of monitoring cameras;
A monitoring camera tracking unit that detects and tracks a predetermined tracking target from an image captured for each monitoring camera;
A deviation determination unit that determines whether or not the tracking target has deviated from the photographing range in all of the plurality of monitoring cameras based on a tracking result of the monitoring camera tracking unit;
When the deviation is determined, other monitoring cameras other than the monitoring camera that finally detected the tracking target are extracted from the position information, and finally from the monitoring camera that detected the tracking target to each of the other monitoring cameras. The expected travel time or the expected travel distance required for the tracking target to move is calculated, and the allowable time from the longest predicted travel time of the calculated predicted travel time or the longest predicted travel distance of the calculated predicted travel distance An allowable time setting section for setting
Even if the elapsed time from the determination of the deviation exceeds the allowable time, when the tracking target is not detected in any of the plurality of monitoring cameras, the tracking target is determined to be untraceable, An untraceable determination unit that outputs the effect,
A monitoring system characterized by comprising: When an output indicating that the tracking is impossible is made, a prediction area estimation unit that estimates a prediction area where the tracking target exists from the position of the monitoring camera that finally detected the tracking target;
A flying device control unit that controls a flying device having an imaging unit that captures an image from above, so that the estimated prediction area can be captured;
The monitoring system according to claim 1, comprising: When an output indicating that the tracking target is impossible is made, a predicted area where the tracking target is present is determined from the position of the monitoring camera that finally detected the tracking target, and the other tracking target is not detected. A prediction area estimation unit that estimates by excluding the imaging range of the surveillance camera;
A display control unit for displaying the estimated prediction area on the display unit;
The monitoring system according to claim 1, comprising: The flying device is a mobile flying device;
The said flight apparatus control part will carry out movement control of the said mobile flight apparatus to the target position which can image | photograph including the said prediction area, if the output to the effect of having determined that the said tracking is impossible is made. The monitoring system described. The prediction area estimation unit repeatedly executes estimation of the prediction area,
The monitoring system according to claim 4, wherein the flying device control unit resets the target position according to a result of the execution. The flight device is a plurality of moored flight devices;
3. The monitoring system according to claim 2, wherein, when an output indicating that the tracking is impossible is performed, the flying device control unit selects and controls shooting of a moored flying device including the prediction area. If none of the plurality of moored flying devices can shoot the predicted area, the estimation of the predicted area is continued and selected when a moored flying device capable of shooting the predicted area is found. The monitoring system according to claim 6.

Images