JP6797860B2 - Water intrusion detection system and its method - Google Patents

Description

The present invention relates to a surveillance system that detects ships, swimmers, drifting objects, etc. on the water from images of surveillance cameras installed on the coast.

In recent years, video cameras with excellent sensitivity and resolution have become available at low cost, and it is possible to capture a wide area such as the sea with a realistic number of cameras and capture an image of an object.

There is known a technique for detecting a ship or the like that has invaded a monitoring area at sea without being affected by waves (see, for example, Patent Documents 1 to 6).

As a technique related to the present invention, there is known a technique for estimating the phase velocity, period, wave height, etc. of a wave on the water surface from a camera image (see, for example, Non-Patent Documents 2 and 3).

Japanese Unexamined Patent Publication No. 2013-181795 Japanese Patent No. 4117703 Japanese Patent No. 3997062 International Publication No. 10/084902 Pamphlet JP-A-2002-279429 Japanese Patent No. 43201 Japanese Patent No. 5709255 Japanese Patent No. 4476517 Japanese Patent No. 4921857 Japanese Patent No. 5021913

Gunnar Farneback, “Two-frame motion estimation based on polynomial expansion”, Scandinavian Conference on Image Analysis, 2003, Internet <URL: http://www.diva-portal.org/smash/get/diva2:273847/FULLTEXT01.pdf ＞ Benetazzo, Alvise, et al, “Observation of extreme sea waves in a space-time ensemble”, Journal of Physical Oceanography Vol.45, No.9, 11 September 2015, Internet <http://journals.ametsoc.org/doi /pdf/10.1175/JPO-D-15-0017.1> Hiroki Hara, Ichiro Fujita, "Application of 2D Fast Fourier Transform in River Surface Flow Analysis Using Space-Time Images", Journal of Hydraulic Engineering, Vol. 54, February 2010, Internet <URL: http: // library .jsce.or.jp/jsce/open/00028/2010/54-0185.pdf ＞ Toru Inaba, 3 outsiders, "Estimation of water depth distribution by analyzing wave field images", 55th Annual Scientific Lecture Meeting of Japan Society of Civil Engineers, 2-5, (September 2000), Internet <http://library.jsce .or.jp/jsce/open/00035/2000/55-2/55-2-0005.pdf ＞

However, in the above-mentioned conventional technique, for example, when a ship and a swimmer (human) or a floating object (object) invade, the ability to distinguish which object is a ship, a human being, or an object is limited. It was. For example, the method of assuming the type from the size, approximate shape, and speed of the object is insufficient in accuracy. For this reason, it has been considered difficult to simultaneously detect ships, people, and objects.
In addition, how far the invaded object is from the land, that is, the degree of invasion into the land, cannot be estimated from the video alone. When accurate camera installation positions and shooting angles of view were available, there was no choice but to refer to them for calculation.

An object of the present invention is to provide a monitoring system that comprehensively detects and warns various objects that may invade from the sea in view of the above problems.

The marine intrusion detection system according to one embodiment detects candidates for invading objects from a visible camera image that monitors the sea, further guides the size, speed, intrusion direction, linearity, etc., and identifies objects to some extent. .. At the same time, it detects invading objects from the far-infrared camera image, and at the same time, distinguishes between ships (high brightness), humans (medium brightness), and floating objects (same brightness as the sea) based on the difference in brightness. At the same time, the movement of the wave in the normal state without an object is analyzed. To analyze the movement of the wave, we use the Fourier transform to observe the periodicity of the temporal change in brightness. In comparison with the movement of waves in the normal state, for example, in the case of ships, there is almost no interlocking with the movement of waves, and since humans move differently from waves, the interlocking is relatively low, such as drums. Since the object is almost floating in the waves, the interlocking with the waves is relatively high. By using these, the accuracy of object identification is improved. The marine intrusion detection system according to the other embodiment automatically tracks an object by using a swivel camera. However, when tracking, the position of the detected object cannot be determined because the angle of view constantly changes. In order to measure the distance to the land without having to set where the land is in advance, the periodicity of the wave is used to guide the position of the object to some extent. Wave periodicity changes such as being disturbed and weakened due to the shallower water depth in places closer to land than offshore. With this change, the distance of the object to the land is estimated and the degree of penetration is automatically derived.

According to the present invention, ships, humans, and suspended matter can be detected with high reliability.

The block diagram which showed the example of the logical configuration of the monitoring system 1 of Example 1. FIG. The functional block diagram of the sea level condition acquisition device 3 of this embodiment. The functional block diagram of the difference method base detector 4 of this Example. The functional block diagram of the detector 5 under the silhouette situation of this embodiment. The functional block diagram of the feature amount-based detector 6 of this Example. The functional block diagram of the tracker 7 of this embodiment. The functional block diagram of the threat evaluator 8 of this embodiment.

The monitoring system of the embodiment of the present invention detects a candidate for an invading object from a visible camera image that monitors the sea, further derives the size, speed, invasion direction, linearity, etc., and identifies the object to some extent. At the same time, it detects invading objects from the far-infrared camera image, and at the same time, distinguishes between ships (high brightness), humans (medium brightness), and floating objects (same brightness as the sea) based on the difference in brightness. At the same time, the movement of the wave in the normal state without an object is analyzed. To analyze the movement of the wave, we use the Fourier transform to observe the periodicity of the temporal change in brightness. In comparison with the movement of waves in the normal state, for example, in the case of ships, there is almost no interlocking with the movement of waves, and since humans move differently from waves, the interlocking is relatively low, such as drums. Since the object is almost floating in the waves, the interlocking with the waves is relatively high. By using these, the accuracy of object identification is improved.

The marine intrusion detection system according to the other embodiment automatically tracks an object by using a swivel camera.
However, when tracking, the position of the detected object cannot be determined because the angle of view constantly changes. In order to measure the distance to the land without having to set where the land is in advance, the periodicity of the wave is used to guide the position of the object to some extent. Wave periodicity changes such as being disturbed and weakened due to the shallower water depth in places closer to land than offshore. With this change, the distance of the object to the land is estimated and the degree of penetration is automatically derived.

FIG. 1 is a block diagram showing an example of a logical configuration of the monitoring system 1 according to the first embodiment of the present invention. The monitoring system 1 of this example integrates and tracks the video source 2, the sea level condition acquisition device 3, the difference method-based detector 4, the silhouette condition detector 5, the feature amount-based detector 6, and their results. The tracking device 7 and the threat assessor 8 are provided.
The video source 2 is a surveillance camera installed on the coast that captures the sea surface, or a device that reproduces the recorded video. Surveillance cameras can be equipped with electric pan heads and electric zoom lenses. The image may be obtained from any of the visible light region, the near infrared region, and the far infrared region, and may be either one channel (gray scale) or multiple channels (color). In addition, shake correction, heat haze correction, automatic white balance, gradation correction, high dynamic range (HDR) composition, etc. can be applied. The video source 2 may also output a plurality of versions of video filtered in the spatial or time domain for the subsequent three detectors. In addition, the input video of a plurality of frames in the past can be freely readable and held in case the input video is required again in the subsequent processing.

As an example, the sea level condition acquisition device 3 automatically estimates the temporal period, spatial period, and amplitude (vertical movement) of the background sea surface wave based on the input image. For these estimations, various methods such as a classical method such as particle image velocimetry (PIV), a stereo image method of Non-Patent Document 2, and a spatiotemporal image method of Non-Patent Document 3 can be used. The processing does not have to be performed all the time, and a human operation may intervene in part or in whole. Further, it is not necessary to perform the entire image frame, and it may be performed only at a few typical locations in the sea surface area. On the contrary, the area of the sea surface in the frame can be estimated through the trial of acquisition by the sea level condition acquisition device 3.

It is possible to verify or complement each other by using the law that holds between the time period, amplitude, and wave number (phase velocity) of the wave, but Non-Patent Document 2 describes a plurality. It is suggested that the law is not always obeyed when the waves overlap. In actual observation of waves near the coast, it is known that the waveform gradient (significant wave height / wavelength of significant waves) is 0.05 or less, and most of them have a peak around 0.01 except for rare high waves. .. In this example, the accuracy is improved by using the amplitude obtained directly from the low depression angle image. That is, the wave image is classified into a bright part and a dark part, and one of the apparent heights (the number of pixels in the vertical direction in the image), for example, the one having the smaller average value or the dispersion value is adopted. The amplitude obtained by this method is the apparent amplitude on the image and is not affected by the error associated with the coordinate transformation between the camera image and the real world.

The difference method-based detector 4 detects pixels whose values change faster than the reference image (background), and is similar to those described in Patent Documents 2 to 6. Waves observed on the surface of the sea photographed in a field of view of several tens of meters to several kilometers are assumed to have a period of several seconds. In the above document, a background image is generated by moving average the input video in a manner that follows the periodic fluctuation of the wave, and the absolute value of the difference between the latest input image and the background image is threshold-processed for each pixel. As a result, a high-speed moving object was detected. In this example, a short-time background image specializing in suppressing only fluctuations in pixel values with a period of several Hz derived from waves and an average of several minutes to several hours can be used to create a moving object other than the background structure. A long-time background image in which all the images are removed is created, and a candidate region (Region of interest) such as a ship is extracted from the difference between the short-time background image and the long-time background image. The candidate area can be represented by the bitmap itself consisting of the above logical values or the attributes of each candidate area. Various attributes of the candidate area are defined in relation to the circumscribing rectangle of the pixel mass having a value of 1 in the bitmap, for example, the coordinates of the center of gravity of the mass, the coordinates of the lowest point, and the circumscribing rectangle. Vertical size, horizontal size, aspect ratio, average brightness, edge amount, number of pixels (area), filling degree, peripheral length, circularity, complexity, etc. Here, the sides of the circumscribing rectangle are always set horizontally or perpendicular to the image frame. The average brightness and the amount of edges are calculated from each pixel corresponding to the mass of interest in the short-time background image or the input image. The filling degree is the ratio of pixels having a value of 1 in the circumscribing rectangle. Further, as one of the attributes of the candidate area, it is preferable to add the apparent distance between the lowermost end of the area and the horizontal line.

In the finite difference method, background update speed and threshold are important controllable parameters. The update speed can be automatically adjusted according to the time cycle obtained by the sea level condition acquisition device 3, the number of detected candidate regions, and the like. In particular, the update speed of the short-time background image (characteristic of the generation filter) should be carefully adjusted so that swimmers and suspended objects remain in the short-time background image and can be detected.

In the threshold value processing, for each pixel of the short-time background image, the difference from the pixel value of the corresponding pixel (block) in the long-time background image of the distribution model to which the pixel belongs is calculated. Then, it is achieved by comparing the absolute value of the difference value with the threshold value based on the variance of the corresponding pixel, and a logical value of 0 or 1 is output. When the pixels in the input video are modeled, the pixels belonging to different distribution models are temporally overlapped in the short-time background image. Therefore, the threshold value can be adjusted based on the variance of the distribution model plus 1/2 of the average difference between the model and the adjacent model. When the input video is color, the threshold value existing for each color is used, and if even one color exceeds the threshold value, it is set to 1 (that is, logical sum), or the sum of the excess of the threshold value is changed. Threshold processing can be performed. Further, there may be two types of threshold values, one with a small number of detection omissions and a large number of false positives, and the other with a large number of false positives but a large number of false detections, and each processing result may be output.

Both short-time and long-time background images may be generated by subjecting the input video to a spatial low-pass filter or downsample processing having characteristics corresponding to the spatial period of the wave, and then performing each time domain processing.

The detector 5 under a silhouette situation detects a dark region as an object candidate from an image in which an object having substantially dark brightness is projected in a background having substantially saturated brightness, and is described in Patent Document 3. It is the same as the one. Such an image is taken, especially at a time when the position of the sun is low, in a situation where the sunlight is reflected directly or on the sea surface and incident on the camera. The detection result is output as a bitmap or an attribute of the candidate area as in the difference method-based detector 4.

The feature amount-based detector 6 detects an object and identifies the type of the object by a process higher than the difference method. The feature amount-based detector 5 of this example acquires information on a candidate area detected by the difference method-based detector 4 and the detector 5 under a silhouette situation, and extracts the feature amount in the vicinity of the area in the input video. Then, using a machine learning method, it is determined whether the feature amount corresponds to a ship, a swimmer, a floating object, or other (sea surface, disturbance, etc.).

The feature amount is not limited to that obtained from one image, but may be a spatiotemporal feature obtained from the latest multiple frames of video, and any well-known feature amount may be available. Similarly, machine learning methods may utilize all well-known features, such as unsupervised learning (clustering) k-means, linear discriminant analysis (LDA), EM algorithms, and supervised learning. Logistic discriminant analysis, support vector machines, decision trees, restricted Boltzmann machines, etc. can be used. However, depending on the application, there is a preferable combination of the feature amount and the learning device. In this example, a random forest is used as the learner, and the features are texton features (Semantic Texton Forests), color histogram, HOG (Histograms of Oriented Gradients), HOF (Histograms of Optical Flow), DOT (Dominant Orientation Templates), MBH (Motion Boundary Histogram), separated lattice hidden Markov model, etc. are used. Before obtaining these features from the input image, a low-pass filter in the spatial region or the time domain, the optical flow processing described in Non-Patent Document 1, and the like can be applied.

The tracker 7 labels the candidate area detected by the detector for each frame and further associates it in the time direction. As a result, swimmers and the like that appear and disappear in the valley of the waves are continuously tracked, and areas that are sporadically falsely detected are removed. Well-known methods such as a Kalman filter, a particle filter, a Bayesian filter, and a mean shift can be applied to the tracker 7 of this example. If necessary for those processes, the tracker 7 is provided with the original image in the candidate area. The tracker can also predict the position in the near future. Buoys, signs, etc. installed at a predetermined position on the sea are removed based on that position. The tracker 7 can track at a position represented by image coordinates, but at a position in the global coordinate system corresponding to the real world based on the camera installation height, depression angle, angle of view (zoom magnification), etc. It is desirable to track. The image coordinates and the global coordinates can be converted by a projection matrix or a homography matrix (when the object position is limited to one plane). The tracker 7 can integrate and track the coordinates of the candidate regions from different detectors, and in that case, information indicating from which detector the candidate regions used for generating the trajectory are provided. It also collects and retains other attributes of the candidate area. Also, through tracking, new attributes such as changes in position from the previous frame can be added.

After identifying what the candidate area is due to, the threat evaluator 8 comprehensively evaluates the threat and issues a multi-stage report in consideration of the tendency of its movement (particularly approaching the land). For example, a candidate area that exists for 5 seconds or more and is not a disturbance of the sea surface is first recognized as some kind of object, and then the size of the candidate area becomes about 20 pixels as the object approaches. success. In addition, by continuing the tracking, it becomes possible to determine the tendency of approaching the land. That is, since the degree of threat can change, the threat evaluator 8 issues an alarm corresponding to the heightened threat. This can be expected to improve the convenience and safety of maritime security.

The identification of the candidate area is performed by a rule-based method set by a person or a well-known machine based on the attributes of the candidate area collected by the tracker 7 and the discrimination result of the feature amount-based detector 6 if available. Achieved using learning techniques. Here, considering the attributes that ships, swimmers, and floating objects will exhibit, the table below shows.

Note that if the object for which the edge is evaluated is a short-term background image, the intelligibility of the outline of the image will differ depending on the degree of movement. That is, if there is a lot of movement, the edge becomes thin, and if there is little movement, the edge becomes closer to the original edge. In practice, it may require more complex decisions (eg, referencing different tables day and night) than Table 1. The result of identification of the candidate region is fed back to the difference method-based detector 4, the detector 5 under the silhouette situation, and the feature amount-based detector 6, and is used for parameter adjustment and online learning (reinforcement learning).

The degree of threat is especially high in situations where a clear intent is presumed. For example, if it is determined that the shore is approached more than 10m in 10 seconds, it is estimated.

The configuration from the sea level condition acquisition device 3 to the threat evaluator 8 can be implemented by using a DSP (Digital Signal Processor), an FPGA (Field-Programmable Gate Array), or another processor specialized in image signal processing. In order to obtain a computing performance of 10000 MMACS (Million Multiply-Accumulates Per Second) or higher, a coordinated design of DSP and FPGA, especially a configuration in which a processing group that consumes memory bandwidth is pipelined on the FPGA. preferable.

FIG. 2 shows a functional block diagram of the sea level condition acquisition device 3 of this embodiment. The horizontal line estimator 31 applies a Canny edge detection filter and a Hough transform to an arbitrary video or short-time background image from the video source 2 to determine the position of a substantially horizontal line segment. The position of the horizon obtained here is appropriately provided from the difference method-based detector 4 to the tracker 7, and detection is not performed in the region above the horizon (mask region). The position of the horizon is used when calculating the apparent distance between the candidate area and the horizon with a tracker 7 or the like.

The simple estimator 32 estimates a pixel model within a plurality of predetermined evaluation regions (pixel blocks) in the input video from the video source, and outputs two threshold values for each region. The image of the sea surface can be represented by a mixed Gaussian distribution model, similar to the background image in the general finite difference method. In this example, the sea surface is assumed, there are bright parts and dark shadow parts corresponding to the undulations of the waves, and the highlight part where sunlight is directly reflected is also assumed, and the distribution is 2 to 3. Existence is presumed.

To determine the threshold, use the mode method that uses the density value of the valley of the histogram, the P-tile method that determines the number of pixels from the lowest density of the histogram according to the area ratio of the area to be separated, and the differential histogram. A differential histogram method that uses the density value that maximizes the value, Otsu's threshold determination method, and a variable threshold method that changes the threshold according to the properties of each part in the image can be used.
In this example, the Otsu method and the fluctuation threshold method are combined. That is, first set an appropriate threshold, distribute each pixel to any distribution model (class) according to the pixel value, accumulate over a sufficient number of frames, and then calculate the number of pixels, average, and variance. To do. Regarding the initial threshold value, since the highlight part is made up of pixels with saturated luminance values, the threshold value for identifying this can be easily determined, and the threshold value for dividing the bright part and the dark part is all. The average can be used. Then, according to Otsu's method, the threshold is updated so as to maximize the ratio of intra-class variance to inter-class variance. These processes are performed for each area. Further, it can be performed using only the luminance value without using the color information. Since the image of the sea surface is comparatively gentle, the distribution model can be performed for each evaluation region set discretely, and the region can be estimated by interpolation. Alternatively, it can be calculated not for each evaluation area but for each pixel.

When the dark part / bright part extractor 33 receives the estimated distribution model from the simple estimator 32, the dark part / bright part extractor 33 selects one of the classes corresponding to the bright part and the dark part. As an example, select the class with the smaller variance or the smaller number of pixels. Then, the number of pixels (height) in the vertical direction of the pixel block belonging to the selected class is calculated in each evaluation region of the input video and its vicinity. This value includes the height and depth of the shadow or illuminated part of the wave when the sea surface is viewed from an angle, and is not the actual height. Here, the reason for choosing the smaller class is that there is a high possibility that multiple waves are connected in the larger class, and such a region is not evaluated incorrectly.

The apparent wave height estimator 34 converts the height of the mass received from the dark / bright area extractor 33 into a wave height using a predetermined conversion formula. The conversion formula is a function of the depression angle and can be corrected empirically, but the smaller the depression angle, the smaller the effect. The depression angle is preset or provided by the calibration executor 72 (described below). Since the wave height is inherently variable, it is desirable to take a plurality of samples, sort them, and perform processing such as averaging between the top percentages and percentages.

The wavenumber and period estimator 35 estimates the wavenumber and period in the vicinity of each region based on the video from the video source 2, and if possible, calculates a more reliable wave height based on them. To do. In addition, when requested by the threat evaluator 8, the wave height, period, etc. of the wave at that location are calculated and provided. The wave height at the location between each region is calculated by interpolating the wave height estimated by the apparent wave height estimator 34. There are various methods for estimating the wave number, etc., but when multiple waves with different directions and periods overlap, it is not easy to estimate from an image with a shallow depression angle. If necessary, assume the existence of only one-way waves coming from offshore to the shore. The period is obtained by detecting a peak by FFTing a time-series pixel value or a classification result for a certain pixel in the region. More simply, it can also be obtained by collecting the positions of the upper end and the lower end of the mass detected by the dark part / bright part extractor 33 in time series and dividing the collection time by the number of intersections with the average value.

The water depth estimator 36 estimates the water depth at that location based on the characteristics of the waves. For example, as in Non-Patent Document 4, it can be calculated by using the following dispersion relation formula of the minute amplitude wave.

Here, T is the wave period, g is the gravitational acceleration, and h is the water depth. The phenomenon that the wave height becomes higher and the wavelength becomes shorter as the water depth becomes shallower is called shallow water deformation, and the same tendency can be seen in complicated irregular waves. Shallow water deformation is noticeably observed only where the water depth is shallower than 1/2 of the wavelength (offshore). The water depth estimator 36 does not need to obtain the absolute value of the water depth, and may only calculate a shallow water count indicating how many times the wave height is higher than that of the offshore area.

FIG. 3 shows a functional block diagram of the difference method-based detector 4 of this embodiment. The difference method-based detector 4 includes a short-time background image generator 41, a long-time background image generator 42, an update coefficient setting device 43, an absolute difference device 44, a binarizer 45, a threshold value setting device 46, and a time filter. A (true value holder) 47 and a labeling device 48 are provided.

The short-time background image generator 41 has a frame memory inside, and each time an image frame is input from the video source 2 at a predetermined rate, the image frame and the image in the frame memory are weighted (updated) by a predetermined weight. It is synthesized with the coefficient ρ ₁ ), output as a background image for a short time, and overwritten in the frame memory. This process is also called a time filter, recursive filter, IIR filter, exponential moving average, etc. As an example, 0. It has a time constant of about several seconds.

The long-time background image generator 42 has a frame memory inside, and each time an image frame is input from the video source 2 at a predetermined rate, the image frame and the image in the frame memory are weighted (updated). It is synthesized with the coefficient ρ ₂ ), output as a background image for a long time, and overwritten in the frame memory. As an example, the long-time background image generator 42 has the same configuration as the short-time background image generator 41, but operates under a reduced frame rate or a relatively small update factor, for example, a few seconds. It has a degree or higher time constant.

The long-time background image generator 42 can use the short-time background image output by the short-time background image generator 41 instead of the image from the video source 2. The long-time background image generator 42 is not limited to simply generating a long-time background image of one frame, but also generates a background image (mean value image) or a dispersion value image of a plurality of frames based on a mixture distribution model. You may. This can be implemented using the well-known codebook method. The result of the simple estimator 32 can be used for modeling.

The update coefficient setting device 43 automatically adjusts the update coefficients (update speed) ρ ₁ and ρ _{2 according} to the time period obtained by the sea level condition acquisition device 3 and the number of detected candidate regions. The update coefficient ρ ₁ is set by, for example, ρ ₁ = β · f so that the update is performed with a time constant similar to the wave period (several seconds to several seconds). Here, f is the frequency of the wave, β is a predetermined coefficient, and can always be adjusted according to the proportion of candidate regions that fail to be tracked by the tracker 7. On the other hand, the update coefficient ρ ₂ is set so that the update is performed with a time constant shorter than the sustainability and longer than the wave period, based on the persistence of the wave state, which is said to be about 20 to 30 minutes. To.

The absolute difference device 44 calculates the absolute value of the difference between the values of the corresponding pixels between the short-time background image and the long-time background image, and outputs it as a difference image. When the input video is color, this processing is performed for each color. Instead of the difference for each pixel, the difference in the histogram near the pixel of interest may be calculated.

The binarizer 45 compares the difference image from the absolute difference device 44 with the threshold value, binarizes it, and outputs the binarized image. When the threshold input video is color, the threshold existing for each color is used, and if even one color exceeds the threshold, it is set to 1 (that is, logical sum), or the excess of the threshold is set. The sum can be further thresholded. Further, the threshold value may be two types, one with a small number of detection omissions and a large number of false positives, and the other with a large number of false positives but a large number of false positives, and the respective processing results may be output.

If a model-based long-term background image is available, the binarizer 45 will use the binarizer 45 for each pixel of the short-time background image to accommodate the corresponding pixels (blocks) in the long-time background image of the distribution model to which that pixel will belong. Calculate the difference from the pixel value. Then, it is achieved by comparing the absolute value of the difference value with the threshold value based on the variance of the corresponding pixel, and a logical value of 0 or 1 is output. When the pixels in the input video are modeled, in the short-time background image, strictly speaking, the pixels belonging to different distribution models are temporally overlapped. Therefore, the threshold value can be adjusted based on the variance of the distribution model plus 1/2 of the average difference between the model and the adjacent model.

The threshold value setting device 46 adaptively sets a threshold value suitable for detecting the candidate region. As an example, by averaging the pixel values (absolute differences) for each of a plurality of predetermined regions in the difference image, the standard deviation in the distribution before the absolute value is obtained, and this standard deviation is multiplied by a predetermined coefficient. Is set as the threshold value. At the position of the pixel that does not belong to the area, the threshold value obtained in the neighboring area is interpolated and applied. The coefficient is set by a person and can be adjusted according to the detection status of the candidate area. When the pixels of the input video are modeled as a normal mixed model, the pixels belonging to different distribution models may overlap in time in the short-time background image. Therefore, the threshold value can be adjusted based on the variance of the distribution model plus 1/2 of the average difference between the model and the adjacent model.

The time filter 47 has a frame memory inside, holds an index of the latest frame in which the pixel has become a true value for each pixel, and updates the index each time a binarized image is received. An image (smoothed binary image) that has a true value if the index is within the latest n frames in the past is output. By this processing, the pixel that has once become the true value maintains the value for at least n frames, and the shape of the mass of the true value pixels approaches the shape of the object. Not limited to the time domain, a spatial domain filter such as a median filter may be applied.

The labeling device 48 extracts a block of pixels having a true value as a candidate region from the binarized image from the time filter 47 by using the 8-nearest neighbor method, the contour tracking method, or the like, and gives an index to them. Get those attributes and output. When this process is intensively performed by the tracking device 7, the labeling device 46 is unnecessary. The labeling device 46 can also perform simple tracking when extracting from the current frame using the extraction result (index table) from the immediately preceding frame.

When the image source 2 is an image obtained from a visible region camera and a far-infrared camera provided separately, the difference method-based detector 4 can independently process both images. If necessary, two sets of configurations from the short-time background image generator 41 to the labeling device 48 may be provided. However, the two sets do not have to have the same configuration, and the short-time background image generator 41, the time filter 47, and the like can be omitted on the side that processes the far-infrared image.

FIG. 4 shows a functional block diagram of the detector 5 under the silhouette condition of this embodiment. The detector 5 under the silhouette situation includes a binarizer 51, a threshold value setting device 52, a labeling device 53, and a time filter 54. The binarizer 51 binarizes the video from the video source 2 in the same manner as the binarizer 45, and outputs a binarized image. However, it is binarized so that it is true when the pixel value is smaller than the threshold value and false when it is greater than or equal to the threshold value. It is sufficient for the binarizer 51 to be able to binarize only the brightness value of the pixel.

The threshold setter 52 provides the threshold value used in the binarizer 51. Since this threshold value only distinguishes between saturation (highlight) and darkness, it can be given fixedly, or the threshold value generated by the simple estimator 32 may be used, and the labeling device 53 may be used. Depending on the size of the detected candidate region, for example, the threshold value may be adjusted to be smaller when a region that is too large compared to the size of the assumed object is detected.

The labeling device 53 processes and labels the binarized image from the binarizing device 51 in the same manner as the labeling device 46. Although not essential, it is desirable for the labeling device 53 to acquire attributes indicating the complexity of the contour of the candidate region, such as the aspect ratio, the filling factor, and the number of contour pixels.

The time filter 54 uses the simple mapping between frames by the labeling device 53 or the tracking result of the tracking device 7 to average the binarized images of the corresponding candidate regions between the frames with their centers of gravity matched. And output as an image of the candidate area. In the silhouette image, the object appears black and the brightness and color information cannot be obtained from the object itself. Therefore, if the object is recognized only from the outline of the binarized image, there is a concern that the accuracy may decrease. On the other hand, the motion blur included in the image obtained by the time filter 54 is expected to give an additional feature amount and help improve the accuracy. Instead of the binarized image, the original image of the candidate region may be averaged, or a time domain operation other than the averaging may be performed. The original image of the candidate area is obtained by a logical product operation of the video frame from the video source 2 and the binarized image of the candidate area. The time filter 54 is not essential.

FIG. 5 shows a functional block diagram of the feature amount-based detector 6 of this embodiment. The feature-based detector 6 identifies an object using a well-known random forest, and is a patch specifier 61, a size normalizer 62, a decision tree executor 63, a probability integrater 64, and a class discriminator 65. , Online learner 66.

The patch specifier 61 applies an image patch appropriately including the candidate region detected by the difference method-based detector 4 and the detector 5 under the silhouette condition, and extracts a partial image from the video of the video source 2. The shape of the patch is usually square. The patch specifier 61 can arbitrarily generate a plurality of image patches as long as the processing capacity of the feature amount-based detector 6 is not exceeded. For example, based on one candidate area, a plurality of versions of patches with slightly different positions and sizes may be applied, or even if there is no candidate area, within the sea level area detected by the sea level condition acquirer 3. Patches may be sequentially scanned and applied. The patch size at that time is set to the size that will be displayed when a ship or the like is present at the patch location (however, it does not fall below the normalized size described later) based on the camera parameters obtained by the tracker 7. To do. On the contrary, when there are too many candidate areas detected from one video frame, it is necessary to select the candidate areas based on the priority according to the size of the area, the request from the threat evaluator 8, or the criteria such as round robin. There is. Further, when the decision tree executor 63 extracts a feature amount from a frame having a different time time or a frame having undergone a different time domain operation, the same image patch is applied to each frame.

The size normalizer 62 normalizes the partial image cut out by the patch specifier 61 to a predetermined size that can be received by the decision tree executor 63. The size may be, for example, 7 × 7 pixels if each pixel is in a 4: 4: 4 format with complete color information.

The decision tree executor 63 traverses each of the T decision trees created by prior learning, and outputs the probability of the class corresponding to the leaf node reached. Branching of the decision tree is performed by evaluating the branching function. As an example, the branch function performs threshold processing on the result of addition / subtraction of the values of specific 1 to 4 pixels in an image patch, and Semantic Texton Forests and the like are known. Each leaf node holds the posterior probability p (c | v) = | Sc | / | S | of the class for the sample used for training, and it is only necessary to read the value. Note that p (c | v) is the probability of being identified in class c when v is input, | S | is the number of samples in the sample set S used for training, and | Sc | is class c of S. The number of samples that belong to. In this example, the class is five or more consisting of ships, people, suspended objects, sea level, and others. As a feature of the decision tree executor 63 of this example, the values (explanatory variables) used by the branch function in some decision trees are not limited to those derived from the image patch, but are the attributes of the candidate area, for example, the position (distance). ), Size, contour complexity, average brightness, etc.

The probability integrater 64 integrates posterior probabilities p obtained from each decision tree for each class. If there is no tree that has reached a certain class c, the probability is 0, and if there are multiple trees, the posterior probabilities p of each are integrated by the arithmetic mean, geometric mean, maximum value, and the like.

The class discriminator 65 determines the class corresponding to one of the largest of the integrated posterior probabilities p and outputs it as an identification result.

The online learning device 66 performs active learning, semi-supervised learning, inductive learning, etc. using the data in operation, or continues learning by the same algorithm as prior offline learning by label propagation to improve performance. Let me. In active learning, some kind of alarm is issued and the result of visually identifying the object by the operator is given to the learning machine. For example, the reliability of the report (probability of output by the class discriminator) is low, the identification is incorrect or it seems to be confusing visually, the operator's subjectivity wants to remember it, etc. Data that can be expected to contribute to the determination of identification boundaries can be fed back. For algorithms such as Adaboost, where the generalization error becomes smaller as training is performed with training data with a large margin, appropriate data should be fed back.

Transductive learning is a concept and method that accumulates labels by operators as test data and minimizes classification errors in this test data. Various well-known methods such as Brown Boost can be used for learning the branch function and posterior probability in the decision tree executor 63 and ensemble learning in the probability integrator 64. Boosting is a meta-algorithm that produces a new version of the classifier by giving a large weight to the training data that was misrecognized by the existing version of the classifier. In label propagation, the identification result of the current class discriminator 65 is used as a temporary label. In a simple example, the posterior probability p held by the leaf node is updated.

FIG. 6 shows a functional block diagram of the tracker 7 of this embodiment. The tracker 7 includes a coordinate system converter 71, a calibration executor 72, a tide level acquirer 73, an attribute integrater 74, and a Kalman filter 75.

The coordinate system converter 71 receives the attributes of the candidate area from the difference method base detector 4 or the feature amount base detector 6, and determines the coordinates and size of the candidate area from the values of the image (scene) coordinates to the global coordinates. Convert to a value. Generally, by using the inverse matrix V of the projection matrix P, the image coordinates expressed in homogeneous coordinates (homogeneous coordinates) are converted into global coordinates.

Here, s is a value corresponding to the reciprocal of the depth in the image coordinates, and is calculated as follows by giving the sea level z _R in the global coordinates of the candidate region to z _world .

For maritime surveillance, the position of the object can be assumed to be Z = 0 (ie 0 above sea level) in global coordinates. The global coordinates of the homogeneous coordinate representation in the above equation become Euclidean coordinates by dividing by the value w = h ₄₁ · u _image + h ₄₁ · v _image + h ₄₃ · s in the bottom row. Giving a constant to z _world in this way is, as a result, equivalent to the following homography transformation or Direct Linear Transform.

_{Here, (u 1, v 1)} .. (u n, v n) is, n pieces image coordinates of points for _{calibration, (x 1, y 1)} .. (x n, y n) they The global coordinates (metric) of the point, n is a natural number of 4 or more.

When the distance to the object is long, the position in the global coordinates can be corrected by using the distance D between the camera and the object, which is estimated from the apparent distance between the object and the horizon. For example, when the position of the camera is the origin of the global coordinates, the distance information of the original coordinates is replaced with D as follows.

The calibration executor 72 calculates the projection matrix P, the homography matrix H, or the camera parameters required for conversion to global coordinates. The projection matrix P is given by the product of the internal camera parameter matrix A and the external camera parameter (motion parameter) matrix M. The internal camera parameter matrix A is determined by the focal length, the aspect ratio of the pixel pitch, and the like, and the external camera parameter matrix M is determined by the camera installation position and the shooting direction. The projection matrix P has 11 degrees of freedom and can be estimated from 6 or more known points using the well-known Z. Zhang and Tsai methods.

The tide level acquirer 73 is provided in the calibration executor 72, and if available, acquires a more accurate sea level and provides it to the calibration executor 72. The water level is estimated based on the height position of floating buoys and signs provided at the specified location on the sea surface, or from the position of the coastline reflected in the image of the image source 2 and the position of the water surface with respect to the artificial structure. It can be estimated using image processing technology. Alternatively, the tide level information may be acquired from the outside at any time, or the tide (astronomical tide) data may be retained internally and read out according to the calendar.

The attribute integrator 74 associates a plurality of candidate regions obtained by the difference method-based detector 4 to the feature amount-based detector 6 with the corresponding candidate regions obtained in the past, and tracks each candidate region with a Kalman filter. In addition to passing to 75, the tracking result is received from the Kalman filter 75, and the attributes of the candidate regions estimated to be the same are integrated, added, or updated. The association is performed by associating the attributes of the candidate area, particularly those having similar position coordinates, sizes, and velocities expressed in global coordinates. Even if candidate regions derived from the same object are obtained from multiple detectors, they are integrated based on the similarity of attributes. If the tracking is successful at least once, new attributes such as trajectory and velocity are added based on the change in position, and each time the tracking is successful thereafter, the attributes are updated or added in the same manner as the other attributes. Even if candidate regions derived from a single object are divided into a plurality of pieces from a certain frame, they can be integrated in consideration of the commonality and size of the trajectories. As the tracking continues, the candidate area turns into an object that is likely to exist. If the tracking is interrupted, the attribute integrater 74 can request the feature-based detector 6 to try to detect a candidate object near the current assumed position. In order to avoid interruption of tracking due to the object going out of the field of view of the video source 2, it is judged by the image coordinates or the global coordinates whether the current assumed position is out of the field of view or is approaching the field of view, and the camera. It is possible to control the electric pan head equipped with.

The Kalman filter 75 receives the position coordinates of the candidate area from the attribute integrator 74, performs Kalman filter processing for each candidate area, and outputs the estimated position. The estimated position has reduced noise. Since the Kalman filter 75 internally estimates the model, the calculated position variance can be used as the association threshold in the attribute integrater 74.

FIG. 7 shows an example of a functional block diagram of the threat evaluator 8 of this embodiment. The threat evaluator 8 includes a far-infrared image brightness evaluator 81, a position change evaluator 82, a size evaluator 83, an aspect ratio evaluator 84, a brightness fluctuation evaluator 85, an edge evaluator 86, a priority evaluator 87, and others. It has an evaluator 88, a classifier 89, and an alarm controller 90. The configuration from the far-infrared image luminance evaluator 81 to the filling degree evaluator 87 calculates a quantitative numerical value such as a feature amount (explanatory variable) or a probability used by the classifier 88.

The far-infrared image brightness evaluator 81 evaluates the brightness in the candidate area of the far-infrared line image among the attributes of the candidate area (object) accumulated by the tracker 7, and outputs a numerical value explaining it. As an example, a value obtained by multiplying the average value of the brightness in the candidate region by a predetermined coefficient is output. The coefficient has the meaning of normalizing the variance of each feature. Alternatively, when the average brightness is obtained, the posterior probabilities that it is a ship, a floating object, or a swimmer may be output respectively. The same applies to other evaluation instruments thereafter.

The position change evaluator 82 calculates the period and width (wave height) of the fluctuation from the time series of the position of the center of gravity, which is one of the attributes of the accumulated candidate region, and combines it with the period and wave height obtained by the sea level acquirer 3. The degree of coincidence, or the degree of linearity and constant velocity of position change is quantified and output. The wave height may be compared with either the apparent wave height or the actual wave height, and coordinate conversion is appropriately performed if necessary for comparison. Alternatively, in the vicinity of the candidate region, the statistical correlation value between the time series of the vertical position of the bright part or the dark part extracted by the dark part / bright part extractor 33 and the time series of the center of gravity position is calculated. May be good. On the other hand, the degree of linearity and constant velocity can be quantified by dividing the average value of the magnitude of acceleration (absolute value or component perpendicular to velocity) by the average velocity, for example. The position of the candidate region used at this time may be the one before being processed by the Kalman filter 75 or the one in the image coordinates.

The size evaluator 83 outputs a value evaluated by time-averaging the size (world coordinates), which is one of the attributes of the accumulated candidate area. The median value may be used instead of the time averaging. The same applies to other evaluators.

The aspect ratio evaluator 84 outputs a value evaluated by time-averaging the aspect ratio, which is one of the attributes of the accumulated candidate area.

The luminance variation evaluator 85 outputs a value obtained by evaluating the degree of variation such as statistical variance and deviation from the time series of average luminance, which is one of the attributes of the accumulated candidate region.

The edge evaluator 86 outputs a value evaluated by time averaging the edge amount, which is one of the attributes of the accumulated candidate area.

The filling degree evaluator 87 outputs a value evaluated by time-averaging the filling degree, which is one of the attributes of the accumulated candidate region.

The other evaluator 88 outputs other feature quantities or parameters of the discriminator 89 based on the attributes of the candidate area and the like. For example, it outputs a signal for switching the classifier according to the type of video source (visible / far red), the feature amount related to sunshine (day / night), or the corresponding amount.

The classifier 89 is a trained classifier constructed using well-known techniques such as case-based inference (k-nearest neighbor method), decision tree, logistic regression, Bayesian inference (including hidden Markov model), and perceptron, and is a candidate. Outputs the area identification (classification) result and / or the probability of each classification. Inside the evaluator 89, parameters and learning machines can be switched according to the output of the other evaluator 88. If the identification result by the feature amount-based detector 6 is available, it may be integrated with the result, and if each evaluation value is obtained in both the far-infrared image and the visible image for a certain candidate region, each evaluation value is obtained. As a result of identifying the above, they may be integrated. When the far-infrared image luminance evaluator 81 to the priority evaluator 87 outputs the probabilities of each classification, the discriminator 89 may be composed of an ensemble learner that integrates them.

The intrusion degree evaluator 90 relates to the degree of invasion into the territorial waters or the degree of approach to the land, or their intention or possibility, from a series of position coordinates processed by the Kalman filter, which is one of the attributes of the accumulated candidate area. Output the evaluation value. In a simple example, the shortest distance between the current position (global coordinates) and the coastline (baseline) or territorial waters line of the map held in advance can be used as the evaluation value. However, it may alert ships that have no intention of invading, such as passing beyond the cape. Therefore, a large number of loci are learned using a well-known machine learning method, and outliers (outliers) that react to trajectories different from the trajectories observed from normal times are detected and destinations are estimated. Therefore, it is desirable that the evaluation value changes according to those values.

The alarm controller 91 outputs a continuous or sufficiently multi-step evaluation value indicating the degree of threat of intrusion based on the identification result of the object by the classifier 89 and the evaluation value by the intrusion degree evaluation device 90. At the same time, an alarm is output each time the evaluation value changes over the set threshold value. The identification result of the classifier 89 usually indicates the probability, but even if the result only points to one selected class, the longer the tracking period by the Kalman filter 75, or the apparent size of the candidate area. It is possible to use reliability that increases as the value increases.

The configurations of the system, the apparatus, and the like according to the present invention are not necessarily limited to those shown above, and various configurations may be used. For example, the video frame and the long-time background image may be subjected to difference processing without using the short-time background image, and although many differences due to waves occur, there is a possibility that they can be separated by machine learning with a feature-based detector 6 or the like. There is.

Further, the present invention is, for example, as a method or device for executing a process according to the present invention, a program for realizing such a method on a computer, a non-transient tangible medium for recording the program, or the like. It can also be provided.

The present invention can be applied to CCTV (Closed-Circuit Television) systems and the like.

1 Surveillance system 2 Surveillance camera device 3 Sea level condition acquirer 4 Difference method base detector 5 Detector under silhouette condition 6 Feature base detector 7 Tracker 8 Threat evaluator

Claims (6)

A sea level conditioner (3) that automatically estimates the wave attributes including the amplitude and period of the water surface wave that is the background of the input video based on the input video from the video source (2).
A finite difference method-based detector (4) that generates a reference image from the input video and detects pixels whose values change at a higher speed than the reference image from the input video.
A detector (5) under a silhouette situation that detects a dark region as an object candidate from the input image showing a background having substantially saturated brightness and an object having substantially dark brightness.
An image feature amount is extracted from the input video, and a feature amount-based detector (6) that outputs the type of the object when the image feature amount corresponding to the type of the object that has been machine-learned in advance is found.
Equipped with a,
The feature amount-based detector (6) acquires information on a candidate region detected by the difference method-based detector (4) and the detector (5) under the silhouette situation, and the darkness in the input video. A water intrusion detection system that extracts the image feature amount in the vicinity of the area . A difference method-based detector (4) that generates a reference image based on the input video from the video source (2) and detects pixels whose values change at a higher speed than the reference image from the input video.
A detector (5) under a silhouette situation that detects a dark region as an object candidate from the input image showing a background having substantially saturated brightness and an object having substantially dark brightness.
An image feature amount is extracted from the input video, and a feature amount-based detector (6) that outputs the type of the object when the image feature amount corresponding to the type of the object that has been machine-learned in advance is found.
The difference method based detectors, the aforementioned under silhouette circumstances detectors and labeling the dark area of the object candidate detected by the feature amount based detectors, and the time direction of the association, integrated the A tracker (7) that updates the attributes of the dark area of the object candidate, and
Equipped with a,
The feature amount-based detector (6) acquires information on a candidate region detected by the difference method-based detector (4) and the detector (5) under the silhouette situation, and the darkness in the input video. A water intrusion detection system that extracts the image feature amount in the vicinity of the area . A difference method-based detector (4) that generates a reference image based on the input video from the video source (2) and detects pixels whose values change at a higher speed than the reference image from the input video.
A detector (5) under a silhouette situation that detects a dark region as an object candidate from the input image showing a background having substantially saturated brightness and an object having substantially dark brightness.
An image feature amount is extracted from the input video, and a feature amount-based detector (6) that outputs the type of the object when the image feature amount corresponding to the type of the object that has been machine-learned in advance is found.
A threat evaluator (8) that identifies the object, continues to track the object, considers the tendency of approaching the land, comprehensively evaluates the threat, and issues a multi-stage report.
With
The feature amount-based detector (6) acquires information on a candidate region detected by the difference method-based detector (4) and the detector (5) under the silhouette situation, and the darkness in the input video. A water intrusion detection system that extracts the image feature amount in the vicinity of the area . A sea level conditioner (3) that automatically estimates the wave attributes including the amplitude and period of the water surface wave that is the background of the input video based on the input video from the video source (2).
A finite difference method-based detector (4) that generates a reference image from the input video and detects pixels whose values change at a higher speed than the reference image from the input video.
A detector (5) under a silhouette situation that detects a dark region as an object candidate from the input image showing a background having substantially saturated brightness and an object having substantially dark brightness.
An image feature amount is extracted from the input video, and a feature amount-based detector (6) that outputs the type of the object when the image feature amount corresponding to the type of the object that has been machine-learned in advance is found.
The difference method based detectors, and the dark area of the object candidate detected by the lower silhouette circumstances detectors and the feature-based detectors labeling, and the time direction of the association, integrated the object candidate The tracker (7) that updates the attributes of the dark area of
Equipped with a,
The feature amount-based detector (6) acquires information on a candidate region detected by the difference method-based detector (4) and the detector (5) under the silhouette situation, and the darkness in the input video. A water intrusion detection system that extracts the image feature amount in the vicinity of the area . A sea level conditioner (3) that automatically estimates the wave attributes including the amplitude and period of the water surface wave that is the background of the input video based on the input video from the video source (2).
A finite difference method-based detector (4) that generates a reference image from the input video and detects pixels whose values change at a higher speed than the reference image from the input video.
A detector (5) under a silhouette situation that detects a dark region as an object candidate from the input image showing a background having substantially saturated brightness and an object having substantially dark brightness.
An image feature amount is extracted from the input video, and a feature amount-based detector (6) that outputs the type of the object when the image feature amount corresponding to the type of the object that has been machine-learned in advance is found.
A threat evaluator (8) that identifies an object, comprehensively evaluates the threat in consideration of the movement tendency of the object, and issues a multi-stage report.
Equipped with a,
The feature amount-based detector (6) acquires information on a candidate region detected by the difference method-based detector (4) and the detector (5) under the silhouette situation, and the darkness in the input video. A water intrusion detection system that extracts the image feature amount in the vicinity of the area . A difference method-based detector (4) that generates a reference image based on the input video from the video source (2) and detects pixels whose values change at a higher speed than the reference image from the input video.
A detector (5) under a silhouette situation that detects a dark region as an object candidate from the input image showing a background having substantially saturated brightness and an object having substantially dark brightness.
An image feature amount is extracted from the input video, and a feature amount-based detector (6) that outputs the type of the object when the image feature amount corresponding to the type of the object that has been machine-learned in advance is found.
The difference method based detectors, labeling the dark area of the object candidate detected by the detector and the characteristic amount based detectors under the silhouettes conditions, and the time direction of the association, and integrated the A tracker (7) that updates the attributes of the dark area of the object candidate, and
A threat evaluator (8) that identifies the object that caused the dark region of the object candidate , comprehensively evaluates the threat in consideration of the movement tendency of the object, and issues a multi-stage report.
Equipped with a,
The feature amount-based detector (6) acquires information on a candidate region detected by the difference method-based detector (4) and the detector (5) under the silhouette situation, and the darkness in the input video. A water intrusion detection system that extracts the image feature amount in the vicinity of the area .