JP7083863B2 - Monitoring method, computer program for realizing the monitoring method, and monitoring system - Google Patents

Description

The present disclosure relates to a technique for detecting suspended matter in a waterway, and more specifically to a technique for suppressing false detection of suspended matter.

Conventionally, snow jam may occur in a waterway that supplies power generation water from a river to a power plant in winter or the like. In addition, jellyfish may flow into the waterways of power plants that use seawater as cooling water.

Patent Document 1 discloses a snow jam detection device. Further, Non-Patent Document 1 discloses a system for detecting the inflow of jellyfish by using an image of a surveillance camera.

Japanese Unexamined Patent Publication No. 2001-64952

Hiroaki Kawasumi "Jellyfish Inflow Detection System Using Surveillance Camera Images", [online], 2001, T.IEE Japan, Vol.121-C No.5, [Searched May 28, 2018 ], Internet <URL: https://www.jstage.jst.go.jp/article/ieejeiss1987/121/5/121_5_854/_pdf>

There is a demand for early detection of snow jams, leaves, jellyfish and other suspended matter. Regarding the detection of snow jam, for example, a detection method using an optical flow can be considered, but erroneous detection may occur depending on the usage environment. Therefore, there is a need for a technique for detecting optical flow at an early stage while suppressing false detection.

The present disclosure has been made in view of the above background, and discloses a technique for detecting suspended matter at an early stage while suppressing false detection.

A monitoring method according to an embodiment is a predetermined step set in a step of receiving input of a plurality of consecutive frame images obtained by photographing a water channel and a first frame image of the plurality of frame images. Using a step of detecting the movement of a number of feature points for each feature point over a plurality of frame images and a trained model prepared in advance, each pixel constituting one frame image of the plurality of frame images , A step of determining whether or not it corresponds to a floating substance in a water channel, a step of calculating the ratio of the number of pixels determined to have a floating substance in one frame image based on the result of the determination, and a feature point. It includes a step of determining whether or not a suspended matter is present in the water channel based on the magnitude relationship between the number and a predetermined number and the ratio of the number of pixels.

In a certain aspect, the step of determining whether or not a floating substance is present in a water channel is that the number of feature points is equal to or greater than a predetermined number and the ratio of the number of pixels is equal to or greater than a predetermined ratio. It includes determining that a suspended matter is present in the waterway based on the fact.

In a certain aspect, the monitoring method is predetermined by shifting the determination process using a predetermined number of frame images out of a plurality of frame images to the next frame image by shifting the first frame image used for the determination process to the next frame image. The number of feature points obtained by repeating the step of accumulating the results obtained by performing the number of times and calculating the total value of the number of feature points and the process of calculating the total value a predetermined number of times. It further includes a step to accumulate the total value. The fact that the number of feature points is equal to or greater than a predetermined number includes that the total value of the number of feature points is equal to or greater than a predetermined threshold value.

According to other embodiments, a computer program for realizing the monitoring method described in any of the above is provided.

According to still another embodiment, a monitoring system is provided that includes a memory that stores the computer program and one or more processors that execute the computer program.

According to a certain embodiment, it is possible to detect suspended matter at an early stage while suppressing false detection.
The above and other objects, features, aspects and advantages of the invention will become apparent from the following detailed description of the invention as understood in connection with the accompanying drawings.

It is a flowchart which shows a part of the processing performed by the processor of the computer system which operates as a monitoring system which detects a floating substance at an early stage. It is a figure for demonstrating snow jam. It is a figure showing the image obtained by taking an image of the water channel which supplies water to a power generation facility (power plant). It is a figure for demonstrating the system configuration of a monitoring system. It is a figure for demonstrating a plurality of reference coordinates (positions) used for setting a feature point. It is a figure for demonstrating an example of the setting method of the initial position of a feature point. It is a figure showing the initial coordinate Ei [1] of a feature point. It is a figure showing the 50th frame image F50. It is a figure showing the 100th frame image F100. It is a figure showing the initial coordinate Ei [1] of a feature point. It is a figure showing the 100th frame image F100'. It is a figure for demonstrating the verification result based on the moving image data at night. It is a figure for demonstrating the verification result based on the moving image data in the daytime. It is a functional block diagram for demonstrating the functional configuration of the server apparatus 100 of the monitoring system 1. It is a flow diagram for demonstrating the flow of the first half part of the process executed in a server apparatus 100. It is a flow diagram for demonstrating the flow of the latter half part of the process executed in a server apparatus 100. It is a figure which showed the typical example of the hardware composition of the server apparatus 100. It is a figure which shows the false positive factor. It is a figure which shows an example of the reference coordinate of a feature point and the arrangement of a frame in the detection using an optical flow. It is a flowchart which shows a part of the processing executed by the processor 151 of the server apparatus 100 which functions as a monitoring system. It is a figure which shows the timing when the snow jam was detected when the security was stopped at 12:30 in a certain power plant. It is a figure which shows an example of the detection result including the case where it was erroneously detected that the snow jam occurred by the water droplets adhering to the camera though there was no security stop in a certain power plant. It is a figure which shows the detection result when there is a false detection factor. It is a figure which shows the transition of the snow area ratio (average for 5 minutes) observed on December 29, 2018. It is a figure which shows the detection result when there is a false detection factor. It is a figure which shows the transition of the snow area ratio (difference from the previous value) observed on December 29, 2018. It is a figure which conceptually shows an example of the method adopted as a measure against false positives by a monitoring system according to a certain embodiment. It is a figure which shows the detection result in the case where the monitoring system uses the snow area ratio as it is for determination with a threshold value of 10%. It is a figure which shows the detection result in the case where the monitoring system uses the cumulative value of the snow area ratio for determination with a threshold value of 140%. It is a figure which shows the detection characteristic by each method of PoC (Proof of Concept) 1 model only, optical flow only, and PoC1 model & optical flow combined use. It is a figure for demonstrating the combination of the evaluation result of a trained model, and an optical flow. It is a figure which shows the timing of the detection of the occurrence of the snow jam when the water intake is stopped by the snow jam on December 28, 2018. It is a figure which shows the timing of the detection of the occurrence of a snow jam when the water intake is stopped by the snow jam on January 24, 2019. It is a figure which shows the state when the false detection by the light of a spotlight occurs on January 15, 2019 even though the snow jam does not occur. It is a figure which shows the state when the false detection by the snow and the cloud of the wall surface reflected on the water surface occurred on January 27, 2019, even though the snow jam did not occur. It is a figure which shows the detection result by the monitoring system in the daytime of February 13, 2019. It is a figure which shows the image which the PoC1 model does not erroneously detect, and the original image, under the situation that erroneous detection is easy when the optical flow model is adopted.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same parts are designated by the same reference numerals. Their names and functions are the same. Therefore, the detailed description of them will not be repeated.

The outline of the technical idea according to the present disclosure will be described with reference to FIG. FIG. 1 is a flowchart showing a part of processing executed by a processor of a computer system operating as a monitoring system for detecting suspended matter at an early stage.

First, the monitoring system accepts the input of the frame image obtained by photographing the inspection target and prepares the processing of the optical flow. More specifically, in step S110, the processor acquires a frame image as an image taken by the camera. In step S120, the processor sets a plurality of feature points in the first region in the v-th frame image of the period u. In step S125, the processor tracks each feature point.

The monitoring system sets criteria based on optical flow. More specifically, in step S130, the processor calculates the number of feature points included in the second region in the frame image. In step S135, the processor sets the judgment criterion A based on the optical flow.

Next, the monitoring system sets the determination criterion B based on the trained model or the pixel count ratio. More specifically, in step S140, the processor executes a floating object determination process for each pixel in the frame image. In step S145, the processor calculates the ratio of the number of pixels determined to be floating matter. In step S150, the processor sets the determination criterion B based on the trained model or the pixel count ratio.

Next, the monitoring system executes the detection process of suspended matter. More specifically, in step S160, the processor determines whether or not the result of the determination process of the suspended matter satisfies the determination criterion A, and the trained model or the ratio of the number of pixels satisfies the determination criterion B. To judge. In step S170, the processor determines that suspended matter is present. In step S180, the processor executes a predetermined notification process. In step S190, the processor sets the next frame image as the processing target.

The order in which the determination criteria A are set may be reversed from the order in which the determination criteria B are set. Therefore, steps S130 and S135 may be performed after steps S140 to S150.

Next, the details of the detection of snow jam will be described with reference to FIGS. 2 to 17.
In the following embodiment, snow jam will be described as an example as a floating substance (an object that causes clogging of the waterway) flowing through the waterway. However, the suspended matter is not limited to snow jam, and for example, in the case of a power plant using seawater as a cooling means, the suspended matter may be a group of jellyfish. Further, the numerical values such as the number of sheets, the number of sheets, and the threshold value shown below are examples and are not limited to the specific numerical values described.

FIG. 2 is a diagram for explaining snow jam.
With reference to FIG. 2, as shown in state (i), the snow on the mountain falls. Then, as shown in state (ii), the fallen snow flows into the waterway. This causes snow jam. Further, as shown in the state (iii), the upper water tank provided upstream of the power plant is clogged with snow jam. This causes water to overflow from the channel, as shown in state (iv). As a result, the water in the channel (cooling water) is no longer supplied to the power plant, and power generation is stopped as shown in the state (v). Therefore, if the occurrence of snow jam can be detected at an early stage, it is possible to prevent the power generation from stopping.

In addition, the place where snow jam occurs is usually a place where the workers of the power plant are not stationed. Therefore, in this example, the occurrence of snow jam is detected by using the image of the waterway captured by the camera.

By the way, from the early stage to the middle stage of the occurrence of snow jam, the waterway is not clogged by the snow jam and there is a water flow. Therefore, snow jam flows through the waterways. On the other hand, in the latter part of the snow jam, the waterways are clogged and the snow begins to stay in one place. In this example, in consideration of such characteristics, snow jam is detected from the early stage to the middle stage of the occurrence of snow jam.

FIG. 3 is a diagram showing an image obtained by imaging a water channel that supplies water to a power generation facility (power plant). Specifically, the image G1 is a diagram showing an image (still image, one frame image) at a certain point in time of a moving image (moving image). More specifically, the image G1 is an image of a place where an intake port is provided in the waterway in winter.

With reference to FIG. 3, a plurality of snow jams (objects displayed in white) are floating on the surface of the water in the waterway. In addition, snow is piled up on the upper part of the intake (upper left part of the image G1).

In such a waterway, the state of the water surface changes due to the waves generated by the water flow itself or the wind. In addition, the water surface is illuminated by nighttime lighting, and nighttime lighting is reflected even at the bottom of the water. In addition, snowfall will occur. As described above, in the waterway, the images captured are not uniform due to the difference in weather conditions including the difference in sunshine conditions including day and night.

In this example, a system capable of accurately detecting the occurrence of snow jam even under such conditions will be described.

<A. System configuration>
FIG. 4 is a diagram for explaining the system configuration of the monitoring system.

With reference to FIG. 4, the surveillance system 1 includes a server device 100, a terminal device 200, and a surveillance camera 300. The surveillance camera 300 is communicably connected to the server device 100. The terminal device 200 is connected to the server device 100 so as to be able to communicate via the network.

The surveillance camera 300 is installed near the water channel 500 in order to image the water channel 500. In this example, the surveillance camera 300 is installed so as to take an image of a place where the intake port 700 is provided in order to monitor the state of the intake port 700 in the water channel 500. The water intake 700 is provided with a fence so that large foreign substances (driftwood, etc.) having a certain shape do not flow into the power generation facility.

The image (moving image) captured by the surveillance camera 300 is sent to the server device 100. The image can be viewed on a device such as a terminal device 200, for example. That is, the worker of the power plant or the like can monitor the state of the water intake 700 by the image.

Further, in the surveillance system 1, the server device 100 detects the occurrence of snow jam by using the image captured by the surveillance camera 300. Specifically, by using an optical flow, which is generally used in the field of motion detection, in this embodiment, the movement of feature points, which will be described later, is tracked. As a result, the server device 100 detects the occurrence of snow jam. Since optical flow is a known technique, it will not be described repeatedly here.

More specifically, in the following, a configuration for determining the presence or absence of snow jam using 100 consecutive frame images as one determination unit will be described as an example. That is, the server device 100 determines whether or not snow jam has occurred based on the first 100 frame images, and then determines whether or not snow jam has occurred based on the next 100 frame images. After that, the server device 100 repeats such a determination process.

<B. Feature point setting and movement detection>
FIG. 5 is a diagram for explaining a plurality of reference coordinates (positions) used for setting feature points.

With reference to FIG. 5, in the case of this example, 40 reference coordinates D1 to D40 are preset (designated) in the server device 100. The reference coordinates are set, for example, by a user (user of the terminal device 200) of the program used for determining snow jam. Alternatively, the designer may set the reference coordinates in advance by default for the program.

The setting of the reference coordinates can be appropriately changed depending on the image pickup target and the image pickup direction. In the case of this example, 40 reference coordinates are set at substantially equal intervals on the upstream side (the lower right region of the image) of the water channel 500. For example, when the user accesses the server device 100 from the terminal device 200, the reference coordinates can be set or the reference coordinates can be changed in the server device 100. In the following, i is a natural number from 1 to 40, and the reference coordinates are expressed using "Di".

FIG. 6 is a diagram for explaining an example of a method of setting an initial position of a feature point.
With reference to FIG. 6, the server device 100 sets the initial coordinates Ei [1] of the feature points based on the reference coordinates Di. Typically, the server device 100 sets the initial coordinates Ei [1] of one feature point in the element region Ai of a predetermined size including the reference coordinate Di. In order to prevent the initial coordinates Ei of the feature points from overlapping, it is preferable that the element regions Ai do not overlap with each other.

The server device 100 sets the initial coordinates Ei [1] of the feature points by, for example, performing predetermined image processing on the frame image obtained by the image pickup by the surveillance camera 300. For example, the server device 100 sets the position where the gradient (change) of the luminance is strongest in the element region Ai to the initial coordinates Ei of the feature point. By such processing, the initial coordinates Ei [1] of 40 feature points are set from the 40 reference coordinates Di.

The setting of the initial coordinates Ei [1] of the feature points can be realized by using, for example, OpenCV (Open Source Computer Vision Library), which is a library for open source computer vision. Specifically, the server device 100 sets the initial coordinates Ei by using the function "cv2.cornerSubPix ()" provided by OpenCV. More specifically, the server device 100 sets the argument of the function to "self.gray_next" (that is, the image is a gray image), and sets (recognizes) the corner from the difference in the brightness of each adjacent pixel. Since the setting of the initial coordinates Ei [1] of the feature points may be performed by using such a known technique, further detailed description will not be repeated here.

Next, the movement mode of the feature points will be described with an example.
(B1. First case)
The mode of movement of the feature points when the snow jam is in a beaded state will be described.

FIG. 7 is a diagram showing the initial coordinates Ei [1] of the feature points.
With reference to FIG. 7, in the first frame image F1, the server device 100 has initial coordinates E1 [1], E2 [1], ..., E8 [1] of 40 feature points based on each reference coordinate Di. ], ..., E16 [1], ..., E33 [1], ..., E40 [1] are set. Since each element area Ai is set in advance, the initial coordinates E1 [1] to E40 [1] of the feature points are set in the predetermined area Q2 (example of the second area). The "first frame image" means the first frame image out of 100 images.

Further, the area Q2 is set so as not to overlap with the area Q1 (example of the first area). The region Q1 is preset so that at least a part of the region Q1 overlaps with the region where the image of the intake port 700 that takes in the water of the water channel 500 into the equipment is displayed. The area Q2 is also set by the user or the designer of the program.

FIG. 8 is a diagram showing the 50th frame image F50.
With reference to FIG. 8, a plurality of feature points are moved within the region Q1. As described above, the movement of such feature points can be detected by using an optical flow. Note that Ei [50] in FIG. 7 represents the coordinates of the feature points in the 50th frame image F50.

FIG. 9 is a diagram showing the 100th frame image F100.
With reference to FIG. 9, more feature points have moved into the region Q1 than in the state of FIG. Based on FIGS. 7, 8 and 9, it can be seen that the feature points move to the region Q1 on the downstream side of the region Q2 with the passage of time in the period of 100 frame images. Note that Ei [100] in FIG. 9 represents the coordinates of the feature points in the 100th frame image F100.

Specifically, in the frame image F100, 15 of the 40 feature points are moved into the region Q1. Further, of the 40 feature points, 8 feature points cannot be tracked on the way and disappear.

(B2. Second case)
The mode of movement of the feature points in the state where the snow jam covers the water surface will be described.

FIG. 10 is a diagram showing the initial coordinates Ei [1] of the feature points.
With reference to FIG. 10, in the first frame image F1', the server device 100 has the initial coordinates E1 [1] and E2 [1] of 40 feature points in the region Q2 based on each reference coordinate Di. , ..., E8 [1], E9 [1], ..., E33 [1], ..., E40 [1] are set.

FIG. 11 is a diagram showing the 100th frame image F100'.
With reference to FIG. 11, many feature points have moved into the region Q1. With the passage of time in the period of 100 frame images, the feature points move to the region Q1 on the downstream side of the region Q2.

Specifically, in the frame image F100', 33 of the 40 feature points are moved into the region Q1. Further, of the 40 feature points, 37 feature points remain without disappearing. When snow jam covers the surface of the water, there is a tendency that there are few characteristic points that disappear.

(B3. Third case)
At the initial stage of snow jam, the number of feature points moved to the region Q1 in the 100th frame image is smaller than that in the first case. In addition, the number of feature points that disappear on the way is also larger than that in the first case.

<C. Judgment criteria setting>
The server device 100 includes "A: initial state of occurrence of snow jam" (hereinafter, also referred to as "A state"), "B: state in which snow jam is connected in beads" (hereinafter, also referred to as "B state"), and ". C: It is configured to determine "a state in which the snow jam covers the water surface" (hereinafter, also referred to as "C state").

Specifically, the server device 100 has the number R of feature points in the region Q1 in the 100th frame image and the number Z of feature points in the entire 100th frame image (hereinafter, also referred to as “remaining number Z”). , The occurrence state of snow jam is determined based on the threshold values Tha and Thb regarding the number R and the threshold values Thc regarding the remaining number Z.

In the following, it should be detected in the order of "A: Initial state of snow jam occurrence", "B: Snow jam is connected in beads", and "C: Snow jam is covering the water surface". An example of a configuration set to have a high priority will be described.

In the case of this example, the threshold value Thc regarding the remaining number Z is set to 30. Further, the threshold value Thb regarding the number R is set to 15. Further, the threshold value Th regarding the number R is set to 5.

If the number of remaining feature points in the 100th frame image is 30 or more (threshold value Thc or more), the server device 100 states that "C: Snow jam covers the water surface" regardless of other conditions. judge. When this determination is made, the server device 100 executes control to stop the water intake from the water intake port 700.

If the number of feature points in the region Q1 in the 100th frame image is 15 or more (threshold value Thb or more), the server device 100 is subjected to the condition that "C: Snow jam is not covering the water surface". Judged as "B: Snow jam is connected in a row". Even when this determination is made, the server device 100 executes control to stop the water intake from the water intake port 700.

If the number of feature points in the region Q1 in the 100th frame image is 5 or more and less than 15 (threshold value Th or more and less than the threshold value Thb), it means that "C: Snow jam covers the water surface". On the condition of, the server device 100 determines "A: initial state of occurrence of snow jam". When this determination is made, the server device 100 notifies the terminal device 200 of a predetermined warning. The notification destination of the warning may be any device registered in advance, and is not limited to the terminal device 200.

<D. Verification>
Verification of the effectiveness of the surveillance system 1 (accuracy of detection by optical flow) using the moving image data (moving image data consisting of a plurality of continuous frame images) obtained by past imaging using the surveillance camera 300. did. In this example, the surveillance camera 300 is assumed to perform imaging 10 times per second.

(D1. First verification result)
FIG. 12 is a diagram for explaining the verification result based on the moving image data at night.

(1) Waterway diary With reference to FIG. 12, it is recorded in the waterway diary that snow jam occurred at 4:05 am within the period T5. The recording in the waterway diary is performed by the on-site observer visually confirming the moving image obtained by the surveillance camera 300 in real time.

(2) Confirmation result at a later date The result of confirming the occurrence state of snow jam by another person watching such a moving image at a later date is described in the time series table of "visual confirmation".

During the period T1 (17289 frame image: about 28.8 minutes), as a result of gazing, it was found that "A: initial state of snow jam occurrence" was intermittently continued. In the period T2 (372 frame images: about 0.6 minutes) immediately after the period T1, as a result of gazing, it was found that the state was "B: Snow jam is connected in a row". In the period T3 (1355 frame image: about 2.2 minutes) immediately after the period T2, it was found that the state was "C: Snow jam covering the water surface".

In the period T4 (1806 frame image: about 3 minutes) immediately after the period T3, it was found that the state was "B: Snow jam is connected in beads". In other words, it was found that the amount of snow jam was once reduced. In the period T5 (703 frame image: about 1.2 minutes) immediately after the period T4, it was found that the state was again "C: a state in which the snow jam covers the water surface". According to the waterway diary, the occurrence of snow jam was recorded during this period T5. In the period T6 (1054 frame image: about 1.8 minutes) immediately after the period T5, it was found that the state was "B: Snow jam is connected in beads". In other words, it was found that the amount of snow jam generated decreased again.

In the period T7 (8893 frame image: about 14.8 minutes) immediately after the period T6, as a result of gazing, the state is again "C: the state where the snow jam covers the water surface". found.

In the period T8 (408 frame images: about 0.7 minutes) immediately after the period T7, the surveillance camera 300 was intentionally moved to take an image other than the intake port 700. Therefore, regardless of the state in which the snow jam is derived, any of the A state, the B state, and the C state is not determined. Immediately after the period T8, the position of the surveillance camera 300 was returned to the original position.

In the period T9 (4190 frame image: about 7 minutes) immediately after the period T8, it was found that the state was "C: Snow jam covering the water surface".

Since each of the frame images cannot be discriminated by human eyes, the above-mentioned number of frames is a reference value.

(3) Detection result using optical flow During the period T1, it was detected that "A: Initial state of snow jam occurrence" was intermittently continued by the state determination based on the movement of the feature points described above. .. Specifically, in the detection of about 172 to 173 times (≈17289 frames ÷ 100 frames), the A state was detected 21 times.

During the period T2, "A: initial state of occurrence of snow jam" and "C: state of snow jam covering the water surface" were detected. Specifically, in the four detections, the A state was detected three times and the C state was detected once. In FIG. 11, it is described as if the A state and the C state occur at the same timing, but this is for convenience, and the occurrence detection timing of the A state and the C state is There is no duplication. This also applies to the subsequent periods T3 to T9.

During the period T3, "A: initial state of occurrence of snow jam", "B: state of beaded snow jam", and "C: state of snow jam covering the water surface" were detected. Specifically, in the 14 detections, the A state was detected 3 times, the B state was detected 8 times, and the C state was detected 3 times.

In the period T4, the A state was detected 6 times, the B state was detected 10 times, and the C state was detected 2 times. In the period T5, the A state was detected once, the B state was detected four times, and the C state was detected once. According to the waterway diary, as mentioned above, the occurrence of snow jam is recorded during this period T5.

Hereinafter, similarly, in the period T6, the A state was detected 4 times and the B state was detected 6 times. In the period T7, the A state was detected 13 times, the B state was detected 65 times, and the C state was detected 6 times. In the period T8, as described above, since the surveillance camera 300 was intentionally moved to take an image other than the intake port 700, any one of the A state, the B state, and the C state was not detected. In the period T9, the A state was detected 3 times, the B state was detected 27 times, and the C state was detected 10 times.

As is clear from the comparison with the visual confirmation results at a later date, it can be said that the detection results by optical flow generally show correct results. In particular, the occurrence of snow jam can be detected before the time of occurrence of snow jam described in the waterway diary. As described above, according to the monitoring system 1, the occurrence of snow jam can be automatically detected.

In addition, when the snow jam is connected in a row, if the snow jam is taken into the equipment, the power generation in the equipment is stopped as described above. However, since the monitoring system 1 can detect that "B: snow jam is in a beaded state" and "C: snow jam is in a state of covering the water surface", these detection timings. By controlling to stop the stop of water intake (water supply to the equipment), it is possible to prevent the power generation from stopping due to the inflow of snow jam into the equipment.

(D2. Second verification result)
FIG. 13 is a diagram for explaining the verification result based on the moving image data in the daytime.

(1) Waterway diary With reference to FIG. 13, it is described in the waterway diary that water intake was stopped between 15:05 and 6 minutes within the period T14. Furthermore, it is stated in the waterway diary that the discharge was started between 15:06 and 7 minutes within the period T15.

(2) Confirmation result at a later date As in FIG. 12, the result of confirming the occurrence state of snow jam by another person watching such a moving image at a later date is described in the time series table of "visual confirmation". is doing.

The period T11 and the period T13 are periods in which the surveillance camera 300 is intentionally moved and images other than the intake port 700 are imaged. Therefore, the state of occurrence of snow jam cannot be determined.

In the period T12 (724 frame image: about 1.2 minutes) between the period T11 and the period T13, the state is "C: Snow jam covering the water surface" as a result of gaze. There was found. Also, in the period T14 (4250 frame image: about 7 minutes) immediately after the period T13, it was found that the state was "C: the state where the snow jam covers the water surface" as in the period T12. ..

In the period T15, since the discharge from the C state was started, it was confirmed that a state similar to the snow jam occurred, although it was not in the snow jam state. Immediately after the period T15, the occurrence of snow jam was not visually recognized, and it was found that water was flowing.

(3) Detection result using optical flow During the period T12, the A state was detected once, the B state was detected four times, and the C state was detected twice. During the period T14, the A state was detected 12 times, the B state was detected 23 times, and the C state was detected twice.

In the period T15 immediately after the start of discharge, none of the A state, B state, and C state was detected. As described above, according to the monitoring system 1 using the optical flow, even if it is "a state similar to snow jam, although it is not a snow jam state" (a state of period T15), it is a "snow jam state". Is not falsely detected. As described above, according to the monitoring system 1, it is possible to perform highly accurate detection even when the water is discharged.

<E. Functional configuration>
FIG. 14 is a functional block diagram for explaining the functional configuration of the server device 100 of the monitoring system 1.

With reference to FIG. 14, the server device 100 is communicably connected to the surveillance camera 300, the water intake device 400, and the terminal device 200. The server device 100 includes a communication IF (Interface) unit 101 and a control unit 102.

The communication IF unit 101 is an interface for the server device 100 to communicate with another device. The communication IF unit 101 includes a receiving unit 111 and a transmitting unit 112.

The control unit 102 controls the overall operation of the server device 100. The control unit 102 is typically realized by the processor 151 (see FIG. 16) executing a program. The control unit 102 includes a reference coordinate setting unit 141, a feature point setting unit 142, a detection unit 143, a determination unit 144, a water supply control unit 145, and a notification unit 146.

The receiving unit 111 continuously receives frame images from the surveillance camera 300 that images the water channel 500 that supplies water to the equipment.

The reference coordinate setting unit 141 sets a plurality of reference coordinates Di based on an external input (for example, a user operation). In addition, when the reference coordinate Di is set by default, external input is not always necessary. Each reference coordinate Di is sent to the feature point setting unit 142.

When the control unit 102 receives an input (for example, a user operation) instructing the start of the snow jam detection process, the feature point setting unit 142 sets the frame image received at the time of the input as the first frame image (100). As the first frame image in the sheet), 40 feature points are set in the area Q2 of the first frame image (see FIG. 6). The coordinates Ei of the set feature points are sent to the detection unit 143.

The detection unit 143 detects the movement of 40 feature points set in the first frame image out of 100 consecutive frame images obtained by imaging for each feature point over 100 frame images. That is, the detection unit 143 tracks the feature points. Note that "detecting each feature point over 100 frame images" means tracking 99 movements (corresponding to the number of frames between frame images in 100 frame images).

The detection unit 143 sends the detection result to the determination unit 144. Specifically, the detection unit 143 sends the remaining number Z in the entire 100th frame image and the number R of feature points in the region Q1 of the 100th frame image to the determination unit 144.

The determination unit 144 states that the snow jam occurrence state is "A: initial state of snow jam occurrence", "B: snow jam is connected in beads", and "C: snow jam is covering the water surface". Which state is determined.

The determination unit 144 is based on the determination that the number of feature points existing in the region Q1 in the last frame image of the 100 frame images is the threshold value Thb (15) or more in relation to the above-mentioned B state. It is determined that snow jam exists in the water channel 500. Specifically, the determination unit 144 determines that the snow jam occurrence state is "B: the snow jam is in a beaded state" (B state).

Further, when the determination unit 144 determines that the number of feature points in the last frame image is the threshold value Thc (30) or more in relation to the above-mentioned C state, snow jam is present in the water channel 500. Is determined. Specifically, the determination unit 144 determines that the snow jam occurrence state is "C: a state in which the snow jam covers the water surface" (C state). More specifically, the determination unit 144 determines that the number of feature points existing in the region Q1 in the last frame image is less than the threshold value Thb (15), but the number of feature points in the last frame image. When it is determined that the threshold value is Thc (30 pieces) or more, it is determined that snow jam is present in the water channel 500. In the example of the present embodiment, since the degree of occurrence of snow jam is larger in the C state than in the B state, the determination of the C state has priority over the determination of the B state.

Further, the determination unit 144 snow jams when the number of feature points existing in the region Q1 in the last frame image (100th frame image) is less than the threshold Thb (15) and the threshold Tha (5) or more. It is determined that the occurrence state of is "A: initial state of occurrence of snow jam" (A state).

When the determination unit 144 determines that the state is B or C, the determination unit 144 transmits a predetermined instruction to the water supply control unit 145. In this case, when the water supply control unit 145 receives the above instruction from the determination unit 144, the water supply control unit 145 transmits a water intake stop command to the water intake device 400 of the equipment via the transmission unit 112. Upon receiving the water intake stop command, the water intake device 400 stops the water supply (water intake) from the water channel 500 to the equipment.

Further, when the determination unit 144 determines that the state is A, the determination unit 144 transmits a predetermined instruction to the notification unit 146. In this case, when the notification unit 146 receives the above instruction from the determination unit 144, the notification unit 146 performs a predetermined notification to the terminal device 200 via the transmission unit 112. Typically, the notification unit 146 notifies the terminal device 200 that a snow jam has occurred.

<F. Control structure>
FIG. 15 is a flow chart for explaining the flow of the first half of the processing executed by the server device 100.

With reference to FIG. 15, in step S101, the server device 100 (specifically, the processor 151) starts acquiring the frame image obtained by the surveillance camera 300. In step S102, the server device 100 resets the values of the predetermined variables u and v to 1.

In step S103, the server device 100 sets 40 feature points in the area Q2 of the v-th frame image in the u-th period. In step S104, the server device 100 increments the value of the variable v. That is, the server device 100 increases the value of v by 1.

In step S105, the server device 100 tracks each feature point in the vth frame image in the uth period. In step S106, the server device 100 determines whether or not the value of the variable v is 100 or more.

When the server device 100 determines that the value of v is less than 100 (NO in step S106), the server device 100 proceeds to the process in step S104. When the server device 100 determines that the value of v is 100 or more (YES in step S106), in step S107, the number of feature points included in the entire v-th frame image (that is, the remaining number Z) is determined. calculate. Next, in step S108, the server device 100 calculates the number R of feature points included in the area Q1 of the v-th frame image. After that, the server device 100 advances the process to step S109 (FIG. 15).

FIG. 16 is a flow chart for explaining the flow of the latter half of the processing executed by the server device 100.

With reference to FIG. 16, in step S109, the server device 100 (specifically, the processor 151) determines whether or not the remaining number Z is 30 (threshold value Thc) or more. When the server device 100 determines that the remaining number Z is 30 or more (YES in step S109), the server device 100 determines that the snow jam generation state is "C: a state in which the snow jam covers the water surface" in step S114. judge. After that, the server device 100 advances the process to step S112.

When the server device 100 determines that the remaining number Z is less than 30 (NO in step S109), whether or not the number R of the feature points included in the region Q1 is 15 (threshold value Thb) or more in step S110. To judge. When the server device 100 determines that the number R is 15 or more (YES in step S110), the server device 100 determines in step S115 that the snow jam generation state is "B: the snow jam is in a beaded state". After that, the server device 100 advances the process to step S112.

When the server device 100 determines that the number R is less than 15 (NO in step S110), whether or not the number R of the feature points included in the region Q1 is 5 (threshold value Tha) or more in step S111. To judge. When the server device 100 determines that the number R is 5 or more (YES in step S111), the server device 100 determines that the snow jam occurrence state is "A: snow jam occurrence initial state" in step S116. After that, the server device 100 advances the process to step S112.

In step S112, the server device 100 increments the value of the variable u. In step S113, the server device 100 resets the value of the variable v to 1. After that, the server device 100 executes the snow jam detection process using the next 100 frame images by advancing the process to step S103 (FIG. 14).

<G. Hardware configuration>
FIG. 17 is a diagram showing a typical example of the hardware configuration of the server device 100.

With reference to FIG. 17, the server device 100 has, as main components, a processor 151 for executing a program, a ROM 152 for storing data non-volatilely, and data or an input device generated by executing the program by the processor 151. It includes a RAM 153 that volatilely stores the data input via the data, an HDD 154 that stores the data non-volatilely, a communication IF (Interface) 155, an operation key 156, a power supply circuit 157, and a display 158. Each component is connected to each other by a data bus. The communication IF 155 is an interface for communicating with other devices.

The processing in the server device 100 is realized by the software executed by each hardware and the processor 151. Such software may be stored in the HDD 154 in advance. In addition, the software may be stored in other storage media and distributed as a program product. Alternatively, the software may be provided as a downloadable program product by an information provider connected to the so-called Internet. Such software is read from the storage medium by a reading device, or downloaded via a communication IF 155 or the like, and then temporarily stored in the HDD 154. The software is read from the HDD 154 by the processor 151 and stored in the RAM 153 in the form of an executable program. Processor 151 executes the program.

Each component constituting the server device 100 shown in the figure is a general one. Therefore, it can be said that an essential part of the present invention is RAM 153, HDD 154, software stored in a storage medium, or software that can be downloaded via a network. Since the operation of each hardware of the server device 100 is well known, the detailed description will not be repeated.

Since the terminal device 200 has the same configuration as the server device 100, the hardware configuration of the terminal device 200 will not be repeatedly described.

<H. Summary>
Extracting a part of the above-mentioned processing is as follows.

(1) The monitoring system 1 is predetermined by a surveillance camera 300 that images a water channel 500 that supplies water to the equipment, and a predetermined frame image that is set in the first frame image among a plurality of continuous frame images obtained by the imaging. The number of feature points existing in the detection unit 143 that detects the movement of the number of feature points for each feature point over the plurality of frame images and the region Q1 in the last frame image of the plurality of frame images. However, it is provided with a determination unit 144 for determining that snow jam is present in the water channel 500 based on the determination that the threshold value is Thb or higher. With such a configuration, it becomes possible to accurately detect the snow jam flowing through the water channel 500.

(2) Further, even when the determination unit 144 determines that the number of the feature points existing in the region Q1 in the last frame image is less than the threshold value Thb, the determination unit 144 of the feature points in the last frame image. When it is determined that the number is equal to or greater than the threshold value Thc, it is determined that snow jam is present in the water channel 500. Also in this case, it is possible to accurately detect the snow jam flowing through the water channel 500.

(3) The monitoring system 1 further includes a water supply control unit 145 that stops the supply of water to the above equipment based on the determination that snow jam exists in the water channel 500. With such a configuration, it is possible to prevent the equipment from stopping.

(4) The monitoring system 1 is a notification unit that performs predetermined notification based on the determination that the number of the feature points existing in the region Q1 in the last frame image is less than the threshold value Thb and equal to or more than the threshold value Tha. Further equipped with 146. With such a configuration, the equipment manager and the like can know the occurrence of snow jam at an early stage.

(5) The monitoring system 1 further includes a feature point setting unit 142 for setting the predetermined number of feature points in the area Q2 of the first frame image. Further, the region Q1 is a region on the downstream side of the water channel 500 with respect to the region Q2.

(6) The feature point setting unit 142 sets each of the plurality of feature points with reference to the predetermined number of predetermined coordinates in the area Q2. According to such a configuration, the server device 100 automatically sets (searches) the feature points only by the user or the like specifying the coordinates.

(7) The feature point setting unit 142 sets one of the feature points in the element region Ai having a predetermined size including the designated coordinates (reference coordinates). According to such a configuration, it is possible to reduce the overlap of the coordinates of the feature points.

(8) The area Q1 partially overlaps with the area where the image of the intake port 700 that takes in the water of the water channel 500 into the above equipment is displayed. Since a fence is installed at the intake 700, snow jam tends to collect at the intake 700. Therefore, according to such a configuration, the occurrence of snow jam can be detected with high accuracy.

<I. Modification example>
(1) In the above, the location of the water intake is imaged by the surveillance camera 300, but the present invention is not limited to this. However, the water intake is often provided with a surveillance camera for the purpose of allowing workers to check the vicinity of the water intake by video, and by using this surveillance camera, the cost can be suppressed. Further, since the floating matter is likely to stay in the intake port 700 due to the fence, it is suitable for detecting the presence or absence of the floating matter as compared with other places.

(2) The equipment that receives water from the waterway 500 is not limited to the power generation equipment (power plant).

<False detection factor>
With reference to FIGS. 18 and 19, false positive factors when using an optical flow or a trained model will be described.

FIG. 18 is a diagram showing a false positive factor. In Photo A, region 1800 shows a state in which the reflection of illumination expands and contracts in the direction of the flow of water. This expansion and contraction can be a factor in false detection of feature points in optical flow. Photo B shows a state in which light is reflected on the surface of the waterway in the daytime. The image C is an image obtained by performing binarization processing by a trained model (sometimes referred to as a “PoC1 model”) using the image data of the photograph B. In the case of the image C based on the photograph B, it is erroneously detected that the snow area ratio at the time of observation is 35.2% as an instantaneous value. Further, when the average value for 5 minutes is used for detecting the occurrence of snow jam, it is erroneously detected as the snow area ratio is 29.3%.

In the present embodiment, the PoC1 model refers to a trained model obtained by training using an image used for detecting snow jam using an image obtained by photographing a certain waterway. .. According to the PoC1 model, the monitoring system 1 reacts sensitively to snow flow, but may erroneously detect reflections from the water surface as snow jam.

The PoC2 model refers to a trained model obtained by additionally learning an image showing a false positive factor or a snow jam image of another place (for example, a power plant) on the PoC1 model. According to the PoC2 model, false positives that occur when the PoC1 model is used can be significantly reduced.

FIG. 19 is a diagram showing an example of reference coordinates of feature points and arrangement of frames in detection using optical flow. The state (A) represents an example of the arrangement of the feature points and the frame 1910 when the suspended matter flows in the lateral direction in the image. The state (B) represents an example of the arrangement of the feature points and the frame 1920 when the floating matter flows in the oblique direction in the image.

When adopting the optical flow model as shown in FIG. 19, it is necessary to set the size, shape and position of the frames 1910 and 1920 as the region Q1 according to the direction in which the suspended matter flows and the speed at which the suspended matter flows. .. However, the flow direction and speed of the suspended matter greatly depend on the flow of the channel to be observed and other actual environments. Therefore, it is difficult to perform pre-tuning before the monitoring system is in operation.

A control structure of a monitoring system according to an embodiment will be described with reference to FIG. FIG. 20 is a flowchart showing a part of the processing executed by the processor 151 of the server device 100 functioning as a monitoring system.

In step S2010, the processor 151 starts acquiring the frame image. The frame image is transmitted to the server device 100 as, for example, an image captured by the surveillance camera 300.

In step S2015, the processor 151 resets the values of the variables u and v to 1 for initialization.

In step S2020, the processor 151 sets a preset number (for example, 40) of feature points in the region Q2 of the v-th frame image in the u-th period.

In step S2025, the processor 151 increments the value of the variable v (v ← v + 1).

In step S2030, the processor 151 tracks each feature point in the vth frame image in the uth period and identifies the coordinates of each feature point.

In step S2035, the processor 151 determines whether or not the value of the variable v is 6 or more. When the value of the variable v is 6 or more (YES in step S2035), the processor 151 switches the control to step S2040 and step S2050. Otherwise (NO in step S2035), processor 151 returns control to step S2025.

In step S2040, the processor 151 calculates the number R of feature points included in the area Q1 of the v-th frame image.

In step S2041, the processor 151 determines whether or not the number of stored array ac5 is 5. When the processor 151 determines that the number of stored array ac5 is 5 (YES in step S2041), the processor 151 switches the control to step S2042. If not (NO in step S2041), processor 151 switches control to step S2043.

In step S2042, the processor 151 deletes and initializes the fifth value in the array ac5.

In step S2043, the processor 151 registers the number R in the first array ac5.

In step S2044, the processor 151 calculates the total value ac5sum of the array ac5.

In step S2045, the processor 151 determines whether or not the number of stored array ac12 is 12. When the processor 151 determines that the number of stored array ac12 is 12 (YES in step S2045), the processor 151 switches the control to step S2046. If not (NO in step S2045), processor 151 switches control to step S2047.

In step S2046, the processor 151 deletes and initializes the twelfth value in the array ac12.

In step S2047, the processor 151 registers the total value ac5sum in the first array ac12.

In step S2048, the processor 151 calculates the total value ac12sum of the array ac12.

In step S2050, the processor 151 performs a snow jam determination by the PoC1 model for each pixel of the v-th frame image.

In step S2051, the processor 151 calculates the pixel number ratio area determined to be snow jam in the v-th frame image.

In step S2052, the processor 151 determines whether or not the previous determination result pre_area is registered. If the previous determination result pre_area is registered (YES in step S2052), the processor 151 switches the control to step S2060. If not (NO in step S2052), processor 151 switches control to step S2053.

In step S2053, the processor 151 inputs the value of the pixel number ratio area calculated in step S2051 into the previous determination result pre_area.

In step S2060, the processor 151 has a threshold value in which the total value ac12sum of the array ac12 is larger than a preset threshold value (for example, 115), and the difference between the pixel number ratio area and the previous determination result pre_area is preset. It is determined whether or not it is larger than (for example, 4). When the total value ac12sum of the array ac12 is larger than the preset threshold value (for example, 115), and the difference between the pixel number ratio area and the previous determination result pre_area is larger than the preset threshold value (for example, 4). (YES in step S2060), processor 151 switches control to step S2070. If not (NO in step S2070), processor 151 switches control to step S2080.

In step S2070, the processor 151 determines that snow jam has occurred in the monitored area. The processor 151 displays on the display 158 of the server device 100 that snow jam has occurred. In another aspect, the processor 151 sends a warning to other information terminals connected to the server device 100 notifying that snow jam has occurred. The warning is transmitted to, for example, a mobile communication terminal used by the administrator of the monitored area.

In step S2080, the processor 151 inputs the value of the pixel number ratio area calculated in step S2051 into the previous determination result pre_area.

In step S2085, the processor 151 increments the value of the variable u by 1.

In step S2090, the processor 151 resets the value of the variable v to 1.
In FIG. 20, the process of deriving the judgment criteria based on the optical flow model (steps S2040 to S2048) and the process of deriving the judgment criteria based on the trained model (PoC1 or PoC2) (steps S2050 to 2053) are Although exemplified in a mode in which they are executed in parallel, in other aspects, these processes may be executed sequentially.

The server device 100 constituting the monitoring system 1 according to the present embodiment suppresses erroneous detection at the time of snow jam judgment by using the judgment standard based on the optical flow model and the judgment standard based on the trained model in combination. However, snow jam can be detected at an early stage. The difference in detection accuracy based on each model will be described below.

<Comparison of detection accuracy of snow jam>
A comparison result of the accuracy of each method for detecting snow jam will be described with reference to FIGS. 21 and 22. FIG. 21 is a diagram showing the timing at which snow jam is detected when the security is stopped at 12:30 at a certain power plant.

[PoC2 model]
The graph (A) shows the detection timing when the PoC2 model is estimated every 4 seconds and the average of 5 minutes is adopted. In this case, the server device 100 determines that the snow jam was detected at 11:11:39.

The graph (B) shows the detection timing when the PoC2 model estimates every 4 seconds and the detection of snow jam is determined by the difference from the previous value. In this case, the server device 100 determines that the snow jam was detected at 11:10:23.

[PoC1 model & optical flow]
The graph (C) and the graph (D) represent the timing when snow jam is detected by the combination of the PoC1 model and the optical flow. The difference between the graph (C) and the graph (D) is the content of tuning (setting) in the optical flow.

More specifically, the graph (C) shows the case where the PoC1 model estimates every 4 seconds and the average for 5 minutes is adopted, and the judgment result by the optical flow in which "tuning 1" is performed is combined. Represents the timing of detection. In this case, the server device 100 determines that the snow jam was detected at 9:45:40 and 11:11:51, respectively. In tuning 1, when the 5-minute average value of the inference result of the PoC1 model is used, the cumulative value of the snow area ratio and the optical flow is adopted as the threshold value.

The graph (D) shows the detection timing when the PoC1 model is estimated every 4 seconds and the average for 5 minutes is adopted, and the judgment result by the optical flow in which "tuning 2" is performed is combined. In this case, the server device 100 determines that the snow jam was detected at 9:45:20 and 11:11:51, respectively. In tuning 2, the difference (change amount) of the inference result of the PoC1 model from the previous value is adopted as the threshold value.

[Optical flow]
The graph (E) shows the detection result of snow jam when only the optical flow is adopted. That is, it is determined that the server device 100 has detected the snow jam at 9:45:40 and 11:11:59, respectively.

As is clear from graphs (A) and (B), snow jam is detected only once when only the PoC2 model is adopted. That is, the snow jam after 9:45 is not detected by the PoC2 model. On the other hand, as is clear from the graphs (C), graphs (D) and graphs (E), according to each method in which optical flow is combined, snow jam after 9:45 is also detected. Has been done.

FIG. 22 is a diagram showing an example of a detection result including a case where a snow jam is erroneously detected due to water droplets adhering to a camera even though there is no safety stop at a certain power plant. That is, according to the diary, it is observed that water droplets are attached to the camera at around 8:08:47. It is recorded that "dripping snow" that did not lead to snow jam was observed at 10:14:31. At the power plant where the observation shown in FIG. 22 was performed, no security suspension was performed.

The graph (A) shows the detection timing when the PoC2 model is estimated every 4 seconds and the average of 5 minutes is adopted. In this case, the server device 100 determines that the snow jam was detected at 8:13:42. At the time when the subsequent snow flow was observed, the server device 100 did not determine that the snow jam was detected, and no erroneous detection occurred at this time.

The graph (B) shows the detection timing when the PoC2 model estimates every 4 seconds and the detection of snow jam is determined by the difference from the previous value. In this case, the server device 100 determines that the snow jam was detected at 8:12:38. Further, it is determined that the server device 100 has detected the snow jam at 10:16:06.

[PoC1 model & optical flow]
The graph (C) and the graph (D) represent the timing when snow jam is detected by the combination of the PoC1 model and the optical flow. The difference between the graph (C) and the graph (D) is the content of tuning in the PoC1 model.

As is clear from the graphs (C) and (D), in both cases, the server device 100 determines that the snow jam was detected at 8:12:54 and 10:16:59.

[Optical flow]
The graph (E) shows the detection result of snow jam when only the optical flow is adopted. That is, it is determined that the server device 100 has detected the snow jam at 8:12:58.

The snow jam detection result will be further described with reference to FIG. 23. FIG. 23 is a diagram showing a detection result when there is a false detection factor.

Graph A shows the snow area for 5 minutes when false detection and snow jam detection are performed when water droplets adhere to the waterway reflected in the conventional monitoring system on one day (January 26, 2019). Shows the transition of the ratio of the average value of. In this case, the monitoring system erroneously detected that a snow jam had occurred at around 7:08:24. The ratio at this time was 1.735 (%). In addition, the monitoring system correctly detects the occurrence of snow jam at around 8:14:18.

In Graph B, on the other day (February 7, 2019), when water droplets adhere to the lens cover of the conventional monitoring system and wind waves are observed, false detection and snow jam detection are performed. It shows the transition of the ratio of the average value of the snow area for 5 minutes at that time. In this case, the monitoring system erroneously detected that a snow jam had occurred at around 17:35:35. The ratio at this time was 1.736 (%). Therefore, the monitoring system according to a certain embodiment may adopt 1.8 (%) as the threshold value of the ratio of the average value.

The detection results on other monitoring days will be described with reference to FIG. 24. FIG. 24 is a diagram showing changes in the snow area ratio (average for 5 minutes) observed on December 29, 2018. More specifically, snow flow began to be observed at around 11:12:34, and at 12:03:32, the waterway reflected in the monitoring system was buried in snow.

The snow jam detection result will be further described with reference to FIG. 25. FIG. 25 is a diagram showing a detection result when there is a false detection factor.

Graph A shows the snow area for 5 minutes when false detection and snow jam detection are performed when water droplets adhere to the lens cover of a conventional monitoring system on one day (January 26, 2019). Shows the transition of the ratio (difference from the previous measurement value). In this case, the monitoring system detected that snow jam occurred at around 8:21:32 and around 10:26:54.

In Graph B, on the other day (February 7, 2019), when water droplets adhere to the lens cover of the conventional monitoring system and wind waves are observed, false detection and snow jam detection are performed. It shows the transition of the ratio of the snow area (difference from the previous measurement value) for 5 minutes at that time. In this case, the monitoring system erroneously detected that snow jam had occurred at around 15:26:42 and around 17:35:35. The maximum value of the ratio at this time was 6.427 (%). Therefore, the monitoring system according to a certain embodiment may adopt 6.5 (%) as the threshold value of the difference between the ratio of the snow area and the previous value.

The detection results on other monitoring days will be described with reference to FIG. 26. FIG. 26 is a diagram showing changes in the snow area ratio (difference from the previous value) observed on December 29, 2018. More specifically, snow flow began to be observed at around 11:12:34, and at 12:03:32, the waterway reflected in the monitoring system was buried in snow.

With reference to FIG. 27, countermeasures against false positives in the monitoring system will be described. FIG. 27 is a diagram conceptually showing an example of a method adopted as a countermeasure against false positives by a monitoring system according to a certain embodiment.

For example, when the number of feature points detected when the monitoring system erroneously detects snow jam in a certain day is 9, it is conceivable to set the threshold value to 10 or more. However, when set in this way, the monitoring system cannot detect small-scale snow jams (for example, the number of detected feature points is 9 or less). Therefore, the monitoring system according to the present embodiment can detect a small-scale snow jam by using the cumulative value.

More specifically, the monitoring system according to the present embodiment accumulates the determination results for 5 times, repeats the accumulation 12 times, and detects whether or not snow jam has occurred using the results. ..

The difference in accuracy of the monitoring system will be described with reference to FIGS. 28 and 29. FIG. 28 is a diagram showing a detection result when the monitoring system uses the number of detected feature points as they are for determination as the threshold value 10. FIG. 29 is a diagram showing a detection result when the monitoring system uses the cumulative value of the number of detected feature points as the threshold value 140 for determination.

As shown in Graph A of FIG. 28, on December 29, 2018, the monitoring system detected the snow jam at around 11:13:21. In reality, snow jam has been observed around 11:12. On the other hand, as shown in Graph B, on February 7, 2019, the number of feature points detected at around 23:36 was 9, but at this time, snow jam was actually observed. Not.

As shown in Graph A of FIG. 29, on December 29, 2018, the monitoring system detected the snow jam at around 11:11:59. In reality, snow jam has been observed around 11:12. On the other hand, as shown in Graph B, on February 7, 2019, the cumulative value of the number of feature points detected around 20:40 was 129, but at this time, it was actually snow jam. Has not been observed.

As is clear from Graph A in FIG. 29, the monitoring system uses the cumulative value of the number of detected feature points to determine the occurrence of snow jam, thereby determining the number of detected feature points as in the case of FIG. 28. Snow jam can be detected earlier than when it is used as it is.

With reference to FIG. 30, the possibility of erroneous detection due to the difference in the method adopted by the monitoring system will be described. FIG. 30 is a diagram showing the detection characteristics of each method of PoC1 model only, optical flow only, and PoC1 model & optical flow combined use. Both methods can detect snow jam. However, the PoC1 model method may cause erroneous detection due to reflection from the water surface in the daytime. The optical flow method can erroneously detect the occurrence of snow jam at night due to reflections from the water surface (see images B and C in FIG. 18). On the other hand, the method in which both methods are used in combination does not cause such false detection.

In this embodiment, two adjustments (tuning 1 and tuning 2) can be applied to the monitoring system.

In tuning 1, when the 5-minute average value of the inference result of the PoC1 model is used, the cumulative value of the snow area ratio and the optical flow is adopted as the threshold value. The relationship at this time is shown as an example by the following equation, a threshold value that was difficult to set by itself for the PoC1 model can be set, and the optical flow can be made smaller than the value when tuned by itself shown in FIG. 29. You can.
・ 5-minute average snow area ratio> 4% and cumulative optical flow> 115
In tuning 2, the difference (change amount) of the inference result of the PoC1 model from the previous value is adopted as the threshold value. In this case, the relationship between the amount of change in the snow area and the cumulative value of the optical flow is shown as an example by the following equation, a threshold value that was difficult to set by itself for the PoC1 model can be set, and the optical flow is also shown in FIG. It can be made smaller than the value when tuned independently.
・ Snow area change> 4% and cumulative optical flow> 115
FIG. 31 is a diagram for explaining a combination of the evaluation result of the trained model and the optical flow. Images A to C are stored at 20 times speed (30 fps). The inference is performed once every 10 fps (once every 6.67 seconds in the original time), and the monitoring system detects the occurrence of snow jam with an average value of 5 minutes. The detection condition in this case is snow area ratio> 9.48%.

Image A represents an image taken in the early stage when snow jam occurred in a certain aspect and the result of image processing. More specifically, the image 3100 shows the detection result by the PoC1 model in the early stage when the snow jam occurred. Image 3110 shows the detection result by the PoC2 model in the early stage when snow jam occurs. In one aspect, the PoC1 model can detect the occurrence of snow jam, but the PoC2 model cannot detect the occurrence of snow jam.

Image B represents an image and the result of image processing when snow or clouds on the wall surface reflected on the water surface are erroneously detected as snow jam in other aspects. More specifically, the image 3120 represents the detection result by the PoC1 model. Image 3130 represents the detection result by the PoC2 model. In this aspect, according to the PoC1 model, the monitoring system erroneously detects that snow jam has occurred on the snow or clouds on the wall surface reflected on the water surface. On the other hand, according to the PoC2 model, the monitoring system can distinguish between the snow flowing on the surface of the channel and the snow reflected on the surface of the channel.

Image C represents an image when the occurrence of snow jam is detected by optical flow. In the determination by the optical flow, the determination is performed every 12 frames (once every 8 seconds in the original time) for the stored video. For the feature points, 42 reference coordinates are given in the first frame, and the monitoring system determines the occurrence of snow jam based on the number of feature points in the region 3140 in the twelfth frame. In this case, for example, if there are eight or more feature points in the area 3140, the monitoring system determines that snow jam has occurred.

The difference in the timing of occurrence of snow jam will be described with reference to FIG. 32. FIG. 32 is a diagram showing the timing of detecting the occurrence of snow jam when water intake is stopped due to snow jam on December 28, 2018. At the time of observation on this day, the angle of view of the camera of the surveillance system was changed three times. At that time, the number of feature points is temporarily changing suddenly.

Graph A shows the detection result of the trained model when the snow area ratio is used as a criterion. The threshold is that the snow area ratio is 9.48% or more. According to the PoC1 model, the surveillance system has detected the occurrence of snow jam at around 2:33:08 on December 28, 2018. According to the PoC2 model, the monitoring system detects the occurrence of snow jam at around 2:36:15 on the same day. Therefore, in this case, the PoC1 model was able to detect the snow jam earlier than the PoC2 model.

Graph B shows the detection result by the optical flow. The threshold has eight feature points. Excluding the time when the angle of view was changed, the monitoring system was able to detect the occurrence of snow jam at around 2:24 according to the optical flow.

FIG. 33 is a diagram showing the timing of detecting the occurrence of snow jam when water intake is stopped due to snow jam on January 24, 2019.

Graph A shows the detection result of the trained model when the snow area ratio is used as a criterion. The detection condition (threshold value) is that the snow area ratio is 9.48% or more. According to the PoC1 model, the monitoring system has detected the occurrence of snow jam at around 0:54:47 on January 24, 2019. According to the PoC2 model, the monitoring system detects the occurrence of snow jam at around 0:56:56 on the same day. Therefore, in this case, the PoC1 model was able to detect the snow jam earlier than the PoC2 model.

Graph B shows the detection result by the optical flow. The threshold has eight feature points. According to Optical Flow, the monitoring system was able to detect the occurrence of snow jam at around 0:54.

FIG. 34 shows a state in which false detection by the light of the spotlight occurs on January 15, 2019 even though snow jam has not occurred.

Graph A shows the detection result of the trained model when the snow area ratio is used as a criterion. The detection condition is snow area ratio ≧ threshold value (9.48%). According to the PoC1 model, the surveillance system falsely detects that a snow jam has occurred around 19:26 on January 15, 2019. According to the PoC2 model, the surveillance system has not determined that a snow jam has occurred. Therefore, in this case, only the PoC1 model is erroneously detected.

Graph B shows the detection result by the optical flow. The detection condition is the number of feature points ≧ threshold value (8). According to the optical flow, the monitoring system does not determine that a snow jam has occurred, as in reality.

FIG. 35 shows a state in which a false detection occurs on January 27, 2019 due to snow or clouds on the wall surface reflected on the water surface even though no snow jam has occurred.

Graph A shows the detection result of the trained model when the snow area ratio is used as a criterion. The detection condition is snow area ratio ≧ threshold value (9.48%). According to the PoC1 model, the surveillance system erroneously detects that a snow jam has occurred at around 11:36 on January 27, 2019. According to the PoC2 model, the surveillance system has not determined that a snow jam has occurred. Therefore, in this case, only the PoC1 model is erroneously detected.

Graph B shows the detection result by the optical flow. The detection condition is the number of feature points ≧ threshold value (8). According to the optical flow, the monitoring system does not determine that a snow jam has occurred, as in reality.

FIG. 36 shows the detection result by the monitoring system in the daytime on February 13, 2019. The condition for detecting the snowboard is the snow area ratio ≧ threshold value (9.48%). With neither the PoC1 model nor the PoC2 model, the monitoring system has determined that snow jam has occurred.

FIG. 37 is a diagram showing an image that the PoC1 model does not erroneously detect and an original image thereof under a situation where erroneous detection is likely to occur when the optical flow model is adopted.

The image A represents an image acquired by taking a picture at the time when the cumulative value shown in the graph B of FIG. 29 reaches 129, that is, taking a picture at about 20:41 on February 7, 2019. Image B is an image obtained when the image is applied to the PoC1 model. According to the image illustrated in FIG. 37, the snow area ratio detected by the PoC1 model is 0.041% under the situation where the optical flow model is easily erroneously detected, and the monitoring system adopting the PoC1 model is almost erroneous. It can be seen that it has not been detected.

As described above, according to the present disclosure, snow jam and other suspended matter can be detected at an early stage by combining the determination using the trained model and the determination using the optical flow. In addition, since the criteria for each determination are combined, it is possible to suppress false detections that may occur in the case of detection using a trained model. This makes it possible to realize a monitoring system that can detect suspended matter at an early stage while suppressing false detection.

In one aspect, the surveillance system 1 is used by a business operator who owns a surveillance camera 300, a server device 100, and a terminal device 200. In another aspect, the image taken by the surveillance camera 300 is sent to the computer of the operator operating the surveillance service, and the computer executes the snow jam detection process as the server device 100, whereby the surveillance system 1 May be configured. In this case, the business operator can receive an image from each electric power company and provide a monitoring service using the image.

It should be considered that the embodiments disclosed this time are exemplary in all respects and not restrictive. The scope of the present invention is shown by the scope of claims rather than the above description, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

The disclosed technical features are applicable as techniques for monitoring places where the presence of obstacles, including suspended matter such as snow, leaves or jellyfish, need to be detected, eg, the intake of a power plant.

1 Monitoring system, 100 server device, 101 communication IF unit, 102 control unit, 111 receiver unit, 112 transmitter unit, 141 reference coordinate setting unit, 142 feature point setting unit, 143 detection unit, 144 judgment unit, 145 water supply control unit, 146 Notification unit, 151 processor, 152 ROM, 153 RAM, 156 operation key, 157 power supply circuit, 158 display, 200 terminal device, 300 surveillance camera, 400 water intake device, 500 water channel, 700 water intake, 800, Q1, Q2 area, 1910, 1920 frames.

Claims (5)

A step that receives input of multiple consecutive frame images obtained by photographing a waterway, and
A step of detecting the movement of a set number of feature points in the first frame image of the plurality of frame images for each feature point over the plurality of frame images.
Using a trained model prepared in advance, a step of determining whether or not each pixel constituting one frame image of the plurality of frame images corresponds to a floating object in the waterway is used.
Based on the result of the determination, a step of calculating the ratio of the number of pixels determined to have a floating substance in the one frame image, and
A step of determining whether or not the suspended matter exists in the water channel in a predetermined state based on the magnitude relationship between the number of feature points and a predetermined threshold value and the ratio of the number of pixels. Including, monitoring method. The step of determining whether or not the suspended matter is present in the water channel in a predetermined state is
Based on the fact that the number of the feature points is equal to or greater than the predetermined threshold value and the ratio of the number of pixels is equal to or greater than the predetermined ratio, the suspended matter is present in the water channel in a predetermined state. The monitoring method according to claim 1, wherein the monitoring method includes determining that the image is being used. It is obtained by performing a determination process using a predetermined number of frame images among the plurality of frame images by shifting the first frame image used for the determination process to the next frame image a predetermined number of times. The step of accumulating the results and calculating the total value of the number of feature points,
Further including a step of accumulating the total value of the number of the feature points obtained by repeating the process of calculating the total value a predetermined number of times.
The monitoring method according to claim 2, wherein the number of the feature points is equal to or more than the predetermined threshold value, the total value of the number of the feature points is equal to or more than the predetermined threshold value. A computer program for realizing the monitoring method according to any one of claims 1 to 3. The memory in which the computer program according to claim 4 is stored and
A monitoring system comprising one or more processors that execute the computer program.

Images