JP6978986B2 - Warning system, warning control device and warning method - Google Patents

Description

The present invention promptly, accurately and efficiently notifies the driver of an alarm due to the intrusion of an object such as a wild animal or a suspicious person into the test course, and suppresses a decrease in test efficiency due to an erroneous notification to the driver. Related to alarm systems, alarm control devices and alarm methods that can be used.

Conventionally, in a vehicle course, various tests such as a high-speed running test of a vehicle need to be performed in a strict secret environment, so that the vehicle is often provided in a place away from the urban area. The test course is equipped with an outer fence to prevent the invasion of wild animals (hereinafter referred to simply as "animals"), but there may be some tears in the outer fence or there may be a lot of snow. In some cases, animals may invade the test course beyond this outer fence. For this reason, in the test course, a plurality of surveillance cameras for monitoring the objects invading the test course are provided, and the observer detects the abnormality while visually recognizing the image captured by the surveillance camera. ..

However, if the image of the surveillance camera is visually confirmed by the observer, the human labor becomes excessive and there is a possibility that the surveillance omission may occur. For example, if an observer overlooks the fact that an animal has climbed over an outer fence and entered the driving path on which the test vehicle travels, the animal may collide with the test vehicle traveling on the traveling path. Therefore, it is conceivable to automatically recognize the animal by using a conventional image processing technique such as template matching, and to cancel the test when the animal invades the test course. It is also conceivable to learn animals using conventional techniques related to deep learning (see Patent Documents 1 and 2) and determine whether or not the input image contains animals.

Japanese Unexamined Patent Publication No. 2015-210824 Japanese Unexamined Patent Publication No. 2017-187954

However, in order to recognize the existence of an animal by an image processing technique such as the above-mentioned template matching, the appearance of the animal must be identifiable and included in the input image. Similarly, when determining the presence or absence of an animal using deep learning, the appearance of the animal must be included in the input image so that it can be discriminated.

However, in many cases, the actual image taken by the surveillance camera installed on the test course does not include the appearance of the animal in a discriminable manner. This is because the test course has a vast area, the number of surveillance cameras that can be installed is limited to a certain number, and animals located far from the surveillance cameras have to be imaged. As a result, the number of pixels related to the animal included in the input image is reduced, and even if template matching or deep learning is used, the animal cannot be detected.

For this reason, if the input image does not sufficiently include the appearance of the animal, priority is given to avoiding collision between the animal and the test vehicle, and the test should be stopped even if it is unclear whether the animal is an animal or not. However, if the test vehicle or observer is mistakenly recognized as an animal and the test is frequently stopped, for example, the test efficiency will be significantly reduced. It is possible to control the focal length of the surveillance camera, but if the focus of the surveillance camera is zoomed up to focus on a specific object, the field of view of the surveillance camera will be limited by that amount, and other cameras will be limited. It is not desirable to rely on zooming in on the focus, as it may lead to missed detection of animals located in the area.

For these reasons, it is an important issue how to promptly and efficiently notify the driver of an alarm due to the intrusion of an object into the test course while preventing a decrease in test efficiency. It should be noted that such a problem also occurs when a suspicious person tries to invade the test course. This is because it is possible that a suspicious person who is trying to obtain information on a test vehicle of a new model may get over the outer fence and invade the test course.

The present invention has been made to solve the above-mentioned problems (problems) of the prior art, and while preventing a decrease in test efficiency, the driver is quickly and efficiently alerted when an object enters the test course. It is an object of the present invention to provide an alarm system, an alarm control device, and an alarm method that can be notified in a targeted manner.

In order to solve the above problems, the present invention comprises a plurality of image pickup devices, an alarm light for notifying a user of an alarm, and an alarm control device communicably connected to the plurality of image pickup devices and the warning light. The warning system has a detection unit that detects an object included in an image captured by each imaging device, and a detection unit that tracks the object detected by the detection unit and has a movement history of the object. The movement history information acquisition unit for acquiring information, the type determination unit for determining the type of the object detected by the detection unit, and the movement history information acquisition unit when the type of the object cannot be determined by the type determination unit. It is characterized by including an alarm determination unit for determining whether or not to perform an alarm using the alarm light based on the movement history information acquired by.

Further, the present invention is characterized in that, in the above invention, the detection unit detects an object included in an image captured by each image pickup device based on a movement vector of pixels between a plurality of images.

Further, in the above invention, in the above invention, the movement history information acquisition unit specifies a rectangular region including the object based on the movement vector of the pixels forming the object detected by the detection unit and a predetermined likelihood. However, it is characterized in that the information related to the movement history of the specified rectangular area is acquired as the movement history information.

Further, in the above invention, in the above invention, the type determination unit inputs an input image including a partial image of a rectangular region in the image to the multi-layer neural network, and the probability for each type output from the multi-layer neural network. Based on the above, the type of the object detected by the detection unit is determined.

Further, the present invention is characterized in that, in the above invention, the multi-layer neural network performs supervised learning with a large number of input images including at least animals, suspicious persons, observers, and vehicles.

Further, the present invention is characterized in that, in the above invention, the multi-layer neural network outputs the probability that the object included in the input image is an animal, a suspicious person, a watchman, or a vehicle.

Further, in the above invention, in the above invention, when the alarm determination unit determines at least that the object detected by the detection unit by the type determination unit is a vehicle or a watchman, the alarm light is turned on. It is characterized in that it is determined not to give the used alarm.

Further, in the above invention, in the above invention, the alarm determination unit uses the alarm light when the object detected by the detection unit is determined to be an animal or a suspicious person by the type determination unit. It is characterized in that it is determined to give an alarm.

Further, in the above invention, in the above invention, when the type determination unit does not specify the type of the object detected by the detection unit, the movement history information of the object excludes a predetermined alarm. If the condition is met, it is determined not to issue an alarm using the alarm light, and if the movement history information of the object does not meet the predetermined alarm exclusion condition, it is determined to issue an alarm using the alarm light. It is characterized by doing.

Further, the present invention has a detection unit that detects an object included in an image captured by a predetermined imaging device, and a movement history that tracks an object detected by the detection unit and acquires movement history information of the object. The movement acquired by the movement history information acquisition unit when the information acquisition unit , the type determination unit that determines the type of the object detected by the detection unit, and the type determination unit cannot determine the type of the object. It is characterized by being provided with an alarm determination unit that determines whether or not to issue an alarm using an alarm light based on historical information.

Further, the present invention provides an alarm in an alarm system having a plurality of image pickup devices, an alarm light for notifying a user of an alarm, and an alarm control device communicably connected to the plurality of image pickup devices and the alarm light. In the method, the alarm control device tracks a detection step of detecting an object included in an image captured by each image pickup device and an object detected by the detection step, and acquires movement history information of the object. The movement history information acquisition step, the type determination step of determining the type of the object detected by the detection step, and the movement history information acquisition step when the type of the object cannot be determined by the type determination step. on the basis of the movement history information, characterized in that it includes a determining alarm determination step whether an alarm using the alarm lamp has.

According to the present invention, it is possible to promptly and efficiently notify the driver of an alarm due to the intrusion of an object into the test course while preventing a decrease in test efficiency. In particular, even when the number of pixels related to the object included in the input image is small, it is possible to give an alarm due to the intrusion of the object into the test course without causing unnecessary test cancellation.

FIG. 1 is an explanatory diagram for explaining an outline of the system configuration and operation of the alarm system according to the present embodiment. FIG. 2 is an explanatory diagram for explaining the concept of the alarm system according to the present embodiment. FIG. 3 is a functional block diagram showing the configuration of the alarm control device shown in FIG. FIG. 4 is a diagram showing an example of type determination using CNN by the type determination unit shown in FIG. FIG. 5 is a flowchart showing a processing procedure of the alarm control device shown in FIG. FIG. 6 is a flowchart showing a processing procedure related to object detection and acquisition of movement history information shown in step S102 of FIG. FIG. 7 is a flowchart showing the alarm determination processing procedure shown in step S104 of FIG. FIG. 8 is a flowchart showing a learning data generation procedure used when performing supervised learning of CNN. FIG. 9 is a diagram showing an example of the hardware configuration of the alarm control device shown in FIG. 1.

Hereinafter, the alarm system, the alarm control device, and the alarm method according to the present embodiment will be described with reference to the attached drawings. In the present embodiment, the case where the present invention is applied to a test course for performing a running test of a vehicle will be shown. Hereinafter, a case where the object A1 is reflected in the input image A will be described.

<Overview of system configuration and operation>
First, an outline of the system configuration and operation of the alarm system according to the present embodiment will be described. FIG. 1 is an explanatory diagram for explaining an outline of the system configuration and operation of the alarm system according to the present embodiment. As shown in FIG. 1, this warning system includes a surveillance camera 10, a warning light 20, a speaker 30, and a warning control device 40.

The surveillance camera 10 is an imaging device that captures an image of an object A1 that invades the test course T from the outside of the test course T, and is arranged around the outer fence of the test course T at regular intervals (for example, every several hundred meters). Has been done. The surveillance camera 10 transmits the input image A to the alarm control device 40 via an IP network such as a LAN (Local Area Network). Although the case where the input image A is transmitted from the surveillance camera 10 to the alarm control device 40 is shown here, it is also possible to transmit a moving image from the surveillance camera 10 to the alarm control device 40. When transmitting a moving image, the alarm control device 40 performs a process of cutting out the input image A from the moving image for each frame.

The warning light 20 is a display device for visually notifying the driver driving the test vehicle, the observer who monitors the test course T, and the like that an abnormality has occurred in the test course T. The warning lights 20 are arranged at regular intervals (for example, every several hundred meters) so that the driver of the test vehicle traveling on the test course T can visually recognize them, and the display is controlled by the warning control device 40. Here, the warning light 20 is displayed in green during the running test of the test vehicle, and the warning light 20 is blinked in red when an event that should stop the running test occurs. Therefore, for example, if the driver of the test vehicle visually recognizes the red blinking of the warning light 20 while traveling on the traveling road, it is immediately determined that the test is canceled and the test vehicle moves to a predetermined position.

The speaker 30 is a notification device for acoustically notifying the driver driving the test vehicle, the observer who monitors the test course T, and the like about the test status on the test course T. When an event that should stop the running test occurs, a predetermined alarm sound is output according to the instruction of the alarm control device 40. Therefore, for example, if the driver of the test vehicle confirms the warning sound while traveling on the traveling road, it is immediately determined that the test is canceled and the vehicle moves to a predetermined position.

When the alarm control device 40 determines that an object A1 such as an animal or a suspicious person has invaded the test course T, the alarm control device 40 controls the display of the alarm light 20 and the output control of the alarm sound from the speaker 30 to the driver of the test vehicle or the like. It is a device that gives an alarm. Further, when the alarm control device 40 cannot specify the type of the object A1 that has invaded the test course T, and it is highly probable that the object A1 is not a legitimate object such as a vehicle or an observer. In addition, an alarm is given to the driver and the like of the test vehicle by controlling the display of the alarm light 20 and controlling the output of the alarm sound from the speaker 30. A detailed description of the alarm control device 40 will be described later.

Here, if the object A1 is imaged by the surveillance camera 10 (S1), the surveillance camera 10 sequentially transmits the captured input image A to the alarm control device 40 (S2). The alarm control device 40 detects the object A1 using a plurality of input images A (S3), and after acquiring the movement history information of the object A1 (S4), the type of the object A1 (animal, observer, suspicious). Person or vehicle) is determined (S5). In this type determination, deep learning using CNN (Convolutional Neural Network) is used. If the type of the object A1 can be identified, the type is output, and if the type cannot be identified, "UNKNOWN (unknown)" indicating that the type of the object A1 cannot be identified is output. After that, it is determined whether or not to issue an alarm based on this type determination result (S6). Specifically, when the type of the object A1 is determined to be "animal" or "suspicious person", it is determined to give an alarm, and the type of the object A1 is determined to be "vehicle" or "watcher". In that case, it is determined that the alarm is not given.

When the type of the object A1 is determined to be "UNKNOWN", it is determined whether or not the movement history of the object A1 corresponds to a predetermined alarm exclusion condition. This warning exclusion condition refers to a condition in which the object A1 is unlikely to collide with the test vehicle and should not be warned. As an example of this alarm exclusion condition, for example, "when the object A1 is stopped at a predetermined monitoring position" or "when the object A1 is orbiting the traveling path at a constant speed". Can be mentioned. This is because the type of the object A1 determined by deep learning is unknown, but the possibility that the object A1 collides with the test vehicle is low. If the movement history of the object A1 meets the predetermined alarm exclusion condition, an alarm is not issued, and if the movement history of the object A1 does not meet the predetermined alarm exclusion condition, an alarm is issued. This is because if the warning is given even when the object A1 is unlikely to collide with the test vehicle, the test efficiency is lowered. If it is determined that an alarm should be issued, an alarm instruction is given to the alarm light 20 and the speaker 30 (S7). The alarm light 20 is turned on in red, and an alarm sound is output from the speaker 30.

<Concept of alarm system>
Next, the concept of the alarm system according to the present embodiment will be described. FIG. 2 is an explanatory diagram for explaining the concept of the alarm system according to the present embodiment. Here, the situation in which the object A1 is reflected in the input image A captured by the surveillance camera 10 is shown.

As described in the background art, in order to accurately detect the object A1 such as an animal or a suspicious person, it is desired that the appearance of the object A1 is included in the input image A to the extent that it can be distinguished. However, since the area of the test course is vast and it is not realistic to install the surveillance cameras 10 in every place around the outer fence of the test course, the surveillance cameras 10 are located at a certain distance from each other (for example, several hundred meters). ) There is no choice but to install it in a remote position. As a result, the number of pixels related to the object A1 included in the input image A is reduced, and a situation occurs in which an animal cannot be detected even if template matching or deep learning is used. Therefore, a situation may occur in which the object A1 collides with a test vehicle traveling on a traveling path. On the other hand, if the object A1 is a test vehicle or a watchman but the test is stopped, the test efficiency is lowered.

Therefore, in the warning system according to the present embodiment, even when the number of pixels related to the object A1 in the input image A is relatively small, the type of the object A1 (animal, observer, suspicious person, or vehicle) can be used as much as possible. ) Make an effort to judge. This is because even if the number of pixels related to the object A1 is small, the feature amount of the object A1 may be actualized. Specifically, the type of the object A1 is determined as much as possible by using deep learning using CNN.

Then, when the type of the object A1 can be specified from the output result of the CNN, it is confirmed whether the type of the object A1 is a vehicle, an observer, an animal, or a suspicious person. As a result, if the type of the object A1 is a vehicle or a watchman, the test is continued and no alarm is given, and if the type of the object A1 is an animal or a suspicious person, an alarm is given to stop the test. ..

On the other hand, it may not be possible to specify the type of the object A1 from the output result of CNN. This is because the number of pixels related to the object A1 is too small, and there is a limit to the type determination by CNN. In such a case, if the movement history of the object A1 meets the alarm exclusion condition, the test is continued and the alarm is not issued. This is because if the warning is given even when the object A1 is unlikely to collide with the test vehicle, the test efficiency is lowered. On the other hand, if the movement history of the object A1 does not meet the alarm exclusion condition, an alarm is given to stop the test.

As a result, even when the object A1 cannot be photographed at an appropriate distance by the surveillance camera 10 and the number of pixels related to the object A1 included in the input image A is small, the test course T is prevented from deteriorating the test efficiency. It is possible to promptly and efficiently notify the driver of an alarm accompanying the intrusion of the object A1 into the driver.

<Configuration of alarm control device 40>
Next, the configuration of the alarm control device 40 shown in FIG. 1 will be described. FIG. 3 is a functional block diagram showing the configuration of the alarm control device 40 shown in FIG. As shown in the figure, the alarm control device 40 includes an input unit 41, a display unit 42, a communication interface unit (hereinafter, simply referred to as “communication I / F unit”) 43, a storage unit 44, and a control unit 45. ..

The input unit 41 is an input device such as a keyboard or a mouse, and the display unit 42 is a display device such as a liquid crystal panel or a display device. The communication I / F unit 43 is an interface unit for communicating with the surveillance camera 10, the alarm light 20, and the speaker 30 via a LAN or the like.

The storage unit 44 is a storage device such as a non-volatile memory or a hard disk device, and stores learning data 44a, movement history information 44b, and the like. The learning data 44a is image data for learning used by CNN, which will be described later, when performing supervised learning. The movement history information 44b is information indicating the movement history of the object A1.

The control unit 45 is a control unit that performs overall control of the alarm control device 40, and includes a detection unit 45a, a movement history information acquisition unit 45b, a type determination unit 45c, an alarm determination unit 45d, and an alarm instruction unit 45e. Have. Although a detailed description thereof will be described later, the CPU (Central Processing Unit) of the alarm control device 40 informs the detection unit 45a, the movement history information acquisition unit 45b, the type determination unit 45c, the alarm determination unit 45d, and the alarm instruction unit 45e. By loading and executing an alarm control program including a corresponding routine from a non-volatile memory or the like onto a RAM (Random Access Memory) which is a main storage device, a process corresponding to each functional unit is formed.

The detection unit 45a is a processing unit that detects an object A1 included in the input image A captured by the surveillance camera 10 based on the movement vector of the pixels between the images of a plurality of consecutive frames (input image A). Specifically, if there are moving pixels in the plurality of input images A, the object A1 is detected based on the moving pixels. The pixel movement vector is a vector (for example, optical flow) between pixels indicating the same location of the object A1.

The movement history information acquisition unit 45b identifies a rectangular area including the object A1 based on the movement vector of the pixels forming the object A1 detected by the detection unit 45a and a predetermined likelihood, and the movement history of the specified rectangular area. This is a processing unit that acquires information related to the above as movement history information. Specifically, the optical flow and likelihood are calculated for each pixel forming the input image A, and the past object extraction results are moved based on the optical flow to extract moving pixels, and a rectangular area as particles is created. Place it on the input image A, move the rectangular area based on the optical flow of the object extraction result position in the rectangular area, and use the likelihood calculated for each rectangular area based on the object extraction results inside and outside the rectangular area. An object A1 was detected using a sampled particle filter, and a partial filter was resampled using the likelihood calculated for each rectangular area based on the color histogram of the object extraction result positions inside and outside the rectangular area. Track the object A1. The specific processing is the same as that of Japanese Patent Application No. 2018-046582. If the rectangular region including the object A1 is specified in this way, for example, the movement history of the center point of the rectangular region (transition of the coordinate position of the center point) is acquired as the movement history information.

The type determination unit 45c is a processing unit that determines the type of the object A1 detected by the detection unit, and performs type determination using a multi-layer convolutional neural network (CNN). As such CNN, "caffe", "TensorFlow", "Chainer", "CNTK" and the like provided as open source products can be used.

Here, the type determination using the above CNN will be further described. FIG. 4 is a diagram showing an example of type determination using CNN by the type determination unit 45c shown in FIG. Here, it is assumed that the supervised learning by CNN50 has already been completed. If the input image A including the object A1 is input to the CNN 50 shown in FIG. 4, the CNN 50 has a probability P1 that the object A1 is an animal, a probability P2 that the object A1 is a suspicious person, and the object A1 is a watcher. The probability P3 and the probability P4 that the object A1 is a vehicle are output. It should be noted that P1, P2, P3, and P4 are values between 0 and 1, the threshold value is "0.8", and it is assumed that a plurality of probabilities do not exceed the threshold value at the same time.

If the probabilities P1, P2, P3 or P4 are "0.8" or more, the corresponding type is specified. If the probability P1 output from CNN50 is "0.8" or more, the object A1 is identified as an animal, and if the probability P2 output from CNN50 is "0.8" or more, the object A1 is identified. Is identified as a suspicious person, and if the probability P3 output from CNN50 is "0.8" or more, the object A1 is identified as a watcher and the probability P4 output from CNN50 is "0. If it is 8 ”or higher, the object A1 is identified as a vehicle.

On the other hand, if all of the probabilities P1, P2, P3, and P4 are less than "0.8", the type is considered to be indefinite (UNKNOWN). As described above, when the number of pixels of the object A1 included in the input image A is very small and the amount of information (feature amount) sufficient to identify the object A1 cannot be obtained, the object A1 is used even if the CNN 50 is used. It is not possible to specify the type of, and the type is undefined.

When the type determination unit 45c determines that the object A1 is an animal or a suspicious person, the alarm determination unit 45d determines that an alarm is given using the alarm light 20 and the speaker 30. Further, when the type determination unit 45c determines that the object A1 is a vehicle or a watchman, it is determined that the alarm light 20 and the speaker 30 are not used to give an alarm. Further, when the type of the object A1 is determined to be "UNKNOWN", it is determined that the alarm is not issued if the movement history of the object A1 corresponds to the alarm exclusion condition, and the movement history of the object A1 becomes the alarm exclusion condition. If not applicable, it is determined to issue an alarm.

The alarm instruction unit 45e is a processing unit that issues an alarm instruction to the alarm light 20 and the speaker 30 when it is determined by the alarm determination unit 45d to issue an alarm. The warning light 20 stores the display mode of the warning light in advance in association with the data pattern of the warning instruction, and displays the display mode in the display mode corresponding to the data pattern included in the warning instruction of the warning instruction unit 45e. Similarly, the speaker 30 stores an output sound pattern in advance in association with the warning instruction data pattern, and outputs the output sound corresponding to the data pattern included in the warning instruction of the alarm instruction unit 45e. The speaker 30 can also output the output sound included in the warning instruction output from the alarm instruction unit 45e.

<Processing procedure of alarm control device 40>
Next, the processing procedure of the alarm control device 40 shown in FIG. 3 will be described. FIG. 5 is a flowchart showing a processing procedure of the alarm control device 40 shown in FIG. As shown in the figure, the alarm control device 40 receives a plurality of input images A from the surveillance camera 10 (step S101). The plurality of input images A are images of consecutive frames in time series. The moving image may be input from the surveillance camera 10 and the input image A corresponding to each frame may be cut out from the moving image.

After that, the alarm control device 40 performs object detection and acquisition of movement history information using a plurality of input images A (step S102), and determines the type of the object A1 in the input image A (step S103). In such type determination, as described above, deep learning using CNN is used. If the type can be specified by such type determination, this type determination result is output. From the CNN, the probability P1 that the type of the object A1 is "animal", the probability P2 that the type of the object A1 is "suspicious person", the probability P3 that the type of the object A1 is "watcher", and the type of the object A1 are. Since the probability P4 of being a "vehicle" is output, when any one of the probabilities is "0.8" or more, which is the threshold value, the type having the probability is specified as the type of the object A1.

After that, the alarm control device 40 makes an alarm determination as to whether or not an alarm should be issued (step S104). Specifically, if it is specified that the type of the object A1 is "vehicle" or "watcher", it is determined that the alarm is not given, and the type of the object A1 is "animal" or "suspicious person". If it is specified, it is determined to give an alarm. Further, if it is determined that the type of the object A1 is "UNKNOWN", it is determined that the alarm is not issued if the movement history of the object A1 corresponds to the alarm exclusion condition. For example, when the object A1 is moving on the traveling path at a constant speed as if it were a vehicle, or when the object A1 is stopped at a predetermined monitoring position as if it were a watchman. Is determined not to give an alarm. If the alarm exclusion condition is not met, it is determined that an alarm will be issued.

Then, when it is determined that an alarm should be given (step S105; Yes), the alarm instruction unit 45e issues an alarm instruction to the alarm light 20 and the speaker 30 (step S106). The alarm light 20 that has received this alarm instruction blinks the alarm light in red, and a predetermined output sound is output from the speaker 30. This allows the driver driving the test vehicle to recognize the cancellation of the test. If it is determined that an alarm should not be given (step S105; No), no alarm instruction is given.

Next, the procedure for detecting the object and acquiring the movement history information shown in step S102 of FIG. 5 will be described. FIG. 6 is a flowchart showing a procedure related to object detection and acquisition of movement history information shown in step S102 of FIG.

As shown in FIG. 6, the alarm control device 40 calculates the optical flow and the likelihood for each pixel of the input image A (step S201), and extracts moving pixels based on the optical flow (step S202). After that, a rectangle as particles is arranged on the input image A (step S203), and the rectangle is moved based on the optical flow (step S204). After that, the geographic history of the movement vector at the center of the rectangle is stored in the storage unit 44 as the movement history information 44b (step S205). As a result, the partial image of the rectangular region can be input to the CNN as the input image A to efficiently perform the type determination, and the movement history information 44b can be acquired.

Next, the procedure for processing the alarm determination shown in step S104 of FIG. 5 will be described. FIG. 7 is a flowchart showing the alarm determination procedure shown in step S104 of FIG. As shown in the figure, when the type of the object A1 is specified by the type determination unit 45c (step S301; Yes), whether or not the type of the object A1 is "animal" or "suspicious person". It is determined (step S302).

As a result, when the type of the object A1 is "animal" or "suspicious person" (step S302; Yes), it is determined that the test should be stopped, that is, an alarm should be given (step S303). On the other hand, when the type of the object A1 is not "animal" or "suspicious person" (step S302; No), it is determined that the test is continued, that is, no alarm is given (step S304). This is because if the test is stopped even when the object A1 is a "vehicle" or a "watcher", the same test must be repeated again, and the test efficiency is lowered.

On the other hand, when the type of the object A1 is not specified by the type determination unit 45c (step S301; No), it is determined whether or not the movement history of the object A1 corresponds to the alarm exclusion condition (step S305). For example, when the object A1 is stopped at a predetermined monitoring position or when the object A1 is orbiting the traveling path at a constant speed, it is determined that the alarm is not issued. As a result, when the movement history of the object A1 corresponds to the alarm exclusion condition (step S306; Yes), it is determined that the test is continued, that is, no alarm is given (step S307). In contrast, the movement history of the object A1 is an alarm. If the exclusion condition is not met (step S306; No), it is determined that the test is stopped, that is, an alarm is given (step S308).

Next, the procedure for generating the learning data 44a used by the CNN when performing supervised learning will be described. FIG. 8 is an explanatory diagram for explaining a procedure for generating learning data 44a used by CNN when performing supervised learning. Here, the image of the object A1 included in the input image A is reduced so that the type of the object A1 can be determined as much as possible even when the number of pixels of the object A1 included in the input image A is small. The learning data 44a used for supervised learning is generated in the state.

As shown in FIG. 8, a composite image of the background image and the animal is generated (step S401), and this composite image is used as a part of the learning data 44a. Similarly, a composite image of the background image and the suspicious person is generated (step S402), and this composite image is used as a part of the learning data 44a. Further, a composite image of the background image and the observer is generated (step S403), and this composite image is used as a part of the learning data 44a. Further, a composite image of the background image and the vehicle is generated (step S404), and this composite image is used as a part of the learning data 44a.

After that, the image of each object A1 is reduced by a predetermined magnification (step S405), and if the reduction ratio is not equal to or less than a predetermined threshold value (step S406; No), the process proceeds to step S401 and the same processing is performed. repeat. Then, when the reduction ratio becomes equal to or less than a predetermined threshold value (step S406; Yes), the process ends. By performing such a series of processes, even when the object A1 included in the input image A is small, it is possible to determine the type.

As described above, in the present embodiment, if the object A1 is imaged by the surveillance camera 10, the input image A captured by the surveillance camera 10 is sequentially transmitted to the alarm control device 40, and the alarm control device 40 is used. Detects an object A1 using a plurality of input images A, acquires movement history information of the object A1, determines the type of the object A1 (animal, observer, suspicious person, or vehicle), and also determines the object. When the type of A1 is not specified, it is determined whether or not the movement history of the object A1 corresponds to the alarm exclusion condition, and if it is determined that an alarm should be issued, an alarm is given to the alarm light 20 and the speaker 30. Since the warning light 20 is turned on in red and the warning sound is output from the speaker 30 by giving an instruction, the driver gives an alarm due to the intrusion of the object A1 into the test course T while preventing the test efficiency from deteriorating. Can be notified quickly and efficiently.

<Relationship with hardware>
Next, the correspondence between the alarm control device 40 according to the present embodiment and the main hardware configuration of the computer will be described. FIG. 9 is a diagram showing an example of a hardware configuration.

A general computer has a configuration in which a CPU 61, a ROM (Read Only Memory) 62, a RAM 63, a non-volatile memory 64, and the like are connected by a bus. A hard disk device may be provided instead of the non-volatile memory 64. For convenience of explanation, only the basic hardware configuration is shown.

Here, the ROM 62 or the non-volatile memory 64 stores a program or the like necessary for starting the operating system (Operating System; hereinafter, simply referred to as “OS”), and the CPU 61 is the ROM 62 or the non-volatile memory when the power is turned on. The OS program is read and executed from the sex memory 64.

On the other hand, various application programs executed on the OS are stored in the non-volatile memory 64, and the CPU 61 executes the application program while using the RAM 63 as the main memory, so that the process corresponding to the application is executed. To.

The alarm processing program of the alarm control device 40 according to the present embodiment is also stored in the non-volatile memory 64 or the like like the other application programs, and the CPU 61 loads and executes the alarm control program. Become. The alarm control device 40 according to the embodiment includes routines corresponding to the detection unit 45a, the movement history information acquisition unit 45b, the type determination unit 45c, the alarm determination unit 45d, and the alarm instruction unit 45e shown in FIG. The alarm control program is stored in the non-volatile memory 64 or the like. When the alarm control program is loaded and executed by the CPU 61, a process corresponding to the detection unit 45a, the movement history information acquisition unit 45b, the type determination unit 45c, the alarm determination unit 45d, and the alarm instruction unit 45e is generated. The learning data 44a, setting data, and the like are stored in the non-volatile memory 64 in advance.

Further, in the above embodiment, the case where the operation is executed on the device as a stand-alone is shown, but the present invention is not limited to this, and may be applied to the case where it is realized in the edge server device of the cloud. can. It can also be applied to distributed processing by a plurality of computers.

Further, in the embodiment of the above embodiment, the case where an object such as an animal or an intruder is targeted for warning is shown, but the present invention is not limited to this, and entry into the test course is not permitted. It can also be applied when various objects such as suspicious objects and drones are targeted for warning.

It should be noted that each configuration shown in the above embodiment is a schematic function, and does not necessarily have to be physically shown. That is, the form of distribution / integration of each device is not limited to the one shown in the figure, and all or part of them may be functionally or physically distributed / integrated in any unit according to various loads and usage conditions. Can be configured.

The warning system, the warning control device, and the warning method of the present invention are useful for promptly and efficiently notifying the driver of an alarm due to the intrusion of an object into the test course while preventing a decrease in test efficiency.

A Input image A1 Object 10 Surveillance camera 20 Indicator light 30 Speaker 40 Alarm control device 41 Input unit 42 Display unit 43 Communication I / F unit 44 Storage unit 44a Learning data 44b Movement history information 45 Control unit 45a Detection unit 45b Movement history information acquisition Unit 45c Type determination unit 45d Alarm determination unit 45e Alarm processing unit 50 CNN
61 CPU
62 ROM
63 RAM
64 Non-volatile memory 65 Bus

Claims (11)

An alarm system including a plurality of image pickup devices, an alarm light for notifying a user of an alarm, and an alarm control device communicably connected to the plurality of image pickup devices and the alarm light.
The alarm control device is
A detector that detects objects included in the image captured by each image pickup device, and
A movement history information acquisition unit that tracks an object detected by the detection unit and acquires movement history information of the object, and a movement history information acquisition unit.
A type determination unit that determines the type of an object detected by the detection unit,
When the type of the object cannot be determined by the type determination unit, an alarm determination is made to determine whether or not to issue an alarm using the alarm light based on the movement history information acquired by the movement history information acquisition unit. An alarm system characterized by having a unit. The detector is
The warning system according to claim 1, wherein an object included in an image captured by each image pickup device is detected based on a movement vector of pixels between a plurality of images. The movement history information acquisition unit is
A rectangular area including an object is specified based on a movement vector of pixels forming an object detected by the detection unit and a predetermined likelihood, and information related to the movement history of the specified rectangular area is acquired as movement history information. The alarm system according to claim 1 or 2, wherein the alarm system is characterized in that. The type determination unit
An input image including a partial image of a rectangular region in the image is input to the multi-layer neural network, and the type of the object detected by the detection unit is determined based on the probability of each type output from the multi-layer neural network. The warning system according to claim 3, wherein the alarm system is characterized in that. The multi-layer neural network is
The warning system according to claim 4, wherein supervised learning is performed using a large number of input images including at least animals, suspicious persons, guards, and vehicles. The multi-layer neural network is
The warning system according to claim 4, wherein the output is the probability that the object included in the input image is an animal, a suspicious person, a guard, or a vehicle. The alarm determination unit
4. The fourth aspect of the present invention is that when the type determination unit determines that the object detected by the detection unit is a vehicle or a watchman, it is determined not to issue an alarm using the warning light. The alarm system described in. The alarm determination unit
The fourth aspect of claim 4, wherein when the type determination unit determines that the object detected by the detection unit is an animal or a suspicious person, it is determined to issue an alarm using the warning light. Alarm system. The alarm determination unit
When the type of the object detected by the detection unit is not specified by the type determination unit, if the movement history information of the object corresponds to a predetermined alarm exclusion condition, an alarm using the alarm light is not performed. The warning system according to claim 4, wherein if the movement history information of the object does not correspond to the predetermined warning exclusion condition, it is determined that the warning using the warning light is performed. A detector that detects an object contained in an image captured by a predetermined image pickup device, and a detector.
A movement history information acquisition unit that tracks an object detected by the detection unit and acquires movement history information of the object, and a movement history information acquisition unit.
A type determination unit that determines the type of an object detected by the detection unit,
An alarm determination unit that determines whether or not to issue an alarm using an alarm light based on the movement history information acquired by the movement history information acquisition unit when the type of the object cannot be determined by the type determination unit. An alarm control device characterized by being equipped with. It is an alarm method in an alarm system having a plurality of image pickup devices, an alarm light for notifying a user of an alarm, and an alarm control device communicably connected to the plurality of image pickup devices and the alarm light.
A detection step in which the alarm control device detects an object included in an image captured by each image pickup device.
A movement history information acquisition step of tracking an object detected by the detection process and acquiring movement history information of the object, and a movement history information acquisition process.
A type determination process for determining the type of an object detected by the detection process, and a type determination process.
When the type of the object cannot be determined by the type determination step, an alarm determination is made to determine whether or not to issue an alarm using the alarm light based on the movement history information acquired by the movement history information acquisition step. An alarm method characterized by including steps.

Images