JP6500428B2 - Moving body detection device, moving body detection method, and moving body detection program - Google Patents

Description

  The present invention relates to a moving object detection apparatus and the like, and relates to, for example, an apparatus for detecting a moving object such as a bird by using an image captured by a visible light camera and an infrared light camera.

  In recent years, it has become a social problem that a bird collides with an aircraft at an airport or a wind turbine with a wind power generation facility (bird collision, bird strike).

  As a solution for preventing a bird strike, for example, a technique of detecting a bird using a radar is generally and widely known (for example, Patent Document 1). This makes it possible to detect birds over a wide range. Then, it is possible to take measures to drive off the bird by the sound of an empty gun in the vicinity where the bird exists.

  However, the technique of detecting a bird using a radar can only estimate the size of an object based on RCS (Radar cross-section). That is, with the technique of detecting a bird using a radar, it was not possible to identify whether an object of the same size as the bird is a bird. Furthermore, in the technique of detecting a bird using a radar, there is a problem that the bird detection device can not be easily installed because the bird detection device becomes large and radio wave use permission is required.

  As another solution to prevent bird strikes, for example, a technique for detecting birds in low airspaces in the vicinity of a demand facility using a monocular visible light camera (also referred to as an EO camera). It has been known. In this case, as compared to the technique of detecting a bird using a radar, the bird detection device is relatively small and thus relatively easy to install. Also, the detection result can be confirmed visually.

  Also, for example, Patent Documents 2 to 4 disclose related techniques of the present invention.

JP, 2009-203873, A International Publication No. 2014/157058 JP, 2013-247492, A JP 2001-357388 A

  However, in the technology that detects birds using a monocular visible light camera, false detection is made for a bird-like moving object or a random-shaped object (sunlight reflection on the sea surface, tree fluctuation, etc.) Had the problem of That is, in the technique of detecting a bird using a monocular visible light camera, there is only a function of shape recognition using an image in the visible light wavelength range. For this reason, in the visible light wavelength region, there is a limit in removing various apparent false detection factors of the image, and certain false detections have been left.

  The present invention has been made in view of such circumstances, and an object of the present invention is to suppress false detection caused by, for example, a mobile object having a shape similar to a bird or an object having a random shape, and accurately An object of the present invention is to provide a moving object detection device and the like capable of detecting a moving object to be detected.

  A moving object detection apparatus according to the present invention includes a visible light image data input processing unit that receives an input of the visible light captured image by a visible light imaging apparatus that captures light in a visible light wavelength band to generate a visible light captured image; An infrared light image data input processing unit that receives an input of the infrared light image pickup image by an infrared light image pickup device which picks up light in an external light wavelength band to generate an infrared light image pickup image, and a moving object to be detected A visible light moving body candidate image extracting unit for extracting a visible light moving body candidate image which is a candidate for an image from the visible light imaging image and detection in the visible light image based on a luminance gradient histogram of the visible light moving body candidate image A visible light moving body candidate determination unit that primarily determines whether or not a detection target moving body candidate that is a target moving body candidate has been detected, and the detection target moving body candidate determined primarily by the visible light moving body candidate determination unit Before the position of A visible light bird coordinate extraction unit that estimates visible light camera coordinates that are coordinates indicated on the basis of a visible light imaging device, and red that is a coordinate that indicates the position corresponding to the visible light camera coordinates on the basis of the infrared light imaging device An infrared light moving body coordinate estimation unit that generates an infrared light estimation image in which an external light camera coordinate is estimated, and the detection target moving body candidate is estimated and arranged on the infrared light camera coordinate; and the infrared light image Based on the similarity between the luminance distribution of the infrared light imaging image received by the data input processing unit and the luminance distribution of the infrared light estimation image output by the infrared light moving body coordinate estimation unit, And an infrared light moving body candidate estimation unit configured to determine whether the detection target moving body candidate is the detection target moving body.

  The moving object detection method according to the present invention includes a visible light image data input processing step of receiving an input of the visible light captured image by a visible light imaging device that captures light in a visible light wavelength band to generate a visible light captured image; An infrared light image data input processing step of receiving an input of the infrared light image pickup image by an infrared light image pickup device which picks up light of an external light wavelength band to generate an infrared light image pickup image; A visible light moving body candidate image extracting step of extracting a visible light moving body candidate image which is a candidate of an image from the visible light imaging image and detection in the visible light image based on a luminance gradient histogram of the visible light moving body candidate image A visible light moving body candidate determination step of temporarily determining whether or not a detection target moving body candidate which is a target moving body candidate has been detected; and the test temporarily determined in the visible light moving body candidate determination step A visible light bird coordinate extracting step of estimating a visible light camera coordinate which is a coordinate indicating a position of a target moving body candidate with respect to the visible light imaging device; a position corresponding to the visible light camera coordinate is the infrared light imaging device To estimate the infrared light camera coordinate which is the coordinate indicated on the basis of the infrared light camera coordinate, and to generate and output an infrared light estimation image in which the detection target moving body candidate is estimated and arranged on the infrared light camera coordinate Step of: brightness distribution of the infrared light imaging image received in the infrared light image data input processing step; and brightness distribution of the infrared light estimation image output in the infrared light moving body coordinate estimation step And an infrared light moving body candidate estimation step of judging whether the detection target moving body candidate is the moving object of the detection target based on the similarity between the two.

  The moving object detection program according to the present invention includes a visible light image data input processing step of receiving an input of the visible light captured image by a visible light imaging device that captures light in a visible light wavelength band to generate a visible light captured image; An infrared light image data input processing step of receiving an input of the infrared light image pickup image by an infrared light image pickup device which picks up light of an external light wavelength band to generate an infrared light image pickup image; A visible light moving body candidate image extracting step of extracting a visible light moving body candidate image which is a candidate of an image from the visible light imaging image and detection in the visible light image based on a luminance gradient histogram of the visible light moving body candidate image A visible light moving body candidate determination step of primarily determining whether or not a detection target moving body candidate which is a candidate of a target moving body is detected, and the visible light moving body candidate determination step A visible light bird coordinate extraction step of estimating a visible light camera coordinate which is a coordinate indicating the position of the detection target moving body candidate with respect to the visible light imaging device; a position corresponding to the visible light camera coordinate is the infrared light An infrared light moving body that estimates an infrared light camera coordinate which is a coordinate shown on the basis of an imaging device, and generates and outputs an infrared light estimated image in which the detection target moving body candidate is estimated and arranged on the infrared light camera coordinates. A luminance estimation distribution of the infrared light imaging image received by the coordinate estimation step, the infrared light image data input processing step, and a luminance distribution of the infrared light estimation image output by the infrared light moving body coordinate estimation step And a computer that executes an infrared light moving body candidate estimation step of determining whether or not the detection target moving body candidate is the moving object of the detection target based on the similarity between

  According to the moving object detection apparatus and the like according to the present invention, false detection due to, for example, a moving object having a shape similar to a bird or an object having a random shape by simultaneously using a visible light captured image and an infrared light captured image Can be accurately detected to detect a moving object to be detected.

It is a block diagram showing composition of a bird detection device in an embodiment of the invention. It is a figure which shows the operation | movement flow of the bird detection apparatus in embodiment of this invention. It is a figure for demonstrating the method to acquire an infrared-light bird candidate image. It is a figure which shows the operation | movement flow of the bird detection apparatus in embodiment of this invention. FIG.

Embodiment
The configuration of the bird detection apparatus 100 according to the embodiment of the present invention will be described.

  FIG. 1 is a block diagram showing the configuration of the bird detection apparatus 100. As shown in FIG. The bird detection device 100 corresponds to the moving object detection device of the present invention.

  The bird detection device 100 is connected to a visible light imaging device (EO (Electric Optical) imaging device) 200 and an infrared light imaging device (IR (Infrared) imaging device) 300. The EO imaging device 200 corresponds to the visible light imaging device of the present invention. The IR imaging device 300 corresponds to the infrared light imaging device of the present invention.

  The EO imaging device 200 captures light in a visible light wavelength band to generate a visible light captured image (EO captured image). The EO imaging device 200 is installed on a fixed base. The EO imaging device 200 shoots an EO captured image by monitoring a fixed azimuth at a fixed point, or by turning and monitoring the entire circumference or a fixed range. In addition, the EO imaging device 200 has a function of transmitting the captured EO captured image to the bird detection device 100 by wire or wirelessly.

  The IR imaging device 300 captures light in the infrared wavelength band to generate an infrared captured image (IR captured image). The IR imaging device 300 is installed on a fixed base. The IR imaging device 300 captures an IR imaging image by monitoring a fixed azimuth at a fixed point, or by turning and monitoring the entire circumference or a predetermined range. Also, the IR imaging device 300 has a function of transmitting the captured IR imaging image to the bird detection device 100 by wire or wirelessly.

  As shown in FIG. 1, the bird detection apparatus 100 includes an EO image data input processing unit 110, an IR image data input processing unit 130, a daytime bird detection processing unit 150, a nighttime bird detection processing unit 170, and an alarm notification. And a unit 190. The EO image data input processing unit 110 corresponds to the visible light image data input processing unit of the present invention. The IR image data input processing unit 130 corresponds to the infrared light image data input processing unit of the present invention.

  As shown in FIG. 1, the EO image data input processing unit 110 is connected to the EO imaging device 200 and the daytime bird detection processing unit 150. The EO image data input processing unit 110 receives an EO captured image from the EO imaging device 200. Specifically, the EO image data input processing unit 110 receives the EO captured image transmitted from the EO imaging device 200, and always receives three frames by the first-in first-out method using an internal temporary recording memory (not shown). Record the minute image data. In addition, the EO captured image temporarily stored in the EO image data input processing unit 110 corrects the distortion of the image due to the rolling shutter phenomenon and the operation error of the turning platform, and then the EO bird candidate of the daytime bird detection processing unit 150 The image is output to the image extraction unit 151.

  As shown in FIG. 1, the IR image data input processing unit 130 is connected to the IR imaging device 300 and the night bird detection processing unit 170. The IR image data input processing unit 130 receives an IR captured image from the IR imaging device 300. Specifically, the IR image data input processing unit 130 receives the IR imaging image transmitted from the IR imaging device 300, and the image data for three frames is always obtained by the first-in first-out method using the internal temporary recording memory. Record The image temporarily stored in the IR image data input processing unit 130 corrects the disturbance of the image due to the rolling shutter phenomenon and the operation error of the turning platform, and then in the daytime, the IR bird coordinate estimation of the daytime bird detection processing unit 150 It is output to the part 156.

  The configuration of the daytime bird detection processing unit 150 will be described. As shown in FIG. 1, the daytime bird detection processing unit 150 is connected to the EO image data input processing unit 110 and the alarm notification unit 190.

  The daytime bird detection processing unit 150 includes an EO bird candidate image extraction unit 151, an EO moving object (bird candidate) information storage unit 152, an EO bird candidate determination unit 153, an EO bird discrimination library 154, and an EO bird coordinate extraction unit 155. And an IR bird coordinate estimation unit 156, a camera coordinate conversion matrix / disparity upper limit information storage unit 157, an IR bird candidate estimation unit 158, and an IR bird luminance distribution library 159.

  As shown in FIG. 1, the EO bird candidate image extraction unit 151 is connected to an EO image data input processing unit 110, an EO moving object (bird candidate) information storage unit 152, and an EO bird candidate determination unit 153. The EO bird candidate image extraction unit 151 extracts a bird candidate image that is a candidate of a bird image as a visible light bird candidate image (EO bird candidate image) from the EO captured image. That is, after the EO bird candidate image extraction unit 151 receives the EO captured image from the EO image data input processing unit 110, the EO bird candidate image extraction unit 151 extracts the outline of the moving object from difference information of successive three frames of images. Extract EO bird candidate images. The visible light bird candidate image corresponds to the visible light moving body candidate image of the present invention. In addition, the EO bird candidate image extraction unit 151 is a histogram of the brightness gradient with respect to the shooting time of the moving object included in the visible light bird candidate image, the relative orientation from the camera, the coordinates of the entire object on the image, and the brightness gradient. The HOG (Histogram of Oriented Gradients) feature amount is collectively stored as EO motion information (visible light motion information) in the EO motion (bird candidate) information storage unit 152. The EO bird candidate image extraction unit 151 corresponds to the visible light moving body candidate image extraction unit of the present invention.

  As shown in FIG. 1, the EO moving object (bird candidate) information storage unit 152 is connected to the EO bird candidate image extraction unit 151. In the EO moving object (bird candidate) information storage unit 152, the EO bird candidate image and the EO moving object information extracted by the EO bird candidate image extracting unit 151 are recorded by the EO bird candidate image extracting unit 151.

  As shown in FIG. 1, the EO bird candidate determination unit 153 is connected to the EO bird candidate image extraction unit 151, the EO bird discrimination library 154, and the EO bird coordinate extraction unit 155. The EO bird candidate determination unit 153 sets a detection target moving body candidate (here, a bird) that is a candidate for a detection target moving object in the visible light image based on the luminance gradient histogram (HOG feature amount) of the EO bird candidate image. First of all, it is determined whether or not. That is, the EO bird candidate determination unit 153 detects a moving object candidate for detection as a candidate for a moving object to be detected in the visible light image based on the luminance gradient histogram (HOG feature value) included in the EO First, it is determined whether or not it has detected. Specifically, based on the luminance gradient histogram (HOG feature value) of the EO motion information, the EO bird candidate determination unit 153 is a binary based on a principle such as a support vector machine (SVM) or a neural network. The discriminator determines, using data in the EO bird discrimination library 154, whether the moving object is a bird (detection target moving object candidate). The EO bird candidate determination unit 153 corresponds to the visible light moving body candidate determination unit of the present invention.

  As shown in FIG. 1, the EO bird discrimination library 154 is connected to the EO bird candidate determination unit 153. The EO bird discrimination library 154 is a database for generating discrimination criteria of the binary discriminator of the EO bird candidate discrimination unit 153. When SVM is adopted, the EO bird discrimination library 154 is configured by image information of a bird and image information other than the bird for comparison.

  As shown in FIG. 1, the EO bird coordinate extraction unit 155 is connected to the EO bird candidate determination unit 153, the IR bird coordinate estimation unit 156, and the night bird detection processing unit 170. The EO bird coordinate extraction unit 155 extracts EO camera coordinates (visible light camera coordinates). The EO camera coordinates are coordinates that indicate the position of the bird (the detection target moving body candidate) primarily determined by the EO bird candidate determination unit 153 with reference to the EO imaging device 200. The EO bird coordinate extraction unit 155 corresponds to the visible light moving object coordinate extraction unit of the present invention.

  As shown in FIG. 1, the IR bird coordinate estimation unit 156 is connected to the EO bird coordinate extraction unit 155, the camera coordinate conversion matrix / disparity upper limit information storage unit 157, the IR bird candidate estimation unit 158, and the night bird detection processing unit 170. It is done. The IR bird coordinate estimation unit 156 estimates IR camera coordinates (infrared light camera coordinates). At this time, the IR bird coordinate estimation unit 156 estimates IR camera coordinates using the information stored in the camera coordinate conversion matrix / disparity upper limit information storage unit 157. The IR camera coordinates are coordinates that indicate the position corresponding to the EO camera coordinates with reference to the IR imaging device 300. Also, the IR bird coordinate estimation unit 156 generates and outputs IR image data corresponding to IR camera coordinates. That is, the IR bird coordinate estimation unit 156 generates and outputs an infrared light estimation image (IR estimation image) in which a bird candidate (detection target moving body candidate) is estimated and arranged on IR camera coordinates. The IR bird coordinate estimation unit 156 corresponds to the infrared light moving body coordinate estimation unit of the present invention.

  In the estimation of IR camera coordinates by the IR bird coordinate estimation unit 156, a camera coordinate conversion matrix between the EO imaging device 200 and the IR imaging device 300 and information of setting values of parallax upper limit are used. These pieces of information are stored in the camera coordinate conversion matrix / disparity upper limit information storage unit 157. The camera coordinate transformation matrix is calculated in advance from the optical characteristics and positional relationship of the EO imaging device 200 and the IR imaging device 300. The parallax upper limit is calculated from the size of the detection candidate bird (bird candidate, detection target moving body candidate) and the optical characteristics of the EO imaging device 200 and the IR imaging device 300.

  As shown in FIG. 1, the camera coordinate conversion matrix / disparity upper limit information storage unit 157 is connected to the IR bird coordinate estimation unit 156. The camera coordinate conversion matrix / parallax upper limit information storage unit 157 stores a camera coordinate conversion matrix between the EO imaging device 200 and the IR imaging device 300 and information of setting values of the parallax upper limit.

  As shown in FIG. 1, the IR bird candidate estimation unit 158 is connected to the IR bird coordinate estimation unit 156, the IR bird luminance distribution library 159, the IR bird discrimination library 174, and the alarm notification unit 190. The IR bird candidate estimation unit 158 is between the brightness distribution of the IR captured image received by the infrared light image data input processing unit 130 and the brightness distribution of the IR estimation image output by the infrared light bird coordinate estimation unit 156. Based on the degree of similarity, whether or not a bird candidate (detection target moving body candidate) is a bird (detection target moving body) is secondarily binary determined. In this binary discrimination, the IR bird candidate estimation unit 158 uses data stored in the IR bird luminance distribution library 159. The IR bird candidate estimation unit 158 finally outputs the moving object secondarily determined as a bird (moving object to be detected) to the alarm notification unit 190 as a bird (moving object to be detected). The IR bird candidate estimation unit 158 corresponds to the infrared light vehicle candidate estimation unit of the present invention.

  As shown in FIG. 1, the IR bird luminance distribution library 159 is connected to the IR bird candidate estimation unit 158. The IR bird luminance distribution library 159 stores data for generating discrimination criteria for binary discrimination. The IR bird candidate estimation unit 158 is between the brightness distribution of the IR captured image received by the infrared light image data input processing unit 130 and the brightness distribution of the IR estimation image output by the infrared light bird coordinate estimation unit 156. Calculation of the degree of similarity using data in the IR bird brightness distribution library 159, and secondarily binary determination whether or not the bird candidate (detection target moving object candidate) is a bird (detection target moving object) Do. The data in the IR bird luminance distribution library 159 is composed of bird luminance distribution information. That is, IR bird images collected and designated in advance (this bird IR image is, for example, data collected manually in advance, but with a number much smaller than the number of data required for bird shape identification processing Good) is stored.

  The configuration of the daytime bird detection processing unit 150 has been described above.

  The configuration of the nighttime bird detection processing unit 170 will be described. As shown in FIG. 1, the nighttime bird detection processing unit 170 is connected to the IR image data input processing unit 130 and the alarm notification unit 190. The nighttime bird detection processing unit 170 includes an IR bird candidate image extraction unit 171, an IR moving object (bird candidate) information storage unit 172, an IR bird candidate determination unit 173, and an IR bird determination library 174.

  As shown in FIG. 1, the IR bird candidate image extraction unit 171 is connected to the IR image data input processing unit 130, the IR moving object (bird candidate) information storage unit 172, and the IR bird candidate determination unit 173. The IR bird candidate image extraction unit 171 receives an infrared light captured image (IR captured image) from the IR image data input processing unit 130, and then calculates the outline of the moving object from difference information of three consecutive frames of images. By extracting, an infrared light bird candidate image (IR bird candidate image) is extracted. The infrared light bird candidate image corresponds to the infrared light moving body candidate image of the present invention. In addition, the IR bird candidate image extraction unit 171 detects the photographing time of the bird (moving object) included in the infrared light bird candidate image, the relative orientation from the camera, the luminance information with the coordinates of the entire object on the image, and the luminance. The histogram (HOG) feature quantities of the gradient are collectively stored in the IR moving body (bird candidate) information storage unit 172 as infrared light moving body information (IR moving body information). The IR bird candidate image extraction unit 171 corresponds to the infrared light moving body candidate image extraction unit of the present invention.

  As shown in FIG. 1, the IR moving object (bird candidate) information storage unit 172 is connected to the IR bird candidate image extraction unit 171. In the IR moving body (bird candidate) information storage unit 172, the IR bird candidate image and IR moving body information extracted by the IR bird candidate image extracting unit 171 are recorded by the IR bird candidate image extracting unit 171.

  As shown in FIG. 1, the IR bird candidate determination unit 173 is connected to the IR bird candidate image extraction unit 171, the IR bird determination library 174, and the alarm notification unit 190. The IR bird candidate determination unit 173 selects a detection target moving body candidate (here, a bird as a candidate for a detection target moving body in the infrared light image based on the brightness gradient histogram (HOG feature amount) of the IR bird candidate image). ) Is determined. That is, based on the luminance gradient histogram (HOG feature value) included in the IR moving object information, the IR bird candidate determining unit 173 detects a moving object candidate for detection which is a candidate for a moving object to be detected in the infrared light image (here, It is determined whether or not a bird is detected. Specifically, the IR bird candidate determination unit 173 uses a binary discriminator based on a principle such as a support vector machine (SVM) or a neural network based on a luminance gradient histogram (HOG feature value) of IR moving object information. Using data in the IR bird discrimination library 174, it is determined whether the moving body is a bird (detection target moving body candidate). The IR bird candidate determination unit 173 corresponds to the infrared light moving body candidate determination unit of the present invention.

  As shown in FIG. 1, the IR bird discrimination library 174 is connected to the IR bird candidate determination unit 173. The IR bird discrimination library 174 is a database for generating discrimination criteria of the binary discriminator of the IR bird candidate discrimination unit 173. When SVM is employed, the IR bird discrimination library 174 is configured by image information of a bird and image information other than a bird for comparison. The image information of this IR bird uses the result of daytime bird detection processing and is accumulated in the IR bird discrimination library 174.

  The configuration of the nighttime bird detection processing unit 170 has been described above.

  As shown in FIG. 1, the alarm notification unit 190 is connected to the IR bird candidate estimation unit 158 and the IR bird candidate determination unit 173. In the daytime bird detection process, when the alarm notification unit 190 determines that the detected moving object candidate is a bird (moving object to be detected) by the IR bird candidate estimation unit 156, the bird (detection object) in the EO captured image To notify that the In the nightly bird detection process, when the IR bird candidate determination unit 173 determines that the bird candidate (detection target moving object candidate) is a bird (detection target moving object), a bird (detection target moving object) in the IR captured image Notify that the The alarm notification unit 190 is a notification unit of the present invention.

  The configuration of the bird detection device 100 has been described above.

  Next, the operation of the bird detection device 100 will be described. Here, the operation (bird detection processing) of the bird detection apparatus 100 will be described separately for daytime and nighttime.

  First, the daytime bird detection processing of the bird detection device 100 will be described with reference to FIG. FIG. 2 is a diagram showing an operation flow of the bird detection device 100, and is a diagram for explaining daytime bird detection processing.

  As shown in FIG. 2, the EO image data input processing unit 110 acquires an EO photographed image from the EO imaging device 200 for each cycle (step (hereinafter, referred to as “S” 01)). At this time, the EO imaging device 200 continuously outputs the captured EO captured image to the EO image data input processing unit 110, for example, at a cycle of 50 frames per second.

  Next, as shown in FIG. 2, the EO bird candidate image extraction unit 151 extracts a bird candidate moving body from the difference image and acquires EO moving body information (S 02).

  More specifically, the EO bird candidate image extraction unit 151 extracts a bird candidate image that is a candidate for a bird image from the EO captured image as a visible light bird candidate image (EO bird candidate image). That is, after receiving the visible light captured image from the EO image data input processing unit 110, the EO bird candidate image extraction unit 151 extracts the outline of the moving object from difference information of images of three consecutive frames. , EO bird candidate image extraction.

  More specifically, the EO bird candidate image extraction unit 151 extracts three images continuously or sequentially extracted at constant intervals based on the EO captured image input from the EO imaging device 200 (each The frames n−1, n, n + 1) are binarized with luminance. Then, the EO bird candidate image extraction unit 151 generates difference images m = n− (n−1) and m + 1 = (n + 1) −n, respectively. Furthermore, the EO bird candidate image extraction unit 151 extracts EO bird candidate images from the three original images by generating a product image D = m * (m + 1) of m and m + 1.

  In addition, the EO bird candidate image extraction unit 151 detects the photographing time of the bird candidate (moving object candidate) included in the EO bird candidate image, the relative orientation from the camera, the luminance information with the coordinates of the entire object on the image, and the luminance The histogram of gradient (HOG) feature quantities are collectively stored as EO motion information (visible light motion information) in the EO motion (bird candidate) information storage unit 152.

  More specifically, for the image of bird candidate (moving object candidate) extracted on D (EO bird candidate image), the EO bird candidate image extracting unit 151 determines the position and shape of the center of gravity on D, and the size Is normalized to a preset reference value to calculate the HOG feature value. Then, the EO bird candidate image extraction unit 151 combines the imaging time and the optical axis orientation of the EO imaging device 200 at the imaging time, and records the result as EO moving object information in the EO moving object (bird candidate) information storage unit 152.

  Next, as shown in FIG. 2, the EO bird candidate determination unit 153 primarily determines whether or not a detection target moving body candidate (bird) which is a detection target moving body candidate is detected in the visible light image. To do (S03).

  That is, the EO bird candidate determination unit 153 applies, to each EO motion information, binary discrimination processing based on a threshold value generated from the database of the EO bird determination library 154 to determine whether the EO motion information is a bird. Make a primary decision whether or not. More specifically, the EO bird candidate determination unit 153 selects a detection target moving body candidate that is a candidate for a detection target moving object in the visible light image based on the luminance gradient histogram (HOG feature amount) of the EO bird candidate image (here Then, it is temporarily determined whether or not a bird is detected. That is, the EO bird candidate determination unit 153 selects a bird candidate (detection target moving object candidate) that is a candidate for a detection target moving object in the EO captured image based on the luminance gradient histogram (HOG feature amount) included in EO moving object information. It is temporarily determined whether or not it has been detected.

  When the EO bird candidate determination unit 153 primarily determines that a bird candidate (detection target moving object candidate) is detected in the visible light image (S03, YES), the EO movement information is diverted to a secondary determination of S06 or lower.

  On the other hand, when the EO bird candidate determination unit 153 primarily determines not to detect a bird candidate (detection target moving object candidate) that is a candidate for a detection target moving object in the EO captured image (S03, NO), the EO movement information is It is deleted (S05).

  Here, for secondary determination, a bird candidate image that is a candidate of a bird image is acquired as an infrared light bird candidate image (IR bird candidate image) from the IR imaging image acquired by the IR imaging device 300 There is a need.

  First, the EO bird coordinate extraction unit 155 acquires EO camera coordinates of the EO moving object in the EO captured image input from the EO imaging device 200. As described above, the EO camera coordinates are coordinates that indicate the position of the bird candidate (the detection target moving object candidate) primarily determined by the EO bird candidate determination unit 153 with reference to the EO imaging device 200.

  In the process of S06, the IR bird coordinate estimation unit 156 estimates IR camera coordinates (also referred to as IR bird coordinates) corresponding to EO camera coordinates (also referred to as EO bird coordinates) (S06). That is, the IR bird coordinate estimation unit 156 calculates the corresponding IR camera coordinates in the EO captured image captured by the IR imaging device 200 with respect to the EO camera coordinates. Specifically, the IR bird coordinate estimation unit 156 estimates IR camera coordinates (infrared light camera coordinates) using the information stored in the camera coordinate conversion matrix / disparity upper limit information storage unit 157.

To that end, IR camera coordinates make use of the following two constraints.
Is the epipolar constraint of the bird candidate position (the position of the object in the image of the object is limited to a straight line due to the positional relationship between the two cameras that capture the same object). Second, the parallax value upper limit on the epipolar line (auxiliary line) is narrowed from the bird's body length.

  Within the range of the above constraint conditions, the IR bird coordinate estimation unit 156 estimates a ring filter result of luminance values and a time series filter result as an IR bird candidate image (a bird candidate image in an IR captured image).

  FIG. 3 is a diagram for explaining a method of acquiring an IR bird candidate image. FIG. 3 shows an EO captured image and an IR captured image obtained by capturing the same bird candidate (detection target moving object).

  As shown in FIG. 3, the bird detection device 100 first performs EO bird detection ((1) in FIG. 3). That is, the EO bird candidate image extraction unit 151 extracts a bird candidate image which is a candidate of a bird image as a visible light bird candidate image (EO bird candidate image) from the visible light captured image.

  Next, as shown in FIG. 3, the IR bird coordinate estimation unit 156 calculates an epipolar line in a state where the EO captured image and the IR captured image are arranged in parallel ((2) in FIG. 3).

  Next, as shown in FIG. 3, the IR bird coordinate estimation unit 156 limits the parallax maximum range ((3) in FIG. 3). That is, the IR bird coordinate estimation unit 156 performs processing of narrowing the parallax value upper limit on the epipolar line from the bird's body length.

  Then, as shown in FIG. 3, the IR bird coordinate estimation unit 156 obtains an IR bird candidate image by calculating a peak value of luminance between two epipolar lines.

  By the way, when two cameras (EO imaging device 200, IR imaging device 300) independently perform pan-tilt turning operation, the visual axis is different, so rotation is performed to calculate the positional relationship of the camera coordinate system in epipolar restraint calculation. Matrix operations are required.

  However, as shown in FIG. 3, this rotation matrix calculation can be omitted and the calculation cost of the epipolar constraint calculation can be reduced by adding the restriction of aligning the visual axes of the EO imaging device 200 and the IR imaging device 300 in parallel. it can.

  Next, referring back to FIG. 2, the IR bird candidate estimation unit 158 performs filtering with IR bird similarity distribution information (S07). More specifically, the IR bird candidate estimation unit 158 determines the primary determination result of the bird candidate from the EO captured image of the moving object to be detected, the brightness distribution of the bird candidate in the IR captured image, and the brightness distribution of the actual bird. Filtering by similarity calculation of to reduce false positives. That is, the IR bird candidate estimation unit 158 calculates the luminance distribution of the IR imaging image received by the infrared light image data input processing unit 130 and the luminance distribution of the IR estimation image output by the infrared light bird coordinate estimation unit 156. Filtering by calculating the degree of similarity between to reduce false positives.

  In this similarity calculation of the luminance distribution, an IR bird image collected and designated in advance (this bird IR image is, for example, data collected in advance manually. However, it is necessary for bird shape identification processing. (The number of data is much smaller than the number of data), and the database (IR bird brightness distribution library 159) in which the brightness distributions are stored. In general, birds have a constant body temperature distribution of 40 ° C to 42 ° C. For this reason, when the imaging setting (shutter speed, brightness, and contrast) in the IR imaging device 300 is fixed, a constant tendency of luminance distribution can be obtained.

  Next, the IR bird candidate estimation unit 158 passes through the filtering process of S07, and is outputted by the infrared light bird coordinate estimation unit 156 and the luminance distribution of the IR captured image received by the infrared light image data input processing unit 130. Based on the degree of similarity between the estimated IR estimated image and the luminance distribution of the IR estimated image, whether or not the detection target moving body candidate is a bird (moving target detection target) is secondarily binary determined (S08).

  When it is determined by the IR bird candidate estimation unit 158 that the bird candidate (detection target moving body candidate) is a bird (detection target moving body) (S08, YES), the IR bird candidate estimation unit 158 generates a bird (detection target) Finally, the moving object secondarily determined as a moving object) is output to the alarm notification unit 190 as a bird (moving object to be detected). Then, the alarm notification unit 190 notifies that a bird (a detection target moving object) appears in the visible light image (S10). Furthermore, the IR bird candidate estimation unit 158 adds the IR bird candidate image to the IR bird luminance distribution library 159 and updates it.

  On the other hand, when it is determined by the IR bird candidate estimation unit 158 that the bird candidate (detection target moving body candidate) is not a bird (detection target moving body) (S08, NO), the EO movement information is deleted (S09).

  The daytime bird detection processing of the bird detection device 100 has been described above.

  Next, nighttime bird detection processing will be described with reference to FIG. FIG. 4 is a diagram showing an operation flow of the bird detection device 100, and is a diagram for explaining a bird detection process at night.

  In order to perform nighttime bird detection, primary judgment of daytime bird detection processing using an EO captured image is applied to an IR captured image.

  In the bird detection process for EO captured images, binary discrimination using the EO bird discrimination library 154 was performed in the shape identification process in the EO bird candidate determination unit 153. However, the existing EO bird discrimination library 154 can not be used for bird detection processing in an IR captured image because the luminance information and the gradient are different from the IR captured image. Also, in general, in order to perform shape identification with high accuracy, it is necessary to prepare a large amount of bird image information.

  Therefore, by storing the IR bird candidate image obtained together with the bird detection in the EO captured image in the daytime in the IR bird discrimination library 174 for night bird detection processing, the efficiency by the automatic learning of the IR bird candidate determination unit 173 And improve identification accuracy.

  Based on the above points, the operation of the night bird detection process will be described according to FIG.

  As shown in Fig. 4, the IR image data input processing unit 110 acquires an EO captured image (visible light captured image) from the EO imaging device 200 for each cycle (step (S21). At this time, the IR imaging device). 300 continuously outputs the captured IR captured image (infrared captured image) to the IR image data input processing unit 170, for example, at a cycle of 50 frames per second.

  Next, as shown in FIG. 4, the IR bird candidate image extraction unit 171 extracts a bird candidate moving object from the difference image and acquires IR moving object information (S22).

  That is, after the IR bird candidate image extraction unit 171 receives an infrared light captured image from the IR image data input processing unit 130, it extracts an outline of a moving object from difference information of images of three consecutive frames. Thus, the infrared bird candidate image (IR bird candidate image) is extracted.

  More specifically, similarly to the EO bird candidate image extraction unit 151, the IR bird candidate image extraction unit 171 continuously or in chronological order on the basis of the IR captured image input from the IR imaging apparatus 300. Three images (each set as frame n-1, n, n + 1) extracted at a constant interval are binarized with luminance. Then, the IR bird candidate image extraction unit 171 generates difference images m = n− (n−1) and m + 1 = (n + 1) −n, respectively. Furthermore, the IR bird candidate image extraction unit 171 extracts IR bird candidate images from the three original images by generating a product image D = m * (m + 1) of m and m + 1.

  In addition, the IR bird candidate image extraction unit 171 detects the photographing time of the bird candidate (moving object candidate) included in the IR bird candidate image, the relative orientation from the camera, the luminance information with the coordinates of the entire object on the image, and the luminance The histogram (HOG) feature quantities of the gradient are collectively stored in the IR moving body (bird candidate) information storage unit 172 as infrared light moving body information (IR moving body information).

  For a moving object extracted on D, the IR bird candidate image extracting unit 171 calculates the HOG feature after normalizing the position of the center of gravity and the shape on D, and the size to a preset reference value. . Then, the IR bird candidate image extraction unit 171 combines the imaging time and the optical axis orientation of the IR imaging device 300 at the imaging time, and records it as IR moving object information in the IR moving object (bird candidate) information storage unit 172.

  Next, as shown in FIG. 4, the IR bird candidate determination unit 173 temporarily determines whether or not a bird candidate (detection target moving object candidate) that is a candidate for a detection target moving object is detected in the infrared light image. (S23).

  That is, the IR bird candidate determination unit 173 applies a binary discrimination process based on a threshold value generated from the database of the IR bird discrimination library 174 to each IR moving body information, thereby determining whether the IR moving body information is a bird Make a primary decision whether or not. More specifically, based on the luminance gradient histogram (HOG feature value) of the IR bird candidate image, the IR bird candidate determination unit 173 detects a detection object moving body candidate that is a candidate for a detection object moving object (herein Then, it is temporarily determined whether or not a bird is detected. That is, the IR bird candidate determination unit 173 selects a bird candidate (detection target moving object candidate) that is a candidate for a detection target moving object in the IR captured image based on the luminance gradient histogram (HOG feature amount) included in IR moving object information. It is temporarily determined whether or not it has been detected.

  If the IR bird candidate determination unit 173 determines that a bird candidate (detection target moving object candidate) that is a candidate for a detection target moving object is detected in the infrared light image (S23, YES), the IR bird candidate determination unit 173 The moving body determined as a bird (moving object to be detected) is finally output to the alarm notification unit 190 as a bird (moving object to be detected). Then, the alarm notification unit 190 notifies that a bird (a detection target moving object) appears in the IR captured image (S25).

  On the other hand, when the IR bird candidate determination unit 173 determines not to detect a bird candidate (detection target moving object candidate) that is a candidate for a detection target moving object in the infrared light image (S23, NO), the IR moving object information is deleted (S24).

  In the database of the IR bird discrimination library 174, IR bird candidate images acquired in the daytime bird detection process are sequentially accumulated, and the database is automatically updated.

  The nighttime bird detection process of the bird detection device 100 has been described above.

<Modification of Embodiment>
Next, a modification of the bird detection device 100 will be described.

  In the variation of the bird detection apparatus 100, the number of the IR imaging apparatuses 200 is plural. Corresponding to this, the IR image data processing unit 130 and the nighttime bird detection processing unit 170 are also plural. At this time, the plurality of IR imaging devices 200 may correspond to different wavelength bands (near infrared, mid infrared, far infrared). The plurality of IR imaging devices 200 are connected to each IR image data input processing unit 130.

  In this case, all the information of the IR captured image output by each IR imaging device 200 is output in parallel to each IR image data input processing unit 130 of the bird detection device 100. At this time, it is possible to select which IR imaging apparatus 300 to process the input by the operation of the user.

  Alternatively, in the processes S06 and S07 of FIG. 2, the filtering accuracy based on the IR bird luminance distribution information can be performed in any combination, so that the discrimination accuracy of the bird candidate can be enhanced.

  Heretofore, modified examples of the bird detection device 100 have been described.

  Note that the processing S10 in FIG. 2 and the processing S25 in FIG. 4 may be configured to display not the alarm notification but the appearance frequency of the bird with respect to the optical axis direction angle of the imaging device on a display or the like.

  The internal program may be replaced by an external storage device, external information connected via a network such as WWW, or hardware such as a gate array.

  Furthermore, the target of detection and discrimination is not limited to birds, and if it is an animal, a human, or any other object capable of defining a standard outer shape and temperature distribution, this is the detection target (target of detection) of this device. Can do.

  As described above, the bird detection apparatus 100 (moving object detection apparatus) according to the embodiment of the present invention includes the EO image data input processing unit 110 (visible light image data input processing unit) and the IR image data input processing unit 130 (infrared Optical image data input processing unit), EO bird candidate image extraction unit 151 (visible light moving body candidate image extraction unit), EO bird candidate determination unit 153 (visible light moving body candidate determination unit), EO bird coordinate extraction unit 155 ( The visible light bird coordinate extraction unit), the IR bird coordinate estimation unit 156 (infrared light moving body coordinate estimation unit), and the IR bird candidate estimation unit 158 (infrared light moving body candidate estimation unit) are provided.

  The EO image data input processing unit 110 receives an input of an EO captured image by the EO imaging device 200 (visible light imaging device) that captures light in a visible light wavelength band to generate an EO captured image (visible light captured image) .

  The IR image data input processing unit 130 captures an image of light in the infrared wavelength band to generate an IR captured image (infrared captured image). Accept input.

  The EO bird candidate image extraction unit 151 extracts an EO bird candidate image (visible light moving body candidate image), which is a candidate of an image of a bird (moving object) to be detected, from the EO captured image.

  The EO bird candidate determination unit 153 primarily determines whether or not a bird (detection target moving object candidate) that is a candidate for a detection target moving object is detected in the EO captured image based on the luminance gradient histogram of the EO bird candidate image. judge.

  The EO bird coordinate extraction unit 155 displays EO camera coordinates (visible light camera coordinates) that indicate the position of the bird (detection target moving object candidate) primarily determined by the EO bird candidate determination unit 153 with reference to the EO imaging device 200. Estimate the coordinates).

  The IR bird coordinate estimation unit 156 estimates IR camera coordinates (infrared light camera coordinates), and estimates an IR estimation image (infrared light estimation image) in which a bird (detection target moving body candidate) is estimated and arranged on the IR camera coordinates. Generate and output. Note that IR camera coordinates are coordinates that indicate a position corresponding to EO camera coordinates with reference to the IR imaging device 300.

  The IR bird candidate estimation unit 158 determines the similarity between the luminance distribution of the IR captured image received by the IR image data input processing unit 130 and the luminance distribution of the IR estimated image output by the IR bird coordinate estimation unit 156. Based on the determination, it is determined whether or not the bird candidate (detection target moving object candidate) is a detection target bird (moving object).

  As described above, the luminance distribution of the IR estimation image obtained by coordinate-converting the EO imaging image obtained by imaging the light in the visible light wavelength band by the EO imaging device 200 on the basis of the IR imaging device 300 and the light in the infrared light wavelength band The bird is detected based on the similarity with the luminance distribution of the IR captured image. That is, the bird detection apparatus 100 acquires temperature information as light characteristics in the infrared light wavelength band, and uses an IR estimation image in which difference information on temperature distribution between birds and non-birds (other than birds) is reflected. , Have detected birds.

  Thereby, it is possible to filter a background image caused by a mobile object having a shape similar to a bird or an object having a random shape (such as the reflection of sunlight, the fluctuation of trees, etc.). As a result, according to the bird detection device 100 in the embodiment of the present invention, a mobile object having a shape resembling a bird or an object having a random shape can be obtained by simultaneously using a visible light captured image and an infrared light captured image. It is possible to accurately detect a moving object to be detected by suppressing false detection caused by reflection of sunlight, fluctuation of trees, and the like.

  In the technology of detecting birds using a monocular visible light camera (background technology), in image processing in which a detection target image is extracted from an acquired image by shape comparison, various things such as fluctuation of trees, white waves, sunlight reflection of sea surface, etc. There is a limit in removing the case where the background becomes close to the shape of an apparent bird, and there is a problem that false detection may occur. On the other hand, in the present invention, information of the characteristics of the background object in the infrared light wavelength range, that is, temperature is acquired, and an IR image in which information of differences in temperature distribution between birds and non-birds is reflected is used. For this reason, the bird detection apparatus 100 of the present invention can filter the background image as described above, and can perform bird detection with higher accuracy as compared to the technology of detecting a bird using a monocular visible light camera. It is.

  In the bird detection apparatus 100 according to the embodiment of the present invention, the EO bird candidate image extraction unit 151 extracts an EO bird candidate image from the EO captured image by extracting an outline of a moving object in the EO captured image. . Thus, the EO bird candidate image can be easily identified and extracted from the EO captured image.

  In addition, the bird detection device 100 according to the embodiment of the present invention includes an alarm notification unit 190 (notification unit). The alarm notification unit 190 indicates that a bird (detection target moving object) appears in the EO image when the IR bird candidate estimation unit 156 determines that the detected moving object candidate is a detection target bird (moving object). Notice. Thereby, the user of the bird notification device 100 can be notified of the presence of a bird.

  In addition, the bird detection device 100 according to the embodiment of the present invention includes an IR bird candidate image extraction unit 171 (infrared light moving body candidate image extraction unit) and an IR bird candidate determination unit 173 (infrared light movement object candidate determining unit) Is equipped. The IR bird candidate image extraction unit 171 extracts an IR bird candidate image (infrared light moving body candidate image) which is a candidate of an image of a detection target bird (moving object) from the IR captured image. The IR bird candidate determination unit 173 determines whether or not a bird (a detection target moving object candidate) is detected in the IR captured image based on the luminance gradient histogram of the IR bird candidate image. Thereby, even at night, birds can be detected.

  In the technique of detecting a bird using a monocular visible light camera (background art), since an EO imaging device is used, the bird detection can not be performed at night. As a general measure, it is conceivable to introduce an IR imaging device and to image a bird at night. However, in order to perform machine learning of bird shape identification processing, it was necessary to prepare a large amount of IR bird images. On the other hand, in the bird detection apparatus 100 of the present invention, a large number of IR bird images corresponding to birds detected by daytime bird detection processing are accumulated and applied to machine learning of nighttime bird shape identification processing. It is possible to solve the problem of having to prepare IR bird images, to automate and streamline machine learning, and to improve shape identification accuracy.

  In addition, the bird detection device 100 according to the embodiment of the present invention includes an alarm notification unit 190 (notification unit). In the alarm notification unit 190, the detection target moving object candidate is a bird (detection target moving object) by the IR bird candidate estimation unit 156 (infrared light moving object candidate estimation unit) or the IR bird candidate determination unit 173 (infrared light moving object candidate determination unit). When it is determined that the EO captured image or the IR captured image is notified that a bird (a detection target moving object) is shown. Thereby, the user of the bird notification device 100 can be notified of the presence of a bird day and night.

  The present invention uses, for example, a device that extracts a specific target from image data and determines whether it is a target set by a user in advance or not, bird strike prevention at an airport, a wind power generation facility (wind turbine) Devices, foreign body (FOD) detectors, farmer's animals and animals damage prevention devices, human detection devices in surveillance camera images, alarm devices when intruders invade warning areas, and functions of the above devices during day and night The present invention can be applied to devices that need to be implemented.

  The present invention has been described above based on the embodiments. The embodiment is an exemplification, and various changes, additions, subtractions, and combinations may be added to the above-described embodiments without departing from the spirit of the present invention. Those skilled in the art will understand that variations to which these changes, additions, and reductions are added are within the scope of the present invention.

DESCRIPTION OF SYMBOLS 100 Bird detection apparatus 110 EO image data input processing part 130 IR image data input processing part 150 Daytime bird detection processing part 151 EO bird candidate image extraction part 152 EO moving object (bird candidate) information storage part 153 EO bird candidate judgment part 154 EO bird Discrimination library 155 EO bird coordinate extraction unit 156 IR bird coordinate estimation unit 157 Camera coordinate transformation matrix / parallax upper limit information storage unit 158 IR bird candidate estimation unit 159 IR bird brightness distribution library 170 Night bird detection processing unit 171 IR bird candidate image extraction unit 172 IR moving object (bird candidate) information storage unit 173 IR bird candidate determination unit 174 IR bird discrimination library 190 alarm notification unit 200 EO imaging device 300 IR imaging device

Claims (7)

A visible light image data input processing unit that receives an input of the visible light captured image by a visible light imaging device that captures light in a visible light wavelength band to generate a visible light captured image;
An infrared light image data input processing unit that receives an input of the infrared light imaging image by an infrared light imaging device that images light in an infrared light wavelength band to generate an infrared light imaging image;
A visible light moving body candidate image extraction unit that extracts a visible light moving body candidate image, which is a candidate for an image of a moving object to be detected, from the visible light imaging image;
Visible light moving body candidate determination to temporarily determine whether or not a detection target moving body candidate which is a detection target moving body candidate is detected in the visible light imaging image based on a luminance gradient histogram of the visible light moving body candidate image Department,
A visible light moving body coordinate extraction unit for estimating a visible light camera coordinate which is a coordinate indicating the position of the detection target moving body candidate determined primarily by the visible light moving body candidate determination unit with the visible light imaging device as a reference;
The infrared camera coordinate which shows the position corresponding to the said visible light camera coordinate on the basis of the said infrared light imaging device is estimated, The said detection target moving body candidate was presumedly arrange | positioned on the said infrared camera coordinate An infrared light moving body coordinate estimation unit that generates and outputs an infrared light estimation image;
Similarity between the luminance distribution of the infrared light imaging image received by the infrared light image data input processing unit and the luminance distribution of the infrared light estimation image output by the infrared light moving body coordinate estimation unit An infrared light moving body candidate estimating unit that determines whether the detection target moving body candidate is a moving object to be detected based on a degree. The visible light moving body candidate image extraction unit according to claim 1, wherein the visible light moving body candidate image is extracted from the visible light pickup image by extracting an outline of a moving object in the visible light pickup image. Motion detection device. Notification notifying that the by infrared body candidate estimation unit, when the detection target moving object candidate is determined to be a moving body of the detection target, the detection target body to the visible light imaging image is captured The moving body detection device according to claim 1 or 2, further comprising: An infrared light moving body candidate image extraction unit which extracts an infrared light moving body candidate image, which is a candidate for an image of a moving object to be detected, from the infrared light imaging image;
An infrared light moving body candidate determination unit that determines whether the detection target moving body candidate has been detected in the infrared light captured image based on a luminance gradient histogram of the infrared light moving body candidate image; The moving body detection apparatus according to 1. By the infrared light body candidate estimation unit or the infrared moving object candidate determination section, the detection when the target moving object candidate is determined to be the detection target body, in the visible light captured image or the infrared light imaging 5. The moving object detection device according to claim 4, further comprising a notification unit that notifies that the detection target moving object is included in an image. A visible light image data input processing step of receiving an input of the visible light captured image by a visible light imaging device that captures light in a visible light wavelength band to generate a visible light captured image;
An infrared light image data input processing step of receiving an input of the infrared light imaging image by an infrared light imaging device for imaging light in an infrared light wavelength band to generate an infrared light imaging image;
A visible light moving body candidate image extracting step of extracting a visible light moving body candidate image, which is a candidate for an image of a moving object to be detected, from the visible light imaging image;
Visible light moving body candidate determination to temporarily determine whether or not a detection target moving body candidate which is a detection target moving body candidate is detected in the visible light imaging image based on a luminance gradient histogram of the visible light moving body candidate image Step and
A visible light moving body coordinate extraction step of estimating a visible light camera coordinate which is a coordinate indicating the position of the detection target moving body candidate primarily determined in the visible light moving body candidate determination step on the basis of the visible light imaging device;
The infrared camera coordinate which shows the position corresponding to the said visible light camera coordinate on the basis of the said infrared light imaging device is estimated, The said detection target moving body candidate was presumedly arrange | positioned on the said infrared camera coordinate An infrared light moving body coordinate estimation step of generating and outputting an infrared light estimation image;
Similarity between the luminance distribution of the infrared light imaging image received by the infrared light image data input processing step and the luminance distribution of the infrared light estimation image output by the infrared light moving body coordinate estimation step An infrared light moving body candidate estimation step of determining whether the detection target moving body candidate is a moving object of the detection target based on a degree. A visible light image data input processing step of receiving an input of the visible light captured image by a visible light imaging device that captures light in a visible light wavelength band to generate a visible light captured image;
An infrared light image data input processing step of receiving an input of the infrared light imaging image by an infrared light imaging device for imaging light in an infrared light wavelength band to generate an infrared light imaging image;
A visible light moving body candidate image extracting step of extracting a visible light moving body candidate image, which is a candidate for an image of a moving object to be detected, from the visible light imaging image;
Visible light moving body candidate determination to temporarily determine whether or not a detection target moving body candidate which is a detection target moving body candidate is detected in the visible light imaging image based on a luminance gradient histogram of the visible light moving body candidate image Step and
A visible light moving body coordinate extraction step of estimating a visible light camera coordinate which is a coordinate indicating the position of the detection target moving body candidate primarily determined in the visible light moving body candidate determination step on the basis of the visible light imaging device;
The infrared camera coordinate which shows the position corresponding to the said visible light camera coordinate on the basis of the said infrared light imaging device is estimated, The said detection target moving body candidate was presumedly arrange | positioned on the said infrared camera coordinate An infrared light moving body coordinate estimation step of generating and outputting an infrared light estimation image;
Similarity between the luminance distribution of the infrared light imaging image received by the infrared light image data input processing step and the luminance distribution of the infrared light estimation image output by the infrared light moving body coordinate estimation step And an infrared light moving body candidate estimating step of determining whether the detection target moving body candidate is the moving object to be detected based on the degree.