KR101852058B1 - unexpected accident detecting method using dual camera - Google Patents

Abstract

The present invention relates to an unexpected situation detection method using a dual camera, using a camera and a thermal image camera to detect and track an object, thus detecting an obstacle, which is not detected when a single camera is operated, and remarkably increasing the levels of accuracy and reliability of determination with respect to occurrence of an unexpected situation due to a huge amount of collected data; configured to determine a reference image according to a visibility condition and displaying an object which is different from the reference image and generating an integrated image, thus efficiently providing information through an image; and allowing an unexpected situation management unit to track the locus of a final object to determine fog, a vehicle accident, icing, reverse running, a fallen object, illegal parking and stopping, a slow vehicle, a passenger, etc., to generate unexpected situation occurrence data informing of the occurrence of the unexpected situation, thus using the huge amount of collected data to quickly and correctly determine the unexpected situation. According to the present invention, the method comprises a photographing step, a calibration step, an image analysis step, and a thermal image analysis step.

Description

[0001] The present invention relates to an unexpected accident detecting method using a dual camera,

The present invention relates to a method of detecting an unexpected situation using a dual camera, and more particularly, to an object tracking method using a visible light camera and a thermal imager to accurately detect an object even in a frustrating clock environment, The present invention relates to an unexpected situation detection method capable of increasing the detection rate and improving the accuracy and reliability of the judgment as to whether or not an unexpected event has occurred.

The demand for intelligent video surveillance technology that detects the occurrence of an event by detecting and tracking an object by analyzing the captured image as the video technology and equipment are developed is increasing exponentially.

Currently, cameras used in intelligent video surveillance technology are widely used in visible light cameras, but such visible light cameras can not detect objects due to various causes such as direct sunlight, scattering of light, insufficient amount of light and illumination, Which has a structural limit of falling.

Accordingly, in order to solve the problem of the visible light camera which can not perform the desired purpose in the weak clock environment, researches on special cameras such as an infrared camera, a thermal camera, and the like are being carried out variously.

On the other hand, intelligent transportation system (ITS, Intelligent Transportation System) has been researched and widely used due to recent development of information communication technology, provision of telematics devices and establishment of ubiquitous environment.

Since the intelligent traffic system (ITS) collects various data related to the vehicle and analyzes and processes vast amount of collected data to determine whether an unexpected situation occurs, and to generate traffic information, Reliability and accuracy of Intelligent Transportation System (ITS) are determined according to how accurate the vehicles can be collected, and accordingly various studies are being conducted to improve the accuracy and efficiency of vehicle detection.

As a result, intelligent video surveillance (ITS) technology is integrated with ITS to monitor the site, and various researches have been conducted on systems for efficiently detecting unexpected situations.

1 is a flowchart showing a vehicle information recognition method using a thermal imaging camera disclosed in Korean Patent No. 10-1370627 (entitled "Method and Apparatus for Recognizing Vehicle Information Using a Thermal Camera").

The vehicle information recognition method (hereinafter referred to as " prior art ") S900 using the thermal imaging camera of Fig. 1 includes a vehicle temperature sensing step S910 of sensing the temperature of the vehicle on which the thermal imaging camera is mounted, (S920) for generating an infrared image according to the temperature sensed by the temperature sensing step (S910), comparing the thermal image generated by the infrared image generating step (S920) with a preset pattern (S930) and a pattern comparison step (S930). If the thermal image is equal to or greater than a threshold value of the predetermined driving pattern, at least one of acceleration, sudden driving, And providing a driving method corresponding to the driving information (S940).

According to the conventional technology (S900) configured as described above, it is possible to detect the temperature of the vehicle using a thermal imaging camera and generate driving information of the vehicle based on the sensed temperature without installing additional devices or systems, thereby reducing the cost And it is possible to generate vehicle driving information even in an environment in which a clock is not secured (unstable) such as fog.

However, since the conventional technology (S900) is configured to visually show an object by measuring the temperature of the vehicle using only a single thermal camera, it has a structural limitation that various information that can be acquired through a general camera can not be collected.

In addition, since the conventional technology (S900) is configured to detect only the vehicle object using the vehicle temperature, information about the surrounding environment such as the road surface state can not be acquired, and only the limited data is collected, .

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems and it is an object of the present invention to provide a camera and an infrared camera capable of detecting objects that can not be detected when the camera is driven by a single camera, The present invention is intended to provide a method for detecting an unexpected situation using a dual camera capable of increasing the accuracy and reliability of judgment as to whether or not an unexpected event occurs, because the amount of collected data is enormous.

In addition, another object of the present invention is to provide a method and apparatus for determining a reference image according to a clock state and displaying an object of an image other than a reference image on a reference image to generate an integrated image, The present invention provides a method for detecting an unexpected situation using a dual camera.

Further, another problem of the present invention is that the unexpected situation management unit tracks the trajectory of the final object to judge whether there is a mist, a vehicle accident, freezing, inversion, falling objects, illegal parking vehicles, slow vehicles, pedestrians, The present invention is to provide a method of detecting an unexpected situation using a dual camera that is capable of quickly and accurately discriminating an unexpected situation by using vast collection data by being configured to generate unexpected occurrence data that an unexpected situation has occurred.

In addition, another object of the present invention is to provide a method and apparatus for displaying all objects detected by a captured image analysis unit and a thermal image analysis unit when an integrated image is generated, And more particularly, to a method for detecting an unexpected situation using a dual camera.

According to an aspect of the present invention, there is provided an image processing method comprising: capturing an image of a predetermined sensing area using a visible light camera and a thermal image of the sensing area using a thermal camera; A calibration step of acquiring calibration information which is relationship information for associating the sensed image with the two-dimensional image of the sensed image and the deteriorated image obtained by the sensing step; Analyzing the photographed image by referring to the calibration information by the calibration step and detecting a first object; A thermal image analysis step of analyzing the thermal image by referring to the calibration information by the calibration step to detect a second object; Wherein the step of generating the integrated image comprises the steps of: determining a reference image to be used as a basis of the integrated image by using any one of the photographed image and the thermal image using a predetermined determination method; An image matching step of generating image matching information by referring to the calibration information, the image matching information being information indicating a relationship between the photographed image and the spatial relations of the degradation images; Comparing the first objects detected by the captured image analyzing step and the second objects detected by the thermal image analyzing step with the first object having position information within a predetermined threshold range, An object comparison step of determining two objects as the same object and discriminating the object as a final object and determining a first object or a second object that is out of a predetermined threshold range as a final object; The objects determined as the final object by the object comparison step are displayed on the reference image determined by the reference image determination step and the temperature measured by the thermal image analysis step is displayed on each displayed object to generate an integrated image And an integrated image processing step.

According to another aspect of the present invention, there is provided a method of determining a reference image, the method comprising: determining a reference image as a captured image when a light amount is equal to or greater than a threshold value; And a second method of determining a reference image as a photographed image when the detected sharpness is greater than or equal to a second threshold value and determining the reference image as a thermal image if the detected sharpness is less than a second threshold value desirable.

Further, in the present invention, the unexpected situation detection method may further include an unexpected situation management step performed after the integrated image generation step and a transmission step performed after the unexpected situation management step, A final object detecting step of detecting a position of each of the final objects displayed in the integrated image generated by the generating step; A trajectory tracking step of tracking a trajectory of each final object using positional information of each of the final objects detected by the final object detecting step; A determining step of determining whether or not an unexpected event occurs using the trajectory information detected by the trajectory tracking step; Further comprising the step of generating an unexpected situation occurrence data which is driven when the unexpected situation is determined by said determining step and generates the unexpected occurrence data including the contents and the position information related to the unexpected situation, A first transmission step of transmitting the unexpected situation occurrence data generated by the situation occurrence data generation step to the local server; and a second transmission step of transmitting the unexpected situation occurrence data transmitted through the first transmission step to the external control center server And a second transmission step.

Further, in the present invention, the determining step may include calculating a quantity of the first objects (N1) detected by the captured image analyzing step and a quantity (N2) of the second objects detected by the thermal image analyzing step, Determining whether a fog exists in the detection area when the absolute value of the difference is less than a predetermined threshold value, thereby determining that the field of view is not secured; A vehicle accident determination step of determining that a vehicle accident occurs when the speed of the final objects detected by the trajectory tracking step is reduced and the speed of any one of the final objects becomes '0'; An icing determining step of determining a final object, which is a second object among the final objects detected only in the thermal image analyzing step, and whose temperature is less than a predetermined threshold and has no motion, as icing; Determining a final object as an inverse car if the final object is detected in the same lane as the opposite direction; Determining a final object that is not present in a previous frame as a falling object when the speed is less than a threshold; An illegal parking discrimination step of judging the final object as an illegal parking space vehicle when a final object displaced on a predetermined shoulder of the sensing area is detected; A pedestrian judging step of judging that the final object of which the speed is less than the threshold and the moving object on the road is a pedestrian; Further comprising a slow-moving-vehicle determining step of determining a slow-moving final object as a slow-moving vehicle in comparison with other final objects among the final objects, wherein the generating of the unexpected-state generating data comprises: , When the event occurs in at least one of the freezing determination step, the backward movement determination step, the falling object determination step, the illegal state parking determination step, the pedestrian determination step, and the slow vehicle determination step .

In the present invention, the method further includes a communication failure determination step performed after the transmission step. The communication failure determination step may include a first ping-test step, a second ping-test step, an external ping- The method according to claim 1, further comprising: a test step, a failure confirmation data transmission step, a reset target determination step, a local server reset step, a failure confirmation data transmission step, and a controller reset step, Performing a first ping-test to check the network status with the controller of the unexpected condition detector that is in the transmitting step, comparing the detected first ping-data with a preset threshold range, and detecting the detected first ping- The second ping-test step determines that the communication failure with the controller has occurred, and the second ping- Test is performed to check the network status of the control center server, and when the detected second ping-data is compared with the threshold range and the detected second ping-data is out of the threshold range, Test, the control center server performs a third ping-test to check the network status with the local server, and compares the detected third ping-data with a threshold range Determining that a communication failure has occurred with the local server when the detected third ping-data is out of the threshold range, and transmitting the failure confirmation data to the local server when the detected third ping- The control center server transmits failure confirmation data indicating that a communication failure has occurred to the local server, Determining if the local server has failed by the first-ping test step, determining that the unexpected state detector is to be reset, and then proceeding with the failure confirmation data transfer step; and 2) If it is determined that a failure has occurred in the test step, it is determined to be a reset target, and then the local server resetting step is performed; and 3) if the failure confirmation data is received from the control center server through the failure confirmation data transmitting step, (T) during a search period (T) during a search period (T) during a search period (T) during a search period (T) If the communication failure has not occurred in the step, the reset target is determined to be the self, and then the local server resetting step is performed, Wherein the resetting of the local server is performed when the reset target is determined to be the local server by the reset target decision step and resets the local server which is the own server itself, Wherein the controller of the unexpected condition detector proceeds when the controller of the unexpected condition detector determines that the local server transmits the fault confirmation data to the controller of the unexpected condition detector, Upon receipt of the failure confirmation data from the local server, it is preferable to reset the controller which is itself.

delete

delete

delete

delete

delete

According to the present invention having the above-described problems and solutions, objects are detected and tracked using a camera and a thermal camera, whereby detection of objects that can not be detected when the camera is driven by a single camera is enabled, The amount of collected data is enormous and it is possible to improve the accuracy and reliability of the judgment as to whether or not an unexpected event occurs.

According to the present invention, a reference image is determined according to a clock state, and an object of an image other than a reference image is displayed on a reference image to generate an integrated image, thereby efficiently providing information through an image .

According to the present invention, the unexpected situation management unit tracks the trajectory of the final object to judge whether there is a mist, a vehicle accident, freezing, reverse run, falling objects, illegal driving, slow moving vehicles, pedestrians, It is possible to quickly and accurately discriminate an unexpected situation by using vast collection data.

In addition, according to the present invention, all the objects detected by the captured image analyzing unit and the thermal image analyzing unit during the integrated image generation are displayed, and the integrated image is generated so that the temperature of each object is displayed, can do.

1 is a flowchart showing a vehicle information recognition method using a thermal imaging camera disclosed in Korean Patent No. 10-1370627 (entitled "Method and Apparatus for Recognizing Vehicle Information Using a Thermal Camera").
2 is a configuration diagram illustrating an unexpected situation detection system according to an embodiment of the present invention.
3 is a perspective view showing the camera unit of Fig.
4 is a block diagram showing the controller of Fig.
FIG. 5 is a diagram illustrating an example of first objects detected by the captured image analysis unit of FIG. 4. FIG.
FIG. 6 is an exemplary view showing second objects detected by the thermal image analysis unit of FIG. 4;
7 is a block diagram showing the integrated image generating unit of FIG.
8 is a block diagram illustrating the object correction module of FIG.
FIG. 9 is an exemplary view showing an image captured by the camera of the present invention when there is a large amount of fog.
10 is an exemplary view showing a thermal image obtained by the thermal imaging camera of the present invention when there is a large amount of fog.
11 is a block diagram showing the unexpected situation management unit of Fig.
12 is a block diagram illustrating the local server of FIG.
13 is a flowchart for explaining an unexpected situation detection method of the present invention.
FIG. 14 is a flowchart showing the integrated image generating step of FIG.
FIG. 15 is a flowchart for explaining the unexpected situation management step in FIG.
FIG. 16 is a flowchart for explaining the communication failure determination step of FIG. 13;

Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.

2 is a configuration diagram illustrating an unexpected situation detection system according to an embodiment of the present invention.

An unexpected situation detection system (1) according to an embodiment of the present invention collects vast amount of data using a camera and a thermal camera to dramatically increase the diversity of traffic information generation and the unexpected situation detection rate.

Also, as shown in FIG. 2, the unexpected state detection system 1 includes the unexpected state detectors 3-1, ..., and 3-N and the local servers 2-1, (2-N), a control center server (5), and a communication network (10).

The camera 3 includes a controller 310 and a camera 330. The camera 330 includes a camera 331, a thermal camera 333, and a tracking camera 335. [

Although not shown in FIG. 2 for convenience of explanation, a wired / wireless communication network (not shown) may be connected between the local server and the unexpected state detectors 3 assigned to the local server to perform data communication.

Also, the local server 2 determines whether or not a communication failure has occurred with the unexpected state detectors 3 assigned to the local server 2. Then, when it is determined that a communication failure has occurred, And transmits the data.

At this time, the unexpected state detector 3 has two communication modules, and the main communication module is used for data transmission / reception and the auxiliary communication module is used for receiving the failure confirmation data from the local server 2. The reason is that even if the local server 2 recognizes the occurrence of a communication failure in the main communication module of the unexpected state detector 3 and the local server 2 transmits the failure confirmation data to the main communication module in which the communication failure has occurred, In the present invention, when the local server 2 transmits the failure confirmation data to the auxiliary communication module of the corresponding controller, the local server 2 transmits a failure notification It is possible to recognize that it has occurred.

The communication network 10 provides a data movement path between the unexpected state detector 3 and the control center server 5 and may be configured as a wide area network (WAN), a mobile communication network, a wired communication network, LTE or the like.

The control center server 5 transmits the photographed image, the thermal image, and the integrated image (the image in which the photographed image and the thermal image are superimposed and corrected) received from the local servers 2-1, ..., and (2-N) And performs monitoring by utilizing it at the same time.

The control center server 5 also generates traffic information by processing and utilizing traffic related information received from the local servers 2-1, ..., and 2-N, and transmits the generated traffic information to the connected To clients. At this time, the traffic-related information is defined as information related to the vehicle and the traffic condition detected by the unexpected state detector 3, the traffic information is defined as information obtained by processing the traffic-related information so as to be provided to the connected client, The technology and the process of generating the traffic information by the server 5 and providing the generated traffic information to the requested client are commonly used in the intelligent traffic system (ITS), and thus the detailed description thereof will be omitted.

In addition, the control center server 5 receives the unexpected occurrence data indicating that an unexpected situation has occurred by the specific unexpected condition sensor from the local servers 2-1, ..., and 2-N, So that a subsequent response can be made.

The local servers 2-1, ..., 2-N mediate data communication between the unexpected situation detectors 3 and the control center server 5 assigned to them.

Also, the local servers 2-1, ..., and 2-N determine the communication failure of the unintentional situation detectors 3 and their own, (Reboot), so that a quick response to a communication failure can be achieved. At this time, the communication failure determination process and technique of the local server will be described in detail below.

The unexpected state detector 3 includes a camera 331 for photographing a road and acquiring an image (hereinafter referred to as a photographed image), and a thermal imager 333 for measuring the same area as the camera 331 and acquiring a thermal image And detects the occurrence of an unexpected situation by detecting and tracking an object by analyzing the captured image and the thermal image received from the camera unit 330 and determines traffic related information And a controller 310 for generating a control signal.

At this time, the traffic related information includes the vehicle speed, the inter-vehicle distance, the occupancy rate, the freezing information, and the accident information.

3 is a perspective view showing a camera and a thermal imaging camera of the camera unit of Fig. 2;

The camera unit 330 of FIG. 3 has been described to illustrate the present invention, and the configuration and the shape of the camera unit 330 are not limited to FIG.

The camera unit 330 includes a housing 3300 in which electric circuits and elements for driving the camera 331 and the thermal imaging camera 333 are installed.

The camera 331 and the thermal imaging camera 333 are installed inside the housing 3300 and the lenses 3303 and 3304 of the camera 331 and the thermal imaging camera 333 are mounted on the front surface of the housing 3300 (3301). At this time, the camera 331 means a normal visible light camera, and the thermal camera 333 means a normal thermal imaging camera.

The camera 331 and the thermal imaging camera 333 are horizontally spaced from each other on the front surface of the housing 3300 so that the same area can be photographed at the same pan-tilt angle.

The unexpected situation detector 3 is connected to the controller 310 via a wired communication network (not shown) to transmit the captured image and the thermal image acquired by the camera 331 and the thermal imaging camera 333 to the local server 2 do.

3, the camera unit 330 further includes a tracking camera 335. The tracking camera 335 includes a PTZ (Pan-Tilt-Zoom) camera and is connected to the controller 310 of FIG. 4, So as to acquire an enlarged image of an area where accurate image reading is required.

Also, the enlarged image acquired by the tracking camera 335 is input to the local server 2.

Although the controller 310 is separately provided from the camera unit 330 for the convenience of explanation in the present invention, the controller 310 may be installed inside the housing 3300 of the camera unit 330 It is natural that it can be manufactured in one piece.

4 is a block diagram showing the controller of Fig.

4, the controller 310 includes a control unit 30, a memory 31, a communication interface unit 32, a calibration processing unit 33, a shot image analysis unit 34, a thermal image analysis unit An integrated image generating unit 36, an unexpected situation managing unit 37, a tracking camera controlling unit 38, a traffic related information generating unit 39, an auxiliary communication interface unit 40 and a reset processing unit 41.

The control unit 30 is an OS (Operating System) of the controller 310. The control unit 30 controls the control objects 31, 32, 33, 34, 35, 36, 37, , And (39).

When the control unit 30 receives the photographed image and the thermal image from the controller 310 of the unexpected state detector 3, the control unit 30 stores 1) the input photographed image and the thermal image in the memory 31, To the local server 2, and 2) inputs the input photographed image to the calibration processing unit 33. [0041]

The control unit 30 also inputs the calibration information generated by the calibration processing unit 33 to the captured image analysis unit 34 and the thermal image analysis unit 35. [

The control unit 30 also inputs the analysis data detected by the photographed image analysis unit 34 and the thermal image analysis unit 35 to the integrated image generation unit 36. [

The control unit 30 also stores the integrated image generated by the integrated image generating unit 36 in the memory 31 and inputs the integrated image to the integrated image generating unit 36. [

The control unit 30 also inputs the integrated image generated by the integrated image generating unit 36 to the unexpected situation management unit 37 and the traffic related information generating unit 39.

The control unit 30 also stores the traffic related information generated by the traffic related information generating unit 39 and the unexpected situation occurrence data generated by the unexpected situation managing unit 37 in the memory 31, 32 to the local server 2.

The control unit 30 crawls the sub communication interface unit 40 and receives the failure confirmation data from the local server 2 to drive the reset processing unit 40.

Upon receipt of the failure confirmation data from the local server 2, the reset processing unit 40 is driven. At this time, upon receipt of the failure confirmation data through the control unit 30, the reset processing unit 4 resets (reboots) the corresponding controller 310, thereby promptly coping with the communication failure.

The memory 31 stores a predetermined image analysis algorithm and a predetermined thermal image analysis algorithm. At this time, the image analysis algorithm is an algorithm for detecting an object such as a vehicle and a pedestrian from a shot image, and a thermal image analysis algorithm is an algorithm for detecting an object from a thermal image.

In the memory 31, a photographed image photographed by the camera 331 and a photographed image obtained by the thermal imaging camera 333 are temporarily stored.

The enlarged image obtained by the tracking camera 335 is temporarily stored in the memory 31. [

The memory 31 also stores traffic related information generated by the traffic related information generating unit 39 and unexpected situation occurrence data generated by the unexpected situation managing unit 37. [

The memory 31 also stores the communication ID and location information of the corresponding controller 310. [

The communication interface unit 32 connects to a wired / wireless communication network (not shown) and transmits / receives data to / from the local server 2.

The calibration processing unit 33 acquires calibration information defined as relationship information for associating the two-dimensional image of the image with the monitoring space through the input image and the information such as the parameter according to the result of the calibration for the thermal image.

In other words, the calibration processing unit 33 adjusts the three-dimensional object and the three-dimensional space on the basis of the photographed image and the deteriorated image obtained through the cameras 331 and 333, It is possible to model the space similar to the actual space by matching the grid corresponding to the ground by the overlay method and matching the objects existing at different positions of the image image through the adjustment of the three dimensional object, It is possible to check the size, position, and speed of the object, which is similar to the actual value.

The calibration information obtained by the calibration processing section 33 is input to the photographed image analysis section 34 and the thermal image analysis section 35.

FIG. 5 is an exemplary view showing first objects detected by the captured image analyzing unit of FIG. 4, and FIG. 6 is an exemplary view showing second objects detected by the thermal image analyzing unit of FIG.

The photographed image analysis unit 34 receives the photographed image photographed by the camera 331 under the control of the control unit 30 and receives the calibration information from the calibration processing unit 33. [

The photographed image analysis unit 34 also detects a first object such as a vehicle, a pedestrian, and a falling object (object) from the photographed image by referring to the input calibration information while using a predetermined image analysis algorithm.

That is, as shown in FIG. 5, the captured image analyzing unit 34 analyzes the input captured image and detects first objects 101 such as a vehicle, a pedestrian, and a falling object in the image frame 100.

The thermal image analysis unit 35 receives the thermal image acquired by the thermal imaging camera 333 under the control of the control unit 30 and receives the calibration information from the calibration processing unit 33. [

In addition, the thermal image analysis unit 35 detects a second object such as a vehicle, a pedestrian, and a freezing section from a thermal image by referring to the input calibration information while using a predetermined thermal image analysis algorithm.

At this time, if the freezing zone is formed on the road surface, since the thermal image analyzing unit 35 of the present invention detects the object using the temperature measurement, it is possible to detect the freezing zone that can not be detected by the captured image analyzing unit 34 .

6, the thermal image analyzing unit 35 can analyze the input thermal image to detect the second object 201 representing the vehicle, the pedestrian, and the ice in the thermal image frame 200 .

As described above, the photographed image analyzing unit 34 and the thermal image analyzing unit 35 of the present invention can improve detection rate and accuracy because (1) objects such as a vehicle and a pedestrian can be detected in duplicate, and 2) An object such as a frost which can not be detected by the image analysis unit 34 can be detected by the thermal image analysis unit 35 and an object such as a falling object that can not be detected by the thermal image analysis unit 35 is captured Since it can be detected by the analyzing unit 34, it is possible to complement each other's disadvantages and significantly increase the detection rate.

7 is a block diagram showing the integrated image generating unit of FIG.

7, the integrated image generation unit 36 includes a reference image determination module 361, an image matching information generation module 362, an object comparison module 363, an integrated image generation module 364, And a correction module 365.

The reference image determination module 361 determines whether to use the reference image to be used as the basis of the integrated image generated by the integrated image generation module 364 as a photographed image or a thermal image by using a predetermined determination method .

For example, the predetermined determination method can be configured as a first method of determining a reference image as a photographed image when the light amount is equal to or greater than a threshold value, and a reference image as a thermal image if the light amount is less than a threshold value. The system may be configured to detect the sharpness of the photographed image, determine the reference image as the photographed image if the detected sharpness is equal to or greater than the threshold value, and determine the reference image as the thermal image if the sharpness is less than the threshold value.

Also, the reference image determination module 361 inputs the determined reference image information to the integrated image generation module 364.

The image matching information generation module 362 generates image matching information capable of mutually matching the image determined as the reference image and the image not the reference image through the monitoring space by referring to the input calibration information.

At this time, the image matching information is defined as information indicating a relation between the photographed image and the spatial relations of the degradation image.

The object comparison module 363 compares the first objects 101 detected by the captured image analysis unit 34 of FIG. 5 and the second object 101 detected by the thermal image analysis unit 35 of FIG. 6 201). Specifically, the location information of the first objects (101) and the location information of the second objects (201) are compared with each other to determine objects having location information within a threshold range as the same object .

The object comparison module 363 determines the first object 101 as a final object if the object is not detected by the thermal image analysis unit 35 but is detected only by the captured image analysis unit 34 .

Also, the object comparison module 363 determines the second object 201 as a final object if an object that is not detected by the captured image analysis unit 34 but is detected only by the thermal image analysis unit 35 exists.

The integrated image generation module 364 displays the objects determined as the final object by the object comparison module 363 on the reference image determined by the reference image determination module 361, And more specifically, the thermal image of the object is linearly matched to the reference image.

That is, the integrated image generated by the integrated image generation module 364 displays all the objects detected by the captured image analysis unit 34 and the thermal image analysis unit 35 in the reference image, Are displayed together.

8 is a block diagram illustrating the object correction module of FIG.

The object correction module 364 performs a function of correcting the final object of the integrated image generated by the integrated image generation module 364.

The object correction module 364 includes a smoothing module 3641, a resolution correction module 3642, and a mapping module 3643 as shown in FIG.

The smoothing module 3641 performs smoothing on the combined image generated by the integrated image generation module 364 so as to remove the blocking phenomenon generated by the resolution difference between the thermal image of the final object and the photographed image.

The resolution correction module 3642 resolves the problem that the visibility of the thermal image is deteriorated due to the resolution difference by converting the resolution of the thermal image obtained by removing the blocking phenomenon by the smoothing module 3641 into the resolution of the captured image .

The mapping module 3643 maps the final object converted to high resolution by the resolution correction module 3642 to the reference image.

FIG. 9 is an exemplary view showing a captured image obtained by the camera of the present invention when there is a large amount of fog, and FIG. 10 is an exemplary view showing a thermal image acquired by the thermal imaging camera of the present invention when there is a large amount of fog.

As shown in FIG. 9, when there is a large amount of fog, no image is captured in the captured image 100 obtained by the camera 331, and even if the captured image analyzing unit 34 analyzes the captured image, Or a failure occurs in detecting only some objects.

However, as shown in FIG. 10, in the thermal camera, the thermal image analysis unit 35 can analyze the thermal image 200 and detect the second objects 201 in the same environment.

Accordingly, the object comparison module 363 determines the second object detected by the thermal image analysis unit 35 as the final object even when there are many fogs, and when the integrated image generation module 365 determines that each of the final objects is displayed It is possible to generate an integrated image.

11 is a block diagram showing the unexpected situation management unit of Fig.

The unexpected situation management unit 37 analyzes the integrated image generated by the unified image generating unit 36 to determine whether an unexpected situation has occurred, and generates an unexpected condition occurrence data when it is determined that an unexpected situation has occurred. At this time, the unexpected situation occurrence data generated by the unexpected situation management unit 37 is transferred to the management server 5 through the communication interface unit 32 under the control of the control unit 30 and is stored in the memory 31 at the same time.

11, the unexpected situation management unit 37 includes a final object detection module 371, a trajectory tracking module 372, a fog determination module 373, a vehicle accident determination module 374, Module 375, an inverse-movement judging module 376, a falling-object judging module 377, an illegal parking judging module 378, a pedestrian judging module 379, a slow-moving vehicle judging module 380, ).

The final object detection module 371 detects the position information of each of the final objects generated by the integrated image generation unit 36. [

The trajectory tracking module 372 tracks the trajectory of each final object using the position information of each of the final objects detected by the final object detection module 371. [

The fog determination module 373 calculates the quantity N1 of the first objects detected by the photographed image analyzer 34 and the quantity N2 of the second objects detected by the thermal image analyzer 35 When the absolute value of the difference of the calculated quantities is less than the predetermined threshold value, it is determined that the visual field is not secured because there are many fog in the corresponding area.

If it is determined by the mist determination module 373 that fog has occurred in the corresponding area, the unexpected occurrence data generation module 381 is driven under the control of the control unit 30.

The vehicle accident determination module 374 analyzes the trajectory information of each final object detected by the trajectory tracking module 372 to determine whether a vehicle accident has occurred.

At this time, the vehicle accident determination module 374 can determine that a vehicle accident has occurred when the speed of the detected final objects gradually decreases to '0' and the speed of subsequent final objects is stopped.

If the vehicle accident determination module 374 determines that a vehicle accident has occurred, the unexpected condition occurrence data generation module 381 is driven under the control of the control unit 30.

The freezing determination module 375 determines that the temperature of the second objects that can not be detected by the photographed image analysis unit 34 and is detected only by the thermal image analysis unit 35 is below the threshold value and does not move, .

When it is determined that the freezing is formed on the road surface by the freezing determination module 375, the unexpected condition generation data generation module 381 is driven under the control of the control unit 30.

The inverse-travel determining module 376 analyzes the trajectory information detected by the trajectory tracking module 372 and determines that the final object is an inverse vehicle when a final object whose direction of movement is opposite on the same lane is detected.

If an inverse vehicle is detected by the inverse-travel-determination module 376, the unexpected-state occurrence data generation module 381 is driven under the control of the controller 30. [

The falling object judging module 377 analyzes the trajectory information detected by the trajectory tracking module 372 and judges that the falling object falls on the road when the speed of the final object is less than the threshold after the last object which is not present on the previous frame is detected do.

The illegal parking distance determination module 378 analyzes the trajectory information detected by the trajectory tracking module 372 to detect illegal parking vehicles.

In addition, when the illegally parked vehicle is detected by the illegal parking differential determination module 378, the unexpected condition generation data generation module 381 is driven under the control of the control unit 30.

The pedestrian judgment module 379 analyzes the trajectory information detected by the trajectory tracing module 372 to judge whether a pedestrian exists on the road.

At this time, the pedestrian judging module 379 judges that the pedestrian is the final object located on the road whose speed is less than the threshold among the detected final objects.

If a pedestrian is detected by the pedestrian judgment module 379, the unexpected condition occurrence data generation module 381 is driven under the control of the control unit 30.

The slow-moving-vehicle determining module 380 analyzes the sign information detected by the sign tracing module 372 to detect the slow-moving vehicle.

At this time, the slowing-down vehicle judging module 380 judges that the slowest vehicle is a slowing-down vehicle compared with other vehicle objects among the detected final objects.

If the slow vehicle detection module 380 detects the slow vehicle, the unexpected situation occurrence data generation module 381 is driven under the control of the control unit 30.

The unexpected state occurrence data generation module 381 includes a mist determination module 373, a vehicle accident determination module 374, an icing determination module 375, an inverse skid determination module 376, a fall determination module 377, Module 378, the pedestrian judgment module 379, and the slow-moving-vehicle judgment module 380. [0060]

In addition, the incoincidence occurrence data generation module 381 generates incoincidence occurrence data including content information, an integrated image frame, and position information related to an unexpected situation at the time of driving.

In addition, the unexpected condition occurrence data generated by the unexpected condition occurrence data generation module 381 is transmitted to the control center server 5 under the control of the control section 30. [

The tracking camera control unit 38 is driven when the unexpected situation occurrence data is generated by the unexpected situation management unit 37 and controls the PTZ (Pan-Tilt-Zoom) of the tracking camera 335 to a position where an unexpected situation occurs And obtain an enlarged image of an area where an unexpected situation has occurred.

The image obtained by the tracking camera 335 under the control of the tracking camera control unit 38 is stored in the memory 31 and transmitted to the control center server 5 at the same time.

The traffic related information generating unit 39 analyzes the position information of the final objects of the integrated image generating unit 36 to generate traffic related information such as the vehicle speed, the vehicle type, the inter-vehicle distance, the accident information, the occupancy rate, and the congestion rate.

The traffic related information generated by the traffic related information generating unit 39 is transmitted to the control center server 5 under the control of the control unit 30. [

The sub communication interface unit 40 receives the failure confirmation data from the local server 2.

That is, the unexpected state detector 3 of the present invention uses the communication interface unit 32 to transmit data including an image to the local server 2, The communication interface unit 32 can receive a communication failure via the interface unit 40 quickly.

The reset processing unit 41 is driven when the failure confirmation data is received from the local server 2 through the sub communication interface unit 40 and controls the power supply of the corresponding controller 310 to reset the controller 310, .

That is, when the failure control data is received from the local server 2 through the sub communication interface 40, the reset controller 40 resets the corresponding controller 310, thereby promptly responding to a communication failure of the controller 310 .

12 is a block diagram illustrating the local server of FIG.

The local server 2 includes a control module 20 and a memory 21, a communication interface module 22, a first ping-test module 23, a second ping-test module 24, a reset target determination module 25, a reset drive module 26, and an inspection module 27.

When the control module 20 receives data from the controller 310 of the unexpected condition detectors 3 allocated through the communication interface module 22, the control module 20 controls the communication interface module 22 to transmit the received data to the control center server 5).

The control module 20 also drives the reset target determination module 25 when it is determined that the failure has occurred by the first ping-test module 23.

The control module 20 also drives the reset target determination module 25 when it is determined that a failure has occurred by the second ping-test module 24.

When the control module 20 is determined to be one of the unexpected events detectors 3 to which the reset target is assigned by the reset target decision module 25, And controls the communication interface module 22 so that the failure confirmation data is transmitted. When the controller 310 of the unexpected state detector 3 receives the failure confirmation data from the local server 2 through the sub communication interface 40 and drives the reset processing unit 41, Ensure that the fault is quickly restored to a normal state.

The control module 20 also controls the reset drive module 26 to reset the local server 2 when the reset target determination module 25 determines that the reset target is itself.

When the control module 20 receives the failure confirmation data from the control center server 5 through the communication interface module 22, the control module 20 inputs the received failure confirmation data to the reset target determination module 25.

The memory 21 stores a ping-test algorithm for performing a ping-test in the first ping-test module 23 and the second ping-test module 24.

The first ping-test module 23 performs a ping-test to check the network status of the unexpected situation detectors 3 with the controllers 310 using a predetermined ping-test algorithm. At this time, a ping is a Unix command that can test whether or not the other computer is operating normally by sending certain test data to a specific computer. Such a ping-test is a method for checking the network status in a network system, The detailed description will be omitted.

The first ping-test module 23 also compares the detected first ping-data to a predetermined threshold range, and if the detected first ping-data is included in the threshold range, It is determined that a communication failure has not occurred, and if the detected first ping-data is out of the threshold range, it is determined that a communication failure has occurred.

When the first ping-test module 23 determines that a communication failure has occurred, the first ping-test module 23 inputs the determined result value to the reset target determination module 25.

The second ping-test module 24 performs a ping-test to check the network status with the control center server 5 utilizing a predetermined ping-test algorithm.

The second ping-test module 24 also compares the detected second ping-data to a predetermined threshold range, and if the detected second ping-data is included in the threshold range, If the detected second ping-data is out of the threshold range, it is determined that a communication failure has occurred to itself.

When the second ping-test module 24 determines that a communication failure has occurred, the second ping-test module 24 inputs the determined result value to the reset target determination module 25.

The reset target determination module 25 determines that the controller 310 of the corresponding unexpected condition detector 3 is to be reset if it is determined that a failure has occurred from the first ping-test module 23.

At this time, the control module 20 transmits the failure confirmation data to the sub communication interface 40 of the controller 310 determined as the reset target by the reset target determination module 25, When receiving the failure confirmation data from the local server 2 via the network 40, resetting is performed so that a quick response to the communication failure can be made.

Also, when it is determined that a failure has occurred from the second ping-test module 24, the reset target determination module 25 determines the local server 2 as its own to be reset, and drives the reset drive module 26.

That is, if it is determined that the communication failure has occurred by the first ping-test module 23, the reset target determination module 25 determines the reset target to the corresponding pending status detector 93, and if the second ping- 24 determines that a communication failure has occurred, it determines itself as the reset target itself (local server 2).

When receiving the failure confirmation data from the control center server 5 via the communication interface module 21, the reset target determination module 25 determines whether or not the second The ping-test module 24 compares whether or not the failure has been judged to have occurred in itself.

Also, the reset target determination module 25 does not perform any other operation if it is determined that the failure has occurred by the second ping-test module 24 during the search period T, If the second ping-test module 24 has not determined that the failure has occurred, it determines the reset target itself (local server 2).

That is, in the unexpected situation detection system according to the present invention, the ping-test for discriminating the communication failure is performed twice in the local server 2 and the control center server 5 to increase the accuracy and reliability of the communication failure judgment, The local server 2 can not detect the communication failure. However, if the communication failure is detected only in the control center server 5, the accuracy and reliability of the determination of the failure can be increased and, at the same time, It is determined that a temporary load has occurred in the local server 2 when a failure is detected only in the control center server 5 and the local server 2 is reset (rebooted) .

The reset drive module 26 can quickly respond to a communication failure by resetting (rebooting) itself if the reset target determined by the reset target determination module 25 is itself (the local server 2).

The checking module 27 detects whether a fault is continuously generated even after the reset is performed by the reset driving module 26. [

If the failure is repeated after the reset, the check module 27 determines that the failure can not be resolved by the reset and generates the failure persistence data.

At this time, the failure persistence data generated by the check module 27 is transmitted to the control center server 5 under the control of the control module 20, and the control center server 5 receives the failure persistence data from the local server 2 Upon receipt of the transmission, notification information including this information is transmitted to an administrator terminal (not shown) of the maintenance company, thereby enabling quick on-site response when the failure can not be solved only by resetting (rebooting).

The control center server 5 is a server that manages and controls the unexpected state detector 3 and receives the unified state transmitted from the unexpected state detector 3 through the local servers 2-1, Monitoring is performed at the same time as the image is saved.

The control center server 5 also generates traffic information by processing and utilizing traffic related information received from the local servers 2-1, ..., and 2-N, Provide information.

In addition, the control center server 5 monitors the integrated image when it receives an unexpected condition occurrence data from the unexpected condition detector 3, and notifies it to an external cooperative agency (police agency, fire department) .

Also, the control center server 5 periodically performs a ping-test to check the network status of each of the local servers 2-1, ..., and (2-N), and if communication with a specific local server If it is determined that the state is a failure, the failure confirmation data is transmitted to the local server.

13 is a flowchart for explaining an unexpected situation detection method of the present invention.

13, the unexpected state detection method S1 includes a photographing step S10, a calibration step S20, a photographed image analysis step S30, a thermal image analysis step S40, an integrated image generation step S50, an unexpected situation management step S60, a traffic related information generating step S70, a transmission step S80, and a communication failure determination step S90.

The photographing step S10 is a step of acquiring a photographed image through photographing by the visible light camera 331 and acquiring a thermal image using the thermal camera 333.

The calibration step S20 is defined as relation information for associating the two-dimensional image of the image with the surveillance space through the information of the image obtained by the imaging step S10 and the parameter according to the result of the calibration for the thermal image The calibration information is generated.

The captured image analysis step S30 refers to the calibration information generated by the calibration step S20 and simultaneously detects a first object such as a vehicle, a pedestrian, and a falling object (object) from the captured image using a predetermined image analysis algorithm .

The thermal image analysis step S40 refers to the calibration information generated by the calibration step S20 and simultaneously detects a second object such as a vehicle, a pedestrian, and an icemaker from the thermal image using a predetermined thermal image analysis algorithm .

The integrated image generation step S50 is a step of generating an integrated image based on the photographed image and the thermal image.

FIG. 14 is a flowchart showing the integrated image generating step of FIG.

The integrated image generation step S50 includes a reference image determination step S51, an image matching step S52, an object comparison step S53, and an integrated image processing step S54, as shown in FIG.

The reference image determination step S51 determines whether to use the reference image to be used as the basis of the integrated image generated by the integrated image generation module 364 as a photographed image or a thermal image by using a predetermined determination method .

For example, the predetermined determination method can be configured as a first method of determining a reference image as a photographed image when the light amount is equal to or greater than a threshold value, and a reference image as a thermal image if the light amount is less than a threshold value. The system may be configured to detect the sharpness of the photographed image, determine the reference image as the photographed image if the detected sharpness is equal to or greater than the threshold value, and determine the reference image as the thermal image if the sharpness is less than the threshold value.

The image matching step S52 refers to the calibration information generated by the calibration step S20 of FIG. 11 described above, and transmits the reference image, which is determined by the reference image determination step S51, And generates image matching information that can be matched with each other.

At this time, the image matching information is defined as information indicating a relation between the photographed image and the spatial relations of the degradation image.

The object comparison step S53 is a step of comparing the positional information of the first objects detected by the captured image analysis step S30 of FIG. 13 and the second objects detected by the thermal image analysis step S40 Specifically, the position information of the first objects and the position information of the second objects are compared with each other, and the objects having the position information within the critical range are identified as the final object, which is the same object.

Also, the object comparison step S53 is not detected in the thermal image analysis step S40, but if there is an object detected only in the shot image analysis step S30, the object comparison step S53 determines the first object as a final object.

In addition, the object comparison step S53 determines the second object as the final object if the object detected in the thermal image analysis step S40 is not detected in the captured image analysis step S30.

 The integrated image processing step S54 displays the objects determined as the final object by the object comparison step S53 on the reference image determined by the reference image determination step S51, And displaying the temperature value measured by the temperature sensor.

That is, the integrated image generated by the integrated image processing step S54 displays all the objects detected by the captured image analysis step S30 and the thermal image analysis step S40 in the reference image, Are displayed together.

FIG. 15 is a flowchart for explaining the unexpected situation management step in FIG.

13, the unexpected situation management step S60 includes a final object detection step S610, a trajectory tracking step S620, a fog determination step S630, a vehicle accident determination step S640, The pedestrian judgment step S690, the slow-moving vehicle judgment step S700, the unexpected state occurrence data generation step S710, the step S650, the step S660, the fall determination step S670, ).

The final object detection step S610 is a step of detecting position information of each of the final objects determined by the integrated image generation step S50.

The trajectory tracking step S620 is a step of tracking the trajectory of each final object using the positional information of each of the final objects detected by the final object detecting step S610.

The fog determination step S630 is a step of calculating the number N1 of the first objects detected by the captured image analysis step S30 and the number N2 of the second objects detected by the thermal image analysis unit 35 When the absolute value of difference in the calculated quantities is less than a preset threshold value, it is determined that there is a lot of mist in the corresponding area and that the field of view is not ensured.

If it is determined in step S630 that a fog has occurred in the corresponding area, the process proceeds to step S710.

The vehicle accident determination step S640 is a step of determining whether a vehicle accident has occurred by analyzing the trajectory information of each final object detected by the trajectory tracking step S620.

At this time, the vehicle accident determination step S640 may determine that a vehicle accident has occurred when the speed of the detected final objects gradually decreases to '0' and the speed of the subsequent final objects is stopped.

If it is determined in step S640 that a vehicle accident has occurred, the process proceeds to step S710.

In the freezing determination step S650, if the temperature of the second objects that can not be detected in the captured image analysis step S30 and detected only by the thermal image analysis step S40 is less than the threshold value and there is no movement, .

If it is determined in step S650 that freezing is formed on the road surface, step S710 is performed.

Step S660 is a step of analyzing the sign information detected by the trajectory tracking step S620 and determining that the final object is an inverse vehicle when a final object whose direction of movement is opposite on the same lane is detected.

If an inverse vehicle is detected by the inverse travel determination step S660, the process proceeds to an unexpected condition occurrence data generation step S710.

The falling object determining step S670 analyzes the trajectory information detected in the trajectory tracking step S620 and determines that the falling object falls on the road when the speed of the final object is less than the threshold after the final object that is not present on the previous frame is detected .

If a falling object is detected by the falling object determining step S670, the unexpected event generating data generating step S710 is performed.

The illegal parking discrimination step S680 is a step of detecting the illegally parked vehicle by analyzing the sign information detected by the trajectory tracking step S620.

If an illegal parking distance is detected by the illegal parking distance determination step (S680), the unexpected parking distance data generation step (S710) is performed.

The pedestrian judgment step S690 is a step of judging whether a pedestrian exists on the road by analyzing the sign information detected by the sign tracing step S620.

At this time, the pedestrian judgment step S690 judges that the final object among the detected final objects, whose velocity is less than the threshold value and is located on the road, is a pedestrian.

If a pedestrian is detected by the pedestrian determination step S690, the process proceeds to an inconsistency occurrence data generation step 710. [

The slow vehicle determining step S700 is a step of detecting the slow vehicle by analyzing the log information detected by the log tracing step S620.

At this time, the slowing-down vehicle judging step S700 judges that the slowest vehicle is the slowing-down vehicle compared with the other vehicle objects among the detected final objects.

If the slowing vehicle is detected by the slowing vehicle determination step S700, the process proceeds to an unexpected condition generation data generation step 710. [

The step S710 of generating the sudden situation occurrence data may include a step S630 for determining whether a mist is present, a step S640 for determining a vehicle accident, a step S650 for determining a freezing state, a step S650 for determining a freezing state, Step S680, a step of judging a pedestrian (S690), and a step S700 of determining a slow-moving vehicle (S700).

In addition, the step of generating an incoincidence occurrence data (S710) is a step of generating unexpected occurrence data including content information, an integrated image frame, and position information related to an unexpected situation at the time of the process.

Referring back to FIG. 13, the traffic-related information generation step S70 may include generating a traffic-related information using the position of the final objects in the integrated image generation step S50, Related information such as information, occupancy rate, congestion rate, and the like.

The transmission step S80 includes a first transmission step of transmitting the unexpected situation occurrence data or traffic related information generated in the unexpected situation management step S60 or the traffic related information creation step S70 to the local server, And a second transmission step to the external control center server for the unexpected occurrence data or traffic related information transmitted through the transmission step.

FIG. 16 is a flowchart for explaining the communication failure determination step of FIG. 13;

The communication failure judgment step S90 may include a first ping-test step S910, a second ping-test step S920, an external ping-test step S930, Step S940, a reset target determination step S950, a local server reset step S960, a failure confirmation data transmission step S970, and a controller reset step S980.

The first ping-test step S910 performs a first ping-test for checking the network status with the controller of the unexpected state sensor that performed the steps 10 (S10) to 80 (S80) Data is compared with a predetermined threshold range and the detected first ping-data is out of the threshold range, it is determined that a communication failure has occurred with the controller.

The second ping-test step (S910) performs a second ping-test for checking the network status with the external control center server by the local server, compares the detected second ping-data with the threshold range, When the 2 ping-data is out of the critical range, it is determined that a communication failure with the corresponding control center server has occurred.

The external ping-test step S930 is a step in which the control center server performs a third ping-test to check the network status with the local server, compares the detected third ping-data with a threshold range, - If the data is outside the critical range, it is determined that a communication failure with the local server has occurred.

The fault acknowledgment data sending step S940 is a step in which, when it is determined that a communication fault with the local server has occurred due to the external ping-test step S930, the fault acknowledging data indicating that a communication fault has occurred is transmitted to the local server.

In the reset target determination step S950, if the local server determines that a failure has occurred in the first-ping test step S910, the unexpected condition detector is determined as a reset target, and then the failure confirmation data transmission step S970 is performed.

Also, if the local server determines that a failure has occurred in the second ping-test step (S920), the reset target determination step (S950) determines itself as a reset target, and then proceeds to a local server reset step (S960).

In addition, when the failure confirmation data is received from the control center server through the failure confirmation data transmission step S950, the reset target determination step S950 is performed during the search period T between the transmission time point and the predetermined threshold time, If the communication failure has not been generated by the second ping-test step (S920) during the search period (T), it is checked whether the communication failure has occurred by the test step (S920) And proceeds to a local server reset step S960.

The local server reset step S960 is performed when the reset target is determined as the local server by the reset target determination step S950, and resets the local server itself.

The failure confirmation data transmission step S970 is performed when the reset target is determined to be the controller of the unexpected state detector by the reset object determination step S950, and the local server transmits the failure confirmation data to the controller of the unexpected state detector.

The controller reset step S980 is a step of resetting the controller, which is itself, when the controller of the unexpected condition sensor receives the failure confirmation data from the local server through the failure confirmation data transmission step S970.

As described above, the unexpected state detection method (S1) of the present invention performs object detection and tracking using the camera (331) and the thermal imaging camera (333), so that detection of objects It is possible to remarkably increase the detection rate and also to increase the accuracy and reliability of the judgment as to whether or not the occurrence of an unexpected situation occurs because the amount of collected data is enormous.

According to another aspect of the present invention, there is provided a method for detecting an unexpected state (S1), wherein a reference image is determined according to a clock state and an object of an image other than a reference image is displayed on a reference image to generate an integrated image, So that it can be efficiently provided.

In the unexpected state detection method (S1) of the present invention, the unexpected situation management unit (37) tracks the trajectory of the final object to determine whether there is a mist, a vehicle accident, freezing, inversion, falling objects, illegally parked vehicles, It is possible to quickly and accurately discriminate an unexpected situation by utilizing a large amount of collected data by constructing an inconstant situation occurrence data that an unexpected situation has occurred.

In addition, the unexpected state detection method (S1) of the present invention displays all the objects detected by the photographed image analysis unit (34) and the thermal image analysis unit (35) when the integrated image is generated, So that the information through the integrated image can be provided in more detail.

S1: Method of detecting an unexpected situation S10: Shooting step S20: Calibration step
S40: Photographed image analysis step S50: Thermal image analysis step S60: Integrated image generation step
S70: Traffic-related information generation step S80:
S51: Reference image determination step S52: Image matching step S53: Object comparison step
S54: Integrated image processing step S610: Final object detection step
S620: trajectory tracking step S630: determination of whether or not fog occurs
S640: Vehicle accident determination step S650: Freezing judgment step S660: Inverse travel judgment step
S670: SUSPECT SUSPECT SIZE S680: INVALID PURPOSE SURFACE DETERMINATION STEP
S690: Pedestrian judgment step S700: Slow vehicle judging step
S710: Step of generating incoincidence occurrence data
1: Emergency situation detection system 3: Emergency situation detector
5: control center server 10: communication network 30:
31: memory 32: communication interface unit 33: calibration processing unit
34: photographed image analysis section 35: thermal image analysis section 36: integrated image generation section
37: an unexpected situation management unit 38: a traffic related information generating unit

Claims (10)

An imaging step of acquiring a photographed image for a predetermined sensing area using a visible light camera and a thermal image for the sensing area using a thermal imaging camera;
A calibration step of acquiring calibration information which is relationship information for associating the sensed image with the two-dimensional image of the sensed image and the deteriorated image obtained by the sensing step;
Analyzing the photographed image by referring to the calibration information by the calibration step and detecting a first object;
A thermal image analysis step of analyzing the thermal image by referring to the calibration information by the calibration step to detect a second object;
And an integrated image generation step,
The integrated image generation step
A reference image determining step of determining any one of the photographed image and the thermal image as a reference image to be used as a basis of an integrated image by using a predetermined determination method;
An image matching step of generating image matching information by referring to the calibration information, the image matching information being information indicating a relationship between the photographed image and the spatial relations of the degradation images;
Comparing the first objects detected by the captured image analyzing step and the second objects detected by the thermal image analyzing step with the first object having position information within a predetermined threshold range, An object comparison step of determining two objects as the same object and discriminating the object as a final object and determining a first object or a second object that is out of a predetermined threshold range as a final object;
The objects determined as the final object by the object comparison step are displayed on the reference image determined by the reference image determination step and the temperature measured by the thermal image analysis step is displayed on each displayed object to generate an integrated image And an integrated image processing step of detecting an unexpected event. The method according to claim 1, wherein the determination method applied by the reference image determination step determines a reference image as an image to be captured if the light amount is equal to or greater than a threshold value, And a second method of determining a reference image as a photographed image when the detected sharpness is greater than or equal to a second threshold value after detecting the sharpness of the photographed image and determining the reference image as a thermal image if the detected sharpness is less than a second threshold value And detecting the unexpected state. The method according to claim 2, further comprising: an unexpected situation management step performed after the unified image generation step; and a transmission step performed after the unexpected situation management step,
The unexpected situation management step
A final object detection step of detecting a position of each of the final objects displayed in the integrated image generated by the integrated image generation step;
A trajectory tracking step of tracking a trajectory of each final object using positional information of each of the final objects detected by the final object detecting step;
A determining step of determining whether or not an unexpected event occurs using the trajectory information detected by the trajectory tracking step;
Further comprising an unexpected situation occurrence data generation step of generating an unexpected occurrence data including content and location information related to an unexpected situation which is driven when the unexpected situation is determined by the determination step,
The transmitting step may include: a first transmitting step of transmitting the unexpected-state occurrence data generated by the unexpected-state generating data generating step to a local server; and a second transmitting step of transmitting the unexpected- And a second transmission step of transmitting the unexpected information to the control center server. The method of claim 3, wherein the determining step
(N1) of the first objects detected by the captured image analyzing step and the quantity (N2) of the second objects detected by the thermal image analyzing step are calculated and then the absolute value of the difference of the calculated quantities is set Determining whether a fog exists in the detection area when the amount of fog is less than a threshold value;
A vehicle accident determination step of determining that a vehicle accident occurs when the speed of the final objects detected by the trajectory tracking step is reduced and the speed of any one of the final objects becomes '0';
An icing determining step of determining a final object, which is a second object among the final objects detected only in the thermal image analyzing step, and whose temperature is less than a predetermined threshold and has no motion, as icing;
Determining a final object as an inverse car if the final object is detected in the same lane as the opposite direction;
Determining a final object that is not present in a previous frame as a falling object when the speed is less than a threshold;
An illegal parking discrimination step of judging the final object as an illegal parking space vehicle when a final object displaced on a predetermined shoulder of the sensing area is detected;
A pedestrian judging step of judging that the final object of which the speed is less than the threshold and the moving object on the road is a pedestrian;
Further comprising a slow-moving-vehicle determining step of determining a slow-moving final object as a slow-moving vehicle in comparison with other final objects among the final objects,
The unexpected situation occurrence data generation step
Wherein at least one of the fog determination step, the vehicle accident determination step, the freezing determination step, the inverse movement determination step, the falling object determination step, the illegal state parking determination step, the pedestrian determination step, And generates the unexpected condition occurrence data when the unexpected condition occurrence occurs. 5. The method according to claim 4, wherein the method further comprises a communication failure determination step after the transmission step,
The communication failure determination step may include a first ping-test step, a second ping-test step, an external ping-test step, a failure confirmation data transmission step, a reset target determination step, a local server reset step, Further comprising a reset step,
Wherein the first ping-test step performs a first ping-test to check the network status of the controller of the unexpected state sensor that the local server proceeds from the photographing step to the transmitting step, Data is compared with a predetermined threshold range, and if the detected first ping-data is out of the threshold range, it is determined that a communication failure has occurred with the controller,
The second ping-test step performs a second ping-test for checking the network status with the control center server by the local server, compares the detected second ping-data with a threshold range, If the ping-data is out of the critical range, it is determined that a communication failure with the corresponding control center server has occurred,
The external ping-test step may include performing a third ping-test to check the network status with the local server by the control center server, comparing the detected third ping-data with a threshold range, - If the data is out of the critical range, it is determined that a communication failure has occurred with the local server,
The failure confirmation data transmission step proceeds when it is determined that a communication failure with the local server has occurred due to the external ping-test step, the control center server transmits failure confirmation data indicating that a communication failure has occurred to the local server,
The reset target determination step
1) if the local server determines that a fault has occurred in the first-ping test step, it determines the unexpected state detector as a reset target, and then proceeds to the failure confirmation data transmission step; 2) If it is determined that the failure has occurred by the control center server, it is determined that the failure has occurred, and then, the local server resetting step is performed, and 3) if the failure confirmation data is received from the control center server through the failure confirmation data transmitting step, (T) during a search period (T) during a search period (T) during a search period (T) during a search period (T) If the communication failure has not occurred, the reset target is determined to be the self, and then the local server reset step is performed,
Wherein the resetting of the local server is performed when the reset target is determined to be the local server by the reset target determination step,
Wherein the failure confirmation data transmission step is performed when a reset target is determined as a controller of the unexpected state detector by a publicity reset object determination step and the local server transmits failure confirmation data to the controller of the unexpected state detector,
Wherein the controller reset step resets the self controller when the controller of the unexpected condition sensor receives the failure confirmation data from the local server through the failure confirmation data transmission step. delete delete delete delete delete

Images