KR102381527B1 - System for estimating traffic volume using stereo image and deep learning and method thereof - Google Patents

Abstract

The present invention relates to a traffic volume calculation system and method using stereo images and deep learning, and to generate a learning model by learning the vehicle from optical images, thermal images, or a combination of various types of vehicles, and optical images The left image and right image captured by each stereo camera equipped with a camera and thermal image sensor are applied to the learning model, respectively, to estimate the vehicle within the target area for traffic volume calculation, and the coordinates of each vehicle estimated from the left image and the right image are calculated Through calculating the difference in position or angle between the left image and the right image, an affine transform parameter is determined, and a vehicle that cannot be estimated from the left image or the right image using the determined inequality transform parameter The present invention relates to a system and method capable of increasing the accuracy of traffic volume calculation by enabling precise vehicle detection by additionally detecting or preventing the vehicle from failing to be found depending on weather conditions or shooting environment.

Description

Traffic volume estimation system and method using stereo image and deep learning

The present invention relates to a system and method for calculating traffic volume using stereo images and deep learning, and more particularly, to generate a learning model by learning the vehicle from optical images, thermal images, or a combination thereof. And, by applying the left image and the right image captured by each stereo camera equipped with an optical image camera and a thermal image sensor to the learning model, respectively, to estimate the vehicle in the traffic volume calculation target area, and to estimate the angle estimated from the left image and the right image An affine transform parameter is determined by calculating the difference in position or angle between the left image and the right image through the coordinates of the vehicle, and estimated from the left image or the right image using the determined unequal image transformation parameter To a system and method capable of increasing the accuracy of traffic volume estimation by enabling precise vehicle detection by additionally detecting failed vehicles or preventing vehicles from failing to be found depending on weather conditions or shooting environments.

Recently, as the performance of hardware such as GPU (Graphics Processing Unit), a device responsible for computer graphics acceleration, has greatly improved, objects are detected and objects are detected through deep learning from images collected using a camera. Research on segmentation of the domain is being actively carried out. For example, research on vehicle detection using deep learning and road-captured images, and traffic volume estimation studies based on vehicle detection, are being actively conducted.

As such, deep learning technology is being actively applied to the task of detecting vehicles and people in images under various lighting conditions or estimating the traffic flow. It is now possible to provide essential road environment information to drivers, such as weather data, traffic conditions, and road surface conditions.

In relation to the above-mentioned traffic volume calculation study, according to the traffic volume information provision system of the Ministry of Land, Infrastructure and Transport, a frequent survey method that is widely conducted for all sections for which basic traffic data is deemed necessary, and a vehicle classification survey equipment installed at a specific point for one year There is a regular survey method that measures the vehicle type, direction, and time zone of passing vehicles for 24 hours a day, 365 days a year.

The occasional survey method and the regular survey method include survey equipment such as Automatic Vehicle Classification (AVC), Vehicle Detection System (VDS), Toll Collection System (TCS), and CCTV (Closed Circuit). TeleVision) is used to conduct a human-based survey and investigators are placed at the survey points to conduct a traffic volume survey by vehicle type, direction, and time period.

The CCTV is spreading because it can read qualitative information about traffic, but as the number of CCTVs increases rapidly, problems with the simple monitoring method that rely only on the eyes of the control personnel begin to appear, and the integrated control for CCTV control Since the center manages 46 CCTVs per person on average and it is not easy to manually detect important events, intelligent CCTV technology that recognizes the overall situation through image analysis is essential. Deep learning can be used.

However, when performing the above-mentioned traffic volume estimation study, when a vehicle was detected using a single CCTV image and deep learning, there was a problem in that the recognition rate was lowered due to noise depending on the shooting environment and low spatial resolution.

In addition, when using CCTV images equipped with only an optical image camera, the vehicle may appear in a protective color depending on weather conditions and shooting environment, making it impossible to detect the vehicle. Therefore, it is necessary to introduce an additional sensor to compensate for this.

In addition, when using stereo CCTV and deep learning, since the characteristics of the left and right images may be different, the accuracy of diagnosis or prediction may differ.

Therefore, in the present invention, the left image and the right image captured by a stereo camera equipped with an optical image camera and a thermal image sensor are respectively applied to a preset learning model to estimate the vehicle within the area to be calculated, and from the left image and the right image The asymmetry map transformation parameter is determined by calculating the difference in position or angle between the left image and the right image through the estimated coordinates of each vehicle, and the left image and the right image are not estimated using the determined asymmetric image transformation parameter. This is to suggest a method that can accurately calculate the traffic volume through precise vehicle detection by additionally detecting unsuccessful vehicles or preventing vehicles from being found depending on weather conditions or shooting environment.

Next, the prior art existing in the technical field of the present invention will be briefly described, and then the technical matters that the present invention intends to achieve differently from the prior art will be described.

First, Japanese Patent Registration No. 5105400 (October 12, 2012) relates to a traffic volume measurement method and a traffic volume measurement device, even when the main vehicle is photographed overlaid or there is a luminous body that is easily confused with a headlight in a vehicle at night. The present invention relates to a traffic volume measuring method and a traffic volume measuring device for accurately counting the traffic volume by accurately determining the presence or absence of another vehicle.

That is, the prior art describes a traffic volume measurement method and a traffic volume measurement device for measuring the traffic volume by analyzing an image obtained by photographing a vehicle passing on the road with a camera.

The prior art has some similarities with the present invention in that it can accurately count the amount of traffic by determining the presence or absence of a vehicle in various shooting environments, but the present invention is a stereo camera equipped with an optical image camera and a thermal image sensor. Since the vehicle is estimated by applying the left image and the right image to a preset learning model, respectively, and the vehicle that cannot be estimated from the left image and the right image is additionally detected by applying an asymmetry transformation, the prior art and the present invention are There are significant structural differences.

In addition, Korean Patent No. 1813783 (2017.12.29.) relates to a method and system for measuring traffic volume using computer vision. A method and system for measuring traffic volume using computer vision that designates an area of interest, recognizes and tracks vehicles moving within the designated area of interest using motion history images, and calculates traffic density according to the vehicles will be.

That is, the prior art describes a method and system for measuring the traffic volume of vehicles using computer vision to provide services such as identification of travel time and delay time, speeding vehicle control, accident area identification, and stolen vehicle management.

The prior art has some similarities with the present invention in that it recognizes and tracks vehicles moving within a designated area of interest and calculates traffic density, but the present invention is photographed with a stereo camera equipped with an optical image camera and a thermal image sensor A vehicle is estimated by applying a left image and a right image to a previously set learning model, and a technical configuration is proposed to calculate the traffic volume by additionally detecting a vehicle that could not be estimated from the left image and the right image through an asymmetry transformation. Therefore, there is a clear difference in composition.

The present invention was created to solve the above problems, and by improving the performance of vehicle detection in the area subject to traffic volume calculation by using a stereo camera equipped with an optical image camera and a thermal image sensor and deep learning, blind spots, weather An object of the present invention is to provide a system and method capable of preventing a vehicle from being found depending on conditions, shooting environment, etc., and improving the accuracy of traffic volume calculation through precise vehicle detection.

In addition, the present invention estimates a vehicle by applying the left image and the right image captured by a stereo camera equipped with an optical image camera and a thermal image sensor to a previously set learning model, respectively, and the vehicle estimated from the left image and the right image, respectively Calculate the difference in position or angle between the left image and the right image through the coordinates to determine an asymmetric map transformation parameter, and add a vehicle that could not be estimated from the left image and the right image using the determined asymmetric image transformation parameter Another object is to provide a system and method that can increase the accuracy of traffic volume estimation by enabling detection by

A traffic volume calculation system using a stereo image and deep learning according to an embodiment of the present invention includes: a vehicle estimator for estimating a vehicle from a left image and a right image taken with a stereo camera; an asymmetry image transformation parameter determining unit for determining an asymmetric image transformation parameter by calculating a difference in position or angle between the left image and the right image through the coordinates of each vehicle estimated from the left image and the right image; a vehicle detector for additionally detecting a vehicle that cannot be estimated from the left image and the right image using the determined asymmetry map transformation parameter; and a traffic volume calculation unit that calculates the traffic volume according to a unit time by using the detection result.

In addition, the left image and the right image taken with the stereo camera include an optical image, a thermal image, or a combination thereof, and the vehicle detection unit is a blind spot where vehicles are overlapped through the optical image, thermal image, or a combination thereof. B, it characterized in that it further comprises preventing the vehicle from being detected according to weather conditions including fog, yellow sand and rain, or the shooting environment including backlight.

In addition, the traffic volume calculation system further includes a learning unit that generates a learning model for vehicle estimation by learning the vehicle from optical images, thermal images, or a combination thereof, obtained by photographing various types of vehicles, wherein the vehicle estimation unit includes: By inputting the left image and the right image of the optical image, the thermal image, or a combination thereof taken by the stereo camera into the learning model for vehicle estimation, respectively, the feature points for the vehicle included in the left image and the right image are extracted, and , estimating the vehicle present in the left image and the right image by configuring a bounding box by connecting the extracted feature points.

In addition, the vehicle detector, confirms the coordinates of the bounding box indicating the vehicle estimated from the left image (or right image), and the vehicle estimated from the left image (or the right image) using the determined asymmetry transformation parameter Check the coordinates of the right image (or left image) corresponding to the coordinates of the bounding box representing By displaying a bounding box on the coordinates of the right image (or left image) to determine that the vehicle is located, and by matching the number of vehicles estimated in the left image and the right image, respectively, the vehicle that cannot be estimated in the left image and the right image It is characterized in that it is further detected.

In addition, the traffic volume calculation system transmits the traffic volume calculation information according to the unit time in the specific traffic volume calculation target area calculated by the traffic volume calculation unit to the traffic volume calculation information use terminal including the government, local government, institution, company, or a combination thereof. It characterized in that it further comprises; a traffic volume calculation information providing unit to provide.

In addition, the traffic volume calculation method using a stereo image and deep learning according to an embodiment of the present invention, in the traffic volume calculation system, the vehicle estimation step of estimating the vehicle from the left image and the right image taken with a stereo camera; In the traffic volume calculation system, an asymmetric map transformation parameter determination step of determining the asymmetric map transformation parameter by calculating the difference in position or angle between the left image and the right image through the coordinates of each vehicle estimated from the left image and the right image ; a vehicle detection step of additionally detecting, in the traffic volume calculation system, a vehicle that cannot be estimated from the left image and the right image by using the determined asymmetric map transformation parameter; and a traffic volume calculation step of calculating, in the traffic volume calculation system, the traffic volume according to a unit time by using the detected result.

In addition, the left image and the right image taken with the stereo camera include an optical image, a thermal image, or a combination thereof, and the vehicle detection step is a blind spot where vehicles are overlapped through the optical image, thermal image or a combination thereof. It characterized in that it further comprises preventing the vehicle from being detected according to the terrain, weather conditions including fog, yellow sand and rain, or the shooting environment including backlight.

In addition, the traffic volume calculation method, in the traffic volume calculation system, a learning step of generating a learning model for vehicle estimation by learning the vehicle from optical images, thermal images, or a combination thereof obtained by photographing various types of vehicles; , In the vehicle estimation step, the left image and the right image of the optical image, thermal image, or a combination thereof captured by the stereo camera are respectively input to the learning model for vehicle estimation in the traffic volume calculation system, and the left image and estimating a vehicle existing in the left image and the right image by extracting feature points for a vehicle included in the right image and connecting the extracted feature points to configure a bounding box.

In addition, the vehicle detection step may include: in the traffic volume calculation system, checking coordinates of a bounding box indicating a vehicle estimated from the left image (or right image); In the traffic volume calculation system, the coordinates of the right image (or the left image) corresponding to the coordinates of the bounding box indicating the vehicle estimated from the left image (or the right image) using the determined asymmetry transformation parameter are confirmed. step; and in the traffic volume calculation system, if there is no bounding box indicating the vehicle estimated in the vehicle estimation step in the coordinates of the checked right image (or left image), a bounding box is displayed in the coordinates of the right image (or left image) and determining that the vehicle is located; further comprising, in the traffic volume calculation system, by matching the number of vehicles estimated in the left image and the right image, respectively, additionally detecting a vehicle that was not estimated in the left image and the right image characterized by doing so.

In addition, the traffic volume calculation method includes, in the traffic volume calculation system, the traffic volume calculation information according to the unit time in the specific traffic volume calculation target area calculated in the traffic volume calculation step, including the government, local government, institution, company, or a combination thereof. It characterized in that it further comprises; providing the traffic volume estimation information provided to the traffic volume estimation information using terminal.

As described above, according to the traffic volume calculation system and method using stereo images and deep learning of the present invention, the left and right images captured by the stereo camera equipped with an optical image camera and a thermal image sensor are applied to a preset learning model. Estimate the vehicle within the area to be calculated by applying each, and calculate the difference in position or angle between the left image and the right image through the coordinates of each vehicle estimated from the left image and the right image to determine an asymmetry transformation parameter, , based on this, it is possible to increase the accuracy of traffic volume calculation by enabling precise vehicle detection by additionally detecting a vehicle that cannot be estimated in the left and right images or preventing the vehicle from being found depending on weather conditions or shooting environment. there is

1 is a view showing the overall configuration of a traffic volume calculation system using a stereo image and deep learning according to an embodiment of the present invention.
2 is a view for explaining a traffic volume calculation process using a stereo image and deep learning according to an embodiment of the present invention.
3 is a diagram illustrating in detail the configuration of a traffic volume calculation system according to an embodiment of the present invention.
4 is a diagram for explaining the configuration and operation of a stereo camera according to an embodiment of the present invention.
5 is a diagram for explaining a vehicle estimation process using a stereo image and deep learning according to an embodiment of the present invention.
6 is a view for explaining a vehicle detection process to which a vehicle estimation result using a stereo image and deep learning and an asymmetry map transformation parameter between a left image and a right image are applied according to an embodiment of the present invention.
7 is a flowchart illustrating in detail an operation process of a method for calculating a traffic volume using a stereo image and deep learning according to an embodiment of the present invention.
8 is a flowchart illustrating in detail a process of detecting all vehicles in a left image and a right image according to an embodiment of the present invention.
9 is a diagram illustrating a hardware structure of a traffic volume calculation system according to an embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings, a preferred embodiment of the system and method for calculating the amount of traffic using a stereo image and deep learning of the present invention will be described in detail. Like reference numerals in each figure indicate like elements. In addition, specific structural or functional descriptions for the embodiments of the present invention are only exemplified for the purpose of describing the embodiments according to the present invention, and unless otherwise defined, all terms used herein, including technical or scientific terms They have the same meaning as commonly understood by those of ordinary skill in the art to which the present invention pertains. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present specification. It is preferable not to

1 is a view showing the overall configuration of a traffic volume calculation system using a stereo image and deep learning according to an embodiment of the present invention.

As shown in FIG. 1 , the present invention is configured to include a traffic volume calculation system 100 , a stereo camera 200 , a database 300 , and a traffic volume estimation information using terminal 400 .

The traffic volume calculation system 100 applies the left image and the right image captured by the stereo camera 200 equipped with an optical image camera and a thermal image sensor to the learning model for vehicle estimation learned in advance, respectively, to the target area for calculating the traffic volume Estimate the vehicle within.

That is, when the traffic volume calculation system 100 detects a vehicle using a single optical image camera and a learning model for vehicle estimation, the vehicle may not be recognized due to a blind spot, or due to noise and low spatial resolution according to the shooting environment. Due to this, the recognition rate of the vehicle may be lowered, and when an optical image camera is used, the vehicle appears in a protective color depending on weather conditions and thus it may be impossible to estimate the vehicle. In addition to the optical imaging camera composed of the camera 220 , the stereo camera 200 additionally mounted with the first thermal image sensor 230 and the second thermal image sensor 240 is used.

Accordingly, the performance of vehicle estimation within the area to be estimated for traffic volume is improved, so that a vehicle located at the rear of a blind spot (for example, due to overlapping vehicles is not identified by either optical imaging camera or thermal imaging sensor, but by other optical imaging cameras or It is possible to prevent the vehicle from being found depending on the thermal image sensor), weather conditions, and shooting environment.

At this time, the learning model for vehicle estimation is generated by learning the vehicle from a plurality of optical images, thermal images, or a combination of various types of vehicles, and the traffic volume calculation system 100 continuously collects related data can be updated.

For the above learning, it is preferable to use a CNN (Convolutional Neural Network) learning network optimized for image learning, but it is not limited thereto. In addition, various learning networks such as ANN (Artificial Neural Network) and DCNN (Deep CNN) can be used. there is.

In addition, the traffic volume calculation system 100 estimates the vehicle from the left image and the right image captured by the stereo camera 200 through the vehicle estimation learning model, and then the coordinates of each vehicle estimated from the left image and the right image By calculating the difference in position or angle between the left image and the right image through to determine the asymmetric image transformation parameter, and based on the determined asymmetric image transformation parameter, additionally detect a vehicle that has not been detected in the left image and the right image. The number of vehicles in the left image and the right image are matched, and based on this, the traffic volume of vehicles passing through the traffic volume calculation target area is calculated.

That is, when the stereo camera 200 and the learning model for vehicle estimation are used, the left image and the right image captured by the stereo camera 200 may have different image characteristics, so the accuracy of diagnosis or prediction may differ. Therefore, in the present invention, the asymmetry transformation parameter is used to detect all the vehicles in the traffic volume calculation target area by complementing them. Accordingly, it is possible to additionally detect a vehicle that has not been estimated in the left image and the right image, thereby increasing the accuracy of calculating the traffic volume.

Here, the affine transform is a function of the space difference between the left image and the right image photographed in the same space through the stereo camera 200 . That is, the asymmetry transformation is a transformation that makes one vector space correspond to another vector space and preserves the ratio of the straight line to the distance, and scales and rotates the left image and the right image captured by the stereo camera 200 . It is a linear transformation such as rotation or translation.

In addition, the traffic volume calculation system 100 may provide the traffic volume calculation information calculated in a specific traffic volume calculation target area to the traffic volume calculation information using terminal 400 through a network.

At this time, the network supports various wired and wireless communication methods such as the Internet, WiFi, 4G, 5G, LTE, etc., thereby transmitting and receiving image data between the traffic volume calculation system 100 and the stereo camera 200, and the traffic volume It enables transmission and reception of data on traffic volume estimation information between the calculation system 100 and the traffic volume estimation information use terminal 400 .

On the other hand, the traffic volume calculation system 100 is configured as an independent platform, and based on the left and right images captured by the plurality of stereo cameras 200, it was described as an example to calculate the traffic volume in each traffic volume calculation target area. , it is revealed that it is possible to calculate the amount of traffic within the corresponding traffic amount calculation target area by being integrally configured with the stereo camera 200 .

The stereo camera 200 is an optical image camera, and a first camera 210 and a second camera 220 for generating a left image and a right image by photographing a traffic volume calculation target area, and a column for the left image and the right image It includes a first thermal image sensor 230 and a second thermal image sensor 240 for generating images, and transmits optical images or thermal images of the left image and right image to the traffic volume estimation system 100 through a network. send.

That is, when only a single optical image camera is used, the stereo camera 200 may not be able to detect a vehicle depending on shooting conditions such as backlight or weather conditions such as fog, yellow sand and rain. In order to prevent this, at least By having two or more optical imaging cameras and thermal imaging sensors, it is possible to accurately detect a vehicle moving in the area subject to the traffic volume calculation (see FIG. 4).

The database 300 is a learning model DB 310 that stores and manages the learning model for vehicle estimation generated by the traffic volume calculation system 100, and collects from each stereo camera 200 in the traffic volume calculation system 100 Image data DB 320 that stores and manages optical images, thermal images, or a combination thereof, and the amount of traffic performed through the vehicle detection result confirmed from the optical image, thermal image, or a combination thereof in the traffic volume calculation system 100 It is configured to include a traffic volume calculation information DB 330 that stores and manages the calculation information.

In addition, although not shown in the drawing, the database 300 may store and manage information (eg, identification information, installation location, etc.) of each stereo camera 200 installed in each region.

The traffic volume calculation information use terminal 400 is a communication terminal such as a smart phone, tablet, PC, etc. used by an administrator or individual user belonging to the government, local government, institution, company, etc., performed in the traffic volume calculation system 100 The traffic volume calculation information of each traffic volume calculation target area is provided from the traffic volume calculation system 100 through the network and is used.

Next, a process of detecting a vehicle and calculating a traffic volume using a stereo image and deep learning in a traffic volume calculation system according to an embodiment of the present invention will be described in more detail with reference to FIG. 2 .

2 is a view for explaining a traffic volume calculation process using a stereo image and deep learning according to an embodiment of the present invention.

As shown in FIG. 2 , the traffic volume calculation system 100 of the present invention generates a learning model for vehicle estimation and stores it in the learning model DB 310 (①). That is, the traffic volume calculation system 100 collects a plurality of optical images, thermal images, or a combination of these images of various types of vehicles, and learns the various types of vehicles from the collected image data. The learning model for estimation is stored and managed in the learning model DB 310 .

In this way, in the state where the learning model for vehicle estimation is created, stored in the database 300 and managed, the traffic volume calculation system 100 is an optical image, a thermal image, or these A left image and a right image are collected for a combination of , and each vehicle is estimated from the left image and the right image by using the collected left image and right image as an input of the learning model for vehicle estimation (②).

Next, the traffic volume calculation system 100 calculates the position or angle difference between the left image and the right image through the coordinates of each vehicle estimated from the left image and the right image to determine an asymmetry transformation parameter (③).

In addition, the traffic volume calculation system 100 additionally detects a vehicle in the left image and the right image using the determined asymmetric map transformation parameter (④). That is, vehicles that have not been detected in the left and right images are additionally detected to match the number of vehicles in the left and right images (refer to FIG. 6).

In addition, the traffic volume calculation system 100 uses the result of the additional detection in step ④, taking into consideration the average speed of the vehicle passing through the traffic volume calculation target area, the vehicle transit time, etc. , 15 minutes, 5 minutes, etc.) calculates the vehicle traffic volume, and stores the calculated traffic volume calculation information in the traffic volume calculation information DB 330 (⑤).

The traffic volume calculation information for each traffic volume calculation target area calculated in this way is transmitted through the network at the request of the manager or individual user belonging to the government, local government, institution, company, etc. (400).

3 is a diagram illustrating in detail the configuration of a traffic volume calculation system according to an embodiment of the present invention.

As shown in FIG. 3 , the traffic amount calculation system 100 determines the learning data collection unit 110 , the learning unit 120 , the image data receiving unit 130 , the vehicle estimation unit 140 , and the asymmetry mapping parameter. It is configured to include a unit 150 , a vehicle detection unit 160 , a traffic quantity calculation unit 170 , a traffic quantity estimation information providing unit 180 , a memory 190 , and the like.

In addition, although not shown in the drawing, the traffic volume calculation system 100 includes a power supply unit for supplying operating power to each component, an input unit for data input for various functions, an update management unit for managing updates of various operation programs, and the traffic calculation The system 100 may further include a control unit that collectively controls the operation of each component.

The learning data collection unit 110 collects image data including a plurality of optical images, thermal images, or a combination thereof of various types of vehicles.

The learning unit 120 learns the vehicle from image data including a plurality of optical images, thermal images, or a combination thereof obtained by photographing various types of vehicles collected by the learning data collection unit 110 to learn vehicle estimation. A model is generated, and the generated learning model for vehicle estimation is stored and managed in the learning model DB 310 .

Meanwhile, the learning data collection unit 110 may continuously collect image data required for the vehicle estimation learning model, and the learning unit 120 updates the vehicle estimation learning model based on the collected image data. can do.

The image data receiving unit 130 collects the left and right images of the optical image, thermal image, or a combination thereof taken by the stereo camera 200, and collects the collected left and right images by each stereo camera ( 200) and stored in the image data DB 320 , and the collected left and right images are provided to the vehicle estimator 140 .

The vehicle estimator 140 estimates a vehicle from the left image and the right image captured by the stereo camera 200 received from the image data collection unit 130 , and estimates of each vehicle estimated from the left image and the right image. The estimation result is output to the disequiangular map transformation parameter determining unit 150 .

That is, the vehicle estimator 140 inputs the left image and the right image of the optical image, the thermal image, or a combination thereof captured by the stereo camera 200 to the learning model for vehicle estimation, respectively, and the left image and extracting a feature point for a vehicle included in the right image, connecting the extracted feature point to configure a bounding box, and estimating a vehicle existing in the left image and the right image.

The asymmetric map transformation parameter determining unit 150 calculates a difference in position or angle between the left image and the right image through the coordinates of each vehicle in the left image and the right image estimated by the vehicle estimator 140, An asymmetry image transformation parameter is determined, and the determined asymmetric image transformation parameter is output to the vehicle detection unit 160 .

The vehicle detection unit 160 additionally detects a vehicle that has not been estimated by the vehicle estimating unit 140 by using the disequiangular map transformation parameter determined by the asymmetric image transformation parameter determining unit 150, and the addition The information on the vehicle detected as , is output to the traffic amount calculating unit 170 .

That is, the vehicle detection unit 160 is a blind spot where vehicles are overlapped in the image data including the optical image, thermal image, or a combination thereof, weather conditions including fog, yellow sand and rain, or shooting environment including backlight. This is to prevent the vehicle from being detected.

To this end, the vehicle detector 160 checks the coordinates of a bounding box indicating a vehicle estimated from the left image (or right image) estimated through the vehicle estimator 140 , and then the amorphous map transformation parameter The coordinates of the right image (or the left image) corresponding to the coordinates of the bounding box indicating the vehicle estimated from the left image (or the right image) are checked using the asymmetric map transformation parameter determined by the determining unit 150 .

Next, when there is no bounding box indicating the vehicle estimated by the vehicle estimator 140 in the coordinates of the checked right image (or left image), the vehicle detection unit 160 determines the corresponding right image (or left image). ) to determine that the vehicle is located by displaying a bounding box on the coordinates of, and by matching the number of vehicles estimated in the left and right images in this way, additionally detects vehicles that were not estimated in the left and right images. .

The traffic volume calculation unit 170 uses the vehicle detection results in the left and right images detected by the vehicle detection unit 160, taking into consideration the average vehicle speed and vehicle transit time passing through the traffic volume calculation target area. The traffic volume is calculated for each unit time, and the calculated traffic volume calculation information is stored and managed in the traffic volume calculation information DB (330).

The traffic volume calculation information providing unit 180 provides the traffic volume calculation information according to the unit time in the specific traffic volume calculation target area calculated by the traffic volume calculation unit 170, including the government, local governments, institutions, companies, or a combination thereof. It is provided to the traffic volume calculation information using terminal 400 .

The memory 190 stores various operation programs used in the traffic volume calculation system 100 , and a left image and a right image including an optical image, a thermal image, or a combination thereof received by the image data receiving unit 130 . , information on the vehicle estimated by the vehicle estimator 140 , the asymmetric map transformation parameter determined by the asymmetric map transformation parameter determiner 150 , additional vehicle information detected by the vehicle detection unit 160 , the It performs a function of temporarily storing the traffic volume calculation information, etc. performed by the traffic volume calculation unit 170 .

4 is a view for explaining the configuration and operation of a stereo camera according to an embodiment of the present invention.

As shown in FIG. 4, the stereo camera 200 according to an embodiment of the present invention is installed on the road of the traffic volume estimation target area, and as an optical image camera, generates a left image by photographing the traffic volume calculation target area. 1 camera 210, a second camera 220 that generates a right image by photographing a traffic volume calculation target area, and a first thermal image that is located in front of the first camera 210 to obtain a thermal image of the left image It consists of a sensor 230 and a second thermal image sensor 240 positioned in front of the second camera 220 to obtain a thermal image of a right image.

At this time, it will be described that the first camera 210 generates a left image as a left camera, and the second camera 220 captures a right image as a right camera for convenience according to a location in the drawing.

The area photographed by the first camera 210 is a solid triangle portion, and the area photographed by the second camera 220 is a dotted triangle portion.

The stereo camera 200 is a left image and a right image captured using the first camera 210 and the second camera 220 in good weather conditions (ie, normal) based on the control of the controller (not shown). is transmitted to the traffic volume calculation system 100 through the network, and when the weather conditions such as fog, yellow sand, rain, etc. deteriorate or the shooting environment such as backlight is not suitable, the first thermal image sensor 230 and the second column Thermal images of the left image and the right image are transmitted through the image sensor 240 to the traffic volume calculation system 100 through a network.

5 is a diagram for explaining a vehicle estimation process using a stereo image and deep learning according to an embodiment of the present invention.

As shown in FIG. 5 , the traffic volume calculation system 100 estimates each vehicle by applying the left image and the right image received from the stereo camera 200 through the network to the vehicle estimation learning model.

At this time, as shown in FIG. 5 ( a ), the traffic volume calculation system 100 is a left image taken by the first camera 210 of the stereo camera 200 and a right image taken by the second camera 220 , as shown in FIG. 5 ( a ). Each vehicle is estimated by applying each image to the learning model for vehicle estimation.

In addition, as shown in (b) of FIG. 5, when the weather conditions such as fog, yellow dust, and rain deteriorate (or when the shooting environment such as backlight is not suitable), the stereo camera ( 200), the thermal image of the left image generated by the first thermal image sensor 230 and the thermal image of the right image generated by the second thermal image sensor 240 are respectively applied to the learning model for vehicle estimation. to estimate each vehicle.

At this time, the estimation result of the vehicle performed by the traffic volume calculation system 100 appears in the form of a bounding box, and the estimation result for the vehicle within the traffic volume calculation target area in the left image and the right image according to conditions such as blind spots, weather conditions, and shooting environment may be different. That is, the number of vehicles estimated from the left image and the right image does not match or the accuracy is low. Accordingly, it is necessary to supplement the vehicle estimation results in the left and right images.

6 is a view for explaining a vehicle detection process to which a vehicle estimation result using a stereo image and deep learning and an asymmetry map transformation parameter between a left image and a right image are applied according to an embodiment of the present invention.

As shown in FIG. 6 , in order to supplement the vehicle estimation results in the left image and the right image described in FIG. 5 , the traffic volume calculation system 100 indicates a vehicle estimated from the left image (or the right image). Check the coordinates of the box, and the coordinates of the right image corresponding to the coordinates of the vehicle identified in the left image using the asymmetric map transformation parameter that calculates the position or angle difference between the left image and the right image through the coordinates of each vehicle. Check it.

Next, if there is no bounding box indicating that the vehicle is estimated at the coordinates of the confirmed right image (or left image), the traffic volume calculation system 100 shows the blue (or red) color shown in (a) and (b) of FIG. 6 . ) Determining that the vehicle is located by displaying a bounding box on the corresponding coordinates as shown by the arrow, repeating this method to match the number of vehicles estimated in the left and right images, respectively, and the vehicle that could not be estimated in the left and right images to detect additionally.

At this time, Fig. 6 (a) is an embodiment showing an ordinary stereo image, a vehicle estimation result using deep learning, and additional detection by applying an asymmetry map transformation parameter, and Fig. 6 (b) is a stereo image and a stereo image at the time of fog generation. It is an embodiment showing the vehicle estimation result using deep learning and additional detection by applying the disequiangular map transformation parameter.

Next, an embodiment of a method for calculating a traffic volume using a stereo image and deep learning according to the present invention configured as described above will be described in detail with reference to FIGS. 7 and 8 . In this case, the order of each step according to the method of the present invention may be changed by the environment of use or by a person skilled in the art.

7 is a flowchart illustrating in detail the operation process of a method for calculating a traffic volume using a stereo image and deep learning according to an embodiment of the present invention, and FIG. 8 is all vehicles in the left image and the right image according to an embodiment of the present invention. It is a flowchart showing the process of detecting in detail.

First, the traffic volume calculation system 100 collects image data including a plurality of optical images, thermal images, or a combination thereof taken of various types of vehicles (S100), and the collected plurality of optical images, thermal images, or A learning step of generating a learning model for vehicle estimation by learning the vehicle from image data including a combination thereof is performed (S200).

In the state that the vehicle estimation learning model is generated and stored in the database 300 through the step S200, the traffic volume calculation system 100 receives the left image and the right image captured by the stereo camera 200 (S300) ). That is, a left image and a right image including an optical image, a thermal image, or a combination thereof are received from the stereo camera 200 .

Next, the traffic volume calculation system 100 applies the left image and the right image received in the step S300 to the learning model for vehicle estimation generated in the step S200, respectively, a vehicle for estimating each vehicle from the left image and the right image An estimation step is performed (S400).

That is, the traffic volume calculation system 100 inputs the left image and the right image of the optical image, the thermal image, or a combination thereof received from the stereo camera 200 to the learning model for vehicle estimation, respectively, and the left image and By extracting feature points for a vehicle included in the right image and connecting the extracted feature points to form a bounding box, vehicles present in the left image and the right image are estimated.

In addition, the traffic volume calculation system 100 calculates the difference in position or angle between the left image and the right image through the coordinates of each vehicle estimated from the left image and the right image through the step S400 to determine the asymmetric map transformation parameter. to perform the step of determining the disequiangular map transformation parameter (S500).

Next, the traffic volume calculation system 100 performs a vehicle detection step of additionally detecting a vehicle that has not been estimated in the left image and the right image by using the asymmetric map transformation parameter determined in the step S500 (S600).

That is, the traffic volume calculation system 100 additionally detects a vehicle that cannot be estimated in the left image and the right image, so that the vehicle is overlapped through the optical image, thermal image, or a combination thereof received from the stereo camera 200 It is to prevent a vehicle from being detected in a blind spot, weather conditions including fog, yellow sand and rain, or a shooting environment including backlight.

When the step S600 is described in more detail with reference to FIG. 8, the traffic volume calculation system 100 checks the coordinates of the bounding box indicating the vehicle estimated in the left image (or right image) (S610), and the S500 The coordinates of the right image (or the left image) corresponding to the coordinates of the bounding box indicating the vehicle estimated from the left image (or the right image) are checked using the asymmetric map transformation parameter determined in the step (S620).

In addition, if there is no bounding box indicating the vehicle estimated in step S400 in the coordinates of the right image (or left image) checked in step S620, the traffic volume calculation system 100 is the coordinates of the right image (or left image). It is determined that the vehicle is located by displaying a bounding box (S630), and by matching the number of vehicles estimated in the left image and the right image, respectively, a vehicle that is not estimated in the left image and the right image is additionally detected (S640).

Referring back to FIG. 7 , the traffic volume calculation system 100 performs a traffic volume calculation step of calculating the traffic volume according to a unit time using the result detected in step S600 ( S700 ).

In addition, the traffic volume calculation system 100 determines whether the traffic volume calculation operation is finished (S800), and repeats steps S300 to S700 until the traffic volume calculation operation ends.

On the other hand, although not shown in the drawing, the traffic volume calculation system 100 calculates the traffic volume calculation information according to the unit time in the specific traffic volume calculation target area calculated in step S700, the government, local government, institution, company, or a combination thereof. The step of providing traffic volume estimation information provided to the terminal 400 using traffic volume estimation information including

9 is a diagram illustrating a hardware structure of a traffic volume calculation system according to an embodiment of the present invention.

As shown in FIG. 9 , the hardware structure of the traffic volume calculation system 100 includes a central processing unit 1000 , a memory 2000 , a user interface 3000 , a database interface 4000 , a network interface 5000 , It is configured to include a web server 6000 and the like.

The user interface 3000 provides input and output interfaces to the user by using a graphical user interface (GUI).

The database interface 4000 provides an interface between a database and a hardware structure. The network interface 5000 provides a network connection between devices owned by a user.

The web server 6000 provides a means for a user to access the hardware structure via a network. Most users can remotely access the web server and use the traffic volume calculation system 100 .

Each step of the above-described configuration or method may be implemented as computer-readable code on a computer-readable recording medium or transmitted through a transmission medium. The computer-readable recording medium is a data storage device capable of storing data that can be read by a computer system.

Examples of computer-readable recording media include, but are not limited to, databases, ROMs, RAMs, CD-ROMs, DVDs, magnetic tapes, floppy disks, and optical data storage devices. The transmission medium may include a carrier wave transmitted through the Internet or various types of communication channels. In addition, the computer readable recording medium may be distributed through a network coupled computer system so that the computer readable code is stored and executed in a distributed manner.

In addition, at least one or more components applied to the present invention may include or be implemented by a processor such as a central processing unit (CPU), a microprocessor, etc. that perform each function, and two or more of the components are one single It may be combined as a component to perform any operation or function for two or more combined components. In addition, a part of at least one or more components applied to the present invention may be performed by other components among these components. In addition, communication between the components may be performed through a bus (not shown).

As such, the present invention estimates a vehicle within a traffic volume calculation target area by applying the left image and the right image captured by a stereo camera equipped with an optical image camera and a thermal image sensor to a preset learning model, respectively, and the left image and the right image An asymmetry map transformation parameter is determined by calculating the difference in position or angle between the left image and the right image through the coordinates of each vehicle estimated above, and based on this, additionally detecting a vehicle that is not estimated in the left image and the right image, or By preventing the vehicle from being found depending on the weather conditions or the shooting environment, it is possible to accurately detect the vehicle and increase the accuracy of the calculation of the traffic volume.

As described above, the present invention has been described with reference to the embodiments shown in the drawings, which are merely exemplary, and those of ordinary skill in the art can make various modifications and equivalent other embodiments therefrom. You will understand that it is possible. Therefore, the technical protection scope of the present invention should be judged by the following claims.

100: traffic volume calculation system 110: learning data collection unit
120: learning unit 130: image data receiving unit
140: vehicle estimating unit 150: non-conformal map transformation parameter determining unit
160: vehicle detection unit 170: traffic volume calculation unit
180: traffic calculation information providing unit 190: memory
200: stereo camera 210: first camera
220: second camera 230: first thermal image sensor
240: second thermal image sensor 300: database
310: learning model DB 320: image data DB
330: traffic volume estimation information DB 400: traffic volume estimation information use terminal

Claims (10)

a vehicle estimator for estimating a vehicle from a left image and a right image taken with a stereo camera;
an asymmetry image transformation parameter determining unit for determining an asymmetric image transformation parameter by calculating a difference in position or angle between the left image and the right image through the coordinates of each vehicle estimated from the left image and the right image;
a vehicle detector for additionally detecting a vehicle that cannot be estimated from the left image and the right image by using the determined asymmetric image transformation parameter; and
Traffic volume calculation system using stereo image and deep learning, characterized in that it comprises; The method according to claim 1,
The left and right images taken with the stereo camera are,
Including optical imaging, thermal imaging, or a combination thereof,
The vehicle detection unit,
Through the optical image, thermal image, or a combination thereof, an unestimated vehicle is added to the left or right image depending on the blind spot where vehicles are overlapped, weather conditions including fog, yellow sand and rain, or the shooting environment including backlight A traffic volume estimation system using stereo images and deep learning, characterized in that it is detected with The method according to claim 1,
The traffic volume calculation system,
It further includes; a learning unit that generates a learning model for vehicle estimation by learning the vehicle from optical images, thermal images, or a combination of various types of vehicles,
The vehicle estimation unit,
By inputting the left image and the right image of the optical image, the thermal image, or a combination thereof taken by the stereo camera into the learning model for vehicle estimation, respectively, the feature points for the vehicle included in the left image and the right image are extracted, and , By connecting the extracted feature points to configure a bounding box, the system for calculating the amount of traffic using stereo image and deep learning, characterized in that it further comprises estimating the vehicle present in the left image and the right image. The method according to claim 1,
The vehicle detection unit,
Check the coordinates of the bounding box indicating the vehicle estimated in the left image (or right image),
Check the coordinates of the right image (or the left image) corresponding to the coordinates of the bounding box indicating the vehicle estimated from the left image (or the right image) using the determined asymmetric map transformation parameter,
If there is no bounding box indicating the vehicle estimated by the vehicle estimation unit in the coordinates of the checked right image (or left image), the vehicle is determined to be located by displaying a bounding box in the coordinates of the right image (or left image),
Traffic volume calculation system using stereo image and deep learning, characterized in that by matching the number of vehicles estimated in the left image and the right image, respectively, to additionally detect a vehicle that is not estimated in the left image and the right image. The method according to claim 1,
The traffic volume calculation system,
Traffic volume calculation information providing unit that provides traffic volume calculation information according to unit time in a specific traffic volume calculation target area calculated by the traffic volume calculation unit to a terminal using traffic volume calculation information including the government, local governments, institutions, companies, or a combination thereof ; Traffic volume calculation system using stereo images and deep learning, characterized in that it further comprises. A vehicle estimation step of estimating a vehicle from a left image and a right image taken with a stereo camera in the traffic volume calculation system;
In the traffic volume calculation system, an asymmetry transformation parameter determination step of determining the asymmetry transformation parameter by calculating the difference in position or angle between the left image and the right image through the coordinates of each vehicle estimated from the left image and the right image ;
a vehicle detection step of additionally detecting, in the traffic volume calculation system, a vehicle that has not been estimated in the left image and the right image by using the determined asymmetric map transformation parameter; and
The traffic volume calculation method using stereo image and deep learning, comprising: in the traffic volume calculation system, a traffic volume calculation step of calculating the traffic volume according to a unit time using the detected result. 7. The method of claim 6,
The left and right images taken with the stereo camera are,
Including optical imaging, thermal imaging, or a combination thereof,
The vehicle detection step is
Through the optical image, thermal image, or a combination thereof, an unestimated vehicle is added to the left or right image depending on the blind spot where vehicles are overlapped, weather conditions including fog, yellow sand and rain, or the shooting environment including backlight A method of calculating traffic volume using stereo images and deep learning, characterized in that it is detected with 7. The method of claim 6,
The traffic volume calculation method is,
In the traffic volume calculation system, a learning step of generating a learning model for vehicle estimation by learning the vehicle from optical images, thermal images, or a combination thereof, obtained by photographing various types of vehicles;
The vehicle estimation step is
In the traffic volume calculation system, the left image and the right image of the optical image, the thermal image, or a combination thereof captured by the stereo camera are respectively input to the learning model for estimating the vehicle, and the vehicle included in the left image and the right image Method for calculating traffic volume using stereo image and deep learning, characterized in that it further comprises estimating the vehicle present in the left image and the right image by extracting the feature points for . 7. The method of claim 6,
The vehicle detection step is
checking, in the traffic volume calculation system, coordinates of a bounding box indicating a vehicle estimated from the left image (or right image);
In the traffic volume calculation system, the coordinates of the right image (or left image) corresponding to the coordinates of the bounding box indicating the vehicle estimated from the left image (or right image) using the determined asymmetry transformation parameter are confirmed step; and
In the traffic volume calculation system, when there is no bounding box indicating the vehicle estimated in the vehicle estimation step in the coordinates of the checked right image (or left image), the bounding box is displayed in the coordinates of the right image (or left image) Determining that the vehicle is located; further comprising,
In the traffic volume calculation system, by matching the number of vehicles estimated in the left image and the right image, respectively, the traffic volume using stereo image and deep learning, characterized in that the vehicle not estimated in the left image and the right image is additionally detected calculation method. 7. The method of claim 6,
The traffic volume calculation method is,
In the traffic volume calculation system, the traffic volume calculation information according to the unit time in the specific traffic volume calculation target area calculated in the traffic volume calculation step is provided to the traffic volume calculation information use terminal including the government, local government, institution, company, or a combination thereof A traffic volume estimation method using stereo images and deep learning, characterized in that it further comprises; providing traffic volume estimation information.

Images