KR20210133827A - Apparatus for learning deep learning model and method thereof - Google Patents

Abstract

Disclosed are a device for learning a deep learning model and a method thereof in accordance with an embodiment. The device for learning a deep learning model comprises: an image acquisition unit for acquiring an image; a model learning unit for constructing image big data for learning based on a reference image obtained from the image acquisition unit, extracting a learning target object from the reference image of the constructed image big data for learning, and learning a deep learning model based on the extracted learning target object; an object inference unit for inferring a target object in a surveillance image provided from the image acquisition unit based on the learned deep learning model and generating accidental situation data with the inferred result; and a false detection definition unit for classifying the generated accidental situation data into right detection data and false detection data and defining the classified false detection data as at least one piece of object false detection data for a predetermined false detection target object. The model learning unit re-learns the deep learning model by adding the defined at least one object false detection data to the image big data for learning. In accordance with an embodiment, the object recognition performance can be improved without hindering learning of existing labeling data.

Description

Apparatus and method for training a deep learning model

The embodiment relates to an apparatus and method for training a deep learning model using false-positive data.

Learning and inference of a deep learning model decomposes videos containing target objects into still images, and then uses all of the labeling data to learn without dividing the target objects into inference and learning data as data for labeling them. The Pascal VOC dataset and COCO dataset, which are open to the public, are labeled with various objects included in tens of thousands of non-continuous still images. Therefore, by dividing the training dataset and the verification dataset and then inferring the verification data, it is possible to check whether the deep learning model is biasedly trained on the training data.

However, in the image-based labeling data, since numerous still images are consecutively connected, the before and after still images show similar image environments. For this reason, even if the learning results are verified by inferring the verification data, it is not easy to determine whether the deep learning model is biased. Therefore, it is necessary to verify the object recognition ability of the deep learning model through field input and monitoring of the deep learning system. Here, due to the difference between the background of the learning dataset and the background shown in the current image, many false detection cases occur in the initial stage of field application of the deep learning system where the initial learning is completed.

These frequent false positives have as much impact on field operations as forward detection performance requirements. It causes frequent alarms that are different from the truth and unnecessary mobilization of on-site personnel, and the false detection data is mixed with the forward detection data, which causes many difficulties in classification, and results in greatly reducing the effectiveness of the forward detection situation.

In addition, in order to recognize a target object in a video and reduce false positives when a large number of false positives occur, it is necessary to prepare additional training materials to reduce false positives in addition to existing training materials. Because this must be preceded, it takes a lot of time and effort to prepare additional learning materials.

The embodiment may provide an apparatus and a method for training a deep learning model using false-positive data.

An apparatus for learning a deep learning model according to an embodiment includes an image acquisition unit for acquiring an image; Building image big data for learning based on the reference image obtained from the image acquisition unit, extracting a learning object from the reference image of the constructed image big data for learning, a deep learning model based on the extracted learning object a model learning unit for learning; an object inference unit for inferring a target object in the surveillance image provided from the image acquisition unit based on the learned deep learning model, and generating emergency situation data as the inferred result; and a false detection definition unit that classifies the generated accidental situation data into forward detection data and false detection data, and defines the classified false detection data as at least one object false detection data for a predetermined false detection target object, , the model learning unit may re-learn the deep learning model by adding the defined at least one object false detection data to the image big data for training.

The false detection definition unit may classify the forward detection data and the false detection data by receiving a designation of a forward detection and a false detection from the user from the accidental situation data.

The predetermined false detection target object includes a first target object and a second target object, and the false detection definition unit provides the false detection data to at least the second target object among the first target object and the second target object. It can be defined as object false detection data for

The false detection data includes a false detection image and labeling information on a false detection target object in the false detection image, and the false detection definition unit labels the corresponding target object when the false detection target object is the first target object Information may be removed, and when the false detection target object is the second target object, an object class name in the labeling information of the corresponding target object may be changed to an error object for indicating that the false detection target object is the false detection target object.

When the false detection target object is the second target object, the false detection definition unit may remove a region remaining from the false detection image except for a region in which the corresponding false detection target object exists.

The false detection definition unit defines a false detection image from which all regions except for the region in which the false detection target object exists, and false detection data including the changed labeling information, as false detection data for a corresponding false detection object target. can

The method for learning a deep learning model according to an embodiment is to construct an image big data for learning based on a reference image obtained from an image acquisition unit, and extract a learning target object from the reference image of the constructed image big data for learning, training a deep learning model based on the extracted learning target object; inferring a target object in the surveillance image provided from the image acquisition unit based on the learned deep learning model, and generating event data as a result of the inference; classifying the generated emergency situation data into forward detection data and false detection data, and defining the classified false detection data as at least one object false detection data for a predetermined false detection target object; and re-learning the deep learning model by adding the defined at least one object false detection data to the training image big data.

In the defining step, the forward detection data and the false detection data may be classified by receiving the forward detection and the false detection designation from the user from the accidental situation data.

The predetermined false detection target object includes a first target object and a second target object, and in the defining step, the false detection data is transmitted to at least the second target object among the first target object and the second target object. It can be defined as object false detection data for

The false detection data includes a false detection image and labeling information on a false detection target object in the false detection image, and in the defining step, when the false detection target object is the first target object, labeling of the target object Information may be removed, and when the false detection target object is the second target object, an object class name in the labeling information of the corresponding target object may be changed to an error object for indicating that the false detection target object is the false detection target object.

In the defining step, when the false detection target object is the second target object, regions other than a region in which the corresponding false detection target object exists in the false detection image may be removed.

In the defining step, the false detection image from which all regions except the region in which the false detection target object exists and the false detection data including the changed labeling information are defined as false detection data for the corresponding false detection object target. can

According to the embodiment, by re-learning and updating the deep learning model using pre-generated labeling data and false-detection data generated in the real-time inference process, the false-detection result can be additionally reflected in the training material without a separate labeling operation. have.

According to the embodiment, it is possible to improve the object recognition performance without hindering the learning of the existing labeling data.

According to the embodiment, it is possible to reduce false positives occurring in the field of application of the deep learning model.

1 is a diagram illustrating an apparatus for training a deep learning model according to an embodiment of the present invention.
FIG. 2 is a diagram showing a detailed configuration of the data construction unit shown in FIG. 1 .
FIG. 3 is a diagram illustrating a detailed configuration of the false detection definition unit illustrated in FIG. 1 .
4A to 4D are diagrams for explaining false detection data according to an embodiment.
5 is a diagram illustrating a method for training a deep learning model according to an embodiment of the present invention.
6A to 6B are graphs showing the false detection status of the deep learning-based tunnel CCTV image retention system.
FIG. 7 is a diagram illustrating false positive samples generated in the system shown in FIG. 5 .
FIG. 8 is a diagram for explaining the similarity between forward detection and false detection of fire and pedestrians.
9 is a diagram illustrating a deep learning learning process including false detection data.
10 is a graph showing the re-inference result for the labeling data of the deep learning model.
11 is a graph showing the monitoring status for field application of the deep learning model.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

However, the technical spirit of the present invention is not limited to some embodiments described, but may be implemented in various different forms, and within the scope of the technical spirit of the present invention, one or more of the components may be selected among the embodiments. It can be combined and substituted for use.

In addition, terms (including technical and scientific terms) used in the embodiments of the present invention may be generally understood by those of ordinary skill in the art to which the present invention pertains, unless specifically defined and described explicitly. It may be interpreted as a meaning, and generally used terms such as terms defined in advance may be interpreted in consideration of the contextual meaning of the related art.

In addition, the terms used in the embodiments of the present invention are for describing the embodiments and are not intended to limit the present invention.

In this specification, the singular form may also include the plural form unless otherwise specified in the phrase, and when it is described as “at least one (or more than one) of A and (and) B, C”, it is combined with A, B, and C It may include one or more of all possible combinations.

In addition, in describing the components of the embodiment of the present invention, terms such as first, second, A, B, (a), (b), etc. may be used.

These terms are only used to distinguish the component from other components, and are not limited to the essence, order, or order of the component by the term.

And, when it is described that a component is 'connected', 'coupled' or 'connected' to another component, the component is not only directly connected, coupled or connected to the other component, but also with the component It may also include a case of 'connected', 'coupled' or 'connected' due to another element between the other elements.

In addition, when it is described as being formed or disposed on “above (above) or under (below)” of each component, the top (above) or bottom (below) is one as well as when two components are in direct contact with each other. Also includes a case in which another component as described above is formed or disposed between two components. In addition, when expressed as “upper (upper) or lower (lower)”, the meaning of not only an upward direction but also a downward direction based on one component may be included.

In the embodiment, after learning the deep learning model using the labeling data generated in advance as training data, the eventual data generated in the real-time inference process is classified into false detection data and positive detection data, and the classified false detection data and labeling We propose a new way to retrain and update a deep learning model using data.

1 is a diagram illustrating an apparatus for training a deep learning model according to an embodiment of the present invention.

Referring to FIG. 1 , an apparatus for learning a deep learning model according to an embodiment of the present invention includes an image acquisition unit 100 , a data construction unit 200 , a model learning unit 300 , an object inference unit 400 , It may include a false detection definition unit 500 .

The image acquisition unit 100 may be installed at different locations to acquire images on the road. The image acquisition unit 100 may include, for example, a Closed Circuit Television (CCTV).

The data construction unit 200 may collect a plurality of reference images under various predetermined environmental conditions and construct image big data for learning including the collected reference images.

The data construction unit 200 extracts an image of a region of interest from a reference image based on the image big data for learning, extracts a learning target object from the extracted image of the region of interest, and sets a learning data set including the extracted learning target object. can be built

The model learning unit 300 may train a deep learning model or a deep learning algorithm based on the constructed training data set.

The model learning unit 300 may retrain the deep learning model based on the false detection data set and the training data set generated in the process of inferring the target object using the learned deep learning model.

The object inference unit 400 infers a target object in the surveillance image provided from the image acquisition unit 100 based on the learned deep learning model, and determines the existence situation based on the inferred target object to determine the target object and the existence situation. It is possible to generate emergency situation data including

The false detection definition unit 500 analyzes the generated incident data, classifies it into forward detection data and false detection data as a result of the analysis, and an object for at least one predetermined false detection target object from the classified false detection data. False positive data can be defined. Here, the false detection target object may include a vehicle, a pedestrian, a fire, and the like. In an embodiment, the false detection target object may include a pedestrian and a fire except for a vehicle that is relatively accurately recognized.

In this case, the false detection definition unit 500 may classify the forward detection data and the false detection data by receiving a designation of the forward detection and the false detection from the user from the generated incident data.

FIG. 2 is a diagram showing a detailed configuration of the data construction unit shown in FIG. 1 .

Referring to FIG. 2 , the data construction unit 200 for constructing a data set according to an embodiment of the present invention may include a collection unit 210 , an extraction unit 220 , and a construction unit 230 . .

The collection unit 210 is connected to the image acquisition unit 100 by wire or wirelessly, and collects the reference images obtained in advance from the image acquisition unit 100 under various predetermined environmental conditions and includes the collected reference images. data can be built. The reference image may be an image including a predetermined target object.

The extractor 220 may extract a region of interest image from each reference image based on the constructed big data of the reference image for learning. For example, the extractor 220 may extract an image of a predetermined region as an ROI image or extract an image of a region designated by a user as an ROI image.

The extractor 220 may extract at least one learning target object from the extracted ROI image and perform labeling on the extracted at least one learning target object.

The reason for performing this labeling is to exclude objects that are not monitored because the extracted learning object may be an insect, bird, or animal rather than a vehicle or a person.

In this case, in order to accurately label the target object, the extractor 220 may perform labeling based on the user's input information on the extracted at least one learning target object. That is, the user directly determines whether the extracted learning target object is a monitoring target or not. Here, the input information may include, for example, various labeling information such as an object type (a vehicle object (Car), a pedestrian object (Person), a fire object (Fire)), a location, and a size.

The construction unit 230 may extract a learning target object to be monitored among the labeled learning target objects to rescue a learning data set including the extracted learning target object.

FIG. 3 is a diagram illustrating a detailed configuration of the false detection definition unit illustrated in FIG. 1 .

Referring to FIG. 3 , a false detection definition unit 500 for defining false detection data according to an embodiment of the present invention may include a collection unit 510 , a classification unit 520 , and a definition unit 530 . can

The collection unit 510 may collect the emergency situation data generated from the object inference unit 400 . The collection unit 510 may irregularly visit the site where the image loss system is operating and collect data on the failure situation detected up to that point.

In this case, the emergency data is data automatically labeled in the object inference process, and may include an image and labeling information on at least one target object in the image.

The classification unit 520 may analyze and classify the accidental situation data into forward detection data and false detection data as a result of the analysis, and classify the classified false detection data into a plurality of object false detection data. For example, the classification unit 520 may classify the classified false detection data into object false detection data for a vehicle, object false detection data for a pedestrian, and object false detection data for fire.

The definition unit 530 may define the false detection data by changing the class name among the labeling information of the target object in the classified plurality of object false detection data to information indicating that the object is a false detection object.

At this time, the labeling class names are vehicle object (Car), pedestrian object (Person), fire object (Fire), which are the target object types that are correctly detected, and vehicle error object (Not_Car) and pedestrian error object (Not_Person), which are the types of the falsely detected target object. ), and a fire error object (Not_Fire).

Through this, in the present embodiment, a frequently occurring false detection can be handled as an exception by adding a comparison object that is falsely detected through a process of modifying the labeling class name.

4A to 4D are diagrams for explaining false detection data according to an embodiment.

4A to 4B , as an example, the definition unit 530 according to the embodiment assigns a class name from the pedestrian object {Person} to the pedestrian error object {Not_Person among the labeling information of the pedestrian false detection data whose target object is a pedestrian. } to define pedestrian false detection data.

As another example, the definition unit 530 according to the embodiment changes the class name among the labeling information of fire false detection data in which the target object is fire from the fire object {Fire} to the fire error object {Not_Fire} to define the false fire detection data. can do.

In this case, in the case of the vehicle object {Car}, since it is more effective to delete the label than to handle an exception through a comparison object, the labeling information is deleted without defining it as false detection data.

Also, a plurality of target objects may exist in one image, and the plurality of target objects may be positively detected or falsely detected. Therefore, in the embodiment, the region in which the false-detected target object exists, that is, all regions except for the false-detected object box are removed.

Through this, the present embodiment prevents re-learning of uncertain inference information or non-detection information, thereby minimizing the possibility of a decrease in the existing positive detection rate due to re-learning of false positive images.

4C to 4D, as an example, the definition unit 530 according to the embodiment removes all areas except for the area in which the pedestrian error object exists from the image of the pedestrian false detection data in which the target object is a pedestrian. can

As another example, the definition unit 530 according to the embodiment may remove all areas other than the area in which the fire error object exists from the image of the fire false detection data in which the target object is fire.

The definition unit 530 may define false detection data including an image in which all regions except for a region in which only a false detection object exists, and the changed labeling information.

5 is a diagram illustrating a method for training a deep learning model according to an embodiment of the present invention.

Referring to FIG. 5 , an apparatus (hereinafter, referred to as a learning apparatus) for training a deep learning model according to an embodiment of the present invention may collect reference images obtained under various predetermined environmental conditions (S511).

Next, the learning apparatus may build big data for learning including the collected reference image (S512).

Next, the learning apparatus extracts an image of a region of interest from each reference image based on the image big data for learning, extracts a learning target object from the extracted image of the region of interest, and generates a training data set including the extracted learning target object. It can be built (S513).

Next, the learning apparatus may train the deep learning model based on the constructed training data set (S514).

On the other hand, the learning apparatus may collect the monitoring image acquired in real time at a predetermined location (S521).

Next, the learning device infers the target object in the surveillance image collected based on the learned deep learning model (S522), determines the existence situation based on the inferred target object, and determines the target object and the existence situation as a result of the determination It is possible to generate the emergency situation data including (S523).

In this case, the learning apparatus may display the monitoring image based on the generated incident data (S524). That is, the learning apparatus may display the target object included in the emergency situation data and the emergency situation by combining it with the monitoring image.

Next, the learning device analyzes the generated incident data and classifies it into forward detection data and false detection data as a result of the analysis, and re-segments only the false detection data among the classified data to classify it into multiple object false detection data. It can be (S525).

Next, the learning apparatus may define the false detection data by inserting information indicating that the object is a false detection object in the labeling class name of the target object in the classified plurality of object false detection data (S526).

Next, the learning apparatus may extend the training data set by adding the defined at least one object false detection data to the training data set ( S527 ).

Next, the learning apparatus may re-train and update the deep learning model based on the extended training dataset, that is, the labeled training dataset and the unlabeled false positive data (S528).

The basic concept and system configuration diagram of the deep learning-based tunnel CCTV image retention system has been completed, and the actual system is being installed and monitored at the tunnel control center site.

1. Basic concept of deep learning-based tunnel CCTV video retention system

The deep learning-based tunnel CCTV image retention system first takes the CCTV still image in the tunnel as an input value at regular frame intervals, and the deep learning model classifies the vehicle, fire, and pedestrian objects that exist in the still image to determine the location and size of the rectangular boundary. It is represented by a bounding box. At this stage, it is possible to detect a disaster situation in which a fire and pedestrians occur. Then, the object is tracked based on the vehicle object recognized in the still image, and the stopping and reverse driving are determined at regular frame periods longer than the object recognition process. In the object tracking process, vehicle objects detected in the previous frame period and the current frame period are compared with each other. Judgment of stopping and reverse running is performed on a pair of vehicle objects having the same object number in the previous frame period and the current frame period. When it is less than 0.75, it is judged as stopping and reverse driving, respectively. This content is the core process of the deep learning-based tunnel CCTV video retention system, and the entire cycle system to be installed at the site receives the CCTV video so that it can simultaneously monitor 9 channels of cameras, and preprocesses the video so that it can be used in the core process. It consists of a core process, and a process of expressing the emergency situation that can be directly checked at the tunnel control center. The deep learning learning process does not learn in the field, it learns in another place, and then periodically updates the deep learning model of the whole cycle system every two weeks to one month.

On the other hand, in machine learning including deep learning, supervised learning learns labeling data including input and output values, and when the data is sufficient, it can be used variously, so it is the most active research It is a field that is becoming The basic process of supervised learning divides data for learning and data for inference, and learns a deep learning model based on the data for learning. And the verification of the deep learning model is to check the re-inference recognition performance of the training data and the recognition performance of the inference data, and when applied to the actual field, it can be predicted whether the deep learning-based system can properly recognize the target. In this process, when evaluating the recognition performance of the deep learning model, the types of the recognized output values are classified into 4 types based on Table 1 and then evaluated.

Category Predicted as Positive Predicted as Negative Positive True Positive (TP) False Negative (FN) negative False Positive (FP) True Negative (TN)

In Table 1, the correct answer (positive) means the object to be recognized, and the wrong answer (negative) means an object that is not a correct answer. false positive). If the wrong answer is correctly recognized, it is a true negative. Most deep learning-based systems set goals in the direction of maximizing positive detection and reducing false positives and non-detections, so recall to evaluate the quantitative recognition rate of positive detection and precision to evaluate the qualitative recognition rate of positive detection. is evaluated as an average precision value using

[Equation 1]

Recall = TP / (TP + FN)

[Equation 2]

Precision = TP / (TP + FP)

[Equation 3]

Average Precision = ∫Precision(Recall)dRecall

Average precision is an indicator that can consider both quantitative and qualitative aspects of forward detection, and can be expressed as a scalar value ranging from 0 to 1, so it is used as a standard indicator to indicate the inference accuracy of deep learning models for labeling data. .

2. Monitoring status of deep learning-based tunnel CCTV video retention system

The deep learning-based tunnel CCTV video retention system was installed and operated at a specific highway tunnel control center. The tunnel control center monitors tunnels and roads based on closed network-based CCTV cameras. Most of the tunnel control centers have installed a disaster detection system according to the guidelines of the Ministry of Land, Infrastructure and Transport. However, the existing emergency detection system shows low reliability due to the frequent occurrence of false positives and requires unnecessary verification and follow-up efforts on the site. This is the majority. For this reason, when monitoring the deep learning-based tunnel CCTV image retention system developed in the present invention, it is the most important goal to check whether a correct detection is achieved, but at the same time, it is also important to reduce the case of being recognized as a false detection. For the deep learning-based tunnel CCTV image retention system, we learned and applied a deep learning model with labeling data and started monitoring. As a result of monitoring the system for 55 days, vehicle objects were generally recognized fairly accurately without false positive objects, but fire objects and pedestrian objects were consistently generating false detections.

6A to 6B are graphs showing the false detection status of a deep learning-based tunnel CCTV image retention system.

6A to 6B, after installing the deep learning-based tunnel CCTV image retention system, monitoring was performed for 55 days, and the 9-channel camera fires and false detections of pedestrians were shown in graphs. Pedestrians were generating up to 1000 false positives, 200 to 400 false positives per day, and fires generated up to 120 false positives and 0 to 40 false positives per day. The number of false positives for pedestrians occurs much more frequently than for fires, because the recognition of pedestrian objects is much more difficult.

Pedestrian objects generally have a smaller size than fire or vehicle objects, and have a vertically elongated shape. In addition, in the event of a vehicle crash, the driver comes out of the vehicle to check the condition of the vehicle, and the image attached to the vehicle is captured by CCTV. This phenomenon is called the overlapping phenomenon between moving objects, and due to the overlapping phenomenon of pedestrian objects and vehicle objects, it is not easy for the deep learning model to distinguish between the shadow of a vehicle or a lane and pedestrians with a general deep learning model learning method. Factors that can cause a fire in a tunnel are mainly fires caused by vehicles. That is, when a fire occurs, the size of the fire object photographed in the CCTV can be seen as the size of the minimum vehicle, and when it occurs large, it is distinguished from the vehicle because it is large enough to occupy the entire area of the CCTV screen. However, it may be mistaken for a fire due to a warning light of the work vehicle or sunlight reflected from the tunnel entrance, and a false detection phenomenon as shown in FIG. 6 could be confirmed.

FIG. 7 is a diagram illustrating false positive samples generated in the system shown in FIG. 5 .

Referring to FIG. 7 , it is a screen in which a fire and pedestrians are falsely detected in the deep learning-based tunnel CCTV image retention system. In the case of fire, as shown in (a) and (c), the shape of the work vehicle's warning light shining inside the tunnel and the back of the large truck at the entrance of the tunnel shining brightly reflected in sunlight are falsely detected as fire. And in the case of pedestrians, as shown in (b) and (d), some shapes in black shading of vehicle objects and white lanes were recognized as pedestrians. The cause of this problem can be determined when compared with the labeling data used for training the deep learning model.

FIG. 8 is a diagram for explaining the similarity between forward detection and false detection of fire and pedestrians.

Referring to FIG. 8 , labeling data and false detection data samples for fire and pedestrians were compared. If you look at the overall picture of each still image, you can see that there is a difference between the labeling data and the false positive data. However, if we compare only the rectangular bounding box, there are similarities, and in the case of fire, the actual fire shape is quite similar to that of the warning light of the work vehicle. Similarly, the pedestrian object is not exactly similar to the black shading of the vehicle and the actual pedestrian, but it is similar enough that the deep learning model can be mistaken for a pedestrian. Therefore, it is not easy to deal with false positives occurring in images other than training only by learning the existing labeling data for the deep learning model.

3. Learning method and evaluation including false detection of deep learning model

The false-detection data generated while monitoring the deep learning-based CCTV image retention system showed the limitations of the deep learning model obtained by learning the labeled data, and showed low system reliability. In order to solve this problem, the present invention proposes a method for learning a deep learning model including false-positive data. And in order to validate the learning method including false-positive data, we measure the re-inference performance on the labeling data and the recognition performance on the false-positive data to measure the deep learning model trained only on the labeling data and the deep learning including the false-positive data. We will compare the models.

3.1 Learning method including false detection of deep learning model

The learning method including false detection of the deep learning model proposed in the present invention proceeds with the same procedure as in FIG. 8, and the securing of false detection data confirmed in the monitoring process of the deep learning-based CCTV image retention system installed in the tunnel control center is presupposed

9 is a diagram illustrating a deep learning learning process including false detection data.

Referring to FIG. 9 , the deep learning model including false detection learning method first trains the deep learning model with labeling data, and then checks the object recognition ability of the deep learning model through verification, and then dips the deep learning model into the system installed at the tunnel control center site. Apply the learning model. Then, the site visits the tunnel control center on an irregular basis, collects and analyzes the data of the accidental situation detected so far, and then classifies the false-detection data and the forward-detection data. Objects to be used for training from false detection data are pedestrians and fires, excluding vehicles that are relatively accurately recognized in the field. ) to create a false detection dataset. The produced false-detection dataset is used for training of the deep learning model like the labeling data, and if there is no abnormality by verifying the deep learning model again, the deep learning model applied to the tunnel control center site is replaced. By repeating this process and adding false positive data to learn, the recognition of false positives in the deep learning model is automatically reduced without system modification. An automatic improvement effect can be expected.

3.2 Deep Learning Model Performance Evaluation

The learning method including false detection of the deep learning model proposed in the present invention learns by adding false detection data generated in the field without adding labeling data, so that it is possible to save time and cost for producing labeling data. To verify this method, the object recognition performance of the deep learning model is evaluated by adding the model trained with the existing labeling data and the false detection data, and then comparing the learned deep learning model. The evaluation of object recognition performance will evaluate the effectiveness of the learning method including false detection of the deep learning model by comparing the re-inference performance of the labeling data and the recognition reduction performance of the false detection data.

3.2.1 Labeling and false-detection big data status

Table 2 shows the status of labeling and false-positive data to be used for deep learning model training. The AA~DD datasets are all manually labeled data, and there are 441,670 vehicles, 49721 pedestrians, and 857 fire objects in a total of 70,914 still images. The learning objects of the deep learning model are vehicles, pedestrians, and fires, and since the images are taken from road tunnels, the number of vehicle objects is the largest in the labeling data. Next, there are many objects in the order of pedestrians and fires. The reason why the number of fire objects is small is that generally, fires rarely occur in road tunnels and it is not easy to secure an original image related to the fire. For this reason, we learn the deep learning model with the problem of decreasing the diversity of fire objects.

In Table 2, false detection data means false positive data automatically derived in the field application stage. FP data 1 is the ratio of the number of false fire detections to the number of false pedestrian detections in a ratio of 1:2, and FP data 2 is for pedestrians. False positive data dominates. FP data 1 is data constructed to examine the effect of simultaneously reducing false fire detection and pedestrian false detection, and FP data 2 is data constructed for the main purpose of reviewing the reduction effect of false pedestrian detection. This configuration is because the number of fire objects in the labeling data is small, so the type of false fire detection that occurs at the site to which the deep learning model is applied is simple, so it is easy to reduce false fire detections. In addition, since the false fire detection in the labeling data was defined as the light of the tunnel warning light, the tail lamp of the vehicle, and the warning light of the working vehicle as the false fire detection, FP data 1 includes 691 object data defined as false fire detections, and FP data 2 Only 22 were included. On the other hand, in the case of pedestrian false positives, FP data 1 includes 1357 data and FP data 2 includes 7999 pedestrian false positives to confirm the effect of reducing false positives according to the scale.

Data category Number of images Car Person Fire False_fire False_person AA 38831 219292 26612 0 181984 0 bb 873 0 0 857 713 0 CC 7471 52296 9042 0 1751 0 DD 23739 175138 11487 0 0 0 FP data1 2041 0 0 0 691 1357 FP data2 8007 0 0 0 22 7999

3.2.2 Data composition and evaluation method according to the deep learning model Based on the labeling data and false detection data described in Table 2 above, a dataset to be used for learning was constructed and divided into three types as shown in Table 3 according to the purpose. In Table 3, Model 1 learns only the labeling data, and it is a trained model to compare Model 2 and Model 3 learned by adding false-positive data. Model 2 added the FP data 1 false detection dataset, and has the purpose of examining the effect of reducing false fire detection and pedestrian false detection at the same time. Finally, Model 3 proceeds with learning by adding FP data 1 and FP data 2 to examine the reduction effect of false pedestrian detection more reliable than Model 2.

model name Including dataset Model1 Labeled dataset Model2 Labeled dataset + FP data1 Model3 Labeled dataset + FP data1 + FP data2

Based on these model conditions, learning proceeds with the deep learning model learning conditions shown in Table 4. The deep learning model uses Faster R-CNN (Regional Convolutional Neural Network), which is widely used in the field of deep learning object recognition. The graphics card uses NVIDIA GTX 1070 8 GB, and deep learning model training and inference were performed in a Linux environment. Each model was trained for 10 epochs.

deep learning model Faster R-CNN GPU NVIDIA GTX 1070 8 GB OS Linuxmint 18.3 Epoch 10

The evaluation of the deep learning model checks the re-inference of the labeling data, the number of recognition of false pedestrian detection data, and the recognition status of the false fire detection data after applying the deep learning model to the field. By reinferring the labeling data, a comparative evaluation is performed to see if the vehicle, fire, and pedestrian object recognition ability is improved compared with Model 1, which learned only the existing labeling data. In the case of pedestrian false detection data, the reduction performance of pedestrian false detections is evaluated according to the number of false pedestrian detections used for training the deep learning model, and the pedestrian false detection data used in Model 2 and the pedestrian false detection data used in Model 3 are evaluated separately. do. In the case of false fire detection, apply Model 2 to the site and monitor it to directly check whether false fire detection has been reduced. 3.2.3 Reinference evaluation of labeling data

The re-inference result for the labeling data according to each deep learning model is shown in FIG. 10 below.

10 is a graph showing the re-inference result for the labeling data of the deep learning model.

Referring to FIG. 10 , in the case of a vehicle and a pedestrian, the object recognition performance of Model 2 and Model 3 is better than that of Model 1. As a result, learning including false-detection data did not impair the object recognition ability of the existing labeling data, but rather improved the object recognition ability. For vehicles, Model 2 and Model 3 showed similar object recognition ability, but in the case of pedestrians, Model 3, which was learned by adding more false-detection data, showed better pedestrian object recognition ability than Model 2. In other words, it can be said that the ability to reduce false positives for pedestrians is improved as the pedestrian object class adds and learns the pedestrian false detection data. On the other hand, the fire object recognition ability has an AP value of 0.9, and Model 1, Model 2, and Model 3 have the same value. From the data given, the fire object was fully trained.

3.2.4 Evaluation of the number of recognition of false positive data

By re-inferring the labeling data, it was possible to confirm the improvement of vehicle object recognition performance, the improvement of pedestrian object recognition performance according to the reduction of false pedestrian detection, and the consistently maintained fire object recognition performance. However, in order to directly check the performance of reducing false positives for pedestrians and fires, it is necessary to directly infer false detection data and to monitor a deep learning model applied to the field including false detections.

The evaluation of the pedestrian false detection reduction performance of the deep learning model was targeted for Model 2 and Model 3 including false detection data as shown in Table 5. We re-inferred on 9358 false-positive data for pedestrians.

target model Number of detected persons as FP 1357 Negative examples 9358 Negative examples Model2 329 3270 Model3 34 466

As a result of re-inference on pedestrian false-detection data, Model 3 showed much better performance in reducing false-detection. First, when inferring 1357 false positives for pedestrians, only 34 false positives occurred in Model 3 compared to 329 in Model 2. And as a result of inferring 9358 false positives for pedestrians, Model 2 generated 3270 false positives for pedestrians, and Model 3 generated only 466 false positives for pedestrians. Comprehensively, Model 2 generated pedestrian false positives at a rate of 24% and 35% of the inferred pedestrian false positive data, whereas Model 3 generated false pedestrian false positives only at 2.5% and 5% of the inferred pedestrian false positives data. Therefore, it can be said that Model 3, which was learned by adding more pedestrian false detection data than Model 2, has excellent performance in reducing false pedestrian detection.

11 is a graph showing the monitoring status for field application of the deep learning model.

Referring to FIG. 11 , it is the monitoring status of the application of the deep learning model to the field to confirm the reduction performance of false fire detection, and it is connected with the monitoring status of the fire in FIG. 5, and until 55 days after the field installation of the deep learning model, Model 1 , and model 2 was used from the 56th day.

As a result, in Model 1, a maximum of 120 false fire detections, usually 0 to 40, occurred, but only 0 to 3 false fire detections occurred after the time when Model 2 was applied. The number of data of false fire detection used in Model 2 is 691 in FP data 1, and this data alone could certainly reduce false fire detection. However, since the number of fire labeling data is not much at 857, it can be judged that the reduction of false fire detection is much easier compared to false detection of pedestrians.

The term '~ unit' used in this embodiment means software or a hardware component such as a field-programmable gate array (FPGA) or ASIC, and '~ unit' performs certain roles. However, '-part' is not limited to software or hardware. The '~ unit' may be configured to reside on an addressable storage medium or may be configured to refresh one or more processors. Thus, as an example, '~' denotes components such as software components, object-oriented software components, class components, and task components, and processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. The functions provided in the components and '~ units' may be combined into a smaller number of components and '~ units' or further separated into additional components and '~ units'. In addition, components and '~ units' may be implemented to play one or more CPUs in a device or secure multimedia card.

Although the above has been described with reference to preferred embodiments of the present invention, those skilled in the art can variously modify and change the present invention within the scope without departing from the spirit and scope of the present invention as set forth in the claims below. You will understand that it can be done.

100: image acquisition unit
200: data construction unit
300: model learning unit
400: object inference unit
500: false positive definition unit

Claims (12)

an image acquisition unit for acquiring an image;
Building image big data for learning based on the reference image obtained from the image acquisition unit, extracting a learning object from the reference image of the constructed image big data for learning, a deep learning model based on the extracted learning object a model learning unit for learning;
an object inference unit for inferring a target object in the surveillance image provided from the image acquisition unit based on the learned deep learning model, and generating emergency situation data as a result of the inference; and
A false detection definition unit for classifying the generated accidental situation data into forward detection data and false detection data, and defining the classified false detection data as at least one object false detection data for a predetermined false detection target object;
The model learning unit,
An apparatus for learning a deep learning model, which re-trains the deep learning model by adding the defined at least one object false detection data to the training image big data. According to claim 1,
The false detection definition unit,
An apparatus for training a deep learning model by receiving a designation of a forward detection and a false detection from a user from the accidental situation data and classifying the forward detection data and the false detection data. According to claim 1,
The predetermined false detection target object includes a first target object and a second target object,
The false detection definition unit,
An apparatus for training a deep learning model, defining the false detection data as object false detection data for at least the second target object among the first target object and the second target object. 4. The method of claim 3,
The false detection data includes a false detection image and labeling information on a false detection target object in the false detection image,
The false detection definition unit,
When the false detection target object is the first target object, the labeling information of the corresponding target object is removed,
When the false detection target object is the second target object, an apparatus for learning a deep learning model that changes an object class name in labeling information of the corresponding target object to an error object for indicating that it is a false detection target object. 5. The method of claim 4,
The false detection definition unit,
When the false detection target object is the second target object, an apparatus for training a deep learning model by removing a region other than a region in which a corresponding false detection target object of the false detection image exists. 6. The method of claim 5,
The false detection definition unit,
A deep learning model that defines false detection data including the false detection image and the changed labeling information from which all areas except the area in which the false detection target object exists are false detection data for the corresponding false detection object target. device for learning. Building image big data for learning based on the reference image obtained from the image acquisition unit, extracting a learning object from the reference image of the constructed image big data for learning, and learning a deep learning model based on the extracted learning object making;
inferring a target object in the surveillance image provided from the image acquisition unit based on the learned deep learning model and generating emergency situation data as the inferred result; and
classifying the generated emergency situation data into forward detection data and false detection data, and defining the classified false detection data as at least one object false detection data for a predetermined false detection target object; and
A method for learning a deep learning model, comprising the step of re-learning the deep learning model by adding the defined at least one object false detection data to the image big data for training. 8. The method of claim 7,
In the above definition step,
A method for training a deep learning model, receiving a designation of a forward detection and a false detection from a user from the accidental situation data and classifying the forward detection data and the false detection data. 9. The method of claim 8,
The predetermined false detection target object includes a first target object and a second target object,
In the above definition step,
A method for training a deep learning model, defining the false detection data as object false detection data for at least the second target object among the first target object and the second target object. 8. The method of claim 7,
The false detection data includes a false detection image and labeling information on a false detection target object in the false detection image,
In the above definition step,
When the false detection target object is the first target object, the labeling information of the corresponding target object is removed,
When the false detection target object is the second target object, a method for training a deep learning model by changing an object class name in labeling information of the corresponding target object to an error object for indicating that it is a false detection target object. 11. The method of claim 10,
In the above definition step,
When the false detection target object is the second target object, the method for training a deep learning model by removing the remaining areas except for the area in which the corresponding false detection target object of the false detection image exists. 12. The method of claim 11,
In the above definition step,
A deep learning model that defines false detection data including the false detection image and the changed labeling information from which all areas except the area in which the false detection target object exists are false detection data for the corresponding false detection object target. How to learn.

Images