KR20220138894A - Prediction and recognition method of road marking information and road maintenance method - Google Patents

Abstract

Disclosed are a vehicle terminal and a method for managing a static object on a vehicle movement path using the same, wherein the vehicle terminal executes the following steps of: extracting a first object image estimated to be a static object on a road having one of predetermined types from a first image taken by a photographing device of a vehicle; selecting a first reference image corresponding to the extracted first object image among a plurality of reference images provided according to traffic laws as images respectively representing static objects; and providing at least one of the extracted first object image and the selected first reference image to another computing device. According to the present invention, provided is a technology for detecting an image of a driver and autonomous vehicle and determining visibility according to a pavement condition of a road using pre-collected information on various static objects related to the road.

Description

Prediction and recognition method of road marking information and road maintenance method

The present invention recognizes road marking information based on AI data that is a static object obtained in relation to road driving such as lane/crosswalk, traffic lights, and road signs using computing technology, and uses the recognized road marking information to recognize the road It relates to technology for generating information for maintenance.

AI data, which is a static object in the driving environment, is an aptitude test item for autonomous vehicles. For autonomous vehicles to drive, the first item in the cognitive-judgment-control stage is cognition, and static objects in the driving environment (lanes, crosswalks, traffic lights, road signs) are in the "Autonomous Driving Vehicle Driver's License Test System" It corresponds to the subject of the department exam. As the first step toward commercialization of autonomous vehicles, the development of static object recognition technology is required.

With the development of autonomous vehicle-related technologies, the sequential compulsory installation of various terminals and sensors necessary for autonomous driving is being promoted at home and abroad. It is expected that laws and systems in the field of security and authentication will be complete.

After the completion of commercialization of the Advanced Driver Assistance System (ADAS), which is a level 2 autonomous driving technology that is at the level of current driving assistance, the implementation of some level 3~4 autonomous driving technologies is expanding, centering on overseas automakers.

The representative autonomous driving technology is the Lane Keeping Assist System (LKAS) technology that recognizes the front lane through an image recognition sensor and maintains the lane by controlling the steering wheel (steering wheel), speed control by recognizing signs, and direction By combining Traffic Sign Recognition (TSR) technology that supports switching and overtaking, the inter-vehicle distance control function and the lane keeping function, it recognizes the driving trajectory of nearby vehicles and maintains a certain distance from the preceding vehicle even when the lane is not visible. There is a traffic jam assist (TJA) technology that maintains the traffic jam. The present invention relates to the lane keeping support system and road sign recognition.

For the production of autonomous vehicles of level 3 or 4 or higher, where the driver's responsibility is rapidly reduced, cameras, radars (Radio Detection and Ranging, RADAR), and LIDAR (Light Detection and Ranging, LIDAR) that recognize static and dynamic obstacles are required. Driving environment recognition sensor and GPS-based positioning, maps with precision that provides information such as road facilities, signals, and signs, V2V (Vehicle to Vehicle) and V2I (Vehicle to Infra), etc. that exchange various facilities, environment and traffic information of the element technology is additionally required.

In order to improve the completeness of autonomous driving technology, it is necessary to improve the recognition performance of automobile AI for static objects on the road such as lanes, crosswalks, and traffic lights/road signs. As the level of autonomous driving increases, the dependence on artificial intelligence will continue to increase. In autonomous driving, static object recognition is very important, and driving by recognizing the lane is a way to improve control. Accordingly, it is necessary to secure a dataset for obtaining lane information and obtain ground truth information.

In order to advance autonomous driving, autonomous cooperative driving, which acquires surrounding infrastructure information and performs control through convergence with own vehicle sensors, is being introduced. As such, autonomous cooperative driving is being introduced, so it is possible to obtain traffic light information through communication between vehicles, but it is difficult to guarantee the integrity of data such as communication delay. Accordingly, the dilemma at intersections can be resolved through recognition of traffic light information, but the final control panel of an autonomous vehicle needs to be judged through the vehicle's magnetic sensor, so development of traffic light recognition using an image sensor is essential. Therefore, high-quality traffic light datasets in many different environments are needed for deep learning-based development. Although various services are possible with the introduction of precision maps, there may be situations in which it is difficult to accurately determine the location of an autonomous vehicle with only a sensor. Accurately stopping at a crosswalk to protect pedestrians is possible even with precision map information, but ultimately, it is necessary to determine the stop line in the vehicle, and high-quality data in many different environments is required.

For autonomous driving and next-generation intelligent transportation service (C-ITS), it is required to recognize various static objects sensed through a camera while driving. In autonomous driving, in order to determine a drivable path and control a lateral vehicle to drive on the corresponding path, it is necessary to recognize various types of lanes in various environments. In addition, the autonomous vehicle needs to sense and recognize traffic lights, road signs, crosswalks, etc. through a camera in order to recognize various road traffic conditions.

Acquiring and recognizing road-related information through a camera while driving a vehicle assumes that the information is maintained undamaged on the road and its surroundings. Over time, signs such as crosswalks, for example, may be damaged, or one-minute signs may be damaged. Such damage needs to be quickly identified and restored, and a technology that supports the identification of this fact is required.

For example, as a result of a survey of 126 'School Zones with Multiple Traffic Accidents' in which two or more traffic accidents occurred or one or more fatalities occurred in one year, 75.4% of the total were problems caused by deterioration of road markings, etc. (2020, Ministry of Public Administration and Security). Therefore, proper monitoring and management of road markings and appropriate driver information assistance means are required.

In the present invention, it is possible to provide a technology for detecting images and determining the degree of visibility of drivers and autonomous vehicles according to the pavement condition of the road by using the information about various static objects related to the road that has already been collected, and it is also difficult to identify It is intended to provide a technology that can supplement and provide display contents in real time through image prediction of road surface markings and lanes, and provide the difference between predicted information and actual observed information to the administrator of the static object management system on the road. .

The static object recognition method on the road provided according to an aspect of the present invention may be implemented using a convolutional neural network (CNN) algorithm. The convolutional neural network is a kind of artificial neural network. The artificial neural network is a statistical learning algorithm inspired by neural networks in biology in machine learning and cognitive science. An artificial neural network is a model that has problem-solving ability by changing the strength of synaptic bonding through learning in which artificial neurons that form a network through synaptic bonding.

The method for generating road management information provided according to an aspect of the present invention may generate and provide information on whether road surface markings need repainting, whether safety signs are aging, whether signal signs are aging, and whether or not anti-slip pavement is aging.

The use of multiple hidden layers in artificial neural networks is a deep learning emerging in big data-based machine learning. Deep learning is easy to find non-linear patterns that are difficult to extract in linear regression analysis. In the case of computer vision, since images are used as input data, a structure different from that when dealing with simple structured data is required, and the convolutional neural network reflects this. A convolutional neural network includes an additional hidden side called convolution and a device for pattern recognition called max pooling.

In the present invention, in order to automatically recognize specific patterns related to the degree of aging and visibility of road marking information in a photographed image and data set, a convolutional neural network technique used for pattern recognition in an image is used.

The convolutional neural network technology adopted according to an aspect of the present invention utilizes an existing image pattern recognition model, and a support vector machine algorithm is commonly used for this structure. The support vector machine algorithm is used to increase the accuracy of analysis by recognizing an object required for analysis among a plurality of objects existing in an image and restricting the position of the object in a box shape.

The method for generating road management information provided in accordance with an aspect of the present invention provides a model for predicting the corresponding road marking information based on the images as well as recognizing the degree of aging of the road markings based on the images, and includes a map learning is used

Unlike unsupervised learning, supervised learning requires that the training data be labeled, and the model is trained based on the "correct answer" of the training data. In the present invention, learning data is generated based on the correct answer data and image data presented by the Road Traffic Act, and based on this, the correlation and pattern between the analysis result of the image-based deep learning model and the aging indication-correct answer data can be found.

Data for sustainable road maintenance and improvement may be provided in accordance with an aspect of the present invention.

In this case, in particular, data for evaluating whether the facility is aging may be provided. As specific technologies for this, ① image prediction technology of aging bridge surface (road surface) markings, ② road facility safety evaluation technology, and ③ performance change estimation and correction through monitoring the degree of visibility and correction of bridge surfaces (road surface) are provided. can do.

Also, in particular, data for supporting autonomous driving may be provided. Specifically, an optimal bridge surface (road surface) pavement luminance and visibility determination technology for an autonomous vehicle may be provided.

The static object prediction method on the road provided according to an aspect of the present invention uses an image recognition-based convolutional neural network algorithm to compare the aged and difficult-to-identify road surface display image with the matching data provided by the Road Traffic Act. Through measurement and prediction, the relevant road surface indication information can be delivered to the driver or autonomous driving service in real time as an API.

In addition, the road facility safety evaluation method provided according to an aspect of the present invention is based on the output obtained from the image prediction technology of the aging bridge surface (road surface) mark, the visibility and It is possible to provide a solution to evaluate the road environment so that the level of safety can be evaluated by presenting the criteria for determining the level of deterioration in stages (best - average - need for improvement - urgently). In addition, it is possible to estimate the performance change of the static object by performing monitoring of the bridge surface (road surface) pavement and the degree of visibility on the static object, and predict the corrective reconstruction time for repairing the leaking static object.

In addition, the monitoring method provided according to one aspect of the present invention manages the collection time of existing data and the safety evaluation history of geo data based on the output obtained from the road facility safety evaluation method It provides real-time information on whether the static object needs to be repackaged or repaired to private and government agencies through the

In addition, the optimal bridge surface (road surface) pavement brightness and visibility determination support information providing method for an autonomous vehicle provided according to an aspect of the present invention is autonomous based on the output of image prediction technology of aging bridge surface (road surface) display Additional information related to the improvement of the recognition rate of the image sensor of the pavement can be provided so as to contribute to the improvement of the accuracy of image recognition through the camera of the driving vehicle.

A vehicle terminal provided according to an aspect of the present invention includes: a processing device; photographing device; and an object information table including data on a plurality of static objects having any one of predetermined types existing on a movement path of a vehicle, and the object information table including geolocations of the plurality of static objects and a storage device in which a plurality of reference reference images provided according to the Road Traffic Act are stored as images representing each of the static objects.

At this time, the difference between the first earth position, which is the earth position of the first static object existing on the predicted first movement path of the vehicle, by the processing device, and the current vehicle position, which is the current earth position of the vehicle obtained from the GPS device starting photographing using the photographing device from a point in time when α has a value within a predetermined threshold value; extracting a first object image estimated to be the static object having any one of the predetermined types from the first image photographed by the photographing device; selecting a first reference reference image corresponding to the extracted first object image from among the reference reference images; and providing at least one of the extracted first object image and the selected first reference reference image to another computing device.

In this case, the other computing device is an autonomous driving device supporting autonomous driving of the vehicle, and the autonomous driving device displays the extracted first object image or the selected first reference reference image on a display device included in the vehicle. may be made to do.

In this case, the selecting of the first reference reference image may include: selecting a first group of static objects existing within a predetermined distance from the first earth location from among the static objects stored in the object information table; selecting a first group of reference reference images corresponding to the selected first group of static objects from among the plurality of reference reference images; and determining, among the selected first group of reference reference images, the one having the highest similarity with the first object image as the first reference reference image.

In this case, the other computing device is a remote server, and the server may include: calculating, by the server, a degree of similarity between the extracted first object image and the selected first reference reference image; calculating an aging degree of the first static object represented by the first object image based on the calculated similarity; and generating a repair or replacement schedule for the first static object according to the calculated degree of deterioration.

In this case, the vehicle terminal further includes a prediction device, and the prediction device includes a neural network learned according to a supervised learning method, and the neural network uses images for learning obtained by shooting an information collection vehicle as input information for learning, It may be supervised learning using a label for each of one or more object images designated in each of the acquired learning images. The extracting of the first object image may include inputting the photographed first image to the learned neural network and using an output of the learned neural network.

In this case, the processing apparatus may include: extracting, from the first image, a first group of object images, which are all object images estimated to be the static objects having any one of the predetermined types; The step of providing information about the first earth position and the extracted first group of object images to the other computing device; may be further executed. The other computing device may be a server configured to access a database including the object information table. In this case, the server selects, among the static objects stored in the object information table, a first group of static objects existing within a predetermined distance from the first earth location; determining whether there is a lost object that is a static object that does not correspond to the object images of the first group among the static objects of the first group; and generating a repair or replacement schedule for the lost object.

According to another aspect of the present invention, there is provided a method for generating a management schedule for a static object on a road, the method comprising: receiving, by a server, a first object image and a first reference reference image from a vehicle terminal; calculating, by the server, a degree of similarity between the extracted first object image and the selected first reference reference image; calculating, by the server, a degree of deterioration of the first static object represented by the first object image based on the calculated similarity; and generating, by the server, a repair or replacement schedule for the first static object according to the calculated degree of deterioration.

In this case, the vehicle terminal determines the geolocations of the plurality of static objects as an object information table including data on a plurality of static objects having any one type among predetermined types existing on the movement path of the vehicle. A plurality of reference reference images provided in accordance with the Road Traffic Act as images representing the object information table including the static objects and each of the static objects are accessed to a storage device stored therein.

And at this time, the vehicle terminal, the difference between the first earth position that is the earth position of the first static object existing on the predicted first movement path of the vehicle and the current vehicle position that is the current earth position of the vehicle obtained from the GPS device starting photographing using the photographing device from a point in time when α has a value within a predetermined threshold value; extracting a first object image estimated to be the static object having any one of the predetermined types from the first image photographed by the photographing device; and selecting a first reference reference image corresponding to the extracted first object image from among the reference reference images.

A method for managing static objects on a vehicle movement path provided according to an aspect of the present invention is an object including data about a plurality of static objects having any one type among predetermined types existing on a vehicle movement path. A vehicle terminal including the object information table including the earth positions of the plurality of static objects as an information table, and a storage device in which a plurality of reference reference images provided according to the Road Traffic Act as an image representing each static object are stored. is intended to be used.

In the method for managing static objects on the vehicle movement path, the vehicle terminal includes: a first earth position that is the earth position of a first static object existing on the predicted first movement path of the vehicle; and a GPS device of the vehicle. starting photographing using the photographing device from a point in time when a difference between current vehicle positions, which is a current earth position, has a value within a predetermined threshold value; extracting, by the vehicle terminal, a first object image estimated to be the static object having any one of the predetermined types from the first image photographed by the photographing device; selecting, by the vehicle terminal, a first reference reference image corresponding to the extracted first object image from among the reference reference images; and providing, by the vehicle terminal, at least one of the extracted first object image and the selected first reference reference image to another computing device.

In addition, the method for managing a static object on a vehicle movement path may include: calculating, by a server, a degree of similarity between the extracted first object image and the selected first reference reference image; calculating, by the server, a degree of deterioration of the first static object represented by the first object image based on the calculated similarity; and generating, by the server, a repair or replacement schedule for the first static object according to the calculated degree of deterioration.

According to an aspect of the present invention, in a crowdsourcing platform system including a source data securing server 1, a learning dataset processing method for processing a learning dataset of artificial intelligence that recognizes a static object in a driving environment can be provided. have. In the processing method, the source data securing server 1 collects black box images based on crowd sourcing, removes data bias from the collected black box images, and de-identifies personal information, so that the collected A data collection step of generating refined image data from the black box image (S10); classifying, by the source data securing server 1, the refined image data based on the large classification categories constituting the learning data, and extracting effective image data from the refined image data (S20); and generating a training dataset from the refined image data and the valid image data (S50).

In this case, in the data collection step, the source data securing server 1 collects image data from a black box installed in a commercial vehicle operated by a driver panel, which is a commercial vehicle driver, and operates a vehicle equipped with a high-definition black box. Compensating the black box image by collecting image data from the moving vehicle; and supplementing, by the source data securing server 1, the black box image with image data purchased from a server of a passenger vehicle transport company.

In this case, in the data collection step, in order for the source data securing server 1 to remove data bias from the collected black box image, the black box is collected from a limited number of participants for each metropolitan area/metropolitan city when the first black box image is collected. collecting images; classifying regions of each image data by matching the collected data of the black box image with GPS information; classifying a road type of a place where each image data is collected by matching the collected data of the black box image with GPS information; classifying the vehicle type of the vehicle providing each of the black box images through a pre-stored vehicle DB; and classifying the collected black box images according to the region, the road type, and the vehicle type, and performing filtering on the collected black box images to prevent bias.

In this case, the data collection step may include: deleting, by the source data securing server 1, a section before and after the starting point and the destination point of each of the collected black box images in order to de-identify the personal information; deleting audio information other than the video from each of the collected black box images; de-identifying the vehicle numbers of the vehicle driving ahead and the vehicle in the next lane from each of the collected black box images; deleting a traffic accident scene from each of the collected black box images; performing pedestrian de-identification processing when stopping at a crosswalk on each of the collected black box images; and de-identifying the photographing time and location information in the black box image from each of the collected black box images.

At this time, the generated learning dataset may be classified into any one category of signs, lanes and crosswalks, traffic lights, and other marks.

In this case, the generated learning data set includes information on the source location, resolution, type of the photographed vehicle, recording time, the type of the photographed road, and the photographed area for the video data of the collected black box image. Information on URLs including the file name, shooting time, shooting location, vehicle speed at the time of shooting, weather at the time of shooting, the number of lanes on the captured road, and labeled images for each image data of the collected black box image Including, the width and height of the bounding box defining the area of the static object extracted from the respective image data, and the x and y coordinates of the upper left corner, and the class ID of the static object included in the bounding box. .

In this case, the large classification categories are further divided into a plurality of subcategories, respectively, and the step of extracting the data comprises building a learning data set for a learning model to be classified based on the large classification category, and the constructed learning data generating a learning model using the set; A static object related to the plurality of sub-categories is extracted from the valid image data by using the generated learning model, and a bounding box is information about a location of the extracted static object in the valid image data. ), generating an annotation on the bounding box.

In this case, the step of extracting the data includes classifying an image corresponding to the annotation and a video corresponding to the image into any one of the large classification categories according to the generated annotation. It may further include; generating a moving picture and pre-processed image data.

At this time, the crowdsourcing platform system further includes a promotion server (2), the promotion server, the personal information use agreement part for processing the personal information use consent process for image use and compensation payment; a reward part that provides cash to the individual in lieu of points according to a given cash reward system; and a content part that provides video collection promotion guide content to the individual.

A method for securing source data according to an aspect of the present invention is, first, a crowdsourcing-based basic data collection method, and includes a technology of pre-processing the collected data by collecting images previously taken by a general user with his/her own black box, and secondly, , includes technology to supplement and collect image data through the hiring of recruited commercial vehicles (taxi, truck, bus) and professional filming panels; have.

The method for generating text and annotations provided according to an aspect of the present invention includes the steps of automatically classifying and searching source data obtained using a dataset authoring tool for AI learning, and recognizing characters and images existing in the source data. It may include the step of automatically generating text and annotations from

The spatial scope for source data essential for object recognition AI learning data in autonomous driving and next-generation intelligent transportation service provided according to an aspect of the present invention includes highways, national roads, urban expressways in metropolitan areas and metropolitan areas, and roads with more than 6 lanes can do.

According to an aspect of the present invention, the content scope of the source data essential for object recognition AI learning data in autonomous driving and next-generation intelligent transportation service is road markings (lanes) and crossing indications, traffic lights, road traffic signs, and regulatory signs. , indicator signs, other signs, road construction section-specific signs, road surface colored guide lines, and ITS facilities.

Specifically, the road markings (lanes) and crossing directions are presented for the purpose of promoting the safety and smooth communication of road traffic and preserving the road structure. It can provide a function to convey the content of regulations or directives to

Specifically, the traffic lights are installed for the purpose of preventing danger on the road and securing traffic safety and smooth communication, and display signals such as progress, stop, change of direction, and caution with letters, symbols, or lights in relation to road traffic to display various types of traffic. It can perform the function of allocating priority to .

Specifically, the road sign is provided for the purpose of providing road users with various information necessary to promote traffic safety and smooth communication in a consistent and unified way and to protect road structures and road facilities. It is a traffic safety facility installed as a traffic safety facility or installed in organic and complementary combination with road markings and signals, providing a function to deliver cautions, regulations, and instructions to road users to achieve the purpose of traffic safety signs. can

Specifically, the source data classified into the other items may be determined as static data that affects the driving decision when the autonomous vehicle is driven. For example, it may be classified as a road construction section exclusive sign, road surface color guide line, and other ITS facilities.

The data collected according to an aspect of the present invention may be a real driving image that can be acquired by an image capturing device installed in a vehicle as a black box image. General black box image data of 30fps, HD (1366x768) or FHD (1920x1080) level of image data can be collected. Image data for learning can be extracted from image data taken from the vehicle and processed into a learning data set.

The black box-based driving video data is, firstly, affected by weather (rain, snow, cloudy, etc.) and time (night, day), secondly, severe light scattering by glass and raindrops, and thirdly, on the performance of the shooting camera module. The quality of the collected video is different depending on the model. Fourth, if the time setting of the black box itself is wrong, it is difficult to map the video recording time. In the case of supporting , there is left and right distortion of the image, and seventh, there is a problem that the time, position, and speed of the image are recorded simultaneously at the top or bottom of the captured image.

The user image collected according to an aspect of the present invention may be a traffic information image collected through crowdsourcing. To this end, the device provided according to the present invention may be configured to provide points and benefits to participants according to participation in traffic information collection.

The apparatus provided according to an aspect of the present invention may be configured to perform data purification on the collected user image data through GPS matching. In this case, it is possible to obtain image data for a given time period, for example, 1,000 hours, in which data deflection is prevented by region, time, weather, vehicle type, and road type.

In this case, the device may perform image de-identification processing for protecting personal information. For example, you can delete origin/destination clips. Image data can be obtained through image extraction from the refined image. Data at the large classification level can be classified through the data refining process. Information such as movement section and weather can be acquired through user input or primary inspection of the collected images.

<Entire system>

The source data collection method provided according to an aspect of the present invention includes the steps of firstly securing a black box image through a crowdsourcing method, and secondly performing GPS matching of the secured black box image to prevent data collection bias. Classification by region, vehicle type, and road type through The method may include supplementing the image data by identifying the requested image from the image obtained from the server of the image supplier and purchasing the identified requested image.

Specifically, when using the crowdsourcing platform, it is possible to obtain a lot of data quickly at a low price, but in order to prevent data bias, it is possible to pre-process the available data before processing the learning data, and to perform a separate personal information use consent procedure can run

Specifically, in the case of using the panel, there is an advantage that it is possible to prevent the collection of biased source data through panel recruitment and the quality of source data can be managed through the installation of separate equipment, but it takes a long time to secure the source data.

Specifically, in the case of the direct data purchase method, it is necessary to investigate the source data securing company, and it is more expensive than crowdsourcing or panel utilization. However, it is possible to obtain video data quickly without hiring a separate panel, and there is no need to hire a separate manpower.

<Crowdsourcing Platform>

The crowdsourcing platform system provided according to an aspect of the present invention includes (1) a promotion server for paying an individual additional points (or cash) according to video provision through additional video collection promotion, (2) large-capacity video data transmission It may include a file upload and additional server storage device provided for, and (3) a source data acquisition server.

The source data securing server collects data using (1) matching parts that match images and driving information with each other by matching and classifying previously collected driving information and image information, and (2) matching region, user, and driving information It may include a deflection removing part that performs a function for preventing deflection.

The promotion server includes (1) the personal information usage agreement part that handles the personal information usage agreement process for video use and compensation payment, and (2) the compensation part that provides cash to the individual in lieu of points according to the given cash compensation system , and (3) a content part for providing video collection promotion guide content to the individual.

The crowdsourcing platform system may be configured to collect 10 times the image data compared to the training data. In this case, it may further include a function of (1) collecting images according to image conditions including static objects, (2) analyzing image metadata for data matching classification, and enabling user input.

The crowdsourcing platform system may collect the image data by using a web hard such as the file upload and an additional server storage device to collect the image data.

In this case, the part of the consent to use of personal information may receive consent to use of image information from the individual.

In this case, the matching part may verify whether the image and the driving information match each other by verifying whether the GPS information is matched, and may accept the primary input of the user input method for identifying information included in the image. The first input may be an input for discriminating whether it is a local road, a highway, or a national road.

In this case, the reward part may be configured to perform a process of providing points or cash rewards to the individual according to the matching result.

In this case, the image data may be uploaded from the individual through a web hard such as the file upload and additional server storage device. The uploaded image can be used by the source data securing server.

In this case, the deflection removing part may perform a deflection prevention function by matching the previously owned driving information and additionally collected image information.

<Complementing video data of panel recruitment method>

The desired image data may not be included in the image data acquired through the crowdsourcing platform system. In order to overcome this, the collected panel may be used in the image data supplementation method according to an aspect of the present invention.

By classifying the source data acquired through the crowdsourcing platform system, and limiting the collected data by city/city/city, highway/national road, weather, and time period, the initial collected data can be supplemented through panel recruitment.

The selection of the panel may be made according to the source data obtained through the crowdsourcing platform system.

The panel may be divided into a commercial vehicle operator panel and a professional panel.

The commercial vehicle driver panel may be selected as a driver with a relatively long driving distance. The professional panel is a kind of hired panel and may mean a panel that collects high-definition image data through the installation of a QHD-class black box. The video data secured by the panels can be submitted online using Webhard or can be collected directly.

<Complementing video data by direct data purchase>

In the case of vehicles used in passenger car transportation business, video recording devices may be mandatory to identify traffic accidents, prevent crime, and secure evidence. Data can be purchased directly for passenger car transport operators such as city/intercity bus transport companies.

Purchased image data and image data may be refined. In the process of refining the purchased image data, original image data can be secured through region, vehicle type, road type classification, and de-identification processing through matching of black box image and GPS information. It is possible to extract an image in units of image frames from the purchased image data, and to classify large classification levels (crosswalks, signs, traffic lights) and secure image data through preprocessing image recognition.

<Data purification process>

The data purification method provided according to an aspect of the present invention may purify image data and image data secured using the above-described crowdsourcing platform system.

The source data securing server may secure original image data through region, vehicle type, road type classification, and de-identification processing through matching of the black box image and GPS information. In addition, image data can be obtained by extracting images in units of image frames and classifying them at the major classification level (crosswalks, signs, traffic lights) through pre-processing image recognition.

The source data securing server includes the steps of (1) collecting crowdsourcing-based black box images, (2) removing data bias in the collected images, (3) de-identification of some information in the collected images (4) extracting an image from the collected image, and (5) obtaining purified data from the extracted image.

The source data securing server performs the steps of (1) limiting the number of participants in each metropolitan area/metropolitan city when collecting the first black box image, (2) matching the reported black box image data and GPS information in order to prevent bias on the collected image data. Through ① metropolitan/metropolitan region classification, ② expressway/national road classification, and ③ vehicle type classification can be performed through the automobile DB.

The source data securing server may execute a process of performing de-identification processing including the following steps on the collected image data. In the first step, the sections before and after the departure and arrival points are deleted from the collected video, in the second step, audio information other than the video is deleted from the collected video, and in the third step, the vehicle of the vehicle driving ahead and the vehicle in the next lane from the collected video Numbers can be de-identified, and in the fourth step, the traffic accident scene is deleted before AI learning data processing, in the fifth step, pedestrian de-identification processing is performed when stopping at the crosswalk, and in the sixth step, the video is taken in the black box Time and location information can be de-identified. The above second to sixth steps may be executed for the driving image in the interval between the departure and arrival of the vehicle.

<Data design for artificial intelligence learning>

Data for artificial intelligence learning provided according to an aspect of the present invention may have the following characteristics.

AI learning data completed by one aspect of the present invention can be provided so that anyone can easily utilize it by opening it to the outside. Therefore, AI learning data can be designed by sufficiently considering data accessibility.

Meta information used in the present invention may be designed to store information for autonomous driving AI learning. In addition, it can be designed in consideration of the functions for the comparison and inspection of the annotation by the operator and the AI recognition result.

AI learning data completed by one aspect of the present invention can be divided into large categories (categories) referred to as signs, lanes and crosswalks, traffic lights, and other signs. The major classification of the sign may be further divided into a middle classification of a regulatory sign and an instruction sign. The large classification of lanes and crosswalks can be further divided into mid-classifications of center line, bus-only lane, U-turn, general lane, road edge area, course change restriction line, staggered crosswalk, diagonal crosswalk, and general crosswalk. The major classification of the traffic lights may be further divided into intermediate classifications of a vehicle signal 4 color light, a vehicle signal 3 color light, a vehicle signal 3 color light, a vehicle signal 2 color light, and a walking signal 2 color light. The major classification of other signs can be further divided into a middle classification of construction signs and variable information boards. Each of the major categories may be divided into one or more predetermined sub-categories, respectively.

In one embodiment of the present invention, in order to collect sufficient learning data for each class by category, it is possible to collect, for example, 3,000 hours or more of video source data using a crowdsourcing-based data collection platform. In an embodiment, the source image data may be classified by region and by classification criteria of a large classification and a medium classification of a category, and learning data may be constructed for each class of the intermediate classification.

<Type of AI data>

Data used in the AI system provided according to an aspect of the present invention may have the following characteristics.

The data set provided according to one aspect of the present invention is (1) original video (FHD resolution), (2) image (jpg), (3) annotation (json) (= annotation), (4) data set list ( json - including class types).

Annotation items include source location for video data, resolution, type of recorded vehicle (truck/bus/car, etc.), recording time, type of recorded road (expressway/national road), and recorded area (metropolitan area/metropolitan area). May be provided for records. In addition, the file name for each image data extracted from the video, the shooting time, the shooting location, the speed of the vehicle when shooting, the weather at the time of shooting, the number of lanes on the taken road, and the labeled image can be provided for recording of URLs. have. In addition, for a number of N classes extracted from each image, the class ID (large class -> middle class -> class) and the location of the bounding box (coordinates (x,y) and width and height of the upper left) are defined and presented. can

In one aspect of the present invention, a json format may be used as the annotation format. The open data set has a form that requires a separate tool such as matlab, but in the present invention, in consideration of versatility, a possible json format can be used without a separate tool.

<Building data for artificial intelligence learning>

The learning data production process provided in accordance with an aspect of the present invention includes a data collection step (S10), a data pre-processing step (S20), a data processing step (S30), a data verification step (S40), and the steps of creating a learning dataset ( S50) may be included.

In the data collection step ( S10 ), crowdsourcing-based black box image collection, bias removal for the collected image, and image de-identification processing may be sequentially performed. This may include the data cleaning process described above.

At this time, data of the user's black box image and the driving time and driving route (when the GPS module is installed) are collected, and the user's driving trip unit OD data (driving time, driving route, mileage) is collected through the crowdsourcing platform. This is possible. It is possible to prevent bias and secure source data by matching the previously owned driving information with the additionally collected image information.

At this time, in order to remove data bias on the collected image data, the number of participants per metropolitan area/metropolitan area is limited when collecting black box images for the first time, and ① metropolitan area/metropolitan area classification, ② highways through matching the reported black box image data and GPS information Classification of vehicle types can be performed through /national road classification, and ③ automobile DB. The collected images may be classified according to regions, road types, and vehicle types, and filtering may be performed on the collected images to prevent bias.

And, as described above, de-identification processing may be performed on the collected image data to protect personal information.

Meanwhile, the following may be considered when extracting still pictures from the acquired moving picture data. It is possible to remove duplicate photos caused by a specific point or traffic congestion by considering the distance based on the vehicle speed from the static object on the road that has already secured and stored its location.

In the data pre-processing step (S20), for the collected and refined image data, lanes/crosswalks, traffic lights, traffic signs, and other signs, which are major categories of learning data categories, are classified, and valid image data is extracted from the image data. can

The data pre-processing step S20 may include the following steps.

That is, in step S21, a training data set for a learning model to perform classification based on the prepared large classification category (for example, for a total of 109 classes for 4 large classifications and 56 medium classifications) is built, and through learning You can create a learning model.

Each of the large classification categories (classes) may be further divided into a plurality of subcategories (medium classification, small classification).

In step S22, static objects related to the plurality of sub-categories are extracted from the valid image data using the generated learning model, and information about the location of the extracted static objects in the valid image data A bounding box and an annotation on the bounding box may be automatically generated.

In step S23, by classifying the corresponding moving picture and the image according to the major classification according to the generated annotation, the moving picture and the pre-processed image data according to the large classification item may be generated.

In the data processing step S30, the inspector may check and correct the bounding box and annotations generated in the data pre-processing step S20. Key-In method, Selection method, Post-verification method, Preset method, Automatic input method, Cross Field Check, Related Field Copy, Reject Packet, Reject Page Various input methods can be supported.

In the data verification step (S40), data verification is performed by performing a full inspection by detecting an error rate using a deep learning algorithm, and the inspector directly confirms and corrects the learning data having abnormalities such as error detection.

Abnormal annotation data can be automatically searched through the deep learning learning model, and the data with errors detected in the learning log can be requested for correction through the data verification process, and a training data set is created based on the final inspection data. And it is possible to perform advanced performance of the deep learning learning model through the final verified training data set.

<Infra for production and storage of learning data>

According to an aspect of the present invention, an infrastructure system for producing and storing learning data may be provided. The infrastructure system may include (1) a process management system to which cooperation between artificial intelligence and workers is applied, and (2) a data collection and analysis system using an artificial intelligence AI recognition engine.

In the existing learning data building process, quality control is usually performed by relying on professional workers in a series of processes such as image processing, index information input and registration.

In the present invention, a plurality of workers and a crowd sourcing method are used so that a lot of learning data can be secured in a short period of time. Therefore, in order to secure data quality, process management automation for process standardization and standardization is required. Therefore, it is possible to secure the best quality of data for artificial intelligence learning by utilizing a process management system equipped with guidelines and guidelines for each target task (for easy reference by workers). A process control system may be a control system in which a person is part of the process.

The learning data generation technology provided according to an aspect of the present invention may consist of steps of data collection, data pre-processing, data processing, data verification, and training data set generation.

In this case, the data processing step may be made of a human inspector's verification and AI's collaborative process. Only the data that has passed the multiple-crossing inspection criteria can be processed after final production by selecting workers with high accuracy as inspectors through internal personnel inspection. All processes of the collected data can be performed by the process control platform and annotation tool. The primary annotation (object extraction, meta, and large-class video classification) is performed by artificial intelligence, and secondary verification and correction can be performed by manpower. After training the final AI algorithm, additional data set validation can be performed to ensure quality.

In this case, the above-described data collection and analysis system, as a system participating in some processes of the process management system, can learn to automatically classify and search target data by applying AI element technology to the acquired AI learning data. .

In the image data pre-processing step, for example, objects for 164 classes are recognized in the collected user video data, and automatic classification and annotation automation functions (meta generation) can be performed according to the major classification of the video data according to the recognized objects. .

A method for securing source data according to an aspect of the present invention is, first, a crowdsourcing-based basic data collection method, and includes a technology of pre-processing the collected data by collecting images previously taken by a general user with his/her own black box, and secondly, , includes technology to supplement and collect image data through the hiring of recruited commercial vehicles (taxi, truck, bus) and professional filming panels; have.

The method for generating text and annotations provided according to an aspect of the present invention includes the steps of automatically classifying and searching source data obtained using a dataset authoring tool for AI learning, and recognizing characters and images existing in the source data. It may include the step of automatically generating text and annotations from

The spatial scope for source data essential for object recognition AI learning data in autonomous driving and next-generation intelligent transportation service provided according to an aspect of the present invention includes highways, national roads, urban expressways in metropolitan areas and metropolitan areas, and roads with more than 6 lanes can do.

According to an aspect of the present invention, the content scope of the source data essential for object recognition AI learning data in autonomous driving and next-generation intelligent transportation service is road markings (lanes) and crossing indications, traffic lights, road traffic signs, and regulatory signs. , indicator signs, other signs, road construction section-specific signs, road surface colored guide lines, and ITS facilities.

Specifically, the road markings (lanes) and crossing directions are presented for the purpose of promoting the safety and smooth communication of road traffic and preserving the road structure. It can provide a function to convey the content of regulations or directives to

Specifically, the traffic lights are installed to prevent danger on the road and secure traffic safety and smooth communication. In relation to road traffic, various traffic types by displaying signals such as progress, stop, change of direction, etc. It can perform the function of allocating priority to .

Specifically, the road sign is provided for the purpose of providing road users with various information necessary to promote traffic safety and smooth communication in a consistent and unified way and to protect road structures and road facilities. It is a traffic safety facility installed as a traffic safety facility or installed in organic and complementary combination with road markings and signals, providing a function to deliver cautions, regulations, and instructions to road users to achieve the purpose of traffic safety signs. can

Specifically, the source data classified into the other items may be determined as static data that affects the driving decision when the autonomous vehicle is driven. For example, it may be classified as a road construction section exclusive sign, road surface color guide line, and other ITS facilities.

The data collected according to an aspect of the present invention may be a real driving image that can be acquired by an image capturing device installed in a vehicle as a black box image. General black box image data of 30fps, HD (1366x768) or FHD (1920x1080) level of image data can be collected. Image data for learning can be extracted from image data taken from the vehicle and processed into a learning data set.

The black box-based driving video data is, firstly, affected by weather (rain, snow, cloudy, etc.) and time (night, day), secondly, severe light scattering by glass and raindrops, and thirdly, on the performance of the shooting camera module. The quality of the collected video is different depending on the model. Fourth, if the time setting of the black box itself is wrong, it is difficult to map the video recording time. In the case of supporting , there is left and right distortion of the image, and seventh, there is a problem that the time, position, and speed of the image are recorded simultaneously at the top or bottom of the captured image.

User images collected according to an aspect of the present invention may be dozens of traffic information images through crowdsourcing. To this end, the device provided according to the present invention may be configured to provide points and benefits to participants according to participation in traffic information collection.

The apparatus provided according to an aspect of the present invention may be configured to perform data purification on the collected user image data through GPS matching. In this case, it is possible to obtain image data for a given time period, for example, 1,000 hours, in which data deflection is prevented by region, time, weather, vehicle type, and road type.

In this case, the device may perform image de-identification processing for protecting personal information. For example, you can delete origin/destination clips. Image data can be obtained through image extraction from the refined image. Data at the large classification level can be classified through the data refining process. Information such as movement section and weather can be acquired through user input or primary inspection of the collected images.

<Entire system>

The source data collection method provided according to an aspect of the present invention includes the steps of firstly securing a black box image through a crowdsourcing method, and secondly performing GPS matching of the secured black box image to prevent data collection bias. Classification by region, vehicle type, and road type through The method may include supplementing the image data by identifying the requested image from the image obtained from the image provider's server and purchasing the identified requested image.

Specifically, when using the crowdsourcing platform, it is possible to obtain a lot of data quickly at a low price, but in order to prevent data bias, it is possible to pre-process the available data before processing the learning data, and to perform a separate personal information use consent procedure can run

Specifically, in the case of using the panel, there is an advantage that it is possible to prevent the collection of biased source data through panel recruitment and the quality of source data can be managed through the installation of separate equipment, but it takes a long time to secure the source data.

Specifically, in the case of the direct data purchase method, it is necessary to investigate the source data securing company, and it is more expensive than crowdsourcing or panel utilization. However, it is possible to obtain video data quickly without hiring a separate panel, and there is no need to hire a separate manpower.

<Crowdsourcing Platform>

The crowdsourcing platform system provided according to an aspect of the present invention includes (1) a promotion server for paying an individual additional points (or cash) according to video provision through additional video collection promotion, (2) large-capacity video data transmission It may include a file upload and additional server storage device provided for, and (3) a source data acquisition server.

The source data securing server collects data using (1) matching parts that match images and driving information with each other by matching and classifying previously collected driving information and image information, and (2) matching region, user, and driving information It may include a deflection removing part that performs a function for preventing deflection.

The promotion server includes (1) the personal information usage agreement part that handles the personal information usage agreement process for video use and compensation payment, and (2) the compensation part that provides cash to the individual in lieu of points according to the given cash compensation system , and (3) a content part for providing video collection promotion guide content to the individual.

The crowdsourcing platform system may be configured to collect 10 times the image data compared to the training data. In this case, it may further include a function of (1) collecting images according to image conditions including static objects, (2) analyzing image metadata for data matching classification, and enabling user input.

The crowdsourcing platform system may collect the image data by using a web hard such as the file upload and an additional server storage device to collect the image data.

In this case, the part of the consent to use of personal information may receive consent to use of image information from the individual.

In this case, the matching part may verify whether the image and the driving information match each other by verifying whether the GPS information is matched, and may accept the primary input of the user input method for identifying information included in the image. The first input may be an input for discriminating whether it is a local road, a highway, or a national road.

In this case, the reward part may be configured to perform a process of providing points or cash rewards to the individual according to the matching result.

In this case, the image data may be uploaded from the individual through a web hard such as the file upload and additional server storage device. The uploaded image can be used by the source data securing server.

In this case, the deflection removing part may perform a deflection prevention function by matching the previously owned driving information and additionally collected image information.

<Complementing video data of panel recruitment method>

The desired image data may not be included in the image data acquired through the crowdsourcing platform system. In order to overcome this, the collected panel may be used in the image data supplementation method according to an aspect of the present invention.

By classifying the source data acquired through the crowdsourcing platform system, and limiting the collected data by city/city/city, highway/national road, weather, and time period, the initial collected data can be supplemented through panel recruitment.

The selection of the panel may be made according to the source data obtained through the crowdsourcing platform system.

The panel may be divided into a commercial vehicle operator panel and a professional panel.

The commercial vehicle driver panel may be selected as a driver with a relatively long driving distance. The professional panel is a kind of hired panel and may mean a panel that collects high-definition image data through the installation of a QHD-class black box. The video data secured by the panels can be submitted online using Webhard or can be collected directly.

<Complementing video data by direct data purchase>

In the case of vehicles used in passenger car transportation business, video recording devices may be mandatory to identify traffic accidents, prevent crime, and secure evidence. Data can be purchased directly for passenger car transport operators such as city/intercity bus transport companies.

Purchased image data and image data may be refined. In the process of refining the purchased image data, original image data can be secured through region, vehicle type, road type classification, and de-identification processing through matching of black box image and GPS information. It is possible to extract an image in units of image frames from the purchased image data, and to classify large classification levels (crosswalks, signs, traffic lights) and secure image data through preprocessing image recognition.

<Data purification process>

The data purification method provided according to an aspect of the present invention may purify image data and image data secured using the above-described crowdsourcing platform system.

The source data securing server may secure original image data through region, vehicle type, road type classification, and de-identification processing through matching of the black box image and GPS information. In addition, image data can be obtained by extracting images in units of image frames and classifying them at the major classification level (crosswalks, signs, traffic lights) through pre-processing image recognition.

The source data securing server includes the steps of (1) collecting crowdsourcing-based black box images, (2) removing data bias in the collected images, (3) de-identification of some information in the collected images (4) extracting an image from the collected image, and (5) obtaining purified data from the extracted image.

The source data securing server performs the steps of (1) limiting the number of participants in each metropolitan area/metropolitan city when collecting the first black box image, (2) matching the reported black box image data and GPS information in order to prevent bias on the collected image data. Through ① metropolitan/metropolitan region classification, ② expressway/national road classification, and ③ vehicle type classification can be performed through the automobile DB.

The source data securing server may execute a process of performing de-identification processing including the following steps on the collected image data. In the first step, the sections before and after the departure and arrival points are deleted from the collected video, in the second step, audio information other than the video is deleted from the collected video, and in the third step, the vehicle of the vehicle driving ahead and the vehicle in the next lane from the collected video Numbers can be de-identified, and in the fourth step, the traffic accident scene is deleted before AI learning data processing, in the fifth step, pedestrian de-identification processing is performed when stopping at the crosswalk, and in the sixth step, the video is taken in the black box Time and location information can be de-identified. The above second to sixth steps may be executed for the driving image in the interval between the departure and arrival of the vehicle.

<Data design for artificial intelligence learning>

Data for artificial intelligence learning provided according to an aspect of the present invention may have the following characteristics.

AI learning data completed by one aspect of the present invention can be provided so that anyone can easily utilize it by opening it to the outside. Therefore, AI learning data can be designed by sufficiently considering data accessibility.

Meta information used in the present invention may be designed to store information for autonomous driving AI learning. In addition, it can be designed in consideration of the functions for the comparison and inspection of the annotation by the operator and the AI recognition result.

AI learning data completed by one aspect of the present invention can be divided into large categories (categories) referred to as signs, lanes and crosswalks, traffic lights, and other signs. The major classification of the sign may be further divided into a middle classification of a regulatory sign and an instruction sign. The large classification of lanes and crosswalks can be further divided into mid-classifications of center line, bus-only lane, U-turn, general lane, road edge area, course change restriction line, staggered crosswalk, diagonal crosswalk, and general crosswalk. The major classification of the traffic lights may be further divided into intermediate classifications of a vehicle signal 4 color light, a vehicle signal 3 color light, a vehicle signal 3 color light, a vehicle signal 2 color light, and a walking signal 2 color light. The major classification of other signs can be further divided into a middle classification of construction signs and variable information boards. Each of the major categories may be divided into one or more predetermined sub-categories, respectively.

In one embodiment of the present invention, in order to collect sufficient learning data for each class by category, it is possible to collect, for example, 3,000 hours or more of video source data using a crowdsourcing-based data collection platform. In an embodiment, the source image data may be classified by region and by classification criteria of a large classification and a medium classification of a category, and learning data may be constructed for each class of the intermediate classification.

<Status of AI Data>

Data used in the AI system provided according to an aspect of the present invention may have the following characteristics.

The data set provided according to one aspect of the present invention is (1) original video (FHD resolution), (2) image (jpg), (3) annotation (json) (= annotation), (4) data set list ( json - including class types).

Annotation items include source location for video data, resolution, type of recorded vehicle (truck/bus/car, etc.), recording time, type of recorded road (expressway/national road), and recorded area (metropolitan area/metropolitan area). May be provided for records. In addition, the file name for each image data extracted from the video, the shooting time, the shooting location, the speed of the vehicle when shooting, the weather at the time of shooting, the number of lanes on the taken road, and the labeled image can be provided for recording of URLs. have. In addition, for a number of N classes extracted from each image, the class ID (large class -> middle class -> class) and the location of the bounding box (coordinates (x,y) and width and height of the upper left) are defined and presented. can

In one aspect of the present invention, a json format may be used as the annotation format. The open data set has a form that requires a separate tool such as matlab, but in the present invention, in consideration of versatility, a possible json format can be used without a separate tool.

<Building data for artificial intelligence learning>

The learning data production process provided in accordance with an aspect of the present invention includes a data collection step (S10), a data pre-processing step (S20), a data processing step (S30), a data verification step (S40), and the steps of creating a learning dataset ( S50) may be included.

In the data collection step ( S10 ), crowdsourcing-based black box image collection, bias removal for the collected image, and image de-identification processing may be sequentially performed. This may include the data cleaning process described above.

At this time, data of the user's black box image and the driving time and driving route (when the GPS module is installed) are collected, and the user's driving trip unit OD data (driving time, driving route, mileage) is collected through the crowdsourcing platform. This is possible. It is possible to prevent bias and secure source data by matching the previously owned driving information with the additionally collected image information.

At this time, in order to remove data bias on the collected image data, the number of participants per metropolitan area/metropolitan area is limited when collecting black box images for the first time, and ① metropolitan area/metropolitan area classification, ② highways through matching the reported black box image data and GPS information Classification of vehicle types can be performed through /national road classification, and ③ automobile DB. The collected images may be classified according to regions, road types, and vehicle types, and filtering may be performed on the collected images to prevent bias.

And, as described above, de-identification processing may be performed on the collected image data to protect personal information.

Meanwhile, the following may be considered when extracting still pictures from the acquired moving picture data. It is possible to remove duplicate photos caused by a specific point or traffic congestion by considering the distance based on the vehicle speed from the static object on the road that has already secured and stored its location.

In the data pre-processing step (S20), for the collected and refined image data, lanes/crosswalks, traffic lights, traffic signs, and other signs, which are major categories of learning data categories, are classified, and valid image data is extracted from the image data. can

The data pre-processing step S20 may include the following steps.

That is, in step S21, a training data set for a learning model to perform classification based on the prepared large classification category (for example, for a total of 109 classes for 4 large classifications and 56 medium classifications) is built, and through learning You can create a learning model.

Each of the large classification categories (classes) may be further divided into a plurality of subcategories (medium classification, small classification).

In step S22, static objects related to the plurality of sub-categories are extracted from the valid image data using the generated learning model, and information about the location of the extracted static objects in the valid image data A bounding box and an annotation on the bounding box may be automatically generated.

In step S23, by classifying the corresponding moving picture and the image according to the major classification according to the generated annotation, the moving picture and the pre-processed image data according to the large classification item may be generated.

In the data processing step (S30), the inspector may check and correct the bounding box and annotation generated in the data preprocessing step (S20). For all confirmation and correction work, all input work is fast and accurate. Key-In method, Selection method, Post-verification method, Preset method, Auto input method, Cross Field Check, Related Field Copy, Reject Packet, Reject Page, etc. method can be supported.

In the data verification step (S40), data verification is performed by performing a full inspection by detecting an error rate using a deep learning algorithm, and the inspector directly confirms and corrects the learning data having abnormalities such as error detection.

Abnormal annotation data can be automatically searched through the deep learning learning model, and the data with errors detected in the learning log can be requested for correction through the data verification process, and a training data set is created based on the final inspection data. And it is possible to perform advanced performance of the deep learning learning model through the final verified training data set.

<Infra for production and storage of learning data>

According to an aspect of the present invention, an infrastructure system for producing and storing learning data may be provided. The infrastructure system may include (1) a process management system to which cooperation between artificial intelligence and workers is applied, and (2) a data collection and analysis system using an artificial intelligence AI recognition engine.

In the existing learning data building process, quality control is usually performed by relying on professional workers in a series of processes such as image processing, index information input and registration.

In the present invention, a plurality of workers and a crowd sourcing method are used so that a lot of learning data can be secured in a short period of time. Therefore, in order to secure data quality, process management automation for process standardization and standardization is required. Therefore, it is possible to secure the best quality of data for artificial intelligence learning by utilizing a process management system equipped with guidelines and guidelines for each target task (for easy reference by workers). A process control system may be a control system in which a person is part of the process.

The learning data generation technology provided according to an aspect of the present invention may consist of steps of data collection, data pre-processing, data processing, data verification, and training data set generation.

In this case, the data processing step may be made of a human inspector's verification and AI's collaborative process. Only the data that has passed the multiple-crossing inspection criteria can be processed after final production by selecting workers with high accuracy as inspectors through internal personnel inspection. All processes of the collected data can be performed by the process control platform and annotation tool. The primary annotation (object extraction, meta, and large-class video classification) is performed by artificial intelligence, and secondary verification and correction can be performed by manpower. After training the final AI algorithm, additional data set validation can be performed to ensure quality.

In this case, the above-described data collection and analysis system, as a system participating in some processes of the process management system, can learn to automatically classify and search target data by applying AI element technology to the acquired AI learning data. .

In the image data pre-processing step, for example, objects for 164 classes are recognized in the collected user video data, and automatic classification and annotation automation functions (meta generation) can be performed according to the major classification of the video data according to the recognized objects. .

According to the present invention, it is possible to provide a technology for detecting images and determining the degree of visibility of drivers and autonomous vehicles according to the pavement condition of the road using the information about various static objects related to the road that has already been collected, and also, the identification is easy. It is possible to supplement and provide display contents in real time through image prediction for difficult road markings and lanes, and to provide a technology that provides the difference between predicted information and actual observed information to the administrator of the static object management system on the road. can

The real-time monitoring service of road bridge surface (road surface) display information provided through the present invention and user information through image prediction will improve public traffic safety.

1 is a diagram for explaining components in a vehicle and a relationship between components in a vehicle and a server according to an embodiment of the present invention.
2 is a flowchart illustrating a method of selecting data to be provided to a server to create a management schedule for a static object on a road according to an embodiment of the present invention.
3 is a flowchart of a method for generating a management schedule for a static object on a road in a server according to an embodiment of the present invention.
4 is a flowchart illustrating a method for generating a repair or replacement schedule for a lost object according to an embodiment of the present invention.
5 is a diagram for explaining a crowdsourcing platform system according to an embodiment of the present invention.
6 is a view for explaining a method of processing a learning dataset according to an embodiment of the present invention.
7 is a diagram for separately explaining a process for securing source image data according to an embodiment of the present invention.
8 schematically illustrates a learning data processing method according to an embodiment of the present invention.
9 is a diagram for explaining a data purification process according to an embodiment of the present invention.
10 is a diagram for explaining annotation according to an embodiment of the present invention.
11 shows information on a class ID and a position of a bounding box extracted from an arbitrary image according to an embodiment of the present invention.
12 is a view for explaining a process of performing the data pre-processing step of step S20 of FIG. 6 .
13 is a view for explaining a process of generating a learning model according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, the present invention is not limited to the embodiments described herein and may be implemented in various other forms. The terminology used in this specification is intended to help the understanding of the embodiment, and is not intended to limit the scope of the present invention. Also, singular forms used hereinafter include plural forms unless the phrases clearly indicate the opposite.

1 is a diagram for explaining components in a vehicle and a relationship between components in a vehicle and a server according to an embodiment of the present invention.

2 is a flowchart illustrating a method of selecting data to be provided to a server to create a management schedule for a static object on a road according to an embodiment of the present invention.

Hereinafter, it will be described with reference to FIGS. 1 and 2 together.

A vehicle terminal 210 and an autonomous driving device 220 may be installed in the vehicle 200 .

The vehicle terminal 210 may include a processing device 211 , a photographing device 212 , a storage device 213 , and a prediction device 214 .

The storage device 213 may store an object information table and a plurality of reference reference images.

The object information table may be an object information table including data on a plurality of static objects having any one type among predetermined types existing on a movement path of a vehicle. The object information table may include geolocations of the plurality of static objects.

The plurality of reference reference images may be provided according to the Road Traffic Act as an image representing each of the static objects.

The prediction device 214 may include a neural network learned according to a supervised learning method.

The neural network uses images for learning acquired by shooting from an information collection vehicle as input information for learning, and supervised learning using a label for each of one or more object images designated in each of the acquired images for learning. can

The autonomous driving device 220 may support autonomous driving of the vehicle 200 .

The processing device 211 may be configured to execute the following steps ( S200 ) to ( S230 ).

In step S200 , the processing device 211 obtains the vehicle 200 from the GPS device and the first earth position that is the earth position of the first static object existing on the predicted first movement path of the vehicle 200 . ) may start photographing using the photographing device 212 from a point in time when the difference between the current vehicle positions, which is the current earth position, has a value within a predetermined threshold value.

In step S210 , the processing device 211 extracts a first object image estimated to be the static object having any one of the predetermined types from the first image photographed by the photographing device 212 . can do.

In this case, step S210 may include a step of using the output of the learned neural network by inputting the first image captured by the processing device 211 to the learned neural network.

In step S220 , the processing device 211 may select a first reference reference image corresponding to the extracted first object image from among the reference reference images.

In step S230, the processing device 211 converts at least one of the extracted first object image and the selected first reference reference image to another computing device (eg, the autonomous driving device 220 or the server 300). ) can be provided to

In this case, when the other computing device is the autonomous driving device 220 , the autonomous driving device 220 displays the extracted first object image or the selected first object image on a display device (not shown) included in the vehicle 200 . The reference reference image may be displayed.

The step S220 may include the following steps S221 to S223.

In this case, the processing device 211 may access the storage device 213 .

In step S221 , the processing device 211 may select a first group of static objects existing within a predetermined distance from the first earth location from among the static objects stored in the object information table.

In step S222 , the processing device 211 may select a first group of reference reference images corresponding to the selected first group of static objects from among the plurality of reference reference images.

In operation S223 , the processing device 211 may determine, as the first reference reference image, the one having the highest similarity with the first object image among the selected first group of reference reference images.

3 is a flowchart of a method for generating a management schedule for a static object on a road in a server according to an embodiment of the present invention.

The following steps ( S300 ) to ( S330 ) may be steps performed by the server 300 when the other computing device of the step ( S230 ) is the remote server 300 .

In step S300 , the server 300 may receive the first object image and the first reference reference image from the vehicle terminal 210 .

In step S310, the server 300 may calculate a degree of similarity between the extracted first object image and the selected first reference reference image.

In step S320 , the server 300 may calculate the degree of deterioration of the first static object indicated by the first object image based on the calculated similarity.

In step S330 , the server 300 may generate a repair or replacement schedule for the first static object according to the calculated degree of deterioration.

4 is a flowchart illustrating a method for generating a repair or replacement schedule for a lost object according to an embodiment of the present invention.

The server 300 may be configured to access a database including the object information table.

In step S400 , the vehicle terminal 210 obtains the vehicle 200 from the GPS device and the first earth position that is the earth position of the first static object existing on the predicted first movement path of the vehicle 200 . ) may start photographing using the photographing device 212 from a point in time when the difference between the current vehicle positions, which is the current earth position, has a value within a predetermined threshold value.

In step S410, the vehicle terminal 210 is all object images estimated to be the static objects having any one of the predetermined types from the first image photographed by the photographing device 212. One group of object images can be extracted.

In step S420 , the vehicle terminal 210 may provide information about the first earth position and the extracted first group object images to the other computing device (ie, the server 300 ).

In step S430 , the server 300 may select a first group of static objects existing within a predetermined distance from the first earth location from among the static objects stored in the object information table.

In step S440 , the server 300 may determine whether there is a lost object that is a static object that does not correspond to the object images of the first group among the static objects of the first group.

In step S450, the server 300 may create a repair or replacement schedule for the lost object.

5 is a diagram for explaining a crowdsourcing platform system according to an embodiment of the present invention.

The crowdsourcing platform system 100 may process a learning dataset of artificial intelligence that recognizes static objects in a driving environment.

The crowdsourcing platform system 100 may include a source data securing server 1 , a promotion server 2 , and a storage device 3 .

The source data securing server 1 may include a matching part 11 and a bias removal part 12 .

The matching part 11 may match the image and the driving information with each other by matching and classifying the previously collected driving information and the image information. In addition, the matching part 11 can authenticate whether the image and driving information match each other by verifying whether the GPS information is matched, and can accept the primary input of the user input method for identifying information included in the image. have. The first input may be an input for discriminating whether it is a local road, a highway, or a national road.

The bias removal part 12 may perform a function for preventing bias in data collection by using region, user, and driving information matching. That is, the deflection removing part 12 may perform a deflection prevention function by matching the previously owned driving information and additionally collected image information.

The promotion server 2 may pay the individual additional points (or cash) according to the provision of the image through the additional image collection promotion. To this end, the promotion server 2 may include a personal information use agreement part 21 , a compensation part 22 , and a content part 23 .

The personal information use consent part 21 may perform an individual's personal information use consent process for image use and compensation payment.

The reward part 22 may perform a process of providing the individual with cash instead of points according to a given cash reward scheme. Alternatively, the reward part 22 may perform a process of providing a point or cash reward to the individual according to the matching result of the matching part 11 of the source data securing server 1 .

The content part 23 may perform a process of providing the image collection promotion guide content to the individual.

The storage device 3 may provide file upload and additional functions for large-capacity image data transmission.

The storage device 3 may include an image DB 31 , a driving technology DB 32 , and a vehicle model identification database 33 .

In the image DB 31 , an original image, for example, a black box image, and a real driving image acquired by an image photographing device installed in a vehicle may be stored. General black box image data of 30fps, HD (1366x768) or FHD (1920x1080) level of image data can be collected. In addition, the image DB 31 may store, for example, a traffic information image collected through crowdsourcing. In addition, images collected through the collected panels may be stored in the image DB 31 .

The driving technology DB 32 may store driving information. The driving information is OD data of the driving trip unit of the user through the crowdsourcing platform, and may include information such as driving time, driving route, and mileage.

The vehicle model identification DB 33 may classify and store the vehicle model with respect to the black box image data.

6 is a view for explaining a method of processing a learning dataset according to an embodiment of the present invention.

In step S10, the source data securing server 1 collects black box images based on crowdsourcing, removes data bias from the collected black box images, and de-identifies personal information, so that the collected Refined image data can be generated from the black box image.

At this time, the source data securing server 1 collects image data from a black box installed in a commercial vehicle operated by a driver panel, which is a commercial vehicle driver, and image data from a vehicle operated by a professional panel operating a vehicle equipped with a high-definition black box. can be collected to supplement the black box image. In addition, the source data securing server 1 may supplement the black box image with image data purchased from a server of a passenger vehicle transport company.

At this time, in the case of video data supplementation, the source data acquired by the source data securing server 1 is first classified, and the collected data is limited by city/city/urban, highway/national road, weather, and time period to supplement the initial collected data through panel recruitment. can do.

In step S20, the source data securing server 1 may classify the refined image data based on the large classification categories constituting the learning data, and extract effective image data from the refined image data.

In step S30, the inspector may check and correct the bounding box and annotation generated in step S20, which is the data pre-processing step. For all confirmation and correction work, all input work is fast and accurate. Key-In method, Selection method, Post-verification method, Preset method, Auto input method, Cross Field Check, Related Field Copy, Reject Packet, Reject Page, etc. method can be supported.

At this time, in step (S30') to perform step (S30), inspector management can be made. The inspector management step may include registering a user, allocating an inspector, and allocating data.

In the data verification step (S40), the data verification performs a full inspection by detecting an error rate using a deep learning algorithm, and the inspector directly confirms and corrects the learning data having an error such as error detection.

In step S50, a training dataset (data sets A, B, etc.) may be generated based on the data on which the verification procedure is completed.

7 is a diagram for separately explaining a process for securing source image data according to an embodiment of the present invention.

In order to secure the source image data, three stages can be divided and explained.

In the first step, the source data securing server 1 may secure basic data based on crowdsourcing. In this case, the crowdsourced image data may be prepared by building an image collection infrastructure, collecting crowdsourcing-based image data, and then refining and classifying the data.

In the second step, as a complement to the first step, it is possible to secure the image data of the panel recruitment method. In this case, after recruiting and selecting a panel (commercial vehicles (taxi, truck, bus, etc.)), a photographing device may be installed in the vehicle of the selected panel. By collecting image data from the installed photographing device, it is possible to additionally prepare image data through the panel.

In the third stage, as a further complement to the first and second stages, data can be directly purchased. In this case, the image data may be additionally prepared by researching the image supplier, identifying the requested image, and purchasing the image data.

The source image data may include crowdsourced image data, panel-collected image data, and purchased image data each secured in the first, second, and third steps described above.

8 schematically illustrates a learning data processing method according to an embodiment of the present invention.

Traffic image data can be collected through crowdsourcing. In this case, points and benefits may be provided to traffic image providers (ie, traffic information collection participants).

The collected image data may be stored in the image DB 31 .

By synthesizing the image DB 31 , the driving record DB 32 , and the vehicle model identification DB 33 , image filtering may be performed under conditions such as region, type, time, date, and the like. This filtered image can be used as source data for AI learning data processing.

9 is a diagram for explaining a data purification process according to an embodiment of the present invention.

The following steps may be steps performed by the source data securing server 1 .

In step S110, a black box image may be collected based on crowdsourcing. In this case, when collecting the first black box image, the black box image may be collected from a limited number of participants for each metropolitan area/metropolitan city.

In step S120, data bias may be removed from the black box image.

Specifically, step S120 may include the following steps S121 to S124.

In step S121, a region (metropolitan area/metropolitan city) of each image data may be classified by matching the collected data of the black box image with GPS information.

In step S122, a road type (highway/national road) of a place where each image data is collected may be classified through the matching of the collected black box image data and GPS information.

In step S123, classification of the vehicle type of the vehicle providing each of the black box images may be performed through a pre-stored vehicle DB (vehicle model identification DB).

In step S124, the collected black box images may be classified according to the region, the road type, and the vehicle type, and the collected black box images may be filtered so that no bias occurs.

In step S130, personal information may be de-identified in the image from which the bias is removed.

Specifically, step S130 may include the following steps S131 to S136.

In step S131, sections before and after the departure point and the destination point of each of the collected black box images may be deleted. In this case, each of the collected black box images may mean an image from which deflection has been removed.

In step S132, audio information other than the video may be deleted from each of the collected black box images.

In step S133, the vehicle number of the vehicle driving ahead and the vehicle in the next lane may be de-identified from each of the collected black box images.

In step S134, a traffic accident scene may be deleted from each of the collected black box images.

In step S135, pedestrian de-identification processing may be performed when the crosswalk is stopped on each of the collected black box images.

In step S136, the photographing time and location information in the black box image in each of the collected black box images may be de-identified.

In step S140 , an image may be extracted from the de-identified image. That is, an image in units of image frames may be extracted.

In step S150, valid image data may be selected from the extracted image, and static objects for each sub-category may be extracted from the selected valid image.

10 is a diagram for explaining annotation according to an embodiment of the present invention.

After the step S150, an annotation on a bounding box, which is information about a position of the extracted static object in the valid image data, may be generated.

Annotation items may be provided for recording video information, image information, and class information.

In the case of video information, the annotation items are the source location, resolution, type of vehicle (truck/bus/car, etc.) for video data, recording time, type of recorded road (expressway/national road), and area ( Metropolitan area/metropolitan area) can be provided for records.

In the case of image information, the annotation items include a file name, shooting time, shooting location, vehicle speed at the time of shooting, weather at the time of shooting, the number of lanes on the captured road, and labeled images for each image data extracted from the video. A URL may be provided for recording.

In addition, the annotation item specifies the class ID (large classification -> medium classification -> class) and the position of the bounding box (coordinates (x,y) and width and height of the upper left) for a number of N classes extracted from each image. can be defined and presented.

11 shows information on a class ID and a position of a bounding box extracted from an arbitrary image according to an embodiment of the present invention.

Reference numerals B1 to B4 denote bounding boxes, and reference numerals C1 to C4 denote annotations for each bounding box B1 to B4.

Annotations may present information about the class ID and the location of the bounding box. For example, reference numeral C1 relates to the bounding box B1, and the class ID is a variable information board, and the location information of the bounding box B1 is (x=570, y=274) (width=618, height=310). can For example, reference numeral C2 relates to the bounding box B2, and the class ID is a center line, and the location information of the bounding box B2 may be (x=242, y=550) (width=523, height=380). .

12 is a view for explaining a process of performing the data pre-processing step of step S20 of FIG. 6 .

In this case, the large classification categories may include lanes/crosswalks, traffic lights, traffic signs, and other signs.

Each of the large classification categories may be further divided into a plurality of sub-categories.

Step S20 may include the following steps S21 to S23.

In step S21, a learning data set for a learning model to be classified based on the large classification category may be constructed, and a learning model may be generated using the constructed learning data set.

In step S22, static objects related to the plurality of sub-categories are extracted from the valid image data by using the generated learning model, and information on the location of the extracted static objects in the valid image data You can create annotations about a bounding box that is (coordinate).

In step S23, according to the generated annotation, by classifying the image corresponding to the annotation and the video corresponding to the image into any one of the major categories, preprocessing with the video according to the major category image data can be created.

13 is a view for explaining a process of generating a learning model according to an embodiment of the present invention.

Steps S10 to S40 are the same as described above with reference to FIG. 6 .

In step S40, the assigned inspector may perform data search and total inspection through coordinates.

Thereafter, when an error is detected by checking the learning log, the data verification step ( S40 ) is performed again, and when no error is detected, a learning model may be generated. In other words, it is possible to create a training dataset through the data that has completed the final inspection, and through this, it is possible to advance the deep learning learning model.

The generated learning dataset may be classified into any one category of signs, lanes and crosswalks, traffic lights, and other signs.

The generated learning data set may include information about the source location, resolution, type of the photographed vehicle, recording time, the type of the photographed road, and the photographed area for the video data of the collected black box image. .

In addition, the generated learning dataset includes a file name for each image data of the collected black box image, shooting time, shooting location, vehicle speed at the time of shooting, weather at the time of shooting, the number of lanes on the captured road, and labeled It may include information about the URL containing the image.

In addition, the generated training dataset includes the width and height of the bounding box defining the area of the static object extracted from the respective image data, and the x and y coordinates of the upper left, and the class ID of the static object included in the bounding box. may include.

By using the above-described embodiments of the present invention, those skilled in the art will be able to easily implement various changes and modifications within the scope without departing from the essential characteristics of the present invention. The content of each claim in the claims may be combined with other claims without reference within the scope that can be understood through this specification.

Claims (8)

processing unit;
photographing device; and
The object information table including geolocations of the plurality of static objects as an object information table including data on a plurality of static objects having any one type among predetermined types existing on the movement path of the vehicle; and a storage device in which a plurality of reference reference images provided according to the Road Traffic Act are stored as images representing each of the static objects.
includes,
the processing device,
The difference between the first earth position, which is the earth position of the first static object existing on the predicted first movement path of the vehicle, and the current vehicle position, which is the current earth position of the vehicle obtained from the GPS device, is within a predetermined threshold. starting photographing using the photographing device from a point in time having a value;
extracting a first object image estimated to be the static object having any one of the predetermined types from the first image photographed by the photographing device;
selecting a first reference reference image corresponding to the extracted first object image from among the reference reference images; and
providing at least one of the extracted first object image and the selected first reference reference image to another computing device;
intended to run
vehicle terminal. According to claim 1,
The other computing device is an autonomous driving device supporting autonomous driving of the vehicle,
the autonomous driving device is configured to display the extracted first object image or the selected first reference reference image on a display device included in the vehicle;
vehicle terminal. According to claim 1,
The step of selecting the first reference reference image,
selecting a first group of static objects existing within a predetermined distance from the first earth location from among the static objects stored in the object information table;
selecting a first group of reference reference images corresponding to the selected first group of static objects from among the plurality of reference reference images; and
determining, as the first reference reference image, the one having the highest similarity with the first object image among the selected first group of reference reference images;
containing,
vehicle terminal. According to claim 1,
The other computing device is a remote server,
The server is
calculating a similarity between the extracted first object image and the selected first reference reference image;
calculating an aging degree of the first static object represented by the first object image based on the calculated similarity; and
generating a repair or replacement schedule for the first static object according to the calculated degree of deterioration;
intended to run
vehicle terminal. According to claim 1,
The vehicle terminal further comprises a prediction device,
The prediction device includes a neural network learned according to a supervised learning method,
The neural network is supervised learning by using the learning images obtained by shooting in the information collection vehicle as input information for learning, and using a label for each of one or more object images designated in each of the acquired learning images. ,
The step of extracting the first object image comprises the step of inputting the photographed first image to the learned neural network and using the output of the learned neural network,
vehicle terminal. According to claim 1,
The processing device is
extracting, from the first image, a first group of object images, which are all object images estimated to be the static objects having any one of the predetermined types; and
providing information about the first earth position and the extracted first group of object images to the other computing device;
can run more
The other computing device is a server configured to access a database including the object information table,
The server is
selecting a first group of static objects existing within a predetermined distance from the first earth location from among the static objects stored in the object information table;
determining whether there is a lost object that is a static object that does not correspond to the object images of the first group among the static objects of the first group; and
generating a repair or replacement schedule for the lost object;
intended to run
vehicle terminal. receiving, by the server, a first object image and a first reference reference image from the vehicle terminal;
calculating, by the server, a degree of similarity between the extracted first object image and the selected first reference reference image;
calculating, by the server, a degree of deterioration of the first static object represented by the first object image based on the calculated similarity; and
generating, by the server, a repair or replacement schedule for the first static object according to the calculated degree of deterioration;
includes,
The vehicle terminal is an object information table including data on a plurality of static objects having any one of predetermined types existing on a movement path of the vehicle, and includes geolocations of the plurality of static objects. The object information table and a plurality of reference reference images provided according to the Road Traffic Act as an image representing each static object are accessed to a storage device stored therein,
The vehicle terminal,
The difference between the first earth position, which is the earth position of the first static object existing on the predicted first movement path of the vehicle, and the current vehicle position, which is the current earth position of the vehicle obtained from the GPS device, is within a predetermined threshold value. starting photographing using the photographing device from a point in time having ;
extracting a first object image estimated to be the static object having any one of the predetermined types from the first image photographed by the photographing device; and
selecting a first reference reference image corresponding to the extracted first object image from among the reference reference images;
intended to run
A method of creating a management schedule for static objects on the road. The object information table including geolocations of the plurality of static objects as an object information table including data on a plurality of static objects having any one type among predetermined types existing on the movement path of the vehicle; and a vehicle terminal including a storage device in which a plurality of reference reference images provided according to the Road Traffic Act are stored as an image representing each static object. As a method,
A difference between a first earth position that is the earth position of a first static object existing on the predicted first movement path of the vehicle by the vehicle terminal and a current vehicle position that is the current earth position of the vehicle obtained from the GPS device is a predetermined difference. starting photographing using the photographing apparatus from a point in time having a value within a threshold value of ;
extracting, by the vehicle terminal, a first object image estimated to be the static object having any one of the predetermined types from the first image photographed by the photographing device;
selecting, by the vehicle terminal, a first reference reference image corresponding to the extracted first object image from among the reference reference images;
providing, by the vehicle terminal, at least one of the extracted first object image and the selected first reference reference image to another computing device;
calculating, by the server, a degree of similarity between the extracted first object image and the selected first reference reference image;
calculating, by the server, a degree of deterioration of the first static object represented by the first object image based on the calculated similarity; and
generating, by the server, a repair or replacement schedule for the first static object according to the calculated degree of deterioration;
containing,
A method for managing static objects on the vehicle movement path.

Images