KR102562757B1 - Prediction and recognition method of road marking information and road maintenance method - Google Patents

Abstract

Extracting a first object image estimated to be a static object on the road having any one of predetermined types from a first image captured by a photographing device of a vehicle, as an image representing each static object, according to the Road Traffic Act selecting a first reference image corresponding to the extracted first object image from among a plurality of reference reference images provided according to the method; and selecting at least one of the extracted first object image and the selected first reference image. Disclosed is a vehicle terminal capable of executing steps provided to other computing devices and a method for managing static objects on a vehicle movement path using the vehicle terminal.

Description

Prediction and recognition method of road marking information and road maintenance method

The present invention uses computing technology to recognize road surface marking information based on static object AI data acquired in relation to road driving such as lanes/crosswalks, traffic lights, road signs, etc. It relates to technology for generating information for maintenance.

AI data, which is a static object in the driving environment, is an aptitude test item for autonomous vehicles. In order for self-driving cars to drive, the first item in the cognitive-judgment-control phase is cognition, and the driving environment static objects (lanes, crosswalks, traffic lights, road signs) Corresponds to the department test items. The development of static object recognition technology is required as the first step for the commercialization of autonomous vehicles.

As the technology related to self-driving cars develops, the sequential mandatory installation of various terminals and sensors necessary for self-driving is being promoted at home and abroad. and laws and systems in the field of security and authentication are expected to be complete.

After completing the commercialization of the Advanced Driver Assistance System (ADAS), which is a level 2 autonomous driving technology that currently assists driving, the implementation of some level 3 to 4 autonomous driving technologies is expanding, centering on overseas automakers.

Representative autonomous driving technologies include the Lane Keeping Assist System (LKAS) technology, which recognizes the lane ahead through an image recognition sensor and controls the steering wheel to maintain the lane, and recognizes signs to adjust speed and direction By combining the Traffic Sign Recognition (TSR) technology that supports switching and overtaking, the headway distance control function, and the lane keeping function, it recognizes the driving trajectory of surrounding vehicles even when the lane is not visible and maintains a certain distance from the vehicle in front. and traffic jam assist (TJA) technology. The present invention relates to the lane keeping assistance system and road sign recognition.

In order to manufacture self-driving cars of level 3 to 4 or higher, where the responsibility of the driver is rapidly reduced, cameras, radars (Radio Detection and Ranging, RADAR), and LIDAR (Light Detection and Ranging, LIDAR) that recognize static and dynamic obstacles are required. Driving environment recognition sensor and GPS-based positioning, precision maps that provide information such as road facilities, signals, and signs, V2V (Vehicle to Vehicle) and V2I (Vehicle to Infra) that exchange various facilities, environments, and traffic information, etc. element technology is additionally required.

In order to improve the completeness of autonomous driving technology, it is necessary to improve the recognition performance of automotive AI for static objects on the road such as lanes, crosswalks, and traffic lights/road signs. As the level of autonomous driving goes up, the dependence on artificial intelligence will continue to increase. In autonomous driving, static object recognition is very important, and driving by recognizing lanes is a way to improve control perfection. Accordingly, it is necessary to secure a dataset and ground truth information to acquire lane information.

In order to advance autonomous driving, autonomous cooperative driving, which acquires surrounding infrastructure information and performs control through convergence with the vehicle sensor, is being introduced, and it is possible to acquire traffic light information through vehicle-to-vehicle communication, raising the level of autonomous driving. As autonomous cooperative driving is being introduced, it is possible to obtain traffic light information through vehicle-to-vehicle communication, but it is difficult to guarantee completeness of data such as communication delay. Therefore, the recognition of traffic light information can solve the dilemma phenomenon at intersections, etc., but the final control panel of an autonomous vehicle needs to be judged through the vehicle's magnetic sensor, so development of traffic light recognition using image sensors is essential. Therefore, for deep learning-based development, high-quality traffic light datasets in many different environments are required. Although various services are possible with the introduction of precision maps, there may be situations where it is difficult to accurately determine the location of an autonomous vehicle using only sensors. Accurately stopping at a crosswalk to protect pedestrians can be done based on precise map information, but it is necessary to finally determine the stop line in the vehicle, and high-quality data is needed in many different environments.

For autonomous driving and next-generation intelligent traffic service (C-ITS), it is required to recognize various static objects sensed by cameras while driving. In autonomous driving, it is necessary to recognize various types of lanes in various environments in order to determine a drivable path and control a lateral vehicle for driving along the corresponding path. In addition, self-driving vehicles need to sense and recognize traffic lights, road signs, and crosswalks through cameras in order to recognize various road traffic conditions.

Obtaining and recognizing road-related information through a camera while a vehicle is driving is based on the premise that the information on the road and its surroundings is maintained undamaged. Over time, signs such as crosswalks, for example, may be damaged or one-minute signs may be damaged. It is necessary to quickly identify and restore these damaged facts, and a technology that supports the identification of these facts is required.

As an example, as a result of a survey of 126 'traffic accident-prone school zones' in which two or more traffic accidents occurred or one or more deaths occurred in one year, 75.4% of the total was found to be a problem caused by obsolescence such as road markings (2020, Ministry of Public Administration and Security). Therefore, appropriate monitoring and management of road markings and appropriate driver information assistance means are required.

In the present invention, it is possible to provide a technique for detecting an image of a driver and an autonomous vehicle according to the pavement condition of the road and determining the degree of visibility using already collected information on various static objects related to the road, and also difficult to identify. Through image prediction of road surface markings, lanes, etc., display contents can be supplemented and provided in real time, and the difference between predicted information and actually observed information is to be provided to the manager of the static object management system on the road. .

A static object recognition method on a road provided according to one aspect of the present invention may be implemented using a convolutional neural network (CNN) algorithm. The convolutional neural network is a kind of artificial neural network. The artificial neural network is a statistical learning algorithm inspired by neural networks in biology in machine learning and cognitive science. An artificial neural network is a model that has problem-solving ability by changing synaptic coupling strength through learning by artificial neurons that form a network by synaptic coupling.

The method for generating road management information provided according to one aspect of the present invention may generate and provide information on whether road surface markings need to be repainted, whether safety signs are obsolete, whether traffic signs are obsolete, and whether anti-skid pavement is obsolete.

The use of multiple hidden layers in an artificial neural network is deep learning, which has recently emerged in big data-based machine learning. Deep learning is easy to find non-linear patterns that are difficult to extract in linear regression analysis, and because of these characteristics, it has become a very powerful technology for computer vision and language learning. In the case of computer vision, since images are used as input data, a different structure is required than when dealing with simple structured data, and convolutional neural networks reflect this. The convolutional neural network includes an additional hidden side called convolution and a device for pattern recognition called max pooling.

In the present invention, convolutional neural network technology used for recognizing patterns in images is used to automatically recognize specific patterns related to the degree of aging and visibility of road marking information in captured images and data sets.

The convolutional neural network technology adopted according to one aspect of the present invention utilizes an existing image pattern recognition model, and a support vector machine algorithm is commonly used in this structure. The support vector machine algorithm is used to increase the accuracy of the analysis by recognizing an object necessary for analysis among a plurality of objects present in the image and constraining the position of the object to a box shape.

The road management information generation method provided according to one point of the present invention not only recognizes the degree of deterioration of road markings based on images, but also provides a model for predicting the corresponding road marking information based on the images, including a map learning is used

Unlike unsupervised learning, supervised learning requires that the training data be labeled, and the model is trained based on the "correct answers" in the training data. In the present invention, learning data is generated based on the correct answer data and image data presented in the Road Traffic Act, and based on this, it is possible to find correlations and patterns between the analysis result of the image-based deep learning model and the aging display-correct answer data.

According to one aspect of the present invention, data for sustainable road maintenance and improvement may be provided.

At this time, in particular, data capable of evaluating whether or not the facility is obsolete may be provided. As specific technologies for this purpose, ① image prediction technology for aging bridge surface (road surface) markings, ② road facility safety evaluation technology, and ③ performance change estimation and correction and reconstruction time prediction technology through monitoring of bridge surface (road surface) pavement and visibility are provided. can do.

In addition, in particular, data for supporting autonomous driving may be provided. Specifically, an optimal bridge (road) pavement luminance and visibility determination technology for autonomous vehicles can be provided.

A method for predicting static objects on a road provided according to one aspect of the present invention uses an image recognition-based convolutional neural network algorithm to compare an aging road surface image, which is difficult to identify, with matching data provided by the Road Traffic Act to determine the degree of aging. Through measurement and prediction, the corresponding road marking information can be delivered to the driver or autonomous driving service through API in real time.

In addition, the road facility safety evaluation method provided according to one aspect of the present invention is based on the output obtained from the image prediction technology of the aging bridge surface (road surface) display, and the visibility and It is possible to provide a solution to evaluate the road environment so that the safety level can be evaluated by presenting the criteria for determining the degree of deterioration in each stage (best - normal - need for improvement - urgent). In addition, it is possible to estimate the performance change of the static object according to the monitoring of the level of pavement and visibility of the static object, and to predict the correction/renovation time for repair of the static object that has deteriorated.

In addition, the monitoring method provided according to one aspect of the present invention manages the collection time of existing data and the history of safety evaluation of geolocation data (Geo data) based on the output obtained from the road facility safety evaluation method. Provides, in real time, whether the static objects need to be repackaged or repaired to private institutions and government agencies.

In addition, according to one aspect of the present invention, the optimal bridge (road) pavement luminance and visibility determination support information providing method for autonomous vehicles is provided based on the output of the image prediction technology of the aging bridge (road) display. Additional information related to the improvement of the recognition rate of the image sensor of the road surface can be provided so that it can contribute to the improvement of the accuracy of image recognition through the camera of the driving vehicle.

A vehicle terminal provided according to one aspect of the present invention includes a processing device; filming device; and an object information table including data on a plurality of static objects having one of predetermined types existing on a moving route of the vehicle, the object information table including geolocations of the plurality of static objects. , and a storage device storing a plurality of standard reference images provided according to the Road Traffic Act as images representing each of the static objects.

At this time, the processing device determines the difference between the first geolocation, which is the geolocation of the first static object existing on the predicted first movement path of the vehicle, and the current vehicle position, which is the current geolocation of the vehicle obtained from the GPS device. starting photographing using the photographing device from the point of time when a has a value within a predetermined threshold value; extracting a first object image estimated to be the static object having one of the predetermined types from a first image captured by the photographing device; selecting a first reference image corresponding to the extracted first object image from among the reference images; and providing at least one of the extracted first object image and the selected first reference reference image to another computing device.

At this time, the other computing device is an autonomous driving device supporting autonomous driving of the vehicle, and the autonomous driving device displays the extracted first object image or the selected first reference reference image on a display device included in the vehicle. may be made to do.

In this case, the selecting of the first reference image may include selecting static objects of a first group existing within a predetermined distance from the first geographic location among the static objects stored in the object information table; selecting a first group of standard reference images corresponding to the selected first group of static objects from among the plurality of standard reference images; and determining, as the first reference reference image, one having the highest similarity to the first object image among the selected reference reference images of the first group.

At this time, the other computing device is a remote server, and the server calculates a similarity between the extracted first object image and the selected first reference reference image; calculating a degree of deterioration of the first static object represented by the first object image based on the calculated similarity; and generating a repair or replacement schedule for the first static object according to the calculated degree of deterioration.

At this time, the vehicle terminal further includes a prediction device, and the prediction device includes a neural network learned according to a supervised learning method, and the neural network uses images for learning obtained by photographing in an information collection vehicle as input information for learning, It may be supervised learning using a label for each of one or more object images specified in each of the acquired training images. The extracting of the first object image may include inputting the photographed first image to the learned neural network and using an output of the learned neural network.

At this time, the processing device may include: extracting object images of a first group, which are all object images estimated to be the static object having one of the predetermined types, from the first image; A step of providing information about the first geolocation and the extracted object images of the first group to the other computing device may be further executed. And, the other computing device may be a server configured to access a database including the object information table. At this time, the server may include: selecting a first group of static objects existing within a predetermined distance from the first geolocation from among the static objects stored in the object information table; determining whether a loss object, which is a static object that does not correspond to the object images of the first group, exists among the static objects of the first group; and generating a repair or replacement schedule for the lost object.

According to another aspect of the present invention, a method for generating a management schedule for static objects on a road includes receiving, by a server, a first object image and a first reference reference image from a vehicle terminal; calculating, by the server, a similarity between the extracted first object image and the selected first reference reference image; calculating, by the server, a degree of deterioration of the first static object represented by the first object image based on the calculated similarity; and generating, by the server, a repair or replacement schedule for the first static object according to the calculated degree of obsolescence.

At this time, the vehicle terminal sets the geolocations of the plurality of static objects as an object information table including data on a plurality of static objects having one of predetermined types existing on a moving route of the vehicle. A storage device in which the object information table including the object information table and a plurality of standard reference images provided according to the Road Traffic Act as images representing the respective static objects is accessed is stored.

At this time, the vehicle terminal determines the difference between the first geolocation, which is the geographic location of the first static object existing on the predicted first movement path of the vehicle, and the current vehicle location, which is the current geographic location of the vehicle obtained from the GPS device. starting photographing using the photographing device from the point of time when a has a value within a predetermined threshold value; extracting a first object image estimated to be the static object having one of the predetermined types from a first image captured by the photographing device; and selecting a first reference image corresponding to the extracted first object image from among the reference images.

A method for managing static objects on a vehicle movement route provided according to one aspect of the present invention is an object including data about a plurality of static objects having one of predetermined types existing on a vehicle movement route. A vehicle terminal including a storage device storing the object information table including the geolocations of the plurality of static objects as an information table and a plurality of standard reference images provided according to the Road Traffic Act as images representing each of the static objects is meant to be used.

In the method for managing a static object on the vehicle movement route, the vehicle terminal determines the first geolocation, which is the geographic location of a first static object existing on the predicted first movement route of the vehicle, and the vehicle obtained from a GPS device. starting photographing using the photographing device from a time point when a difference between the current geographic location and the current vehicle position is within a predetermined threshold value; extracting, by the vehicle terminal, a first object image estimated to be the static object having one of the predetermined types from a first image photographed by the photographing device; selecting, by the vehicle terminal, a first standard reference image corresponding to the extracted first object image from among the standard reference images; and providing, by the vehicle terminal, at least one of the extracted first object image and the selected first reference reference image to another computing device.

The static object management method on the vehicle movement path may further include calculating, by a server, a degree of similarity between the extracted first object image and the selected first reference reference image; calculating, by the server, a degree of deterioration of the first static object represented by the first object image based on the calculated similarity; and generating, by the server, a repair or replacement schedule for the first static object according to the calculated degree of obsolescence.

According to one aspect of the present invention, in a crowdsourcing platform system including a source data securing server 1, a learning dataset processing method for processing an artificial intelligence learning dataset that recognizes static objects in a driving environment can be provided. there is. In the processing method, the source data securing server 1 collects black box images based on crowd sourcing, removes data bias from the collected black box images, and de-identifies personal information, thereby processing the collected black box images. A data collection step of generating refined image data from the black box image (S10); Classifying, by the original data securing server 1, the refined image data based on major classification categories constituting learning data, and extracting valid image data from the refined image data (S20); and generating a learning dataset from the refined image data and the effective image data (S50).

At this time, in the data collection step, the source data securing server 1 collects image data from a black box installed in a commercial vehicle driven by a driver panel, a commercial vehicle driver, and a professional panel that operates a vehicle equipped with a high-definition black box. supplementing the black box image by collecting image data from the running vehicle; and supplementing, by the source data securing server 1, the black box image with image data purchased from a server of a passenger transport service provider.

At this time, in the data collection step, in order to remove data bias from the collected black box images, the source data securing server 1 collects the first black box images from the limited number of participants for each metropolitan area/metropolitan city. Collecting images; Classifying a region of each image data through matching the collected data of the black box image and GPS information; Classifying a road type of a place where each image data was collected through matching the collected black box image data and GPS information; Classifying the car type of the vehicle providing each of the black box images through a pre-stored car DB; and classifying the collected black box images according to the region, the road type, and the vehicle type, and performing filtering on the collected black box images to prevent bias.

At this time, the data collection step may include, by the source data securing server 1, deleting sections before and after the source and destination points of each of the collected black box images in order to de-identify the personal information; deleting audio information other than video from each of the collected black box images; De-identifying the license plate number of the front driving vehicle and the next lane vehicle in each of the collected black box images; deleting traffic accident scenes from each of the collected black box images; Performing pedestrian de-identification processing when stopping at a crosswalk in each of the collected black box images; and de-identifying the photographing time and location information in the black box image from each of the collected black box images.

In this case, the generated learning data set may be classified into any one category among signs, lanes and crosswalks, traffic lights, and other signs.

At this time, the generated learning dataset includes information about the source location, resolution, type of vehicle captured, recording time, type of road captured, and information about the area captured for the video data of the collected black box image. and information about the URL including the file name, shooting time, shooting location, vehicle speed at the time of shooting, weather at the time of shooting, the number of lanes on the road taken, and the labeled image for each image data of the collected black box video. Including, the width and height of the bounding box defining the region of the static object extracted from each image data, and the x, y coordinates of the upper left corner, and the class ID of the static object included in the bounding box may be included. .

In this case, each of the major classification categories is further divided into a plurality of subcategories, and the step of extracting the data comprises constructing a learning data set for a learning model to be classified based on the major classification category, and constructing the built learning data. creating a learning model using the set; Static objects related to the plurality of subcategories are extracted from the valid image data using the generated learning model, and a bounding box, which is information about a position where the extracted static objects exist in the valid image data, is provided. ), and generating an annotation related to the bounding box.

At this time, in the step of extracting the data, according to the generated annotation, by classifying an image corresponding to the annotation and a video corresponding to the image into any one of the major classification categories, It may further include; generating a video and pre-processed image data according to the method.

At this time, the crowdsourcing platform system further includes a promotion server 2, and the promotion server includes a personal information utilization agreement part for processing an individual's personal information utilization agreement process for video utilization and reward payment; a compensation part that provides cash instead of points to the individual according to a given cash compensation system; and a content part providing video collection promotion guide content to the individual.

The source data securing method according to one aspect of the present invention is, first, a crowdsourcing-based basic data collection method, which includes a technique of pre-processing the collected data by collecting images previously taken by a general user with a black box; Third, it can include a method of directly purchasing the requested video through an investigation of the institution holding the video data. there is.

The text and annotation generation method provided according to one aspect of the present invention includes the steps of automatically classifying and searching source data obtained using an AI learning dataset authoring tool and recognizing characters and images present in the source data, thereby It may include automatically generating text and annotations from

The spatial range of source data essential for object recognition AI learning data in autonomous driving and next-generation intelligent transportation services provided according to one aspect of the present invention includes highways, national highways, urban expressways in the metropolitan and metropolitan areas, and roads with 6 or more lanes. can do.

According to one aspect of the present invention, the scope of content for source data essential for object recognition AI learning data in autonomous driving and next-generation intelligent transportation services is road marking (lane) and crossing indications, traffic lights, road traffic signs, and regulatory signs. , instruction signs, other signs, signs dedicated to road construction sections, road color guidance lines, and ITS facilities.

Specifically, the road surface markings (lanes) and crossing indications are presented for the purpose of promoting the safety and smooth communication of road traffic and preserving the road structure, and are independently or supplemented with traffic safety signs and signals to help road users It can provide the function of delivering the contents of regulations or instructions to the

Specifically, the traffic lights are installed for the purpose of preventing danger on the road and securing traffic safety and smooth communication, and displaying signals such as progress, stop, change of direction, caution, etc. with letters, symbols, or lights in relation to road traffic for various types of traffic. It can perform the function of assigning priority to.

Specifically, the road signs are provided for the purpose of providing road users with various information necessary to promote traffic safety and smooth communication in a consistent and unified way and to protect road structures and road facilities. As a traffic safety facility that is installed as a road surface or organically and complementarily installed with road markings and signals, it provides the function of conveying the contents of caution, regulation, instructions, etc. to road users so that the purpose of traffic safety signs can be achieved. can

Specifically, the original data classified into the other items may be determined as static data that affects driving decisions when driving an autonomous vehicle. For example, it can be classified as a road construction section dedicated sign, road surface color guidance line, and other ITS facilities.

Data collected according to one aspect of the present invention may be actual driving images obtained by an image capturing device installed in a vehicle as a black box image. 30 fps, HD (1366x768) or FHD (1920x1080) level video data, which are general black box video data, can be collected. Image data for learning may be extracted from image data captured in a vehicle and processed into a learning data set.

Black box-based driving video data is, firstly, affected by weather (rain, snow, cloudy, etc.) and time (night, daytime), secondly, light smearing caused by glass and raindrops is severe, and thirdly, the camera module performance is affected. Fourth, if the black box's own time setting is incorrect, it is difficult to map the video recording time. Fifth, a separate external module is required to record GPS information (in most cases, a separate GPS module). Sixth, a wide viewing angle In the case of supporting, there is a left and right distortion of the image, and seventh, there is a problem that the image capture time, location, speed, etc. are simultaneously recorded at the top or bottom of the captured image.

According to one aspect of the present invention, the collected user image may be a traffic information image collected through crowdsourcing. To this end, the device provided according to the present invention may provide points and benefits to participants according to their participation in traffic information collection.

An apparatus provided according to one aspect of the present invention may be configured to perform data refinement through GPS matching on collected user image data. In this case, image data for a given time, for example, 1,000 hours, in which data bias is prevented for each region, time, weather, vehicle type, and road type, may be obtained.

At this time, the device may perform image de-identification processing for personal information protection. For example, you can delete origin/destination clips. Image data can be secured through image extraction from refined images. Through the data refinement process, the large classification level data can be classified. Information such as movement section and weather can be obtained through user input or primary inspection of the collected images.

<whole system>

The source data collection method provided according to one aspect of the present invention includes, firstly, first securing a black box image through a crowdsourcing method, and second, performing GPS matching on the obtained black box image to prevent data collection bias. Third, selecting a panel, installing a photographing device in the vehicle of the selected panel, and collecting image data from the installed photographing device to supplement the image data through the panel. Fourth, The method may include supplementing image data by identifying a requested image from an image acquired from a server of a video provider and purchasing the identified requested video.

Specifically, when using the crowdsourcing platform, it is possible to quickly secure a lot of data at a low price, but in order to prevent data bias, it is possible to pre-process available data before processing learning data, and to perform a separate consent procedure for use of personal information. can run

Specifically, in the case of using the panel, it has the advantage of being able to prevent source data bias collection through panel recruitment and the advantage of being able to manage the quality of source data through the installation of separate equipment, but it takes a long time to obtain source data.

Specifically, in the case of the direct data purchase method, it is necessary to investigate the source data securing company, and it is expensive compared to crowdsourcing or panel utilization. However, it is possible to secure image data quickly without hiring a separate panel, and there is no need to hire additional personnel.

<Crowdsourcing Platform>

The crowdsourcing platform system provided according to one aspect of the present invention includes (1) a promotion server for paying additional points (or cash) according to video provision to individuals through additional video collection promotion, (2) transmission of large-capacity video data. It may include a file upload and additional server storage device provided for, and (3) a source data acquisition server.

The original data securing server collects data by using (1) a matching part that matches images and driving information by matching and classifying previously collected driving information and image information, and (2) region, user, and driving information matching. It may include a deflection removal part that performs a function for preventing deflection.

The promotion server includes (1) a personal information usage consent part that handles the individual's personal information use agreement process for video use and reward payment, (2) a reward part that provides cash instead of points to the individual according to a given cash reward system , and (3) a content part for providing video collection promotion guide content to the individual.

The crowdsourcing platform system may be configured to collect image data 10 times greater than learning data. In this case, functions of (1) collecting images according to image conditions including static objects, (2) analyzing image metadata for data matching classification, and enabling user input may be further included.

The crowdsourcing platform system may collect the image data by using web hard such as the file upload and additional server storage devices for image data collection.

In this case, the consent to use of personal information part may receive consent to use of image information from the individual.

At this time, the matching part can verify whether the GPS information matches, thereby authenticating whether the video and the driving information match each other, and can accept a primary input of a user input method to identify information included in the video. The first input may be an input for distinguishing whether it is a local road, a highway, or a national road.

At this time, the compensation part may be configured to perform a process of providing points or cash compensation to the individual according to the matching result.

At this time, image data may be uploaded from the individual through a web hard such as the file upload and additional server storage device. The uploaded video can be used by the original data securing server.

At this time, the deflection removal part may perform a deflection prevention function by matching previously held driving information with additionally collected image information.

<Supplementation of video data of panel recruitment method>

Image data obtained through the crowdsourcing platform system may not include desired image data. In order to overcome this problem, a recruited panel may be used in the method for supplementing image data according to one aspect of the present invention.

The original data obtained through the crowdsourcing platform system can be classified, and the initial collected data can be supplemented through panel recruitment by limiting collected data by city/outside city, highway/national road, weather, and time zone.

Selection of the panel may be made according to original data acquired through the crowdsourcing platform system.

Panels can be divided into commercial vehicle driver panels and specialized panels.

The driver panel of the commercial vehicle may be selected for a driver having a relatively long driving distance. The professional panel is a kind of hired panel and may mean a panel that collects high-definition image data through the installation of a QHD-level black box. The video data secured by the panelists can be received online using webhard or directly collected.

<Supplementation of video data by purchasing data directly>

In the case of vehicles used in the passenger car transport business, video recording devices can be installed for the purpose of identifying traffic accident situations, preventing crimes, and securing evidence. Data can be purchased directly from passenger vehicle transport operators, such as city/intercity bus transport operators.

The purchased image data and image data may be refined. In the process of refining the purchased image data, it is possible to secure original image data through region, vehicle type, road type classification, and de-identification processing through matching black box image and GPS information. It is possible to extract an image frame unit image from the purchased image data, classify a large classification level (crosswalk, sign, traffic light) and secure image data through preprocessing image recognition.

<Data cleaning process>

A data purification method provided according to one aspect of the present invention may refine image data and image data secured using the crowdsourcing platform system described above.

The source data securing server may secure original image data through region, vehicle type, road type classification, and non-identification processing through matching black box image and GPS information. In addition, it is possible to secure image data by extracting image frame units and classifying large classification levels (crosswalks, signs, traffic lights) through preprocessing image recognition.

The original data securing server includes (1) collecting black box images based on crowdsourcing, (2) removing data bias from the collected images, (3) de-identifying some information from the collected images. , (4) extracting images from the collected images, and (5) obtaining refined data from the extracted images.

The source data securing server, in order to prevent bias in collected image data, (1) limiting the number of participants by metropolitan area/metropolitan city when collecting black box images for the first time, (2) matching reported black box image data and GPS information Through, it is possible to perform ① metropolitan/metropolitan area classification, ② highway/national highway classification, and ③ vehicle type classification through the car DB.

The source data securing server may execute a process of performing de-identification processing including the following steps on the collected image data. In the first step, the sections before and after the point of departure and destination are deleted from the collected images, in the second step, audio information other than the images are deleted from the collected images, and in the third step, the forward driving vehicle and the vehicle in the next lane are deleted from the collected images. The number can be de-identified, in the fourth step, the traffic accident scene is deleted before AI learning data processing, in the fifth step, pedestrian de-identification is performed when stopping at a crosswalk, and in the sixth step, a black box video is captured Time and location information can be de-identified. The second to sixth steps above may be executed for driving images in intervals other than the departure and arrival of the vehicle.

<Designing Data for AI Learning>

The artificial intelligence learning data provided according to one aspect of the present invention may have the following characteristics.

The AI learning data completed by one aspect of the present invention can be disclosed to the outside and provided so that anyone can easily utilize it. Therefore, AI learning data can be designed with sufficient consideration of data accessibility.

Meta information used in the present invention can be designed to store information for self-driving AI learning. In addition, it can be designed considering the function for comparison and inspection of annotation by the operator and AI recognition result.

AI learning data completed by one aspect of the present invention can be divided into major categories (categories) referred to as signs, lanes and crosswalks, traffic lights, and other signs. The main classification of the signs may be further divided into a middle category of regulatory signs and instruction signs. The main classification of lanes and crosswalks can be further divided into intermediate categories such as center lines, bus-only lanes, U-turns, general lanes, road edge areas, route change restrictions, staggered crosswalks, diagonal crosswalks, and general crosswalks. The main classification of the traffic lights may be further divided into secondary categories such as a 4-color vehicle signal light, a 3-color vehicle signal light, a 3-color vehicle signal light, a 2-color vehicle signal light, and a 2-color walking signal light. The major classification of other signs may be further divided into a middle category of construction signs and variable information boards. Each of the intermediate categories may be divided into one or more predetermined subcategories.

In one embodiment of the present invention, in order to collect sufficient learning data for each class for each category, for example, more than 3,000 hours of video source data may be collected using a crowdsourcing-based data collection platform. In an embodiment, source image data may be classified according to regions and classification criteria for major categories and intermediate categories, and training data may be constructed for each intermediate category class.

<Type of AI data>

Data used in the AI system provided according to one aspect of the present invention may have the following characteristics.

Data sets provided according to one aspect of the present invention include (1) original video (FHD resolution), (2) image (jpg), (3) annotation (json) (= annotation), (4) data set list ( json-class types included).

Annotation items include source location for video data, resolution, type of vehicle (truck/bus/passenger car, etc.), recording time, type of road (highway/national highway), and region (metropolitan/metropolitan area) captured. may be provided for the record. In addition, it can be provided for the recording of URLs including the file name, shooting time, shooting location, vehicle speed at the time of shooting, weather at the time of shooting, number of lanes on the road taken, and labeled images for each image data extracted from the video. there is. In addition, for a number of N classes extracted from each image, the class ID (major classification -> middle classification -> class) and the location of the bounding box (coordinates (x,y) and width and height of the upper left corner) are defined and presented. can

In one aspect of the present invention, a json format may be used as an annotation format. An open data set has a form that requires a separate tool such as matlab, but in the present invention, a json format that is possible without a separate tool can be used in consideration of versatility.

<Building Data for AI Learning>

The learning data production process provided according to one aspect of the present invention includes data collection step (S10), data preprocessing step (S20), data processing step (S30), data verification step (S40), and learning data set generation step ( S50) may be included.

In the data collection step (S10), crowd sourcing-based black box image collection, bias removal on the collected image, and image de-identification processing may be sequentially performed. This may include the data cleansing process described above.

At this time, the black box video for the user and the driving time and driving route (when the GPS module is installed) of the video are collected, and the user's driving trip unit OD data (driving time, driving route, driving distance) is collected through a crowdsourcing platform this is possible It is possible to prevent bias and secure original data by matching the previously held driving information with additionally collected image information.

At this time, in order to remove data bias for the collected video data, the number of participants per metropolitan area/metropolitan city is limited when the black box video is collected for the first time, and the reported black box video data and GPS information are matched to ① classify the metropolitan area/metropolitan city area, ② highway / National highway classification, and ③ Car type classification can be executed through the vehicle DB. Collected images may be classified according to region, road type, and vehicle type, and filtering may be performed on the collected images to prevent bias.

And, as described above, de-identification processing may be performed on the collected image data to protect personal information.

Meanwhile, the following may be considered when extracting still pictures from acquired video data. Duplicate photos occurring at a specific point or traffic congestion can be removed by considering the distance based on the speed of the vehicle from the static object target on the road whose location has already been secured and stored.

In the data pre-processing step (S20), for the collected and refined image data, lanes/crosswalks, traffic lights, traffic signs, and other signs, which are major categories of learning data, are classified, and valid image data is extracted from the image data. can

The data preprocessing step (S20) may include the following steps.

That is, in step S21, a learning data set (eg, for a total of 109 classes for 4 major classifications and 56 intermediate classifications) for a learning model to be classified based on the prepared major classification categories is built, and through learning A learning model can be created.

Each of the major classification categories (classes) may be further divided into a plurality of subcategories (middle class and small class).

In step S22, static objects related to the plurality of subcategories are extracted from the valid image data by using the generated learning model, and information about the position where the extracted static objects exist in the valid image data A bounding box and annotations related to the bounding box may be automatically generated.

In step S23, the corresponding video and image are classified according to the major classification according to the generated annotation, thereby generating video and preprocessed image data according to the major classification.

In the data processing step (S30), the inspector may check and modify the bounding box and annotation generated in the data preprocessing step (S20). Key-In method, Selection method, Post-verification method, Preset method, Automatic input method, Cross Field Check, Related Field Copy, Reject Packet, Reject Page It can support various input methods such as

In the data verification step (S40), the data verification may proceed with a procedure in which a total inspection is performed by error rate detection using a deep learning algorithm, and an inspector directly checks and corrects abnormal learning data such as error detection.

Abnormal annotation data can be automatically searched through the deep learning learning model, data with errors detected in the learning log can be requested for correction through the data verification process, and a training data set is created through the data that has been finally inspected. And it is possible to perform advanced deep learning learning model through the finally verified training data set.

<Infrastructure for producing and storing learning data>

According to one aspect of the present invention, an infrastructure system for producing and storing learning data may be provided. The infrastructure system may include (1) a process management system applying artificial intelligence and cooperation between workers and (2) a data collection and analysis system using an artificial intelligence AI recognition engine.

In the existing learning data construction process, quality control relied on professional workers in a series of processes such as image processing, index information input and registration.

In the present invention, a large number of workers and a crowd sourcing method are used to secure a lot of learning data in a short period of time. Therefore, in order to secure the quality of data, it is necessary to standardize the process and automate process management for standardization. Therefore, it is possible to secure the best quality of AI learning data by utilizing a process management system equipped with guidelines and guidelines for each target task (so that workers can easily refer to them). A process control system can be a control system that involves people as part of the process.

Learning data generation technology provided according to an aspect of the present invention may be composed of steps of data collection, data preprocessing, data processing, data verification, and learning data set generation.

At this time, the data processing step may consist of a verification of a human inspector and a collaborative process of AI. Through inspection by internal personnel, workers with high accuracy are selected as inspectors, and only data that passes multiple cross-examination standards can be processed for final production completion. All processes of collected data can be performed by process management platform and annotation tool. The first annotation (object extraction, meta, and large category video classification) is performed by artificial intelligence, and the second verification and correction can be performed by human resources. After training the final AI algorithm, data set validation can be additionally performed to ensure quality.

At this time, the above-described data collection and analysis system is a system that participates in some processes of the process management system, and learns to automatically classify and search target data by applying AI element technology to the acquired AI learning data. .

In the video data pre-processing step, objects for, for example, 164 classes are recognized in the collected user video data, and automatic classification and annotation automation functions (meta creation) can be performed according to the major classification of video data according to the recognized objects. .

The source data securing method according to one aspect of the present invention is, first, a crowdsourcing-based basic data collection method, which includes a technique of pre-processing the collected data by collecting images previously taken by a general user with a black box; Third, it can include a method of directly purchasing the requested video through an investigation of the institution holding the video data. there is.

The text and annotation generation method provided according to one aspect of the present invention includes the steps of automatically classifying and searching source data obtained using an AI learning dataset authoring tool and recognizing characters and images present in the source data, thereby It may include automatically generating text and annotations from

The spatial range of source data essential for object recognition AI learning data in autonomous driving and next-generation intelligent transportation services provided according to one aspect of the present invention includes highways, national highways, urban expressways in the metropolitan and metropolitan areas, and roads with 6 or more lanes. can do.

According to one aspect of the present invention, the scope of content for source data essential for object recognition AI learning data in autonomous driving and next-generation intelligent transportation services is road marking (lane) and crossing indications, traffic lights, road traffic signs, and regulatory signs. , instruction signs, other signs, signs dedicated to road construction sections, road color guidance lines, and ITS facilities.

Specifically, the road surface markings (lanes) and crossing indications are presented for the purpose of promoting the safety and smooth communication of road traffic and preserving the road structure, and are independently or supplemented with traffic safety signs and signals to help road users It can provide the function of delivering the contents of regulations or instructions to the

Specifically, the traffic lights are installed to prevent danger on the road and ensure traffic safety and smooth communication, and display signals such as progress ㅇ stop ㅇ direction change ㅇ caution with letters ㅇ symbols or lights in relation to road traffic, so that various types of traffic flow It can perform the function of assigning priority to.

Specifically, the road signs are provided for the purpose of providing road users with various information necessary to promote traffic safety and smooth communication in a consistent and unified way and to protect road structures and road facilities. As a traffic safety facility that is installed as a road surface or organically and complementarily installed with road markings and signals, it provides the function of conveying the contents of caution, regulation, instructions, etc. to road users so that the purpose of traffic safety signs can be achieved. can

Specifically, the original data classified into the other items may be determined as static data that affects driving decisions when driving an autonomous vehicle. For example, it can be classified as a road construction section dedicated sign, road surface color guidance line, and other ITS facilities.

Data collected according to one aspect of the present invention may be actual driving images obtained by an image capturing device installed in a vehicle as a black box image. 30 fps, HD (1366x768) or FHD (1920x1080) level video data, which are general black box video data, can be collected. Image data for learning may be extracted from image data captured in a vehicle and processed into a learning data set.

Black box-based driving video data is, firstly, affected by weather (rain, snow, cloudy, etc.) and time (night, daytime), secondly, light smearing caused by glass and raindrops is severe, and thirdly, the camera module performance is affected. Fourth, if the black box's own time setting is incorrect, it is difficult to map the video recording time. Fifth, a separate external module is required to record GPS information (in most cases, a separate GPS module). Sixth, a wide viewing angle In the case of supporting, there is a left and right distortion of the image, and seventh, there is a problem that the image capture time, location, speed, etc. are simultaneously recorded at the top or bottom of the captured image.

User images collected according to one aspect of the present invention may be dozens of traffic information images through crowdsourcing. To this end, the device provided according to the present invention may provide points and benefits to participants according to their participation in traffic information collection.

An apparatus provided according to one aspect of the present invention may be configured to perform data refinement through GPS matching on collected user image data. In this case, image data for a given time, for example, 1,000 hours, in which data bias is prevented for each region, time, weather, vehicle type, and road type, may be obtained.

At this time, the device may perform image de-identification processing for personal information protection. For example, you can delete origin/destination clips. Image data can be secured through image extraction from refined images. Through the data refinement process, the large classification level data can be classified. Information such as movement section and weather can be obtained through user input or primary inspection of the collected images.

<whole system>

The source data collection method provided according to one aspect of the present invention includes, firstly, first securing a black box image through a crowdsourcing method, and second, performing GPS matching on the obtained black box image to prevent data collection bias. Third, selecting a panel, installing a photographing device in the vehicle of the selected panel, and collecting image data from the installed photographing device to supplement the image data through the panel. Fourth, The method may include supplementing image data by identifying a requested image from an image obtained from a server of an image supplier and purchasing the identified requested image.

Specifically, when using the crowdsourcing platform, it is possible to quickly secure a lot of data at a low price, but in order to prevent data bias, it is possible to pre-process available data before processing learning data, and to perform a separate consent procedure for use of personal information. can run

Specifically, in the case of using the panel, it has the advantage of being able to prevent source data bias collection through panel recruitment and the advantage of being able to manage the quality of source data through the installation of separate equipment, but it takes a long time to obtain source data.

Specifically, in the case of the direct data purchase method, it is necessary to investigate the source data securing company, and it is expensive compared to crowdsourcing or panel utilization. However, it is possible to secure image data quickly without hiring a separate panel, and there is no need to hire additional personnel.

<Crowdsourcing Platform>

The crowdsourcing platform system provided according to one aspect of the present invention includes (1) a promotion server for paying additional points (or cash) according to video provision to individuals through additional video collection promotion, (2) transmission of large-capacity video data. It may include a file upload and additional server storage device provided for, and (3) a source data acquisition server.

The original data securing server collects data by using (1) a matching part that matches images and driving information by matching and classifying previously collected driving information and image information, and (2) region, user, and driving information matching. It may include a deflection removal part that performs a function for preventing deflection.

The promotion server includes (1) a personal information usage consent part that handles the individual's personal information use agreement process for video use and reward payment, (2) a reward part that provides cash instead of points to the individual according to a given cash reward system , and (3) a content part for providing video collection promotion guide content to the individual.

The crowdsourcing platform system may be configured to collect image data 10 times greater than learning data. In this case, functions of (1) collecting images according to image conditions including static objects, (2) analyzing image metadata for data matching classification, and enabling user input may be further included.

The crowdsourcing platform system may collect the image data by using web hard such as the file upload and additional server storage devices for image data collection.

In this case, the consent to use of personal information part may receive consent to use of image information from the individual.

At this time, the matching part can verify whether the GPS information matches, thereby authenticating whether the video and the driving information match each other, and can accept a primary input of a user input method to identify information included in the video. The first input may be an input for distinguishing whether it is a local road, a highway, or a national road.

At this time, the compensation part may be configured to perform a process of providing points or cash compensation to the individual according to the matching result.

At this time, image data may be uploaded from the individual through a web hard such as the file upload and additional server storage device. The uploaded video can be used by the original data securing server.

At this time, the deflection removal part may perform a deflection prevention function by matching previously held driving information with additionally collected image information.

<Supplementation of video data of panel recruitment method>

Image data obtained through the crowdsourcing platform system may not include desired image data. In order to overcome this problem, a recruited panel may be used in the method for supplementing image data according to one aspect of the present invention.

The original data obtained through the crowdsourcing platform system can be classified, and the initial collected data can be supplemented through panel recruitment by limiting collected data by city/outside city, highway/national road, weather, and time zone.

Selection of the panel may be made according to original data acquired through the crowdsourcing platform system.

Panels can be divided into commercial vehicle driver panels and specialized panels.

The driver panel of the commercial vehicle may be selected for a driver having a relatively long driving distance. The professional panel is a kind of hired panel and may mean a panel that collects high-definition image data through the installation of a QHD-level black box. The video data secured by the panelists can be received online using webhard or directly collected.

<Supplementation of video data by purchasing data directly>

In the case of vehicles used in the passenger car transport business, video recording devices can be installed for the purpose of identifying traffic accident situations, preventing crimes, and securing evidence. Data can be purchased directly from passenger vehicle transport operators, such as city/intercity bus transport operators.

The purchased image data and image data may be refined. In the process of refining the purchased image data, it is possible to secure original image data through region, vehicle type, road type classification, and de-identification processing through matching black box image and GPS information. It is possible to extract an image frame unit image from the purchased image data, classify a large classification level (crosswalk, sign, traffic light) and secure image data through preprocessing image recognition.

<Data cleaning process>

A data purification method provided according to one aspect of the present invention may refine image data and image data secured using the crowdsourcing platform system described above.

The source data securing server may secure original image data through region, vehicle type, road type classification, and non-identification processing through matching black box image and GPS information. In addition, it is possible to secure image data by extracting image frame units and classifying large classification levels (crosswalks, signs, traffic lights) through preprocessing image recognition.

The original data securing server includes (1) collecting black box images based on crowdsourcing, (2) removing data bias from the collected images, (3) de-identifying some information from the collected images. , (4) extracting images from the collected images, and (5) obtaining refined data from the extracted images.

The source data securing server, in order to prevent bias in collected image data, (1) limiting the number of participants by metropolitan area/metropolitan city when collecting black box images for the first time, (2) matching reported black box image data and GPS information Through, it is possible to perform ① metropolitan/metropolitan area classification, ② highway/national highway classification, and ③ vehicle type classification through the car DB.

The source data securing server may execute a process of performing de-identification processing including the following steps on the collected image data. In the first step, the sections before and after the point of departure and destination are deleted from the collected images, in the second step, audio information other than the images are deleted from the collected images, and in the third step, the forward driving vehicle and the vehicle in the next lane are deleted from the collected images. The number can be de-identified, in the fourth step, the traffic accident scene is deleted before AI learning data processing, in the fifth step, pedestrian de-identification is performed when stopping at a crosswalk, and in the sixth step, a black box video is captured Time and location information can be de-identified. The second to sixth steps above may be executed for driving images in intervals other than the departure and arrival of the vehicle.

<Designing Data for AI Learning>

The artificial intelligence learning data provided according to one aspect of the present invention may have the following characteristics.

The AI learning data completed by one aspect of the present invention can be disclosed to the outside and provided so that anyone can easily utilize it. Therefore, AI learning data can be designed with sufficient consideration of data accessibility.

Meta information used in the present invention can be designed to store information for self-driving AI learning. In addition, it can be designed considering the function for comparison and inspection of annotation by the operator and AI recognition result.

AI learning data completed by one aspect of the present invention can be divided into major categories (categories) referred to as signs, lanes and crosswalks, traffic lights, and other signs. The main classification of signs may be further divided into a middle category of regulatory signs and instruction signs. The main classification of lanes and crosswalks can be further divided into intermediate categories such as center lines, bus-only lanes, U-turns, general lanes, road edge areas, route change restrictions, staggered crosswalks, diagonal crosswalks, and general crosswalks. The main classification of traffic lights may be further divided into secondary categories such as 4-color vehicle signal lights, 3-color vehicle signal lights, 3-color vehicle signal lights, 2-color vehicle signal lights, and 2-color walking signal lights. The major classification of other signs may be further divided into a middle category of construction signs and variable information boards. Each of the middle categories may be divided into one or more predetermined sub-categories.

In one embodiment of the present invention, in order to collect sufficient learning data for each class for each category, for example, more than 3,000 hours of video source data may be collected using a crowdsourcing-based data collection platform. In an embodiment, source image data may be classified according to regions and classification criteria for major categories and intermediate categories, and training data may be constructed for each intermediate category class.

<Current status of AI data>

Data used in the AI system provided according to one aspect of the present invention may have the following characteristics.

Data sets provided according to one aspect of the present invention include (1) original video (FHD resolution), (2) image (jpg), (3) annotation (json) (= annotation), (4) data set list ( json-class types included).

Annotation items include source location for video data, resolution, type of vehicle (truck/bus/passenger car, etc.), recording time, type of road (highway/national highway), and region (metropolitan/metropolitan area) captured. may be provided for the record. In addition, it can be provided for the recording of URLs including the file name, shooting time, shooting location, vehicle speed at the time of shooting, weather at the time of shooting, number of lanes on the road taken, and labeled images for each image data extracted from the video. there is. In addition, for a number of N classes extracted from each image, the class ID (major classification -> middle classification -> class) and the location of the bounding box (coordinates (x,y) and width and height of the upper left corner) are defined and presented. can

In one aspect of the present invention, a json format may be used as an annotation format. An open data set has a form that requires a separate tool such as matlab, but in the present invention, a json format that is possible without a separate tool can be used in consideration of versatility.

<Building Data for AI Learning>

The learning data production process provided according to one aspect of the present invention includes data collection step (S10), data preprocessing step (S20), data processing step (S30), data verification step (S40), and learning data set generation step ( S50) may be included.

In the data collection step (S10), crowd sourcing-based black box image collection, bias removal on the collected image, and image de-identification processing may be sequentially performed. This may include the data cleansing process described above.

At this time, the black box video for the user and the driving time and driving route (when the GPS module is installed) of the video are collected, and the user's driving trip unit OD data (driving time, driving route, driving distance) is collected through a crowdsourcing platform this is possible It is possible to prevent bias and secure original data by matching the previously held driving information with additionally collected image information.

At this time, in order to remove data bias for the collected video data, the number of participants per metropolitan area/metropolitan city is limited when the black box video is collected for the first time, and the reported black box video data and GPS information are matched to ① classify the metropolitan area/metropolitan city area, ② highway / National highway classification, and ③ Vehicle type classification can be executed through the vehicle DB. Collected images may be classified according to region, road type, and vehicle type, and filtering may be performed on the collected images to prevent bias.

And, as described above, de-identification processing may be performed on the collected image data to protect personal information.

Meanwhile, the following may be considered when extracting still pictures from acquired video data. Duplicate photos occurring at a specific point or traffic congestion can be removed by considering the distance based on the speed of the vehicle from the static object target on the road whose location has already been secured and stored.

In the data pre-processing step (S20), for the collected and refined image data, lanes/crosswalks, traffic lights, traffic signs, and other signs, which are major categories of learning data, are classified, and valid image data is extracted from the image data. can

The data preprocessing step (S20) may include the following steps.

That is, in step S21, a learning data set (eg, for a total of 109 classes for 4 major classifications and 56 intermediate classifications) for a learning model to be classified based on the prepared major classification categories is built, and through learning A learning model can be created.

Each of the major classification categories (classes) may be further divided into a plurality of subcategories (middle class and small class).

In step S22, static objects related to the plurality of subcategories are extracted from the valid image data by using the generated learning model, and information about the position where the extracted static objects exist in the valid image data A bounding box and annotations related to the bounding box may be automatically generated.

In step S23, the corresponding video and image are classified according to the major classification according to the generated annotation, thereby generating video and preprocessed image data according to the major classification.

In the data processing step (S30), the inspector can check and modify the bounding box and annotation created in the data preprocessing step (S20). All checks and corrections are performed through various inputs such as Key-In method, Selection method, Post-verification method, Preset method, Automatic input method, Cross Field Check, Related Field Copy, Reject Packet, Reject Page, etc. for fast and accurate input. method can be supported.

In the data verification step (S40), the data verification may proceed with a procedure in which a total inspection is performed by error rate detection using a deep learning algorithm, and an inspector directly checks and corrects abnormal learning data such as error detection.

Abnormal annotation data can be automatically searched through the deep learning learning model, data with errors detected in the learning log can be requested for correction through the data verification process, and a training data set is created through the data that has been finally inspected. And it is possible to perform advanced deep learning learning model through the finally verified training data set.

<Infrastructure for producing and storing learning data>

According to one aspect of the present invention, an infrastructure system for producing and storing learning data may be provided. The infrastructure system may include (1) a process management system applying artificial intelligence and cooperation between workers and (2) a data collection and analysis system using an artificial intelligence AI recognition engine.

In the existing learning data construction process, quality control relied on professional workers in a series of processes such as image processing, index information input and registration.

In the present invention, a large number of workers and a crowd sourcing method are used to secure a lot of learning data in a short period of time. Therefore, in order to secure the quality of data, it is necessary to standardize the process and automate process management for standardization. Therefore, it is possible to secure the best quality of AI learning data by utilizing a process management system equipped with guidelines and guidelines for each target task (so that workers can easily refer to them). A process control system can be a control system that involves people as part of the process.

Learning data generation technology provided according to an aspect of the present invention may be composed of steps of data collection, data preprocessing, data processing, data verification, and learning data set generation.

At this time, the data processing step may consist of a verification of a human inspector and a collaborative process of AI. Through inspection by internal personnel, workers with high accuracy are selected as inspectors, and only data that passes multiple cross-examination standards can be processed for final production completion. All processes of collected data can be performed by process management platform and annotation tool. The first annotation (object extraction, meta, and large category video classification) is performed by artificial intelligence, and the second verification and correction can be performed by human resources. After training the final AI algorithm, data set validation can be additionally performed to ensure quality.

At this time, the above-described data collection and analysis system is a system that participates in some processes of the process management system, and learns to automatically classify and search target data by applying AI element technology to the acquired AI learning data. .

In the video data pre-processing step, objects for, for example, 164 classes are recognized in the collected user video data, and automatic classification and annotation automation functions (meta creation) can be performed according to the major classification of video data according to the recognized objects. .

According to the present invention, it is possible to provide a technique for detecting an image of a driver and an autonomous vehicle according to the pavement condition of the road and determining the degree of visibility by using information on various static objects related to the road that has already been collected, and also for identification. Through image prediction of difficult road markings, lanes, etc., the display contents can be supplemented and provided in real time, and the difference between the predicted information and the actually observed information can be provided to the manager of the static object management system on the road. can

Due to the real-time monitoring service of the road bridge surface (road surface) display information provided through the present invention and user information through image prediction, it will be possible to improve public traffic safety.

1 is a diagram for explaining a relationship between components in a vehicle and a server and components in a vehicle according to an embodiment of the present invention.
2 is a flowchart illustrating a method of selecting data to be provided to a server to generate a management schedule of static objects on a road according to an embodiment of the present invention.
3 is a flowchart of a method for generating a management schedule of static objects on a road in a server according to an embodiment of the present invention.
4 is a flowchart illustrating a method of generating a repair or replacement schedule for a lost object according to an embodiment of the present invention.
5 is a diagram for explaining a crowdsourcing platform system according to an embodiment of the present invention.
6 is a diagram for explaining a learning dataset processing method according to an embodiment of the present invention.
7 is a diagram for separately explaining a process for obtaining source image data according to an embodiment of the present invention.
8 schematically illustrates a learning data processing method according to an embodiment of the present invention.
9 is a diagram for explaining a data purification process according to an embodiment of the present invention.
10 is a diagram for explaining annotations according to an embodiment of the present invention.
11 illustrates information about a location of a class ID and a bounding box extracted from an image according to an embodiment of the present invention.
FIG. 12 is a diagram for explaining the process of performing the data pre-processing step of step S20 of FIG. 6 .
13 is a diagram for explaining a process of generating a learning model according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, the present invention is not limited to the embodiments described herein and may be implemented in various other forms. Terms used in this specification are intended to aid understanding of the embodiments, and are not intended to limit the scope of the present invention. Also, the singular forms used herein include the plural forms unless the phrases clearly dictate the contrary.

1 is a diagram for explaining a relationship between components in a vehicle and a server and components in a vehicle according to an embodiment of the present invention.

2 is a flowchart illustrating a method of selecting data to be provided to a server to generate a management schedule of static objects on a road according to an embodiment of the present invention.

Hereinafter, it will be described with reference to FIGS. 1 and 2 together.

A vehicle terminal 210 and an autonomous driving device 220 may be installed in the vehicle 200 .

The vehicle terminal 210 may include a processing device 211 , a photographing device 212 , a storage device 213 , and a prediction device 214 .

An object information table and a plurality of reference reference images may be stored in the storage device 213 .

The object information table may be an object information table including data about a plurality of static objects having one of predetermined types existing on a moving route of the vehicle. The object information table may include geolocations of the plurality of static objects.

The plurality of standard reference images may be provided according to the Road Traffic Act as images representing the respective static objects.

The prediction device 214 may include a neural network trained according to a supervised learning method.

The neural network is supervised using images for learning acquired by photographing from an information collection vehicle as input information for learning, and using a label for each of one or more object images specified in each of the acquired images for learning. can

The autonomous driving device 220 may support autonomous driving of the vehicle 200 .

The processing device 211 may be configured to execute steps S200 to S230 below.

In step S200, the processing device 211 determines the vehicle 200 obtained from the GPS device and the first geolocation, which is the geographic location of a first static object existing on the predicted first movement route of the vehicle 200. The photographing using the photographing device 212 can be started from the time when the difference between the current vehicle position, which is the current geographic position of ), has a value within a predetermined threshold value.

In step S210, the processing device 211 extracts a first object image estimated to be the static object having any one of the predetermined types from the first image photographed by the photographing device 212. can do.

At this time, step S210 may include a step in which the processing device 211 inputs the photographed first image to the learned neural network and uses the output of the learned neural network.

In step S220, the processing device 211 may select a first reference image corresponding to the extracted first object image from among the reference images.

In step S230, the processing device 211 transfers at least one of the extracted first object image and the selected first reference reference image to another computing device (eg, the autonomous driving device 220 or the server 300). ) can be provided.

At this time, when the other computing device is the autonomous driving device 220, the autonomous driving device 220 displays the extracted first object image or the selected first object image on a display device (not shown) included in the vehicle 200. It may be configured to display a fiducial reference image.

The step S220 may include the following steps S221 to S223.

At this time, the processing device 211 may access the storage device 213 .

In step S221, the processing device 211 may select a first group of static objects existing within a predetermined distance from the first geolocation from among the static objects stored in the object information table.

In step S222, the processing device 211 may select a first group of reference images corresponding to the selected first group of static objects from among the plurality of reference images.

In step S223, the processing unit 211 may determine, as the first reference image, one having the highest similarity with the first object image among the selected reference images of the first group.

3 is a flowchart of a method for generating a management schedule of static objects on a road in a server according to an embodiment of the present invention.

Steps S300 to S330 below may be steps performed by the server 300 when the other computing device of the step S230 is the remote server 300 .

In step S300 , the server 300 may receive a first object image and a first reference reference image from the vehicle terminal 210 .

In step S310, the server 300 may calculate a similarity between the extracted first object image and the selected first reference reference image.

In step S320, the server 300 may calculate the degree of deterioration of the first static object represented by the first object image based on the calculated similarity.

In step S330, the server 300 may create a repair or replacement schedule for the first static object according to the calculated degree of deterioration.

4 is a flowchart illustrating a method of generating a repair or replacement schedule for a lost object according to an embodiment of the present invention.

The server 300 may be configured to access a database including the object information table.

In step S400, the vehicle terminal 210 determines the vehicle 200 obtained from the first geolocation, which is the geographic location of a first static object existing on the predicted first movement path of the vehicle 200, and the GPS device. The photographing using the photographing device 212 can be started from the time when the difference between the current vehicle position, which is the current geographic position of ), has a value within a predetermined threshold value.

In step S410, the vehicle terminal 210 determines all object images estimated to be the static object having one of the predetermined types from the first image photographed by the photographing device 212. One group of object images can be extracted.

In step S420, the vehicle terminal 210 may provide information about the first geolocation and the extracted object images of the first group to the other computing device (ie, the server 300).

In step S430, the server 300 may select a first group of static objects existing within a predetermined distance from the first geolocation from among the static objects stored in the object information table.

In step S440, the server 300 may determine whether there is a loss object, which is a static object that does not correspond to the object images of the first group, among the static objects of the first group.

In step S450, the server 300 may generate a repair or replacement schedule for the lost object.

5 is a diagram for explaining a crowdsourcing platform system according to an embodiment of the present invention.

The crowdsourcing platform system 100 may process an artificial intelligence learning dataset that recognizes static objects in a driving environment.

The crowdsourcing platform system 100 may include a source data securing server 1 , a promotion server 2 , and a storage device 3 .

The source data acquisition server 1 may include a matching part 11 and a deflection removal part 12 .

The matching part 11 may match the previously collected driving information and the video information to each other by matching and classifying the collected driving information and driving information. In addition, the matching part 11 can verify whether the video and driving information match each other by verifying whether the GPS information matches, and can accept a primary input of a user input method to identify information included in the video. there is. The first input may be an input for distinguishing whether it is a local road, a highway, or a national road.

The deflection removal part 12 may perform a function to prevent bias in data collection by using region, user, and driving information matching. That is, the deflection removal part 12 can perform a deflection prevention function by matching pre-owned driving information with additionally collected image information.

The promotion server 2 may pay additional points (or cash) according to the video provision to the individual through additional video collection promotion. To this end, the promotion server 2 may include a personal information utilization agreement part 21, a reward part 22, and a content part 23.

Part 21 of consent to use of personal information may perform a process of consent to use of personal information of individuals for video use and payment of compensation.

The compensation part 22 may perform a process of providing cash instead of points to the individual according to a given cash compensation scheme. Alternatively, the compensation part 22 may perform a process of providing points or cash compensation to the individual according to the matching result of the matching part 11 of the source data securing server 1 .

The content part 23 may perform a process of providing video collection promotion guide content to the individual.

The storage device 3 may provide file upload and additional functions for transmitting large-capacity image data.

The storage device 3 may include an image DB 31 , a driving technology DB 32 , and a vehicle model identification database 33 .

The image DB 31 may store original images, for example, actual driving images acquired by an image capturing device installed in a vehicle as a black box image. 30 fps, HD (1366x768) or FHD (1920x1080) level video data, which are general black box video data, can be collected. Also, traffic information images collected through crowdsourcing, for example, may be stored in the image DB 31 . In addition, images collected through recruited panels may be stored in the image DB 31 .

The driving technology DB 32 may store driving information. The driving information is OD data of a driving trip unit of a corresponding user through a crowdsourcing platform, and may include information such as driving time, driving route, and driving distance.

The vehicle model identification DB 33 may classify and store the vehicle model with respect to the black box image data.

6 is a diagram for explaining a learning dataset processing method according to an embodiment of the present invention.

In step S10, the source data securing server 1 collects black box images based on crowdsourcing, removes data bias from the collected black box images, and de-identifies personal information, thereby removing the collected black box images. Refined image data can be generated from the black box image.

At this time, the original data acquisition server 1 collects image data from a black box installed in a commercial vehicle operated by a driver panel, a commercial vehicle driver, and image data from a vehicle operated by a professional panel operating a vehicle equipped with a high-definition black box. It is possible to supplement the black box image by collecting. Further, the source data securing server 1 may supplement the black box image with image data purchased from a server of a passenger transport service provider.

At this time, in the case of video data supplementation, the original data acquired by the source data securing server 1 is first classified, and the collected data is limited to city/outside, highway/national road, weather, and time zone, and the initial collected data is supplemented through panel recruitment. can do.

In step S20, the original data securing server 1 classifies the refined image data based on the large classification categories constituting the learning data, and extracts valid image data from the refined image data.

In step S30, the inspector may check and modify the bounding box and annotation generated in step S20, which is the data preprocessing step. All checks and corrections are performed through various inputs such as Key-In method, Selection method, Post-verification method, Preset method, Automatic input method, Cross Field Check, Related Field Copy, Reject Packet, Reject Page, etc. for fast and accurate input. method can be supported.

At this time, in step (S30') to perform step (S30), inspector management may be made. The inspector management step may include registering a user, allocating an inspector, and allocating data.

In the data verification step (S40), data verification can be performed by performing a total inspection by detecting an error rate using a deep learning algorithm, and proceeding with a procedure in which the inspector directly checks and corrects the learning data with errors such as error detection.

In step S50, a learning data set (data set A, B, etc.) may be created based on the data for which the verification procedure has been completed.

7 is a diagram for separately explaining a process for obtaining source image data according to an embodiment of the present invention.

In order to secure source image data, three steps can be largely classified and explained.

In the first step, the original data securing server 1 may secure basic data based on crowdsourcing. At this time, after constructing an image collection infrastructure and collecting crowdsourcing-based image data, the crowdsourced image data may be prepared by refining and classifying the data.

In the second step, as a supplement to the first step, image data of the panel recruiting method may be secured. At this time, after recruiting and selecting panels (commercial vehicles (taxi, truck, bus), etc.), a photographing device may be installed in the vehicle of the selected panel. By collecting image data from the installed photographing device, image data may be additionally prepared through a panel.

In step 3, as an additional complement to steps 1 and 2, data can be purchased directly. At this time, image data may be additionally prepared by researching image suppliers, identifying requested images, and purchasing image data.

The original image data may include crowdsourced image data, panel collection image data, and purchased image data obtained in the above-described first step, second step, and third step, respectively.

8 schematically illustrates a learning data processing method according to an embodiment of the present invention.

Traffic image data can be collected through crowdsourcing. In this case, points and benefits may be provided to traffic image providers (ie, traffic information collection participants).

Collected image data may be stored in the image DB 31 .

By combining the image DB 31, driving record DB 32, and vehicle type identification DB 33, image filtering can be performed based on conditions such as region, type, time, and date. The filtered image can be used as source data for AI learning data processing.

9 is a diagram for explaining a data purification process according to an embodiment of the present invention.

The following steps may be steps performed by the original data securing server 1.

In step S110, black box images may be collected based on crowdsourcing. In this case, when the black box image is first collected, the black box image may be collected from a limited number of participants for each metropolitan area/metropolitan city.

In step S120, data bias can be removed from the black box image.

Specifically, step S120 may include steps S121 to S124 below.

In step S121, the region (metropolitan area/metropolitan city) of each image data may be classified through matching the collected black box image data and GPS information.

In step S122, the road type (highway/national road) of the place where each image data was collected can be classified through matching the collected black box image data and GPS information.

In step S123, it is possible to classify the car type of the vehicle providing each of the black box images through a pre-stored car DB (vehicle type identification DB).

In step S124, the collected black box images may be classified according to the region, the road type, and the vehicle type, and filtering may be performed on the collected black box images to prevent bias.

In step S130, personal information may be de-identified in the image from which the deflection is removed.

Specifically, step S130 may include steps S131 to S136 below.

In step S131, it is possible to delete the section before and after the point of departure and destination of each black box image collected. In this case, each of the collected black box images may refer to an image from which deflection is removed.

In step S132, audio information other than video may be deleted from each of the collected black box images.

In step S133, vehicle numbers of vehicles traveling in front and vehicles in the next lane may be de-identified in each of the collected black box images.

In step S134, the traffic accident scene may be deleted from each of the collected black box images.

In step S135, pedestrian de-identification processing may be performed when stopping at a crosswalk in each of the collected black box images.

In step S136, the photographing time and location information in the black box image may be de-identified in each of the collected black box images.

In step S140, an image may be extracted from the de-identified image. That is, an image in units of image frames may be extracted.

In step S150, valid image data may be selected from the extracted image, and static objects for each subcategory may be extracted from the selected valid image.

10 is a diagram for explaining annotations according to an embodiment of the present invention.

After step S150, an annotation about a bounding box, which is information about a location where the extracted static object exists in the valid image data, may be generated.

Annotation items may be provided for recording video information, image information, and class information.

In the case of video information, annotation items include source location for video data, resolution, type of vehicle (truck/bus/passenger car, etc.), recording time, type of road (highway/country road), and region ( metropolitan area/metropolitan area) may be provided for records.

In the case of image information, the annotation items include the file name for each image data extracted from the video, shooting time, shooting location, vehicle speed at the time of shooting, weather at the time of shooting, number of lanes on the road taken, and labeled images. Can be provided for logging of URLs.

In addition, the annotation item includes the class ID (major classification -> middle classification -> class) and the location of the bounding box (coordinates (x,y) and width and height of the upper left corner) for a number of N classes extracted from each image. can be defined and presented.

11 illustrates information about a location of a class ID and a bounding box extracted from an image according to an embodiment of the present invention.

Reference numerals B1 to B4 denote bounding boxes, and reference numerals C1 to C4 denote annotations for each bounding box B1 to B4.

Annotation can present information about the class ID and location of the bounding box. For example, reference numeral C1 relates to the bounding box B1, the class ID is a variable information board, and the location information of the bounding box B1 is (x = 570, y = 274) (width = 618, height = 310). can For example, reference numeral C2 relates to the bounding box B2, the class ID is the center line, and the location information of the bounding box B2 may be (x = 242, y = 550) (width = 523, height = 380) .

FIG. 12 is a diagram for explaining the process of performing the data pre-processing step of step S20 of FIG. 6 .

In this case, the major classification categories may include lanes/crosswalks, traffic lights, traffic signs, and other signs.

Each of the major classification categories may be further divided into a plurality of subcategories.

Step S20 may include steps S21 to S23 below.

In step S21, a learning data set for a learning model to be classified based on the major classification category may be built, and a learning model may be created using the built learning data set.

In step S22, static objects related to the plurality of subcategories are extracted from the valid image data by using the generated learning model, and information on locations where the extracted static objects exist within the valid image data. You can create an annotation about a bounding box that is (coordinates).

In step S23, according to the generated annotation, the image corresponding to the annotation and the video corresponding to the image are classified into any one of the major classification categories, thereby pre-processing the video according to the major classification category. image data can be created.

13 is a diagram for explaining a process of generating a learning model according to an embodiment of the present invention.

Steps S10 to S40 are the same as those described above with reference to FIG. 6 .

In step S40, the assigned inspector may perform data search and total inspection through coordinates.

Thereafter, when an error is detected by checking the learning log, step S40, which is a data verification step, is performed again, and when an error is not detected, a learning model may be generated. That is, a learning dataset can be created through data that has undergone final inspection, and through this, it is possible to perform advanced deep learning learning models.

The generated training data set may be classified into any one category among signs, lanes and crosswalks, traffic lights, and other signs.

The generated learning dataset may include information about the source location, resolution, type of vehicle captured, recording time, type of road captured, and information about the area captured for the video data of the collected black box image. .

In addition, the generated learning dataset includes the file name, shooting time, shooting location, vehicle speed at the time of shooting, weather at the time of shooting, number of lanes on the road where the image was taken, and labeling of each image data of the collected black box images. You can include information about the URL containing the image.

In addition, the generated training dataset includes the width and height of the bounding box defining the region of the static object extracted from each image data, the x, y coordinates of the upper left corner, and the class ID of the static object included in the bounding box. can include

Using the above-described embodiments of the present invention, those belonging to the technical field of the present invention will be able to easily implement various changes and modifications without departing from the essential characteristics of the present invention. The content of each claim of the claims may be combined with other claims without reference relationship within the scope understandable through this specification.

Claims (8)

processing device;
filming device;
predictor; and
An object information table including data on a plurality of static objects having one of predetermined types existing on a moving route of a vehicle, the object information table including geolocations of the plurality of static objects; and a storage device storing a plurality of standard reference images provided according to the Road Traffic Act as images representing each of the static objects.
Including,
The processing device,
The difference between the first geolocation, which is the geolocation of the first static object existing on the predicted first movement path of the vehicle, and the current vehicle location, which is the current geolocation of the vehicle obtained from the GPS device, is within a predetermined threshold value. starting photographing using the photographing device from a time point having a value;
extracting a first object image estimated to be the static object having one of the predetermined types from a first image captured by the photographing device;
selecting a first reference image corresponding to the extracted first object image from among the reference images; and
providing at least one of the extracted first object image and the selected first reference reference image to another computing device;
is supposed to run
The prediction device includes a neural network learned according to a supervised learning method,
The neural network uses learning images obtained by photographing from an information collection vehicle as input information for learning, and is supervised using a label for each of one or more object images designated in each of the acquired learning images. ,
The step of extracting the first object image includes inputting the captured first image to the learned neural network and using an output of the learned neural network;
The trained neural network is trained using a predetermined training data set,
The method for generating the learning data set,
Data collection, which collects images captured by a black box installed in vehicles including the information collection vehicle, removes data bias from the captured images, and de-identifies personal information on the images from which the bias is removed. Step (S10);
As a data pre-processing step (S20) of extracting valid image data from the de-identified images in the data collection step, creating a learning model through learning by constructing a learning data set based on the prepared large classification category (S21 ), generating a bounding box, which is information about the location of the static object in the valid image data, and an annotation for the bounding box using the generated learning model (S22), and in the generated annotation The data pre-processing step (S20) including the step (S23) of classifying the captured images according to the major classification category and generating a video and pre-processed image data according to the major classification category;
A data processing step (S30) of providing the generated bounding box and annotation to a device used by the inspector so that the inspector can check the generated bounding box and annotation;
Annotation data with errors is automatically detected using a deep learning algorithm, and the annotation data with errors is detected so that the inspector can check and correct the annotation data with errors. Provided to the device used by the inspector, data verification step (S40); and
a learning data set creation step (S50) of generating the learning data set based on the data for which the verification process by the data processing step and the data verification step is completed;
Including,
In the data collection step,
Image data is collected from a black box installed in a commercial vehicle operated by a driver panel, a commercial vehicle driver, and image data is collected from a vehicle driven by a professional panel equipped with a high-definition black box to supplement the collected black box image. doing; and
Supplementing the collected black box image with image data purchased from a server of a passenger transport service provider;
Including more,
When collecting black box images for the first time, the black box images are collected from a limited number of participants per metropolitan area/metropolitan city,
The step of removing data bias from the collected black box images and de-identifying personal information,
Classifying a region of each image data through matching the collected data of the black box image and GPS information;
Classifying a road type of a place where each image data was collected through matching the collected black box image data and GPS information;
Classifying the car type of the vehicle providing each of the black box images through a pre-stored car DB;
classifying the collected black box images according to the region, the road type, and the vehicle type, and performing filtering on the collected black box images to prevent bias;
deleting sections before and after the point of departure and destination of each of the collected black box images;
deleting audio information other than video from each of the collected black box images;
De-identifying the license plate number of the forward driving vehicle and the next lane vehicle in each of the collected black box images;
deleting traffic accident scenes from each of the collected black box images;
Performing pedestrian de-identification processing when stopping at a crosswalk in each of the collected black box images; and
De-identifying the shooting time and location information in the black box image from each of the collected black box images
Characterized in that it further comprises,
vehicle terminal. According to claim 1,
The other computing device is an autonomous driving device supporting autonomous driving of the vehicle,
The autonomous driving device is configured to display the extracted first object image or the selected first reference reference image on a display device included in the vehicle.
vehicle terminal. According to claim 1,
The step of selecting the first reference reference image,
selecting static objects of a first group existing within a predetermined distance from the first geolocation from among the static objects stored in the object information table;
selecting a first group of standard reference images corresponding to the selected first group of static objects from among the plurality of standard reference images; and
determining an image having the highest similarity with the first object image among the selected reference images of the first group as the first reference image;
including,
vehicle terminal. According to claim 1,
The other computing device is a remote server,
The server,
calculating a similarity between the extracted first object image and the selected first reference reference image;
calculating a degree of deterioration of the first static object represented by the first object image based on the calculated similarity; and
generating a repair or replacement schedule for the first static object according to the calculated degree of deterioration;
which is supposed to run
vehicle terminal. delete According to claim 1,
The processing device,
extracting object images of a first group, which are all object images estimated to be the static object having one of the predetermined types, from the first image; and
providing information about the first geolocation and the extracted object images of the first group to the other computing device;
You can run more
The other computing device is a server configured to access a database including the object information table,
The server,
selecting static objects of a first group existing within a predetermined distance from the first geolocation from among the static objects stored in the object information table;
determining whether a loss object, which is a static object that does not correspond to the object images of the first group, exists among the static objects of the first group; and
generating a repair or replacement schedule for the lost object;
which is supposed to run
vehicle terminal. delete delete

Images