KR20210090040A - Online annotation system for obtaining specific image data and method thereof - Google Patents

Abstract

An objective of the present invention is to annotate required specific information online to efficiently secure specific data when obtaining image data in a vehicle. Since a user personally views with eyes and annotates in real time, reliability is high and data of a desired type can be obtained. The present invention is useful in collecting rare specific objects and environments which do not appear often, uses only short videos and CAN information extracted through online annotation in comparison to conventional offline annotation using the entire recorded videos to reduce manpower, time, and costs, and allows quick development and verification by quickly securing data for a specific case.

Description

ONLINE ANNOTATION SYSTEM FOR OBTAINING SPECIFIC IMAGE DATA AND METHOD THEREOF

The present invention relates to an online annotation system and method for acquiring specific image data, and more particularly, to obtain specific image data that enables efficient specific data acquisition by annotating specific information required online when acquiring image data from a vehicle. It relates to an Online Annotation system and method for

In situations where there are many factors that require attention around the vehicle, the driver tends to slow down. Especially when turning left or right in a narrow alley or intersection, drivers are always nervous about the possibility that an object approaching from an area where the driver's view is not sufficiently secured may collide with the vehicle.

On the other hand, image data annotation is essential for developing and verifying machine learning-based image recognition algorithms as a task to create ground truth.

The existing method generates ground truth by annotating the image offline after collecting image data from the vehicle, and the generated data is used for image application development and verification.

There is a recent Auto Annotation method, but only pre-learned recognizable information is available.

In the case of Offline Annotation, it takes a lot of time because it requires review of all image data. In the case of Auto Annotation, it is difficult to collect objects and specific data that are not within the recognizable range, and it is dependent on the performance of the recognition system. There was a necessary problem.

Republic of Korea Patent Publication Registration No. 10-2002225

An embodiment of the present invention enables efficient specific data to be secured by annotating specific information required online when acquiring image data from a vehicle, and since the user directly sees and performs real-time annotation, the reliability is high and data of the desired type In contrast to the existing Offline Annotation that uses the entire recorded image, it is useful for collecting rare specific objects and environments that can acquire We aim to provide an online annotation system and method for acquiring specific image data that can reduce time and cost and enable rapid development and verification by securing fast data for a specific case.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

The online annotation system for acquiring specific image data according to an embodiment of the present invention selects the specific type of object information when recognizing a specific type of object set by a user through image data captured by a camera while driving a vehicle The personal identification item including the event generation unit that receives input and stores the occurrence of an event and the specific type of object information in the personal identification item, image data captured through the camera, driving information data of the vehicle, and event occurrence information When the image data, the vehicle driving information data, and the personal identification item are stored in the data collection unit and the data collection unit for storing It may include a data extraction unit.

In an embodiment, the event generator may receive specific data information to be acquired in advance, set a personal identification item in which the specific data information is to be stored, and set a type of the specific data type.

In an embodiment, the data collection unit may synchronize and store the image data, the driving information data of the vehicle, and the personal identification item.

In one embodiment, the data extraction unit receives the vehicle movement distance between before and after the occurrence of the event is set in advance, and extracts the frame extraction start point and frame using the vehicle speed information and the frame information per second can be computed at the end point of

In one embodiment, the data extraction unit, the section image between the start point of the frame extraction and the end point of the frame extraction, the driving information data of the vehicle in the section image, and the personal identification item in the section image can be synchronized and saved.

In the online annotation method for acquiring specific image data according to another embodiment of the present invention, when a specific type of object set by a user is recognized through image data captured by a camera while driving a vehicle, the event generating unit generates the specific type The first step of selecting and inputting the object information of the personal identification item and storing the specific type of object information according to the occurrence of an event in the personal identification item, including image data captured through the camera, driving information data of the vehicle, and event occurrence information In the second step of storing the personal identification item in the data collection unit, and when the image data, the vehicle driving information data, and the personal identification item are stored in the data collection unit, an event is generated from the image data in the data extraction unit A third step of extracting and storing a frame of a predetermined portion including information may be included.

In one embodiment, the first step comprises the step of receiving the specific data information to be acquired in advance, the step of receiving the setting of a personal identification item in which the specific data information is to be stored, the type of the specific data type It may include the step of receiving the setting.

In an embodiment, the second step may include synchronizing and storing the image data, the driving information data of the vehicle, and the personal identification item.

In one embodiment, the third step comprises the steps of receiving the vehicle movement distance between before and after the event occurrence in advance, the starting point of frame extraction using the vehicle speed information and frame per second information and It may include calculating an end point of frame extraction.

In one embodiment, in the third step, a section image between the start point of the frame extraction and the end point of the frame extraction, the vehicle driving information data in the section image, and the personal identification in the section image It may include synchronizing and storing the item.

This technology makes it possible to secure specific data efficiently by annotating specific information required online when acquiring image data from a vehicle. Because the user directly sees and performs real-time annotation, the reliability is high and it is possible to acquire the desired type of data. It is useful for collecting rare specific objects and environments that can and do not appear frequently. Annotation manpower, time, and cost using only short images and CAN information extracted through Online Annotation compared to the existing Offline Annotation using the entire recorded image. This has the effect of enabling rapid development and verification by rapidly securing data for a specific case.

In addition, various effects identified directly or indirectly through this document may be provided.

1 is a block diagram illustrating an online annotation system for acquiring specific image data according to an embodiment of the present invention.
2 is a diagram for explaining specific data setting in an online annotation system for acquiring specific image data according to an embodiment of the present invention.
3 is a diagram for explaining storage of an event image in an online annotation system for acquiring specific image data according to an embodiment of the present invention.
4 is a flowchart illustrating an online annotation method for acquiring specific image data according to an embodiment of the present invention.

Hereinafter, some embodiments of the present invention will be described in detail with reference to exemplary drawings. In adding reference numerals to the components of each drawing, it should be noted that the same components are given the same reference numerals as much as possible even though they are indicated on different drawings. In addition, in describing the embodiment of the present invention, if it is determined that a detailed description of a related known configuration or function interferes with the understanding of the embodiment of the present invention, the detailed description thereof will be omitted.

In describing the components of the embodiment of the present invention, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the component from other components, and the essence, order, or order of the component is not limited by the term. In addition, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having meanings consistent with the meanings in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

Hereinafter, embodiments of the present invention will be described in detail with reference to FIGS. 1 to 3 .

1 is a block diagram illustrating an Online Annotation system for acquiring specific image data according to an embodiment of the present invention, and FIG. 2 is a specific data setting in the Online Annotation system for acquiring specific image data according to an embodiment of the present invention , and FIG. 3 is a diagram for explaining storage of an event image in the Online Annotation system for acquiring specific image data according to an embodiment of the present invention.

Referring to FIG. 1 , the online annotation system for acquiring specific image data according to an embodiment of the present invention acquires specific image data by annotating specific information in a driving situation of a vehicle, an event generator 100, It may be configured to include a data collection unit 300 and a data extraction unit 500 .

The event generating unit 100 operates in a vehicle in real time, and when a specific type of object set by the user is recognized through image data captured by a camera while driving the vehicle, the specific type of object information is selected and inputted to the individual. Whether an event has occurred and information about a specific type of object may be stored in the identification item.

The event generating unit 100 may set specific data information to be acquired in advance, set a personal identification item in which specific data information is to be stored, and set a specific data type.

For example, referring to FIG. 2 , a user (a vehicle driver) can set specific data information to be acquired in advance, sets a personal identification item (CAN ID) in which the information is to be stored, and sets the type setting unit 130 ), which is a specific data type, can be set to a plurality of types.

As a specific data type, Type 1 may be set as a sign, Type 2 as a tractor, Type 3 as a kangaroo, Type 4 as an elk, Type N as a damaged sign, etc. may be preset.

By doing so, it can be useful to collect rare objects, certain objects, and certain environmental data that do not appear often.

When specific data information to be acquired is set through the event generating unit 100, the type of the type appears based on the event generation option set by the user during vehicle driving as Online Annotation. Whether an event has occurred and a specific data type can be stored in the personal identification item 110 by inputting it as a shortcut key.

The data collection unit 300 operates in a vehicle in real time, and includes image data captured through a camera, vehicle driving information data and event occurrence information transmitted through C (Chassis) CAN communication. Personal identification item 110 can be saved. In this case, the data collection unit 300 may synchronize and store the image data, the vehicle driving information data, and the personal identification item 110 .

The data extraction unit 500 operates when the image data, the vehicle driving information data, and the personal identification item 110 are stored in the data collection unit 300, and extracts a frame of a certain portion including the event occurrence information from the image data. It can be extracted and saved.

The data extraction unit 500 searches the personal identification item 110, checks whether an event has occurred, and if it does, crops the corresponding frame location-based image and vehicle driving information data and stores it by type.

Referring to FIG. 3 , the data extracting unit 500 performs an image data frame 511 , a vehicle operation data frame 521 , and a personal setting data frame 531 including a personal identification item in a synchronized state in the T frame. If an event occurs, the frame between 'T-A', which is the frame before the event, and 'T+B', which is the frame after the event, is cropped based on the 'T' frame, which is the time of occurrence of the event, and cropped image data The frame 513, the cropped vehicle operation data frame 523, and the cropped personal setting data frame 533 may be synchronized and stored.

In the above-described data cropping method, first, based on the event occurrence frame, the vehicle movement distance is set in advance in a section between before the event occurrence frame and after the event occurrence frame, and the vehicle speed information and frames per second (fps; frame per second) information can be used to calculate the start point of frame extraction and the end point of frame extraction.

If the start point of frame extraction is set to 'A' and the end point of frame extraction is set to 'B', the number of frames at point 'A' and the number of frames at point 'B' are respectively through [Equation 1] can be calculated

[Equation 1]

Frame rate = fps * (vehicle travel distance / vehicle speed)

Next, the data extraction unit 500 performs the image data frame 513 in the period 'T-A' to 'T+B', and the period 'T-A' to 'T+B' based on the event occurrence frame 'T'. The vehicle operation data frame 523 of 'T-A' can be stored in the database by synchronizing the personal identification item frame 533 of the 'T+B' section.

At this time, the image data frame 513 from 'T-A' to 'T+B', the vehicle operation data frame 523 from 'T-A' to 'T+B', and 'T-A' to ' The personal identification item frame 533 of the T+B' section may be stored for each type.

For example, with respect to type 1 data, the image data frame 513 in the section 'T-A' to 'T+B' and the vehicle operation data frame 523 in the section 'T-A' to 'T+B' , 'T-A' to 'T+B' section personal identification item frame 533 may be separately stored, and for type 2 data, image data frame from 'T-A' to 'T+B' section (513), the vehicle operation data frame 523 of the section 'T-A' to 'T+B', and the personal identification item frame 533 of the section 'T-A' to 'T+B' may be stored separately. And for type N data, the image data frame 513 of the section 'T-A' to 'T+B', the vehicle operation data frame 523 of the section 'T-A' to 'T+B', ' The personal identification item frame 533 of the 'T+B' section from T-A' may be stored separately.

Hereinafter, an online annotation method for acquiring specific image data according to another embodiment of the present invention will be described in detail with reference to FIG. 5 . 4 is a flowchart illustrating an online annotation method for acquiring specific image data according to an embodiment of the present invention.

Hereinafter, it is assumed that the online annotation system for acquiring specific image data of FIG. 1 performs the process of FIG. 4 .

First, it includes the step of setting specific data information to be acquired in advance, and it is possible to set a personal identification item in which specific data information is to be stored, and set a specific data type type (S110).

Next, when a specific type of object set by the user is recognized through image data captured by the camera while driving the vehicle (S120), vehicle operation information is received (S130), and the event generator 100 receives the specific type of object Information can be selected and inputted (S140), and whether an event has occurred and information about a specific type of object according to the event occurrence can be stored in the personal identification item 110 (S150).

Subsequently, the personal identification item 110 including the image data captured by the camera, the vehicle driving information data, and the event occurrence information may be synchronized and stored in the data collection unit 300 ( S160 ).

Then, when the image data, vehicle driving information data, and personal identification items are stored in the data collection unit 300, the data extraction unit 500 extracts a frame of a certain portion including the event occurrence information from the image data (S170). ), which can be stored in the database (S180), the vehicle movement distance between before and after the event is set in advance, and the starting point of frame extraction and frame extraction using vehicle speed information and frame per second information The end point can be calculated.

Subsequently, it is possible to synchronize and store the section image between the start point of frame extraction and the end point of frame extraction, vehicle driving information data in the section image, and personal identification items in the section image.

According to the Online Annotation system and method for acquiring specific image data as described above, when acquiring image data from a vehicle, it is possible to secure specific data efficiently by annotating specific information required online. Because real-time annotation is performed, the reliability is high, desired type of data can be acquired, and it is useful for collecting rare specific objects and environments that do not appear frequently. Annotation manpower, time, and cost are reduced by using only extracted short images and CAN information, and rapid development and verification are possible by securing fast data for a specific case.

Meanwhile, the online annotation method for obtaining specific image data according to steps S110 to S180 according to the present invention may be programmed and stored in a computer-readable recording medium.

The above description is merely illustrative of the technical spirit of the present invention, and various modifications and variations will be possible without departing from the essential characteristics of the present invention by those skilled in the art to which the present invention pertains.

Therefore, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain, and the scope of the technical spirit of the present invention is not limited by these embodiments. The protection scope of the present invention should be construed by the following claims, and all technical ideas within the equivalent range should be construed as being included in the scope of the present invention.

100: event generation unit 300: data collection unit
500: data extraction unit

Claims (11)

When a specific type of object set by the user is recognized through the image data captured by the camera while the vehicle is driving, the specific type of object information is selected and input, and whether an event has occurred and the specific type of object information is displayed in the personal identification item. an event generator to store;
a data collection unit for storing the personal identification item including image data captured by the camera, driving information data of the vehicle, and event occurrence information; and
When the image data, the vehicle driving information data, and the personal identification item are stored in the data collection unit, a data extraction unit for extracting and storing a frame of a certain portion including event occurrence information from the image data
Online Annotation system for acquiring specific image data, characterized in that it comprises a. The method according to claim 1,
The event generating unit,
Online Annotation system for acquiring specific image data, characterized in that receiving specific data information to be acquired in advance, receiving a personal identification item in which the specific data information is to be stored, and receiving a type of specific data type. The method according to claim 1,
The data collection unit,
Online Annotation system for acquiring specific image data, characterized in that the image data, the driving information data of the vehicle, and the personal identification item are synchronized and stored. The method according to claim 1,
The data extraction unit,
Specific characterized in that the vehicle movement distance between before and after the event is set in advance, and a start point of frame extraction and an end point of frame extraction are calculated using the vehicle speed information and frame per second information. Online Annotation system for image data acquisition. 5. The method according to claim 4,
The data extraction unit,
A specific image characterized in that the segment image between the start point of the frame extraction and the end point of the frame extraction, the driving information data of the vehicle in the segment image, and the personal identification item in the segment image are synchronized and stored Online Annotation System for Data Acquisition. When a specific type of object set by the user is recognized through the image data captured by the camera while driving the vehicle, the event generator selects and receives information about the specific type of object, and determines whether an event occurs in the personal identification item and the specific type accordingly A first step of storing the object information of the;
a second step of storing the personal identification item including image data captured by the camera, driving information data of the vehicle, and event occurrence information in a data collection unit; and
When the image data, the driving information data of the vehicle, and the personal identification item are stored in the data collection unit, a third step of extracting and storing a frame of a certain portion including the event occurrence information from the image data in the data extraction unit
Online Annotation method for acquiring specific image data, characterized in that it comprises a. 7. The method of claim 6,
The first step is
Specific data information to be acquired includes the step of receiving in advance setting, the step of receiving a personal identification item to be stored in the specific data information is stored, the step of receiving a setting of the type of the specific data type, characterized in that it comprises the steps of: Online Annotation method for image data acquisition. 7. The method of claim 6,
The second step is
Online Annotation method for acquiring specific image data, comprising the step of synchronizing and storing the image data, the driving information data of the vehicle, and the personal identification item. 7. The method of claim 6,
The third step is
Receiving the vehicle movement distance between before the event occurrence and after the event occurrence is set in advance, and calculating a start point of frame extraction and an end point of frame extraction using the vehicle speed information and frame per second information Online Annotation method for acquiring specific image data, characterized in that. 10. The method of claim 9,
The third step is
and synchronizing and storing the section image between the start point of the frame extraction and the end point of the frame extraction, the driving information data of the vehicle in the section image, and the personal identification item in the section image. Online Annotation method for acquiring specific image data. A computer-readable recording medium in which a program for executing the Online Annotation method for acquiring specific image data according to any one of claims 6 to 10 is recorded.

Images