KR102389998B1 - De-identification processing method and a computer program recorded on a recording medium to execute the same - Google Patents

Abstract

The present invention provides a de-identification processing method where personal information from data for machine learning of artificial intelligence (AI) is subjected to de-identification processing. The de-identification processing method comprises the following steps of: allowing a learning data generator to identify an object included in a previously collected 2D image for machine learning of AI; and allowing the learning data generator to subject a part of the identified object to de-identification processing in correspondence to a type of the identified object. According to the present invention, an object included in the collected 2D image is identified and a part of the identified object can be subjected to de-identification processing in correspondence to a type of the identified object. That is, the whole portion of the identified object is not subjected to de-identification processing, but only a partial area of the identified object is selectively subjected to de-identification processing, thereby preventing personal information from being leaked and enhancing learning efficiency of machine learning.

Description

De-identification processing method and a computer program recorded on a recording medium to execute the same

The present invention relates to the processing of data for artificial intelligence (AI) learning. More specifically, it relates to a de-identification processing method for de-identifying personal information in data for machine learning of artificial intelligence (AI), and a computer program recorded on a recording medium for executing the de-identification processing method.

Artificial intelligence (AI) refers to a technology that artificially implements some or all of human learning ability, reasoning ability, and perception ability using computer programs. In relation to artificial intelligence (AI), machine learning refers to learning to optimize parameters with given data using a model composed of multiple parameters. Such machine learning is classified into supervised learning, unsupervised learning, and reinforcement learning according to the type of data for learning.

In general, the design of data for artificial intelligence (AI) learning proceeds in the stages of data structure design, data collection, data purification, data processing, data expansion, and data verification.

To describe each step in more detail, the design of the data structure is made through the definition of an ontology, a definition of a classification system, and the like. Data collection is performed by collecting data through direct shooting, web crawling, or association/professional organizations. Data purification is performed by removing duplicate data from the collected data and de-identifying personal information. Data processing is performed by performing annotations and inputting metadata. Data expansion is performed by performing ontology mapping, and supplementing or extending the ontology as necessary. And, the verification of the data is performed by verifying the validity according to the set target quality using various verification tools.

Meanwhile, automatic driving of a vehicle refers to a system capable of driving by determining the vehicle itself. Such autonomous driving may be divided into gradual stages from non-automation to full automation according to the degree to which the system is involved in driving and the degree to which the driver controls the vehicle. In general, the stage of autonomous driving is divided into six levels classified by the Society of Automotive Engineers (SAE) International. According to the six levels classified by the International Association of Automobile Engineers, Level 0 is non-automated, Level 1 is driver assistance, Level 2 is partial automation, Level 3 is conditional automation, Level 4 is highly automated, and Level 5 is The steps are fully automated steps.

Autonomous driving of a vehicle is performed through mechanisms of perception, localization, path planning, and control. Currently, several companies are developing to implement cognitive and path planning among autonomous driving mechanisms using artificial intelligence (AI). And, the data used for machine learning of artificial intelligence (AI) that can be used for autonomous driving consists of a large number ranging from a few thousand to a maximum of several million.

As such, since the collected data used for machine learning of artificial intelligence (AI) is generally images captured by a camera, it is time-series continuously collected data, and similar images continuously exist depending on the shooting environment. there are many Accordingly, there is a problem that the concentration of work is lowered because workers who perform annotation work have to continuously perform repetitive work on similar images in the process of curation for inputting meta data. there was.

In addition, the importance of the collected artificial intelligence (AI) data used for machine learning differs depending on whether it includes an object directly related to machine learning, the target weather information, and location information. Here, when a large number of data of relatively low importance exist in the data used for machine learning of artificial intelligence (AI), there is a problem in that the learning efficiency of machine learning is lowered.

In addition, among the collected data used for machine learning of artificial intelligence (AI), data containing personal information such as license plates and pedestrian faces must be de-identified to prevent personal information from being leaked. Typically, the de-identification processing method may be a blurring processing or a deep fake processing. However, although this de-identification processing method can prevent personal information leakage, there is a problem in that the learning efficiency of machine learning is lowered because the object becomes unclear or is replaced with another image.

Korean Patent Application Laid-Open No. 10-2020-0042629, ‘Method for generating touch-based annotations and images in mobile devices for artificial intelligence learning, and device therefor’, (published on April 24, 2020)

One object of the present invention is to provide a de-identification processing method for de-identifying personal information in data for machine learning of artificial intelligence (AI).

Another object of the present invention is to provide a computer program recorded on a recording medium to execute a de-identification processing method for de-identifying personal information in data for machine learning of artificial intelligence (AI).

In order to achieve the technical task as described above, the present invention provides a learning data generating device that identifies an object included in a 2D image collected in advance for machine learning of artificial intelligence (AI). and de-identifying, by the apparatus for generating learning data, a portion of the identified object corresponding to the type of the identified object.

Specifically, the de-identification processing includes performing the de-identification processing by blurring a part of the identified object, extracting a landmark from the identified object, and extracting the land Blur processing can be performed on the mark.

The de-identification process performs the de-identification process by blurring a part of the identified object, extracting an edge of the identified object, and blurring based on the extracted edge processing can be performed.

In the de-identification process, blur processing may be performed so as to be spaced apart from the extracted edge by a preset number of threshold pixels.

The de-identification processing may include de-identifying a portion of the identified object, extracting an edge of the identified object, and changing a pattern of the extracted edge.

In the de-identification process, the pattern of the extracted edge may be changed in consideration of at least one of a body shape and a face shape of a region subject to machine learning with respect to the extracted edge.

In the de-identification process, a part of the identified object is deep-fake-processed to perform the de-identification process, and a landmark is extracted from the identified object, the extracted land You can replace the mark with another image that is appropriate for that type of landmark.

In the de-identification process, the extracted landmark may be replaced with a landmark image having a similarity higher than a preset threshold value among landmark images stored in advance according to the landmark type.

In the de-identification processing step, after performing the primary de-identification processing by deep-fake processing a portion of the identified object, a portion of the primary de-identification processing object is blurred (blurring) can be processed to perform secondary de-identification processing.

In order to achieve the technical problem as described above, the present invention proposes a computer program recorded on a recording medium to execute a de-identification processing method capable of de-identifying a part of an object. The computer program is combined with a computing device comprising a memory, a transceiver, and a processor for processing instructions resident in the memory, so that the processor is an artificial intelligence (AI) Identifying an object included in a 2D image previously collected for machine learning, and the processor de-identifies a portion of the identified object corresponding to the type of the identified object In order to execute the processing step, it may be a computer program recorded on a recording medium.

According to embodiments of the present invention, an object included in a collected 2D image may be identified, and a portion of the identified object may be de-identified in response to the identified object type. That is, by selectively de-identifying only some areas of the identified objects without de-identifying the entire identified object, it is possible to improve the learning efficiency of machine learning while preventing personal information leakage.

1 is a block diagram of an artificial intelligence learning system according to an embodiment of the present invention.
2 is a logical configuration diagram of an apparatus for collecting learning data according to an embodiment of the present invention.
3 is a hardware configuration diagram of an apparatus for collecting learning data according to an embodiment of the present invention.
4 is a logical configuration diagram of an apparatus for generating learning data according to an embodiment of the present invention.
5 is a hardware configuration diagram of an apparatus for generating learning data according to an embodiment of the present invention.
6 is a flowchart illustrating a guide providing method according to an embodiment of the present invention.
7 is a flowchart illustrating a data purification method according to an embodiment of the present invention.
8 is a flowchart illustrating a de-identification processing method according to an embodiment of the present invention.
9 and 10 are exemplary views for explaining a guide providing method according to an embodiment of the present invention.
11 is an exemplary view for explaining a guide providing method according to another embodiment of the present invention.
12 is an exemplary diagram for explaining a data purification method according to an embodiment of the present invention.
13 to 16 are exemplary views for explaining a de-identification processing method according to an embodiment of the present invention.
17 is a flowchart illustrating a data purification method according to an embodiment of the present invention.
18 is an exemplary diagram for explaining a process of refining data according to an embodiment of the present invention.

It should be noted that the technical terms used herein are used only to describe specific embodiments, and are not intended to limit the present invention. In addition, the technical terms used in this specification should be interpreted as meanings generally understood by those of ordinary skill in the art to which the present invention belongs, unless otherwise specifically defined in this specification, and excessively inclusive It should not be construed in the meaning of a human being or in an excessively reduced meaning. In addition, when the technical terms used in the present specification are incorrect technical terms that do not accurately express the spirit of the present invention, they should be understood by being replaced with technical terms that can be correctly understood by those skilled in the art. In addition, the general terms used in the present invention should be interpreted according to the definition in the dictionary or according to the context before and after, and should not be interpreted in an excessively reduced meaning.

Also, as used herein, the singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as "consisting of" or "having" should not be construed as necessarily including all of the various components or various steps described in the specification, and some of the components or some steps are included. It should be construed that it may not, or may further include additional components or steps.

Also, terms including ordinal numbers such as first, second, etc. used herein may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but another component may exist in between. On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that no other element is present in the middle.

Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the accompanying drawings, but the same or similar components are assigned the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. In addition, in the description of the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, it should be noted that the accompanying drawings are only for easy understanding of the spirit of the present invention, and should not be construed as limiting the spirit of the present invention by the accompanying drawings. The spirit of the present invention should be interpreted as extending to all changes, equivalents or substitutes other than the accompanying drawings.

As described above, since workers who perform annotation work have to continuously perform repetitive tasks on similar images in the process of curation for inputting meta data, work concentration is lowered. There was a problem. In addition, when there are a lot of data of relatively low importance in the data used for machine learning of artificial intelligence (AI), there is a problem in that the learning efficiency of machine learning is lowered. In addition, the de-identification processing method can prevent personal information leakage, but there is a problem in that the learning efficiency of machine learning is lowered because the object becomes unclear or is replaced with another image.

In order to overcome this limitation, the present invention provides guide information that can be helpful in annotation work of data for machine learning artificial intelligence (AI) together with 2D images, and collects 2D images in advance. We intend to propose means for refining according to importance and de-identifying personal information in data for machine learning.

1 is a block diagram of an artificial intelligence learning system according to an embodiment of the present invention.

1, the artificial intelligence learning system according to an embodiment of the present invention includes a plurality of learning data collection devices 100a, 100b, ..., 100n; 100), a learning data generating device 200, a plurality of It may be configured to include the annotation devices 300a, 300b, ..., 300n; 300 and the artificial intelligence learning device 400 .

As such, since the components of the AI learning system according to an embodiment are merely functionally distinct elements, two or more components are integrated with each other in the actual physical environment, or one component is the actual physical environment. may be implemented separately from each other.

When explaining each component, the learning data collection device 100 collects data for machine learning artificial intelligence (AI) that can be used for autonomous driving, a lidar installed in a vehicle, a camera ( A device that collects data in real time from at least one of a camera), a radar, an ultrasonic sensor, a rain sensor, a position measuring sensor, and a speed detecting sensor.

Characteristically, the learning data collection apparatus 100 according to various embodiments of the present invention extracts information on the collected 2D images, and frames per second of the collected 2D images based on the information on the collected 2D images The number of frames per second may be determined, and 2D images corresponding to the determined number of frames per second may be transmitted to the training data generating apparatus 200 .

That is, the learning data collection apparatus 100 may determine the number of frames per second of the 2D images to be transmitted to the learning data generating apparatus 200 based on the similarity between consecutive 2D images among the collected 2D images.

In addition, the learning data collection device 100 calculates and calculates the degree of influence on machine learning through metadata including at least one of speed information, weather information, sensor operation information, and GPS coordinate information at the time of collecting the 2D image The number of frames per second of the 2D images to be transmitted to the learning data generating apparatus 200 may be determined based on the obtained influence.

The types of sensors that are control objects of the learning data collection device 100 and installed in a vehicle to acquire, photograph, or detect data for machine learning include lidar, camera, radar, and ultrasonic sensor. (Ultrasonic sensor), a rain sensor (rain sensor), may include one or more of a position measurement sensor and a speed detection sensor, but is not limited thereto. In addition, the control target of the learning data collection device 100 and the sensor installed in the vehicle to acquire, photograph, or detect data for machine learning is not limited to being provided one for each type, and a plurality of sensors even of the same type are provided. can be

With the following configuration, the learning data generating device 200 is a device that can be used to design and generate data for machine learning artificial intelligence (AI) that can be used for autonomous driving of a vehicle.

As such, the learning data generating apparatus 200 is basically a device distinct from the learning data verifying apparatus 400, but in an actual physical environment, the learning data generating apparatus 200 and the learning data verifying apparatus 400 are one device. It may be integrated and implemented.

Characteristically, the learning data generating apparatus 200 according to embodiments of the present invention extracts a singularity by analyzing the 2D images collected in advance for machine learning of artificial intelligence (AI), The extracted singularity may be processed by the annotation apparatus 300 as guide information for performing an annotation operation, and may be provided together with 2D images.

In addition, the learning data generating apparatus 200 according to embodiments of the present invention receives shooting information related to the 2D image together with the 2D image collected for machine learning of artificial intelligence (AI) from the learning data collecting device, and shooting information may be analyzed to extract a singularity, and the extracted singularity may be processed into guide information for the annotation operation to be performed by the annotation apparatus 300 and provided to the annotation apparatus 300 together with 2D images.

In addition, the learning data generating apparatus 200 according to embodiments of the present invention analyzes the 2D images collected in advance for machine learning of artificial intelligence (AI) and evaluates the importance, and the 2D images collected according to the evaluated importance At least one of the 2D images may be refined.

In addition, the learning data generating apparatus 200 according to embodiments of the present invention identifies an object included in a 2D image collected in advance for machine learning of artificial intelligence (AI), and identifies an object corresponding to the identified type of object It is possible to de-identify part of the object.

The training data generating device 200 having such a characteristic transmits and receives data to and from the training data collection device 100 , the annotation device 300 , and the artificial intelligence learning device 400 , and performs an operation based on the transmitted/received data Any device that can do this is acceptable. For example, the learning data generating apparatus 200 may be any one of a fixed computing device such as a desktop, a workstation, or a server, but is not limited thereto.

With the following configuration, the annotation device 300 is a local computing device that can be used to perform an annotation operation on 2D images or 3D point cloud data distributed by the training data generating device 200 . As such, all or a part of the annotation device 300 may be a device in which an annotator performs an annotation operation through a clouding service.

Specifically, the annotation apparatus 300 may output one 2D image or 3D point cloud data to be annotated among the 2D images or 3D point cloud data received from the training data generating apparatus 200 to the display.

The annotation device 300 may select a tool according to a signal input from the user through the input/output device. Here, the tool is a tool for setting a bounding box specifying one or more objects included in a 2D image or 3D point cloud data.

The annotation device 300 may receive coordinates according to the selected tool through the input/output device. In addition, the annotation apparatus 300 may specify an object included in the 2D image or 3D point cloud data by setting a bounding box based on the input coordinates. Here, the bounding box is an area for specifying an object to be subjected to artificial intelligence (AI) learning among objects included in the image. As such, the bounding box may have a shape of a rectangle or a cube, but is not limited thereto.

For example, the annotation device 300 receives two coordinates through an input/output device, and uses the input two coordinates as the coordinates of the upper-left vertex and the coordinates of the lower-right vertex in the 2D image. Bounding based on a rectangle By setting the box, you can specify the object included in the 2D image. In this case, the two coordinates may be set by the user inputting one type of input signal twice (eg, clicking the mouse), or the user may inputting two types of input signals once (eg, dragging the mouse). However, the present invention is not limited thereto.

The annotation apparatus 300 may generate 2D image or 3D point cloud data to be annotated, or metadata for a set object according to a signal input from a user through the input/output device. Here, the metadata is 3D point cloud data or 2D image and data for describing an object specified from 3D point cloud data or 2D image. Such metadata includes a category of an object specified from 3D point cloud data or a 2D image, the ratio of the object cut by the angle of view, the ratio of the object to another object or object, the tracking ID of the object, the time the image was taken, the image may include, but are not limited to, the weather conditions on the day that ' was taken, file size, image size, copyright holder, resolution, bit value, aperture transmission amount, exposure time, ISO sensitivity, focal length, aperture opening number, angle of view, white Balance, RGB depth, class name, tag, shooting location, type of road, road surface information or traffic jam information may be further included.

The annotation apparatus 300 may generate an annotation work result based on an object set from the 2D image or 3D point cloud data and the generated metadata. In this case, the annotation operation result may have a JSON (Java Script Object Notation) file format, but is not limited thereto. The annotation apparatus 300 may transmit the generated annotation work result to the training data generating apparatus 200 . In addition to the generated annotation work result, the annotation apparatus 300 may transmit a 2D image or 3D point cloud data in which an object is set to the learning data generating apparatus 200 for verification.

Characteristically, the annotation device 300 according to an embodiment of the present invention provides 2D image or 3D point cloud data to be annotated according to a signal input from a user through an input/output device, or metadata for a set object ( metadata), guide information provided from the training data generating apparatus 200 may be output together.

Here, the guide information may be a section corresponding to a point in time when a photographing environment is changed among the 2D images, and may be displayed together with the 2D images so that a worker performing an annotation operation can identify it.

The annotation apparatus 300 according to an embodiment of the present invention does not transmit the 2D image or 3D point cloud data in which the result of the annotation work and the object are set to the learning data generating apparatus 200, but the input/output constituting the annotation apparatus 300 Control data of the device may be transmitted to the training data generating device 200 .

Here, the control data of the input/output device is data stored in time-series of one or more signals input by the user to control the input/output device while the annotation device 300 performs an annotation operation on the 2D image or 3D point cloud data. can be Here, the user may be referred to as an operator, performer, labeler, or data labeler, but is not limited thereto.

For example, when the annotation device 300 is driven by an operating system according to an event-driven architecture, one or more signals included in the control data of the input/output device are the annotation device 200 . It may be an event message generated by the operating system in response to the control of the input/output device. Also, the annotation device 300 may generate control data of the input/output device by duplicating a system queue in which event messages generated by the operating system are stored in a first-in first-out structure.

As a more specific example, when the operating system of the annotation device 300 corresponds to Windows, the control data of the input/output device includes WM_LBUTONDOWN generated in response to a left button click of the mouse, WM_KEYDOWN generated in response to a keyboard input, etc. Event messages may be included.

The annotation device 300 having the above-described characteristics may be any device as long as it is a device capable of transmitting and receiving data to and from the training data generating device 200 and performing an operation based on the transmitted/received data. For example, the annotation device 300 may be a desktop, a stationary computing device such as a workstation or a server, or a smart phone, a laptop, a tablet, a phablet, or a portable multimedia player. It may be any one of a portable computing device such as a portable multimedia player (PMP), a personal digital assistant (PDA), or an e-book reader (E-book reader).

With the following configuration, the artificial intelligence learning device 400 is a device that can be used for machine learning of artificial intelligence (AI) that can be used for autonomous driving of a vehicle.

Specifically, the artificial intelligence learning apparatus 400 may transmit a requirement for achieving the purpose of artificial intelligence (AI) that can be used for autonomous driving of a vehicle to the learning data generating apparatus 200 . The artificial intelligence learning apparatus 400 may receive data for artificial intelligence (AI) learning from the learning data generating apparatus 200 . In addition, the artificial intelligence learning apparatus 400 may machine-learning artificial intelligence (AI) that can be used for autonomous driving of a vehicle by using the received artificial intelligence (AI) learning data.

As such, the artificial intelligence learning apparatus 400 may be any device as long as it is capable of transmitting and receiving data to and from the learning data generating apparatus 200 and performing an operation using the transmitted/received data. For example, the artificial intelligence learning device 400 may be any one of a desktop, a workstation, or a stationary computing device such as a server, but is not limited thereto.

As described above, the learning data collection device 100 , the learning data generating device 200 , the plurality of annotation devices 300 , and the artificial intelligence learning device 400 have a security line directly connecting the devices, a common Data may be transmitted/received using a network in which one or more of a wired communication network or a mobile communication network is combined.

For example, public wired networks include Ethernet, x Digital Subscriber Line (xDSL), Hybrid Fiber Coax (HFC), and Fiber To The Home (FTTH). However, it is not limited thereto. In addition, the mobile communication network includes Code Division Multiple Access (CDMA), Wideband CDMA, WCDMA, High Speed Packet Access (HSPA), Long Term Evolution, LTE) and 5th generation mobile communication may be included, but are not limited thereto.

2 is a logical configuration diagram of an apparatus for collecting learning data according to an embodiment of the present invention.

As shown in FIG. 2 , the learning data collection apparatus 100 according to an embodiment of the present invention includes a communication unit 105 , an input/output unit 110 , a similarity calculation unit 115 , an influence level calculation unit 120 , It may be configured to include a frame determining unit 125 , a data providing unit 130 , and a storage unit 135 .

As such, the components of the learning data collection apparatus 100 merely represent functionally distinct elements, so that two or more components are integrated with each other in the actual physical environment, or one component is implemented with each other in the actual physical environment. It may be implemented separately.

Each component is described. The communication unit 105 may transmit/receive data to/from multiple sensors installed in the vehicle and the learning data generating device 200 .

Specifically, the communication unit 105 detects from a lidar, a camera, a radar, an ultrasonic sensor, a rain sensor, a position measurement sensor, and a speed sensor installed in the vehicle. It is possible to receive data, 3D point cloud data, 2D image, distance information, weather information, location information, speed information, and the like.

Also, the communication unit 105 may transmit sensed data, 3D point cloud data, 2D image, distance information, weather information, location information, and speed information to the learning data generating apparatus 200 under the control of the data providing unit 130 . .

Here, the communication unit 105 may transmit the 2D images to the learning data generating apparatus 200 according to the number of frames per second of the 2D images determined by the frame determiner 125 .

With the following configuration, the input/output unit 110 may receive a signal from a user through a user interface (UI) or may output an operation result to the outside.

Specifically, the input/output unit 110 provides a basic size of a buffer for storing sensed data, 3D point cloud data, 2D image, distance information, weather information, location information, and speed information, and priority of data to be preferentially stored in the buffer. The ranking can be input from the user.

Also, the input/output unit 110 may receive, from the user, a threshold range, which is a size range for determining the number of frames per second of 2D images to be transmitted to the learning data generating apparatus 200 from among the 2D images. That is, the input/output unit 110 receives from the user the number of frames per second of the 2D images to be transmitted to the learning data generating apparatus 200 matching the similarity or influence calculated by the similarity calculating unit 115 or the influence level calculating unit 120 . can be input.

With the following configuration, the similarity calculator 115 may calculate the similarity between consecutive 2D images among the collected 2D images. In particular, the similarity calculator 115 may generate a red, green, blue (RGB) histogram for a pixel in a continuous 2D image, and calculate the similarity by comparing the generated RGB histogram. Here, the RGB histogram is a graph showing the brightness distribution of each primary color (RGB) in the image. For example, in an RGB histogram, the horizontal axis represents the brightness level of a color, and the vertical axis represents the number of pixels assigned to the color's brightness level. The higher the number of pixels, the brighter and darker the color can be expressed. In this way, the similarity calculator 115 may calculate the similarity by comparing the color saturation and grayscale state of the continuous 2D image through the RGB histogram, and the tendency of white balance.

Also, the similarity calculator 115 may extract an edge of each successive 2D image and calculate the similarity between successive 2D images based on the amount of edge change between the successive 2D images. Here, the similarity calculator 115 may extract an edge with respect to the identified object region or may extract an edge with respect to an object included in the entire 2D image. In this case, the similarity calculator 115 may calculate the similarity by comparing the extracted moments of the edges.

The influence calculator 120 may calculate the influence on machine learning through the collected meta data together with the collected 2D image.

Specifically, the influence calculation unit 120 calculates the influence on machine learning through metadata including at least one of speed information, weather information, sensor operation information, and GPS coordinate information at the time of collecting the 2D image. can

That is, when the moving speed of the vehicle for collecting learning data is higher than the preset threshold, the influence calculator 120 determines the first influence, and the moving speed of the learning data collecting device is lower than the preset threshold. In this case, it may be determined as a second influence level that is lower than the first influence level.

For example, when the speed of the vehicle collecting learning data through speed information is high, the influence calculation unit 120 calculates the influence low because there is a high possibility that the amount of change between consecutive 2D images is small, and the moving speed When is low speed, it is highly likely that the amount of change between 2D images is large, so that the degree of influence can be calculated to be high.

The frame determiner 125 is the number of frames per second of the 2D images to be transmitted to the learning data generating apparatus 200 based on the similarity or influence calculated by the similarity calculator 115 and the influence calculator 120 described above. (frame per second) can be determined.

Here, the frame determiner 125 may determine the number of frames per second for each section of the 2D images. That is, the frame determiner 125 may group the 2D images according to the similarity between consecutive 2D images, and apply a preset number of frames per second to the similarity of each group.

In addition, the frame determiner 125 may group the 2D images according to the degree of influence between successive 2D images, and apply a preset number of frames per second to the influence level of each group.

Also, the frame determiner 125 may determine the number of frames per second for all 2D images. That is, a similarity average value of successive 2D images among the 2D images collected by the learning data generating apparatus 200 may be calculated, and a preset number of frames per second matching the calculated similarity average value may be applied.

In addition, the frame determiner 125 calculates an average value of influence of consecutive 2D images among the 2D images collected in the learning data generating device 200, and a preset number of frames per second matching the calculated average value of influence. can be applied.

The data providing unit 130 may provide the 2D images corresponding to the number of frames per second determined by the frame determining unit 125 to the learning data generating apparatus 200 through the communication unit 105 . That is, the data providing unit 130 may apply the number of frames per second determined by the frame determiner 125 to the collected 2D images, and transmit the applied 2D images to the learning data generating apparatus 200 .

The storage unit 135 may store data necessary for the operation of the learning data collection apparatus 100 .

Specifically, the storage unit 135 may be configured to include a buffer for storing sensed data, 3D point cloud data, 2D image, distance information, weather information, location information, and speed information. In addition, the storage unit 135 may be configured to include a database for storing basic data and rules for calculating the degree of similarity or influence and determining the number of frames per second.

Hereinafter, hardware for implementing the logical components of the learning data collection apparatus 100 as described above will be described in more detail.

3 is a hardware configuration diagram of an apparatus for collecting learning data according to an embodiment of the present invention.

As shown in Figure 3, the learning data collection device 100 is a processor (Processor, 150), a memory (Memory, 155), a transceiver (Transceiver, 160), an input/output device (Input/output device, 165), a data bus (Bus, 170) and may be configured to include a storage (Storage, 175).

The processor 150 may implement the operations and functions of the learning data collection apparatus 100 based on an instruction according to the software 180a in which the method according to the embodiments of the present invention is implemented residing in the memory 155 . . The memory 155 may be loaded with software 180a in which methods according to embodiments of the present invention are implemented.

The transceiver 160 is a lidar (lidar), a camera (camera), a radar (radar), an ultrasonic sensor (ultrasonic sensor), a rain sensor (rain sensor), a position measurement sensor, a speed detection sensor and a learning data generating device 200 can send and receive data. The input/output device 165 may receive data necessary for the operation of the learning data collection device 100 , and may output sensed data, 3D point cloud data, 2D image, distance information, weather information, location information, speed information, and the like. The data bus 170 is connected to the processor 150 , the memory 155 , the transceiver 160 , the input/output device 165 and the storage 175 , and is a movement path for transferring data between each component. can play a role.

The storage 175 stores an application programming interface (API), a library file, a resource file, etc. necessary for the execution of the software 180a in which the method according to the embodiments of the present invention is implemented. can be saved The storage 175 may store the software 180b and the database 185 in which the method according to the embodiments of the present invention is implemented.

The database 185 may store rules and basic data for calculating the degree of similarity or influence, and determining the number of frames per second of 2D images according to the degree of similarity or influence.

According to an embodiment of the present invention, the software 180a, 180b for implementing the control method of the sensors resident in the memory 155 or stored in the storage 175 is the processor 150 artificial intelligence (AI). ) of 2D images for machine learning, extracting information on the collected 2D images, and the number of frames per second of collected 2D images based on the information on the collected 2D images (frame per second) second) may be a computer program recorded on a recording medium to execute the step of determining.

More specifically, the processor 150 may include an application-specific integrated circuit (ASIC), another chipset, a logic circuit, and/or a data processing device. The memory 155 may include read-only memory (ROM), random access memory (RAM), flash memory, memory cards, storage media, and/or other storage devices. The transceiver 160 may include a baseband circuit for processing wired/wireless signals. The input/output device 165 includes an input device such as a keyboard, a mouse, and/or a joystick, and a liquid crystal display (LCD), an organic light emitting diode (OLED) and/or an input device such as a joystick. Alternatively, an image output device such as an active matrix OLED (AMOLED) may include a printing device such as a printer or a plotter.

When the embodiment included in this specification is implemented in software, the above-described method may be implemented as a module (process, function, etc.) that performs the above-described function. The module resides in the memory 155 and may be executed by the processor 150 . The memory 155 may be internal or external to the processor 150 , and may be connected to the processor 150 by various well-known means.

Each component shown in FIG. 3 may be implemented by various means, for example, hardware, firmware, software, or a combination thereof. In the case of implementation by hardware, an embodiment of the present invention provides one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs ( Field Programmable Gate Arrays), a processor, a controller, a microcontroller, a microprocessor, etc. may be implemented.

In addition, in the case of implementation by firmware or software, an embodiment of the present invention is implemented in the form of a module, procedure, function, etc. that performs the functions or operations described above, and is stored in a recording medium readable through various computer means. can be recorded. Here, the recording medium may include a program command, a data file, a data structure, etc. alone or in combination. The program instructions recorded on the recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. For example, the recording medium includes a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a compact disk read only memory (CD-ROM), a digital video disk (DVD), and a floppy disk. Magneto-Optical Media, such as a disk, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions may include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes such as those generated by a compiler. Such hardware devices may be configured to operate as one or more software to perform the operations of the present invention, and vice versa.

Hereinafter, the configuration of the learning data generating apparatus 200 as described above will be described in more detail.

4 is a logical configuration diagram of an apparatus for generating learning data according to an embodiment of the present invention.

As shown in FIG. 4 , the learning data generating device 200 includes a communication unit 205 , an input/output unit 210 , a singularity extraction unit 215 , a guide information processing unit 220 , an importance evaluation unit 225 , and an image. It may be configured to include a refiner 230 , an object identification unit 235 , a de-identification processing unit 240 , and a storage unit 245 .

As such, the components of the learning data generating apparatus 200 merely represent functionally distinct elements, so that two or more components are integrated with each other in the actual physical environment, or one component is implemented with each other in the actual physical environment. It may be implemented separately.

When each component is described, the communication unit 205 may transmit/receive data to and from one or more of the learning data collection apparatus 100 , the annotation apparatus 300 , and the artificial intelligence learning apparatus 400 .

Specifically, the communication unit 205 may receive 2D images and 3D point cloud data from the learning data collection apparatus 100 . Here, the 2D images may be images captured by a camera fixedly installed in the vehicle in order to machine-learning artificial intelligence (AI) that can be used for autonomous driving of the vehicle. In addition, the 3D point cloud data may be point cloud data acquired through a lidar fixedly installed in a vehicle in order to machine-learning artificial intelligence (AI) that can be used for autonomous driving of a vehicle.

The communication unit 205 may distribute and transmit a plurality of 2D images or 3D point cloud data to be annotated to a plurality of annotation devices 300 . The communication unit 205 may receive an annotation work result from each of the plurality of annotation apparatuses 300 . Also, the communication unit 205 may receive control data of the input/output device from each of the plurality of annotation devices 300 . Here, the control data of the input/output device includes the annotation device 300 that has received one or more of the 2D images or 3D point cloud data. In the process of performing an annotation operation on the received 2D image or 3D point cloud data, the user may use the annotation device It may be data stored in time series of one or more signals input to control the input/output device constituting the 300 .

In addition, the communication unit 205 may transmit data for artificial intelligence (AI) learning to the artificial intelligence learning apparatus 300 .

With the following configuration, the input/output unit 210 may receive a signal from a user through a user interface (UI) or output an operation result to the outside.

Specifically, the input/output unit 210 may receive a control signal for designing a data structure for artificial intelligence (AI) learning from a user. The input/output unit 210 may receive input from a user, such as a quota for distributing annotation tasks to the plurality of annotation apparatuses 300 .

With the following configuration, the singularity extraction unit 215 may analyze a continuous 2D image among the collected 2D images, and extract a time point at which the photographing environment changes as a singularity. Specifically, the singularity extractor 215 evaluates the similarity between consecutive 2D images among the collected 2D images, and determines a continuous 2D image having a similarity higher than a preset threshold as the time point at which the shooting environment changes. there is. In this case, the singularity extractor 215 may generate a red, green, blue (RGB) histogram for pixels in a continuous 2D image, and calculate a similarity by comparing the generated RGB histogram. Here, the RGB histogram is a graph showing the brightness distribution of each primary color (RGB) in the image. For example, in an RGB histogram, the horizontal axis represents the brightness level of a color, and the vertical axis represents the number of pixels assigned to the color's brightness level. The higher the number of pixels, the brighter and darker the color can be expressed. As such, the singularity extractor 215 may calculate the similarity by comparing the color saturation and gradation state of the continuous 2D image through the RGB histogram, and the tendency of white balance.

In addition, the singularity extractor 215 may divide each of the collected 2D images into a plurality of regions, analyze each of the divided regions to identify an environmental variable, and extract a singularity based on the amount of change in the environmental variable. In this case, the singularity extractor 215 may calculate the environment variable based on the brightness of the divided area or the RGB (Red, Green, Blue) value. For example, the singularity extraction unit 215 divides each of the collected 2D images into two regions in the vertical direction, and when the change amount of the environment variable in the upper region of the continuous 2D image is higher than a preset threshold, the weather It can be recognized as a changing time point, and when the amount of change of the environmental variable in the lower area is higher than a preset threshold value, it can be recognized as a time when the road changes.

In addition, the singular point extractor 215 may analyze the shooting information and extract a point at which the shooting environment changes as the singular point. Here, the photographing information may be a value sensed by sensors installed in the learning data collection apparatus 100 . Specifically, the singularity extraction unit 215 determines whether the lighting device installed in the learning data collection device 100 is turned on or off through the illuminance sensor installed in the learning data collection device 100, and whether the lighting device is turned on or off. It is possible to determine the time at which the 2D image was photographed, or the time at which the location changes. For example, the singular point extractor 215 may recognize a time when the lighting device is turned on or off as a time when night and day change or a time when entering a tunnel. In addition, the singularity extractor 215 may determine when the weather conditions of the 2D image change through a rain sensor installed in the learning data collection apparatus 100 . In addition, the singularity extractor 215 may determine when the weather conditions of the 2D image change through the amount of change in focus of the camera installed in the learning data collection apparatus 100 .

In the following configuration, the guide information processing unit 220 designates a section corresponding to a point in time when the shooting environment is changed among the collected 2D images, and processes the designated section into guide information to annotate the 2D images together with the 2D images. can be provided to In addition, the guide information processing unit 220 designates a section in which the environment variable of each divided region of the collected 2D images is higher than a preset threshold value, and processes the designated section into guide information to annotate the 2D images together with the 2D images ( 300) can be provided. In addition, the guide information processing unit 220 may designate a section in which the shooting environment is changed, process the designated section as guide information, and provide the 2D images together with the annotation apparatus 300 . For example, when the guide information processing unit 220 outputs one 2D image or 3D point cloud data to be annotated to the display, the 2D image corresponding to the section in which the viewpoint or the shooting environment is changed is a 2D image in which the viewpoint is changed. It can be printed together with possible indications.

With the following configuration, the importance evaluation unit 225 extracts an object included in each of the collected 2D images, calculates a similarity between the extracted object and an object corresponding to a preset type require, importance can be assessed. That is, the importance of each 2D image in machine learning may be determined according to whether or not an object related to machine learning is actually included. Accordingly, the importance evaluator 225 may extract an object included in the collected 2D images and evaluate the importance by comparing the extracted object with an object required for actual machine learning.

In addition, the importance evaluation unit 225 may evaluate the importance based on each 2D image collected together with the 2D images and an environmental factor at the time of photographing. That is, the importance evaluator 225 may evaluate, among the 2D images, a 2D image different from the weather information, which is a target of machine learning, as an image with a low importance as a target of refining. Also, the importance evaluation unit 225 may evaluate, among the 2D images, a 2D image that is different from a photographing point of time that is a target of machine learning as an image of low importance that is a target of refinement. In addition, the importance evaluation unit 225 may evaluate a 2D image having GPS coordinates located within a preset threshold distance from Global Positioning System (GPS) coordinates, which are the target of machine learning, as an image with low importance to be refined. In addition, the importance evaluation unit 225 extracts an object included in each of the collected 2D images, and converts 2D images in which the number of extracted objects is lower than a preset threshold number to an image of low importance to be refined. can be evaluated In addition, the importance evaluation unit 225 selects a 2D image including an object corresponding to a preset type require from among the 2D images in less than a preset required number of 2D images to be refined. image can be evaluated.

The image refiner 230 may refine at least one 2D image among the collected 2D images according to the importance evaluated by the importance evaluation unit 225 . In more detail, the image refiner 230 may refine a 2D image in which a similarity between an object included in the collected 2D image and an object corresponding to a set request type is lower than a preset threshold value. In addition, the image refiner 230 may refine the 2D image whose importance is lower than a preset threshold based on environmental factors at the time the 2D image was captured, that is, weather information, the shooting time, GPS coordinates, and the number of objects. .

With the following configuration, the object identification unit 235 may identify an object from the sensed data, 3D point cloud data, 2D image, and distance information collected by the learning data collection apparatus 100 .

Basically, the object identification unit 235 according to an embodiment of the present invention sets an object area in which an object is predicted to exist in a 2D image based on the three-dimensional coordinates of points included in the 3D point cloud data. can In this case, the object region may be a two-dimensional region composed of vertices and edges connecting the vertices to each other.

The number of vertices constituting the object region may be configured in advance by the learning data generating apparatus 200 in response to computing power.

However, when the importance of the 2D image captured by the camera is changed, the object identification unit 235 may adjust the number of vertices constituting the object region.

For example, the object identification unit 235 may increase the number of vertices constituting the object region in proportion to the size of a period during which a camera installed in a vehicle captures a 2D image.

As another example, the object identification unit 235 may increase the number of vertices constituting the object area in proportion to the moving speed of the vehicle in which the camera is installed.

Meanwhile, in order to identify the object in the 2D image from the 3D point cloud data, the object identification unit 235 may identify points forming a cluster within a preset threshold range among points included in the 3D point cloud data.

The object identification unit 235 may identify the object type based on a width on an X-axis, a height on a Y-axis, and a depth on the Z-axis of the identified cluster. In more detail, the object identification unit 235 determines the width on the X-axis and the height on the Y-axis of the identified cluster based on a rate relation of width, height, and depth provided in advance for each type of object in the database. and the type of object corresponding to the depth on the Z-axis may be identified.

The object identification unit 235 may rotate the 3D model previously provided in the database according to the type of the identified object according to the direction of the optical axis of the camera which photographed the corresponding 2D image in 3D. In addition, the object identification unit 235 may identify a two-dimensional shape viewed from the optical axis direction of the camera as an object region of the three-dimensionally rotated 3D model.

In addition, the object identification unit 235 may configure the vertices and edges by reflecting the two-dimensional shape identified as the object area in the 2D image, thereby setting the object area in which the object is predicted to exist in the 2D image.

On the other hand, the object identification unit 235 extracts an edge with respect to the identified object region, and determines whether an object detection rule corresponding to the type of object and the extracted edge pattern from the database exists. Based on the identified object area can be verified. In this case, the object detection rule is a rule that is distributed by the learning data generating device 200 and enumerates edge patterns classified by type of object to verify the object identified in the 2D image.

Also, when no object is identified in the 2D image, the object identification unit 235 may remove the 3D point cloud data acquired at the same time as the 2D image in which the object is not identified from the storage unit 245 . .

When the object identification unit 235 sets the object area using the 3D model, the de-identification processing unit 240 may perform de-identification processing on the area corresponding to the non-identification processing area previously assigned to the 3D model. .

Specifically, the de-identification processing unit 240 performs de-identification processing by blurring a part of the identified object, extracting a landmark from the identified object, and blurring the extracted landmark Ring processing can be performed. For example, when the identified object is a person, the non-identification processing unit 240 extracts the eyes, nose, and mouth corresponding to the landmark of the person, and selectively blurs only the extracted eyes, nose, and mouth, so that the entire face Learning efficiency can be improved compared to blurring.

In addition, the de-identification processing unit 240 performs de-identification processing by blurring a part of the identified object, extracting the edge of the identified object, and performing blurring processing based on the extracted edge can be done On the other hand, if the entire identified object is blurred, the edge of the object becomes unclear, which affects the annotation work. Accordingly, the non-identification processing unit 240 may extract an edge of the object and perform blurring processing to be spaced apart from the extracted edge by a preset number of threshold pixels. For example, when the identified object is a person, the non-identification processing unit 240 extracts an edge corresponding to the face shape of the person, and blurs the edge so as to be spaced apart from the edge by the preset number of critical pixels to the inside, De-identification processing can be performed in a line that does not interfere with the human shape.

In addition, the de-identification processing unit 240 de-identifies a part of the identified object, but may extract an edge of the identified object and change the pattern of the extracted edge. Specifically, the non-identification processing unit 240 may change the pattern of the extracted edge in consideration of at least one of a body shape and a face shape of a region to be machine learning as the extracted edge. For example, when the target region for machine learning is Korea, the non-identification processing unit 240 may change and apply the extracted edge pattern of the identified object to the average body shape and average face shape of Korea.

In addition, the de-identification processing unit 240 performs a de-identification process by deep-fake processing a part of the identified object, extracting a landmark from the identified object, and the extracted landmark can be replaced with another image suitable for that landmark type. Specifically, the non-identification processing unit 240 may replace the extracted landmark with a landmark image having a similarity higher than a preset threshold value among landmark images stored in advance to match the landmark type. For example, when the identified object is a face, the non-identification processing unit 240 extracts eyes, nose, and mouth, and has a high similarity compared to the extracted eyes, nose, and mouth and the previously stored eye, nose, and mouth images. The eye, nose, and mouth images can be replaced with the extracted eye, nose, and mouth images.

In addition, the de-identification processing unit 240 deep-fakes a portion of the identified object to perform the first de-identification process, and then blurs a portion of the first de-identified object. can be processed to perform secondary de-identification processing.

Hereinafter, hardware for implementing the logical components of the apparatus 200 for generating learning data as described above will be described in more detail.

5 is a logical configuration diagram of an apparatus for generating learning data according to an embodiment of the present invention.

As shown in FIG. 5 , the learning data generating device 200 includes a processor 250 , a memory 255 , a transceiver 260 , an input/output device 265 , and a data bus. (Bus, 270) and storage (Storage, 275) may be configured to include.

The processor 250 may implement the operations and functions of the learning data generating apparatus 200 based on an instruction according to the software 280a in which the method according to the embodiments of the present invention is implemented residing in the memory 255 . . In the memory 255, the software 280a in which the method according to the embodiments of the present invention is implemented may be loaded. The transceiver 260 may transmit/receive data to and from the learning data collection apparatus 100 , the annotation apparatus 300 , and the artificial intelligence learning apparatus 400 . The input/output device 265 may receive data necessary for the operation of the learning data generating device 200 , and output collected and preprocessed 2D images, 3D point cloud data, and annotation work results. The data bus 270 is connected to the processor 250 , the memory 255 , the transceiver 260 , the input/output device 265 , and the storage 275 . can play a role.

The storage 275 stores an application programming interface (API), a library file, a resource file, etc. necessary for the execution of the software 280a in which the method according to the embodiments of the present invention is implemented. can be saved The storage 275 may store the software 280b in which the method according to the embodiments of the present invention is implemented. Also, the storage 275 may store information necessary for performing the method according to the embodiments of the present invention.

According to an embodiment of the present invention, the software (280a, 280b) for implementing the guide providing method, residing in the memory 255 or stored in the storage 275, the processor 250 is artificial intelligence (Artificial Intelligence, AI) ) for machine learning (machine learning), extracting a singularity by analyzing the 2D images collected in advance, and processing the extracted singularity into guide information for an annotation device to perform an annotation operation, the 2D It may be a computer program recorded on a recording medium to execute the step of providing the images together.

According to another embodiment of the present invention, the software (280a, 280b) for implementing the guide providing method, resident in the memory 255 or stored in the storage 275, the processor 250 is artificial intelligence from the learning data collection device. Receiving photographing information related to the collected 2D image together with the 2D image collected for machine learning of (Artificial Intelligence, AI), extracting a singularity by analyzing the photographing information, and the extracted The singularity may be a computer program recorded on a recording medium in order to execute the step of processing the singularity into guide information for the annotation device to perform an annotation operation, and providing the 2D images together.

According to another embodiment of the present invention, the software (280a, 280b) for implementing the data purification method, residing in the memory 255 or stored in the storage 275, the processor 250 is artificial intelligence (Artificial Intelligence, AI) ) to analyze the 2D images collected in advance for machine learning and evaluate the importance and to perform the steps of refining at least one of the 2D images of the collected 2D images according to the evaluated importance It may be a computer program recorded on a recording medium.

According to another embodiment of the present invention, the software (280a, 280b) for implementing the de-identification processing method, resident in the memory 255 or stored in the storage 275, the processor 250 is artificial intelligence (Artificial Intelligence) , AI) identifying an object included in a previously collected 2D image for machine learning and de-identifying a part of the identified object in response to the type of the identified object It may be a computer program recorded on a recording medium to execute the processing step.

More specifically, the processor 250 may include an application-specific integrated circuit (ASIC), another chipset, a logic circuit, and/or a data processing device. The memory 255 may include read-only memory (ROM), random access memory (RAM), flash memory, memory cards, storage media, and/or other storage devices. The transceiver 260 may include a baseband circuit for processing wired and wireless signals. The input/output device 265 includes an input device such as a keyboard, a mouse, and/or a joystick, and a liquid crystal display (LCD), an organic light emitting diode (OLED) and/or an input device such as a joystick. Alternatively, an image output device such as an active matrix OLED (AMOLED) may include a printing device such as a printer or a plotter.

When the embodiment included in this specification is implemented in software, the above-described method may be implemented as a module (process, function, etc.) that performs the above-described function. Modules reside in memory 255 and may be executed by processor 250 . The memory 255 may be internal or external to the processor 250 , and may be coupled to the processor 250 by various well-known means.

Each component shown in FIG. 5 may be implemented by various means, for example, hardware, firmware, software, or a combination thereof. In the case of implementation by hardware, an embodiment of the present invention provides one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs ( Field Programmable Gate Arrays), a processor, a controller, a microcontroller, a microprocessor, etc. may be implemented.

In addition, in the case of implementation by firmware or software, an embodiment of the present invention is implemented in the form of a module, procedure, function, etc. that performs the functions or operations described above, and is stored in a recording medium readable through various computer means. can be recorded. Here, the recording medium may include a program command, a data file, a data structure, etc. alone or in combination. The program instructions recorded on the recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. For example, the recording medium includes a magnetic medium such as a hard disk, a floppy disk, and a magnetic tape, an optical recording medium such as a compact disk read only memory (CD-ROM), a digital video disk (DVD), and a floppy disk. Magneto-Optical Media, such as a disk, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions may include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes such as those generated by a compiler. Such hardware devices may be configured to operate as one or more software to perform the operations of the present invention, and vice versa.

6 is a flowchart illustrating a guide providing method according to an embodiment of the present invention.

Referring to FIG. 6 , the learning data generating apparatus 200 according to an embodiment of the present invention may analyze a continuous 2D image among the collected 2D images, and extract a time point at which the shooting environment changes as a singularity ( S110). Specifically, the learning data generating apparatus 200 evaluates the similarity between consecutive 2D images among the collected 2D images, and determines a continuous 2D image having a similarity higher than a preset threshold value as the time point at which the shooting environment changes. can Also, the learning data generating apparatus 200 may divide each of the collected 2D images into a plurality of regions, analyze each of the divided regions to identify an environmental variable, and extract a singularity based on the amount of change in the environmental variable. In addition, the learning data generating apparatus 200 may analyze the photographing information and extract a time point at which the photographing environment changes as a singular point. Here, the photographing information may be a value sensed by sensors installed in the learning data collection apparatus 100 . Specifically, the learning data generating device 200 determines whether the lighting device installed in the learning data collecting device 100 is turned on or off through the illuminance sensor installed in the learning data collecting device 100 , and turning on or turning off the lighting device. Based on the presence or absence, it is possible to determine when the 2D image is photographed or when to change the location.

Next, the learning data generating apparatus 200 designates a section corresponding to a point in time at which the shooting environment changes among the collected 2D images, and processes the designated section into guide information to the annotation device 300 along with the 2D images. can be provided (S120). In addition, the learning data generating apparatus 200 may designate a section in which an environment variable of each of the divided regions of the collected 2D images is higher than a preset threshold value, and process the designated section as guide information. Also, the learning data generating apparatus 200 may designate a section in which the shooting environment is changed, and process the designated section as guide information.

Next, the learning data generating apparatus 200 may provide the guide information processed in step S120 together with the 2D image to the annotation apparatus 300 ( S130 ).

7 is a flowchart illustrating a data purification method according to an embodiment of the present invention.

Referring to FIG. 7 , the learning data generating apparatus 200 extracts an object included in each of the collected 2D images, and a degree of similarity between the extracted object and an object corresponding to a preset type require It is possible to evaluate the importance by calculating (S210). In addition, the learning data generating apparatus 200 may evaluate the importance based on each 2D image collected together with the 2D images and an environmental factor at the time of the photographing. Also, the learning data generating apparatus 200 may evaluate, among 2D images, a 2D image that is different from a photographing time that is a target of machine learning as an image with a low importance as a target of refining. In addition, the learning data generating apparatus 200 may evaluate a 2D image having GPS coordinates located within a preset threshold distance from Global Positioning System (GPS) coordinates, which are the target of machine learning, as an image of low importance to be refined. . In addition, the learning data generating apparatus 200 extracts an object included in each of the collected 2D images, and selects a 2D image in which the number of extracted objects is lower than a preset threshold number, an image of low importance to be refined can be evaluated as In addition, the learning data generating apparatus 200 is a 2D image containing an object corresponding to a preset type require from among the 2D images in a preset required number or less, the importance to be refined is It can be evaluated as a low image.

Next, the learning data generating apparatus 200 may refine at least one 2D image among the 2D images collected according to the importance evaluated in step S210 ( S220 ). In more detail, the learning data generating apparatus 200 may refine a 2D image in which a similarity between an object included in the collected 2D image and an object corresponding to a set request type is lower than a preset threshold value. In addition, the learning data generating device 200 may refine the 2D image whose importance is lower than a preset threshold based on environmental factors at the time the 2D image was taken, that is, weather information, the shooting time, GPS coordinates, and the number of objects. there is.

Next, the training data generating apparatus 200 may provide the 2D image refined in step S220 to the annotation apparatus 300 ( S230 ).

8 is a flowchart illustrating a de-identification processing method according to an embodiment of the present invention.

Referring to FIG. 8 , the learning data generating apparatus 200 may identify an object from the sensed data, 3D point cloud data, 2D image, and distance information collected by the learning data collecting apparatus 100 ( S310 ). The learning data generating apparatus 200 may identify the type of object based on a width on an X-axis, a height on a Y-axis, and a depth on the Z-axis of the identified cluster. In more detail, the learning data generating apparatus 200 determines the width on the X-axis and the Y-axis of the identified cluster based on a rate relation of width, height, and depth provided in advance for each type of object in the database. It is possible to identify the type of object corresponding to the height and depth on the Z-axis.

Next, if the learning data generating apparatus 200 sets the object region using the 3D model in step S310, de-identification processing may be performed on the region corresponding to the non-identification processing region previously assigned to the 3D model. . Specifically, the learning data generating apparatus 200 performs de-identification processing by blurring a part of the identified object, extracts a landmark from the identified object, and Blur processing can be performed. The learning data generating apparatus 200 performs de-identification processing by blurring a part of the identified object, extracting the edge of the identified object, and performing blurring processing based on the extracted edge can be done In addition, the learning data generating apparatus 200 may de-identify a part of the identified object, but may extract an edge of the identified object and change the pattern of the extracted edge. Specifically, the learning data generating apparatus 200 may change the pattern of the extracted edge in consideration of at least one of a body shape and a face shape of a region to be machine learning as the extracted edge. In addition, the learning data generating apparatus 200 deep-fakes a part of the identified object to perform de-identification processing, extracts a landmark from the identified object, and the extracted landmark can be replaced with another image suitable for that landmark type. Specifically, the learning data generating apparatus 200 may replace the extracted landmark with a landmark image having a similarity higher than a preset threshold value among landmark images stored in advance according to the landmark type.

Next, the training data generating apparatus 200 may provide the 2D image de-identified in step S320 to the annotation apparatus 300 ( S330 ).

9 and 10 are exemplary views for explaining a guide providing method according to an embodiment of the present invention.

9 and 10 , the learning data generating apparatus 200 according to an embodiment of the present invention extracts a singularity by analyzing the collected 2D images, and the extracted singularity is annotated by the annotating device 300 By processing the guide information for performing and providing it with 2D images, it is possible to improve the convenience and concentration of workers performing the annotation work.

For example, as shown in FIG. 9 , the training data generating apparatus 200 may divide the continuous 2D images (a) and (b) into two regions in a vertical direction, respectively. That is, the learning data generating apparatus 200 may divide the upper regions A1 and B1 into regions capable of recognizing the time of change of weather, and the lower regions A2 and B2 into regions capable of recognizing the time of change of road. .

Thereafter, as shown in (a) of FIG. 10 , the learning data generating apparatus 200 generates an RGB histogram of the upper region for two consecutive images in order to recognize the upper region, that is, a weather change time, and compares the RGB histograms. there is.

And, as shown in (b) of FIG. 10 , the learning data generating apparatus 200 recognizes a point T where the amount of change of the RGB histogram is higher than a preset threshold as a time point when the weather changes, and identifies a point when the weather changes as a singularity point. As a result, the annotation device may process it into guide information for performing an annotation operation, and provide it to the annotation device 300 together with 2D images.

11 is an exemplary view for explaining a guide providing method according to another embodiment of the present invention.

Referring to FIG. 11 , the learning data generating apparatus 200 according to an embodiment of the present invention extracts singularities by analyzing photographing information received together with the collected 2D image, and the annotating apparatus 300 for the extracted singularities By processing the guide information for performing the annotation work and providing it together with 2D images, it is possible to improve the convenience and concentration of workers performing the annotation work.

Specifically, the learning data generating device 200 determines whether the lighting device installed in the learning data collecting device 100 is turned on or off through the illuminance sensor installed in the learning data collecting device 100 , and turning on or turning off the lighting device. It is possible to determine when the 2D image was photographed based on the presence or absence.

For example, as shown in (a) of FIG. 11 , when the time point at which the learning data collection device 100 is photographing is daytime, the lighting device is turned off, and as shown in (b), the learning data collection device When it is night when 100 is photographing, it can be confirmed that the lighting device is turned on.

In this way, the learning data generating device 200 can check the shooting time of the corresponding 2D image through whether the lighting device installed in the learning data collecting device 100 for collecting the learning data is turned on or off, and the shooting time is a singular point. The annotation device may process it into guide information for performing an annotation operation, and may provide it to the annotation device 300 together with 2D images.

12 is an exemplary diagram for explaining a data purification method according to an embodiment of the present invention.

12 , the learning data generating apparatus 200 according to an embodiment of the present invention analyzes the collected 2D images to evaluate the importance, and at least one 2D image among the 2D images collected according to the evaluated importance By refining , it is possible to improve the learning efficiency of machine learning by refining 2D images with relatively low importance.

For example, as shown in (a), when the object to be machine learning is a car (object1), the learning data generating apparatus 200 extracts an object included in each of the collected 2D images, It is possible to evaluate the importance by calculating the similarity between the extracted object and a preset request type, that is, an object corresponding to a vehicle.

At this time, as shown in (b), when the extracted object is a human, the learning data generating apparatus 200 refines the 2D image because the degree of similarity of the object corresponding to the set relic type is lower than the preset threshold value. can do.

13 to 16 are exemplary views for explaining a de-identification processing method according to an embodiment of the present invention.

13 to 16 , the learning data generating apparatus 200 according to an embodiment of the present invention identifies an object included in a collected 2D image, and a part of the identified object corresponding to the identified type of object can be de-identified. That is, by selectively de-identifying only some areas of the identified objects without de-identifying the entire identified object, it is possible to improve the learning efficiency of machine learning while preventing personal information leakage.

For example, as shown in FIG. 13, when a face is recognized as an object in a 2D image, if the recognized object is blurred, the face shape itself becomes unclear, so it is used as training data. If used, there may be a problem that the learning efficiency is lowered.

Therefore, as shown in FIG. 14 , the learning data generating apparatus 200 performs de-identification processing by blurring a part of the identified object, but extracts the edge of the identified object, and extracts Blur processing can be performed based on the removed edge. That is, the learning data generating apparatus 200 may perform the blurring process to be spaced apart from the extracted edge by the preset number of threshold pixels to prevent the edge of the recognized object from becoming unclear, thereby increasing learning efficiency.

Also, as shown in FIGS. 15 and 16 , the learning data generating apparatus 200 may extract a landmark from the identified object. In addition, the learning data generating apparatus 200 may perform a blurring process on the extracted landmark to prevent an edge of a recognized object from becoming unclear, thereby increasing learning efficiency.

17 is a flowchart illustrating a data purification method according to an embodiment of the present invention.

Referring to FIG. 17 , the apparatus 100 for collecting learning data according to an embodiment of the present invention may collect 2D images for machine learning of artificial intelligence (AI) ( S100 ). In addition, the learning data collection apparatus 100 may collect sensed data, 3D point cloud data, distance information, weather information, location information, speed information, and the like.

Next, the learning data collection apparatus 100 may extract information about the collected 2D images (S200). That is, the learning data collection apparatus 100 may calculate a similarity between consecutive 2D images among the collected 2D images. In particular, the similarity calculator 115 may generate a red, green, blue (RGB) histogram for a pixel in a continuous 2D image, and calculate the similarity by comparing the generated RGB histogram. Also, the learning data collection apparatus 100 may extract an edge of each successive 2D image and calculate a similarity between successive 2D images based on an edge change amount between successive 2D images. Here, the similarity calculator 115 may extract an edge with respect to the identified object region or may extract an edge with respect to an object included in the entire 2D image. In this case, the similarity calculator 115 may calculate the similarity by comparing the extracted moments of the edges. In addition, the learning data collection apparatus 100 may calculate the degree of influence on machine learning through the collected meta data together with the collected 2D image. Specifically, the learning data collection device 100 calculates the degree of influence on machine learning through metadata including at least one of speed information, weather information, sensor operation information, and GPS coordinate information at the time of collecting the 2D image. can

Next, the learning data collection apparatus 100 may determine the number of frames per second of the 2D images based on the calculated similarity or influence. Here, the learning data collection apparatus 100 may determine the number of frames per second for each section of the 2D images. That is, the frame determiner 125 may group the 2D images according to the similarity between consecutive 2D images, and apply a preset number of frames per second to the group. Also, the learning data collection apparatus 100 may determine the number of frames per second for all 2D images. That is, a similarity average value of successive 2D images among the 2D images collected by the learning data generating apparatus 200 may be calculated, and a preset number of frames per second matching the similarity average value may be applied.

Next, the learning data collection apparatus 100 may provide 2D images corresponding to the determined number of frames per second to the learning data generating apparatus 200 . That is, the learning data collection apparatus 100 may transmit the applied 2D images to the learning data generating apparatus 200 by applying the determined number of frames per second among the collected 2D images.

18 is an exemplary diagram for explaining a process of refining data according to an embodiment of the present invention.

Referring to FIG. 18 , the learning data collection apparatus 100 according to an embodiment of the present invention determines the number of frames per second of the collected 2D images based on information about the collected 2D images, It is possible to improve learning efficiency by refining unnecessary data among the collected data.

More specifically, the learning data collection apparatus 100 calculates a similarity between two consecutive 2D images among the collected 2D images, or performs machine learning through metadata collected together with the collected 2D images. The number of frames per second of 2D images can be determined based on the influence by calculating the degree of influence.

For example, as shown in (a) of FIG. 18 , the learning data collection apparatus 100 may calculate a similarity between two consecutive 2D images among the collected 2D images. In this case, the similarity may be calculated by generating a red, green, blue (RGB) histogram for a pixel in a continuous 2D image, and comparing the generated RGB histogram to calculate the similarity.

And as shown in (b), the learning data collection apparatus 100 may control the number of frames per second of the 2D images to be provided to the learning data generating apparatus 200 according to the similarity between two consecutive 2D images.

As described above, although preferred embodiments of the present invention have been disclosed in the present specification and drawings, it is in the technical field to which the present invention pertains that other modifications based on the technical idea of the present invention are possible in addition to the embodiments disclosed herein. It is obvious to those with ordinary knowledge. In addition, although specific terms have been used in the present specification and drawings, these are only used in a general sense to easily explain the technical contents of the present invention and help the understanding of the present invention, and are not intended to limit the scope of the present invention. Accordingly, the above detailed description should not be construed as restrictive in all respects but as exemplary. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included in the scope of the present invention.

Training data collection device: 100 Training data generation device: 200
Annotation device: 300 AI learning device: 400
Communication unit: 105, 205 Input/output unit: 110, 210
Similarity calculator: 115 Influence calculator: 120
Frame determining unit: 125 Data providing unit: 130
Singularity extraction unit: 215 Guide information providing unit: 220
Importance evaluation unit: 225 Image refinement unit: 230
Object identification unit: 235 Non-identification processing unit: 240
Storage: 135, 245

Claims (10)

identifying, by the learning data generating apparatus, an object included in a 2D image previously collected for machine learning of artificial intelligence (AI); and
de-identifying, by the learning data generating device, a portion of the identified object in response to the type of the identified object;
including,
The de-identification processing step is
A portion of the identified object is de-identified, but an edge of the identified object is extracted, and a pattern of the extracted edge is changed,
The de-identification processing step is
De-identification processing method, characterized in that the extracted edge is changed in a pattern of the extracted edge in consideration of at least one of a body shape and a face shape of a region subject to machine learning. The method of claim 1, wherein the de-identification processing comprises:
A part of the identified object is subjected to blurring to perform the de-identification process, a landmark is extracted from the identified object, and a blurring process is performed on the extracted landmark A method of de-identifying processing. The method of claim 1, wherein the de-identification processing comprises:
A part of the identified object is subjected to blurring to perform the de-identification process, an edge of the identified object is extracted, and the blurring process is performed based on the extracted edge, characterized in that A non-identifying method of processing. The method of claim 3, wherein the de-identification processing comprises:
The de-identification processing method, characterized in that blur processing is performed to be spaced apart from the extracted edge by a preset number of threshold pixels. The method of claim 1, wherein the de-identification processing comprises:
A part of the identified object is deep-fake-processed to perform the de-identification process, but a landmark is extracted from the identified object, and the extracted landmark is applied to the corresponding landmark type. A de-identification processing method, characterized in that it is replaced with another image that fits. The method of claim 5, wherein the de-identification processing comprises:
De-identification processing method, characterized in that the extracted landmark is replaced with a landmark image having a similarity higher than a preset threshold value among landmark images stored in advance to match the landmark type. The method of claim 1, wherein the de-identification processing comprises:
After performing a first de-identification process by deep-fake processing a part of the identified object, a second de-identification process is performed by blurring a part of the first de-identified object A non-identification processing method, characterized in that performing a. memory;
transceiver; and
Combined with a computing device configured to include a processor (processor) for processing instructions resident in the memory,
identifying, by the processor, an object included in a 2D image previously collected for machine learning of artificial intelligence (AI); and
and executing, by the processor, de-identifying a portion of the identified object corresponding to the type of the identified object;
The de-identification processing step is
A portion of the identified object is de-identified, but an edge of the identified object is extracted, and a pattern of the extracted edge is changed,
The de-identification processing step is
A computer program recorded on a recording medium, characterized in that the extracted edge is changed in a pattern of the extracted edge in consideration of at least one of a body shape and a face shape of a region subject to machine learning. The method of claim 8, wherein the de-identification processing comprises:
A part of the identified object is subjected to blurring to perform the de-identification process, a landmark is extracted from the identified object, and a blurring process is performed on the extracted landmark A computer program recorded on a recording medium. The method of claim 8, wherein the de-identification processing comprises:
A part of the identified object is subjected to blurring to perform the de-identification process, an edge of the identified object is extracted, and the blurring process is performed based on the extracted edge, characterized in that A computer program recorded on a recording medium.

Images