KR20230053279A - Method for providing a meta data curation guide based on sensor information and a computer program recorded on a recording medium for executing the same - Google Patents

Abstract

Disclosed is a data refinement method of a collection device of artificial intelligence (AI) machine learning data. The method may comprise: a step of collecting, by an annotation device, a plurality of images for an object, which is a target of annotation work for AI learning; a step of acquiring, by the annotation device, filming time information of the collected images; a step of comparing, by the annotation device, environment information for the plurality of images based on the filming time information of the images; and a step of determining, by the annotation device, a change in the environment information of the plurality of images. Accordingly, the present invention can generate metadata for the images with the changed environment information, and curation guidance, thereby improving the convenience of a curation process.

Description

Method for providing a meta data curation guide based on sensor information and a computer program recorded on a recording medium for executing the meta data curation guide based on sensor information same}

The present invention relates to the processing of artificial intelligence (AI) learning data. More specifically, it relates to a method for providing a metadata curation guide based on sensor information and a computer program recorded on a recording medium to execute the method.

Artificial intelligence (AI) refers to a technology that artificially implements some or all of human learning abilities, reasoning abilities, and perception abilities using computer programs. In relation to artificial intelligence (AI), machine learning refers to learning to optimize parameters with given data using a model composed of multiple parameters. Such machine learning is classified into supervised learning, unsupervised learning, and reinforcement learning according to the form of learning data.

In general, designing data for artificial intelligence (AI) learning proceeds in the steps of data structure design, data collection, data refinement, data processing, data expansion, and data verification.

To describe each step in more detail, data structure design is performed through ontology definition, classification system definition, and the like. Data collection is performed by collecting data through direct filming, web crawling, or associations/professional organizations. Data purification is performed by removing redundant data from collected data and de-identifying personal information. Data processing is performed by performing annotation and inputting metadata. Data extension is performed by performing ontology mapping and supplementing or extending the ontology as needed. In addition, data verification is performed by verifying validity according to the set target quality using various verification tools.

Data processing is also referred to as data labeling. More specifically, the annotation work of data processing is performed by a worker processing a bounding box for an object included in an image. Here, the bounding box is an area for specifying the position and shape of an object included in the image.

In addition, the metadata input operation of data processing is performed by a worker inputting image information, copyright information, shooting condition information, annotated object information, environment information, and the like. For example, image information may include a file name, file size, and image size. Copyright information may include information about the copyright holder of the image. The photographing condition information may include resolution, bit value, aperture transmittance, exposure time, ISO sensitivity, focal length, aperture value, angle of view, white balance, and RGB depth. Object information may include annotation type, coordinate value, area size, class name, tag item, clipping, major classification, middle classification, small classification, and instance information. In addition, the environment information may include a shooting date and time, a shooting location, and weather information.

The number and type of information that can be input in the metadata input process of data processing may vary depending on the purpose of artificial intelligence (AI) learning. For example, in the case of artificial intelligence (AI) learning data for performing automatic driving, environment information may further include the type of road on which an image is captured, road surface information, traffic congestion information, and the like. can

As described above, data processing is performed on a large number of data ranging from a few thousand to several million. Accordingly, various means capable of more easily processing a large number of data and more easily verifying the processed data are required.

The raw data collected through the learning data collection device is provided to the annotation server after the data collection server performs de-identification and duplicate data removal on numerous raw data. The annotation server provides primitive data received from the data collection server to numerous labelers (annotators) to perform annotation and input meta data.

Since the raw data collected by the learning data collection device is generally data collected continuously in time series and is very similar to the shooting environment in many cases, even if the labeler provides a simple guide in the process of entering meta data (i.e., curation), the queue The convenience of the simulation process can be greatly improved.

Korean Patent Laid-open Publication No. 10-2020-0042629, ‘Method and apparatus for generating touch-based annotations and images in mobile devices for artificial intelligence learning’, (published on April 24, 2020)

One object of the present invention is to provide a method for providing a metadata curation guide based on sensor information.

Another object of the present invention is to provide a computer program recorded on a recording medium to execute a method capable of automatically adding metadata.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

In order to achieve the technical problem as described above, the present invention proposes a method for providing a metadata curation guide based on sensor information. The method includes the steps of, by an annotation device, collecting a plurality of images of an object to be annotated for artificial intelligence (AI) learning; obtaining, by the annotation device, photographing time point information of the collected images; Comparing, by the annotation device, environment information of the plurality of images based on photographing time information of the images; and determining, by the annotation device, a change in environment information of the plurality of images.

Here, the plurality of images may be images continuously collected in time series, and the photographing time information is collected through at least one of an illuminance sensor for controlling a vehicle light, a photographing setting value of a camera, and a rain sensor of the vehicle. information may be included.

In addition, the step of determining the change of the environment information may include any one of a time point when the light of the vehicle is turned on or off by the illuminance sensor, a time point at night/day change, and a time point entering a tunnel.

In order to achieve the technical problem as described above, the present invention proposes a computer program recorded on a recording medium to execute the processing method as described above. The program may include a memory; transceiver; and a processor configured to process instructions resident in the memory. In addition, the computer program may include: collecting, by the processor, an annotation device, a plurality of images of an object to be annotated for artificial intelligence (AI) learning; obtaining, by the processor, photographing time point information of the collected image; comparing, by the processor, environment information of the plurality of images based on photographing time information of the images; It may be a computer program recorded on a recording medium to execute; and determining, by the processor, a change in environmental information of the plurality of images.

Details of other embodiments are included in the detailed description and drawings.

According to conventional means, when AI learning data is requested from a customer who wants to learn AI, after that, data structure design, data collection, data refinement, data processing, It took a relatively long time to expand data, verify data, generate artificial intelligence (AI) learning data based on the results, and deliver it to customers.

In contrast, according to the embodiments of the present invention, big data built in advance in response to the needs of customers who want to learn artificial intelligence (AI) is simply mined or filtered for artificial intelligence (AI) learning. By generating data and immediately delivering the generated AI learning data, the period required to generate and deliver AI learning data can be greatly shortened.

In particular, the present invention is a data collection server, prior to providing primitive data to the annotation server, since it can provide information about the point in time when the shooting environment of raw data, including primitive data, is changed, the labeler detects that the shooting environment of raw data has changed. There is an effect of performing curation with high concentration only on the viewpoint.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

It should be noted that the technical terms used in this specification are only used to describe specific embodiments and are not intended to limit the present invention. In addition, technical terms used in this specification should be interpreted in terms commonly understood by those of ordinary skill in the art to which the present invention belongs, unless specifically defined otherwise in this specification, and are overly inclusive. It should not be interpreted in a positive sense or in an excessively reduced sense. In addition, when the technical terms used in this specification are incorrect technical terms that do not accurately express the spirit of the present invention, they should be replaced with technical terms that those skilled in the art can correctly understand. In addition, general terms used in the present invention should be interpreted as defined in advance or according to context, and should not be interpreted in an excessively reduced sense.

Also, singular expressions used in this specification include plural expressions unless the context clearly indicates otherwise. In this application, terms such as "consisting of" or "having" should not be construed as necessarily including all of the various components or steps described in the specification, and some of the components or steps are included. It should be construed that it may not be, or may further include additional components or steps.

Also, terms including ordinal numbers such as first and second used in this specification may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but other components may exist in the middle. On the other hand, when a component is referred to as “directly connected” or “directly connected” to another component, it should be understood that no other component exists in the middle.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings, but the same or similar components are given the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. In addition, in describing the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description will be omitted. In addition, it should be noted that the accompanying drawings are only for easily understanding the spirit of the present invention, and should not be construed as limiting the spirit of the present invention by the accompanying drawings. The spirit of the present invention should be construed as extending to all changes, equivalents or substitutes other than the accompanying drawings.

Processing of images to create data for artificial intelligence (AI) learning is performed on a large number of images ranging from a few thousand to several million. Therefore, according to the conventional means, it took a relatively long time to generate AI learning data and deliver it to the customer after receiving the customer's request to learn artificial intelligence (AI).

In order to solve these difficulties, the present invention intends to propose various means capable of more easily processing a large number of images and more easily verifying the processed data.

The learning data collection device according to the present invention is described based on being applied to a vehicle, but is not limited to being installed in a vehicle, and is a concept including various embodiments for collecting artificial intelligence learning data.

An artificial intelligence learning system according to an embodiment of the present invention may include a learning data generating device, one or more annotation devices, a learning data verification device, and an artificial intelligence learning device.

In addition, the artificial intelligence learning system according to another embodiment of the present invention includes a plurality of groups (Group-a, Group-b..., Group-m) consisting of one or more annotation devices and learning data verification devices, a learning data generating device and It may be configured to include an artificial intelligence learning device.

Since the components of the artificial intelligence learning system according to various embodiments are merely functionally distinct elements, two or more components are integrated and implemented in the actual physical environment, or one component is implemented in the actual physical environment. may be implemented separately from each other.

The description of each component is a device that can be used to design and generate artificial intelligence (AI) learning data. The learning data generating device is basically a device that is distinguished from the learning data verifying device, but the learning data generating device and the learning data verifying device may be integrated into one device in an actual physical environment.

Characteristically, the learning data generating device according to an embodiment of the present invention may preemptively generate AI learning data before a request for AI learning data is received from the AI learning device. Big data can be built.

In addition, the learning data generating device generates AI learning data by mining or filtering preemptively built big data when a request value according to the customer's needs is received from the artificial intelligence learning device, The generated artificial intelligence (AI) learning data may be transmitted to an artificial intelligence learning device.

Any device capable of transmitting and receiving data to and from an annotation device, a learning data verification device, and an artificial intelligence learning device, and performing calculations based on the transmitted and received data may be used as the learning data generation device having such characteristics. . For example, the learning data generating device may be any one of a fixed computing device such as a desktop, workstation, or server, but is not limited thereto.

With the following configuration, the annotation device is a device that can be used to annotate images distributed by the learning data generating device.

Characteristically, the annotation device according to an embodiment of the present invention has a tracking ID (tracking ID) capable of tracking a specified object from each image (ie, an image to be annotated) distributed by the learning data generating device. tracking identifier) can be assigned. In addition, the annotation device may automatically assign environment information to each image distributed by the learning data generating device.

Any device capable of transmitting/receiving data to and from the learning data generating device and the learning data verifying device and performing calculations based on the transmitted/received data may be used as the annotation device having such characteristics. For example, the annotation device may be a stationary computing device such as a desktop, workstation, or server, or a smartphone, laptop, tablet, phablet, or portable multimedia player. Player, PMP), personal digital assistants (PDA), or e-book reader (E-book reader), and may be any one of mobile computing devices.

In addition, all or part of the annotation device may be a device in which an annotation worker performs annotation through a clouding service.

With the following configuration, the learning data verification device is a device that can be used to verify artificial intelligence (AI) learning data. That is, the learning data verification device is a device capable of verifying whether the annotation job result generated by the annotation device meets a pre-set target quality or whether the annotation job result is valid for artificial intelligence (AI) learning.

Specifically, the training data verification device may receive an annotation job result from the annotation device. Here, the annotation work result may include coordinates of an object specified by an annotation worker and metadata about an image or object. The metadata of the annotation work result may include image information, copyright information, photographing condition information, specific object information, and environment information, but is not limited thereto. Such an annotation work result may have a JSON (Java Script Object Notation) file format, but is not limited thereto.

The learning data verification device may inspect the received annotation work result. To this end, the learning data verification device may perform inspection using a script for annotation work results. Here, the script is a code for verifying whether or not the target quality previously set for the annotation work result meets or whether the data is valid.

Characteristically, the learning data verification apparatus according to the embodiments of the present invention can provide a user interface (UI) that can more easily inspect the tracking ID and environment variables of objects included in the annotation work result.

Specifically, in outputting a user interface (UI) capable of verifying the metadata of the annotation work result, the training data verification device displays a cropped image of an object to which the same tracking ID is assigned from images included in an image group. ) are collected, the cropped images collected on the user interface (UI) may be sequentially arranged and output according to the order of the images taken.

Here, the image group is a set of annotated images. Such image groups may be formed for each annotation device or project. For example, the image group includes images taken consecutively in time series by the same camera as the camera that took the image to be annotated, and other images arranged adjacent to the camera that took the image to be annotated. Images spatially continuously photographed by a camera may be included, but are not limited thereto.

Also, the cropped image is an image obtained by cropping a part of an image that is an annotation work target based on coordinates set to specify an object by annotation.

In addition, the learning data verification device cross-contrasts the patterns of RGB (Red, Green, Blue) values of the objects included in each of the cropped images and the pattern of the edge, targeting cropped images of objects to which the same tracking ID has been assigned. , it can independently inspect whether the same tracking ID is assigned to each object.

Conversely, the learning data verification apparatus cross-contrasts the RGB value pattern and the edge pattern of objects to which different tracking IDs have been assigned, targeting the cropped images of objects to which different tracking IDs have been assigned. You can also check whether different tracking IDs are assigned correctly.

In this case, the pattern of RGB values means a pattern of a ratio of RGB values to pixels in an image and an arrangement order of RGB values. And, the pattern of the edge means a pattern for the number of pixels included in an enclosure closed by the edge extracted from the image and the relative position where the edge is located in the image.

And, when the learning data verification device determines that it is incorrect to assign the same tracking ID or that different tracking IDs are incorrect, the object to which the tracking ID determined to be incorrect is assigned on the user interface (UI). information can be output.

In this way, the learning data verification device outputs a user interface (UI) in which cropped images of a plurality of objects assigned the same tracking ID are successively arranged, and the user including the result of self-checking whether the tracking ID of the objects is correctly assigned By outputting the interface (UI), the user of the learning data verification device can easily check and inspect a plurality of objects determined to be the same at a glance.

On the other hand, in outputting a user interface (UI) capable of verifying the metadata of the annotation work result, the learning data verification device is expressed as a figure having a preset color corresponding to a value included in each metadata. In order to be possible, each of the images included in the image group may be graphicalized and included in the user interface (UI).

For example, the training data verification apparatus may express each of the images included in the image group as a single line having a color corresponding to a value included in the metadata. The learning data verification apparatus may construct a bar-type graphic by continuously arranging each line segment in the direction of the width of the line segment according to the sequence of images captured. Also, the learning data verification device may output the configured bar-shaped graphic by including it in the user interface (UI).

In this way, since the learning data verification apparatus outputs a user interface (UI) in which a plurality of images are simply graphic according to the value of metadata, the user of the learning data verification apparatus can easily check the metadata of the plurality of images at a glance. It is possible to intuitively identify an image having a specific metadata value different from other images.

And, the learning data verification device may transmit annotation work results and inspection results received from the annotation devices to the learning data generating device.

Any device capable of transmitting and receiving data to and from an annotation device and a learning data generating device and performing an operation based on the transmitted and received data may be used as the learning data verification device having the characteristics described above. For example, the learning data verification device may be any one of fixed computing devices such as a desktop, workstation, or server, but is not limited thereto.

With the following configuration, the artificial intelligence learning device is a device that can be used to develop artificial intelligence (AI).

Specifically, the artificial intelligence learning device may transmit a request value including requirements that AI learning data must satisfy in order for AI to achieve a development purpose, to the learning data generating device. The artificial intelligence learning device may receive artificial intelligence (AI) learning data from the learning data generating device. In addition, the artificial intelligence learning device may machine learn the artificial intelligence (AI) to be developed using the received artificial intelligence (AI) learning data.

As such, the artificial intelligence learning device may be any device capable of transmitting and receiving data to and from the learning data generating device and performing calculations using the transmitted and received data. For example, the artificial intelligence learning device may be any one of a desktop, a workstation, or a fixed computing device such as a server, but is not limited thereto.

As described above, the learning data generating device, one or more annotation devices, the learning data verification device, and the artificial intelligence learning device use a network in which at least one of a secure line, a common wired communication network, or a mobile communication network is directly connected between devices. to send and receive data.

For example, public wired communication networks may include Ethernet, x Digital Subscriber Line (xDSL), Hybrid Fiber Coax (HFC), and Fiber To The Home (FTTH). It may be, but is not limited thereto. In addition, in the mobile communication network, Code Division Multiple Access (CDMA), Wideband CDMA (WCDMA), High Speed Packet Access (HSPA), Long Term Evolution, LTE) and 5th generation mobile telecommunication may be included, but is not limited thereto.

The annotation device may include a communication unit, an input/output unit, a storage unit, an object specification unit, a metadata generation unit, and a result generation unit.

Since the components of the annotation device are merely functionally distinct elements, two or more components may be integrated and implemented in an actual physical environment, or one component may be implemented separately from each other in an actual physical environment. There will be.

Describing each component, the communication unit may transmit and receive data to and from the learning data generating device and the learning data verifying device.

Specifically, the communication unit may receive a plurality of images distributed by the learning data generating device. Here, the image is an image that is a target of annotation work for artificial intelligence (AI) learning. Such a plurality of images may be received individually or collectively.

And, the communication unit may transmit the annotation work result to the learning data verification device. Such an annotation work result may have a JSON file format, but is not limited thereto.

With the following configuration, the input/output unit may receive a signal from a user through a user interface (UI) or output an operation result to the outside. Here, the user means a person who performs annotation work. Such a user may be referred to as a worker, performer, labeler, data labeler, and the like, but is not limited thereto.

Specifically, the input/output unit may output an image to be an annotation work. The input/output unit may receive a control signal for setting a bounding box from a user. In addition, the input/output unit may overlay and output the bounding box on the image.

The input/output unit may receive a control signal for setting metadata of an image from a user.

With the following configuration, the storage unit can store data required for annotation work.

Specifically, the storage unit may store an image received through the communication unit. Also, the storage unit may store project properties, image properties, or user properties received through the communication unit.

With the following configuration, the object specifying unit may specify an object included in an image that is a target of annotation work.

Specifically, the object specification unit may load an image that is a target of annotation work into memory. The object specifying unit may select a tool according to a user's control signal input through the input/output unit. Here, the tool is a tool for setting a bounding box that specifies one or more objects included in the image.

The object specifying unit may receive coordinates through the selected tool according to a control signal input through the input/output unit. The object specifying unit may specify an object included in the image by setting a bounding box based on the input coordinates.

Here, the bounding box is an area for specifying an object to be learned by artificial intelligence (AI) among objects included in the image. Such a bounding box may have a rectangle or polygon shape, but is not limited thereto.

For example, the object specification unit receives two coordinates from the user through the input/output unit, and sets a bounding box based on a rectangle having the input two coordinates as the coordinates of the top left vertex and the coordinates of the bottom right vertex in the image. Thus, an object included in the image can be specified.

In this case, the two coordinates may be set by the user inputting one type of input signal twice (eg, mouse click) or by the user inputting two types of input signal once (eg, mouse drag). It may, but is not limited thereto.

With the following configuration, the metadata generation unit may generate metadata for an image that is a target of annotation work.

Here, the metadata may include image information, copyright information, shooting condition information, annotated object information, and environment information, but is not limited thereto. Image information included in metadata may include a file name, file size, and image size. Copyright information may include information about the copyright holder of the image. The photographing condition information may include resolution, bit value, aperture transmittance, exposure time, ISO sensitivity, focal length, aperture value, angle of view, white balance, and RGB depth. Object information may include annotation type, coordinate value, area size, class name, tag item, clipping, major classification, middle classification, small classification, and instance information. The environmental information may include shooting date and time, shooting location, weather information, road type, road surface information, and traffic jam information. In addition, the number and type of information that may be included in metadata is not limited thereto.

The metadata generator according to an embodiment of the present invention may set specific values to be included in the metadata according to a control signal input from the user, but characteristically, the metadata generator does not rely on the user's control signal, and the tracking ID of the object and environmental information may be automatically determined or assigned. In other words, the metadata generation unit may generate metadata by including the coordinates in the image of the specified object, the automatically assigned tracking ID, and the automatically determined environment information.

Specifically, the metadata generation unit may automatically assign a tracking ID to an object specified by the object specification unit. Here, the tracking ID is a unique identifier of an object assigned to track an object from an image group. An image group is a set of annotated images. For example, the image group includes images taken consecutively in time series by the same camera as the camera that took the image to be annotated, and other images arranged adjacent to the camera that took the image to be annotated. Images spatially continuously photographed by a camera may be included, but are not limited thereto.

The metadata generator according to an embodiment of the present invention may assign a tracking ID based on the color of the object. Specifically, the metadata generator may identify a pattern of RGB values for an area occupied by an object in an image. The metadata generation unit may generate an identifier corresponding to the identified pattern of RGB values and assign it as the tracking ID of the object specified by the object specification unit. Here, the pattern of RGB values means a pattern of ratios of RGB values to pixels and arrangement order of RGB values.

In this case, before the metadata generation unit newly generates the identifier, the pattern of RGB values for the objects specified from the images included in the image group and the pattern of RGB values identified from the object specified by the object specifying unit are determined. Similar objects can be searched by comparing each other. Also, when a similar object is searched for from the images included in the image group, the metadata generator may assign a tracking ID assigned to the searched similar object to the tracking ID of the object specified by the object specification unit.

The metadata generation unit according to another embodiment of the present invention may assign a tracking ID based on the shape of the object. Specifically, the metadata generation unit may extract an edge of an image (edge detection). The metadata generation unit may identify only the edge of the object specified by the object specification unit from all edges of the extracted image. The metadata generation unit may generate an identifier corresponding to the edge pattern of the identified object and assign it as the tracking ID of the object specified by the object specification unit. Here, the pattern of the edge means a pattern for the number of pixels included in the area closed by the edge extracted from the image and the relative position of the edge within the image.

In this case, before the metadata generator newly generates the identifier, the edge pattern of the objects specified from the images included in the image group and the edge pattern of the object identified from the object specified by the object specifying unit are mutually related. You can search for similar objects by comparing them. Also, when a similar object is searched for from the images included in the image group, the metadata generator may assign a tracking ID assigned to the searched similar object to the tracking ID of the object specified by the object specification unit.

The metadata generation unit according to another embodiment of the present invention may assign a tracking ID based on symbols or characters included in objects. Specifically, the metadata generating unit may identify whether a symbol or character is included in an area occupied by an object in an image. The metadata generator may search objects specified from the images included in the image group for similar objects including the same symbol or character as the symbol or character identified from the object specified by the object specification unit. Also, when a similar object is searched for from the images included in the image group, the metadata generator may assign a tracking ID assigned to the searched similar object to the tracking ID of the object specified by the object specification unit.

Meanwhile, the metadata generating unit may automatically determine environment information about an image that is a target of annotation work. Here, the environment information refers to information related to a place or time point at which an image was captured by a camera. Such environmental information may include, but is not limited to, information on the date and time of image capture, location, weather information, road type, road surface, and traffic congestion. It doesn't.

The metadata generation unit according to an embodiment of the present invention compares the existing image with the image to be annotated, and applies the environment information given to the existing image as the environment information of the image to be annotated. can

More specifically, the metadata generation unit compares the existing image with the image to be annotated, and determines whether or not the environmental information given to the existing image can be applied as the environmental information of the image to be annotated. can Here, the existing image means an image taken prior to the image to be annotated by the same camera as the camera that captured the image to be annotated.

If there are a plurality of images taken prior to the image to be annotated, the metadata generation unit includes an object to which the same tracking ID as the object specified by the object specification unit is assigned among the plurality of images. can be selected as an existing image.

Alternatively, if there are a plurality of images taken prior to the image to be annotated, the metadata generation unit may select, as an existing image, an image whose shooting time is closest to the image to be annotated from among the plurality of images. there is.

Thereafter, the metadata generating unit may identify a background area occupied by an object to which the same tracking ID as an object of an existing image is removed from the image to be annotated. The metadata generating unit may identify an area occupied by an object to which a tracking ID identical to that of the object of the image to be annotated is removed from the selected existing image.

And, based on the result of comparing the background area of the image to be annotated with the background area of the existing image, the metadata generation unit converts environmental information given to the existing image to the image to be annotated. It can be judged whether or not it is applicable with environmental information.

That is, the metadata generation unit removes the area occupied by the object to which the same ID is assigned to each of the preceding image (i.e., the previous image) and the succeeding image (annotation target image). are compared with each other, and based on the comparison result, it is determined whether or not environmental information previously assigned to an image captured in advance can be applied as environment information of an image captured later.

As an example of a specific determination, the metadata generation unit assigns a predetermined value to the existing image based on the pattern of RGB values for the background area of the image to be annotated and the pattern of RGB values for the background area of the existing image. It is possible to determine whether the applied environmental information can be applied as the environmental information of the image to be annotated.

Unlike this, the metadata generation unit annotates the environment information given to the existing image based on the edge pattern of the background area of the image to be annotated and the edge pattern of the background area of the existing image. It can be determined whether or not it is applicable as environment information of an image that is a target of .

Alternatively, the metadata generation unit sets the average brightness value or average gamma value of the background area of the image to be annotated and the average brightness value or average gamma value of the background area of the existing image. Based on this, it may be determined whether or not environmental information previously assigned to an existing image can be applied as environmental information of an image to be annotated.

In addition, when it is determined that the metadata generator can apply the environmental information previously assigned to the existing image as the environmental information of the image to be annotated, the metadata creation unit converts the environmental information previously assigned to the existing image to the target of the annotation work. Metadata can be created by including it as environment information of the image.

In summary, when a plurality of images correspond to images continuously photographed by the same camera, the metadata generation unit encloses environment information of an image photographed by following the environment information given to the image photographed in advance among the plurality of images. It is possible to automatically generate metadata for images taken later by including it as .

On the other hand, if the metadata generation unit determines that it is impossible to apply the environmental information already given to the existing image as the environmental information of the image to be annotated, the characteristic of one or more reference points set in the image to be annotated is Based on this, it is possible to automatically determine the environmental information about the image to be annotated.

More specifically, the metadata generating unit may set one or more reference points in an image to be annotated. A number of images used in machine learning have a relatively similar composition so that artificial intelligence (AI) can achieve a specific purpose. A reference point according to an embodiment of the present invention may be one or more points in an image set to correspond to the learning purpose of artificial intelligence (AI). In addition, the metadata generation unit may determine environment information of an image to be annotated based on the color, brightness, or gamma of each pixel of one or more set reference points.

The metadata generation unit according to an embodiment of the present invention may extract an edge of an image to be annotated and divide the image into a plurality of regions based on the extracted edge. The metadata generating unit may set one or more reference points for each of the plurality of divided areas. The metadata generation unit may identify a background area by removing an area occupied by an object from an image that is an annotation target based on the extracted edge. The metadata generation unit may determine the directionality of the object within the image based on the class assigned to the object. For example, when the class assigned to the object is a vehicle, the metadata generation unit may determine the direction in which the wheels of the object are directed in the image as the ground. The metadata generator may identify a ground area and an air area from among a plurality of areas included in the identified background area, based on the determined object directionality. Here, the ground area is an area closest to the ground among a plurality of areas divided by edges. The aerial area is an area furthest from the ground among a plurality of areas divided by edges. In addition, the metadata generator determines road surface information constituting environmental information based on pixels of reference points set in the ground area, and weather conditions constituting environment information based on pixels of reference points set in the air area weather) information.

The metadata generation unit according to another embodiment of the present invention may set one or more reference points for each of the four line segments constituting the border of the image to be annotated. The metadata generation unit may determine the directionality of the object within the image based on the class assigned to the object. The metadata generation unit may identify a ground segment and an air segment among four line segments based on the determined object direction. Here, the ground segment is a line segment closest to the ground among four line segments constituting the edge of the image. The mid-air segment is the farthest line segment from the ground among the four line segments constituting the edge of the image. The metadata generator may determine road surface information constituting environment information based on pixels of a reference point set within a ground segment, and may determine weather information constituting environment information based on pixels of a reference point set within an aerial segment. .

On the other hand, the metadata generation unit sets a reference point at the same location for each of the image to be annotated and the existing image included in the image group, and the pixel of the reference point set in the image to be annotated is the same as that of the existing image. When the color, brightness, and gamma values of the pixels are within a predetermined allowable range in contrast to the pixels of the set reference point, the preset environment information of the existing image may be determined as the environment information of the image to be annotated. If the difference between the color, brightness, and gamma values of the pixels of the reference point set identically in the image to be annotated and the existing image is outside the allowable range, the metadata generator notifies that the environment information must be set manually. The message may be output through a user interface (UI).

With the following configuration, the result generation unit may generate an annotation work result based on the metadata generated by the metadata generation unit. In this case, the annotation work result may have a JSON file format, but is not limited thereto.

In addition, the result generation unit may directly transmit the generated annotation work result to the learning data generation device or to the learning data verification device through the communication unit.

Environment information on continuous images based on sensor information of an annotation device according to an embodiment of the present invention may be assigned.

Here, the environment information refers to information related to a place or time point at which an image was captured by a camera. For example, the environment information may include information on the date and time of image capture, location, weather information, road type, road surface, and traffic congestion. Not limited.

According to an embodiment of the present invention, based on the annotation device sensor information, an existing image and an image to be annotated are compared with each other, and environmental information given to the existing image is converted to the environment of the image to be annotated. information can be applied.

Specifically, it is possible to identify a background area from which an area occupied by an object (an object assigned the same tracking ID as an object of an existing image) is removed from an image to be annotated based on the annotation device sensor information. Based on the annotation device sensor information, it is possible to identify a background area from which an area occupied by an object (an object to which the same tracking ID as the object of the image to be annotated) is removed is removed from the existing image.

Based on the annotation device sensor information, based on the result of comparing the background area of the image to be annotated with the background area of the existing image, the environmental information previously given to the existing image is converted to the image to be annotated. can be applied to the environmental information of

A pattern of RGB values for the background area of the image to be annotated based on the sensor information of the annotation device. Based on the pattern of RGB values for the background area of the existing image, it is possible to determine whether the environment information given to the existing image can be applied as the environment information of the image to be annotated.

An edge pattern for the background area of the image that is the subject of annotation work based on the annotation device sensor information. Based on the pattern of the edge of the background area of the existing image, it may be determined whether or not the environment information given to the existing image can be applied as the environment information of the image to be annotated.

In the case where it is impossible to apply environmental information previously assigned to an existing image based on annotation device sensor information according to an embodiment of the present invention as environmental information of an image to be annotated, within the image to be annotated Based on the characteristics of one or more set reference points, environmental information about an image to be annotated may be determined.

A number of images used in machine learning have a relatively similar composition so that artificial intelligence (AI) can achieve a specific purpose. Therefore, the reference point according to an embodiment of the present invention may be one or more points set in correspondence with the learning purpose of artificial intelligence (AI).

An annotation device based on sensor information may extract an edge of an image to be annotated, and divide the image into a plurality of regions based on the extracted edge. One or more reference points may be set for each of a plurality of divided regions based on the annotation device sensor information.

Based on the annotation device sensor information, the orientation of the object within the image can be determined based on the class assigned to the object. For example, if the class assigned to the object is a vehicle, the direction in which the wheels of the object are directed in the image may be determined as the ground.

Based on the orientation of the object determined based on the annotation device sensor information, it is possible to identify a ground area and an air area among a plurality of areas. Here, the ground area is an area closest to the ground among a plurality of areas divided by edges. The aerial area is an area furthest from the ground among a plurality of areas divided by edges.

Then, based on the annotation device sensor information, road surface information constituting environmental information is determined based on pixels of reference points set in the ground area, and environment information is determined based on pixels of reference points set in the air area. It is possible to determine constituting weather information.

One or more reference points may be set for each of the four line segments constituting the edge of the image to be annotated based on the annotation device sensor information. Based on the annotation device sensor information, the orientation of the object within the image can be determined based on the class assigned to the object.

Based on the orientation of the object determined based on the annotation device sensor information, a ground segment and an air segment may be identified among four line segments. Here, the ground segment is a line segment closest to the ground among four line segments constituting the edge of the image. The mid-air segment is the farthest line segment from the ground among the four line segments constituting the edge of the image.

Based on the annotation device sensor information, road surface information constituting environmental information is determined based on pixels of a reference point set in a ground segment, and weather information constituting environmental information is determined based on pixels of a reference point set in an aerial segment. can

A learning data verification device according to an embodiment of the present invention based on sensor information may provide a user interface (UI) capable of easily inspecting metadata of annotation work results.

Specifically, the user interface (UI) by graphicalizing the images included in the image group so that it can be expressed as a figure corresponding to the value included in the metadata based on the learning data verification device sensor information. can be included in

For example, each of the images included in the image group based on the learning data verification device sensor information may be expressed as a line segment having a color corresponding to a value included in metadata. Based on the learning data verification device sensor information, bar-shaped graphics can be configured by continuously arranging each line segment in the width direction of the line segment according to the image before and after the images are taken. In addition, the configured bar-shaped graphic based on the learning data verification device sensor information may be included in the user interface (UI) and output.

Accordingly, according to an embodiment of the present invention, metadata of a plurality of images can be easily checked at a glance, and an image having a specific metadata value different from other images can be intuitively identified.

Hereinafter, a method for providing a metadata curation guide based on sensor information according to various embodiments of the present invention as described above and a computer program recorded on a recording medium to execute the method will be described in detail.

Prior to providing primitive data to the annotation server, the data collection server may provide information about a time point at which the photographing environment of raw data including primitive data is changed. Through this, the labeler can perform curation with high concentration only at the time when the shooting environment of raw data has changed.

First of all, the data collection server generates primitive data by refining raw data (ie, image) (de-identification, removal of similar data, etc.) from an edge device (eg, vehicle) that has collected raw data. Photographing time information may be received.

The shooting time information includes the information collected from the illuminance sensor that controls the vehicle light, the camera's shooting setting value, and the vehicle's rain sensor.

The data collection server may estimate a time point at which the received photographing time information changes a time point at which a photographing environment changes, and may reflect this to primitive data.

For example, a time point at which lights of a vehicle are turned on/off by an illuminance sensor may be a time point at which night/day changes or a time point at which a vehicle enters a tunnel or the like.

As an extreme example, the point at which the camera's focus dramatically changes in a short period of time can be the point at which weather conditions change.

The data collection server may set a singularity to be noted when labeler performs curation based on the capture time of the image in which the shooting environment is changed, and then provide the information to the annotation server.

As described above, although preferred embodiments of the present invention have been disclosed in the present specification and drawings, it is in the technical field to which the present invention belongs that other modified examples based on the technical spirit of the present invention can be implemented in addition to the embodiments disclosed herein. It is self-evident to those skilled in the art. In addition, although specific terms have been used in the present specification and drawings, they are only used in a general sense to easily explain the technical content of the present invention and help understanding of the present invention, but are not intended to limit the scope of the present invention. Accordingly, the foregoing detailed description should not be construed as limiting in all respects and should be considered illustrative. The scope of the present invention should be selected by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

Claims (5)

Collecting, by an annotation device, a plurality of images of an object to be annotated for artificial intelligence (AI) learning;
obtaining, by the annotation device, information on a capturing point of time of the collected image;
Comparing, by the annotation device, environment information of the plurality of images based on photographing time information of the images; and
A method for providing a metadata curation guide based on sensor information, including determining, by the annotation device, a change in environmental information of the plurality of images.
According to claim 1,
The plurality of images,
A method for providing a metadata curation guide based on sensor information, characterized in that the images are continuously collected in time series.
According to claim 1,
The photographing time information,
A method for providing a metadata curation guide based on sensor information, characterized in that it includes information collected through at least one of an illuminance sensor that controls vehicle light, a shooting set value of a camera, and a rain sensor of a vehicle.
According to claim 3,
The step of determining the change of the environmental information,
Providing a metadata curation guide based on sensor information, characterized in that it is determined including any one of the time when the light of the vehicle is turned on or off by the illuminance sensor, the time of night / day change, and the time of entering the tunnel method.
memory;
transceiver; and
In combination with a computing device configured to include a processor for processing instructions resident in the memory,
Collecting, by the processor, a plurality of images of an object to be annotated for artificial intelligence (AI) learning by the annotation device;
acquiring, by the processor, photographing time point information of the collected image;
comparing, by the processor, environment information of the plurality of images based on photographing time information of the images; and
A computer program recorded on a recording medium to execute, by the processor, determining a change in environmental information of the plurality of images.