KR102259207B1 - Automatic taging system and method thereof - Google Patents

Abstract

Disclosed is an automatic tagging system and method thereof. Here, the automatic tagging system includes an input unit for receiving a still image, a physical information extraction unit for inferring physical information including visual characteristics of an object by analyzing the still image, and an attribute of the still image by analyzing the still image. And a semantic information extracting unit for inferring corresponding semantic information, and an automatic tagging unit for tagging the still image by integrating metadata of the still image, the physical information, and the semantic information.

Description

Automatic tagging system and its method {AUTOMATIC TAGING SYSTEM AND METHOD THEREOF}

The present invention relates to an automatic tagging system and method thereof.

As services such as cloud computing and social networks become popular, still images and video contents produced by individuals are exponentially increasing. In order to efficiently manage such video contents, there is a need for a system that automatically analyzes video contexts and attaches tags to them. The aforementioned context refers to context and context information of video content.

Recently, as the field of image signal processing and machine learning has developed, many algorithms have been developed that can automatically segment the background and foreground of a given image when an image is input and recognize objects included in the image.

Accordingly, a conventional image processing system extracts physical information and semantic information of backgrounds and objects belonging to image content. It can be used in applications such as automatic tagging and search of images.

In the conventional automatic tagging of video content, the meta data recorded at the time the content was created, that is, using only the time, location, exposure, and photographing equipment, or a method of utilizing collective intelligence, has been mainly made.

In addition, conventionally, a method in which a user directly inputs a tag into video content is mainly used. It can be viewed as a method of utilizing collective intelligence by inducing information sharing between users mainly.

As such, conventionally, automatic tagging of video content is limited to physical information in the content, even when manual tagging of a human, that is, inducing collective intelligence or individual, or automatically tagging.

Therefore, the technical problem to be achieved by the present invention is to provide a system and method for automatically tagging by inferring physical information, semantic information, and meta data by receiving a still image to be stored by a user and automatically understanding the image context. will be.

According to one feature of the present invention, the automatic tagging system includes an input unit for receiving a still image, a physical information extracting unit for inferring physical information including visual characteristics of an object by analyzing the still image, and analyzing the still image. Thus, a semantic information extracting unit that infers semantic information corresponding to an attribute of an image describing an abstract concept or situation, and the still image by integrating the metadata of the still image, the physical information, and the semantic information Includes an automatic tagging unit for tagging on.

The physical information extraction unit,

A background separation module for separating a background foreground from the still image, and an object recognition module for recognizing an object by extracting a feature from a divided image in which the background foreground is separated.

The background separation module,

A super pixel is generated by combining pixels having similar characteristics among the pixels of the still image, a feature vector is extracted from the super pixel, and the background foreground to which the super pixel is combined is separated using the feature vector. You can create an image.

The background separation module,

Feature vectors including color, texture, shape, location, and visual words can be extracted.

The object recognition module,

In the divided image, a plurality of objects may be recognized through object classification.

The semantic information extraction unit,

An image property may be extracted by applying a predefined generation model to the still image and the background foreground image.

According to another feature of the present invention, the automatic tagging method includes receiving, by an automatic tagging system, a still image, interpreting the still image to infer physical information including visual characteristics of the object, and interpreting the still image. Inferring semantic information corresponding to an attribute of an image describing an abstract concept or situation, and tagging the still image by integrating the metadata of the still image, the physical information, and the semantic information Including,

The step of inferring the physical information and the step of inferring the semantic information may be performed simultaneously in parallel.

Inferring the physical information,

Separating a background foreground from the still image, and recognizing an object by extracting a feature from the divided image from which the background foreground is separated.

The separating step,

Generating a super pixel by combining pixels having similar characteristics among the pixels of the still image, extracting a feature vector from the super pixel, and the background foreground to which the super pixel is combined using the feature vector It may include generating the divided image.

Recognizing the object,

And extracting features for object recognition including features that are invariant to color, pixel brightness, slope, size, and rotation from the divided image, and recognizing the object by passing the extracted features through a mechanical learning algorithm. I can.

Inferring the semantic information,

An image attribute may be extracted by applying a predefined generated model to the still image and a background foreground image separated from the still image.

Inferring the semantic information,

The still image and the attribute describing the abstract concept or situation of the still image may be trained to infer the image attribute.

According to an embodiment of the present invention, since physical information and semantic information can be inferred from still images to understand and auto-tag images, the inconvenience of directly tagging the user is eliminated, and accurate and efficient image retrieval is possible. It is possible.

In addition, since the system automatically tags and stores still images input to the server from services such as cloud computing or social networks, it is possible to efficiently search for images desired by the user in the future.

In addition, it can be used as an application that adds a simple description of a corresponding image based on a tag attached to a user's still image and a natural language processing algorithm.

1 is a block diagram showing the configuration of an automatic tagging system according to an embodiment of the present invention.
2 is a conceptual diagram of automatic tagging according to an embodiment of the present invention.
3 is a flowchart illustrating an automatic tagging method according to an embodiment of the present invention.
FIG. 4 is a detailed flowchart illustrating step S103 of FIG. 3.
5 is a detailed flowchart illustrating step S105 of FIG. 3.
6 is a detailed flowchart illustrating step S107 of FIG. 3.
7 is an exemplary diagram of a generation model for semantic information extraction according to an embodiment of the present invention.
8 is a schematic diagram of an automatic tagging system according to another embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the present invention. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part "includes" a certain component, it means that other components may be further included rather than excluding other components unless specifically stated to the contrary.

In addition, the terms "... unit" and "... module" described in the specification mean a unit that processes at least one function or operation, which may be implemented by hardware or software, or a combination of hardware and software.

Hereinafter, an automatic tagging system and method according to an embodiment of the present invention will be described in detail with reference to the drawings.

1 is a block diagram showing the configuration of an automatic tagging system according to an embodiment of the present invention, and FIG. 2 is a conceptual diagram of automatic tagging according to an embodiment of the present invention.

First, referring to FIG. 1, when the automatic tagging system 1 stores or uploads a still image owned by an individual to an arbitrary storage device, the automatic tagging system 1 automatically includes physical information and semantic information in the image. After inferring, tag information is generated along with meta data and managed efficiently. That is, the automatic tagging system 1 may be mounted on an image storage device (not shown) or may be connected to an image storage device (not shown).

In addition, the automatic tagging system 1 may be mounted on a cloud computing server (not shown) or a social network server (not shown).

The automatic tagging system 1 uses image signal processing and machine learning to recognize multiple objects in a still image, infer class information, and infer semantic information of the background and foreground and automatically tag them.

The automatic tagging system 1 uses independent modules for extracting physical and semantic information of still images, and learns each module using an algorithm based on image signal processing and machine learning. In addition, since meta data is included in the image, a separate learning algorithm is not used. For the method of extracting physical information, the background foreground separation and object recognition are used, and the semantic information is extracted as a method for deep learning. You can use a generation model based on it.

The automatic tagging system 1 includes a still image input unit 100, a physical information extraction unit 200, a semantic information extraction unit 300, and an automatic tagging unit 400.

Here, the still image input unit 100 receives a still image from a user terminal (not shown).

In this way, when a still image is input, the physical information extracting unit 200 and the semantic information extracting unit 300 infer physical information and semantic information in parallel, respectively.

When a still image is input, the physical information extracting unit 200 extracts physical information using a background foreground separation and an object recognition algorithm. That is, a feature is extracted from a still image to separate the background and the foreground, and the feature is extracted from the divided area to recognize an object. The physical information extraction unit 200 includes a background separation module 210 and an object recognition module 230.

The semantic information extraction unit 300 extracts semantic information from a still image. That is, semantic information is inferred by modeling the relationship between the still image and the attribute.

Here, the semantic information corresponds to an attribute corresponding to a still image. Examples of attributes are abstract information such as'with mane','young boy', etc. It is possible to describe the video context only by listing the attributes. It is information that is clearly distinguished from low-dimensional information corresponding to the visual characteristics of an object.

There are also ways to understand the video and find its properties. For example, as abstract information such as'round sphere shape','cloudy','clear','set sunset','mammal','hairy', it is highly dependent on the visual information of the object. It is information that is clearly distinguished from low-dimensional information.

In fact, semantic information has a greater influence than physical (low-dimensional) information when conversations occur between people or when explaining a specific situation to the other person.

As shown in (a) of FIG. 2, although physical information is present in a given still image, more semantic information is actually contained.

2B, the physical information extraction unit 200 extracts features from the still image and separates the background foreground, and then divides the still image into a background foreground image and a separated image. The physical information extracting unit 200 recognizes an object that is physical information by extracting features from an image in which the background foreground is separated. Here, the object may be an adult male or a child wearing glasses in FIG. 2A.

In addition, the semantic information extraction unit 300 extracts features from the still image input in FIG. 2A, and generates attributes corresponding to semantic information by understanding objects and situations through a generation model. In other words, it recognizes events and generates semantic information such as picnics, family, and daughters.

The automatic tagging unit 400 adds the reliability of the inferred physical information and semantic information and the given metadata along with the still image, and then finally performs automatic tagging. That is, the automatic tagging unit 400 generates a physical information tag such as “people, grass, and trees” as shown in (c) of FIG. 2. And it creates semantic information tags such as'family, picnic, adult men, girls, wearing glasses, sunny days, and running'.

The operation of the automatic tagging system 1 based on the configuration described with reference to FIGS. 1 and 2 will be described as follows. In this case, the same reference numerals are used for descriptions of the same components as in FIGS. 1 and 2.

3 is a flowchart illustrating an automatic tagging method according to an embodiment of the present invention, FIG. 4 is a detailed flowchart illustrating step S103 of FIG. 3, and FIG. 5 is a detailed flowchart illustrating step S105 of FIG. 3, and FIG. Is a detailed flowchart showing step S107 of FIG. 3, and FIG. 7 is an exemplary diagram of a generation model for semantic information extraction according to an embodiment of the present invention.

3, the still image input unit 100 receives a still image (S101). The physical information extracting unit 200 separates the background foreground from the input still image (S103).

Here, the background separation module 210 of the physical information extraction unit 200 separates the background and the foreground from the still image to recognize not only the object but also the region corresponding to the background in the still image and recognize it as a single object. Perform the pretreatment process.

At this time, there are various methods for separating the background and the foreground. According to an embodiment, a method based on'correlation clustering' may be used to allow general application to an object not included in the training data. The general sequence of background/foreground separation is to first convert the pixels of a still image into a larger unit called a super pixel, and then combine the super pixels to create a larger segmentation area. The partitioned regions are combined as much as possible to satisfy the condition that the boundary lines of objects other than the object to which they belong, and finally created partitions are classified as objects.

Here, referring to FIG. 4, when a still image is input, the background separation module 210 combines pixels having similar properties to generate super pixels (S201). The super pixel is a preprocessing process for efficiently separating the background foreground, and according to an embodiment, an ultrametric contour map (UCM) is used. In order to separate the background and foreground, a feature vector consisting of color, texture, shape, position, and visual word is extracted from the super pixels (S203). Then, the extracted feature vectors are input to a correlation clustering algorithm, and background foreground separation is performed (S205). Correlation clustering is an algorithm that is trained based on an energy minimization technique, looking at a feature vector of a single super pixel and a feature vector between two adjacent super pixels and combining them in the direction in which the energy is minimized.

In order to obtain good performance, it is important to design a higher order term, which is a combination of 3rd or higher order super pixels. Here, the higher order term is assumed to have a hierarchical structure such as a mass made of a combination of super pixels in a still image, a larger mass made of a combination of masses, and an object when large masses are combined. It is a constraint that allows them to be properly combined.

The result obtained from separating the background and foreground corresponds to a divided area as large chunks that make up an object (S207). At this time, a restricted Boltzmann machine (RBM) is used to improve the existing higher order term. RBM is an undirected graph that expresses probability as energy. It is a generation model that minimizes energy for training data by learning the structure of training data by the non-linear learning method. The probability distribution of RBM is given as follows.

Here, p(v) is a probability distribution for v, v is a visible node, and h is a hidden node.

The background separation module 210 receives features of the polynomial super pixels as input, and if the input super pixels fall into the same area of the object, a small amount of energy is given to combine them, and if they belong to different objects, a large amount of energy is given to combine them. Do not. According to Equation 1, the smaller the energy, the larger the value of p(v) (the probability of v), which means that the probability of the same object is higher.

Since RBM is learned by the comparative history learning technique, it is possible to effectively design higher order terms without extra great effort.

Again, referring to FIG. 3, the object recognition module 230 recognizes an object from a divided area (or divided image) output from the background separation module 210 (S105 ).

Here, the object recognition module 230 uses an object classifier that can be used in general, and in one embodiment, a support vector machine (SVM) may be used. At this time, the SVM (support vector machine) is one of the machine learning algorithms that selects and separates the support vectors and the largest-margin hyperplane among the hyperplanes separating data.

In this case, referring to FIG. 5, the object recognition module 230 classifies which object the divided regions generated by the background foreground separation algorithm correspond to using an object classifier.

When a segmented image is input, the object recognition module 230 extracts features for object recognition (S301), and may use color, pixel brightness, slope, and Scale Invariant Feature Transform (SIFT). SIFT is to extract features that are invariant in size and rotation, and apply them to detection or recognition. The SVM for object classification may use a radius basis function (RBF) kernel.

That is, the object recognition module 230 may recognize multiple objects from the still image by classifying the divided regions (S303) after separating the background foreground (S305). Conventionally, in general object recognition, one object could be recognized from one still image.

Again, referring to FIG. 3, the semantic information extracting unit 300 generates an image attribute (attribute) (S107).

In this case, referring to FIG. 6, the semantic information extraction unit 300 receives a still image and uses a generation model based on deep learning technology to extract attributes (S401, S403) describing an abstract concept or situation ( S405) to generate an image property, that is, an attribute. This generated property is used for automatic tagging. In this case, semantic features may be extracted from a still image and semantic features may be extracted from a background foreground image.

Here, the generated model is trained by receiving data of two modes, a still image and an attribute, and modeling the input data (image or attribute) of each mode using several latent layers.

There are two main things that can be done through a generative model, that is, one of the proposed graph models in the machine learning field. When a still image comes in, it creates the attributes of the image, namely, the sunset, the clear weather, the mountains, and the sea. And when an attribute is given as an input, a still image containing the attributes or having a high similarity context is created or retrieved.

At this point, you have to go through the process of "training" the generated model. The pursuit of machine learning is to obtain a function that gives a desired output when given an input, that is, by optimizing the objective function mainly based on statistics and probability. This requires training data (experience) that covers both inputs and outputs. As humans learn through experience, the generative model (machine) optimizes its own parameters through training data, and eventually learns a function that gives an appropriate output to the input when it comes in.

That is, the generated model receives image properties, and receives the still image and the properties of the still image in the training process. And when the model is generated using many training still image-property pairs, after training, when a completely new still image comes in, properties can be created using parameters optimized in the machine training process. There will be. At this time, the attributes are divided into a training attribute that trains the generated model in advance and a test image when a completely new type of still image is entered for actual use.

At the top of the model, a latent layer is added to connect the two modes. When a still image comes in, the generated model creates a property that is highly related to a still image, and conversely, when a number of attributes are received as input, it retrieves a still image that is highly related to the input properties.

In general, the latent layer connected to the top serves as a bridgehead for modeling the relationship between the two types of inputs, but there is a limit when the input data are inconsistent and have various distributions. For example, there is consistency when image data consisting only of natural landscapes comes in, but in reality, there may be image data including natural landscapes, urban backgrounds, and indoor images.

According to an embodiment, in order to better model the relationship between various types of images (natural scenery, urban background, indoors, people, objects) and input queries, one latent layer at the top of the model is expanded to a large number and each Distribution of different layers, for example, natural scenery, urban background, and indoor images are modeled.

In this case, referring to FIG. 7, there are two generation models for still image data and attribute data. Then, the latent layers at the top of the two generation models are connected by a gating function. Latent layers of each generative model are sequentially trained for each layer using RBM, and h_natural^3, h_urban^3, h_indoor^3, h_i^2, h_t^2 at the top of the entire model are learned as a mixture of RBMs. The entire training process of this generative model follows the non-historical learning technique.

Again, referring to FIG. 3, the automatic tagging unit 400 automatically generates a tag by integrating the physical information inferred in step S105 and the semantic information inferred in step S107 together with the metadata of the still image (S109). ). That is, the inferred physical information and semantic information are collected, and a total of three types of information, up to the metadata included in the still image, are automatically tagged on the image and then stored.

Meanwhile, FIG. 8 is a schematic diagram of an automatic tagging system according to another embodiment of the present invention. A still image input unit 100, a physical information extraction unit 200, and semantic information of the automatic tagging system described with reference to FIG. 1 Represents a device that can be used to perform at least some of the functions of the extraction unit 300 and the automatic tagging unit 400.

Referring to FIG. 8, the automatic tagging system 500 includes a processor 501, a memory 503, at least one storage device 505, an input/output (I/O) interface 507, and a network interface ( 509).

The processor 501 may be implemented as a central processing unit (CPU) or other chipset, a microprocessor, and the like, and the memory 503 includes dynamic random access memory (DRAM), Rambus DRAM (RDRAM), and synchronous DRAM ( SDRAM), static RAM (SRAM), etc. may be implemented in a medium such as RAM.

The storage device 505 includes a hard disk, a compact disk read only memory (CD-ROM), a CD rewritable (CD-RW), a digital video disk ROM (DVD-ROM), a DVD-RAM, and a DVD-RW disk. , It may be implemented as an optical disk such as a blue-ray disk, a flash memory, or a permanent or volatile storage device such as various types of RAM.

In addition, the I/O interface 507 allows the processor 501 and/or the memory 503 to access the storage device 505, and the network interface 509 is the processor 501 and/or the memory 503 ) To access the network (not shown).

In this case, the processor 501 is a program command for implementing at least some of the functions of the still image input unit 100, the physical information extraction unit 200, the semantic information extraction unit 300, and the automatic tagging unit 400. Is loaded into the memory 503 to control the operation described with reference to FIG. 1 to be performed.

In addition, the memory 503 or the storage device 505 interlocks with the processor 501 to provide a still image input unit 100, a physical information extraction unit 200, a semantic information extraction unit 300, and an automatic tagging unit 400. Can be performed.

The processor 501, the memory 503, the storage device 505, the I/O interface 507, and the network interface 509 shown in FIG. 8 may be implemented in one computer or distributed among a plurality of computers. It can also be implemented.

The embodiments of the present invention described above are not implemented only through an apparatus and a method, but may be implemented through a program that realizes a function corresponding to the configuration of the embodiment of the present invention or a recording medium in which the program is recorded.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concept of the present invention defined in the following claims are also provided. It belongs to the scope of rights.

Claims (12)

An input unit that receives a still image uploaded by a user terminal,
A physical information extraction unit for inferring physical information including visual characteristics of an object by analyzing the still image,
A semantic information extraction unit that interprets the still image to infer semantic information corresponding to an attribute of an image describing an abstract concept or situation, and
An automatic tagging unit for tagging the still image by integrating metadata of the still image, the physical information, and the semantic information,
The semantic information extraction unit,
By learning various still image data and attribute data representing an abstract concept of the still image data or describing a situation using a deep learning technology, a generation model modeled the relationship between the still image data and the attribute data is derived,
Extracting attribute data related to the input still image as the semantic information by using the still image input from the input unit as an input of the generated model,
The physical information extraction unit,
A background separation module for classifying a background foreground from the still image into a background and a foreground in pixel units, and
An object recognition module for recognizing an object by extracting a feature from the divided image from which the background foreground is separated,
The background separation module,
A super pixel is generated by combining pixels having similar characteristics among the pixels of the still image, a feature vector is extracted from the super pixel, and the background foreground to which the super pixel is combined is separated using the feature vector. Automatic tagging system that generates video. delete delete The method of claim 1,
The background separation module,
An automatic tagging system that extracts feature vectors including color, texture, shape, location, and visual words. The method of claim 4,
The object recognition module,
An automatic tagging system for recognizing a plurality of objects through object classification in the divided image. The method of claim 1,
The semantic information extraction unit,
An automatic tagging system for extracting attribute data related to each of the still image and the background foreground image by using the still image and the background foreground image as inputs of the generated model. A step in which the automatic tagging system receives a still image,
Interpreting the still image to infer physical information including visual characteristics of the object,
Interpreting the still image to infer semantic information corresponding to an attribute of an image describing an abstract concept or situation, and
Integrating metadata of the still image, the physical information, and the semantic information, and tagging the still image,
The step of inferring the physical information and the step of inferring the semantic information are performed simultaneously in parallel,
Inferring the semantic information,
By learning various still image data and attribute data representing an abstract concept of the still image data or describing a situation using a deep learning technology, a generation model modeling the relationship between the still image data and the attribute data is derived,
Extracting attribute data related to the received still image as the semantic information by using the received still image as an input of the generated model,
Inferring the physical information,
Separating a background foreground from the still image, and
Recognizing an object by extracting a feature from the divided image from which the background foreground is separated,
The separating step,
Generating a super pixel by combining pixels having similar characteristics among the pixels of the still image,
Extracting a feature vector from the super pixel, and
Generating a segmented image in which the background foreground to which the super pixels are combined is separated using the feature vector
Automatic tagging method comprising a. delete delete The method of claim 7,
Recognizing the object,
Extracting features for object recognition including features that are invariant in color, pixel brightness, gradient, size, and rotation from the divided image, and
Recognizing an object by passing the extracted features through a mechanical learning algorithm
Automatic tagging method comprising a. The method of claim 10,
Inferring the semantic information,
An automatic tagging method for extracting image attribute data related to each of the still image and the background foreground image by using the still image and a background foreground image separated from the still image as an input of the generated model.
delete

Images