KR102258420B1 - Animaiton contents resource service system and method based on intelligent information technology - Google Patents

Abstract

The present invention relates to an intelligent information technology-based animation content resource service system and method, and more particularly, to recognize an object from an animation resource based on an object recognition and tracking algorithm, and provide it by classification, storage, and search, and the user's search It is an intelligent information technology-based animation content resource service system and method that recommends animation resources with high relevance based on information.

Description

Animation contents resource service system and method based on intelligent information technology

The present invention relates to an intelligent information technology-based animation content resource service system and method, and more particularly, to recognize an object from an animation resource based on an object recognition and tracking algorithm, and provide it by classification, storage, and search, and the user's search The present invention relates to an intelligent information technology-based animation content resource service system and method for recommending animation resources with high relevance based on information.

Animation, known as so-called cartoon film, refers to a filming company that makes motionless objects appear to be moving or films made in such a way.In Korea, the demand and related technologies have rapidly developed since the first animation was screened in the mid-1950s. I'm doing it.

In addition, as the global video market is growing and the number of visitors and page views for video-related services continues to increase, related business and technology development is continuously increasing. Therefore, video consumption continues to increase due to the increase of smart devices. It is expected to do.

As an example of animation that is well known to the general public, cartoon movies such as Pororo, Pikachu, and Janggu generally consume 6 months in preparation for production, and production costs are between 60 million won and 100 million won per episode, and sales at distribution companies range from 5 million won per episode. The cost is set at 10 million won.

However, although animation images are produced with such a large cost and time, many animation sources, that is, animation resources, and image objects produced to create the animation, are generally destroyed and discarded after production of the animation image.

Therefore, in the case of future production of a sequel or another person seeking a contract or permission with the copyright holder to do business using the character (image object) of the animation, although it was an animation source that has already been worked, a new work and creation are required. There were also many cases, and there was a significant problem in wasting time, manpower, and costs accordingly.

However, there is no technology for a service that can recycle and share animation sources for mutual symbiosis and profit sharing in the animation distribution ecosystem to solve this problem, and therefore, image objects such as image objects that can be used in the future from animation sources are not available. There is a need to develop a technology for a system that can extract, classify, store, and search data, and provide it to users who need it.

Meanwhile, image recognition technology generally refers to a technology that recognizes what an object appears in an image, and has developed a machine learning technology in the field of vision that recognizes images based on a large amount of image data.

In particular, deep learning technology, which has developed the existing neural network technology, has shown a rapid performance improvement, and is being actively researched as it has been applied to almost all fields requiring artificial intelligence such as vision and natural language processing.

After Professor Jeffrey Hinton of the University of Toronto introduced deep learning technology in 2012, the performance improved significantly, and after 2015, the performance of deep learning in some fields, such as image classification, has been evaluated as surpassing humans. The related technology has been developed to that extent, and the object recognition method has been developed from R-CNN to SSD based on CNN.

However, this conventional intelligent information-based image recognition technology has a problem that takes a long time because it recognizes each object for the entire image without a special object, and an introduction to image recognition technology based on intelligent information specialized for animation. Nothing happened.

Accordingly, there is a need to develop a technology for an intelligent information technology-based animation content recognition technology optimized for animation for extracting and classifying data such as image objects that can be used in the future from the animation source described above.

The present invention is an object of the present invention to provide an intelligent information technology-based animation content resource service system and method.

The object of the present invention described above is a source input unit for inputting an animation source including at least one of an animation resource image, a scene image, and an image, and a data classification unit for classifying image objects included in an animation source input to the source input unit. And, a metadata processing unit that extracts metadata for each of the recognized objects, and extracts keywords and categories for each of the recognized objects, and generates label information for identifying the extracted keywords and categories. An intelligent information technology-based animation content resource service including a labeling processing unit and a database unit for storing the animation source, the recognized and classified image object, the metadata, and label information including the keyword and category respectively. This can be achieved by providing a system.

Here, the data classification unit is a resource processing unit that classifies according to the resource image when the animation source is a resource image, and when the animation source is an image, a representative frame selected after selecting a representative frame from all frames included in the image A clustering processing unit for extracting a scene image, and an image object tracking unit for classifying a scene image input to the source input unit or a scene image extracted from the clustering processing unit into image objects included in the scene image, respectively.

In addition, the image object tracking unit may perform AI image learning to distinguish and classify the image objects.

Here, the AI image learning may be a learning operation according to Faster Regions with Convolution Neural Networks (R-CNN) or You Only Look Once (YOLO) techniques.

In addition, the clustering processing unit extracts only each object region from the entire image frame, then classifies frames having a similarity of the object regions greater than or equal to a preset reference similarity into a similar frame group, and a plurality of frames included in the similar frame group Among them, the frame having the most object area may be selected as the representative frame.

In addition, the database unit may classify and store the label information by category when storing the label information so that keywords belonging to the same category can be simultaneously extracted or searched during a future search.

In addition, the animation source, the image object, and the animation source stored in the database unit to receive search information including at least one of a keyword, an image, and an image input by a user and provide an animation source corresponding to the search information. It may further include a search unit capable of searching the metadata and the label information.

Also, based on search information input to the search unit, animation source, image object, metadata, and label information may be searched in the database unit and transmitted to the search unit.

In addition, the search recommendation processing unit extracts or recommends the animation source and image object in the order of high relevance to the search information based on the search information input to the search unit, or belongs to the same category as the input search information. It may further include a search recommendation processing unit for extracting or recommending an animation source having label information and an image object.

In addition, the search recommendation processing unit may collect and store user information accessing the search unit, search information searched by the user through the search unit, and keywords according to the search information, and the user's specific search information and keywords When the number of inputs is greater than or equal to a preset reference value, the user's search taste information is calculated based on the search information and the keyword, and when an animation source corresponding to the taste information is input to the database, the animation source is recommended to the user. can do.

The object of the present invention described above is the step of inputting an animation source including at least one of an animation resource, an image, and an image, the step of recognizing and classifying each image object included in the input source, and each of the recognized Extracting metadata for an object of, extracting keywords and categories for each of the recognized objects, and generating label information to uniquely identify the extracted keywords and categories; and Also achieved by providing an intelligent information technology-based animation content resource service method comprising storing the animation source, the image object recognized and classified, the metadata, the keyword, the category, and the label information, respectively. Can be.

Here, the step of inputting a keyword, image, or image to be searched, and searching and providing the stored animation source and image object in the order of relevance to the keyword, image, or image based on the keyword, image, or image. It may contain more.

According to the intelligent information technology-based animation content resource service system and method according to the present invention, it is possible to recycle and share animation sources, thereby preventing wasting of time, manpower, and costs, and achieving mutual symbiosis and profit sharing of the animation distribution ecosystem. Can be.

In addition, it is possible to easily and efficiently classify each object by recognizing and classifying the object through image learning in the animation source.

In addition, it is easy to manage data by classifying keywords and categories for each object, storing and managing by category, and providing easy search and recommendation functions by providing keywords and image objects belonging to the same category when users search. have.

In addition, since the clustering processor extracts only each object region from the entire image frame, determines only the similarity between object regions, and classifies it into a similar frame group, fast image recognition optimized for animation content is possible.

In addition, it is possible to provide an effective recommendation function by collecting and evaluating user's search information, etc., and recommending animation sources necessary for the corresponding user.

1 is a block diagram illustrating an intelligent information technology-based animation content resource service system according to the present invention.
FIG. 2 shows an example of a resource image of an original animation of the intelligent information technology-based animation content resource service system shown in FIG. 1.
FIG. 3 shows an example of a scene image of an original animation of the intelligent information technology-based animation content resource service system shown in FIG. 1.
FIG. 4 shows an example of an image frame extracted from an image of the intelligent information technology-based animation content resource service system shown in FIG. 1
FIG. 5 shows an image object extracted from a scene image or video frame of the intelligent information technology-based animation content resource service system shown in FIG. 1.
6 shows an example of an image object provided through a search unit of the intelligent information technology-based animation content resource service system shown in FIG. 1.
FIG. 7 is a flowchart of a method for servicing an animation content resource based on intelligent information technology shown in FIG. 1.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the embodiments described below are only described in detail enough to allow those of ordinary skill in the art to carry out the invention easily, and this does not mean that the protection scope of the present invention is limited. Does not. In addition, in describing various embodiments of the present invention, the same reference numerals will be used for components having the same technical characteristics.

1 is a block diagram of an intelligent information technology-based animation content resource service system according to the present invention, and with reference to this, an intelligent information technology-based animation content resource service system according to the present invention will be described.

The intelligent information technology-based animation content resource service system 1 of the present invention receives animation sources from producers, copyright holders, etc., extracts resource images or image objects from each source, classifies and stores, and stores each Provides a resource or image object and the original image, metadata, keywords and categories related thereto, and for this purpose, the source input unit 100, the data classification unit 200, the metadata processing unit 300, the labeling processing unit 400, It includes a database unit 500, and may further include a search unit 600 and a search recommendation processing unit 700.

2 shows an example of a resource image of an original animation, and FIG. 3 shows an example of a scene image of an original animation.

2 and 3, an animation source may be input to the source input unit 100, and the animation source may be a resource image of an original animation, as shown in FIG. 2, and shown in FIG. As in the example described above, it may be a scene image such as a thumbnail or a captured main scene, and may be a full image or a partially edited partial image.

4 illustrates an example of an image frame extracted from an image, and FIG. 5 illustrates an image object extracted from a scene image or an image frame, and the data classification unit 200 will be described with reference to this.

The data classification unit 200 serves to extract and recognize or classify necessary data from an animation source, and may include a resource processing unit 210, a clustering processing unit 220, and an image object tracking unit 230 for this purpose. have.

When the animation source input to the source input unit 100 is a resource image as illustrated in FIG. 2, the resource processing unit 210 may classify the resource images, respectively.

When the animation source input to the source input unit 100 is an image, the clustering processing unit 220 may analyze the image and extract a scene image.

In detail, when the animation source is an image, the clustering processor 220 may recognize all frames of the image constituting the image, respectively. Thereafter, similar frames may be classified in all frames of the image. Here, each object is first extracted and identified for all frames of the image, and then the extracted objects are compared to perform clustering.

In other words, in the case of analyzing an entire frame of an image, in the prior art, object recognition such as a face or an object is performed for each frame, and the results are stored respectively, whereas in the present invention, the result is stored in the database unit 500 later, Since only the animation image object to be provided to the user who needs it in the future needs to be recognized, only the objects are extracted in advance, and the similarity is determined by comparing only the objects with each other, and then frames with the similarity greater than or equal to a specific similarity are classified into a similar frame group. In comparison, there is an effect that the clustering speed can be increased by several tens of times or more.

Here, the method of extracting and recognizing the image object may be a learning operation according to the Faster R-CNN (Faster Regions with Convolution Neural Networks) of TensorFlow, which is an AI image object recognition learning method, or You Only Look Once (YOLO) method. .

When the clustering method of the clustering processor 220 is described, the clustering processor 220 may extract a plurality of frames from the entire image frame, as shown in FIG. 4. Thereafter, an object region may be first extracted from the plurality of frames. After extracting the object regions, the object regions are compared to each other, and if the similarity between each object region is greater than or equal to a preset reference similarity, for example, if the match rate of color and outline is more than 70%, frames containing each object region are similarly framed. It can be set as a group.

Then, in order to allow the image object tracking unit 230 to extract the image object from each similar frame group later, the frame with the largest number of object regions among the similar frame groups is compared to represent the similar frame group. Frame can be selected.

These representative frames are scene images as in the example shown in FIG. 3, and these representative frames, that is, scene images, are selected from each of a plurality of similar frame groups, and the selected plurality of scene images are transferred to the image object tracking unit 230. Can be.

The image object tracking unit 230 may extract each image object included in the scene image, recognize and classify each image object. In this case, the scene image may be a scene image (for example, a thumbnail) input to the source input unit 100. Meanwhile, when the image itself is input to the above-described source input unit 100, a representative selected by the clustering processing unit 220 It may be a scene image that is a frame.

Here, the image object tracking unit 230 may perform AI image learning in order to learn, distinguish, extract, recognize, and classify each object from the scene image. This AI image learning may be a learning operation according to TensorFlow's Faster Regions with Convolution Neural Networks (R-CNN) or You Only Look Once (YOLO) technique, and is also a result of image learning by the image object tracking unit 230. An image object such as the example shown in FIG. 5 may be extracted.

The metadata processing unit 300 may extract the animation source input through the source input unit 100 and metadata for each image object extracted through the above process.

Here, the metadata is information on original data input when an animation source is input through the source input unit 100, and refers to information such as a maker, production date, production company, data size, and data file name, and each image object is When extracted, not only the above information on the extracted image object but also information on which original data was extracted may be included. Such metadata may be provided together when an image object is provided to a user in the future.

The labeling processing unit 400 may label image objects extracted and classified by the data classification unit 200 respectively (each resource image also corresponds to each image object in the end) so as to be uniquely identified.

In other words, the above-described data classification unit 200 only classifies each image object by object, but there is no information on what kind of object each object is, and the labeling processor 400 identifies each of these objects. Label information including keywords and categories may be given as to what the image object is and where it belongs so that it can be uniquely identified.

Here, the labeling processing unit 400 may automatically assign a keyword and a category to each image object, and the keyword may be manually assigned or changed by an administrator or the like in the future.

For example, the labeling processor 400 recognizes each image object and assigns the keyword'strawberry' to the first image object, assigns the keyword'banana' to the second image object, and assigns the keyword'banana' to the third image object. You can assign the keyword'apple'. At the same time, a category of'fruit' may be assigned to each of the first to third image objects.

In addition, the keyword'mugunghwa' may be assigned to the fourth image object, the keyword'rose' may be assigned to the fifth image object, and the keyword'lily' may be assigned to the sixth image object. At the same time, a category of'flower' may be assigned to each of the fourth to sixth image objects.

Such an operation may be referred to as a labeling operation, and through such labeling operation, the user or the search recommendation processing unit 700 to be described later can uniquely identify each image object by the keyword and category, and also by the recommendation algorithm. Can be used.

Meanwhile, the labeling processing unit 400 may generate image object information called a tag in order to quickly search for each image object.

The database unit 500 may store an animation source, each image object, metadata, and label information, respectively, and may provide each image object, metadata, label information, etc. when a user searches for and requests information. .

The database unit 500 may classify and store the label information by category when storing the label information so that keywords belonging to the same category can be simultaneously extracted or searched during a future search.

The search unit 600 is provided for a user to search for and receive image objects stored in the database unit 500, and the search unit 600 is a user device connected to the database unit 500 or It may be a terminal.

The search unit 600 receives search information including at least one of a keyword, an image, and an image input by the user, and provides an animation source corresponding to the search information based on the received search information. Animation sources, image objects, metadata, and label information stored in the base unit 500 may be searched.

6 shows an example of an image object provided through a search unit of the present invention. Referring to this, the image object provided through the search unit 600 is not only the image object itself, but a label including metadata (file surface, file size, etc.) of the image object, keywords, and categories, as shown in the illustrated example. Information may be provided together.

The search unit 600 may directly search for the keyword input by the user in the database unit 500, but other users may not always be able to input only the correct keyword. You can also search 500 information. For example, a user may input an image, an image, or other types of search information, and when such search information is input, the search recommendation processing unit 700 to search the database unit 500 based on the search information. It may further include.

The search recommendation processing unit 700 may search the animation source, image object, metadata, and label information of the database unit 500 based on the search information input to the search unit 600 and transmit the search information to the search unit 600. have.

In this case, the search recommendation processing unit 700 may extract or recommend the animation source and the image object in the order of high relevance to the search information based on the search information input to the search unit 600.

In addition, the search recommendation processing unit 700 may extract or recommend an animation source and an image object having label information belonging to the same category as the input search information input to the search unit 600.

For example, when a keyword is input to the search unit 600, for example, when the keyword'strawberry' is input, the search recommendation processing unit 700 does not search only the image object corresponding to the keyword, but In one example, since the same category of'fruit' was assigned to'strawberry','banana', and'apple', an image object with the keyword of'strawberry' is provided, and at the same time, an image of'banana' or'apple' belonging to the same category You can recommend whether or not to receive the object as well.

For another example, when the keyword'Pororo' is input into the search unit 600, the search recommendation processing unit 700 does not search only the image object corresponding to the keyword'Pororo', but the'Pororo' and It is possible to recommend whether to receive an image object having the same category, for example, an image object such as'Luffy' or'Eddy', which are characters appearing on the same animation.

Meanwhile, when the search information is not a keyword, the search recommendation processing unit 700 may extract a keyword through AI image learning. For example, there may be cases where the user has an image or part of Pororo, but does not know the name of the character or image itself, etc. When input to 600, the search recommendation processing unit 700 may extract each object from the image or image through AI image learning, and then search the database unit 500 based on the extracted object.

In this case, the AI image learning may be a learning operation according to Faster Regions with Convolution Neural Networks (R-CNN) of TensorFlow or You Only Look Once (YOLO) technique.

The search recommendation processing unit 700 may collect and store user information accessing the search unit 600, search information that the user searches through the search unit 600, and keywords according to the search information. This is to recommend an animation source to the user by determining the user's preference for searching.

To explain this, the search recommendation processing unit 700 identifies user information accessing the search unit 600 and collects what type of search information the user inputs, search results, etc., and the number of specific search information or keyword input When the number of the specific keyword input is greater than or equal to a preset reference value (for example, when the number of input specific keywords is 70% or more compared to the number of accesses), the user's search preference information may be calculated and generated based on the search information and the keyword.

Here, when the user accesses the search unit 600, if the user's search taste information is generated, the keyword or result is immediately clicked without typing the keyword according to the taste information in the database unit 500. On the other hand, when an animation source corresponding to the keyword, which was not previously stored in the database unit 500, is input through the source input unit 100, it is notified through an alarm, etc., and is recommended to facilitate the user. It is possible to provide the necessary animation source.

As described above, the intelligent information technology-based animation content resource service system 1 according to the present invention has been described, and next, an intelligent information technology-based animation content resource service method according to the present invention will be described.

7 is a flowchart illustrating an intelligent information technology-based animation content resource service method according to the present invention. Referring to this, the intelligent information technology-based animation content resource service method of the present invention includes the step (s10) of inputting an animation source including at least one of an animation resource, an image, and an image, and an image object included in the input source. Recognizing and classifying each of (s20), extracting metadata for each of the recognized objects (s30), extracting keywords and categories for each of the recognized objects (s40), and , Generating label information to uniquely identify the extracted keyword and category (s50), the animation source, the recognized and classified image object, the metadata, the keyword and category, and Including the step (s60) of storing the label information, respectively, the step (s70) of inputting a keyword, image, or image to be searched, and the association with the keyword, image, or image based on the keyword, image, or image. It may further include the step (s80) of searching for and providing the stored animation source and image object in the order of FIG.

Here, the method may be implemented by the intelligent information technology-based animation content resource service system 1 according to the present invention, and the situation implemented in the system is illustrated for ease of description, but is not limited thereto. , It can be implemented by any system that can perform the same function.

The step (s10) of inputting an animation source including at least one of an animation resource, an image, and an image may be performed by the source input unit 100. In this case, one of a resource, an image, and an image may be input to the source input unit 100, and a resource, an image, or an image may be inputted in combination.

The step (s20) of recognizing and classifying image objects included in the input source may be performed by the data classifying unit 200. At this time, when the input source is an image, each object can be tracked through image analysis and recognized for each object. On the other hand, when the input source is an image, frames constituting the image are determined according to similarity. After classifying into a similar frame group, a representative image is selected from the similar frame group, and each object is tracked through image analysis on the representative image, and each object can be recognized.

In the step of extracting metadata for each recognized object (s30), information on the input source and metadata for each object, that is, the creator, production date, original data name, etc., may be extracted. It may be performed in the data processing unit 300.

The step of extracting the keyword and category for each recognized object (s40) and the step of generating label information to uniquely identify the extracted keyword and category (s50) are performed in the labeling processing unit 400. In the steps s40 and s50, keywords and categories for each object may be extracted, and such information, that is, label information may be generated.

Here, in the step (s20) of recognizing and classifying each image object described above, only the image object is recognized, but since there is no information on what the recognized image object is, each of the recognized image objects can be uniquely identified. It is to create and give label information so that it is possible.

In the step (s60) of storing the animation source, the respectively recognized and classified image object, the metadata, the keyword, the category, and the label information, the above-described image object and information related thereto are stored. In addition, the step (s60) may be performed in the database unit 500.

The step (s70) of inputting a keyword, image, or image to be searched may be performed by the search unit 600, and through the search unit 600, a user may input a keyword to be searched.

The step (s80) of searching and providing the stored animation source and image object in the order of relevance to the keyword, image, or image based on a keyword, image, or image (s80) is performed to the search recommendation processing unit 700 and the search unit 600. The search recommendation processing unit 700 searches the database unit 500 based on a keyword, image, or video input to the search unit 600, and animation through the search unit 600 in the order of relevance. Source and image objects can be provided.

As described above, the method for servicing an intelligent information technology-based animation content resource according to the present invention has been described, and the creator and the user use the system and method for providing an intelligent information technology-based animation content resource according to the present invention to recycle and share animation sources. As a result, waste of time, manpower and cost can be prevented, and mutual symbiosis and profit sharing of the animation distribution ecosystem can be achieved.

In addition, it is possible to easily and efficiently classify each object by recognizing and classifying the object through image learning in the animation source. In addition, it is easy to manage data by classifying keywords and categories for each object, and storing and managing each category. , By providing keywords and image objects belonging to the same category when the user searches, easy search and recommendation functions can be provided, and the clustering processing unit extracts only each object region for the entire image frame and determines only the similarity between object regions to be similar. Since it is classified into frame groups, it is possible to recognize fast images optimized for animation content, and by collecting and evaluating user's search information and recommending animation sources necessary for the user, an intelligent information technology-based animation that can receive an effective recommendation function. Content resource services can be provided.

As described above, the intelligent information technology-based animation content resource service system and method according to the present invention have been described. However, for those of ordinary skill in the art to which the embodiment belongs, the present embodiment is not changed to the technical idea or essential features, and other specific forms. It will be appreciated that it can be implemented with. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting.

100: source input
200: data classification unit
300: metadata processing unit
400: labeling processing unit
500: database part
600: search section
700: search recommendation processing unit

Claims (11)

A source input unit to which an animation source including an image is input;
A data classification unit for classifying image objects included in an animation source input to the source input unit;
A metadata processing unit for extracting metadata for each classified image object;
A labeling processor for extracting keywords and categories for each classified image object, and generating label information for identifying the extracted keywords and categories;
When the animation source, the image object, the metadata, and label information including the keyword and category are stored respectively, and when the label information is stored so that keywords belonging to the same category can be simultaneously extracted or searched for a future search A database unit for classifying and storing by category;
Search for the animation source, the image object, the metadata, and the label information stored in the database to receive search information including an image input by a user and provide an animation source corresponding to the search information. A search unit that can do it; And
A search recommendation processing unit for searching animation source, image object, metadata, and label information to the database unit based on search information input to the search unit and transmitting it to the search unit,
The data classification unit extracts only each object region from the entire image frame and classifies frames with a matching rate of color and outline, which are a preset reference similarity degree, into a similar frame group, with a similarity between each object region being 70% or more, and within the similar frame group. A clustering processing unit for extracting a scene image of the selected representative frame after selecting a frame having the largest object area among a plurality of included frames as a representative frame, and a scene image extracted from the clustering processing unit as an image object included in the scene image. It includes an image object tracking unit to classify each,
The image object tracking unit performs AI image learning to distinguish and classify the image objects, respectively,
The search recommendation processing unit extracts or recommends the animation source and the image object in the order of high relevance to the search information based on search information input to the search unit, or a label belonging to the same category as the input search information Extract or recommend animation sources and image objects with information,
The search recommendation processing unit collects and stores user information accessing the search unit, search information searched by the user through the search unit, and keywords according to the search information, and the number of times the user's specific search information and keyword is input If it is more than a preset reference value, the user's search taste information is calculated based on the search information and the keyword, and when an animation source corresponding to the taste information is input to the database, a corresponding animation source is recommended to the user. Based on intelligent information technology, animation content resource service system. delete delete The method of claim 1,
The AI image learning is an intelligent information technology-based animation content resource service system, characterized in that the learning operation according to Faster R-CNN (Faster Regions with Convolution Neural Networks) or YOLO (You Only Look Once) technique. delete delete delete delete delete delete delete

Images