KR20220107952A - Appratus and method for monitoring creator`s copyright - Google Patents

Abstract

In accordance with one aspect of the present disclosure, a copyright monitoring apparatus includes: at least one memory storing at least one instruction; and at least one processor connected with the memory. The at least one processor can be configured to execute the at least one instruction: receive a user identification (ID), a first image created by the user, and a character tag for the first image from the user; calculate the number of accumulated images while mapping the first image with each user ID and each character tag; determine a copyright check algorithm based on the number of accumulated images; receive a second image created by another user; and determine whether the second image violates the copyright of the first image based on the copyright check algorithm. The copyright check algorithm includes a first copyright check algorithm based on supervised learning, and a second copyright check algorithm based on similarity. Therefore, the present invention is capable of efficiently monitoring copyrights through the application of different copyright check algorithms.

Description

Apparatus and method for copyright monitoring of creators

The present disclosure relates to a copyright monitoring apparatus and method of a creator.

Content analysis, search, and provision are rapidly developing due to the development of deep learning-based artificial intelligence technology, and are expected to be applied and utilized in various platform fields. For example, in order to reduce uncertainty about the quality and taste of experiential contents, user-customized contents may be provided based on artificial intelligence technology in the decision-making process of consumption of digital contents. User-customized content is emerging as an important keyword in artificial intelligence technology, especially content-related technology. Artificial intelligence technology is an essential element in the content market where economies of scale are important.

An example of digital content is creations uploaded by creators. Creators are building their image portfolios by uploading images such as characters and illustrations they have created through various image platforms. On the other hand, there is a need to protect creations uploaded to such image platforms, but the existing image platforms do not provide a function of protecting creations through copyright inspection of uploaded creations.

Republic of Korea Patent Registration 10-1557321 Korean Patent Registration 10-2052534

Various examples of the present disclosure are intended to provide a copyright monitoring apparatus and method capable of efficiently monitoring copyright by applying different copyright checking algorithms according to the number of raw data.

Technical problems to be achieved in various examples of the present disclosure are not limited to those mentioned above, and other technical problems not mentioned are to those of ordinary skill in the art from various examples of the present disclosure to be described below. can be considered by

In one aspect of the present disclosure, at least one memory (memory) for storing at least one instruction (instruction); and at least one processor coupled to the memory, wherein the at least one processor executes the at least one instruction: a user identification from a user, a first image created by the user and the receive a character tag for the first image; calculating the accumulated number of images while mapping the first image for each of the user ID and the character tag; determine a copyright checking algorithm based on the accumulated number of images; receive a second image created by another user; and to determine whether the second image violates the copyright of the first image based on the copyright check algorithm, wherein the copyright check algorithm is a first copyright check algorithm based on supervised learning and a second copyright based on similarity It is a copyright monitoring device, including a checking algorithm.

For example, the at least one processor executes the at least one instruction: if the number of accumulated images is greater than or equal to a preset threshold, it is determined by the first copyright check algorithm, and the number of accumulated images is equal to or greater than the preset threshold. If less than, it may be further configured to determine with the second copyright check algorithm.

For example, the first copyright check algorithm generates a feature map based on a convolution filter for the user ID and the accumulated images for each character tag and pre-stored training data, and classifies the feature map as a classifier. to generate an output value indicating whether or not the user is a character, train the convolution filter and the classifier until the loss function of the output value is minimized, and based on the convolution filter and the classifier, the It may be determined whether the second image violates the copyright of the first image.

For example, the second copyright check algorithm extracts an edge and a feature point for each of the first image and the second image, and calculates a similarity between the first image and the second image based on the edge and the feature point and, when the similarity is equal to or greater than a preset threshold similarity value, it may be determined that the second image violates the copyright of the first image.

For example, the at least one processor executes the at least one instruction to: extract a unified action unit (AU) from a character object included in the first image and the second image, wherein the unified AU is a human includes AUs and character AUs; and generating a pre-processed image obtained by filtering the human AU from the first image and the second image, wherein the copyright check algorithm may be performed on the pre-processed image.

For example, the human AU may include a plurality of eyebrows AUs, a plurality of eyes AUs, and a plurality of eyebrows AUs having different distances and ratios between a plurality of feature points included in each of the eyebrows, eyes, nose, mouth, and cheeks of the character object. at least one of a nose AU, a plurality of mouth AUs, and a plurality of cheeks AU, wherein the character AU may include at least one of a background object AU and an emotion object AU included in the first image and the second image have.

For example, the emotion object AU corresponds to an object including an edge at least partially overlapping with the character object among objects other than the character object in the first image and the second image, and the background object AU is The first image and the second image may correspond to objects other than the character object and the emotion object AU.

In another aspect of the present disclosure, there is provided a copyright monitoring method performed by a copyright monitoring device, comprising: receiving a user ID (identification) from a user, a first image created by the user, and a character tag for the first image; calculating the accumulated number of images while mapping the first image for each of the user ID and the character tag; determining a copyright check algorithm based on the accumulated number of images; receiving a second image created by another user; and determining whether the second image violates the copyright of the first image based on the copyright check algorithm, wherein the copyright check algorithm is a first copyright check algorithm based on supervised learning and a first copyright check algorithm based on similarity. 2 It is a copyright monitoring method, including a copyright check algorithm.

For example, in the determining of the copyright check algorithm, if the number of accumulated images is greater than or equal to a preset threshold, it is determined by the first copyright check algorithm, and if the number of accumulated images is less than the preset threshold, the second copyright It can be determined by the inspection algorithm.

For example, the first copyright check algorithm generates a feature map based on a convolution filter for the user ID and the accumulated images for each character tag and pre-stored training data, and classifies the feature map as a classifier. to generate an output value indicating whether or not the user is a character, train the convolution filter and the classifier until the loss function of the output value is minimized, and based on the convolution filter and the classifier, the 2 It is determined whether the image violates the copyright of the first image, and the second copyright check algorithm extracts an edge and a feature point for each of the first image and the second image, and based on the edge and the feature point The similarity between the first image and the second image may be calculated, and when the similarity is equal to or greater than a preset threshold similarity value, it may be determined that the second image violates the copyright of the first image.

The various examples of the present disclosure described above are only some of the preferred examples of the present disclosure, and various examples in which the technical features of various examples of the present disclosure are reflected are detailed descriptions to be detailed below by those of ordinary skill in the art. can be derived and understood based on

According to various examples of the present disclosure, the following effects are obtained.

According to various examples of the present disclosure, a copyright monitoring apparatus and method capable of efficiently monitoring copyright by applying different copyright checking algorithms according to the number of raw data may be provided.

Effects that can be obtained from various examples of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned are clearly derived to those of ordinary skill in the art based on the detailed description below. and can be understood

The accompanying drawings are provided to help understanding of various examples of the present disclosure, and various examples of the present disclosure are provided together with the detailed description. However, the technical features of various examples of the present disclosure are not limited to specific drawings, and the features disclosed in each drawing may be combined with each other to constitute a new embodiment. Reference numerals in each drawing refer to structural elements.
1 is a block diagram of a copyright monitoring apparatus according to an example of the present disclosure.
2 is a functional block diagram of a copyright monitoring apparatus according to an example of the present disclosure.
3 is a functional block diagram of a copyright monitoring apparatus according to another example of the present disclosure.
4 to 7 are for explaining an action unit (AU) defined according to various examples of the present disclosure.
8 is a block diagram of a server according to an example of the present disclosure.
9 is for explaining emotion category labeling.
10 is a flowchart of a copyright monitoring method according to an example of the present disclosure.
11 is a flowchart of a first copyright check algorithm according to an example of the present disclosure.
12 is a flowchart of a second copyright check algorithm according to an example of the present disclosure.

Hereinafter, implementations according to the present invention will be described in detail with reference to the accompanying drawings. DETAILED DESCRIPTION The detailed description set forth below in conjunction with the appended drawings is intended to describe exemplary implementations of the present invention, and is not intended to represent the only implementation forms in which the present invention may be practiced. The following detailed description includes specific details in order to provide a thorough understanding of the present invention. However, it will be apparent to one of ordinary skill in the art that the present disclosure may be practiced without these specific details.

Since various examples according to the concept of the present invention can have various changes and have various forms, various examples are illustrated in the drawings and described in detail in the present disclosure. However, this is not intended to limit various examples according to the concept of the present invention to specific disclosed forms, and includes changes, equivalents, or substitutes included in the spirit and scope of the present invention.

Terms such as first or second may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from other components, for example, without departing from the scope of rights according to the concept of the present invention, a first component may be named as a second component, Similarly, the second component may also be referred to as the first component.

In various examples of the present disclosure, “/” and “,” should be construed as indicating “and/or”. For example, “A/B” may mean “A and/or B”. Furthermore, “A, B” may mean “A and/or B”. Furthermore, “A/B/C” may mean “at least one of A, B, and/or C”. Furthermore, “A, B, and C” may mean “at least one of A, B and/or C”.

In various examples of the present disclosure, “or” should be construed as indicating “and/or”. For example, “A or B” may include “only A”, “only B”, and/or “both A and B”. In other words, “or” should be construed as indicating “additionally or alternatively”.

1 is a block diagram of a copyright monitoring apparatus according to an example of the present disclosure.

Referring to FIG. 1 , the copyright monitoring apparatus 100 according to an example of the present disclosure is to provide a copyright monitoring service capable of monitoring whether a copyright of a creation uploaded by a creator is violated, and for this purpose, at least one transceiver 110 , at least one memory 120 , and at least one processor 130 .

The transceiver 110 may be connected to the processor 130 and may transmit and/or receive wired/wireless signals or data. For example, the transceiver 110 may be connected to a user terminal and/or a server through a wired/wireless communication network.

Here, the wireless communication network may include a mobile communication network, a wireless LAN, a local area wireless communication network, and the like. For example, wireless communication networks include LTE, LTE Advance (LTE-A), code division multiple access (CDMA), wideband CDMA (WCDMA), universal mobile telecommunications system (UMTS), Wireless Broadband (WiBro), and Global System (GSM). for Mobile Communications) and the like. For example, the wireless communication network may include at least one of wireless fidelity (WiFi), Bluetooth, Bluetooth low energy (BLE), Zigbee, near field communication (NFC), and radio frequency (RF).

Here, the wired communication network may include at least one of a Universal Serial Bus (USB), a Universal Synchronous/Asynchronous Receiver Transmitter (USART), and an Ethernet.

Each of the at least one transceiver 110 may correspond to the above-described wired/wireless communication network.

The transceiver 110 may include a transmitter and a receiver. The transceiver 110 may be used interchangeably with a radio frequency (RF) unit. The transceiver 110 may transmit and receive various signals or data with the user terminal and/or the server under the control of the processor 130 .

The memory 120 may be connected to the processor 130 and may store various information related to the operation of the processor 130 . For example, the memory 120 provides instructions for performing some or all of the processes controlled by the processor 130 , or for performing the descriptions, functions, procedures, suggestions, methods, and/or operational flowcharts of the present disclosure. It may store software code including (instructions).

The processor 130 controls the transceiver 110 and/or the memory 120 , and executes instructions stored in the memory 120 to implement the descriptions, functions, procedures, suggestions, methods, and/or operational flowcharts of the present disclosure. can be configured. For example, the processor 130 may receive a signal through the transceiver 110 and store information included in the signal in the memory 120 . In addition, the processor 130 may provide a copyright monitoring service based on the instructions stored in the memory 120 . In addition, the processor 130 may generate a signal by processing information stored in the memory 120 , and then transmit the generated wireless signal through the transceiver 110 .

Hereinafter, various embodiments of the copyright monitoring apparatus 100 and method will be described. Various embodiments related to the copyright monitoring apparatus 100 and method disclosed below may be performed based on the processor 130 executing the above-described instructions.

The functional units included in the copyright monitoring device 100 disclosed below are software or a combination of hardware and software for implementing hardware or instructions including the above-described transceiver 110, memory 120, and processor 130. can be implemented.

2 is a functional block diagram of a copyright monitoring apparatus according to an example of the present disclosure.

Referring to FIG. 2 , the copyright monitoring apparatus 100 according to an example of the present disclosure includes a communication unit 140 , an image mapping unit 150 , a copyright check algorithm determining unit 160 , and a copyright monitoring unit 170 . .

The communication unit 140 may correspond to the above-described transceiver 110 . The communication unit 140 receives a user ID, a first image created by the user, and a character tag for the first image from the user. In the present disclosure, all images created by a user are collectively referred to as a first image for convenience of distinction from images created by other users, and the first image does not refer to only a specific image created by the user, but all images created by the user. is a concept that refers to The character tag may be a tag designated or selected by a user to indicate a character included in the first image.

The communication unit 140 transmits the received data to the image mapping unit 150 .

The image mapping unit 150 calculates the accumulated number of images while mapping the first image for each received user ID and character tag. That is, the first image is stored for each character tag designated by the user for the same user, and the accumulated image means the first image stored for each character tag for the same user.

The copyright check algorithm determining unit 160 determines a copyright check algorithm based on the number of accumulated images. In the present disclosure, the copyright check algorithm includes a first copyright check algorithm based on supervised learning and a second copyright check algorithm based on similarity, and a detailed description thereof will be described later.

Specifically, if the number of accumulated images is greater than or equal to a preset threshold value, the copyright check algorithm determining unit 160 determines with the first copyright check algorithm, and if the number of accumulated images is less than a preset threshold, it is determined with the second copyright check algorithm. Therefore, the copyright monitoring device 100 determines whether the copyright is violated based on supervised learning if the number of accumulated images is sufficient, and if not, determines whether the copyright is violated based on the similarity, so the number of raw data Different copyright check algorithms can be applied accordingly, enabling efficient copyright monitoring. In particular, when the number of raw data is insufficient to derive the performance of supervised learning, a similarity-based algorithm can be used.

The copyright check algorithm determining unit 160 determines that the number of accumulated images is insufficient and, after determining with the second copyright check algorithm, when the number of accumulated images increases due to an increase in the number of stored first images, the first copyright check in the second copyright check algorithm can be determined by an algorithm.

Subsequently, the communication unit 140 receives the second image created by another user, and transmits the second image to the copyright monitoring unit 170 .

The copyright monitoring unit 170 determines whether the second image violates the copyright of the first image based on the copyright inspection algorithm determined by the copyright inspection algorithm determining unit 160 .

According to an embodiment, the copyright monitoring unit 170 may determine whether a copyright is violated by using the first image and the second image.

For example, the copyright monitoring unit 170 performs the first copyright check algorithm when it is determined as the first copyright check algorithm. The first copyright checking algorithm is based on supervised learning.

First, the copyright monitoring unit 170 generates a feature map based on a convolution filter with respect to the accumulated image (ie, the first image) and pre-stored training data for each user ID and character tag. Here, the pre-stored training data is training data stored in the memory, and in particular, may be an image corresponding to the same character tag.

Next, the copyright monitoring unit 170 generates an output value indicating whether the user's character by inputting the feature map to the classifier. Specifically, the classifier includes an input layer, a plurality of hidden layers, and an output layer, and the output layer may include two neurons to indicate whether the user is a character or not. An output value is generated from each neuron.

Next, the copyright monitoring unit 170 trains the convolution filter and the classifier until the loss function of the output value is minimized. For example, the copyright monitoring unit 170 may perform learning through a known back propagation operation. Through learning, the filter value of the convolution filter is learned to extract the feature that can best represent the accumulated image, and the weight and bias of each layer included in the classifier are learned to improve classification performance. In particular, the filter value of the convolution filter may be learned to extract a feature that can best represent the character object included in the accumulated image.

Finally, the copyright monitoring unit 170 determines whether the second image violates the copyright of the first image based on the learned convolution filter and the learned classifier.

For example, the copyright monitoring unit 170 performs a second copyright check algorithm when it is determined as the second copyright check algorithm. The second copyright checking algorithm is based on similarity.

First, the copyright monitoring unit 170 extracts edges and feature points for each of the first image and the second image.

Next, the copyright monitoring unit 170 calculates a similarity between the first image and the second image based on the edge and the feature point. For example, the copyright monitoring unit 170 may determine the similarity for each edge, determine the similarity for each feature point, and calculate the similarity between the first image and the second image by synthesizing the calculated similarity.

The copyright monitoring unit 170 determines that the second image violates the copyright of the first image when the similarity is greater than or equal to a preset threshold similarity value.

3 is a functional block diagram of a copyright monitoring apparatus according to another example of the present disclosure. Hereinafter, a detailed description of the content overlapping with the above-described content will be omitted.

Referring to FIG. 3 , the copyright monitoring apparatus 100 according to another example of the present disclosure includes the communication unit 140 , the image mapping unit 150 , the copyright check algorithm determining unit 160 and the copyright monitoring unit 170 of FIG. 2 . ), and may further include an AU extractor 151 and a preprocessor 152 .

The AU extraction unit 151 extracts an integrated action unit (AU) from the first image and the second image. The copyright monitoring unit 170 may determine whether the copyright is violated based on the AU extracted from the AU extraction unit 151 . Hereinafter, the integrated AU defined in the present disclosure will be described in more detail.

4 to 7 are for explaining an AU defined according to various examples of the present disclosure.

The integrated AU is extracted from an image (eg, all image data included in the present disclosure, such as a first image and a second image), and may include human AU and character AU.

The human AU may correspond to an element representing an expression, such as facial features, etc., and the character AU corresponds to an element that is not related to the expression of the face but directly or indirectly represents the emotion of the character, and for convenience, each of the human AU and the character It may be referred to as an enemy AU.

Referring to FIG. 4 , the human AU is a plurality of eyebrow AUs and a plurality of eyes having different distances and ratios between a plurality of feature points included in each of the eyebrows, eyes, nose, mouth, and cheeks of the character object included in the image. It may include at least one of an AU, a plurality of nose AUs, a plurality of mouth AUs, and a plurality of cheek AUs.

The eyebrow AU, eye AU, nose AU, mouth AU, and cheek AU may include at least one of a plurality of feature points and edges included in each of the eyebrows, eyes, nose, mouth, and cheeks of the character object. Eyebrow AU, eye AU, nose AU, mouth AU, and cheek AU may be variously set or defined according to emotions.

Referring to FIG. 5 , for example, the character object corresponding to the sadness emotion category is a frown eyebrow AU 311 , closed eyes AU 321 , an enlarged nose AU 331 , and/or an M-shaped mouth AU 341 ). and the like.

For example, the character object corresponding to the angry emotion category may include angry eyes AU 322 , enlarged nose AU 332 , and/or open mouth AU 342 .

For example, the character object corresponding to the joy emotion category may include a smiling eye AU 323 and/or a U-shaped mouth AU 343 .

6 and 7 , the character AU may include at least one of a background object AU and an emotion object AU included in an image. The background object AU and the emotion object AU may each include at least one of a plurality of feature points and edges included in each of the above-described background object and emotion object. The background object AU and the emotion object AU may be variously set or defined according to emotions.

More specifically, the emotion object AU may correspond to an object including an edge at least partially overlapping with the character object among objects other than the character object in the image. Also, the background object AU may correspond to objects other than the character object and the emotion object AU in the image.

For example, the emotional object AU may include a tear AU 411 , an angry display AU 412 , an angry AU 413 , and a sweat drop AU 414 .

For example, the background object AU may include a flower pattern AU 421 , a text AU 422 , and the like. Here, the text AU 422 may include, for example, a text representing joy, such as 'I'm so happy', as illustrated in the case of a joy emotion category.

The above-described integrated AUs are merely exemplary, and in addition, various elements capable of effectively expressing a corresponding emotion for each emotion category may be set or defined as the integrated AU.

Referring back to FIG. 3 , the AU extraction unit 151 extracts the above-described various integrated AUs from the character objects included in the first image and the second image. Each extracted integrated AU may be classified according to the above-described human AU and character AU.

The preprocessor 152 generates a preprocessed image obtained by filtering human AUs from the first image and the second image. In other words, the preprocessor 152 generates a preprocessed image by removing the character AU from among the extracted integrated AU and filtering only the human AU.

Thereafter, the above-described copyright check algorithm is performed on the pre-processed image generated by the pre-processing unit 152 . That is, the copyright monitoring unit 170 may perform the first copyright check algorithm or the second copyright check algorithm on the pre-processed image other than the first image and the second image.

For example, the copyright monitoring unit 170 may perform the first copyright check algorithm on the preprocessed image. The copyright monitoring unit 170 may determine whether the above-described feature map and output value generation, the learning operation of the convolution filter and the classifier, and copyright violation, based on the pre-processed image.

For example, the copyright monitoring unit 170 may perform the second copyright check algorithm on the preprocessed image. The similarity determination of the copyright monitoring unit 170 may be performed in units of the above-described AUs. For example, the copyright monitoring unit 170 may determine the similarity of edges and/or feature points for each of a plurality of AUs included in the human AU, and calculate a similarity value for each of the plurality of AUs.

When the ratio of the similarity value exceeding the preset threshold similarity value among the calculated similarity values is A[%] of the total ratio (where A is a real number included in 0 to 100), the copyright monitoring unit 170 controls the second image may be determined to violate the copyright of the first image.

The copyright monitoring apparatus 100 according to various examples of the present disclosure described above may improve the efficiency of copyright inspection by applying different copyright inspection algorithms in consideration of the number of raw data. In addition, when the copyright check algorithm is performed on the pre-processed image from which the AU is extracted, the operation speed of the copyright check algorithm may be improved.

Hereinafter, a server including the above-described copyright monitoring apparatus 100 will be described.

8 is a block diagram of a server according to an example of the present disclosure.

Referring to FIG. 8 , the server includes a copyright monitoring device 100 and a character emotion recognition and tagging device 200 .

The server 10, based on the copyright monitoring device 100, checks whether the uploaded image from the user terminal violates the copyright, or based on the character emotion recognition and tagging device 200, the image uploaded from the user terminal It is possible to recognize the emotion of the character included in the , and tag the recognized emotion.

Hereinafter, various operation examples of the character emotion recognition and tagging apparatus 200 will be described.

The character emotion recognition and tagging apparatus 200 receives input image data including a character object from a user terminal. In the present disclosure, the character object may be an object corresponding to a character or emoticon created by a creator rather than a live action. The input image data includes a character object, and may include an emotion object for indicating the emotional state of the corresponding character and/or a background object corresponding to a background image.

Character emotion recognition and tagging apparatus 200, based on a feature of the input image data received from the user terminal and a pre-learned classification model, the character object included in the input image data, the emotion of one of a plurality of emotion categories can be classified into categories. In other words, the character emotion recognition and tagging apparatus 200 may recognize and classify the emotion of the character object by using the characteristics of the input image data as input data of the pre-learned classification model.

A plurality of emotion categories for emotion recognition and classification of character objects may be set or defined in various ways. For example, the emotion categories are Joy, Desire, Excitement, Surprise, Angry, Fear, Sadness, Boredom, Tiredness, Disappointment, Shame, Spout, Comfort, Worry, Lovingness, Insidiousness, Hunger, Pain, Embarrassment, and Playfulness. etc. may be set or defined.

Whether to be classified into any one emotion category among the emotion categories as in the above example may be performed based on various artificial intelligence (AI) algorithms. In other words, the learning algorithm for learning the previously learned classification model may include various machine learning algorithms such as supervised learning, unsupervised learning, reinforcement learning, and transfer learning.

The feature for learning the classification model may be the above-described integrated AU of the character object. That is, the character emotion recognition and tagging apparatus 200 may learn a classification model based on the integrated AU of the character object, and may generate a pre-learned classification model as a result of the learning. In the present disclosure, AU may mean one element formed by at least one of an edge extracted from a character object and a feature point included in the edge.

9 is for explaining emotion category labeling.

Referring to FIG. 9 , each of the above-described integrated AUs may be labeled with at least one of a plurality of emotion categories. In this case, the training image data of the classification model may be, for example, (emotion category, unified AU) pairs. That is, when training the classification model, supervised learning using the integrated AU labeled in the emotion category as training image data may be used.

The character emotion recognition and tagging apparatus 200 may generate tagged input image data by tagging one classified emotion category to input image data. Here, the tagged input image data may include input image data and classified information on one emotion category.

The character emotion recognition and tagging apparatus 200 may transmit the generated tagged input image data to the user terminal.

According to the above-described character emotion recognition and tagging apparatus 200 , emotions of objects included in creations, such as characters or emoticons, in particular, not in real life, can be effectively recognized and classified. Specifically, the character emotion recognition and tagging apparatus 200 may more precisely recognize and classify the emotion of a character object by considering a character element in a feature used in an AI algorithm such as machine learning.

Alternatively, according to another example of the present disclosure, the character emotion recognition and tagging apparatus 200 may select a feature to be used for emotion classification when emotion classification of input image data is performed. Here, the selected feature may be any one of character AUs.

Specifically, the character emotion recognition and tagging apparatus 200 may receive classification option data from the user terminal. Here, the classification option data may be selection information for selecting at least one of the background object AU and the emotion object AU.

The character emotion recognition and tagging apparatus 200 may select an AU corresponding to the received selection information among the background object AU and the emotion object AU.

The character emotion recognition and tagging apparatus 200 may classify the character object into one emotion category based on the human AU, the AU corresponding to the received selection information, and a pre-learned classification model.

Accordingly, the character emotion recognition and tagging apparatus 200 according to another example of the present disclosure may allow a creator to select a feature for emotion recognition and classification, and perform emotion recognition and classification based on the selected feature, thereby making the machine It can increase the satisfaction of learning.

The server may search for similar images corresponding to the classified and tagged emotion categories through the character emotion recognition and tagging apparatus 200 for market research of content, and transmit the searched similar images to the user terminal.

Also, the server may determine whether content copyright is violated by performing a similarity check between the similar image searched for through the copyright monitoring device 100 and the uploaded image. Accordingly, if it is determined through the similarity check that the content copyright is violated, the server may transmit a separate warning message to the user terminal.

Alternatively, according to another example of the present disclosure, the character emotion recognition and tagging apparatus 200 may perform reclassification by adding a feature according to the evaluation degree of the classified emotion.

Specifically, the character emotion recognition and tagging apparatus 200 may receive evaluation data for the tagged input image data. Here, the evaluation data may be data obtained by quantifying the user's satisfaction with the primarily classified emotions.

Character emotion recognition and tagging apparatus 200, if the received evaluation data is less than a preset threshold value, based on at least one of human AU, background object AU, and emotion object AU and a pre-learned classification model, the character object is one It can be classified into emotion categories. For example, when the maximum value of the evaluation value corresponding to the evaluation data is 10 points, the preset threshold value is 5 points, and the value of the evaluation value corresponding to the received evaluation data is 4 points, the character emotion recognition and tagging apparatus 200 ) may be characterized by at least one of a human AU, a background object AU, and an emotion object AU to perform emotion classification again.

According to another example of the present disclosure described above, the character emotion recognition and tagging apparatus 200 may be particularly useful when the emotion classification performed primarily features only human AU. That is, the character emotion recognition and tagging apparatus 200 primarily performs emotion classification based on human AU. User satisfaction can be increased by re-classifying.

Hereinafter, a copyright monitoring method performed by the copyright monitoring apparatus 100 will be described.

10 is a flowchart of a copyright monitoring method according to an example of the present disclosure.

Referring to FIG. 10 , in S110 , the copyright monitoring apparatus 100 receives a user ID, a first image created by the user, and a character tag for the first image from the user.

In S120, the copyright monitoring apparatus 100 calculates the accumulated number of images while mapping the first image for each user ID and character tag.

In S130, the copyright monitoring apparatus 100 determines a copyright check algorithm based on the number of accumulated images. The copyright check algorithm includes a first copyright check algorithm based on supervised learning and a second copyright check algorithm based on similarity. In S130, the copyright monitoring apparatus 100 may determine with the first copyright check algorithm if the number of accumulated images is equal to or greater than a preset threshold, and may determine with the second copyright check algorithm if the number of accumulated images is less than the preset threshold.

In S140, the copyright monitoring device 100 receives the second image created by another user.

In S150, the copyright monitoring apparatus 100 determines whether the second image violates the copyright of the first image based on the copyright check algorithm.

11 is a flowchart of a first copyright check algorithm according to an example of the present disclosure.

Referring to FIG. 11 , in S210 , the copyright monitoring apparatus 100 generates a feature map based on a convolution filter for accumulated images and pre-stored training data for each user ID and character tag.

In S220, the copyright monitoring apparatus 100 generates an output value indicating whether the user's character by inputting the feature map to the classifier.

In S230, the copyright monitoring apparatus 100 trains the convolution filter and the classifier until the loss function of the output value is minimized.

In S240, the copyright monitoring apparatus 100 determines whether the second image violates the copyright of the first image based on the convolution filter and the classifier.

12 is a flowchart of a second copyright check algorithm according to an example of the present disclosure.

Referring to FIG. 12 , in S310 , the copyright monitoring apparatus 100 extracts edges and feature points for each first image and second image.

In S320, the copyright monitoring apparatus 100 calculates the similarity between the first image and the second image based on the edge and the feature point.

In S330, the copyright monitoring apparatus 100 determines whether the similarity is equal to or greater than a preset threshold similarity value.

In S340 , the copyright monitoring apparatus 100 determines that the second image violates the copyright of the first image when it is determined that the similarity is equal to or greater than a preset threshold similarity value.

Since examples of the proposed method in the above description may also be included as one of the implementation methods of the present disclosure, it is obvious that they may be regarded as a kind of proposed method. In addition, the above-described proposed methods may be implemented independently, but may also be implemented in the form of a combination (or merge) of some of the proposed methods.

The examples of the present disclosure disclosed as described above are provided to enable those skilled in the art to make and practice the present disclosure. Although the above has been described with reference to examples of the present disclosure, those skilled in the art may variously modify and change the examples of the present disclosure. Accordingly, this disclosure is not intended to be limited to the examples set forth herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

10: Server
100: copyright monitoring device
110: transceiver 120: processor
130: memory
200: character emotion recognition and tagging device

Claims (10)

at least one memory for storing at least one instruction; and
At least one processor coupled to the memory,
The at least one processor executes the at least one instruction to:
receiving a user identification (ID) from a user, a first image created by the user, and a character tag for the first image;
calculating the accumulated number of images while mapping the first image for each of the user ID and the character tag;
determine a copyright checking algorithm based on the accumulated number of images;
receive a second image created by another user; and
and determine whether the second image violates the copyright of the first image based on the copyright checking algorithm;
The copyright check algorithm includes a first copyright check algorithm based on supervised learning and a second copyright check algorithm based on similarity,
Copyright monitoring device.
According to claim 1,
The at least one processor executes the at least one instruction to:
If the number of accumulated images is greater than or equal to a preset threshold, it is determined by the first copyright check algorithm,
Further configured to determine with the second copyright check algorithm if the number of accumulated images is less than the preset threshold value,
Copyright monitoring device.
3. The method of claim 2,
The first copyright check algorithm,
A feature map is generated based on a convolution filter with respect to the user ID and accumulated images for each character tag and pre-stored training data, and the feature map is input to a classifier to indicate whether the character is the user's character. and training the convolution filter and the classifier until the loss function of the output value is minimized,
determining whether the second image violates the copyright of the first image based on the convolution filter and the classifier,
Copyright monitoring device.
3. The method of claim 2,
The second copyright check algorithm,
When an edge and a feature point are extracted for each of the first image and the second image, a similarity between the first image and the second image is calculated based on the edge and the feature point, and the similarity is greater than or equal to a preset threshold similarity value Determining that the second image violates the copyright of the first image,
Copyright monitoring device.
According to claim 1,
The at least one processor executes the at least one instruction to:
Extracting an integrated action unit (AU) from the character object included in the first image and the second image,
wherein the integrated AU includes human AU and character AU; and
further configured to generate a filtered pre-processing image of the human AU from the first image and the second image,
The copyright check algorithm is performed on the pre-processed image,
Copyright monitoring device.
6. The method of claim 5,
The human AU is a plurality of eyebrows AUs, a plurality of eyes AUs, a plurality of noses AUs, and a plurality of different distances and ratios between a plurality of feature points included in each of the eyebrows, eyes, nose, mouth and cheeks of the character object. at least one of a mouth AU and a plurality of ball AUs;
The character AU includes at least one of a background object AU and an emotion object AU included in the first image and the second image,
Copyright monitoring device.
7. The method of claim 6,
The emotion object AU corresponds to an object including an edge at least partially overlapping with the character object among the remaining objects except for the character object in the first image and the second image,
The background object AU corresponds to the remaining objects except for the character object and the emotion object AU in the first image and the second image,
Copyright monitoring device. A copyright monitoring method performed by a copyright monitoring device, comprising:
receiving a user ID (identification) from a user, a first image created by the user, and a character tag for the first image;
calculating the accumulated number of images while mapping the first image for each of the user ID and the character tag;
determining a copyright check algorithm based on the accumulated number of images;
receiving a second image created by another user; and
determining whether the second image violates the copyright of the first image based on the copyright check algorithm;
The copyright check algorithm includes a first copyright check algorithm based on supervised learning and a second copyright check algorithm based on similarity,
How to monitor copyrights.
9. The method of claim 8,
In the determining of the copyright check algorithm, if the number of accumulated images is greater than or equal to a preset threshold, it is determined by the first copyright check algorithm, and if the number of accumulated images is less than the preset threshold, it is determined by the second copyright check algorithm. doing,
How to monitor copyrights.
10. The method of claim 9,
The first copyright check algorithm,
A feature map is generated based on a convolution filter with respect to the user ID and accumulated images for each character tag and pre-stored training data, and the feature map is input to a classifier to indicate whether the character is the user's character. and training the convolution filter and the classifier until the loss function of the output value is minimized,
determining whether the second image violates the copyright of the first image based on the convolution filter and the classifier;
The second copyright check algorithm,
When an edge and a feature point are extracted for each of the first image and the second image, a similarity between the first image and the second image is calculated based on the edge and the feature point, and the similarity is greater than or equal to a preset threshold similarity value Determining that the second image violates the copyright of the first image,
How to monitor copyrights.

Images