KR20230115028A - Method and apparatus for determining region with high complexity from image of contents to insert identification information therein - Google Patents

Abstract

A method is provided that includes: extracting a unit cut included in an image from the image of content including a plurality of unit cuts; determining at least one complex area with high complexity among the areas within the unit cut as an area where identification information associated with the content (e.g., a watermark including identification information of a user who viewed the content) is inserted. Therefore, it is possible to ensure the robustness of identification information against piracy.

Description

Method and apparatus for determining high complexity region from image of content to insert identification information

The present disclosure relates to a method and apparatus for determining a region of high complexity from an image of content, and more particularly, to a region of high complexity suitable for inserting identification information from an image of content such as webtoon content including a plurality of cuts. It relates to a method and apparatus for determining.

Recently, interest in a so-called webtoon content providing service that provides content such as cartoons and cartoons online is increasing. These webtoon contents express characters, animals, objects, etc. as characters, and provide readers with something to read by including humor, satire, or a certain plot through fingerprints or lines.

These webtoon contents can be read by the reader by scrolling the corresponding webtoon contents. The webtoon content includes a plurality of cuts, and the reader can browse the webtoon content by sequentially checking the plurality of cuts through scrolling.

Webtoon content is digitized content and is provided in the form of, for example, an image, and secondary copying and processing of webtoon content in the form of such an image is relatively easy. In order to prevent illegal copying and processing of webtoon contents, predetermined identification information can be inserted into webtoon contents, and illegal copying and processing are performed by extracting such identification information from illegally copied and processed webtoon contents. One subject can be detected.

Such identification information inserted into webtoon contents must have toughness to effectively cope with illegal copying and processing, and must not be easily recognized by pirates or readers of illegally copied and processed webtoon contents.

On the other hand, Korean Patent Publication No. 10-2011-0123393 (published on November 15, 2011) discloses a technology for providing cartoons in the form of mobile digital contents through direct online transactions.

The information described above is for illustrative purposes only and may include material that does not form part of the prior art, and may not include what the prior art may suggest to those skilled in the art.

In one embodiment, a unit cut included in a corresponding image is extracted from an image of content including a plurality of unit cuts (ie, scene cuts), and at least one complex region having a higher complexity is selected from among regions within the unit cut. It is possible to provide a method for determining an area where identification information associated with (eg, a watermark including identification information of a user who browses content) is to be inserted.

In one embodiment, in extracting a unit cut in an image of content to determine a region for inserting identification information, a rule-based method based on pixel analysis of an image or an object or scene cut It is possible to provide a method of using a deep learning model trained to detect .

An embodiment is a rule-based method based on analyzing various parameters of an image to determine a complex region having a high complexity score within an extracted unit cut as a region for inserting identification information, a rule-based method A method of using a deep learning model trained to estimate the complexity determined according to , or a method of using an attention map obtained through a classification model of an image may be provided.

In one aspect, a method for determining a complex region from an image of content including a plurality of unit cuts, performed by a computer system, comprising: extracting unit cuts included in the image from the image; A method for determining a complicated region is provided, comprising determining at least one complex region having a higher complexity among regions within the cut as a region into which identification information associated with the content is to be inserted.

The identification information associated with the content to be inserted into the complex area may include information for identifying a user who viewed the content as a watermark inserted into the content.

The image is a service cut image that is a unit image provided to the user when the content is provided to the user, and the service cut image may include at least one unit cut among the plurality of unit cuts.

The image is a single episodic image including the plurality of unit cuts, the episodic image is generated by connecting a plurality of the service cut images, and the extracting includes the episodic image including at least one unit cut. However, it may include dividing into a plurality of divided images having a predetermined degree of overlap, and extracting each of the plurality of unit cuts by extracting a unit cut and a part of the unit cut from each of the divided images. .

The extracting may include extracting, as the unit cut, a region having unequal pixel values in a horizontal axis direction of the image.

The extracting of the unit cut may include determining a region having the same pixel values in the horizontal axis direction of the image as a background, and determining a predetermined length in the vertical axis direction of the region having unequal pixel values in the horizontal axis direction of the image. If it is equal to or greater than the value of , extracting a region having unequal pixel values as the unit cut may be included.

In the extracting, the unit cut may be extracted by extracting a region including the object or scene cut from the image using a deep learning model pretrained to detect the object or scene cut from the image.

The determining may include dividing the extracted unit cut into a plurality of tiles, determining a complexity of each tile of the divided tiles, and determining a complexity of each tile within the unit cut based on the determined complexity of each tile. Determining at least one patch having a higher complexity score, wherein each of the patches is composed of a predetermined number of tiles among the tiles, and determining an area of the unit cut corresponding to the determined patch as the complex area. steps may be included.

The determining of the patch may include calculating the complexity score of each patch based on the complexity of each tile in a sliding window method in which a size of each tile is a stride; and and determining at least one patch having a higher calculated complexity score.

The complexity score of each patch composed of a predetermined number of tiles among the tiles may be calculated based on the sum of the complexity of each of the tiles belonging to each patch.

The determining of the patch may include determining whether each patch composed of a predetermined number of tiles among the tiles intersects a speech bubble area included in the unit cut, and ii) a composing area included in the unit cut. The method may further include determining at least one patch having a higher complexity score based on at least one condition of intersecting with and iii) intersecting with a face region included in the unit cut.

Determining the complexity of each tile comprises: i) entropy associated with each tile, ii) edge density associated with each tile, iii) NZR of a discrete cosine transform (DCT) coefficient associated with each tile The complexity of each tile may be determined based on at least two of (non-zero-rate) and iv) variance associated with each tile.

The step of determining the complexity of each tile may include standardizing i) to iv) based on a preset mean and standard deviation, and excluding the maximum and minimum values among the normalized i) to iv), and The step of determining the complexity of each tile based on two values may be included.

In the step of determining the complexity of each tile, a deep learning model pretrained to estimate the complexity of each tile determined using a rule-based decision model as a ground truth is used. The complexity of each tile may be determined.

The rule-based decision model is: i) the entropy associated with each tile, ii) the edge density associated with each tile, iii) the NZR of the discrete cosine transform (DCT) coefficient associated with each tile (non -zero-rate) and iv) the complexity of each tile based on at least two of the variance associated with each tile.

The determining step may include inputting the extracted unit cut to a classification model pretrained to classify an image, obtaining feature maps associated with the unit cut of a plurality of middle layers of the classification model, Visualizing the feature maps, generating attention maps corresponding to the feature maps, and at least one indicating a region having a higher activation degree from the attention map obtained by averaging or summing the attention maps. and determining an area of the unit cut corresponding to a patch of the complex area.

The content is webtoon content, and each of the unit cuts may be a scene cut constituting the webtoon content.

In another aspect, a computer system for determining a complex region from an image of content including a plurality of unit cuts includes at least one processor implemented to execute instructions readable by the computer system, The processor of extracts, from the image, a unit cut included in the image, and determines at least one complex region having a higher complexity among regions within the unit cut as a region into which identification information associated with the content is to be inserted. A computer system is provided.

A re-drawing attack and/or a signal attack ( It is possible to guarantee robustness of identification information against illegal copying and processing of contents such as signal attack.

By inserting identification information, such as a watermark, into a complicated area with high complexity in a scene cut, it may be difficult for an illegal copyer or a viewer to read pirated and processed webtoon content to recognize the inserted identification information.

In order to determine a complex region with a high complexity score within a scene cut, a rule-based method based on analysis of various parameters of an image is used, thereby increasing the sophistication of the complexity region determination.

In addition, in order to determine a complex region with a high complexity score within a scene cut, a method using a deep learning model learned to estimate complexity determined according to a rule-based method or an attention map obtained through a classification model of an image By using a method using , it is possible to increase the speed of complex region determination.

1 illustrates a method of determining a complex region and inserting identification information into an image of content, according to an exemplary embodiment.
2 illustrates a computer system performing a method of determining a complex region for an image of content for embedding identification information, according to one embodiment.
3 is a flowchart illustrating a method of determining a complex region for an image of content for embedding identification information, according to an exemplary embodiment.
4 is a flowchart illustrating a method of extracting a unit cut from an image of content that is an episodic image, according to an example.
5 is a flowchart illustrating a method of extracting a unit cut from an image of content through a rule-based method according to an example.
6 is a flowchart illustrating a method of dividing an image of content into a plurality of tiles, calculating a complexity of each tile, and determining a patch having a higher complexity score, according to an example.
7 is a flowchart illustrating a method of determining a patch having a higher complexity score according to an example.
8 is a flowchart illustrating a method of determining a complexity of each tile of an image of content, according to an example.
9 is a flowchart illustrating a method of determining a complex region of an image of content through an attention map-based method according to an example.
10 illustrates a method of extracting a unit cut from an image of content and determining a complex region for inserting identification information from the unit cut, according to an example.
11 illustrates a method of extracting a unit cut from an image of content that is an episodic image, according to an example.
12 illustrates a method of determining a complexity of each tile of an image of content using a rule-based method, according to an example.
13 illustrates a method of determining a patch corresponding to a complex region of an image of content using a sliding window method according to an example.
14 is a flowchart illustrating a method of determining a complex region of an image of content using an attention map-based method according to an example.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 illustrates a method of determining a complex region and inserting identification information into an image of content, according to an exemplary embodiment.

Referring to FIG. 1, for a method of determining a region for inserting identification information such as a watermark in an image of content, for example, webtoon content, including a plurality of unit cuts (ie, scene cuts), Explain.

Content may include a plurality of unit cuts. Each of these unit cuts is a scene cut constituting a scene of content such as webtoon content, and may have a semantic relationship by being sequentially exposed when content is viewed through a terminal such as a user terminal. Unit cuts of content may be sequentially arranged, and when the unit cuts are sequentially exposed, organically connected stories may be formed. A user viewing the content can enjoy the content by understanding the story of the content.

For example, the content is webtoon content, and may include digital content type cartoons provided over the Internet based on a wired/wireless network. Unit cuts of content may be sequentially provided as the user scrolls the screen of the user terminal, for example. However, the content of the embodiment may include any content configured in the form of an image, not just such webtoon content.

Content may be provided to a user terminal, which is a client, through a content providing server. The content providing server may be a service platform providing webtoon content.

In a detailed description to be described later, a “unit cut” is a “scene cut” representing one scene of content, and may mean a block image (ie, a scene cut image) into which the scene is included.

Content may be provided to a user (ie, a user terminal) as a predetermined unit image, and a basic image provided from a service providing such content may be referred to as a service cut image. This service cut image may be configured to include at least one unit cut, and/or may additionally include a portion of at least one unit cut.

Meanwhile, an image configured (created) by connecting these service cut images may be referred to as an episodic image. The episode image may be a single image including a plurality of unit cuts, and may be configured by connecting service cut images of one episode of content, such as webtoon content, for example.

The image 10 of content in the embodiment shown in FIG. 1 is an image including unit cut(s), and may be such a service cut image or an episode image.

The computer system 100 may extract the unit cut 20 from the image 10, determine at least one complex region 30 having a higher complexity among regions within the unit cut 20, and determine the Identification information associated with content may be inserted into the complex area 30 .

The identification information associated with the content to be inserted into the determined complex area 30 may include information for identifying a user (user terminal and/or user account) who viewed the content as a watermark inserted into the content.

Specifically, the identification information may include, for example, content viewer information for content. Content viewing information may include information about a user who has viewed or purchased (or requested to view or purchase) content. Content viewing information is information for identifying a corresponding user and may include a user ID. In this case, the user ID may be a user identification ID registered when subscribing to a content providing service in order to view content. This user identification ID may be an ID logged in by a user when browsing content. In other words, the identification information may be information for transaction tracking.

Alternatively, the identification information may be configured to include different types of information in addition to the information for transaction tracking described above. For example, the identification information may include copyright holder information for content. By including the copyright owner information in the identification information, the copyright owner can be confirmed by the identification information when a dispute arises in relation to content including the identification information.

Alternatively, the identification information may be configured to provide the content or include information related to other content available from the provider and/or user side.

Since contents such as webtoons are composed of images, illegal copying or processing can be easily performed. Accordingly, the identification information may include information for preventing such illegal copying and processing and identifying an illegal copyer (a leaker or a processor). Identification information such as a watermark may be inserted into an image of content so that a general content viewer does not interfere with viewing the content and an illegal copyer cannot know where the identification information is inserted.

In addition, since illegal copyers often re-edit and share content, identification information needs to be inserted into an image of content to have robustness against a redraw attack or a signal attack by illegal copyers. A redrawing attack may include, for example, an act of re-editing or re-translating a speech bubble area and/or a typeface area included in an image of content, an act of resizing an image, an act of cropping an image (removing blank space, etc.), and the like. Signal attacks include, for example, adding noise to the image of the content, applying a filter, performing blurring processing, converting color, changing resolution, performing denoising processing, image compression processing, and the like. It may include the act of re-editing the image.

In the embodiment, an area in which the color of the image 10 of the content is monotonous, that is, a complex area 30 having a higher complexity is determined instead of an area 5 having a low complexity to be determined as an area for inserting identification information. can If the identification information is inserted into the low-complexity area 5, this identification information can be easily recognized by the viewer of the content due to the noise caused by the identification information, and will be vulnerable to redrawing attacks and signal attacks by pirates. can

The complex area 30 may be determined as an area other than the speech bubble area and the compose area of the unit cut 20 in order to secure the robustness of the identification information against the above-described illegal copyer's redrawing attack. In addition, the complexity region 30 may be determined as a region of high complexity within the unit cut 20 in order to secure the robustness of identification information against the aforementioned illegal copyer's signal attack and to secure the invisibility of identification information.

As such, in the embodiment, rather than the unit cut 20 or a fixed area of the image 10, a complex area 30 having a different complexity within the unit cut 20 is determined, and identification information is provided in the corresponding area 30. By making it inserted, it can be difficult for an illegal copyer to grasp an insertion pattern of identification information or perform an attack to remove identification information.

The computer system 100 extracts the unit cut 20 from the image 10 and determines the complex region 30 within the unit cut 20 with reference to FIGS. 2 to 14 to be described later. Explain in detail.

2 illustrates a computer system performing a method of determining a complex region for an image of content for embedding identification information, according to one embodiment.

The computer system 100 is a device that performs an operation for determining a complex region 30 within the unit cut 20 of the image 10 of content as an area for inserting identification information such as the aforementioned watermark. can

The computer system 100 extracts the unit cut 20 from the image 10 and inserts identification information associated with the content into at least one complex region 30 having a higher complexity among regions within the unit cut 20. area can be determined.

The computer system 100 may be configured to include at least one computing device, and the computing device may be implemented as a terminal or server of a user or administrator who manages content.

As shown, the computer system 100 may include a memory 130, a processor 120, a communication unit 110, and an input/output interface 140.

The memory 130 is a computer-readable recording medium and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Here, the ROM and the non-perishable mass storage device may be separated from the memory 130 and included as a separate permanent storage device. Also, an operating system and at least one program code may be stored in the memory 130 . These software components may be loaded from a computer-readable recording medium separate from the memory 130 . The separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD/CD-ROM drive, and a memory card. In another embodiment, software components may be loaded into the memory 130 through the communication unit 110 rather than a computer-readable recording medium.

The processor 120 may be configured to process commands of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to the processor 120 by the memory 130 or the communication unit 110 . For example, processor 120 may be configured to execute instructions received according to program code loaded into memory 130 .

The communication unit 110 may be a component for the computer system 100 to communicate with other devices (user terminals or other servers). In other words, the communication unit 110 is a hardware module or network such as an antenna, a data bus, a network interface card, a network interface chip, and a networking interface port of the computer system 100 that transmits/receives data and/or information to other devices. It may be a software module such as a device driver or networking program.

The input/output interface 140 may be a means for interfacing with an input device such as a keyboard or mouse and an output device such as a display or speaker.

The processor 120 may manage components of the computer system 100, execute a program or application for extracting the unit cut 20 and determining the complexity region 30, and execute the program Alternatively, operations required for application execution and data processing may be processed. The processor 120 may be at least one processor (such as a CPU or GPU) of the computer system 100 or at least one core within the processor.

Also, in embodiments, computer system 100 and processor 120 may include more components than those shown. Components included in the processor may be part of the aforementioned processor 120 or functions implemented by the processor 120 . In other words, the components included in the processor 120 represent different functions performed by the processor 120 according to a control instruction according to an operating system code or at least one computer program code. can be

A specific method of extracting the unit cut 20 from the image 10 and determining the complex region 30 within the unit cut 20 according to the operation of the processor 120 of the computer system 100 will be described later. It will be described in more detail with reference to FIGS. 2 to 14 .

Since descriptions of the technical features described above with reference to FIG. 1 may be applied as they are to FIG. 2 , duplicate descriptions will be omitted.

In the detailed description to be described later, an operation performed by the computer system 100 or the processor 120 or its components may be described as an operation performed by the computer system 100 for convenience of description.

3 is a flowchart illustrating a method of determining a complex region for an image of content for embedding identification information, according to an exemplary embodiment.

Referring to FIG. 3 , the computer system 100 extracts a unit cut 20 from an image 10 of content, and selects at least one complex region 30 having a higher complexity among regions within the unit cut 20. A method of determining a region in which identification information associated with content is to be inserted will be described in more detail.

As described above, the image 10 of content may include a plurality of unit cuts.

In operation 310 , the computer system 100 may extract a unit cut 20 included in the image 10 from the image 10 . The computer system 100 may extract each unit cut of a plurality of unit cuts from the image 10 of content.

The computer system 100, for example, from the image 10, which is a service cut image, which is a unit image provided to the user (user terminal or user account) when content is provided to the user, at least one service cut image included. Unit cut 20 can be extracted. The service cut image may include at least one unit cut and/or a part of at least one unit cut.

Meanwhile, the image 10 may be a plurality of service cut images, and in this case, the computer system 100 may extract each unit cut of a plurality of unit cuts included in the content from the plurality of service cut images.

Each extracted unit cut 20 may be a target for determining a complex region 30 into which identification information is to be inserted.

In step 320, the computer system 100 inserts identification information associated with the content into at least one complex region 30 having a higher complexity among the regions in the unit cut 20 extracted in step 310. area can be determined. The computer system 100 may determine an area excluding the speech bubble area and the typesetting area of the unit cut 20 from among areas included in the unit cut 20 as the complex area 30, and the area excluding the speech bubble area and the typesetting area. Among them, a region having a higher complexity may be determined as the complex region 30 . The composing area may be an area in which text is inserted in the unit cut 20 . An area of low complexity, such as a background of a scene or a single color area within the unit cut 20 may not be determined as the complex area 30 .

In step 330 , the computer system 100 may insert identification information associated with the content into the determined complex region 30 . Identification information inserted into the complex area is a watermark inserted into the content and may include information for identifying a user who viewed the content. As for the identification information, since the contents described with reference to FIG. 1 can be applied as they are, duplicate descriptions will be omitted.

The computer system 100 may determine a complex area 30 for each of the unit cuts extracted from the image 10 and insert identification information into the determined complex area 30 . Alternatively, the computer system 100 may determine the complex region 30 only for some of the unit cuts, and insert identification information into the determined complex region 30 .

There may be a plurality of complex areas 30, and identification information may be inserted for each of the plurality of complex areas.

In the embodiment, the identification information is provided in the complex area 30 determined by considering the complexity of the image 10, that is, the complexity in the unit cut 20 included in the image 10, rather than a fixed position in the image 10. can be inserted, making it difficult for an illegal copyer of content to recognize identification information or to grasp an insertion pattern of identification information.

In this regard, FIG. 10 illustrates a method of extracting a unit cut from an image of content and determining a complex region for inserting identification information from the unit cut, according to an example.

As shown, the content may be composed of a plurality of service cut images 1000 . The aforementioned image 10 may be each of the plurality of service cut images 1000 or the service cut images 1000 . Each service cut image includes at least one unit cut, but may further include a part of the at least one unit cut.

The computer system 100 may extract unit cuts 1010 - 1 to 1010 - 3 from the plurality of service cut images 1000 . Each of the unit cuts 1010-1 to 1010-3 may include at least one of a character, an object such as a person or an object, a speech bubble area, and a composing area.

The computer system 100 may detect a speech bubble area and a composing area from each of the extracted unit cuts 1010-1 to 1010-3. For example, a speech bubble area 1020-1 in the unit cut 1010-1 and a composing area 1020-2 in the unit cut 1010-3 may be detected.

The computer system 100 may determine a complex region for each unit cut by excluding the detected speech bubble region 1020-1 and composing region 1020-2. As shown, a complex region 1030-1 for unit cut 1010-1, a complex region 1030-2 for unit cut 1010-2, and a complex region 1030-3 for unit cut 1010-3. An area 1030-3 may be determined as an area for inserting identification information. The complex region determined in each unit cut is a region of a predetermined size and may be a region with the highest complexity in the corresponding unit cut.

A more specific method for the computer system 100 to extract the unit cut 20 from the image 10 or a more specific method for determining the complex region 30 having a higher complexity within the unit cut 20 will be described later. It is described in more detail with reference to FIGS. 4 to 9 and 11 to 14 .

The above description of the technical features described above with reference to FIGS. 1 and 2 may be applied to FIGS. 3 and 10 as it is, and thus duplicate descriptions are omitted.

4 is a flowchart illustrating a method of extracting a unit cut from an image of content that is an episodic image, according to an example.

The above-described content image 10 may be a single episodic image including a plurality of unit cuts of the content, and the computer system 100 extracts the unit cut 20 from the episodic image, and unit cut 20 It is possible to determine the complex region 30 from.

In relation to this, FIG. 11 illustrates a method of extracting a unit cut from an image of content that is an episodic image, according to an example.

At step 410 , computer system 100 may compose episodic image 1100 . For example, the computer system 100 may configure (generate) a single episode image 1100 including a plurality of unit cuts of content by connecting the plurality of service cut images 1000 described above. Alternatively, the computer system 100 may extract unit cuts from the already created episode image 1100 without directly composing the episode image 1100 .

In step 420, the computer system 100 may divide the episodic image 1100 into a plurality of divided images including at least one unit cut but having a predetermined degree of overlap. Each of the divided images may include at least one unit cut and/or a portion of the at least one unit cut (ie, a portion of the unit cut), and an overlapping region with an adjacent divided image according to the predetermined degree of overlap is defined. can include The desired degree of redundancy may be suitably set by computer system 100 .

The computer system 100 creates divided images such that the unit cuts included in the episode image 1100 overlap according to a predetermined degree of overlap, so that the unit cuts (ie, each A unit cut may be extracted even from a unit cut including only a part of a service cut image).

In step 430, the computer system 100 extracts a unit cut included in each divided image and a part of the unit cut from each of the divided images in step 420, thereby extracting a plurality of images included in the episode image 1100. Each of the unit cuts may be extracted. At this time, each extracted unit cut may be a target for determining the complex region 30 described above.

As shown in FIG. 11 , in the case where the episode image 1100 includes unit cuts (unit cuts 1, 2, 3...), the episode image 1100 is composed of a plurality of divided images with a predetermined degree of overlap. , unit cuts and parts of the unit cuts can be extracted from each of the divided images, and as parts of the extracted unit cuts are combined, unit cuts (unit cuts 1, 2, 3...) Each of can be extracted.

That is, in the example shown in FIG. 11 , parts of unit cut 1 and unit cut 2 are extracted from the first divided image, part of unit cut 2 and part of unit cut 3 are extracted from the second divided image, and third division By extracting a part of unit cut 3 from the image, all unit cuts (unit cuts 1, 2, 3...) can be extracted. Since the divided image has overlap, the divided cuts from which only part of the image is extracted may be merged into one divided cut.

In this case, the extracted unit cuts (unit cuts 1, 2, 3, ...) may have connectivity (ie, order) that the original episode image 1100 has.

Each extracted unit cut may be each target for determining the complex region 30 as shown in FIG. 10 described above.

Above, the description of the technical features described above with reference to FIGS. 1 to 3 and 10 may be applied to FIGS. 4 and 11 as it is, and thus duplicate descriptions are omitted.

5 is a flowchart illustrating a method of extracting a unit cut from an image of content through a rule-based method according to an example.

Referring to FIG. 5, a method of extracting a unit cut 20 using a rule-based method from an image 10 of content such as the above-described service cut image(s) 1000 or episode image 1000 explain in more detail.

A rule-based method may be a method based on pixel analysis of image 10 .

At step 510 , computer system 100 may determine a background and unit cut 20 of image 10 based on the pixel values of image 10 . The background of the image 10 may be a solid color area within the image 10 that is distinct from the unit cut 20 .

Specifically, in operation 512 , the computer system 100 may determine a region having the same pixel values in the horizontal axis direction of the image 10 as the background of the image 10 . In other words, the computer system 100 may determine, as the background of the image 10, an area representing the same color because pixel values of all horizontal lines of the image 10 are the same.

In operation 514 , the computer system 100 may extract, as the unit cut 20 , a region in which pixel values in the horizontal axis direction of the image 10 are not the same. In other words, the computer system 100 may determine an area represented by multiple colors in the horizontal line as the unit cut 20 because the pixel values of the entire horizontal line of the image 10 are not the same. At this time, the computer system 100 extracts the region having unequal pixel values as the unit cut 20 only when the length in the vertical axis direction of the region having unequal pixel values in the horizontal axis direction is equal to or greater than a predetermined value. can do.

In this way, through the rule-based method, the computer system 100 can determine as the background of the image 10 an area in which all pixels in the horizontal line of the image 10 have the same color, and An area in which pixels in a horizontal line do not have the same color may be determined as the unit cut 20 . In other words, an area from a horizontal line of the image 10 that is not the background of the image 10 to a horizontal line that becomes the background of the image 10 may be determined as the unit cut 20 . However, the computer system 100 extracts the area as the unit cut 20 for determining the complex area 30 when the length in the vertical axis direction is less than a predetermined value, even if the color of the pixels existing in the horizontal line is not the same. may not That is, if the length in the direction of the vertical axis is too short, it is inappropriate to insert identification information, and thus such an area may not be extracted by the unit cut 20 .

In addition, the computer system 100 includes an area in which pixels in a horizontal line do not have the same color (first scene) and an area in which pixels in the next horizontal line do not have the same color (second scene). When the distance in the vertical axis direction between scenes) is less than or equal to a predetermined value, “first scene + second scene” may be extracted as one unit cut 20 . In other words, if the background of the image 10 between scenes (that is, the area where pixels in the horizontal line of the image 10 have the same color) is short, the computer system 100 integrates the scenes to form an integrated scene as a unit cut ( 20) can be extracted.

On the other hand, the computer system 100 from the image 10 of the content such as the service cut image (s) 1000 or the episode image 1000 through a method using a deep learning model rather than the above-described rule-based method. Unit cut 20 can be extracted.

At this time, the computer system 100 extracts a region including the object or scene cut from the image 10 using a deep learning model pretrained to detect the object or scene cut from the image, thereby generating the unit cut 20. can be extracted. An object may be a character, person, or object. Also, the object may be a speech bubble or typesetting. A deep learning model pretrained to detect scene cuts may be trained to detect scene cuts (unit cuts) from webtoon contents using webtoon contents including a plurality of scene cuts as training data.

Computer system 100 may extract unit cut 20 from image 10 using this pretrained deep learning model. The computer system 100 may extract the unit cut 20 more quickly through a method using a deep learning model, compared to a rule-based method.

The description of the technical features described above with reference to FIGS. 1 to 4, 10 and 11 may be applied to FIG. 5 as it is, and thus duplicate descriptions are omitted.

6 is a flowchart illustrating a method of dividing an image of content into a plurality of tiles, calculating a complexity of each tile, and determining a patch having a higher complexity score, according to an example.

Referring to FIG. 6, a method for determining a complex region 30 by calculating the complexity of regions of the unit cut 20 (or image 10) extracted by the computer system 100 will be described in more detail. .

In step 610, the computer system 100 may divide the unit cut 20 extracted from the image 10 into a plurality of tiles. For example, the computer system 100 may divide the unit cut 20 into rectangular or square tiles of a certain size. Tiles may not overlap each other.

At step 620, the computer system 100 may determine the complexity of each tile of the divided tiles. The computer system 100 may calculate the complexity of each tile by analyzing the divided tiles.

In operation 630, the computer system 100 may determine at least one patch having a higher complexity score within the unit cut 20 based on the determined complexity of each tile. The computer system 100 may determine a complexity score for each patch including a predetermined number of tiles among the tiles of the unit cut 20 and determine at least one patch having a higher complexity score. That is, the computer system 100 may determine one patch having the highest complexity score or n patches having two or more higher complexity scores.

In step 640 , the computer system 100 may determine a region of the unit cut 20 corresponding to the patch determined in step 630 as the complex region 30 . The computer system 100 may allow the aforementioned identification information to be inserted into the determined complex region 30 .

In operation 630, in determining at least one patch having a higher complexity score, the computer system 100 may further consider a predetermined priority condition. For example, the computer system 100 may consider the predetermined priority condition in calculating a complexity score for each patch. Alternatively, the computer system 100 may consider the predetermined priority condition in order to select patch(s) corresponding to the complex region 30 from among the determined n patches.

For example, in determining at least one patch having a higher complexity score, the computer system 100 determines that each patch composed of a predetermined number of tiles among the tiles of the unit cut 20, i) unit cut Whether it intersects the speech bubble area included in (20), ii) whether it intersects the composing area included in the unit cut 20, and iii) intersects the face area, character, or other object included in the unit cut 20 At least one patch having a higher complexity score may be determined further based on at least one condition among whether or not the patch is performed.

In relation to i), the computer system 100 lowers the complexity score of the patch when the patch intersects the speech bubble area included in the unit cut 20, or determines the final complexity of the patch among the n patches described above. It may be excluded so that it is not determined as a patch for determining the area 30 .

In relation to ii), the computer system 100 also lowers the complexity score of the patch when the patch intersects the literate area included in the unit cut 20, or the patch is selected as the final n patches. It may be excluded from being determined as a patch for determining the complex region 30 .

In relation to iii), the computer system 100 may also increase the complexity score of the patch when the patch intersects a face region, character, or other object included in the unit cut 20 .

Alternatively, in determining a patch for determining the complex region 30, at least one of the above-described priority conditions may be applied in preference to a complexity score calculated based on the complexity of each tile. For example, when determining a patch for determining the complex region 30, according to i), first patches that do not intersect the speech bubble region are preferentially determined (that is, patches that intersect the speech bubble region are preferentially excluded) , among the second patches after the second patches that do not intersect with the literate area are determined according to ii) among the first patches (that is, among the patches remaining after the patches intersecting the next literate area are excluded), each Patch(s) having a higher complexity score calculated based on the complexity of the tile may be determined as a patch for determining the complexity region 30 .

Alternatively, among the determined n patches, first patches that do not intersect the speech bubble area according to i) are preferentially determined, and among the first patches, the second patch(s) that do not intersect the composing area according to ii) are determined. It may be determined as a patch for determining the complex region 30 .

As such, in the embodiment, the computer system 100 may determine a patch for determining the complex region 30 according to a policy of priority conditions that may be set in various ways.

The description of the technical features described above with reference to FIGS. 1 to 5, 10 and 11 may be applied to FIG. 6 as it is, and thus duplicate descriptions are omitted.

7 is a flowchart illustrating a method of determining a patch having a higher complexity score according to an example.

Referring to FIG. 7 , a method of calculating a complexity score of a patch based on the complexity of each tile and, accordingly, determining a patch for determining the complex region 30 will be described in more detail.

In step 710, the computer system 100 may calculate a complexity score of each patch based on the complexity of each tile in a sliding window manner in which the size of each tile is a stride. there is.

In step 720, the computer system 100 may determine at least one patch having a higher complexity score calculated for each patch. The determined patch may be a patch for determining the complex region 30 .

In this regard, FIG. 13 illustrates a method of determining a patch corresponding to a complex region of an image of content using a sliding window method according to an example. This sliding window method can calculate complexity scores for all patches at high speed.

As shown, the unit cut 1300 may be divided into a plurality of tiles. The computer system 100 may calculate the complexity of each tile 1310 of the divided tiles. The patch 1320 may be composed of a predetermined number of tiles, and for example, four tiles may constitute one patch 1320 as shown. The size of the patch 1320 may be different according to embodiments. The complexity score of the patch 1320 may be calculated based on the sum or average of the respective complexities of the tiles belonging to the patch 1320, and the additive complexity sum or average may be the complexity score of the patch 1320.

The computer system 1320 may calculate the complexity score of each patch in a sliding window manner in the direction shown in FIG. 13 and determine patch(s) having a higher calculated complexity score.

In calculating the complexity score of each patch in a sliding window method, a patch corresponding to the above-described priority conditions (eg, i) or ii)) is selected from patches for determining the complexity region 30. may be excluded.

The description of the technical features described above with reference to FIGS. 1 to 6, 10, and 11 may be applied to FIGS. 7 and 13 as they are, and thus duplicate descriptions are omitted.

Below, a method of calculating the complexity of each tile of the unit cut 1300 will be described in detail with reference to FIGS. 8 and 12 .

First, a method of calculating the complexity of each tile using a rule-based method will be described.

In this regard, FIG. 8 is a flowchart illustrating a method of determining a complexity of each tile of an image of content, according to an example. 12 illustrates a method of determining a complexity of each tile of an image of content using a rule-based method, according to an example.

Computer system 100, in computing the complexity of each tile of unit cut 1300, determines i) the entropy associated with each tile, ii) the edge density associated with each tile, and iii) the discrete value associated with each tile. The complexity of each tile may be determined based on at least one of a non-zero-rate (NZR) of a cosine transform (DCT) coefficient and iv) a variance associated with each tile. Alternatively, the computer system 100 may determine the complexity of each tile based on a combination of at least two of i) to iv).

Regarding i), the entropy associated with each tile can be higher as the colors used within that tile vary. However, such entropy may exhibit a high value even in a region including a gradation.

Regarding ii), the edge density associated with each tile may become higher as the amount of edges present in the tile increases. However, such an edge density may represent an excessively low value in a region including a gradation.

Regarding iii), the DCT-NZR associated with each tile may increase as the number of frequencies used to represent the image of the tile increases. However, this DCT-NZR may show a high value in a region including a simple curve.

Regarding iv), the variance associated with each tile may represent the color distribution within the tile. However, such dispersion may exhibit a high value in a region where colors are simply divided.

As described above, i) to iv) may represent the complexity of each tile, but there is a possibility that the complexity may not be accurately reflected, so a combination of at least two may be used to calculate the tile complexity.

At step 810, computer system 100 may standardize i) to iv) associated with each tile based on a predetermined mean and standard deviation. That is to say, each of i) to iv) can be converted to a standardized value. The preset average and standard deviation may be reference values pre-calculated by analyzing tiles of a plurality of contents (ie, other webtoon contents). In calculating the reference value, tiles of a single color white and a single color black may be excluded from the calculation.

In step 820, the computer system 100 may determine the complexity of each tile based on the middle two values selected by excluding the maximum and minimum values of standardized i) to iv). For example, the complexity of each tile may be calculated based on the average or sum of the middle two values.

As shown in the example of FIG. 12A, the median value of one tile may be determined using the median entropy and edge density, excluding the maximum value of variance and the minimum value of DCT-NZR, and the sum of entropy and edge density or can be calculated as an average. Alternatively, as shown in the example of FIG. 12B, the median value of one tile may be determined using DCT-NZR and variance, which are median values, excluding edge density, which is the maximum value, and entropy, which is the minimum value, and DCT-NZR and variance, which are median values. can be calculated as the sum or average of

Accordingly, by excluding two values that deviate from i) to iv) and calculating the complexity of the tile, the complexity of the tile can be more accurately calculated. Therefore, the complexity score of the patch determined based on the complexity of the tile is also a unit The degree of complexity within the cut 20 can be accurately reflected.

In short, computer system 100 may include a rules-based decision model that determines the complexity of each tile using the parameters i) through iv) described above.

Meanwhile, in the graphs illustrated in FIGS. 12A and 12B , the x-axis may represent a standardized complexity value. In other words, the x-axis may represent a standardized value of parameters i) to iv). The y-axis may represent the number of images (or tiles).

Next, we describe how to determine the complexity of each tile using a deep learning model.

The computer system 100 may determine the complexity of each tile using a deep learning model pretrained to estimate the complexity of each tile determined by the above-described rule-based decision model as a ground truth (GT). . The deep learning model may be trained to set the complexity of each tile determined by the above-described rule-based decision model as a label and estimate it.

As described above, a rule-based decision model can be used to determine at least two of i) the entropy associated with each tile, ii) the edge density associated with each tile, iii) the DCT-NZR associated with each tile, and iv) the variance associated with each tile. It may be configured to determine the complexity of each tile based on.

The deep learning model may be, for example, an EfficientNet-based model.

The computer system 100 determines the complexity of each tile using a deep learning model instead of the above-described rule-based decision model, so that the complexity of each tile can be determined at high speed.

Meanwhile, the unit cut 20 or 1300, which is the object of complexity calculation and determination of the complexity region 30 described above with reference to FIGS. 6 to 8, may be the image 10 itself, depending on the embodiment. In this case, An operation of extracting the unit cut 20 from the image 10 may not be performed. A description similar to that described above may also be applied to this, and redundant description will be omitted.

The description of the technical features described above with reference to FIGS. 1 to 7, 10, 11, and 13 may also be applied to FIGS. 8 and 12 as they are, and thus duplicate descriptions are omitted.

Hereinafter, a method of determining a complex region in the unit cut 20 or image 10 using the attention map will be described in more detail with reference to FIGS. 9 and 14 .

In this regard, FIG. 9 is a flowchart illustrating a method of determining a complex region of an image of content through an attention map-based method according to an example. 14 is a flowchart illustrating a method of determining a complex region of an image of content using an attention map-based method according to an example.

At step 910, the computer system 100 may input the unit cut 20 into a classification model pre-trained to classify the image. Meanwhile, what is input to the classification model may be the image 10 instead of the unit cut 20 . At this time, an operation of extracting the unit cut 20 from the image 10 may not be performed. In other words, steps 910 to 950 may determine the complex region 30 directly from a service cut image or an episodic image. The classification model is a deep learning model for image classification and may be, for example, VGG.

In step 920, the computer system 100 may obtain feature maps associated with unit cuts of a plurality of middle layers of the classification model.

In operation 930, the computer system 100 may generate attention maps corresponding to the feature maps by synthesizing the feature maps. That is to say, computer system 100 may generate attention maps, which visualize the feature maps.

In step 940, the computer system 100 may determine a patch representing a region having a higher activation degree (eg, a higher region displayed brightly) from an attention map obtained by averaging or summing the generated attention maps. . A patch may have any size.

In operation 950 , the computer system 100 may determine a region of the unit cut 10 or image 10 corresponding to the determined patch as the complex region 30 .

In FIG. 14, Original corresponds to the original image 10 or the unit cut 20 and may represent an input to a classification model. Pool 1 to Pool 5 may represent attention maps generated from feature maps of middle layers of the classification model. Since the size of the image of the attention map may decrease from Pool 1 to Pool 5, Pool 1 to Pool 5 may be resized to have the same size.

An attention map combining resized attention maps is shown as Sum. The computer system 100 may determine an area of the unit cut 10 or image 10 corresponding to the most activated area from the sum attention map as the complex area 30 .

The computer system 100 may determine an activated region (patch) using a convolution operation (convolution function), and may also determine a plurality of n patches having higher activation degrees. The aforementioned identification information may be inserted into the complex region 30 determined in this way.

Through the method using the attention map of the embodiment, it is possible to more quickly determine the complex region 30 from the image 10 compared to the above-described rule-based method or method using a deep learning model.

The description of the technical features described above with reference to FIGS. 1 to 8 and 10 to 13 may be applied to FIGS. 9 and 14 as they are, and thus duplicate descriptions are omitted.

The devices described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in the embodiments include a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), and a programmable PLU (programmable logic unit). logic unit), microprocessor, or any other device capable of executing and responding to instructions. The processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. The device can be commanded. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. can be embodied in Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. In this case, the medium may continuously store a program executable by a computer or temporarily store the program for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or combined hardware, but is not limited to a medium directly connected to a certain computer system, and may be distributed on a network. Examples of the medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROM and DVD, magneto-optical media such as floptical disks, and ROM, RAM, flash memory, etc. configured to store program instructions. In addition, examples of other media include recording media or storage media managed by an app store that distributes applications, a site that supplies or distributes various other software, and a server.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (19)

A method for determining a complex region from an image of content including a plurality of unit cuts, performed by a computer system, comprising:
extracting unit cuts included in the image from the image; and
Determining at least one complex region having a higher complexity among regions within the unit cut as a region into which identification information associated with the content is to be inserted.
Including, how to determine the complex area. According to claim 1,
The identification information associated with the content to be inserted into the complex area includes information for identifying a user who has viewed the content as a watermark inserted into the content. According to claim 1,
The image is a service cut image that is a unit image provided to the user when the content is provided to the user,
The method of determining a complex region, wherein the service cut image includes at least one unit cut among the plurality of unit cuts. According to claim 3,
The image is a single episode image including the plurality of unit cuts,
The episode image is generated by connecting a plurality of the service cut images,
The extraction step is
dividing the episode image into a plurality of divided images including at least one unit cut and having a predetermined degree of overlap; and
Extracting each of the plurality of unit cuts by extracting a unit cut and a portion of the unit cut from each of the divided images.
Including, how to determine the complex area. According to claim 1,
The extraction step is
Extracting, as the unit cut, a region in which pixel values in the horizontal axis direction of the image are not the same
Including, how to determine the complex area. According to claim 5,
The step of extracting the unit cut,
determining a region having the same pixel values in a horizontal axis direction of the image as a background; and
extracting, as the unit cut, an area having unequal pixel values when a length in a vertical axis direction of an area having unequal pixel values in the horizontal axis direction of the image is equal to or greater than a predetermined value;
Including, how to determine the complex area. According to claim 1,
The extraction step is
A method for determining a complex region, wherein the unit cut is extracted by extracting a region including the object or scene cut from the image using a deep learning model pretrained to detect the object or scene cut from the image. According to claim 1,
The determining step is
Dividing the extracted unit cut into a plurality of tiles;
determining a complexity of each tile of the divided tiles;
determining at least one patch having a higher complexity score within the unit cut, based on the determined complexity of each tile, wherein each patch is composed of a predetermined number of tiles among the tiles; and
Determining an area of the unit cut corresponding to the determined patch as the complex area
Including, how to determine the complex area. According to claim 8,
The step of determining the patch is,
calculating the complexity score of each patch, based on the complexity of each tile, using a sliding window method in which a size of each tile is a stride; and
Determining at least one patch having a higher calculated complexity score
Including, how to determine the complex area. According to claim 8,
The complexity score of each patch consisting of a predetermined number of tiles among the tiles is calculated based on the sum of the complexity of each of the tiles belonging to each patch. According to claim 8,
The step of determining the patch is,
Each patch composed of a predetermined number of tiles among the tiles, i) whether or not it intersects the speech bubble area included in the unit cut, ii) whether or not it intersects the composing area included in the unit cut, and iii) Determining at least one patch having a higher complexity score, further based on at least one condition of whether the patch intersects a face region included in the unit cut.
Including, how to determine the complex area. According to claim 8,
The step of determining the complexity of each tile,
i) the entropy associated with each tile, ii) the edge density associated with each tile, iii) the non-zero-rate (NZR) of the discrete cosine transform (DCT) coefficient associated with each tile, and iv) the A method for determining complexity regions, wherein the complexity of each tile is determined based on at least two of the variances associated with each tile. According to claim 12,
The step of determining the complexity of each tile,
standardizing the i) to iv) based on a preset mean and standard deviation; and
Determining the complexity of each tile based on the middle two values selected by excluding the maximum value and the minimum value among the normalized i) to iv)
Including, how to determine the complex area. According to claim 8,
The step of determining the complexity of each tile,
A complexity region in which the complexity of each tile is determined using a deep learning model pretrained to estimate the complexity of each tile determined using a rule-based decision model as a ground truth. How to determine. According to claim 14,
The rule-based decision model is: i) the entropy associated with each tile, ii) the edge density associated with each tile, iii) the NZR of the discrete cosine transform (DCT) coefficient associated with each tile (non -zero-rate) and iv) determining the complexity of each tile based on at least two of the variance associated with each tile. According to claim 1,
The determining step is
inputting the extracted unit cuts to a pre-learned classification model to classify images;
obtaining feature maps associated with the unit cut of a plurality of middle layers of the classification model;
generating attention maps corresponding to the feature maps, which visualize the feature maps; and
Determining a region of the unit cut corresponding to at least one patch representing a region having a higher activation degree, as the complex region, from an attention map obtained by averaging or summing the attention maps.
Including, how to determine the complex area. According to claim 1,
The content is webtoon content,
Each of the unit cuts is a scene cut constituting the webtoon content, a method for determining a complex region. A program recorded on a computer readable recording medium for executing the method of any one of claims 1 to 17 in the computer system. A computer system for determining a complex region from an image of content including a plurality of unit cuts,
at least one processor implemented to execute instructions readable by the computer system
including,
The at least one processor,
Extracting unit cuts included in the image from the image;
A computer system that determines at least one complex region having a higher complexity among regions within the unit cut as a region into which identification information associated with the content is to be inserted.

Images