KR20240082137A - Method and apparatus for providing text information including text extracted from content including image - Google Patents

Abstract

Identify content including images uploaded to the content server, extract text from images included in the identified content, and convert text information including the extracted text into text information associated with the content, provided by the administrator of the content or the content. A method of providing text information is provided, including providing text information upon consumer request.

Description

A method and device for providing text information including text extracted from content including an image {METHOD AND APPARATUS FOR PROVIDING TEXT INFORMATION INCLUDING TEXT EXTRACTED FROM CONTENT INCLUDING IMAGE}

The present disclosure relates to a method and device for providing text information including text extracted from content including an image, by extracting text corresponding to the line from content including cuts and lines such as webtoon content, and extracting the corresponding text. Related to a method and device for providing text information including.

Interest in services that provide content containing images online, such as comics, cartoons, or webtoon services, is increasing. Images included in such content include dialogue uttered by characters in the content to progress the story of the content or text to explain the story, and also include text unrelated to the progress of the story. Text that is unrelated to the progress of the story is, for example, text included in sound effects, background decoration, or objects in an image, and unlike dialogue uttered by a character or text to explain the story, it does not affect the progress of the story of the content. .

Text extracted from the content may be provided together with the content as information related to the content, for example, provided to the manager upon request from the manager managing the content, or provided to the consumer upon request from the consumer consuming the content. It can be.

At this time, the text provided upon request is text information related to the content and does not simply represent all text extracted from the content, but is required to be processed to effectively provide information related to the content, such as the story of the content. Additionally, when extracting text from content, it is necessary to exclude text that is unrelated to the story progress from extraction so that only meaningful text is included in the text information.

Korean Patent No. 10-2374280 (registration date March 10, 2022) discloses a system and method for blocking text extracted from images.

The information described above is for illustrative purposes only and may include content that does not form part of the prior art and may not include what the prior art would suggest to a person skilled in the art.

One embodiment identifies content including an image uploaded to a content server, extracts text from the image included in the identified content, and uses text information including the extracted text as text information associated with the content. A method of providing text information can be provided, including providing it at the request of the administrator or the consumer of the content.

In one embodiment, a plurality of cuts included in an image are detected, a cut image including each cut is generated, text is extracted from the cut images corresponding to the plurality of cuts, and text is included in each cut image. A method of detecting a dialogue region, extracting text included in the dialogue region for each detected dialogue region using OCR (Optical Haracter Recognition), and generating text information based on the extracted text can be provided.

In one aspect, a method of providing text information associated with content, performed by a computer system, comprising: identifying content that includes an image uploaded to a content server; extracting text from the image included in the content; A method of providing text information is provided, including the step of extracting and providing text information including the extracted text as text information associated with the content.

The image includes a plurality of cuts of the content in an order and text including a line of the content, the extracted text is the line extracted from the text included in the image, and the text information includes the line. It may include each line of the plurality of lines included, and order information of each line.

Extracting the text may include detecting the plurality of cuts in the image; generating each cut image including each of the plurality of cuts; and extracting text from cut images corresponding to the plurality of cuts.

The plurality of cuts may be included in the image in a vertically scrolling order, and each cut image may be configured to further include a blank area of a predetermined size above and below each cut.

Extracting text from the cut images may include detecting a dialogue area including the dialogue for each cut image; Extracting text included in the dialogue region for each detected dialogue region using OCR (Optical Haracter Recognition); and generating the text information based on text extracted for each detected dialogue area, wherein the dialogue area includes a speech bubble included in the image, a monologue or narration by a speaker or character of the content. , or an area containing text explaining the content, and the text information may include information about which dialogue area and which cut the text extracted for each detected dialogue area was extracted as the order information. .

The order information may further include row information within the corresponding dialogue region of the text extracted for each detected dialogue region.

Each of the plurality of cuts is assigned a first order number, which is higher as it is on the upper side in the vertical direction within the image, and is higher as it is on the left or right side at the same position in the vertical direction, and is assigned a first order number in each cut image. Each of the detected metabolic regions is assigned a second order number that is ahead as it is on the upper side in the vertical direction within each cut image, and is assigned a second order number in which the order number is ahead as it is on the left or right side at the same position in the up and down direction, and the detected To each line of text extracted for each dialogue area, a third order number, which is earlier as it is located at the top in the vertical direction, is assigned as the line information, and the text information is the order information for the text extracted from each of the dialogue areas. It may include the first sequence, the second sequence, and the third sequence.

The step of extracting text from the cut images may include, if the detected dialogue region is a first region including the monologue or narration or a second region including the explanatory text, in the first region or the second region. It further includes generating a corresponding virtual speech balloon, wherein the order information is based on the order in the image of speech balloons corresponding to the detected dialogue area and speech balloons including the virtual speech balloon, wherein the detection The text extracted for each dialogue area may include information about which of the speech bubbles it was extracted from.

Extracting text from the cut images may include generating one integrated dialogue region image by integrating the dialogue regions detected from the cut images; And with respect to the dialogue regions included in the integrated dialogue region image, the step may include extracting text included in the dialogue region for each dialogue region using OCR (Optical Haracter Recognition).

Detecting the dialogue area may include detecting areas containing text in each cut image; Among the areas, identifying a non-dialogue area, which is an area including text corresponding to the background of each cut, text representing sound effects of the content, and text determined to be unrelated to the story of the content; And it may include detecting regions excluding the non-metabolic region among the regions as metabolic regions containing metabolism.

In the providing step, the text information is provided to the administrator terminal according to a request from an administrator terminal that manages the content, and the method of providing the text information allows the text information to be inspected for the administrator terminal. It further includes providing a function to enable the inspection, wherein the function to enable the inspection includes a first function to enable the text information to be edited, a second function to enable the text information to be downloaded, and the text information to be updated. It may include at least one of the third functions for setting availability.

The function enabling inspection includes the first function, and the step of providing the function enabling inspection includes, at the manager terminal, a first cut selected by the manager among the plurality of cuts and the selected first cut. displaying the text information including the dialogue extracted from the cut; providing a first user interface for editing the displayed text information; and providing a second user interface that enables switching from the first cut to a second cut, which is another cut among the plurality of cuts.

The providing step may provide audio information corresponding to the text information to the consumer terminal according to a request from the consumer terminal consuming the content.

The providing step includes: calling the text information associated with the content when viewing the content is requested from the consumer terminal; Recognizing a cut being viewed by the consumer terminal among the plurality of cuts; It may include outputting audio information corresponding to a portion of the text information corresponding to the recognized cut from the consumer terminal.

The method of providing the text information includes monitoring whether the content on the content server is updated or deleted; When an update to the content is identified, extracting text from the image included in the updated content; and when deletion of the content is identified, deleting the text information associated with the content.

The method of providing the text information includes determining a speaker of the content who uttered the text extracted for each detected dialogue region - the speaker is expressed in association with a speech bubble corresponding to the detected dialogue region in the image. determined based on at least one of a speaker image and a color or shape of a speech bubble corresponding to the detected dialogue region, wherein the text information generated based on the text extracted for each detected dialogue region is determined by the speaker Additional information may be included.

In another aspect, a computer system providing text information associated with content, comprising at least one processor configured to execute instructions readable by the computer system, the at least one processor configured to: upload to a content server; A computer system is provided that identifies content including an image, extracts text from the image included in the content, and provides text information including the extracted text as text information associated with the content. .

Text information that includes only text corresponding to the dialogue of the content, excluding text corresponding to the background of each cut included in the content, text representing the sound effects of the content, and text judged to be unrelated to the story of the content, This is text information related to content and can be provided to managers or consumers of content.

Administrators can receive text information related to the content and inspect and edit the text extracted from each cut, and consumers of the content can be provided with audio information corresponding to the text information related to the viewed content when viewing the content. .

The dialogue area can be detected from each cut image corresponding to each cut of the content, and the text can be extracted for each detected dialogue area. It is possible to determine from which dialogue area and from which cut the extracted text was extracted, and within the dialogue area. The order information indicating which row it belongs to can be processed into text information related to the content along with the extracted text.

Figure 1 shows a method of extracting text from an image of content and providing text information including the extracted text, according to an embodiment.
2 illustrates a computer system, a consumer terminal, and a content server performing a method for providing text information, according to one embodiment.
Figure 3 is a flowchart showing a method of extracting text from an image of content and providing text information including the extracted text, according to an embodiment.
Figure 4 is a flowchart illustrating a method for extracting text from an image of content, according to an example.
FIG. 5 is a flowchart illustrating a method of generating an integrated dialogue region image by integrating dialogue regions extracted from cut image(s) and extracting text from the integrated dialogue region image, according to an example.
FIG. 6 is a flowchart illustrating a method of generating text information including extracted text and order information associated with the extracted text, according to one example.
Figure 7 is a flowchart showing a method for detecting a metabolic region from a cut image, according to an example.
FIG. 8 is a flowchart illustrating a method of providing text information associated with content to a consumer terminal of a consumer consuming content, according to an example.
FIG. 9 is a flowchart illustrating a method of generating text information or deleting text information by re-extracting text according to update or deletion of content, according to an example.
FIG. 10A shows a cut image corresponding to a cut of content, according to an example.
Figure 10b shows a method of extracting a cut from an image of content, according to an example.
FIG. 11A shows a method for detecting a metabolic region from a cut image, according to an example.
FIG. 11B shows a method of detecting text from a dialogue area that is a speech bubble (or a virtual speech balloon), according to an example.
Figure 12 shows a method of determining the order of cuts of content and dialogue regions included in each cut, according to an example.
Figure 13 shows a method of providing text information, according to an example.
Figures 14 and 15 show a method of viewing and reviewing text information at an administrator terminal, according to an example.
Figure 16 shows a method of providing audio information corresponding to text information to a consumer terminal, according to an example.
Figure 17 shows a method of extracting text from an image of content that is webtoon content and providing text information including the extracted text, according to an example.
Figure 18 shows a method of determining the speaker or character who uttered the text included in the dialogue area, according to an example.
Figure 19 shows an integrated metabolic region image, according to one example.

Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

Figure 1 shows a method of extracting text from an image of content and providing text information including the extracted text, according to an embodiment.

Referring to FIG. 1, a method of extracting text from content including an image 10 and generating and providing text information 50 including the extracted text will be described.

The content includes an image 10 and may be, for example, webtoon content. Webtoon content may include comics in the form of digital content provided through the Internet based on wired/wireless networks.

Such webtoon content can be read by consumers (or readers) by scrolling the webtoon content. Webtoon content includes multiple cuts, and readers can view webtoon content by sequentially checking multiple cuts through scrolling.

The content described in the embodiment may represent one episode of a work composed of a plurality of episodes. For example, content may represent a specific episode of a webtoon work.

The depicted image 10 may be at least part of the content. This image 10 may include a plurality of cuts 20-1 to 20-N of content having an order and text including dialogue of the content. Dialogue may include at least one word uttered by a speaker (or character) included in the content. Dialogue may include a monologue or narration by the speaker or character of the content. Alternatively, the dialogue may include at least one word that describes the story of the content, rather than a word uttered by a specific speaker or character of the content. Words that describe the story of such content may be uttered by the author or narrator of the content. The dialogue included in the image 10 may exclude text unrelated to the progress of the story from among the text included in the image 10.

A computer system (e.g., a computer system 100, which will be described later with reference to FIG. 2) can extract text included in the image from the image 10 of the content and generate text information 50 including the text. there is.

The text extracted from the image 10 may be dialogue extracted from the text included in the image 10. As a result, the extracted text can consist of only the text necessary to explain the story of the content. The computer system can detect dialogue regions 12, 22, and 24 containing dialogue from the image 10 and extract text for each dialogue region. The computer system can detect a dialogue region for each of the cuts 20-1 to 20-N and extract text for each dialogue region.

As shown, text information 50 generated including the extracted text includes information indicating which cut the extracted text was extracted from (e.g., cut N, N is an integer), and which dialogue region it was extracted from. It may include information indicating the dialogue area K (e.g., dialogue area K, K is an integer), and information indicating which row of the extracted text line is (e.g., [R], R is an integer). In this way, the text information 50 may include extracted text and order information associated with the extracted text. In other words, the text information 50 may be composed of each line of a plurality of lines included in the dialogue of the content, and order information of each line.

The text information 50 is configured to include meaningful text explaining the story of the content, and may be provided to an administrator who manages the content. The manager can utilize the text information 50 when servicing content to consumers. For example, upon request, a consumer of content may be provided with audio information corresponding to the text information 50 when viewing the content.

A more specific method of the computer system extracting text from the image 10 of the content and generating and providing text information 50 including the extracted text will be described in more detail with reference to FIGS. 2 to 19 to be described later. .

2 illustrates a computer system, a consumer terminal, and a content server performing a method for providing text information, according to one embodiment.

Computer system 100 may be a computing device that performs tasks necessary to perform the method of providing text information of the embodiment.

Computer system 100 may be configured to include at least one computing device. The computer system 100 can detect the dialogue area (12, 22, 24, etc.) from the image 10 included in the content, extract the text included in the dialogue area (12, 22, 24, etc.), and extract Text information 50 including text can be generated, and the generated text information 50 can be provided to an administrator terminal (not shown) that manages the content or a consumer terminal 160 that consumes the content.

The computer system 100 may be the manager terminal described above, or another computer device or server that communicates with the manager terminal. The administrator terminal may provide tools for viewing and inspecting the provided text information 50.

Meanwhile, the computer system 100 may identify content uploaded to the content server 150 and perform the above-described text extraction and provision of text information 50.

The content server 150 is a server where content is managed. Content can be uploaded to the content server 150, and the uploaded content can be updated or deleted. The content server 150 may be a service platform that provides content, such as webtoon content, or may be a part of a service platform.

Below, the more detailed configuration of the computer system 100 will be described in more detail.

As shown, the computer system 100 may include a memory 130, a processor 120, a communication unit 110, and an input/output interface 140.

The memory 130 is a computer-readable recording medium and may include a non-permanent mass storage device such as random access memory (RAM), read only memory (ROM), and a disk drive. Here, the ROM and the non-perishable mass recording device may be separated from the memory 130 and included as a separate permanent storage device. Additionally, an operating system and at least one program code may be stored in the memory 130. These software components may be loaded from a computer-readable recording medium separate from the memory 130. Such separate computer-readable recording media may include computer-readable recording media such as floppy drives, disks, tapes, DVD/CD-ROM drives, and memory cards. In another embodiment, software components may be loaded into the memory 130 through the communication unit 110 rather than a computer-readable recording medium.

The processor 120 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Commands may be provided to the processor 120 by the memory 130 or the communication unit 110. For example, the processor 120 may be configured to execute instructions received according to program code loaded into the memory 130.

The communication unit 110 may be a component that allows the computer system 100 to communicate with other devices (such as user terminals or other servers). That is, the communication unit 110 is a hardware module or network, such as an antenna, a data bus, a network interface card, a network interface chip, and a networking interface port, of the computer system 100 that transmits/receives data and/or information to and from other devices. It may be a software module such as a device driver or networking program.

The input/output interface 140 may be a means for interfacing with an input device such as a keyboard or mouse and an output device such as a display or speaker.

The processor 120 may manage the components of the computer system 100, detecting the dialogue areas 12, 22, 24, etc. from the image 10 of the above-described content, extracting text for each dialogue area, and extracting the text. A program or application for generating and providing information 50 can be executed, and operations necessary for executing the program or application and processing data can be processed. The processor 120 may be at least one processor (such as a CPU or GPU) of the computer system 100 or at least one core within the processor.

Additionally, in embodiments, computer system 100 and processor 120 may include more components than those shown. For example, the processor 120 may include components that train a transformation model and perform functions for performing the image transformation method of the embodiment using the learned transformation model. These components of the processor 120 may be part of the processor 120 or may be functions implemented by the processor 120. The components included in the processor 120 may be expressions of different functions performed by the processor 120 according to control instructions according to the code of the operating system or the code of at least one computer program. .

The illustrated consumer terminal 160 may be a user terminal of a consumer viewing content. The user terminal is a smart device such as a smart phone, a personal computer (PC), a laptop computer, a laptop computer, a tablet, an Internet of Things device, or a wearable computer ( wearable computer), etc. In addition, the consumer terminal 160 may be any electronic device (for example, an e-book reader, etc.) capable of viewing content that is webtoon content.

As described above, the content server 150 may be a service platform that provides content or may be part of a service platform. Depending on the embodiment, the content server 150 may include the computer system 100 or the computer system 100 may include the content server 150.

The consumer terminal 160 and the content server 150 are computer systems and may include similar components to the computer system 100, so redundant descriptions will be omitted.

A more specific method of providing text information 50 associated with content through the computer system 100 and the operations of the computer system 100, consumer terminal 160, and content server 150 will be described in FIGS. 3 to 3 below. This is explained in more detail with reference to 19.

In the detailed description to be described later, operations performed by the computer system 100 or the processor 120 or their components may be described as operations performed by the computer system 100 for convenience of explanation.

Figure 3 is a flowchart showing a method of extracting text from an image of content and providing text information including the extracted text, according to an embodiment.

At step 310, computer system 100 may identify content including image 10 that is a target for generating text information 50. For example, computer system 100 may identify content that includes image 10 uploaded to content server 150 . The computer system 100 may periodically (e.g., 1 hour) monitor the content server 150 and determine whether content including the image 10 is uploaded to the content server 150 or a previously uploaded image 10 is uploaded to the content server 150. When an update of the included content is identified, the corresponding content can be identified as a target for generating text information 50. Alternatively, the computer system 100 may be notified that content including the image 10 has been uploaded (or updated) from the content server 150, and the computer system 100 may be notified that the content including the image 10 has been uploaded (or updated) accordingly. Content can also be identified.

At step 320, computer system 100 may extract text from image 10 included in the content. For example, computer system 100 may extract text from image 10 using optical haracter recognition (OCR). As described above, the image 10 is at least part of the webtoon content and may include a plurality of cuts of the content in an order and text including dialogue of the content. At this time, the computer system 100 may extract dialogue from text included in the image. Computer system 100 may extract cuts and dialogue regions from image 10 and extract dialogue from image 10 using a learning model pre-trained to extract text from the dialogue regions.

The computer system 100 may generate text information 50 including the extracted text. For example, the computer system 100 may generate text information 50 including the extracted text by processing it to include order information. The order information included in the text information 50 is, for example, information indicating which cut the extracted text was extracted from (e.g., cut N, N is an integer), information indicating which dialogue area it was extracted from ( For example, it may include a dialogue area K (K is an integer), and information indicating which row of the extracted text line is (for example, [R], R is an integer).

A more detailed method of extracting text from the image 10 will be described in more detail with reference to FIGS. 4 to 7 and FIGS. 10 and 11, which will be described later.

At step 330, computer system 100 may provide text information 50, including the text extracted by step 320, as text information 50 associated with content.

For example, the computer system 100 may provide text information 50 to the administrator terminal according to a request from the administrator terminal that manages content (step 332). The administrator terminal may provide tools for viewing and inspecting the provided text information 50.

In relation to this, Figures 14 and 15 show a method of viewing and reviewing text information at an administrator terminal, according to an example.

14 and 15 show a screen of an administrator terminal displaying tools for viewing and inspecting text information 50.

The computer system 100 may provide a function that enables inspection of text information 50 for an administrator terminal.

In FIG. 14, a list of text information 50 is shown in a tool for viewing and inspecting text information 50. As shown, text information (e.g., ID, content name, number, episode name, content posting date, text information modification date, text information modifier, etc.) is displayed through the first UI 1410. 50) can be sorted. When one piece of text information 1420 is selected from the list, the administrator terminal can be switched to a screen for inspecting the text information 1420. For example, the manager terminal can be switched to a screen like the one shown in FIG. 15.

The tool for viewing and inspecting the text information 50 is a function that enables inspection of the text information 50. The first function 1430 allows editing the text information 50, and downloads the text information 50. It may include at least one of a second function 1440 for enabling and a third function 1450 for setting whether the text information 50 can be updated.

For example, when the first function 1430 is selected, the manager terminal can be switched to a screen for inspecting the corresponding text information 50. For example, the manager terminal may be switched to a screen like the one shown in FIG. 15.

Through the second function 1440, text information 50 can be downloaded to the administrator terminal as, for example, an Excel file.

When the third function 1450 is turned on, updating the text information 50 may be prohibited. When the third function 1450 is turned on, text information may not be changed or deleted even if the content is updated or deleted. The third function 1450 may be OFF by default.

Below, with reference to FIG. 15, the method of inspecting the text information 50 will be described in more detail. In Figure 15, text information 1430 is selected from the text information list.

As shown, the administrator terminal can display the work name 1510 of the content and the episode name 1520 of the content. When button 1530 is selected, the episode list of the corresponding work may be called. The administrator can inspect text information 50 of other episodes of the same work through the episode list.

On the inspection screen, a cut image area 1540 may be displayed. The cut image area 1540 may include text information 1590 including a selected cut of content and text extracted from the cut, and a cut number ('0'). Text information 1590 may be configured to be editable by an administrator. Accordingly, the text information 1590 can correct text incorrectly extracted by the computer system 100. On the inspection screen, the remote control UI may be further displayed. The remote control UI includes UI (1550) for moving to another cut (the administrator can directly enter the cut number), UI (1560) for reflecting modified text information (1590), and modified text information (1590). It may include a UI 1570 for reflecting the service more quickly (e.g., with the highest priority) and a UI 1580 for moving to the episode list.

The functions and UI of the tool for viewing and inspecting the text information 50 described above may be provided to the administrator terminal under control by the computer system 100.

In other words, the computer system 100 is provided at the administrator terminal, text information including a first cut ('0') selected by the administrator among a plurality of cuts of content and a line extracted from the first cut ('0') ( 1590) can be displayed. At this time, a user interface for editing the displayed text information 1590 may be provided on the administrator terminal. Additionally, the computer system 100 may further provide a user interface 150 that enables switching from the first cut ('0') to the second cut, which is another cut among the plurality of cuts.

As such, in the embodiment, the text information 50 generated and provided by the computer system 100 can be inspected by the administrator using a tool through the administrator terminal, and the inspected text information 50 is used to provide content. can be reflected in services for

Additionally, the computer system 100 may provide text information 50 to the consumer terminal 160 in response to a request from the consumer terminal 160 consuming content (step 334).

For example, the computer system 100 may provide audio information corresponding to the text information 50 to the consumer terminal 160 according to a request from the consumer terminal 160 consuming content. The audio information may be text included in the text information 50 converted into audio. For example, text included in the text information 50 may be converted into audio using TTS (Text to Speech) technology.

In relation to this, FIG. 8 is a flowchart illustrating a method of providing text information associated with content to a consumer terminal of a consumer consuming content, according to an example.

In an embodiment, the consumer terminal 160 that reads content may receive audio information obtained by converting text extracted from the image 10 of the content into audio based on text information. Provision of audio information to the consumer terminal 160 may be provision of a reading service for lines of content.

In step 810, the computer system 100 may call text information 50 associated with the content as viewing of the content is requested from the consumer terminal 160. For example, the consumer terminal 160 can view content through a 'webtoon application', a dedicated application for viewing webtoon content, and accordingly, the computer system 100 (or content server 150) provides text information. (50) can be called.

In step 820, the computer system 100 (or content server 150) may recognize a cut that the consumer terminal is viewing among a plurality of cuts of content. For example, the computer system 100 (or content server 150) recognizes a cut of content displayed on the screen of the consumer terminal 160, or cuts the content displayed on the screen of the consumer terminal 160. When a cut is selected (or touched) by the user, it can be recognized. Alternatively, when a cut of content is displayed at a specific location (eg, central area) on the screen of the consumer terminal 160, the computer system 100 (or content server 150) may recognize the cut.

In step 830, the computer system 100 (or content server 150) may output audio information corresponding to a portion of the text information 50 corresponding to the recognized cut from the consumer terminal 160. . In other words, the consumer terminal 160 can output audio that reads lines corresponding to the content cut being displayed.

If the corresponding cut is no longer displayed on the consumer terminal 160 or deviates from a specific location (eg, the center area) by a certain amount, audio output may be stopped.

Meanwhile, depending on the embodiment, the 'reader function' may be selected and executed first in the consumer terminal 160, and accordingly, the above-described steps 820 and 830 may be performed. Cut transition in the consumer terminal 160 may be performed automatically.

Relatedly, Figure 16 illustrates a method of providing audio information corresponding to text information to a consumer terminal, according to an example.

In Figure 16, when the first cut of content is recognized (or when the 'reader function' is executed and the first cut is displayed on the consumer terminal 160), audio information output to the consumer terminal 160 is displayed. indicates. As shown, prior to reading the text extracted from the cut, there is first a guidance phrase (“Please use AI to...”; “The reading of the lines will now begin.”), the “title of the work” and the “title” of the work. The “episode name” may be output as a voice first. After audio information corresponding to the text in the recognized cut is output, a guidance phrase such as “Please move to the next cut” may be output as a voice.

As described with reference to FIGS. 14 and 15, text information 50 is provided to the administrator terminal and can be inspected for provision of services. The reviewed text information 50 may be served to the consumer terminal 160 for viewing content as described above with reference to FIGS. 8 and 16 .

The description of the technical features described above with reference to FIGS. 1 and 2 can also be applied to FIGS. 3, 8, and 14 to 16, so overlapping descriptions will be omitted.

Figure 4 is a flowchart illustrating a method for extracting text from an image of content, according to an example.

Computer system 100 may extract cuts and dialogue regions from image 10 and extract dialogue from image 10 using a learning model pre-trained to extract text from the dialogue regions.

The learning model is an artificial intelligence (AI) model, and may be, for example, an artificial neural network or deep learning-based model. The learning model may be pre-trained to extract cuts from images 10 of content such as webtoon content, and may also be pre-trained to extract text corresponding to dialogue among the text included in each cut from each cut. there is. For example, the learning model may be pre-trained to detect a dialogue region containing dialogue from each cut and extract text from the detected dialogue region. For example, steps to be described later with reference to FIGS. 4 to 7 may be performed using this learning model.

At step 410, computer system 100 may detect a plurality of cuts in image 10 of content.

In step 420, the computer system 100 may generate each cut image including each cut of the plurality of cuts. Computer system 100 can detect cuts from image 10 using a pre-trained learning model and generate a cut image configured to include each cut. The learning model may be trained to generate a cut image from the image 10. For example, in the case of the image 10 of webtoon content, cuts may be divided into square areas, etc., and the computer system 100 can detect a plurality of cuts of the content by recognizing these divided cuts.

Relatedly, FIG. 10A shows a cut image corresponding to a cut of content, according to an example.

In the illustrated example, a plurality of cuts included in the content image 10 may be included in the image 10 in a scrolling order in the vertical direction. At this time, each cut image including each cut may be configured to further include a blank area 1010 of a predetermined size above and below each cut.

As shown, the cut image 1000-1 may include a cut area 1020 corresponding to a cut and a blank area 1010 around the cut area 1020. Depending on the embodiment, the blank area 1010 may exist not only above and below the cut area 1020 but also on the left and right sides.

The computer system 100 generates the cut image 1000-1 to include a blank area 1010 around the cut area 1020, so that the dialogue area (or speech bubble) located across the border of each cut is displayed in the cut image. It can be sufficiently included in (1000-1).

In other words, the computer system 100 detects a plurality of cuts from the image 10 and divides or crops the image 10 to generate a cut image to include each cut, such as a blank area between cuts ( 1010) can be included in the cut image, and thus, it is possible to prevent the dialogue area or speech bubble spanning the cut line between cuts from being omitted from the cut image.

For example, the blank area 1010 included in the cut image 1000-1 may have a certain size. Alternatively, the first cut image may be created by cutting the middle of the cut line of the first cut and the cut line of the second cut, which is the next cut, and the middle of the cut line of the first cut and the cut line of the 0th cut, which is the previous cut. It may be possible. At this time, the cut image may have different sizes depending on the gap between cuts.

Meanwhile, depending on the content, the cut may not be divided into square areas, etc. Relatedly, Figure 10b shows a method of extracting a cut from an image of content, according to an example.

As shown in FIG. 10B, cuts are not distinguished by a visually recognizable cut line, but may be distinguished by a gradient, a solid background, or other background transition effects. The above-described learning model can be configured to detect a cut from the image 10 and generate a cut image even in this case. For example, the computer system 100 may determine the middle of the gradient area between cuts as the boundary of the cut image. Alternatively, the computer system 100 may determine the boundary of the cut image based on the gradient or solid color present in the image 10 over a certain length. For example, the computer system 100 may determine the boundary of the cut image to be in the middle when a gradient or a single color exists in the image 10 over a certain length. Meanwhile, gradients or solid colors over a certain length can be treated similarly to the blank area 1010 described above.

The computer system 100 may assign an order to the extracted cut or a cut image including the cut according to the order of the cut. For example, the computer system 100 may assign a sequence number to the extracted cut or a cut image including the cut. This order or number can be used as order information included in the text information 50.

Meanwhile, the computer system 100 may extract an area containing only a speech bubble or dialogue area as a cut, depending on the configuration of the content image 10. For example, there may be a case where only a speech bubble or dialogue area exists in a specific area, and a certain area around the area (top and bottom) has a gradient or a solid color. In this case, the computer system 100 may display only a speech bubble or dialogue area. The area can be extracted as a cut. These cuts do not include cut lines. The computer system 100 may generate a virtual cut line when an area containing only a speech bubble or dialogue area is extracted as a cut. Cuts including these virtual cut lines may be assigned an order like other cuts. In other words, a cut or cut image including a virtual cut line may also be assigned a sequence number like a cut or cut image including an actual cut line.

In step 430, the computer system 100 may extract text from cut images corresponding to a plurality of cuts. The computer system 100 may extract text from cut images using a pre-trained learning model. For example, the learning model may be trained to detect a dialogue area containing dialogue from a cut image, and the computer system 100 may extract text using OCR from the dialogue area detected using this learning model. You can.

Below, with reference to steps 432 to 436, a method for detecting a dialogue region from a cut image and extracting text from the dialogue region will be described in more detail.

At step 432, the computer system 100 may detect a dialogue region containing dialogue in the content for each cut image. The computer system 100 may detect the metabolic region from each cut image using a pre-trained learning model.

The dialogue area is a speech bubble included in the image 10 of the content, an area containing a monologue or narration by the speaker or character of the content (i.e., an area not separated by text or speech bubbles uttered by the speaker or character), or the content It may be an area containing explanatory text (an area not separated by speech bubbles, for example, text uttered by the writer). Here, the speaker or character may be a character in the content. The learning model is pre-trained to detect dialogue areas from such cut images, and may be pre-trained using images including cuts included in multiple webtoon contents.

Text included in the dialogue area can be considered text necessary to explain the story of the content.

Relatedly, FIG. 11A shows a method for detecting a metabolic region from a cut image, according to an example.

In Figure 11a, a cut image 1100 is shown. The computer system 100 may detect the metabolic region 1110 from the cut image 1100. The metabolism area 1110 may include a first dialogue area 1110-1 and a third dialogue area 1110-3, which represent speech bubbles, and a second metabolism area 1110-2, which is a dialogue area not separated by speech bubbles. You can. The second dialogue area 1110-2 may be an area containing a monologue or narration by the speaker or character of the above-described content, or an area containing text explaining the content.

The computer system 100 uses a pre-trained learning model to create metabolic regions including a first metabolic region 1110-1 and a third metabolic region 1110-3 as metabolic regions representing speech bubbles that exist in various forms. It can be detected. Additionally, the computer system 100 may include a second dialogue area 1110-2, which is a dialogue area containing dialogue that is not distinguished by speech bubbles using a pre-trained learning model. The learning model creates a second dialogue area (1110- It may be trained to detect 2).

Meanwhile, the computer system 100 may identify a non-dialogue area 1120 that does not contain dialogue among the text included in the cut image 1100. The non-communicating area 1120 is, for example, an area containing text included in an object (character, etc.) in the cut image 1100 (e.g., the first non-communicating area 1120-1, the third non-communicating area 1120- 3) and a fourth non-metabolic region (1120-4)). Additionally, the non-communicating area 1120 may include, for example, an area containing a pattern or text included in the background of the cut (for example, the second non-communicating area 1120-2). For example, an area containing text corresponding to a sound effect included in the cut image 1100 may be identified as a non-dialogue area 1120. Sound effects may be a text representation of sounds, effects, etc. generated according to the movement of objects included in the content.

In an embodiment, for example, the text included in an object such as a flyer is text that is not related to the story delivery of the content, and the area containing the text may be identified as the first non-metabolic area 1120-1. . In addition, the text included in the character's clothes is also text that is not related to the story delivery of the content, and the areas containing the text are the third non-dialogue area (1120-3) and the fourth non-dialogue area (1120-4). ) can be identified. In addition, since the pattern or text expressed in the background of the cut is also text that is not related to the story delivery of the content, the area containing the pattern or text may be identified as the second non-dialogue area 1120-2.

In an embodiment, the computer system 100 may exclude the non-dialogue area 1120 and detect only the dialogue areas 1110 among areas containing text.

Relatedly, FIG. 7 is a flowchart showing a method of detecting a metabolic region from a cut image, according to an example.

In detecting the dialogue area from the cut image 1100, the computer system 100 may detect areas containing text in each cut image 1100 in step 710. The detected regions may include a metabolic region 1110 and a non-metabolic region 1120.

In step 720, the computer system 100 selects, among the detected areas, text corresponding to the background of each cut (i.e., a pattern or text expressed in the background), text representing the sound effect of the content, and a story of the content. is a non-communicable area 1120, which is an area containing text determined to be irrelevant (e.g., the same text included in the above-described first to fourth non-communicable areas 1120-1 to 1120-4) can be identified.

In step 730, the computer system 100 may detect regions excluding the identified non-metabolic region 1120 among the regions as metabolic regions 1110 containing metabolism. Meanwhile, the above-described learning model for identifying the metabolic region 1110 may be trained in advance to distinguish the metabolic region 1110 and the non-metabolic region 1120.

Meanwhile, depending on the embodiment, unlike what is described in steps 720 and 730, the computer system 100 does not identify the non-metabolic region 1120 among the regions detected in step 710, but directly identifies the metabolic region ( 1110) can also be detected.

Below, we explain in more detail how to extract text from the detected dialogue region.

In step 434, the computer system 100 may extract text included in the dialogue region for each detected dialogue region using OCR. That is, computer system 100 can extract text for each speech region. As the text contained in the dialogue area is extracted for each dialogue area, text that is not included in a dialogue area but is included in the same line of the cut image (or text contained in another dialogue area located on the same line) is incorrectly detected through OCR. Recognition can be prevented. The lines included in one dialogue area constitute lines that form one meaning, and by extracting the text included in the dialogue area for each dialogue area, the extracted text can more accurately explain the story of the content. In other words, the accuracy of the order of dialogue represented by the text can be guaranteed by extracting the text included in the dialogue area for each dialogue area.

Relatedly, FIG. 11B shows a method of detecting text from a dialogue area that is a speech bubble (or a virtual speech balloon), according to an example.

As shown, text can be extracted through OCR for the recognized dialogue region 1110-2. The depicted metabolic region 1110-2 may include a total of three lines 1140: FFF..., GGG..., and HHH.... Each of the extracted lines of text may be associated with order information indicating which line each line corresponds to in the corresponding dialogue area 1110-2. Accordingly, text information 50 may include such order information.

Text can be extracted in a similar manner for the dialogue areas 1110-1 and 1110-3 separated by speech bubbles.

Meanwhile, the dialogue area 1110-2 in the illustrated example may be a dialogue area corresponding to the first area including the above-described monologue or narration or the second area including text explaining the content. The computer system 100 may generate a virtual speech bubble 1130 corresponding to the first or second area. The virtual speech bubble 1130 may be created to provide order information for the dialogue area.

In other words, the order information included in the text information 50, which will be described later, is based on the order in the image 10 of speech balloons including a speech balloon corresponding to the detected dialogue area and a virtual speech balloon, for each detected dialogue area. The extracted text may include information about which of the speech bubbles it was extracted from. In other words, the sequence number of the dialogue area included in the order information can be sequentially assigned to both the dialogue area separated by speech bubbles and the dialogue area not separated by speech bubbles.

In step 436, the computer system 100 may generate text information 50 based on text extracted for each detected dialogue region.

The text information 50 may include information about which dialogue region and which cut the text extracted for each detected dialogue region was extracted as order information. Additionally, the order information may be configured to further include row information within the corresponding dialogue region of the text extracted for each detected dialogue region.

For example, order information included in the text information 50 may include, for example, information indicating which cut the extracted text was extracted from (e.g., cut N, N is an integer), and information indicating which dialogue region it was extracted from. It may include information (e.g., dialogue area K, K is an integer), and information indicating which row of the extracted text line is (e.g., [R], R is an integer).

For example, in each of the plurality of cuts of the image 10 of the content, the first cut is earlier in the image 10 as it is on the upper and lower side, and in the same position in the vertical direction, the first cut is earlier in the left or right side. A turn number may be assigned.

In addition, in each of the metabolic regions detected in each cut image, there is a second order number that is earlier in the upper and lower direction in each cut image, and is earlier in the left or right position at the same position in the upper and lower direction. can be assigned.

Additionally, each line of text extracted for each detected dialogue region may be assigned a third order number, which is earlier as it is located at the top in the vertical direction, as row information.

The text information 50 may include the first order number, the second order number, and the third order number for the text extracted from each of the dialogue areas as order information. For example, as described above with reference to FIG. 1, the text information 50 includes the cut number (first sequence) of the cut containing the extracted text, the number of the dialogue area (second sequence), and the number of each line of the text ( 3rd turn) may be included.

In relation to this, Figure 12 illustrates a method of determining the order of cuts of content and dialogue regions included in each cut, according to an example.

In FIG. 12 , an example in which a sequence number is assigned as order information to cuts and dialogue regions included in an image 1200 of webtoon content is shown.

As shown, the cut number (C1 -> C5) (first order number) that is higher on the upper and left side may be assigned. Likewise, the higher it is on the top and the left, the higher the metabolic region number (1 -> 6) (second order number) can be assigned. The dialogue area number may be assigned to a speech bubble identified as a dialogue area, but as shown, it may also be assigned to a dialogue area that is not distinguished by a speech bubble (see dialogue area no. 6). At this time, the computer system can generate a virtual speech bubble corresponding to a dialogue area that is not divided into speech bubbles, and assign a dialogue area number (6) to this virtual speech bubble.

Meanwhile, unlike the illustrated example, depending on the embodiment, the dialogue region numbers may be ordered for each cut rather than for all cuts.

Additionally, unlike the illustrated example, a higher cut number (first order number) may be assigned as the cut number is located on the upper or right side. Likewise, the higher the location on the upper and right side, the higher the metabolic region number (second order number) may be assigned. The dialogue area number may be assigned to a speech bubble identified as a dialogue area.

In other words, the direction of reading content (e.g., books, comics, articles, etc.) may be different for each country or cultural area, and the above-mentioned first and second order numbers may vary depending on the direction of reading content in the country or cultural area. can be decided.

According to the order information for cuts and dialogue areas shown in FIG. 12, order information included in the finally generated text information 50 may be determined.

As described above with reference to steps 410 to 430, in an embodiment, computer system 100 does not extract text directly from image 10, but first cuts text from image 10 through object detection. detecting and detecting a metabolic region from the cut image including the cut; Next, OCR can be used to extract text from the dialogue area by dialogue area.

Accordingly, in the embodiment, unnecessary text that does not correspond to 'dialogue' is not extracted from the image 10, and the accuracy of the order of dialogue in the extracted text can be guaranteed.

The description of the technical features described above with reference to FIGS. 1 to 3, 8, and 14 to 16 can also be applied to FIGS. 4, 7, and 10 to 12, so overlapping descriptions will be omitted. .

FIG. 5 is a flowchart illustrating a method of generating an integrated dialogue region image by integrating dialogue regions extracted from cut image(s) and extracting text from the integrated dialogue region image, according to an example.

Referring to FIG. 5, the method for detecting a metabolic region from a cut image will be described in more detail.

In step 510, the computer system 100 may integrate the metabolic regions detected from the cut images to generate one integrated metabolic region image.

In step 520, the computer system 100 may extract text included in each metabolic region for each metabolic region using OCR for the metabolic regions included in the generated integrated metabolic region image.

Relatedly, Figure 19 shows an integrated metabolic region image, according to one example.

As shown, the integrated dialogue area image 1900 may include a plurality of dialogue areas (eg, a plurality of speech bubbles). For example, the computer system 100 may generate one integrated metabolic region image 1900 by integrating metabolic regions detected from a plurality of cut images. Depending on the embodiment, the computer system 100 may generate an integrated metabolic region image 1900 by integrating metabolic regions detected from one cut image. The integrated dialogue area image 1900 may include only dialogue areas by excluding the area containing objects other than the dialogue area and the blank area from the cut image. For example, the computer system can mask a dialogue area (speech balloon, etc.) in a cut image, and create an integrated dialogue area image 1900 by cutting and integrating images corresponding to the masked dialogue area. Accordingly, for the dialogue area image 1900, which is one image, the text included in each dialogue area can be extracted using OCR. Accordingly, in the task of extracting the text contained in each dialogue area for each dialogue area, the text contained in each dialogue area is processed by processing only a smaller number of dialogue area images (1900) without the need to process a large number of images. can be extracted, and thus, the network resources used for image processing and the resources used for text extraction can be saved. For example, extracting text from a dialogue region by reconstructing the detected dialogue region into a dialogue region image 1900 is performed by processing each cut image without reconstructing the detected dialogue region into a dialogue region image 1900. Compared to extracting text (i.e. regular serial processing), it can improve operation speed by about 11 times.

Meanwhile, in another embodiment, the computer system 100 may perform the tasks of recognizing dialogue areas (speech balloons, etc.) for a cut image and extracting text for each dialogue area in parallel. In other words, the computer system 100 can perform an object detection task to recognize dialogue areas (speech balloons, etc.) and a character recognition task to extract text from each dialogue area in parallel. For example, computer system 100 may detect a dialogue region from a cut image, extract text from the detected dialogue region, and detect other dialogue regions while extracting text from the detected dialogue region. In this way, performing the dialogue area detection process and text extraction process in parallel can improve work speed compared to performing the dialogue area detection process and text extraction process serially. For example, such parallel processing can improve work speed by 2 to 5 times compared to serial processing.

However, in the above parallel processing, text extraction (recognition) takes more time than detection of the dialogue region, so a bottleneck may occur in text extraction. Accordingly, the above-mentioned parallel processing may be slower than the operation of extracting text from the dialogue region by reconstructing the above-described detected dialogue region into the dialogue region image 1900. In other words, the time required for text extraction can be reduced by reconstructing the detected dialogue region into a dialogue region image 1900 and extracting text from the dialogue region.

The description of the technical features described above with reference to FIGS. 1 to 4, 7, 8, 10 to 12, and 14 to 16 can also be applied to FIGS. 5 and 19, so the description is redundant. is omitted.

FIG. 6 is a flowchart illustrating a method of generating text information including extracted text and order information associated with the extracted text, according to one example.

As described above with reference to FIGS. 4 and 11B, at step 610, computer system 100 determines that the detected area of dialogue includes a first region or a first region containing the above-described monologue or narration or a second region containing descriptive text of the content. In the case of a dialogue area corresponding to area 2, a virtual speech bubble 1130 corresponding to the first area or the second area can be created. The virtual speech bubble 1130 may be created to provide order information for the dialogue area.

In step 620, the computer system 100 may generate text information 50 including order information considering the generated virtual speech bubbles. In other words, the order information included in the text information 50 is extracted for each detected dialogue area based on the order in the image 10 of speech balloons including a speech balloon and a virtual speech balloon corresponding to the detected dialogue area. It may include information about which of the speech bubbles the text was extracted from. In other words, the sequence number of the dialogue area included in the order information can be sequentially assigned to both the dialogue area separated by speech bubbles and the dialogue area not separated by speech bubbles.

The description of the technical features described above with reference to FIGS. 1 to 5, 7, 8, 10 to 12, 14 to 16, and 19 can also be applied to FIG. 6, so the description is redundant. is omitted.

FIG. 9 is a flowchart illustrating a method of generating text information or deleting text information by re-extracting text according to update or deletion of content, according to an example.

The content may be content that consumers can view, such as webtoon content, for example. Such content may be uploaded to the content server 150 and then served to consumers.

Content may be updated or deleted by deleting, changing, or adding at least part of the content by the author or the administrator of the content. When content is updated, the updated content may be uploaded to the content server 150.

At step 910, computer system 100 may monitor whether content on content server 150 is updated or deleted. For example, computer system 100 may periodically (e.g., once an hour) monitor content server 150 and thereby identify updates and deletions of content previously uploaded to content server 150. Alternatively, the computer system 100 may receive notification from the content server 150 that content has been updated or deleted, and may identify whether the content has been updated or deleted accordingly.

At step 920, if an update to the content is identified, the computer system 100 may extract text from the image included in the updated content. In other words, the computer system 100 can re-identify the updated content as content subject to text extraction, re-extract text from the image of the updated content, and regenerate text information including the re-extracted text. Meanwhile, in some embodiments, computer system 100 may identify updated portions of content, and re-extraction of text may be performed only with respect to these identified updated portions.

At step 930, computer system 100 may delete text information 50 associated with the content once deletion of the content is identified.

Meanwhile, steps 920 and 930 can be performed only when the third function 1450 for setting whether the text information 50 described above with reference to FIG. 15 can be updated is set to OFF. In other words, when the third function 1450 is set to ON, updating of pre-registered text information may be prohibited, and therefore, the computer system 100 changes or deletes text information even if the content is updated or deleted. You may not.

As such, in the embodiment, the text information is regenerated or deleted in consideration of the update and deletion of content in the content server 150, so that the text information can quickly reflect the update and deletion of content.

The description of the technical features described above with reference to FIGS. 1 to 8, 10 to 12, 14 to 16, and 19 can also be applied to FIG. 9, so overlapping descriptions will be omitted.

Figure 13 shows a method of providing text information, according to an example.

In FIG. 13 , cut images (or cuts) 1310 extracted from images of content from which text is to be extracted and text information 1320 including text extracted from the cut images 1310 are shown.

The computer system 100 can detect dialogue regions of the cut images 1310 and extract text included in each dialogue region for each dialogue region.

The computer system 100 may generate text information 1320 including the extracted text and order information about the text. Text information 1320 may be named alternative text.

The text information 1320 may include cut information (cut number) to which the extracted text belongs, dialogue region information (dialogue region number) to which the extracted text belongs, and line information (line number) of each line of the extracted text. .

As shown, text information 1320 can be configured to be expanded by selecting '>'. The text information 1320 may further include location information (coordinates) within the cut or within the image including the cut. This positional information may include at least one of the position of the cut (i.e., the position of the cut line), the position of the dialogue area (speech balloon, etc.), and the position of each line of text included in the dialogue area. This location information can be used as information for determining order information included in the above-described text information. For example, depending on the coordinates indicated by the location information, the order of cuts, the order of the dialogue area, and the order of each line of text included in the dialogue area may be determined. The textlist can display the line number of each line of text included in the dialogue area. The dialogue area number and/or cut number may be displayed above the line number. Through the line number, it can be identified whether the text within the dialogue area has a line break.

In other words, 'cut -> dialogue area -> text (each line of text) included in the dialogue area' can be displayed in the text information 1320 to have a hierarchical relationship. Text information 1320 can be used as meta information to provide the above-described audio information, etc. to consumers of content.

The description of the technical features described above with reference to FIGS. 1 to 12, 14 to 16, and 19 can also be applied to FIG. 13, so overlapping descriptions will be omitted.

Figure 17 shows a method of extracting text from an image of webtoon content and providing text information including the extracted text, according to an example.

In Figure 17, in the case where the content is webtoon content, a webtoon application 1710 running on the consumer's consumer terminal 160, a webtoon server 1720 corresponding to the content server 150, and a computer system of the embodiment ( Example operations of the text information management tool 1730 (shown as an AI alternative text management tool in FIG. 17) running on 100) (or administrator terminal) will be described in more detail. 'Text information' is shown as alternative text in FIG. 17.

As shown, on the webtoon application 1710 side, the webtoon application 1710 may run a content viewer according to a consumer's request to view content. At this time, when the above-mentioned 'reader function (screen reader)' is used in the webtoon application 1710, the webtoon application 1710 may request an alternative text, which is text information related to the content, from the content server 150. . Accordingly, audio information corresponding to the alternative text may be provided to the webtoon application 1710 from the content server 150, and therefore, a reading service for the alternative text may be provided to the webtoon application 1710. If the content is a cuttoon with separate cuts, audio information corresponding to the alternative text of the cut being viewed in the webtoon application 1710 may be provided to the webtoon application 1710.

The webtoon server 1720 may store a copy of the alternative text to provide services to the webtoon application 1710. A copy of this alternative text may be provided from text information management tool 1730.

Content including images may be uploaded to the webtoon server 1720, and the text information management tool 1730 may identify such content as content subject to text extraction and generate alternative text by extracting text from the image of the content. there is.

Meanwhile, content can be modified (updated) or deleted in the webtoon server 1720, and the text information management tool 1730 can recognize such update or deletion and create or delete alternative text for the content. Modification (update) or deletion of content may be transmitted immediately or periodically from the webtoon server 1720 to the text information management tool 1730. Meanwhile, as described above, if the alternative text is locked, the above-described alternative text may not be recreated or deleted.

The text information management tool 1730 may generate alternative text for content when there is a modification in the AI modeling (e.g., when there is an update to the above-described learning model, etc.). Even in this case, if the alternative text is locked, the above-described alternative text may not be regenerated.

The text information management tool 1730 is a function for inspecting the generated alternative text and may provide a function to edit (add/modify/delete) the alternative text. Accordingly, inaccuracies in the generation of alternative text by the AI model can be corrected by the administrator.

The computer system 100 including the text information management tool 1730 may store the original text of the alternative text, and this alternative text may be transmitted to the webtoon server 1720 periodically (eg, every hour). However, with reference to FIG. 15, when the UI 1570 is selected to reflect the above-described text information to the service more quickly (e.g., with the highest priority), the (verified) alternative text is immediately sent to the webtoon server 1720. It can be delivered.

The description of the technical features described above with reference to FIGS. 1 to 16 and 19 can also be applied to FIG. 17 , so overlapping descriptions will be omitted.

Figure 18 shows a method of determining a speaker or character who uttered text included in a dialogue area, according to an example.

The computer system 100 may determine the speaker of the content who uttered the extracted text for each detected dialogue region described above. The speaker may be a character in the content or a character included in the content.

Computer system 100 may determine the speaker of the content using, for example, the learning model described above. This learning model may be pre-trained using a number of webtoon images, and may be pre-trained to estimate who uttered the text in the dialogue area such as a speech bubble included in the webtoon image.

For example, as in the illustrated example, the computer system 100 may configure at least one of a speaker image expressed in association with a speech bubble corresponding to a dialogue area detected in the content image 10 and the color or shape of a speech bubble corresponding to the detected dialogue area. You can determine the speaker based on one.

For example, images 1810-1 and 1820 representing the speaker may be displayed around the speech bubble or dialogue area, and the computer system 100 may display each of the dialogue areas 1810-2 and 1820-2 accordingly. You can determine who this utterance came from. Alternatively, the speech bubble corresponding to the dialogue area may have a different shape depending on the speaker, and the computer system 100 can determine from whom each of the dialogue areas 1820 was spoken according to the shape of the speech bubble. Alternatively, the computer system 100 may determine who uttered each of the dialogue areas 1820 according to whom the tail of the speech bubble is directed toward. Alternatively, the speech bubble corresponding to the dialogue area may have a different color depending on the speaker, and the computer system 100 may determine from whom each of the dialogue areas 1830 was spoken according to the color of the speech balloon. .

In an embodiment, the text information 50 generated based on the text extracted for each detected dialogue region may further include information about the determined speaker. Accordingly, the text information 50 may further include information about who uttered the text in addition to information about the order of the text extracted from the content.

The description of the technical features described above with reference to FIGS. 1 to 17 and 19 can also be applied to FIG. 18, so overlapping descriptions will be omitted.

The device described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, the devices and components described in the embodiments include a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), and a programmable logic unit (PLU). It may be implemented using one or more general-purpose or special-purpose computers, such as a logic unit, microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. It can be embodied in . Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. At this time, the medium may continuously store a computer-executable program, or temporarily store it for execution or download. In addition, the medium may be a variety of recording or storage means in the form of a single or several pieces of hardware combined. It is not limited to a medium directly connected to a computer system and may be distributed over a network. Examples of media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, And there may be something configured to store program instructions, including ROM, RAM, flash memory, etc. Additionally, examples of other media include recording or storage media managed by app stores that distribute applications, sites that supply or distribute various other software, or servers.

As described above, although the embodiments have been described with limited examples and drawings, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims (19)

A method of providing text information associated with content, performed by a computer system, comprising:
identifying content including images uploaded to a content server;
extracting text from the image included in the content; and
Providing text information including the extracted text as text information associated with the content.
A method of providing text information, including. According to paragraph 1,
The image includes a plurality of cuts of the content in an order and text including dialogue of the content,
The extracted text is the dialogue extracted from the text included in the image,
The text information includes each line of a plurality of lines included in the dialogue, and order information of each line. According to paragraph 2,
The step of extracting the text is,
detecting the plurality of cuts in the image;
generating each cut image including each cut of the plurality of cuts; and
Extracting text from cut images corresponding to the plurality of cuts
A method of providing text information, including. According to paragraph 3,
The plurality of cuts are included in the image in an order of scrolling up and down,
A method of providing text information, wherein each cut image is configured to further include a blank area of a predetermined size above and below each cut. According to paragraph 3,
The step of extracting text from the cut images is,
detecting a dialogue region containing the dialogue for each cut image;
Extracting text included in the dialogue region for each detected dialogue region using OCR (Optical Haracter Recognition); and
Generating the text information based on the text extracted for each detected dialogue region
Including,
The dialogue area is an area containing a speech bubble included in the image, a monologue or narration by a speaker or character of the content, or an area containing text explaining the content,
The text information includes, as the order information, information about which dialogue region and which cut the text extracted for each detected dialogue region was extracted from. According to clause 5,
The order information further includes row information within the corresponding dialogue area of the text extracted for each detected dialogue area. According to clause 6,
Each of the plurality of cuts is assigned a first order number, which is earlier as it is on the upper side in the vertical direction within the image, and is assigned a first order number that is earlier as it is on the left or right side at the same position in the vertical direction,
In each of the metabolic regions detected in each cut image, there is a second order number that is ahead the closer it is to the top in the vertical direction within each cut image, and is ahead the closer it is to the left or right at the same position in the up and down direction. assigned,
To each line of text extracted for each detected dialogue region, a third order number, which is higher as it moves upward in the vertical direction, is assigned as the line information,
The text information includes, as the order information, the first order number, the second order number, and the third order number for the text extracted from each of the dialogue areas. According to clause 5,
The step of extracting text from the cut images is,
If the detected dialogue area is a first area containing the monologue or narration or a second area containing the explanatory text, generating a virtual speech bubble corresponding to the first area or the second area
It further includes,
The order information is based on the order in the image of speech balloons including the speech balloon corresponding to the detected dialogue area and the virtual speech balloon, to which of the speech balloons is the text extracted for each detected dialogue area. A method of providing text information, including information about whether it was extracted from. According to clause 5,
The step of extracting text from the cut images is,
Generating one integrated metabolic region image by integrating metabolic regions detected from the cut images; and
For the dialogue regions included in the integrated dialogue region image, extracting text included in the dialogue region for each dialogue region using OCR (Optical Haracter Recognition)
A method of providing text information, including. According to clause 5,
The step of detecting the metabolic region is,
detecting areas containing text in each cut image;
Among the areas, identifying a non-dialogue area, which is an area including text corresponding to the background of each cut, text representing sound effects of the content, and text determined to be unrelated to the story of the content; and
Detecting regions excluding the non-metabolic region among the regions as metabolic regions containing metabolism.
A method of providing text information, including. According to paragraph 2,
The providing step includes providing the text information to the manager terminal according to a request from the manager terminal that manages the content,
Providing a function to enable inspection of the text information for the manager terminal
It further includes,
The function that enables the above inspection is,
Providing text information, including at least one of a first function for enabling editing of the text information, a second function for enabling downloading of the text information, and a third function for setting whether the text information can be updated. method. According to clause 11,
The function enabling the inspection includes the first function,
The step of providing the function that enables the inspection is,
Displaying, at the administrator terminal, the text information including a first cut selected by the administrator among the plurality of cuts and dialogue extracted from the selected first cut;
providing a first user interface for editing the displayed text information; and
Providing a second user interface that allows switching from the first cut to a second cut that is another cut among the plurality of cuts.
A method of providing text information, including. According to paragraph 2,
The providing step includes providing audio information corresponding to the text information to the consumer terminal according to a request from the consumer terminal consuming the content. According to clause 13,
The steps provided above are:
When viewing the content is requested from the consumer terminal, calling the text information associated with the content;
Recognizing a cut being viewed by the consumer terminal among the plurality of cuts;
Outputting audio information corresponding to a portion of the text information corresponding to the recognized cut from the consumer terminal.
A method of providing text information, including. According to paragraph 1,
Monitoring whether the content is updated or deleted on the content server;
When an update to the content is identified, extracting text from the image included in the updated content; and
If deletion of the content is identified, deleting the text information associated with the content.
A method of providing text information, further comprising: According to clause 5,
Determining a speaker of the content who uttered the text extracted for each detected dialogue area - the speaker is displayed in the speaker image expressed in association with a speech bubble corresponding to the detected dialogue area in the image and the detected dialogue area. Determined based on at least one of the color or shape of the corresponding speech bubble -
It further includes,
The text information generated based on the text extracted for each detected dialogue region further includes information about the determined speaker. A program recorded on a computer-readable recording medium for executing the method of claim 1 on the computer system. In a computer system that provides text information associated with content,
At least one processor implemented to execute instructions readable by the computer system
Including,
The at least one processor,
Identify content, including images uploaded to a content server;
A computer system for extracting text from the image included in the content and providing text information including the extracted text as text information associated with the content. According to clause 18,
The image includes a plurality of cuts of the content in an order and text including dialogue of the content,
The extracted text is the dialogue extracted from the text included in the image,
The text information includes each line of the plurality of lines included in the dialogue, and order information of each line,
The at least one processor,
Detecting the plurality of cuts in the image, generating each cut image including each cut of the plurality of cuts, and extracting text from the cut images corresponding to the plurality of cuts,
For each cut image, a dialogue area containing the dialogue is detected, text included in the dialogue area for each detected dialogue area is extracted using OCR, and based on the text extracted for each detected dialogue area. to generate the text information,
The dialogue area is an area containing a speech bubble included in the image, a monologue or narration by a character of the content, or an area containing text explaining the content,
The computer system wherein the text information includes information about which dialogue region and which cut the text extracted for each detected dialogue region was extracted as the order information.

Images