KR20230086502A - Method and system for deleting text in image - Google Patents

Abstract

The present disclosure relates to a method of deleting text in an image, performed by at least one processor. The text deletion method includes the steps of receiving an image including one or more text objects, receiving a text area mask corresponding to the image, inputting the received image and the received text area mask to a machine learning model, and inputting the received image to a machine learning model. and outputting a correction image in which at least a part of one or more text objects is deleted from the received image, based on a region surrounding one or more strokes in the machine learning model, the reference image and the reference text corresponding to the reference image. and a generation model trained to receive a region mask and generate a first reference corrected image based on a region surrounding one or more reference strokes in the reference image.

Description

Method and system for deleting text in an image {METHOD AND SYSTEM FOR DELETING TEXT IN IMAGE}

The present invention relates to a method of deleting text in an image, and more particularly, by receiving an image including one or more text objects, inputting the received image and the received text area mask to a machine learning model, A method and system for outputting a corrected image from which at least a part of a text object is deleted.

STR (Scene Text Removal) has recently been attracting attention as an important technique for hiding personal information such as ID numbers, phone numbers, and vehicle numbers in images. Currently, many STR studies use deep learning methods, but inaccurate and unsatisfactory results have been obtained, such as blurring of some areas where text has been removed. In addition, conventional methods cannot selectively erase characters in a specific area, so application fields are greatly limited.

Furthermore, models designed for the conventional STR require additional improvement steps or sub-models, resulting in a complicated configuration and slow speed. On the other hand, if the speed is high, there is a problem that it is difficult to design as an actual model. In addition, since conventional STR technologies do not use the same standardized training data set and evaluation data set, it is difficult to evaluate their superiority.

In order to solve the above problems, various embodiments of the present disclosure provide a method for deleting text in an image, a computer program stored in a recording medium, and an apparatus (system).

The present disclosure may be implemented in a variety of ways, including a method, apparatus (system) or computer program stored on a readable storage medium.

According to one embodiment of the present disclosure, a method of deleting text in an image, performed by at least one processor, is provided. The text deletion method includes the steps of receiving an image including one or more text objects, receiving a text area mask corresponding to the image, inputting the received image and the received text area mask to a machine learning model, and inputting the received image to a machine learning model. and outputting a correction image in which at least a part of one or more text objects is deleted from the received image, based on a region surrounding one or more strokes in the machine learning model, the reference image and the reference text corresponding to the reference image. and a generation model trained to receive a region mask and generate a first reference corrected image based on a region surrounding one or more reference strokes in the reference image.

According to one embodiment of the present disclosure, a computer program stored in a computer readable recording medium is provided to execute a method of deleting text in an image on a computer.

According to one embodiment of the present disclosure, an information processing system includes a memory and at least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory, the at least one program comprising: Receive an image containing one or more text objects, receive a text area mask corresponding to the image, input the received image and the received text area mask to a machine learning model, and input the received image and the received text area mask to the area around the one or more strokes in the received image. Based on, instructions for outputting a corrected image from which at least a part of one or more text objects is deleted from the received image, and the machine learning model receives a reference image and a reference text area mask corresponding to the reference image as input, and a generating model learned to generate a first reference corrected image based on a region surrounding one or more reference strokes in the reference image.

According to some embodiments of the present disclosure, a text object in an image may be naturally deleted.

According to some embodiments of the present disclosure, learning efficiency may be improved by learning a machine learning model based on a region surrounding a text stroke.

According to some embodiments of the present disclosure, since the machine learning model does not explicitly create a text stroke mask, there is no need for a sub-model of the machine learning model and a faster model can be designed with fewer parameters. .

According to some embodiments of the present disclosure, since a process of reconstructing an area not masked by a text area mask is omitted in the machine learning model, the quality of a text deletion operation may be improved.

1 shows an example of a machine learning model trained to delete text in an image according to an embodiment of the present disclosure.
2 is a schematic diagram illustrating a configuration in which an information processing system according to an embodiment of the present disclosure is communicatively connected with a plurality of user terminals.
3 is a block diagram showing internal configurations of a user terminal and an information processing system according to an embodiment of the present disclosure.
4 is a configuration diagram of a machine learning model according to an embodiment of the present disclosure.
5 is a configuration diagram of a spatial concentration model according to an embodiment of the present disclosure.
6 illustrates an example of generating a corrected image according to an embodiment of the present disclosure.
7 shows an example of a reference intermediate image according to an embodiment of the present disclosure.
8 illustrates a structure of a nested connection according to an embodiment of the present disclosure.
9 illustrates quality comparison results between a method of deleting an image in text according to an embodiment of the present disclosure and other methods.
10 illustrates an example of a method of deleting text in an image according to an embodiment of the present disclosure.

Hereinafter, specific details for the implementation of the present disclosure will be described in detail with reference to the accompanying drawings. However, in the following description, if there is a risk of unnecessarily obscuring the gist of the present disclosure, detailed descriptions of well-known functions or configurations will be omitted.

In the accompanying drawings, identical or corresponding elements are given the same reference numerals. In addition, in the description of the following embodiments, overlapping descriptions of the same or corresponding components may be omitted. However, omission of a description of a component does not intend that such a component is not included in an embodiment.

Advantages and features of the disclosed embodiments, and methods of achieving them, will become apparent with reference to the following embodiments in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms, but only the present embodiments make the present disclosure complete, and the present disclosure does not extend the scope of the invention to those skilled in the art. It is provided only for complete information.

Terms used in this specification will be briefly described, and the disclosed embodiments will be described in detail. The terms used in this specification have been selected from general terms that are currently widely used as much as possible while considering the functions in the present disclosure, but they may vary according to the intention of a person skilled in the related field, a precedent, or the emergence of new technologies. In addition, in a specific case, there is also a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the invention. Therefore, the terms used in the present disclosure should be defined based on the meaning of the terms and the general content of the present disclosure, not simply the names of the terms.

Expressions in the singular number in this specification include plural expressions unless the context clearly dictates that they are singular. Also, plural expressions include singular expressions unless the context clearly specifies that they are plural. When it is said that a certain part includes a certain component in the entire specification, this means that it may further include other components without excluding other components unless otherwise stated.

In addition, the term 'module' used in the specification means a software or hardware component, and a 'module' performs certain roles. However, 'module' is not meant to be limited to software or hardware. A 'module' may be configured to reside on an addressable storage medium and may be configured to reproduce one or more processors. Thus, as an example, a 'module' includes components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, or variables. The functionality provided within the components and 'modules' may be combined into a smaller number of components and 'modules' or further separated into additional components and 'modules'.

According to one embodiment of the present disclosure, a 'module' may be implemented with a processor and a memory. 'Processor' should be interpreted broadly to include general-purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, and the like. In some circumstances, 'processor' may refer to an application specific integrated circuit (ASIC), programmable logic device (PLD), field programmable gate array (FPGA), or the like. 'Processor' refers to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors in conjunction with a DSP core, or a combination of any other such configurations. You may. Also, 'memory' should be interpreted broadly to include any electronic component capable of storing electronic information. 'Memory' includes random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable-programmable read-only memory (EPROM), It may also refer to various types of processor-readable media, such as electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, and the like. A memory is said to be in electronic communication with the processor if the processor can read information from and/or write information to the memory. Memory integrated with the processor is in electronic communication with the processor.

In the present disclosure, 'each of a plurality of A' and/or 'each of a plurality of A' may refer to each of all components included in a plurality of A's, or each of some components included in a plurality of A's. can For example, each of a plurality of pixels may refer to each of all pixels included in the plurality of pixels or each of some pixels included in the plurality of pixels.

1 illustrates an example of a machine learning model trained to delete text within an image 110 according to one embodiment of the present disclosure. As shown, the machine learning model includes a generation model (G) 130 for generating a correction image 140 from which one or more text objects in the image 110 are deleted and a correction image for the advancement of the generation model 130. 140 may include a discrimination model (D) for determining whether a correct image (Real image) or a fake image (Fake image). For example, as shown in FIG. 1 , one or more text objects may include Welcome to Our Porch Sit Long Talk Much Laugh Often. 1, the machine learning model is illustrated as including both the generating model 130 and the discriminating model 160, but is not limited thereto. For example, the machine learning model may include only the generative model 130 .

According to an embodiment, the generation model 130 takes the image 110 and the text area mask 120 corresponding to the image 110 as inputs, and at least a portion of one or more text objects is deleted from the image 110. The correction image 140 may be output. Here, the text area mask 120 can be used to efficiently learn a machine learning model by detecting the location of one or more text objects included in the image 110 . For example, the text area mask 120 may be used to specify a location of a text object. As shown, the 'white' region of the text area mask 120 represents a text region, and the 'black' region represents a non-text region. In addition, a text area mask 120 generated in advance using the image 110 may be input to the machine learning model.

According to an embodiment, the generation model 130 receives the image 110 and the text area mask 120 corresponding to the image 110, and generates the surrounding areas 172 and 174 of one or more strokes in the image 110. It can be learned to generate the corrected image 140 based on . Here, the stroke peripheral areas 172 and 174 may refer to areas adjacent to the outer stroke (and/or outer contour) of the text object in the image 110 . For example, the area 172 surrounding the first stroke may refer to an area adjacent to the outer stroke of 'Welcome To Our Porch', which is a part of the text object in the image 110 . As another example, the area 174 surrounding the second stroke may refer to an area adjacent to the outer stroke of 'Sit Long Talk Much Laugh Often', which is the remaining part of the text object in the image 110 .

According to an embodiment, the discrimination model 160 may be configured to learn the distribution of the ground truth image by using the ground truth image 150 from which at least a part of the text object has been deleted in advance. Here, the distribution of the reference correct answer images may indicate a probability distribution to be included in the reference correct answer images, and may be determined using a plurality of reference correct answer images. Then, the discrimination model 160 may determine whether the corrected image 140 is a correct image or not based on whether the corrected image 140 is included in the correct answer distribution. When the discrimination model 160 determines that the corrected image 140 is the correct image, the machine learning model is trained so that the generating model 130 outputs the corrected image 140 close to the reference correct image 150 . That is, the machine learning model may be configured so that the generation model 130 outputs the corrected image 140 that is the same as or very similar to the reference correct image 150 .

On the other hand, in FIG. 1, the term referring to the data used for inference of the machine learning model of the generative model 130 and the data used for learning of the machine learning model is an image 110, a text area mask 120, and a corrected image ( 140) and the surrounding areas 172 and 174 of the stroke. However, in FIGS. 2 to 10 , terms referring to data used in the inference process and terms referring to data used in the learning process may be used separately. For example, data used in the inference process of the machine learning model may be described as 'image', 'text area mask', 'correction image', and 'surrounding area of stroke'. On the other hand, the data used for machine learning model learning are 'reference image', 'reference text area mask', 'reference correction image', 'region around reference stroke', 'reference label mask', and 'reference correct answer image'. can be listed. That is, the data term 'X' (eg, image, etc.) used in the inference process and the data term used in the corresponding learning process may be defined as 'reference X' (eg, reference image, etc.). .

2 is a schematic diagram illustrating a configuration in which an information processing system 230 according to an embodiment of the present disclosure is connected to communicate with a plurality of user terminals 210_1, 210_2, and 210_3. Information processing system 230 may include a system and/or device(s) for deleting text within an image. In one embodiment, the information processing system 230 is one or more server devices and/or devices capable of storing, providing, and executing computer-executable programs (eg, downloadable applications) and data related to deleting text within an image. database, or one or more distributed computing devices and/or distributed databases based on cloud computing services. The information processing system 230 may provide information corresponding to a signal input through an application (eg, an artificial intelligence application, etc.) or perform a corresponding process. For example, the information processing system 230 may process a plurality of user terminals 210_1, 210_2, and 210_3 through an arbitrary application configured to provide a service related to image processing (eg, processing of deleting text in an image). can control.

The information processing system 230 may communicate with a plurality of user terminals 210_1 , 210_2 , and 210_3 through the network 220 . The network 220 may be configured to enable communication between the plurality of user terminals 210_1, 210_2, and 210_3 and the information processing system 230. Depending on the installation environment, the network 220 may be, for example, a wired network such as Ethernet, a wired home network (Power Line Communication), a telephone line communication device and RS-serial communication, a mobile communication network, a wireless LAN (WLAN), It may consist of a wireless network such as Wi-Fi, Bluetooth, and ZigBee, or a combination thereof. The communication method is not limited, and a plurality of user terminals (210_1, 210_2) as well as a communication method utilizing a communication network (eg, mobile communication network, wired Internet, wireless Internet, broadcasting network, satellite network, etc.) that the network 220 may include. , 210_3) may also include short-range wireless communication.

In FIG. 2, a mobile phone terminal 210_1, a tablet terminal 210_2, and a PC terminal 210_3 are illustrated as examples of user terminals, but are not limited thereto, and the user terminals 210_1, 210_2, and 210_3 may perform wired and/or wireless communication. It may be any computing device capable of this and capable of installing and running a translation application, an image processing application, an artificial intelligence application, and the like. For example, the user terminal includes a smart phone, a mobile phone, a navigation device, a computer, a laptop computer, a digital broadcasting terminal, a PDA (Personal Digital Assistants), a PMP (Portable Multimedia Player), a tablet PC, and a game console. , a wearable device, an internet of things (IoT) device, a virtual reality (VR) device, an augmented reality (AR) device, and the like. In addition, although three user terminals 210_1, 210_2, and 210_3 are shown in FIG. 2 to communicate with the information processing system 230 through the network 220, it is not limited thereto, and a different number of user terminals may be connected to the network ( 220) to communicate with the information processing system 230.

In one embodiment, the user terminals 210_1, 210_2, and 210_3 transmit a data request related to a method of deleting text in an image to the information processing system 230 through the network 220, and from the information processing system 230. Data related thereto (eg, an image in which a text object is deleted) may be received.

Figure 3 is a block diagram showing the internal configuration of the user terminal 210 and the information processing system 230 according to an embodiment of the present disclosure. The user terminal 210 may refer to any computing device capable of executing a translation application, a text editing application, an image processing application, and/or an artificial intelligence application, and capable of wired/wireless communication. For example, in FIG. It may include a mobile phone terminal 210_1, a tablet terminal 210_2, and a PC terminal 210_3. As shown, the user terminal 210 may include a memory 312 , a processor 314 , a communication module 316 and an input/output interface 318 . Similarly, the information processing system 230 may include a memory 332 , a processor 334 , a communication module 336 and an input/output interface 338 . As shown in FIG. 3, the user terminal 210 and the information processing system 230 are configured to communicate information and/or data through the network 220 using respective communication modules 316 and 336. It can be. In addition, the input/output device 320 may be configured to input information and/or data to the user terminal 210 through the input/output interface 318 or output information and/or data generated from the user terminal 210.

The memories 312 and 332 may include any non-transitory computer readable media. According to one embodiment, the memories 312 and 332 are non-perishable mass storage devices such as random access memory (RAM), read only memory (ROM), disk drives, solid state drives (SSDs), flash memory, and the like. (permanent mass storage device) may be included. In another embodiment, a non-perishable mass storage device such as ROM, SSD, flash memory, disk drive, etc. may be included in the user terminal 210 and/or the information processing system 230 as a separate permanent storage device distinct from memory. there is. In addition, the memories 312 and 332 include an operating system and at least one program code (for example, a translation application, a text editing application, an image processing application, and/or an artificial intelligence application installed and driven in the user terminal 210). code) can be stored.

These software components may be loaded from a computer readable recording medium separate from the memories 312 and 332 . A recording medium readable by such a separate computer may include a recording medium directly connectable to the user terminal 210 and the information processing system 230, for example, a floppy drive, a disk, a tape, a DVD/CD- It may include a computer-readable recording medium such as a ROM drive and a memory card. As another example, software components may be loaded into the memories 312 and 332 through a communication module rather than a computer-readable recording medium. For example, at least one program is loaded into the memories 312 and 332 based on a computer program installed by files provided by developers or a file distribution system that distributes application installation files through the network 220 . It can be.

The processors 314 and 334 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to processors 314 and 334 by memories 312 and 332 or communication modules 316 and 336 . For example, processors 314 and 334 may be configured to execute instructions received according to program codes stored in a recording device such as memory 312 and 332 .

The communication modules 316 and 336 may provide configurations or functions for the user terminal 210 and the information processing system 230 to communicate with each other through the network 220, and the user terminal 210 and/or information processing. System 230 may provide configurations or functions for communicating with other user terminals and/or other systems (eg, separate cloud systems, etc.). For example, a request or data generated by the processor 314 of the user terminal 210 according to a program code stored in a recording device such as the memory 312 (eg, data associated with a text object in an image) is transmitted through a communication module It may be transferred to the information processing system 230 through the network 220 under the control of 316 . Conversely, a control signal or command provided under the control of the processor 334 of the information processing system 230 passes through the communication module 336 and the network 220 and through the communication module 316 of the user terminal 210. It may be received by the user terminal 210 .

The input/output interface 318 may be a means for interfacing with the input/output device 320 . As an example, the input device may include an inertial sensor such as an acceleration sensor or a gyroscope (ie, an angular velocity sensor), an optical sensor such as an optical camera or an IR camera, or a distance sensor such as a ToF, LiDAR sensor, or depth camera. . Additionally, the input device may include a device such as a camera, keyboard, microphone, mouse, etc. including an audio sensor and/or an image sensor, and the output device may include a device such as a display, speaker, or haptic feedback device. . As another example, the input/output interface 318 may be a means for interface with a device in which a configuration or function for performing input and output is integrated into one, such as a touch screen. For example, a service configured using information and/or data provided by the information processing system 230 when the processor 314 of the user terminal 210 processes a command of a computer program loaded into the memory 312. A screen or the like may be displayed on the display through the input/output interface 318 . Although the input/output device 320 is not included in the user terminal 210 in FIG. 3 , it is not limited thereto, and the user terminal 210 and the user terminal 210 may be configured as one device. In addition, the input/output interface 338 of the information processing system 230 is connected to the information processing system 230 or means for interface with a device (not shown) for input or output that the information processing system 230 may include. can be In FIG. 3, the input/output interfaces 318 and 338 are shown as separate elements from the processors 314 and 334, but are not limited thereto, and the input/output interfaces 318 and 338 may be included in the processors 314 and 334. there is.

The user terminal 210 and the information processing system 230 may include more components than those shown in FIG. 3 . However, there is no need to clearly show most of the prior art components. According to one embodiment, the user terminal 210 may be implemented to include at least some of the aforementioned input/output devices 320 . In addition, the user terminal 210 may further include other components such as a transceiver, a global positioning system (GPS) module, a camera, various sensors, and a database. For example, the user terminal 210 may be implemented such that various components such as a camera module, a touch panel, and an input/output port are further included in the user terminal 210 .

According to one embodiment, the processor 314 of the user terminal 210 may be configured to run any application capable of operating or controlling the user terminal 210 . For example, a translation application, a text editing application, an image processing application, and/or an artificial intelligence application may be configured to operate. At this time, codes associated with the application and/or program may be loaded into the memory 312 of the user terminal 210 . While the application and/or program is running, the processor 314 of the user terminal 210 receives information and/or data provided from the input/output device 320 through the input/output interface 318 or through the communication module 316. Information and/or data may be received from the information processing system 230 , and the received information and/or data may be processed and stored in the memory 312 . Additionally, such information and/or data may be provided to information processing system 230 via communication module 316 .

While a program for a text editing application or the like is running, the processor 314 receives information input or selected through an input device such as a camera, microphone, or the like including a touch screen, keyboard, audio sensor, and/or image sensor connected to the input/output interface 318. It can receive text, image, video, voice, etc., and store the received text, image, video, and/or voice in the memory 312 or through the communication module 316 and the network 220 to the information processing system ( 230) can be provided.

The processor 334 of the information processing system 230 may be configured to manage, process and/or store information and/or data received from a plurality of user terminals including the user terminal 210 and/or a plurality of external systems. can Information and/or data processed by the processor 334 may be provided to the user terminal 210 through the communication module 336 and the network 220 . In one embodiment, the processor 334 of the information processing system 230 may receive a request for deleting text in an image from the user terminal 210 through the communication module 336 and the network 220 . The processor 334 of the information processing system 330 processes information and/or data (eg, text) through an output device such as a display output capable device (eg, a touch screen, a display, etc.) of the user terminal 210. may be configured to output a deleted image). Alternatively, when receiving a request for text deletion, the processor 314 of the user terminal 210 may be configured to perform text deletion processing using a machine learning model stored in the memory 312 of the user terminal 210. there is. That is, an inference process of a machine learning model for deleting text in an image may be performed in a user terminal.

4 is a configuration diagram of a machine learning model according to an embodiment of the present disclosure. As shown, the machine learning model receives a reference image 420 and a reference text area mask 410 corresponding to the reference image 420 and is trained to generate a first reference corrected image 440 (or generator) (430). Here, the reference corrected image 440 may refer to a non-text image from which text objects in the reference image 420 are deleted. According to one embodiment, the generative model 430 may be implemented with an FCN-ResNet18 backbone, and the size of the kernel is 4, the stride is 2, and the padding is 1. It may include five convolution layers 432 paired with the deconvolution layers 434, but is not limited thereto, and may include arbitrary models having various sizes.

According to an embodiment, objective functions of the generative model 430 and the discriminant model (eg, the discriminant model 150) are as Equations 1 and 2, respectively.

In Equations 1 and 2, G and D refer to a generative model 430 and a discriminant model, respectively. Also, x and y denote a reference image 420 and a ground truth image 460 concatenated with the reference text area mask 410 , respectively.

According to an embodiment, the generation model 430 may perform convolution and deconvolution on the reference image 420 concatenated with the reference text area mask 410 . In this case, the convolution layers 432 included in the generation model 430 use the reference image 420 and the reference text area mask 410 to determine the surrounding area of one or more reference strokes in the reference image 420 (eg : A Spatial Attention model 436 configured to determine information about the surrounding areas 172 and 174 of the stroke. In this disclosure, the spatial concentration model 436 may be denoted as a spatial concentration module or Region Aware Mask Guided Attention (RAMA). A detailed description of the spatial concentration model 436 is described later in FIG. 5 .

According to an embodiment, the generation model 430 uses a reference correct image 460 corresponding to the first reference corrected image 440 and a loss function 450 defined based on the first reference corrected image 440. so it can be learned. In this case, the value of the loss function 450 may be calculated based on the difference between the reference correct image 460 and the first reference correction image 440 and the intersection area of the input reference text area mask 410. . For example, the loss function (L _RR or L _M ) 450 may be defined as Equation 3.

In Equation 3, I _out(i) represents the ith (i is a natural number) deconvolution layers 434 of the generative model 430 . In Equation 3, M _i and I _gt(i) represent the text area mask 410 and the reference answer image 460 scaled to the same scale as I _out(i) . In the present disclosure, L _M defined by Equation 3 may be expressed as L _RR or Region Regression Loss (RRL).

,

및

는 각각 0.6, 0.8 및 1.0으로 설정될 수 있다.According to an embodiment, one or more second reference correction images may be extracted from the convolution layers 432 and/or the deconvolution layers 434 of the generation model 430 . For example, as shown in FIG. 4 , at least some portions 438_1, 438_2, and 438_3 of the deconvolution layer may be extracted as a second reference correction image. Here, the size of each of the one or more second reference corrected images may be different from that of the first reference corrected image 440 . The one or more extracted second reference corrected images may be used to define loss function 450 . For example, deconvolution layers 434 may be used to compute the value of loss function 450 . In this disclosure, a third deconvolution layer 438_1 , a fourth deconvolution layer 438_2 , and a fifth deconvolution layer 438_3 may be used to define the loss function 450 . in this case,

,

and

may be set to 0.6, 0.8 and 1.0, respectively.

According to one embodiment, the total loss function L _G of the generative model 430 may be defined as Equation 4.

는 수학식 10에서 정의되는 함수를 지칭할 수 있다. 또한,

,

,

,

,

및

는 대응 로스 함수(로스)의 가중치를 나타낼 수 있다. 예를 들어,

,

,

,

,

및

는 각각 100, 0.5, 50.0, 25.0, 1 및 1로 설정될 수 있다. 이하에서는, 각 함수가 상세히 정의될 수 있다.Here, L _adv represents the objective function of the generative model 430 defined in Equation 1,

may refer to a function defined in Equation 10. also,

,

,

,

,

and

may represent the weight of the corresponding loss function (loss). for example,

,

,

,

,

and

may be set to 100, 0.5, 50.0, 25.0, 1 and 1, respectively. In the following, each function may be defined in detail.

L _p can be used to make the generated reference corrected image 440 more realistic. This can be used to reduce the difference between the high-level features of two images extracted using a known, pre-trained ImageNet. Specifically, a loss function L _p for using two composite outputs generated using the reference text area mask 410 and the reference text stroke mask (not shown) may be designed. Here, the reference text stroke mask may refer to a mask generated by the generation model 430 based on a pixel value difference between the reference image 420 and the reference correct image 460 corresponding to the reference corrected image 440. there is. L _p may be defined as Equation 7 by Equations 5 and 6.

In Equations 5 to 7, I _in and I _out represent the input image and output image of the generated model, respectively, and I _boxComp and I _strokeComp are synthesized with the reference text area mask 410 M _box and the reference text stroke mask M _Storke , respectively. It can refer to the created image. I _gt is the reference answer image 460, and A _n may indicate activation of the n-th layer in the network. In addition, pool1, pool2, and pool3 of VGG-16 pretrained on ImageNet, which is already known, can be used.

번째 레이어의 활성화를 나타낸다. 또한, L_p와 동일한 레이어가 이용될 수 있다.L _s in Equation 4 can also be used to encourage better visual quality and feature consistency of the reference calibrated image 440. This can be calculated using the gram matrix of the feature map. As with L _p , two composite outputs may be used. L _s may be defined by Equation 8. where ф(x)=x ^T x is the gram matrix operator, and A _n is the VGG-16

Indicates the activation of the second layer. Also, the same layer as L _p may be used.

When synthetic output is used for L _t in Equation 4 for noise removal, a synthetic image generated using the reference text area mask 410 may be used. Here, L _t may be defined by Equation 9.

Here, I _comp may refer to a composite image, and i and j may indicate pixel positions.

5 is a configuration diagram of a spatial concentration model 510 according to an embodiment of the present disclosure. As shown, the spatial concentration model 510 takes a reference image 514 and a reference text area mask 512 as inputs to determine information 516 about the surrounding area of one or more reference strokes in the reference image 514. can According to an embodiment, the spatial concentration model 510 performs convolution on the input reference image 514 and the reference text area mask 512, respectively, and then performs max pooling on the convolved reference image 514. (Max Pooling) and Average Pooling can be performed. Then, by concatenating the reference text area mask 512 with the reference image 514, information 516 about the surrounding area of one or more reference strokes may be determined. For example, a feature on which max pooling is performed on the reference image 514, a feature on which average pooling is performed on the reference image 514, and the text area mask 512 are combined, and the combined information Information 516 on the surrounding area of one or more reference strokes may be determined by performing a convolution operation on the reference stroke. With this configuration, the spatial concentration model 510 can be intensively learned for the surrounding area of the stroke (or text object).

According to an embodiment, the information 516 on the surrounding area of one or more reference strokes is a probability value that each of a plurality of pixels included in the reference image 514 is included in the surrounding area of the one or more reference strokes, that is, the attention score. It may include an attention score map having (Attention Score). Additionally, the attention score map can be displayed as a heatmap for visualization. In this case, the higher the probability that each of the plurality of pixels included in the reference image is included in the surrounding area of the reference stroke, the corresponding pixel on the heat map is displayed in red, and the lower the probability, the corresponding pixel is displayed in blue.

According to one embodiment, the spatial concentration model 510 may be learned using a loss function defined based on the reference label mask 526 and the information 516 on the surrounding area of one or more reference strokes. The reference label mask 526 may refer to an area corresponding to the area around the reference stroke (eg, the area around the stroke 172 or 174). According to one embodiment, the reference label mask 526 may refer to a previously created label mask for the reference image 514 . According to one embodiment, a reference based on a pixel value difference between a reference image 514 and a reference correct image (eg, reference correct image 460) corresponding to a reference corrected image (eg, reference corrected image 440). A text stroke mask (not shown) may be created. Then, the area of intersection between the outside of the generated reference text stroke mask and the inside of the reference text area mask 512 corresponds to the surrounding area of one or more reference strokes (eg, the surrounding area of the stroke 172, 174). may be created as a label mask 526 . That is, the spatial concentration model 510 may be learned according to a weak supervision method using the reference label mask 526 .

According to an embodiment, the value of the loss function 530 may be calculated based on the reference label mask 526 and the information 516 on the surrounding area of one or more determined reference strokes. For example, loss function 530 can be defined as Equation 10.

In Equation 10, M ⁱ and M ⁱ _box represent the ith pixel of the reference label mask 526 and the reference text area mask 512, respectively, and score ⁱ may represent the ith pixel of the attention score map. In addition, the form of M and score can be expressed as (H _n , W _n , 1). Here, H _n and W _n may indicate the size of the feature map of the nth layer.

According to one embodiment, the spatial concentration model 510 may update the reference text area mask 512 . Since the spatial concentration model 510 uses the information of the text region of the reference image 514 (i.e., the portion corresponding to the 'white' region of the text region mask 512 in the reference image 514), the feature during execution I need to track the textarea within the map. At this time, the spatial concentration model 510 updates the reference text area mask 512 without distortion and tracks all areas affected by the masked area (ie text area) in the reference image 514, In convolutional layers, the concept of a receptive field can be used. The reference text area mask 512 can be updated by two convolutional layers. For example, each layer may have one 3x3 convolution filter in which each element is fixed to 1/9. The updated reference text area mask 522 can be defined as Equation (11).

는 컨볼루션 연산자를 나타내고, K는 각 요소(element)가 고정된 컨볼루션 레이어의 가중치를 나타낸다.In Equation 11, M _in and M _out represent the reference text area mask 512 and the updated reference text area mask 522 of the convolution layer, respectively. also

denotes a convolution operator, and K denotes a weight of a convolution layer in which each element is fixed.

According to an embodiment, an area indicated by the reference text area mask 512 (ie, a 'white' area) may be designed to have a larger value than an external area (ie, a 'black' area). For example, a mask value (ie, M _in or M _out ) may be higher when a region with many reactive fields is included than when it is not otherwise. Additionally, mask values can be automatically resized to prevent them from becoming too small or too large.

According to an embodiment, the generation model 510 concatenates the information 516 of the surrounding area of the reference stroke, the reference image 514, and a layer obtained by convolving the reference image 514 to form a reference intermediate image ( 524) can be created. For example, a reference intermediate image 524 is generated by multiplying the information 516 of the surrounding area of the reference stroke by a layer obtained by convolving the reference image 514 and adding the reference image 514 to the multiplied information. can Then, the generation model 510 generates a region masked by the reference text area mask 512 in the reference image 514 based on the reference intermediate image 524 (ie, the 'white color' of the reference text area mask 512). ' area) and the text object included in the corresponding part can be deleted. Meanwhile, in FIG. 5 , the reference image 514 is shown as an RGB image for convenience of description, but is not limited thereto. For example, the reference image 514 input to the generation model 510 may refer to a feature map of the reference image 514 obtained in a previous layer.

6 illustrates an example of generating a corrected image according to an embodiment of the present disclosure. As described above, the machine learning model may determine information (eg, information 516 ) about an area around one or more reference strokes based on information about the reference image 610 and the reference text area mask 620 . . Then, an image 630 in which one or more text objects (or strokes) in the reference text area (or text area mask) are deleted may be generated using the information about the area around the determined one or more reference strokes. In this case, image characteristics (eg, color, intensity, texture, etc.) of a region around the reference stroke may be used to delete one or more text objects. That is, the image characteristics of the area around the reference stroke may be applied to the area corresponding to the text object (ie, the stroke area). In this way, since the image characteristics of the area around the reference stroke are used to delete one or more text objects, one or more text objects in the reference image 610 can be deleted more naturally.

According to one embodiment, the machine learning model is an image 630 from which one or more text objects have been deleted and an image 640 including an area outside the reference text area mask 620 in the reference image 610 (i.e., the reference text The first reference corrected image 650 may be output using information on an area not masked by the area mask 620 . For example, the machine learning model may output the first reference corrected image 650 by combining the image 630 from which one or more text objects are deleted and the image 640 including the external area.

7 shows an example of a reference intermediate image 720 according to an embodiment of the present disclosure. As described above with reference to FIG. 6 , the spatial concentration model included in the machine learning model may generate the reference intermediate image 720 based on information about the surrounding area of the reference stroke. For example, the spatial concentration model combines information about the area around the reference stroke (eg, information 516 about the area around the reference stroke) and the reference image 710 to obtain a reference intermediate image 720. can create

According to an embodiment, as shown in the reference intermediate image 720, with respect to pixels included in at least a portion of the reference image 710, a heat map indicating a probability that each pixel is included in an area surrounding one or more reference strokes. this can be displayed. For example, the higher the probability that each pixel is included in the peripheral area of one or more reference strokes, the closer to red, and the lower the probability that each pixel is included in the peripheral area of one or more reference strokes, the closer to blue color may be displayed.

8 illustrates a structure of a nested connection according to an embodiment of the present disclosure. Here, the overlapping connection may be implemented using a dense connection already known in the art in order to improve the quality of a non-stroke area. Specifically, the structure of overlapping connection may include a transforming block and an up-sampling block. In order to replace large-kernel convolution, lateral connection, which is already known in the art, can be applied to such a dense connection. As shown, the upsampled feature map may be concatenated with the transformed feature map.

9 illustrates quality comparison results between a method of deleting an image in text according to an embodiment of the present disclosure and other methods. From left to right, an image 910, a correct image 920, a correction image 930 generated by EnsNet known in the art, a correction image 940 created by MTRNet known in the art, A correction image 950 generated by MTRNet++ known in the art, a correction image 960 created by EraseNet known in the art, and a method for deleting text in an image according to an embodiment of the present disclosure. Image 970 is shown.

According to an embodiment, both a synthetic data set and a real data set may be used for training and evaluation of the machine learning model of the present disclosure. For example, a synthetic data set might contain about 800,000 images, consisting of 8,000 textless images. 95% of images are randomly selected for training, 10,000 images are selected for testing, and the rest can be used for validation. The background images of the training data set and the test data set may be mutually exclusive. As another example, a real data set manually generated from Chinese and English text images may include 2,749 training images and 813 test images.

According to an embodiment, pre-processing may be performed on the image 910 or a reference image (eg, the reference image 420 or the reference image 514) in order to perform the machine learning model of the present disclosure. A stroke-level segmentation mask may be required to train a machine learning model. However, existing data sets (i.e., image 910 or reference image) do not provide a stroke-level segmentation mask (eg, text stroke mask or reference text stroke mask). Accordingly, a stroke-level segmentation mask may be generated by calculating a pixel value difference between the image 910 and the correct answer image 920 (or the reference image and the reference correct answer image). The threshold can be set to 25 to suppress noise. The synthetic data set and the real data set can be combined, a total of 738,113 images are used for learning, and a test set can be used separately to distinguish the performance of the real data set and the synthetic data set.

According to an embodiment, an evaluation method using an auxiliary text detector may be proposed for evaluation of the machine learning model of the present disclosure. The auxiliary text detector may obtain detection results from the corrected images 930 to 970 from which text is removed. Then, the performance of the model can be evaluated by calculating the Precision value, Recall value, and F-score value. The lower the Precision value, the Recall value, and the F-score value, the better the text can be erased. Additionally, an evaluation method used for image inpainting may be proposed for evaluation of the machine learning model of the present disclosure. Here, in order to evaluate the quality of the corrected images 930 to 970, Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), Mean Squared Error (MSE), Average Gray-level Error (AGE), Percentage of Error Pixels (pEP) and Percentage of Clustered Error Pixels (pCEP) may be used. Higher values of PSNR and SSIM and lower values of other indicators may mean that the quality of the corrected images 930 to 970 is better. Also, PSNR, SSIM and AGE may be used. Since PSNR is calculated by MSE, for various comparisons, MSE can be replaced with MAE (Mean Absolute Error).

According to an embodiment, for the implementation of the machine learning model of the present disclosure, a model for 6 epochs with a batch size of 30 in a data set in which the synthetic data set and the real data set are combined can learn Here, the Adam optimizer with β(0.9, 0.999) can be used. The initial learning rate is set to 0.0005, which can be divided by 5 for every 50,000 steps. To do this, PyTorch and NVIDIA Tesla M40 GPUs can be used.

According to an embodiment, EnsNet may be adopted as a criterion to verify the effect of the spatial concentration model of the present disclosure (eg, the spatial concentration model 510) and the nested connection described in FIG. 8 . Since the loss function RRL (Region Regression Loss) defined by the spatial concentration model and Equation 3 uses information associated with text area masks (eg, reference text area masks 410 and 512), the efficiency of text area masks is also discussed. do. Quantitative results are shown in Table 1.

Method Image Eval Detection Eval PSNR SSIM MAE AGE P R F Original Image 79.76 67.24 72.97 BaseLine (EnsNet) 31.05/32.99 94.78/95.16 0.0116/0.0076 2.6657/1.8542 73.07 54.73 62.59 BaseLine+M 35.65/37.28 96.67/96.78 0.0069/0.0039 1.5015/0.8929 63.35 33.22 43.58 BaseLine+M+RAMA 36.12/38.45 97.29/97.46 0.0071/0.0037 1.5106/0.8353 39.07 6.03 10.45 BaseLine(Unet++)+M 37.20/38.40 97.32/97.38 0.0052/0.0034 1.1266/0.7880 50.44 15.07 23.21 BaseLine(Unet++)+M+RAMA 37.27/38.78 97.43/97.48 0.0055/0.0034 1.1634/0.7827 31.59 3.99 7.09 BaseLine+M+RRL -/40.82 -/98.19 -/0.0030 -/0.6565 27.46 3.09 5.56 BaseLine+M+RAMA+RRL -/41.13 -/98.36 -/0.0028 -/0.6348 24.45 2.41 4.39 BaseLine(Unet++)+M+RRL -/40.94 -/98.18 -/0.0029 -/0.6520 10.14 0.69 1.28 BaseLine+M(Unet++)+M+RAMA+RRL -/41.40 -/98.34 -/0.0028 -/0.6462 2.86 0.25 0.45

Table 1 shows the results of an ablation study. The left side of '/' indicates the evaluation result using the output image. The right side of '/' indicates the evaluation result using the composite image generated using the area (box) mask. Here, M is the text area mask, RAMA is the spatial concentration model proposed in the present disclosure, and RRL is the area regression loss. If RRL is used, unmasked areas of the corrected image 970 may become meaningless. Therefore, methods using RRL without synthesis are not evaluated. As a result, Table 1 shows that the proposed loss function RRL significantly improves the quantitative performance in both Image Eval and Detection Eval. In addition, RRL shows that machine learning models can help focus text regions. In Table 1, models with RAMA (i.e., spatially focused models) can achieve higher PSNR and SSIM values. In addition, lower MAE and AGE values than those of the reference model with box mask can be obtained. In particular, in detection evaluation, RAMA obtains much lower Precision, Recall and F-score values than the reference model. The outputs of the generators were partially compared to analyze the detection improvements in detail. Specifically, a composite image is created using a stroke mask and a label mask. The image to be compared appears different only in the masked area. The quantitative results are shown in Table 2. It shows that the RAMA module can improve the image quality of the stroke area.

Method Image Eval BaseLine + M 38.90/44.41 97.34/98.86 0.0026/0.0012 0.6117/0.2812 BaseLine + M + RAMA 44.41/45.16 98.80/99.15 0.0016/0.0012 0.3645/0.2703

Table 2 shows the results of the ablation study. The left side of '/' indicates the evaluation result using the stroke synthesis image. The right side of '/' represents the evaluation result using the synthesized image generated using the label mask. Here, M may refer to a text area (box) mask, and RAMA may refer to a spatial concentration model of the present disclosure.

Method Train
Data Input
Size Image-Eval Detection-Eval PSNR SSIM MAE AGE P R F Original Image 512 79.8 67.2 73.0 Scene Text Eraser Real 256 25.47/- 90.14/- - 6.0069/- 40.9 5.9 10.2 EnsNet Real 512 29.53/- 92.74/- -/- 4.1600/- 68.7 32.8 44.4 MTRNet Syn(75%) 256 -/- -/- -/- -/- - - - MTRNet++ Syn(95%) 256 -/- -/- -/- -/- - - - EraseNet Real 512 32.30/37.26 95.42/96.86 -/- 3.0174/- 53.2 4.6 8.5 EnsNet Real+Syn 512 31.05/32.99 94.78/95.16 0.0116/0.0076 2.6657/1.8542 73.1 54.7 62.6 MTRNet Real+Syn 256 30.61/36.06 89.85/95.72 0.0184/0.0055 3.9225/1.2137 Real+Syn 512 32.46/36.89 95.86/96.41 0.0137/0.0044 3.1222/0.9720 69.8 41.1 51.7 MTRNet++ Real+Syn 256 35.29/37.40 96.31/96.68 0.0053/0.0047 1.2636/1.0889 Real+Syn 512 34.86/36.50 96.32/96.51 0.0060/0.0057 1.4836/1.3509 58.6 20.5 30.4 EraseNet Real+Syn 512 30.54/37.16 96.27/97.53 0.0147/0.0051 3.0736/1.2043 40.8 6.3 10.9 Ours Real+Syn 512 -/41.13 -/98.36 -/0.0028 -/0.6348 24.4 2.4 4.4 Ours (Unet++) Real+Syn 512 -/41.40 -/98.34 -/0.0028 -/0.6462 2.9 0.2 0.5

Table 3 shows the comparison results for SCUT-EnsText. The left side of '/' indicates the evaluation result using the output image. The right side of '/' represents the evaluation result using the composite image generated using the box mask. SSIM at the top collected from EraseNet means MSSIM, and SSIM at the middle and bottom means SSIM. In addition, Real means SCUT-EnsText, Syn means Oxford synthetic Dataset, P means Precision, R means Recall, and F means F-score. The best performance for each metric is shown in bold.

Method Input
Size Image-Eval Detection-Eval PSNR SSIM MAE AGE P R F Original Image 512 71.3 51.5 59.8 EnsNet 512 36.67/39.74 97.71/97.94 0.0059/0.0033 1.2504/0.7737 55.1 14.0 22.3 MTRNet 256 30.96/37.69 90.95/95.83 0.01855/0.0051 4.1711/1.2198 512 35.49/40.03 97.10/97.69 0.0100/0.0035 2.1809/0.8033 58.6 13.7 22.2 MTRNet++ 256 37.40/40.26 97.02/97.25 0.0036/0.0033 0.8577/0.7885 512 38.31/40.64 97.82/97.94 0.0034/0.0032 0.0806/0.7308 64.0 15.9 25.4 EraseNet 512 34.35/41.73 98.01/98.62 0.0106/0.0028 1.8150/0.6621 30.8 0.6 1.2 Ours 512 -/43.35 -/98.61 -/0.0024 -/0.5625 20.6 0.2 0.3 Ours (Unet++) 512 -/43.91 -/98.75 -/0.0023 -/0.5329 12.4 0.1 0.2

Table 4 shows the comparison results for Oxford. The left side of '/' indicates the evaluation result using the output image. The right side of '/' indicates an evaluation result using a composite image generated using a box mask. Here, P means Precision, R means Recall, and F means F-score. All methods were trained on a combined data set of synthetic and real data sets.

Input Size GMAC GPU Time(ms) Params EnsNet 512 21.11 12.03 12.4M MTRNet 256 18.93 21.89 50.3M 512 75.73 51.28 MTRNet++ 256 126.20 69.76 18.7M 512 504.81 238.68 EraseNet 512 92.87 47.35 17.8M Ours 512 21.19 14.46 12.4M Ours (Unet++) 512 67.08 33.72 14.1M

Table 5 shows the comparison results for GMAC (multiply-accumulate operation), GPU Time, and parameter size. In general, overlapping connections with lateral connections can improve performance in all metrics. However, it can be confirmed that the GPU time of the model slightly increases due to the increase in complexity. FIG. 10 shows an example of a method 1000 for deleting text in an image according to an embodiment of the present disclosure. Method 1000 may be performed by at least one processor (eg, processor 334) of an information processing system. As shown, method 1000 may include receiving an image including one or more text objects ( S1010) may be initiated.

According to an embodiment, the processor may receive a text area mask corresponding to the image (S1020). Then, by inputting the received image and the received text area mask to a machine learning model, a corrected image in which at least a part of one or more text objects has been deleted from the received image based on the area surrounding one or more strokes in the received image. can be output (S1030). In this case, the machine learning model receives a reference image and a reference text area mask corresponding to the reference image, and generates a generation model that is trained to generate a first reference correction image based on an area surrounding one or more reference strokes in the reference image. can include

According to an embodiment, the generating model may include a spatial attention model configured to determine information about a region surrounding one or more reference strokes in the reference image by using the reference image and the reference text region mask. In this case, the processor may generate a reference text stroke mask based on a pixel value difference between the reference image and a ground truth image corresponding to the first reference correction image. Then, an intersection area between the outside of the created reference text stroke mask and the inside of the reference text area mask can be created as a reference label mask corresponding to the surrounding area of one or more reference strokes.

According to an embodiment, the spatial concentration model may be learned using a first loss function defined based on a reference label mask and information about a region surrounding one or more determined reference strokes. In this case, the information on the surrounding area of at least one reference stroke may include an attention score map having a probability value that each of a plurality of pixels included in the reference image is included in the surrounding area of the at least one reference stroke. can

According to an embodiment, the generation model may be learned using a reference correct answer image corresponding to the first reference corrected image and a second loss function defined based on the first reference corrected image. Here, the value of the second loss function may be calculated based on the difference between the correct reference image and the first reference correction image and the intersection area of the received text area mask. Then, the processor may extract one or more second reference corrected images from at least some layers of the generated model. In this case, the size of each of the one or more second reference corrected images is different from the size of the first reference corrected image, and the extracted second reference corrected image may be used to define the second loss function.

According to an embodiment, the machine learning model may be configured to output a first reference corrected image by using information about a region around one or more reference strokes and information about a region outside a reference text region mask in the reference image. . Additionally, the machine learning model further includes a discrimination model configured to determine whether the first reference corrected image is included in the distribution of reference correct answer images in order to enhance the generation model, and the distribution of the reference correct answer images uses a plurality of reference correct answer images. can be determined by

The methods, acts or techniques of this disclosure may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or combinations thereof. Those skilled in the art will appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchange of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design requirements imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementations should not be interpreted as causing a departure from the scope of the present disclosure.

In a hardware implementation, the processing units used to perform the techniques may include one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs) ), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, and other electronic units designed to perform the functions described in this disclosure. , a computer, or a combination thereof.

Accordingly, the various illustrative logical blocks, modules, and circuits described in connection with this disclosure may be incorporated into a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or may be implemented or performed in any combination of those designed to perform the functions described in A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, eg, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other configuration.

In firmware and/or software implementation, the techniques include random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), PROM ( on a computer readable medium, such as programmable read-only memory (EPROM), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or optical data storage device, or the like. It can also be implemented with stored instructions. Instructions may be executable by one or more processors and may cause the processor(s) to perform certain aspects of the functionality described in this disclosure.

Although the embodiments described above have been described as utilizing aspects of the presently-disclosed subject matter in one or more stand-alone computer systems, the disclosure is not limited thereto and may be implemented in conjunction with any computing environment, such as a network or distributed computing environment. . Further, aspects of the subject matter in this disclosure may be implemented in a plurality of processing chips or devices, and storage may be similarly affected across multiple devices. These devices may include PCs, network servers, and portable devices.

Although the present disclosure has been described in relation to some embodiments in this specification, various modifications and changes may be made without departing from the scope of the present disclosure that can be understood by those skilled in the art. Moreover, such modifications and variations are intended to fall within the scope of the claims appended hereto.

110: image
120: text area mask
130: generative model
140: correction image
160: discrimination model

Claims (20)

A method of deleting text in an image, performed by at least one processor, comprising:
receiving an image containing one or more text objects;
receiving a text area mask corresponding to the image; and
At least a portion of the one or more text objects is deleted from the received image based on an area surrounding one or more strokes in the received image by inputting the received image and the received text area mask to a machine learning model. Including the step of outputting the correction image,
The machine learning model,
A text in an image comprising a generation model that receives a reference image and a reference text area mask corresponding to the reference image, and is trained to generate a first reference corrected image based on an area around one or more reference strokes in the reference image. how to delete it.
According to claim 1,
The generating model generates text in an image, including a spatial attention model configured to determine information about a region around one or more reference strokes in the reference image using the reference image and the reference text region mask. How to delete.
According to claim 2,
generating a reference text stroke mask based on a pixel value difference between the reference image and a ground truth image corresponding to the first reference correction image; and
generating an intersection area between an outside of the generated reference text stroke mask and an inside of the reference text area mask as a reference label mask corresponding to a peripheral area of the one or more reference strokes; How to delete.
According to claim 3,
Wherein the spatial concentration model is learned using a first loss function defined based on the reference label mask and information about regions surrounding the determined one or more reference strokes.
According to claim 2,
The information on the surrounding area of the one or more reference strokes includes an attention score map having a probability value that each of a plurality of pixels included in the reference image is included in the surrounding area of the one or more reference strokes. , how to delete text within an image.
According to claim 2,
The machine learning model is configured to output the first reference corrected image by using information about a region around the one or more reference strokes and information about a region outside a reference text region mask in the reference image. How to delete.
According to claim 1,
Wherein the generating model is learned using a reference correct answer image corresponding to the first reference corrected image and a second loss function defined based on the first reference corrected image.
According to claim 7,
The value of the second loss function is calculated based on a difference between the correct reference image and the first reference corrected image and an intersection area of the received text area mask.
According to claim 7,
Extracting one or more second reference corrected images from at least some layers of the generated model;
text in an image, wherein the size of each of the one or more second reference corrected images is different from the size of the first reference corrected image, and the extracted one or more second reference corrected images are used to define the second loss function. how to delete it.
According to claim 1,
The machine learning model further includes a discrimination model configured to determine whether the first reference corrected image is included in a distribution of reference correct answer images in order to advance the generation model,
The distribution of the reference correct answer images is determined using a plurality of reference correct answer images,
How to delete text within an image.
A computer program stored in a computer-readable recording medium to execute the method according to any one of claims 1 to 10 on a computer.
As an information processing system,
Memory; and
at least one processor connected to the memory and configured to execute at least one computer readable program contained in the memory
including,
The at least one program,
receive an image containing one or more text objects;
Receive a text area mask corresponding to the image;
At least a portion of the one or more text objects is deleted from the received image based on an area surrounding one or more strokes in the received image by inputting the received image and the received text area mask to a machine learning model. Including instructions for outputting a correction image,
The machine learning model,
An information processing system comprising: a generation model that receives a reference image and a reference text area mask corresponding to the reference image, and is trained to generate a first reference correction image based on an area around one or more reference strokes in the reference image; .
According to claim 12,
wherein the generating model includes a spatial concentration model configured to determine information about a region surrounding one or more reference strokes in the reference image using the reference image and the reference text region mask.
According to claim 13,
The at least one program,
generating a reference text stroke mask based on a pixel value difference between the reference image and a reference correct answer image corresponding to the first reference correction image;
and instructions for generating an intersection area between an exterior of the generated reference text stroke mask and an interior of the reference text area mask as a reference label mask corresponding to a peripheral area of the one or more reference strokes.
According to claim 14,
The information processing system of claim 1 , wherein the spatial concentration model is learned using a first loss function defined based on the reference label mask and information on a region surrounding the determined one or more reference strokes.
According to claim 13,
The information processing system, wherein the information on the surrounding area of the at least one reference stroke includes an attention score map having a probability value that each of a plurality of pixels included in the reference image is included in the surrounding area of the at least one reference stroke.
According to claim 13,
Wherein the machine learning model is configured to output the first reference corrected image by using information on a peripheral region of the one or more reference strokes and information on an external region of a reference text region mask in the reference image.
According to claim 12,
The generating model is learned using a reference correct image corresponding to the first reference corrected image and a second loss function defined based on the first reference corrected image;
The information processing system of claim 1 , wherein a value of the second loss function is calculated based on a difference between the reference correct answer image and the first reference corrected image and an intersection area of the received text area mask.
According to claim 18,
The at least one program,
Includes instructions for extracting one or more second reference corrected images from at least some layers of the generated model;
A size of each of the one or more second reference corrected images is different from the size of the first reference corrected images, and the extracted one or more second reference corrected images are used to define the second loss function. .
According to claim 12,
The machine learning model further includes a discrimination model configured to determine whether the first reference corrected image is included in a distribution of reference correct answer images in order to advance the generation model,
The distribution of the reference correct answer images is determined using a plurality of reference correct answer images,
information processing system.

Images