KR20220046808A - Method and Apparatus for Video Inpainting - Google Patents

Abstract

Disclosed are a method and apparatus for video inpainting, generating a reconstructed video with high quality by performing video inpainting based on inter-frame and/or intra-frame correlation. According to one aspect of the present disclosure, a method of restoring a region occluded by an object included in a video comprises: selecting a target frame from which the object is to be removed and a reference frame including information about the region in the target frame occluded by the object from among frames of the video; and inpainting the target frame by performing at least one of inter-inpainting and intra-inpainting according to the number of frames selected as the reference frame.

Description

Image restoration method and apparatus {Method and Apparatus for Video Inpainting}

This embodiment relates to an image restoration method and apparatus.

The content described in this section merely provides background information on the present invention and does not constitute the prior art.

Various subtitles are added to the video to help viewers understand it or to add fun. For example, in the case of importing foreign video, subtitles translated from the voice of the performer are added. Also, in a noisy environment, subtitles help to convey the content.

However, in some cases, subtitles can become an obstacle to video viewing. For example, a viewer watching an image for learning a foreign language may be interrupted by subtitles. In addition, excessive subtitles cover part of the screen, making it difficult to focus on the video. Furthermore, if foreign language subtitles are added and the subtitles are not understood, it is better not to have subtitles.

When the subtitle and the video exist as separate channels, it is not difficult to edit or remove the subtitle. However, there is a problem in the case where the subtitles are integrated with the video and the original video before adding the subtitles is not retained. In this case, the subtitles are removed by blurring the region containing the subtitles, adding a translucent and/or opaque band over the region containing the subtitles, and in extreme cases cutting the entire region containing the subtitles from the video. You are destroying the image to cover it up.

Accordingly, there is an increasing demand for a technique for reconstructing an area covered by subtitles without damaging an image.

With the existing image restoration technology, a diffusion-based method that restores the occluded region by referring to information on surrounding pixels of the region to be removed, and a screen divided into several regions and then selected an appropriate region There is a patch-based method that replaces a region, and the like.

An embodiment of the present disclosure selects a target frame from which subtitles are to be removed from an input image and a reference frame to obtain information about an area covered by the subtitles, and based on this, image restoration based on inter-frame correlation and/or to provide an image restoration method and apparatus for generating a restored image having high quality by performing intra-frame correlation-based image restoration.

According to an aspect of this embodiment, in the method of reconstructing an area covered by an object included in an image, a target frame from which the object is to be removed from among the frames of the image and within the target frame covered by the object selecting a reference frame including information on a region; and performing at least one of inter-inpainting and intra-inpainting according to the number of frames selected as the reference frame to restore the target frame. .

According to another aspect of this embodiment, in the apparatus for reconstructing a region obscured by an object included in an image, information about a target frame from which the object is to be removed and a region within the target frame obscured by the object is provided. a frame selection unit for selecting a reference frame including; and an inpainting unit configured to generate a restored frame by performing at least one of inter-inpainting and intra-inpainting according to the number of reference frames. do.

As described above, according to the embodiment of the present disclosure, a target frame from which a caption is to be removed from an input image and a reference frame from which information about an area covered by the caption are obtained are selected, and based on this, a reference frame is selected. By performing inter-frame correlation-based image restoration and/or intra-frame correlation-based image restoration, it is possible to generate a restored image having high quality.

Furthermore, according to the embodiment of the present disclosure, since the original image before editing can be obtained from the image on which the subtitle editing is completed, there is an effect that the original image purchase cost, the original image storage cost, and the image processing cost can be reduced.

1 is a block diagram schematically showing an image restoration apparatus according to an embodiment of the present disclosure.
2 to 4C are exemplary views for explaining the frame selector according to the first embodiment of the present disclosure.
5 is an exemplary view for explaining a frame selector according to a second embodiment of the present disclosure.
6 is a flowchart illustrating a frame selection process according to a second embodiment of the present disclosure.
7 is an exemplary diagram for explaining a network of an inter recovery unit according to an embodiment of the present disclosure.
8A to 8B are exemplary diagrams for explaining calculation of a similarity between a target frame and a reference frame according to an embodiment of the present disclosure.
9 is an exemplary diagram illustrating feature indexing of a reference frame for attention matching according to an embodiment of the present disclosure.
10 is an exemplary diagram for explaining an autoencoder network having an asymmetric input/output structure according to an embodiment of the present disclosure.
11 is an exemplary diagram for explaining an intra restoration unit according to an embodiment of the present disclosure.
12 is an exemplary diagram for explaining a coarse prediction unit according to an embodiment of the present disclosure.
13 is an exemplary view for explaining a precision processing unit according to an embodiment of the present disclosure.
14 is an exemplary diagram for explaining merge network learning according to an embodiment of the present disclosure.

Hereinafter, some embodiments of the present disclosure will be described in detail with reference to exemplary drawings. In adding reference numerals to the components of each drawing, it should be noted that the same components are given the same reference numerals as much as possible even though they are indicated on different drawings. In addition, in describing the present disclosure, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present disclosure, the detailed description thereof will be omitted.

In addition, in describing the components of the present disclosure, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the elements from other elements, and the essence, order, or order of the elements are not limited by the terms. Throughout the specification, when a part 'includes' or 'includes' a certain element, this means that other elements may be further included, rather than excluding other elements, unless otherwise stated. . In addition, the '... Terms such as 'unit' and 'module' mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software.

Hereinafter, various embodiments of the present disclosure will be described using a subtitle as an object to be removed by the image restoration apparatus. However, this is only for convenience of description, and the present disclosure is not limited to these embodiments. For example, the image restoration apparatus according to an embodiment of the present disclosure may remove a specific trademark and/or logo from an image.

1 is a block diagram schematically showing an image restoration apparatus according to an embodiment of the present disclosure.

Referring to FIG. 1 , an image restoration apparatus 10 according to an embodiment of the present disclosure includes an input frame DB (DB) 100, a frame metadata DB (DB) 110, and a mask generation unit. unit 120, a frame selection unit 130, an inpainting unit 140, a frame merging unit 150, and an output frame DB (output frame DB, 160) all or includes some Not all blocks shown in FIG. 1 are essential components, and in another embodiment, some blocks included in the image restoration apparatus 10 may be added, changed, or deleted. For example, when only one of inter restoration and intra restoration is selectively performed on a target frame, the image restoration apparatus may not include the frame merging unit 150 .

The input frame DB 100 stores the input image including the subtitle to be removed as an image in units of frames.

The frame meta information DB 110 extracts and stores scene change information, subtitle location information, text information used for subtitles, and/or font information used for subtitles from the input image.

The mask generator 120 generates a mask image representing the caption area based on the text and font used for the caption.

The frame selector 130 selects a target frame from which a caption is to be removed and a reference frame from which information on an area covered by the caption is obtained from among the input frames, based on the frame meta information and/or the mask image. do. A detailed description of the frame selector 130 will be described with reference to FIGS. 2 to 6B .

The restoration unit 140 removes the subtitle from the target frame based on the target frame, the reference frame, and the mask image, and restores an area covered by the subtitle. In this disclosure, 'restore' is used synonymously with inpainting, which reconstructs pixels in an area obscured by the subtitle 202 with reference to pixels in other frames and/or pixels in other positions in other input frames. do. The restoration unit includes an inter restoration unit 142 and an intra inpainting unit 144 . The inter reconstructor 142 reconstructs the target frame based on inter-frame correlation, whereas the intra reconstructor 144 reconstructs the target frame based on intra-frame correlation. do. Hereinafter, inter-frame correlation-based restoration is referred to as inter restoration, and intra-frame correlation-based restoration is referred to as intra restoration.

The restoration unit 140 according to an embodiment of the present disclosure performs at least one of an inter restoration and an intra restoration based on the information on the target frame and the reference frame selected by the frame selection unit 130 . A detailed description of the inter restoration unit 142 and the intra restoration unit 144 will be described with reference to FIGS. 7 to 13 .

The frame merging unit 150 synthesizes the output of the inter restoration unit 142 and the output of the intra restoration unit 144 , or outputs one of the output of the inter restoration unit 142 and the output of the intra restoration unit 144 . Select it and use it as a restoration frame. A detailed description of the frame merging unit 150 will be described with reference to FIG. 14 .

The output frame DB 160 stores a frame of an output image from which subtitles are removed.

Hereinafter, a frame selector according to a first embodiment of the present disclosure will be described with reference to FIGS. 2 to 4C .

2 to 4C are exemplary views for explaining the frame selector according to the first embodiment of the present disclosure.

Referring to FIG. 2 , the frame selector 130 according to an embodiment of the present disclosure performs a target frame and a target frame from a previous frame positioned before the target frame and/or a subsequent frame positioned later than the target frame in time. A forward reference frame and a backward reference frame having high similarity are selected, and similar region information for the reference frames is extracted. Here, the forward reference frame refers to a frame having a high similarity to the target frame among previous frames, and the backward reference frame refers to a frame having a high similarity to the target frame among subsequent frames.

The frame selector 130 provides the target frame, the reference frame, and similar region information on the reference frames as an input to the deep neural network provided in the reconstruction unit 140 .

Hereinafter, before describing a method of extracting similar region information by the frame selector 130 according to an embodiment of the present disclosure, terms used in the present disclosure will be described with reference to FIG. 3 .

Referring to FIG. 3A , the detection region 302 indicates a region in which a caption is detected within a frame or a specific region 300 of the frame. According to an embodiment of the present disclosure, the frame selector 130 may detect a caption using optical character recognition (OCR), but is not limited to this example. A neighbor area 304 indicates an area separated by a predetermined distance from the detection area 302 and/or a rectangular area including the corresponding area.

Referring to FIG. 3B , a frame or a specific area 300 of a frame is divided into an outer area 300a, an inner area 300b, and a caption area 300c based on the detection area 302 and the caption. can be The outer region 300a refers to an area in which a caption is not detected, that is, an area outside the detection area 302 . The inner region 300b refers to a region that does not correspond to a caption within the detection region 302 . The caption area 300c is an area corresponding to a caption within the detection area 302 and refers to an area in which caption removal and restoration are to be performed.

A method of extracting similar region information by the frame selector 130 according to an embodiment of the present disclosure will be described with reference to FIGS. 4A to 4C based on the above terms. Meanwhile, a method of extracting similar region information from two previous frames will be described below, but this is for convenience of description, and this method may be equally applied to one or more previous frames and/or one or more subsequent frames.

4A illustrates an example in which a circular object moves to the lower right corner of a frame over time. 4a (c) shows the target frame from which the subtitle is to be removed at time t (f _t , 420), and (a) and (b) of FIG. 4A show the previous frames (f _t-2, f _t-1 , 400 and 410). 4A (a) and (b) show an example in which the caption is added to the previous frames 400 and 410, that is, the detection regions 402 and 412 are included in the previous frames 400 and 410, but , this is for convenience of explanation, and subtitles may not be added to the previous frames 400 and 410 , or subtitles with different contents from the target frame 420 may be added.

Referring to FIG. 4B , the frame selector 130 according to an embodiment of the present disclosure uses the surrounding area 424 of the target frame 420 to select the target frame 420 from previous frames 400 and 410 . Template matching is performed to find similar regions 430 and 440 having the highest similarity to .

Meanwhile, although FIG. 4B shows an example of performing template matching using all of the pixels surrounding the detection area 422 , template matching may be performed by dividing the pixels surrounding the detection area 422 into blocks. Furthermore, the method of performing template matching encompasses all methods that can be easily employed by those skilled in the art, and is not limited to a specific method.

The frame selector 130 according to an embodiment of the present disclosure may calculate the similarity between specific regions using sum of absolute differences (SAD) and/or mean square difference (MSD), but is not limited to these examples. , any person skilled in the art will be able to calculate the similarity between specific regions using other methods. Meanwhile, the frame selector 130 does not calculate the similarity of the target frame 420 and the inner regions of the previous frames 400 and 410 .

The frame selector 130 according to an embodiment of the present disclosure selects previous frames 400 and 410 in which similar regions 430 and 440 having a similarity higher than a preset threshold similarity are detected as backward reference frames. In other words, when there is no region having a similarity higher than the preset threshold similarity in the previous frames 430 and 440, the previous frame is not selected as a backward reference frame.

The frame selector 130 according to an embodiment of the present disclosure extracts similar region information and distance information between the target frame and the previous frame with respect to the previous frames 400 and 410 selected as the backward reference frame.

The similar region information may include a degree of similarity, location information of the similar region, and/or information on one or more regions subdivided from the similar region.

Referring to FIG. 4C , the frame selector 130 according to an embodiment of the present disclosure divides the similar regions 430 and 440 into external regions 430a and 440a, internal regions 430b and 440b, and a caption region 440c. ), and a different index and/or weight is given to each area. For example, a value of “2” is assigned to the outer regions 430a and 440a, a value of “1” is assigned to the inner regions 430b and 440b, and a value of “0” is assigned to the subtitle region 440c. can be given The frame selector 130 may re-classify the similar regions 430 and 440 into valid regions, important regions, reference regions, and the like, by using the assigned indexes and/or weights.

On the other hand, the frame selector 130 according to another embodiment of the present disclosure selects a reference frame by using a motion vector of a video codec instead of template matching, information on a similar region, and a target Distance information between the frame and the reference frame may be extracted.

Hereinafter, a frame selector according to a second embodiment of the present disclosure will be described with reference to FIGS. 5 to 6 .

5 is an exemplary view for explaining a frame selector according to a second embodiment of the present disclosure.

5, the frame selection unit 130 according to an embodiment of the present disclosure includes a flow control unit 500, a backward reference frame queue 510, a target frame list 520, and a forward reference frame queue ( 530) in whole or in part. Not all blocks shown in FIG. 5 are essential components, and in another embodiment, some blocks included in the frame rate improving apparatus 10 may be added, changed, or deleted.

The flow control unit 500 determines a target frame, a backward reference frame, and a forward reference frame from among the input frames based on frame meta information. In this case, the reference frame refers to a frame that is adjacent in time to the target frame and has a pixel value with high correlation that can replace the subtitle of the target frame.

Specifically, the reference frame does not include the subtitle or has a subtitle region that does not overlap the subtitle region of the target frame even if it does include the subtitle. In addition, the backward reference frame refers to a reference frame located temporally earlier than the target frame, and the forward reference frame refers to a reference frame located temporally later than the target frame.

According to an embodiment of the present disclosure, the flow control unit 500 determines the target frame by sequentially checking the input frames in chronological order, and the closest backward reference frame and/or forward direction in time based on the determined target frame. Allows the reference frame to be loaded into the backward reference frame queue 510 and/or the forward reference frame queue 530 .

According to an embodiment of the present disclosure, the flow controller 500 determines a target frame and a reference frame in units of scenes based on scene change information provided from the frame meta information DB 110 . That is, the flow control unit 500 determines a target frame and a reference frame among input frames in the same scene. Specifically, when a scene change occurs, the flow control unit 500 initializes the backward reference frame queue 510 and the forward reference frame queue 530 so that a frame having a low correlation with the target frame is not used as the reference frame.

According to an embodiment of the present disclosure, the flow control unit 500 configures N (N = natural number) frames as one frame processing unit, the number of target frames (T), and the number of backward reference frames (B) , and when the sum of the number F of omnidirectional reference frames becomes N, the target frame and/or the reference frame are provided to at least one of the inter restoration unit 142 and the intra restoration unit 144 .

Specifically, when there is no reference frame within the N frame processing units (B + F = 0), the flow control unit 500 selects M (M = natural number) target frames among T target frames in the target frame list 520 . It is provided to the intra restoration unit 144 and added to the backward reference frame queue 510 in order to use the restored M frames as reference frames. On the other hand, when there are reference frames within the N frame processing units (B + F > 0), the flow control unit 500 converts the T target frames, the B backward reference frames, and the F forward reference frames to the inter restoration unit. (142). In this case, the flow control unit 500 may also provide the T target frames to the intra restoration unit 144 .

A detailed description of a process in which the flow controller 500 determines the target frame and the reference frame will be described with reference to FIGS. 6A and 6B .

The backward reference frame queue 510 stores the backward reference frame determined by the flow control unit 500 . The forward reference frame queue 530 stores the forward reference frame determined by the flow control unit 500 . Each of the reference frame queues 510 and 530 inserts a new reference frame and/or removes the oldest reference frame based on a preset maximum number of frames, so that the backward reference frame closest in time to the target frame and/or Alternatively, the forward reference frame may be loaded into the backward reference frame queue 510 and/or the forward reference frame queue 530 . Each of the reference frame queues 510 and 530 provides at least one backward reference frame and/or at least one forward reference frame to the inter restoration unit 142 under the control of the flow control unit 500 .

The target frame list 520 stores the target frame determined by the flow control unit 500 . The target frame list 520 provides at least one target frame to at least one of the inter restoration unit 142 and the intra restoration unit 144 under the control of the flow control unit 500 .

6 is a flowchart illustrating a frame selection process according to a second embodiment of the present disclosure.

The flow control unit 500 checks whether a subtitle to be removed exists in the n (n = natural number)-th frame among the input frames (S600). If there is no subtitle to be removed in the nth frame, the flow controller 500 adds the nth frame to the backward reference frame queue 510 (S602), increases n by 1, and restarts the frame selection process from the beginning. (S604). That is, the flow control unit 500 sequentially checks the input frames in chronological order to confirm whether there is a subtitle to be removed in the frame.

When there is a caption to be removed in the nth frame, the flow controller 500 checks whether a backward reference frame exists in the backward reference frame queue 510 (S610).

If there is no backward reference frame, the flow control unit 500 detects a backward reference frame from among the frames for which subtitle removal has been previously processed, and adds it to the backward reference frame queue 510 ( S612 ). Specifically, the flow control unit 500 detects a frame having a caption area that does not overlap the caption area of the nth frame from among the frames for which caption removal has been previously processed as a backward reference frame, and stores the frame in the backward reference frame queue 510 . add

The flow control unit 500 detects the forward reference frame from among the frames located later in time from the nth frame, and adds it to the forward reference frame queue 530 ( S620 ). Specifically, the flow control unit 500 sequentially checks whether subtitles exist from the n+1th frame to the frame at the time before the next scene change occurs, and sets the frame without the subtitle to be removed to the forward reference frame queue ( 530) is added. That is, the flow control unit 500 adds to the omnidirectional reference frame queue 530 from the omnidirectional reference frame that is closest in time order from the nth frame, and adds omnidirectional reference frames less than or equal to the preset maximum number of frames to the omnidirectional reference frame queue. (530) is added.

The flow control unit 500 adds the nth frame to the target frame list 520 (S630).

The flow control unit 500 is the sum of the number of target frames stored in the target frame list 520 and the number of reference frames stored in the backward reference frame queue 510 and the forward reference frame queue 530 (T+B+F). ) is greater than N, which is a frame processing unit (S640).

When the sum of the number of target frames and the number of reference frames is smaller than N, which is a frame processing unit, the flow control unit 500 checks whether the nth frame is the last frame (S642). Here, the last frame means the last frame among all input frames or the last frame before the next scene change occurs.

If the nth frame is not the last frame, the flow control unit 500 increases n by 1 and restarts the frame selection process from the beginning (S604). That is, the flow controller 500 adjusts the number of frames to be restored based on the frame processing unit and the scene change time.

When the sum of the number of target frames and the number of reference frames is greater than N, which is a frame processing unit, or when the nth frame is the last frame, the flow controller 500 checks whether the number of reference frames is 0.

When the number of reference frames is 0, the flow control unit 500 provides M (M = natural number) frames among the target frames stored in the target frame list 520 to the intra restoration unit 144, and sets the M frames to the target. It is removed from the frame list 520 (S652). Accordingly, the intra restoration unit 144 performs intra restoration on the M target frames.

The flow control unit 500 adds the M frames restored by the intra restoration unit 144 to the backward reference frame queue 510 for use as reference frames (S654).

If the number of reference frames is not 0 as a result of the check in step S650 or after adding the reference frames to the backward reference frame queue 510 in step S654, the flow controller 500 checks whether the number of target frames is 0 (S660) .

When the number of target frames is not 0, the flow control unit 500 inter-restores the target frame stored in the target frame list 520 and the reference frame stored in the backward reference frame queue 510 and the forward reference frame queue 530 . It is provided to the unit 142, and the target frame list 520 is initialized (S662). Accordingly, the inter restoration unit 142 performs intra restoration on all target frames stored in the target frame list 520 . Meanwhile, the flow control unit 500 according to another embodiment of the present disclosure may provide the target frame stored in the target frame list 520 to the intra restoration unit 144 before initializing the target frame list 520 . .

The flow control unit 500 checks whether the n-th frame is the last frame (S670). Here, the last frame means the last frame among all input frames or the last frame before the next scene change occurs.

If the nth frame is not the last frame, the flow control unit 500 increases n by 1 and restarts the frame selection process from the beginning (S604). That is, the flow control unit 500 selects one or more target frames from which subtitles are to be removed from frames located later in time from the nth frame, and selects a frame having high correlation with the target frames as a reference frame.

If the n-th frame is not the last frame, the flow control unit 500 ends the frame selection process. On the other hand, when the n-th frame is the last frame before the next scene change occurs, the flow control unit 500 initializes the backward reference frame queue 510 and the forward reference frame queue 530, and starts the frame selection process for the first time. can be redone from

Hereinafter, an inter restoration unit according to an embodiment of the present disclosure will be described with reference to FIGS. 7 to 10 .

The inter reconstructor 142 obtains new pixel values to replace the pixel value of the subtitle position to be removed from the target frame from the reference frame with high correlation with the target frame by using the similarity between the target frame and the reference frame, and synthesizes them to perform local image restoration.

7 is an exemplary diagram for explaining a network of an inter recovery unit according to an embodiment of the present disclosure.

Referring to FIG. 7 , the inter reconstructor 142 uses an encoder network 700 to 706 learned with a neural network-based autoencoder, and features a target frame and a reference frame. ) as a value feature, a key feature, and a query feature.

The inter restoration unit 142 performs attention matching between the query characteristics of the surrounding region adjacent to the subtitle region in the target frame and the key characteristics of the reference frame using an attention matching unit 710, and An attention score is calculated. The inter reconstructor 142 finds a similar region having high similarity to the detection region of the target frame from the reference frame based on the attention score. The attention matching unit 710 according to an embodiment of the present disclosure will be described later with reference to FIGS. 8A to 9 .

The inter reconstructor 142 generates a compensated value feature for filling the subtitle region of the target frame, and inputs it to a decoder network 720 to generate a final reconstructed pixel.

Hereinafter, the attention matching unit 710 according to an embodiment of the present disclosure will be described with reference to FIGS. 8A to 9 .

First, a method for calculating the similarity between a target frame and a reference frame by using the modified 3D convolution by the attention matching unit 710 according to an embodiment of the present disclosure will be described.

8A to 8B are exemplary diagrams for explaining calculation of a similarity between a target frame and a reference frame according to an embodiment of the present disclosure.

8A illustrates a similarity calculation method using characteristic information of an inpainting area according to an embodiment of the present disclosure.

A conventional attention matching unit obtains a degree of similarity between a key feature vector and a query feature vector, that is, a degree of similarity between vectors, by using matrix multiplication. However, since this method only calculates the correlation for a point that has passed through the neural network, that is, a small region, it is impossible to calculate the similarity between large regions using surrounding pixels to be restored. On the other hand, the attention matching unit 710 according to an embodiment of the present disclosure uses a convolution to use a similarity between a key feature matrix and a query feature matrix, that is, between a matrix look for similarity.

Referring to FIG. 8A , an encoder according to an embodiment of the present disclosure receives a target frame and a reference frame having a size of (W, H) and generates a three-dimensional matrix having a size of (W', H', F). create Here, W' and H' are frame sizes scaled by the encoder, and F is the number of features extracted by the encoder. Similarly, when the size of the reconstructed area included in the target frame is (iW, iH), the size of the reconstructed area scaled by the encoder is (iW', iH'). Accordingly, the query feature matrix obtained by extracting only the 3D matrix of the region to be restored from the 3D matrix of the target frame has a size of (iW', iH', F). On the other hand, the key feature matrix is a three-dimensional matrix for the reference frame, and has a size of (W', H', F).

The attention matching unit 710 according to an embodiment of the present disclosure calculates a similarity matrix by using a modified 3D convolution between the query feature matrix and the key feature matrix. At this time, if the modified three-dimensional convolution is expressed as an equation, it is shown in Equation 1.

Here, S is a similarity matrix, Q is a query feature matrix, and K is a key feature matrix.

The attention matching unit 710 according to an embodiment of the present disclosure converts the similarity matrix into probability information by applying a SoftMax or SparseMax function to the similarity matrix.

As described above, the conventional attention matching unit calculates only the similarity between specific points of the target frame and the reference frame by using the matrix product between the two-dimensional matrices after converting the three-dimensional matrix into a two-dimensional matrix, but in an embodiment of the present disclosure Accordingly, the attention matching unit 710 may calculate the similarity between the specific region of the target frame and the reference frame by using the convolution between the 3D matrices.

8B illustrates a similarity calculation method using characteristic information of a neighbor area according to an embodiment of the present disclosure.

As described in FIG. 8A , the attention matching unit 710 according to an embodiment of the present disclosure extracts features of a restoration area to be restored and calculates a degree of similarity. On the other hand, referring to FIG. 8B , the attention matching unit 710 according to another embodiment of the present disclosure generates a query feature matrix by extracting features of a region surrounding the restored region, not the restored region.

Hereinafter, a method for the attention matching unit 710 according to an embodiment of the present disclosure to adaptively perform feature indexing according to an estimated amount of movement between a target frame and a reference frame will be described.

9 is an exemplary diagram illustrating feature indexing of a reference frame for attention matching according to an embodiment of the present disclosure.

The conventional attention matching unit 710 indexes and uses all available features of the reference frame to perform attention matching with the features of the restored region of the target frame. However, statistically, since the amount of movement occurring between adjacent frames in the image sequence is small, there is a high possibility of obtaining a high attention score when calculating using features extracted from the location adjacent to the restoration area of the target frame among all available features of the reference frame. big. That is, indexing the features of the reference frame into the entire frame area requires a lot of computation, and may cause unexpected quality degradation due to incorrect attention matching.

The attention matching unit 710 according to an embodiment of the present disclosure variably performs feature indexing of the reference frame by estimating the amount of movement of the target frame and the reference frame.

Referring to FIG. 9 , the attention matching unit 710 according to an embodiment of the present disclosure is effective based on regions 912 , 922 , and 932 at the same location as the restoration region 902 of the target frame within the reference frame. The valid areas 914, 924, and 934 are extended to the periphery. In this case, the effective region means a region to be subjected to feature indexing. That is, the attention matching unit 710 performs feature indexing on features included in the effective regions 914 , 924 , and 934 .

The attention matching unit 710 according to an embodiment of the present disclosure estimates the amount of movement between the target frame and the reference frame, and determines the extent of expansion of the effective area based on the estimated amount of movement. That is, the attention matching unit 710 adaptively adjusts the sizes of the effective regions 914 , 924 , and 934 according to the estimated amount of movement between the target frame and the reference frame. For example, the attention matching unit 710 reduces the size of the effective region when the estimated amount of movement between the target frame and the reference frame is small, and increases the size of the effective region when the amount of movement estimated between the target frame and the reference frame is large. increase

Referring to FIG. 9 , the first reference frame feature map 910 is a feature map extracted from a reference frame with a small amount of motion, and the second reference frame feature map 920 is extracted from a reference frame with an intermediate amount of movement. It is a feature map, and the third reference frame feature map 930 represents a feature map extracted from a reference frame having a large amount of motion. Therefore, as shown in FIG. 9 , the size of the effective region 914 of the first reference frame feature map 910 is determined to be the smallest, and the size of the effective region 934 of the third reference frame feature map 930 is the largest. is decided

The attention matching unit 710 according to an embodiment of the present disclosure may estimate the amount of movement based on a temporal distance between the target frame and the reference frame. For example, the attention matching unit 710 may estimate that the greater the temporal distance between the target frame and the reference frame, the greater the amount of movement. The attention matching unit 710 according to another embodiment of the present disclosure may estimate a movement amount according to how much pixels in an area adjacent to the restored area of the target frame have moved from the reference frame. The method of estimating the amount of movement of the attention matching unit 710 is not limited to the above-described example, and any person skilled in the art may estimate the amount of movement between frames using other methods.

Hereinafter, an autoencoder network having an asymmetric input/output structure according to an embodiment of the present disclosure will be described with reference to FIG. 10 .

10 is an exemplary diagram for explaining an autoencoder network having an asymmetric input/output structure according to an embodiment of the present disclosure.

The conventional autoencoder network is designed in a symmetrical structure in which the resolution of the input image and the resolution of the output image are the same. Therefore, as the resolution of the input image increases, the amount of computation and memory usage of the network increases significantly. The process is divided into 4 steps.

Here, the downsampler downsamples an input image having a size of (W, H) including a reconstructed area to a size of (W', H') and input it to the autoencoder network. The autoencoder network performs restoration on the restoration area included in the input image, and may correspond to the network of the inter restoration unit 142 as described in the description with reference to FIG. 7 . The upsampler upsamples the output of the autoencoder network having the size of (W', H') and restores the output image to the original resolution of the size (W, H). The synthesizing unit generates a final reconstructed image by synthesizing a reconstructed area of the output image with a region excluding the reconstructed area from the input image.

Referring to FIG. 10 , the autoencoder network 1000 according to an embodiment of the present disclosure has an asymmetric input/output structure in which a resolution of an input image and a resolution of an output image are different. The output terminal of the decoder network 1010 according to an embodiment of the present disclosure includes an upsampling network layer 1020 capable of outputting an image having the same resolution as that of the input image. That is, according to an embodiment of the present disclosure, the process of upsampling the output image to restore the original resolution is not performed by a separate upsampler interworking with the autoencoder network 1000 , but a decoder in the autoencoder network 1000 . It is performed in the upsampling network layer 1020 fused to the network 1010 . Accordingly, the autoencoder network can learn image restoration and upsampling together, so that a higher resolution result can be obtained than general upsampling methods such as bicubic.

Hereinafter, an intra restoration unit according to an embodiment of the present disclosure will be described with reference to FIGS. 11 to 13 .

11 is an exemplary view for explaining an intra restoration unit according to an embodiment of the present disclosure.

The intra restoration unit 144 according to an embodiment of the present disclosure obtains, from another position in the target frame, a new pixel value to replace the pixel value of the subtitle position to be removed from the target frame, by using the similarity within the target frame. By synthesizing, local image reconstruction is performed.

Referring to FIG. 11 , the intra reconstruction unit 144 according to an embodiment of the present disclosure includes a coarse prediction unit 1100 and a refinement processing unit 1110 . The coarse prediction unit 1100 primarily generates a coarse predicted frame based on the target frame and the mask image. The precision processing unit 1110 receives an inaccurate prediction frame and finally generates an inpainted frame.

12 is an exemplary diagram for explaining a coarse prediction unit according to an embodiment of the present disclosure.

The coarse prediction unit 1100 according to an embodiment of the present disclosure roughly predicts a new pixel value for replacing the region from which the caption is removed by analyzing the image characteristics of the target frame.

Referring to FIG. 12 , the coarse prediction unit 1100 according to an embodiment of the present disclosure includes an encoder network 1200 and a decoder network 1210 trained by a neural network-based autoencoder. The coarse prediction unit 1100 receives a target frame and a mask image, and generates a coarse prediction frame in which the caption removal region is replaced with a new pixel value.

13 is an exemplary view for explaining a precision processing unit according to an embodiment of the present disclosure.

13, the precision processing unit 1110 according to an embodiment of the present disclosure uses the encoder networks 1300 to 1308 learned with a neural network-based autoencoder, a value feature as a feature of a target frame, Extract a key feature, and a query feature.

The precision processing unit 1110 performs attention matching between the query characteristics of the subtitle removal area and the key characteristics of other areas in the target frame using an attention matching unit 1310, and performs an attention score (attention matching). score) is calculated. The precision processing unit 1110 searches for a similar region having high similarity to the subtitle region in the target frame based on the attention score. The attention matching unit 1310 according to an embodiment of the present disclosure performs attention matching in the same manner as the attention matching unit 710 of the inter restoration unit 142 as described in the description with reference to FIGS. 8A to 9 . can

The precision processing unit 1110 generates a compensation value feature vector for filling the subtitle region of the target frame, and inputs it to the decoder network 1320 to generate a final reconstructed pixel.

Hereinafter, a frame merging unit according to an embodiment of the present disclosure will be described with reference to FIG. 14 .

The frame merging unit 150 according to an embodiment of the present disclosure synthesizes the output of the inter restoration unit 142 and the output of the intra restoration unit 144 , or the output of the inter restoration unit 142 and the intra restoration unit 144 . ), one output is selected and used as the final output frame.

The frame merging unit 150 according to an embodiment of the present disclosure may synthesize two reconstructed frames using the attention score or select one reconstructed frame from among the two reconstructed frames. Specifically, the frame merging unit 150 may select a restored frame output by the restoration unit for which the inter restoration unit 142 and the intra restoration unit 144 has calculated a higher attention score as the final output frame. Alternatively, the frame merging unit 150 may generate a final output frame by synthesizing two reconstructed frames by the ratio of the attention scores calculated by the inter reconstructor 142 and the intra reconstructor 144, respectively.

For example, when the attention score of the inter restoration unit 142 is 80 and the attention score of the intra restoration unit 144 is 30, the frame merging unit 150 receives the restored frame output from the inter restoration unit 142 . A final reconstructed frame may be generated by selecting it as the final output frame or synthesizing the reconstructed frame output by the inter reconstructor 142 and the reconstructed frame output by the intra reconstructor 144 at a ratio of 80:30.

The frame merging unit 150 according to another embodiment of the present disclosure may synthesize two reconstructed frames using a pretrained model or select one reconstructed frame from among the two reconstructed frames. Specifically, the frame merging unit 150 uses a pre-learned network for segmentation from the reconstructed frame output by the inter reconstructor 142 and the reconstructed frame output by the intra reconstructor 144, respectively. to extract The frame merging unit 150 compares the extracted features and selects a reconstructed frame from which more features are extracted as the final output frame, or generates a final output frame by synthesizing two reconstructed frames according to a ratio from which features are extracted.

For example, the frame merging unit 150 may use VGGNet 16 as a pre-trained model. The frame merging unit 150 inputs the restored frame output by the inter restoration unit 142 and the restored frame output by the intra restoration unit 144 into VGGNet to extract features. The frame merging unit 150 determines which reconstructed frame is better reconstructed by comparing the extracted feature values. For example, when the total sum of features or energy extracted from the restored frame output by the inter restoration unit 142 is greater than the total sum of features extracted from the restored frame output by the intra restoration unit 144, the inter The restored frame output by the restoration unit 142 is used as the final output frame.

The frame merging unit 150 according to another embodiment of the present disclosure may learn a neural network to synthesize two reconstructed frames or select one reconstructed frame from among the two reconstructed frames. Specifically, the frame merging unit 150 performs a merge network to select a better restored frame among the restored frame output by the inter restoration unit 142 and the restored frame output by the intra restoration unit 144 . can learn

14 is an exemplary diagram for explaining merge network learning according to an embodiment of the present disclosure.

Referring to FIG. 14 , the merge network 1400 is a Convolutional Neural Networks (CNN)-based artificial neural network. When two frames are input, one frame is output. According to an embodiment of the present disclosure, the learning unit 1410 calculates a loss between a frame output from the merging network and an original frame or GT (Ground Truth) frame, and merges in a direction in which the loss is reduced. The network 1400 is trained. That is, the learning unit 1410 trains the merging network 1400 so that the merging network 1400 outputs a frame similar to the original frame or the GT frame.

According to an embodiment of the present disclosure, the learner 1410 may calculate a difference between the frame output from the merge network and the original frame or the GT frame before the subtitle is edited as a loss. Specifically, the learning unit 1410 calculates a loss by using a method such as Mean Absolute Difference (MAD), Mean Square Difference (MSD), etc. or the sum or energy of features extracted using a pretrained network. The difference can be calculated as a loss.

Although it is described that each process is sequentially executed in FIGS. 6A and 6B , this is merely illustrative of the technical idea of an embodiment of the present disclosure. In other words, those of ordinary skill in the art to which an embodiment of the present disclosure pertain change the order described in FIGS. 6A and 6B without departing from the essential characteristics of an embodiment of the present disclosure, or perform one or more processes Since various modifications and variations can be applied by executing in parallel, FIGS. 6A and 6B are not limited to a time-series order.

Various implementations of the systems and techniques described herein may be implemented in digital electronic circuitry, integrated circuitry, field programmable gate array (FPGA), application specific integrated circuit (ASIC), computer hardware, firmware, software, and/or combination can be realized. These various implementations may include being implemented in one or more computer programs executable on a programmable system. The programmable system includes at least one programmable processor (which may be a special purpose processor) coupled to receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device. or may be a general-purpose processor). Computer programs (also known as programs, software, software applications or code) contain instructions for a programmable processor and are stored on a "computer-readable recording medium".

The computer-readable recording medium includes all types of recording devices in which data readable by a computer system is stored. These computer-readable recording media are non-volatile or non-transitory, such as ROM, CD-ROM, magnetic tape, floppy disk, memory card, hard disk, magneto-optical disk, and storage device. It may be a medium, and may further include a transitory medium such as a data transmission medium. In addition, the computer-readable recording medium may be distributed in a network-connected computer system, and the computer-readable code may be stored and executed in a distributed manner.

Various implementations of the systems and techniques described herein may be implemented by a programmable computer. Here, the computer includes a programmable processor, a data storage system (including volatile memory, non-volatile memory, or other types of storage systems or combinations thereof), and at least one communication interface. For example, a programmable computer may be one of a server, a network appliance, a set-top box, an embedded device, a computer expansion module, a personal computer, a laptop, a Personal Data Assistant (PDA), a cloud computing system, or a mobile device.

The above description is merely illustrative of the technical idea of this embodiment, and a person skilled in the art to which this embodiment belongs may make various modifications and variations without departing from the essential characteristics of the present embodiment. Accordingly, the present embodiments are intended to explain rather than limit the technical spirit of the present embodiment, and the scope of the technical spirit of the present embodiment is not limited by these embodiments. The protection scope of this embodiment should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be interpreted as being included in the scope of the present embodiment.

10: image restoration device 100: input frame DB
110: frame meta information DB 120: mask generator
130: frame selection unit 140: restoration unit
142: inter restoration unit 144: intra restoration unit
150: frame merging unit 160: output frame DB

Claims (11)

A method of restoring an area covered by an object included in an image, the method comprising:
selecting a reference frame including information on a target frame from which the object is to be removed and a region within the target frame covered by the object from among the frames of the image; and
Reconstructing the target frame by performing at least one of inter-inpainting and intra-inpainting according to the number of frames selected as the reference frame
Image restoration method comprising a. According to claim 1,
The selection process is
calculating a similarity between the target frame and a frame adjacent to the target frame by using pixels adjacent to an area in which the object is detected in the target frame; and
selecting a frame having the similarity higher than a preset threshold similarity as a reference frame, and generating information on a similarity region having the highest similarity within the reference frame;
The restoration process is
The image restoration method according to claim 1, wherein the target frame is restored by receiving information on the target frame, the reference frame, and the similar region. 3. The method of claim 2,
The creation process is
The image restoration method, characterized in that the similar region is divided into at least one region according to the position of the object in the reference frame, and a different weight is assigned to each divided region. According to claim 1,
The selection process is
a first selection process of checking frames of the image in chronological order, selecting a frame not including the object as a backward reference frame, and selecting a frame including the object as a target frame; and
A second selection process of selecting a frame not including the object as a forward reference frame by checking at least one frame located later in time from the target frame in chronological order
Image restoration method comprising a. According to claim 1,
The selection process is
An image restoration method, wherein the reference frame is selected from among the previously processed frames in which restoration is completed, and a previously processed frame having a restored region that does not overlap a region covered by the object in the target frame is selected as the reference frame. According to claim 1,
The selection process is
The image restoration method of claim 1, wherein the target frame and the reference frame are selected from among frames in the same scene based on the scene change information extracted from the image. The method of claim 1,
The restoration process is
When the sum of the number of target frames and the number of reference frames is greater than a preset frame processing unit, at least one of the inter restoration and the intra restoration is performed. According to claim 1,
The restoration process is
The image restoration method according to claim 1, wherein the intra restoration is performed when there is no frame selected as the reference frame, and the inter restoration is performed when there is a frame selected as the reference frame. According to claim 1,
The restoration process is
generating a restored frame by performing the intra restoration on some of the target frames when there is no frame selected as the reference frame; and
A process of performing the inter restoration on the remaining frames among the target frames by using the restored frame as the reference frame
Image restoration method comprising a. In the device for restoring the area covered by the object included in the image,
a frame selection unit for selecting a reference frame including information on a target frame from which the object is to be removed and a region within the target frame obscured by the object; and
an inpainting unit generating a restored frame by performing at least one of inter inpainting and intra inpainting according to the number of reference frames;
Image restoration apparatus comprising a. A computer program stored in a computer-readable recording medium to execute each process included in the image restoration method according to any one of claims 1 to 9.

Images