KR20220129852A - Image processing apparatus and method for generating information beyond image area - Google Patents

Abstract

The present invention relates to an image processing technique for generating information outside an image area. In accordance with the present invention, an image processing device receives a video having a first screen standard; sets an image area corresponding to a difference from a second screen standard defined differently from the first screen standard as an unsecured area based on the first screen standard; detects each similar frame by frame included in the inputted video of the first screen standard to set the frames to a similar frame group; generates an image with respect to the unsecured area by referring to the similar frame group with respect to each frame included in the inputted video of the first screen standard; and outputs a video having the second screen standard from an original image included in the video of the first screen standard and an image created for the unsecured area. Therefore, the present invention is capable of improving quality satisfaction which a user can experience as a result.

Description

Image processing apparatus and method for generating information beyond image area

The present invention relates to image processing technology, and more particularly, to an image processing apparatus and method for referencing various frames in a moving picture or generating information outside an image area based on artificial intelligence, and a recording medium recording the method.

With the development of communication technology, image processing technology, and hardware, it has become possible to process higher quality images in real time. As a result, many changes were made to the standard of the image. For example, in the past, an aspect ratio of a moving picture was generally 4:3, but nowadays, a wide aspect ratio such as 16:9 is preferred. Accordingly, when a video produced in the past is played back on a newer device having a new aspect ratio, the left and right areas of the video reproducing device face the reality that the corresponding area must be empty without information to be displayed. In order to solve this problem, unless a stretch that changes the ratio of an image or a crop that cuts out a part of an image is used, the image reproducing apparatus typically fills in the region without image information and displays it in black. . This lack of information has been supplemented by letterbox or pillarbox methods that fill in the blank spaces with black.

In addition, images produced in the past did not have good aspect ratio as well as the quality of the image itself. For example, the latest display device designed based on high resolution pixels due to low resolution or bit rate. When the video is played back in , the coarseness of the image quality becomes conspicuous.

In other words, it is difficult to meet consumers' heightened expectations and expectations when trying to use images produced in the past in the latest devices. Accordingly, various ideas have been proposed to utilize the low-quality images of the past in the latest image reproducing apparatuses supporting high-quality images. In the prior art literature presented below, a conversion technique for improving the quality of such an image is introduced.

"Low-resolution image loss information estimation method and high-resolution image conversion method", Korean Patent Laid-Open No. 2011-0026942

The technical problem to be solved by the present invention is that when a video according to the past screen standard is reproduced by an image reproducing device according to the latest screen standard, the conventional technology simply adjusts the size of the video based on one side of the past video However, since there is no information to display in the area outside the other side, the problem of wasted resources of the latest video playback device is solved by using the letterbox or pillarbox method that displays black and to overcome the weakness that the quality satisfaction experienced by users is rather lowered.

In order to solve the above technical problem, an image processing method according to an embodiment of the present invention includes the steps of: (a) receiving, by an image processing apparatus, a video having a first screen standard; (b) setting, by the image processing apparatus, an image area corresponding to a difference between the first screen standard and a second screen standard defined differently from the first screen standard as an unsecured area; (c) detecting, by the image processing apparatus, each similar frame for each frame included in the inputted moving picture of the first screen standard, and setting a similar frame group; (d) generating, by the image processing apparatus, an image of the unsecured area by referring to the similar frame group for each frame included in the inputted moving picture of the first screen standard; and (e) outputting, by the image processing apparatus, a moving picture having the second screen standard from the original image included in the moving picture of the first screen standard and the image generated for the unsecured area.

In the image processing method according to an embodiment, in the step (b) of setting the area as an unsecured area, the second screen standard is set as a target and is selected from among aspect ratio, resolution, and angle of view. An image area that is not possessed by the first screen standard due to a difference in screen standards including at least one may be set as an unsecured area.

In the image processing method according to an embodiment, the step (c) of setting a similar frame group is 2 using at least one of feature matching, template matching, and histogram comparison. A similarity value may be calculated by comparing adjacent frames.

In the image processing method according to an embodiment, the step (c) of setting a similar frame group includes: (c1) temporally preceding or temporally preceding the reference frame with respect to one reference frame included in the moving picture of the first screen standard; calculating similarities with subsequent adjacent frames, respectively; (c2) when the similarity calculated from the neighboring frames is equal to or greater than the threshold, setting the neighboring frame equal to or greater than the threshold as a new reference frame, calculating the similarity with the neighboring frames again, and checking whether the newly calculated similarity is equal to or greater than the threshold setting only frames having a similarity greater than or equal to a threshold as a group of similar frames to the first reference frame by repeating the process in series; and (c3) deriving a similar frame group for each frame by performing steps (c1) and (c2) for all frames included in the moving picture of the first screen standard.

Also, in the step (c) of setting the similar frame group, the similarity value between two adjacent frames is matched with an identifier pair of the corresponding frame and stored in a look-up table, but two new When calculating the similarity between adjacent frames, the lookup table is first searched for, and if there is a previously stored similarity value, the stored similarity value is read and used, and the similarity is calculated only when there is no previously stored similarity value. can be stored in a table.

In the image processing method according to an embodiment, in the step (c) of setting the group of similar frames, (c4) sequentially calculating the similarity between adjacent frames for all frames included in the moving picture of the first screen standard. to do; and (c5) setting a similar frame group including only frames having a similarity greater than or equal to the threshold for each frame section in which the calculated similarity value is equal to or greater than the threshold value for successive time series.

In the image processing method according to an embodiment, the step (d) of generating the image for the unsecured area includes (d1) included in the similar frame group with respect to the current frame included in the input video of the first screen standard. extending the current frame using a geometric relationship between the similar frame and the current frame; and (d2) generating an image for the unsecured area by performing the frame expansion of step (d1) on all frames included in the inputted moving picture of the first screen standard.

In addition, the step (d1) of extending the current frame may extend the current frame by extracting a keypoint from each of the similar frame and the current frame and matching both images based on the extracted keypoint. .

Furthermore, the step (d) of generating the image for the non-secured area includes: (d3) when the image for the non-secured area is not generated through the frame expansion of the (d1) step, a Generative Adversarial Network (GAN) or auto-encoder The method may further include a step of replenishing the image for the unsecured area using (Autoencoder).

In the image processing method according to an embodiment, the step (e) of outputting a video having a second screen standard includes: (e1) the original video and the unsecured video for all frames included in the video having the first screen standard, respectively. and converting the image generated for the region into an image having the second screen standard by combining the image.

In addition, the step (e) of outputting a moving picture having a second screen standard includes: (e2) when the quality difference between the image generated for the unsecured area and the original image is equal to or greater than the acceptable limit, the resolution of the inferior image or The method may further include supplementing a bit rate from the similar frame.

Furthermore, the step (e) of outputting the moving picture having the second screen standard includes: (e3) smoothing the boundary between the image generated for the unsecured area and the original image included in the moving picture of the first screen standard. It may further include;

On the other hand, below, a computer-readable recording medium in which a program for executing the above-described image processing methods is recorded on a computer is provided.

In order to solve the above technical problem, an image processing apparatus according to an embodiment of the present invention includes an input unit for receiving a video having a first screen standard; a memory for storing a program for converting the input video having the first screen standard into a video having a second screen standard defined differently from the first screen standard; and a processor executing the program stored in the memory, wherein the program stored in the memory is an image corresponding to a difference from a second screen standard defined differently from the first screen standard based on the first screen standard The area is set as an unsecured area, and similar frames are detected for each frame included in the input video of the first screen standard, respectively, and set as a group of similar frames, and the inputted video included in the video of the first screen standard For each frame, an image of the unsecured area is generated by referring to the similar frame group, and the second screen standard is obtained from the original image included in the moving picture of the first screen standard and the image generated for the unsecured area. Contains a command to output a video with

In the image processing apparatus according to an embodiment, the program stored in the memory includes at least one of an aspect ratio, a resolution, and an angle of view by targeting the second screen standard. An image area that is not possessed by the first screen standard due to a difference in screen standards may be set as an unsecured area.

In the image processing apparatus according to an embodiment, the program stored in the memory determines a similarity with adjacent frames temporally preceding or following the reference frame with respect to one reference frame included in the moving picture of the first screen standard. If the similarity calculated from the adjacent frames is equal to or greater than the threshold, the adjacent frame greater than or equal to the threshold is set as a new reference frame, the similarity with the adjacent frames is calculated again, and whether the newly calculated similarity is greater than or equal to the threshold is checked. a process of sequentially repeating the process of setting only frames having a similarity greater than or equal to a threshold as a group of similar frames to the first reference frame, and calculating the similarity for all frames included in the moving picture of the first screen standard; A similar frame group may be derived for each frame by performing a process of setting the similar frame group.

In the image processing apparatus according to an embodiment, the program stored in the memory sequentially calculates similarities between adjacent frames for all frames included in the moving picture of the first screen standard, and the calculated similarity is equal to or greater than a threshold value. For this time-series continuous frame section, a similar frame group including only frames having a similarity degree equal to or greater than a threshold value for each section can be set.

In the image processing apparatus according to an embodiment, the program stored in the memory uses a geometric relationship between the similar frame included in the similar frame group and the current frame with respect to the current frame included in the input video of the first screen standard. to extract a keypoint from each of the similar frame and the current frame, extend the current frame by matching both images based on the extracted keypoint, and all frames included in the input video of the first screen standard By performing a process of extending the current frame with respect to , it is possible to generate an image for the unsecured area.

In addition, the program stored in the memory uses a generative adversarial network (GAN) or an autoencoder when the image for the unsecured region is not generated through the process of extending the current frame. can be supplemented.

In the image processing apparatus according to an embodiment, the program stored in the memory combines the original image and the image generated for the unsecured area with respect to all frames included in the moving picture of the first screen standard, respectively, to obtain the second screen It can be converted to an image having a standard.

Embodiments of the present invention can actively and effectively respond to changes in screen standards by expanding an image based on information in an adjacent frame in the video, but by supplementing the image for an area that has not been expanded through artificial intelligence. It is possible to prevent letterboxes or pillarboxes that are displayed in black because there is no information to display in some areas in the video according to the change in video quality. can do it

1 is an exemplary diagram for explaining the basic idea of the present invention for generating an image with a changed screen standard by acquiring information outside the image area.
2 is a flowchart illustrating an image processing method for generating information outside an image area according to an embodiment of the present invention.
3 is a diagram for explaining a process of detecting a similar frame in an image processing method according to an embodiment of the present invention.
4 is a diagram for explaining a process of setting a similar frame group in an image processing method according to an embodiment of the present invention.
5 and 6 are diagrams illustrating a process of generating an image for an unsecured area in an image processing method according to an embodiment of the present invention and a program code therefor.
7 and 8 are exemplary diagrams and block diagrams for explaining a process of replenishing a blank area in an image by using a deep neural network, respectively.
9 is a block diagram illustrating an image processing apparatus for generating information outside an image area according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, detailed descriptions of well-known functions or configurations that may obscure the gist of the present invention in the following description and accompanying drawings will be omitted. In addition, throughout the specification, 'including' a certain component does not exclude other components unless otherwise stated, but means that other components may be further included.

Also, terms such as first, second, etc. may be used to describe various elements, but the elements should not be limited by the terms. The above terms may be used for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

The terms used in the present invention are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as "comprises" or "comprises" are intended to designate that the specified feature, number, step, operation, component, part, or combination thereof exists, but is one or more other features or It should be understood that the existence or addition of numbers, steps, operations, components, parts, or combinations thereof does not preclude the possibility of addition.

Unless specifically defined otherwise, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having meanings consistent with the context of the related art, and unless explicitly defined in the present application, they are not to be interpreted in an ideal or excessively formal meaning. .

1 is an exemplary diagram for explaining the basic idea of the present invention for generating an image with a changed screen standard by acquiring information outside the image area.

FIG. 1A exemplifies some frames of the Barcelona Olympic marathon final video, and it is intended to generate a video having a wide aspect ratio by changing the current aspect ratio. To this end, it was noted that information outside the field of view (angle of view) of the image being reproduced at a given reproduction time point is contained in the image before/after the reproduction time point in many cases. That is, the field of view (angle of view) of the image being reproduced at the specific playback time may be expanded by using information included in frames before and after the specific playback time.

Referring to FIG. 1B, it shows that a larger image is generated by combining the information obtained from the frames before and after the specific time point illustrated in FIG. 1A and the frame of the original specific time point. In this way, information outside the image area at a specific time can be obtained, and when the search for adjacent front/rear frames is repeatedly extended, a larger image can also be expected.

Furthermore, it is possible to acquire information on an area that cannot be obtained from adjacent front/back frames by using artificial intelligence. Referring to FIG. 1C , as exemplified by FIG. 1B , a result of generating a larger image than a large image generated from information obtained from frames before and after a given frame is exemplified. The image of FIG. 1C could be obtained by estimating the area outside the angle of view using artificial intelligence and creating a new image.

In other words, it is possible to actively and effectively respond to changes in screen standards by utilizing information in adjacent frames within a video and supplementation of video through artificial intelligence. In the embodiments of the present invention presented below, specific technical means for implementing this idea will be described with reference to the drawings.

2 is a flowchart illustrating an image processing method for generating information outside an image area according to an embodiment of the present invention.

In step S210, the image processing apparatus receives a video having a first screen standard. Here, the moving picture may be understood as set data formed by continuously connecting a plurality of still image frames. Accordingly, each frame has a time-series relationship, and for convenience of explanation, a frame that is temporally ahead of one frame selected at a specific point in time is named a preceding frame, and a frame that is temporally behind is named a succeeding frame.

In step S220, the image processing apparatus sets an image area corresponding to a difference from a second screen standard defined differently from the first screen standard as an unsecured area based on the first screen standard. In this process, targeting the second screen standard, the first screen standard is retained due to a difference in screen standards including at least one of aspect ratio, resolution, and angle of view. An image area that is not available can be set as an unsecured area. Through this process, it is specified which area the information does not have in the original image corresponds to.

In step S230, the image processing apparatus detects a similar frame for each frame included in the input video of the first screen standard, and sets the similar frame group. That is, it is checked whether frames similar to the reference frame exist by searching for a preceding frame or a succeeding frame adjacent to one reference frame. tied and managed. By performing this process on all frames in the moving picture, a similar frame group with a high degree of similarity to the frame at a specific time is derived. In this process, a similarity value may be calculated by comparing two adjacent frames using at least one of feature matching, template matching, and histogram comparison.

Referring to FIG. 3 for explaining a process of detecting a similar frame in an image processing method according to an embodiment of the present invention, five frames are exemplified along the time axis. For example, based on the frame f _t corresponding to time t, frames f _t-1 and f _t+1 are selected at times t-1 and t+1 adjacent thereto, respectively, and the similarity is compared with the reference frame f _t . Calculate. If the calculated similarity is equal to or greater than a preset threshold, similarity evaluation is performed again on frames f _t-2 and f _t+2 of time t-2 or t+2 adjacent to the adjacent frame. In this way, adjacent frames similar to the reference frame f _t are searched and only temporally continuous similar frames are included in the similar frame group. Here, the similar frame group refers to a set of candidates that can be used to obtain information on an area outside the screen standard in the subsequent reference frame f _t .

On the other hand, in order to detect a similar frame for each frame included in the moving picture of the first screen standard and set it as a similar frame group, various embodiments may be utilized in terms of implementation.

As a first embodiment, first, with respect to one reference frame included in a moving picture of the first screen standard, similarities with adjacent frames temporally preceding or following the reference frame are calculated, respectively. Then, when the degree of similarity calculated from the adjacent frames is equal to or greater than the threshold, the adjacent frame greater than the threshold is set as a new reference frame, the degree of similarity with the adjacent frames is calculated again, and the newly calculated similarity is greater than or equal to the threshold. By repeating the process sequentially, only frames having a similarity greater than or equal to a threshold are set as a group of similar frames to the first reference frame. Now, a similar frame group can be derived for each frame by performing a process of calculating the degree of similarity with the adjacent frames for all frames included in the moving picture of the first screen standard and setting the similar frame group. .

In order to reduce the repetition of the operation and increase data reuse efficiency, the similarity value between two adjacent frames is matched with the identifier pair of the corresponding frame and stored in a look-up table. In the case of calculating the similarity between the two groups, the lookup table is first searched, and if there is a previously stored similarity value, the stored similarity value is read and used. It is preferable to do

4 is a view for explaining a process of setting a similar frame group in an image processing method according to an embodiment of the present invention, and illustrates a lookup table. Referring to FIG. 4 , two pairs of consecutive frames are formed, an identifier pair of the frame is set as an index, and a similarity value corresponding thereto is stored. As the similarity calculation process is repeated, a similarity value is recorded and a check for forming a similarity frame group is performed. For example, assuming that the threshold for determining the similarity is set to '0.7', successive sections of (f ₁ ,f ₂ ), (f ₂ ,f ₃ ), (f ₃ ,f ₄ ) are 'similar frames' It can be confirmed that group #1' was formed. However, the similarity corresponding to (f ₄ ,f ₅ ) did not satisfy the threshold criterion, and again, the continuous sections of (f ₅ ,f ₆ ) and (f ₆ ,f ₇ ) were 'similar frame group #2' was formed. Therefore, by referring to the lookup table generated in this way, it is possible to quickly identify which adjacent frames are similar to the frame at a specific time, and when the screen standard of the image is changed later, it can be used as candidate data for obtaining information outside the region in the original image. .

As a second embodiment for detecting similar frames for each frame included in the moving picture of the first screen standard and setting them as a group of similar frames, a method of calculating the degree of similarity between frames in advance according to time sequence can be utilized. First, for all frames included in the moving picture of the first screen standard, similarities between adjacent frames are sequentially calculated. Then, with respect to frame sections in which the calculated similarity value equal to or greater than the threshold value is continuous in time series, a similar frame group including only frames having a similarity level equal to or greater than the threshold value for each section may be set. That is, while the first embodiment described above searches for adjacent previous/next preceding/following frames from one reference frame, the second embodiment operates in a way that calculates the similarity between adjacent frames in one direction. Of course, even in this second embodiment, the lookup table introduced above can be used to manage the calculated similarity.

Returning to FIG. 2 , in step S240 , the image processing apparatus generates an image of the unsecured area with reference to the similar frame group for each frame included in the inputted moving picture of the first screen standard. In this process, with respect to the current frame included in the input video of the first screen standard, the current frame is expanded by using a geometric relationship between the similar frame included in the similar frame group and the current frame. Then, the above-described current frame expansion process is performed on all frames included in the inputted moving picture of the first screen standard to generate an image for the unsecured area.

It is assumed that a similar frame group including a plurality of similar frames having a high similarity to the current frame is set when the current frame (a reference basic frame) is to be extended. In a situation in which the current frame is extended by selecting one frame from the similar frame group, even if the contents of the current frame and the similar frame are the same (for example, a frame including a human running scene in the image), the frame is shot The frame characteristics may be different from each other depending on the angle, perspective, illuminance, etc. that the camera faces at the moment the camera is directed. Therefore, when extending the current frame, it is necessary to match the characteristics of the frame. For example, when a cube exists, the shape of the cube may appear as a square, a trapezoid, or an arbitrary rectangle depending on the angle the camera faces.

In order to merge the current frame and the similar frame into one frame, various corrections regarding characteristic factors including camera angle and perspective between the two frames are required. From an implementation point of view, a mathematical transformation such as stretching or reducing an image is required for correction, and correction can be effectively performed using a series of processing steps exemplified below.

First, an artificial neural network is constructed. Then, the basic three-dimensional model (cube, cube, cylinder, sphere, etc.) and its combined configuration model are artificially generated using a computer using images viewed from various angles and perspectives, and then taught through an artificial neural network. Then, the artificial neural network learns the concept of angle and perspective, and when it receives an image of an arbitrary angle and perspective as input, it is in a state in which it can output an image of a specific angle and perspective. Now, by taking two images as input and learning an artificial neural network that outputs the angle and perspective difference between the images, it is used to transform similar frames and merge the two frames into one by matching the angle and perspective features with the basic frame. can

5 and 6 are diagrams illustrating a process of generating an image for an unsecured area in an image processing method according to an embodiment of the present invention and a program code therefor.

Previously, it has been described that in step S240 of FIG. 2 , an image for an unsecured area can be generated with reference to a similar frame group with respect to each frame included in the inputted moving picture of the first screen standard. To this end, the current frame may be extended by extracting a keypoint from each of the similar frame and the current frame and matching both images based on the extracted keypoint. Referring to FIG. 5, the process of generating an image larger than the original image from information obtained from preceding and following frames adjacent to a given reference frame is exemplified. It shows that a larger image was created by using the information obtained from them.

Meanwhile, in FIG. 6 , it can be seen that blank areas in which information in the image is not secured are shown in the lower left and upper right corners. Hereinafter, technical means for supplementing this will be introduced.

7 and 8 are exemplary diagrams and block diagrams for explaining a process of replenishing a blank area in an image by using a deep neural network, respectively.

In the embodiments of the present invention, an image of an unsecured area is generated with reference to a similar frame group. Nevertheless, it may not be possible to obtain information on an unreserved area from a similar frame group for a required screen standard. That is, with respect to the current frame included in the input video of the first screen standard, the current frame is extended using the geometric relationship between the similar frame included in the similar frame group and the current frame, but the image for the unsecured area is not generated. In this case, an image for the unsecured area may be supplemented using a generative adversarial network (GAN) or an autoencoder.

Referring to FIG. 7 , compared with FIG. 6 , it can be confirmed that the upper left area 710 and the lower right area 720 are supplemented. Referring to FIG. 8 , first, a plurality of adjacent images including an image with empty information are input as an input image 810 . Then, by receiving the clue information 830 for the vacant image in which the information is empty through the deep neural network 820, an image that can supplement the empty information is newly generated, and then as the output image 840. print out In this case, the deep neural network 820 may be implemented using a GAN or an autoencoder.

Now, finally, in step S250 of FIG. 2 , the image processing apparatus outputs the moving picture having the second screen standard from the original image included in the moving picture of the first screen standard and the image generated for the unsecured area. Here, for all frames included in the moving picture of the first screen standard, the original image and the image generated for the unsecured area may be combined, respectively, to be converted into an image having the second screen standard.

In the case where it is determined as a frame with high similarity, three types can be considered. The first type is a case in which the image is taken while the shooting means (camera) is fixed, the second type is when the shooting means is moved up/down/left/right, and the third type is when the shooting device is zoomed in. (zoom-in) / zoom-out (zoom-out) is the case of shooting.

In the first case, for example, a scene in which the presenter speaks in a news broadcast in the studio. When comparing similarity, pixels with a large difference between frames are discarded and only pixels with little difference are used to improve the resolution. For example, if it is determined that four similar frames exist in a temporally continuous section of the reference frame, 1 by 1 image information can be improved to 4 by 4 image information by using a similar frame group composed of these similar frames. .

In the second or third case, the reference frame may be extended based on preceding and following frames adjacent to the reference frame. For example, an image having an aspect ratio of 16:9 may be generated by connecting images having an aspect ratio of 4:3.

If the quality of the video is improved using the image creation method suggested above, the resolution may be inconsistent or a problem may occur in which a heterogeneous color (e.g., black) appears in some areas of the supplemented image unintentionally. have. Therefore, image post-processing or filtering is required to resolve such heterogeneity. From an implementation point of view, when the quality difference between the image generated for the unsecured region and the original image is equal to or greater than an acceptable threshold, the inferior image resolution or bit rate may be supplemented from the similar frame. Also, a boundary between the image generated for the unsecured area and the original image included in the moving picture of the first screen standard may be smoothed.

9 is a block diagram illustrating an image processing apparatus 920 for generating information outside an image region according to an embodiment of the present invention. function was shown. Therefore, in order to avoid duplication of description, only the general structure thereof is outlined herein.

The input unit 921 is a means for receiving a moving picture 910 having the first screen standard. The memory 923 stores a program for converting the moving picture 910 having the first screen standard input through the input unit 921 into the moving picture 930 having a second screen standard defined differently from the first screen standard. . The processor 925 is a subject that executes the program stored in the memory 923 . Here, the program stored in the memory 923 sets an image area corresponding to a difference between the first screen standard and a second screen standard defined differently based on the first screen standard as an unsecured area, and A similar frame is detected for each frame included in the moving picture of the first screen standard, and is set as a similar frame group, and the similar frame group is generated for each frame included in the inputted moving picture of the first screen standard. generating an image of the non-secured area with reference to, and outputting a moving picture having the second screen standard from the original image included in the moving picture of the first screen standard and the image generated for the non-secured area.

The program stored in the memory 923 targets the second screen standard, and due to a difference in screen standards including at least one of an aspect ratio, a resolution, and an angle of view, 1 The image area that the screen standard does not have can be set as an unsecured area.

The program stored in the memory 923 calculates a similarity with adjacent frames temporally preceding or following the reference frame with respect to one reference frame included in the moving picture of the first screen standard, and the adjacent frames When the degree of similarity calculated from is equal to or greater than the threshold, by setting the adjacent frame greater than or equal to the threshold as a new reference frame, calculating the degree of similarity with the adjacent frames again, and repeatedly repeating the process of checking whether the newly calculated degree of similarity is greater than or equal to the threshold. a process of setting only frames having a similarity equal to or greater than a threshold value as a similar frame group to the first reference frame, calculating the similarity for all frames included in the moving picture of the first screen standard, and setting the similar frame group as the similar frame group; It is possible to derive a similar frame group for each frame by performing it.

The program stored in the memory 923 sequentially calculates a degree of similarity between adjacent frames for all frames included in the moving picture of the first screen standard, and a frame section in which the calculated similarity is greater than or equal to a threshold value is continuous in time series can be set as a similar frame group including only frames having a similarity level equal to or greater than a threshold value for each section.

The program stored in the memory 923 uses a geometrical relationship between the similar frame and the current frame included in the similar frame group with respect to the current frame included in the inputted moving picture of the first screen standard, the similar frame and the current frame. Extracting a keypoint from each, extending the current frame by matching both images based on the extracted keypoint, and extending the current frame with respect to all frames included in the input video of the first screen standard By performing the process, an image of the unsecured area may be generated. In addition, the program stored in the memory 923 is stored in the non-secured region using a Generative Adversarial Network (GAN) or an autoencoder when the image for the non-secured region is not generated through the process of extending the current frame. You can supplement the video.

The program stored in the memory 923 may convert an image having the second screen standard by combining the original image and the image generated for the unsecured area, respectively, for all frames included in the moving picture of the first screen standard. have.

According to embodiments of the present invention, it is possible to actively and effectively respond to changes in the screen standard by expanding an image based on information in an adjacent frame in the moving picture, but by supplementing the image for the area that has not been expanded through artificial intelligence. It is possible to prevent letterbox or pillarbox that is displayed in black because there is no information to display in some areas of the image according to the change of the screen standard. can improve

Meanwhile, the embodiments of the present invention can be implemented as computer-readable codes on a computer-readable recording medium. The computer-readable recording medium includes all types of recording devices in which data readable by a computer system is stored.

Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. In addition, the computer-readable recording medium may be distributed in a network-connected computer system, and the computer-readable code may be stored and executed in a distributed manner. And functional programs, codes, and code segments for implementing the present invention can be easily inferred by programmers in the technical field to which the present invention pertains.

In the above, various embodiments of the present invention have been mainly examined. Those of ordinary skill in the art to which the present invention pertains will understand that the present invention can be implemented in modified forms without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments are to be considered in an illustrative rather than a restrictive sense. The scope of the present invention is indicated in the claims rather than the foregoing description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

810: input image
820: deep neural network 830: vacant image clue information
840: output image
910: first standard video
920: image processing device 921: input unit
923: memory 925: processor
930: 2nd standard video

Claims (20)

(a) receiving, by the image processing apparatus, a video having a first screen standard;
(b) setting, by the image processing apparatus, an image area corresponding to a difference between the first screen standard and a second screen standard defined differently from the first screen standard as an unsecured area;
(c) detecting, by the image processing apparatus, each similar frame for each frame included in the inputted moving picture of the first screen standard, and setting a similar frame group;
(d) generating, by the image processing apparatus, an image of the unsecured area by referring to the similar frame group for each frame included in the inputted moving picture of the first screen standard; and
(e) outputting, by the image processing apparatus, a moving picture having the second screen standard from the original image included in the moving picture of the first screen standard and the image generated for the unsecured area; . The method of claim 1,
Step (b) is,
An image that is not possessed by the first screen standard due to a difference in screen standards including at least one of aspect ratio, resolution, and angle of view, targeting the second screen standard An image processing method for setting an area as an unsecured area. The method of claim 1,
Step (c) is,
An image processing method of calculating a similarity value by comparing two adjacent frames using at least one of feature matching, template matching, and histogram comparison. The method of claim 1,
Step (c) is,
(c1) calculating similarities with adjacent frames temporally preceding or following the reference frame with respect to one reference frame included in the moving picture of the first screen standard;
(c2) when the similarity calculated from the neighboring frames is equal to or greater than the threshold, setting the neighboring frame equal to or greater than the threshold as a new reference frame, calculating the similarity with the neighboring frames again, and checking whether the newly calculated similarity is equal to or greater than the threshold setting only frames having a similarity greater than or equal to a threshold as a group of similar frames to the first reference frame by repeating the process in series; and
(c3) performing steps (c1) and (c2) for all frames included in the moving picture of the first screen standard to derive a similar frame group for each frame; 5. The method of claim 4,
Step (c) is,
A similarity value between two adjacent frames is matched with an identifier pair of the corresponding frame and stored in a look-up table,
When the similarity between two adjacent frames is newly calculated, the lookup table is first searched, and if there is a previously stored similarity value, the stored similarity value is read and used, and the similarity is calculated only when the previously stored similarity value does not exist. and storing it in the lookup table. The method of claim 1,
Step (c) is,
(c4) sequentially calculating similarities between adjacent frames for all frames included in the moving picture of the first screen standard; and
(c5) setting a similar frame group including only frames having a similarity greater than or equal to the threshold for each frame section in which the calculated similarity value is greater than or equal to the threshold value for successive time series; The method of claim 1,
Step (d) is,
(d1) expanding the current frame by using a geometric relationship between the current frame and the similar frame included in the similar frame group with respect to the current frame included in the input video of the first screen standard; and
(d2) performing the frame expansion of step (d1) on all frames included in the inputted moving picture of the first screen standard to generate an image for the unsecured area; 8. The method of claim 7,
The step (d1) is,
An image processing method for extending the current frame by extracting a keypoint from each of the similar frame and the current frame and matching both images based on the extracted keypoint. 8. The method of claim 7,
(d3) if the image for the non-secured region is not generated through the frame extension of step (d1), supplementing the image for the non-secured region using a Generative Adversarial Network (GAN) or an autoencoder; Further comprising, an image processing method. The method of claim 1,
Step (e) is,
(e1) converting an image having the second screen standard by combining the original image and the image generated for the unsecured region for all frames included in the moving picture of the first screen standard, respectively; processing method. 11. The method of claim 10,
(e2) supplementing the inferior image resolution or bit rate from the similar frame when the quality difference between the image generated for the unsecured area and the original image is equal to or greater than an acceptable threshold; , image processing method. 11. The method of claim 10,
(e3) smoothing the boundary between the image generated for the unsecured area and the original image included in the moving picture of the first screen standard; A computer-readable recording medium in which a program for executing the method of any one of claims 1 to 12 on a computer is recorded. an input unit for receiving a video having a first screen standard;
a memory for storing a program for converting the input video having the first screen standard into a video having a second screen standard defined differently from the first screen standard; and
A processor that executes the program stored in the memory;
The program stored in the memory is
An image area corresponding to a difference between the first screen standard and a second screen standard defined differently based on the first screen standard is set as an unsecured area, and a frame ( frame) is detected for each similar frame, set as a similar frame group, and for each frame included in the input video of the first screen standard, an image for the unsecured area is generated with reference to the similar frame group, and a command for outputting a moving picture having the second screen standard from the original image included in the moving picture of the first screen standard and the image generated for the unsecured area. 15. The method of claim 14,
The program stored in the memory is
An image that is not possessed by the first screen standard due to a difference in screen standards including at least one of aspect ratio, resolution, and angle of view, targeting the second screen standard An image processing apparatus for setting an area as an unsecured area. 15. The method of claim 14,
The program stored in the memory is
For one reference frame included in the moving picture of the first screen standard, a degree of similarity with adjacent frames temporally preceding or following the reference frame is calculated, respectively;
If the degree of similarity calculated from the adjacent frames is equal to or greater than the threshold, the process of setting the adjacent frame equal to or greater than the threshold as a new reference frame, calculating the degree of similarity with the adjacent frames again, and checking whether the newly calculated similarity is equal to or greater than the threshold is chained By repeatedly repeating, only frames with similarity greater than or equal to the threshold are set as similar frame groups to the first reference frame,
An image processing apparatus for deriving a similar frame group for each frame by performing a process of calculating the degree of similarity and setting the similarity frame group for all frames included in the moving picture of the first screen standard. 15. The method of claim 14,
The program stored in the memory is
For all frames included in the moving picture of the first screen standard, the similarity between adjacent frames is sequentially calculated,
An image processing apparatus for setting a similar frame group including only frames having a similarity equal to or greater than the threshold value for each frame section in which the calculated similarity value is equal to or greater than the threshold value is continuous in time series. 15. The method of claim 14,
The program stored in the memory is
Extracting a keypoint from each of the similar frame and the current frame by using a geometric relationship between the similar frame included in the similar frame group and the current frame with respect to the current frame included in the input video of the first screen standard, Expand the current frame by matching both images based on the extracted feature points,
An image processing apparatus for generating an image of the unsecured area by performing a process of extending the current frame with respect to all frames included in the inputted moving picture of the first screen standard. 19. The method of claim 18,
The program stored in the memory is
An image processing apparatus for replenishing an image for the unsecured region by using a generative adversarial network (GAN) or an autoencoder when the image for the unsecured region is not generated through the process of extending the current frame. 15. The method of claim 14,
The program stored in the memory is
An image processing apparatus for converting an image having the second screen standard by combining an original image and an image generated for the unsecured area for all frames included in the moving picture of the first screen standard, respectively.

Images