KR102492430B1 - Image processing apparatus and method for generating information beyond image area - Google Patents

Abstract

The present invention relates to an image processing technology for generating information outside an image area, wherein an image processing device receives a video having a first screen standard, and a second screen defined differently from the first screen standard based on the first screen standard The image area corresponding to the difference from the standard is set as an unsecured area, each similar frame is detected for each frame included in the video of the input first screen standard, and set as a similar frame group, and the input first screen For each frame included in the standard video, a similar frame group is referred to to create an image for the unsecured area, and a second screen standard is generated from the original video included in the video of the first screen standard and the image generated for the unsecured area. Outputs a video with

Description

Image processing apparatus and method for generating information beyond image area

The present invention relates to image processing technology, and more particularly, to an image processing device and method for generating information outside an image area based on artificial intelligence or referring to various frames in a video, and a recording medium recording the method.

With the development of communication technology, image processing technology and hardware, it has become possible to process higher quality images in real time. As a result, many changes have occurred in the standards of images. For example, in the past, it was common for a video to have an aspect ratio of 4:3, but nowadays a wide aspect ratio such as 16:9 is preferred. Therefore, when a video produced in the past is played on a latest device having a new aspect ratio, the reality is faced with the fact that the left and right areas of the video reproducing device must be left empty without information to be displayed. In order to solve this problem, unless a stretch that changes the ratio of an image or a crop that cuts out a part of an image is not used, an image reproducing device usually fills an area without image information with black and displays it. . Since there was no information like this, it was supplemented with a letterbox or pillarbox method that fills the empty space with black.

In addition, images produced in the past had poor quality as well as aspect ratio. When the video is played back in , the poor picture quality becomes noticeable.

In other words, when trying to use images produced in the past on the latest devices, it is difficult to meet the heightened expectations and expectations of consumers. Accordingly, various ideas have been proposed for utilizing old low-quality images in the latest image reproducing apparatus supporting high-quality images. In the prior art literature presented below, a conversion technique for improving the quality of such an image is introduced.

"Loss information estimation method of low resolution image and high resolution image conversion method", Korean Patent Publication No. 2011-0026942

The technical problem to be solved by the present invention is that the conventional technology simply adjusts the size of the video based on one side of the past video in playing a video according to the screen standard of the past in a video reproducing apparatus according to the latest screen standard. However, there is no information to be displayed in the area outside the other side, so it solves the problem of wasting resources of the latest video playback device by using a letterbox or pillarbox method that is displayed in black, even though the latest device is used. and overcome the weakness that the quality satisfaction experienced by users is rather degraded.

In order to solve the above technical problem, an image processing method according to an embodiment of the present invention includes: (a) receiving a video having a first screen standard by an image processing device; (b) setting, by the image processing device, an image area corresponding to a difference between the first screen standard and a second screen standard defined differently from the first screen standard as an unsecured area; (c) detecting similar frames for each frame included in the input moving picture of the first screen standard by the image processing device and setting them as a similar frame group; (d) generating, by the image processing device, an image for the unsecured area by referring to the similar frame group for each frame included in the input video of the first screen standard; and (e) outputting, by the image processing device, a video having the second screen standard from an original video included in the video of the first screen standard and an image generated for the unsecured area.

In the image processing method according to an embodiment, the step (b) of setting the unsecured area to the second screen standard among aspect ratio, resolution, and angle of view Due to a difference in screen standards including at least one, an image area not possessed by the first screen standard may be set as an unsecured area.

In the image processing method according to an embodiment, the step (c) of setting the group of similar frames to 2 frames using at least one of feature matching, template matching, and histogram comparison. A similarity value may be calculated by comparing two adjacent frames.

In the image processing method according to an embodiment, the step (c) of setting the group of similar frames may include (c1) temporally preceding one reference frame included in the moving picture of the first screen standard, or Calculating similarities with subsequent adjacent frames, respectively; (c2) If the similarity calculated from the adjacent frames is equal to or greater than the threshold, setting adjacent frames equal to or greater than the threshold as a new reference frame, calculating the similarity with each adjacent frame again, and checking whether the newly calculated similarity is equal to or greater than the threshold setting only frames whose degree of similarity is equal to or higher than a threshold value as a group of similar frames with respect to the first reference frame by repeating the process serially; and (c3) deriving a similar frame group for each frame by performing steps (c1) and (c2) for all frames included in the video of the first screen standard.

In addition, in the step (c) of setting a similar frame group, the similarity value between two adjacent frames is matched with an identifier pair of the corresponding frame and stored in a look-up table, and two newly When calculating the similarity between adjacent frames, the lookup table is first inquired, and if there is a pre-stored similarity value, the stored similarity value is read and used, and the similarity is calculated only when the pre-stored similarity value does not exist. can be stored in a table.

In the image processing method according to an embodiment, the step (c) of setting the group of similar frames includes: (c4) sequentially calculating the similarity between adjacent frames for all frames included in the video of the first screen standard; doing; and (c5) setting a similar frame group including only frames having a similarity equal to or greater than the threshold value for each section of frame sections in which the calculated similarity value is equal to or greater than the threshold value.

In the image processing method according to an exemplary embodiment, the step (d) of generating an image of an unsecured area includes a current frame included in the input video of the first screen standard of (d1) included in the similar frame group. extending the current frame by using a geometric relationship between the similar frame and the current frame; and (d2) generating an image for the unsecured area by performing the frame expansion of step (d1) on all frames included in the input video of the first screen standard.

In addition, in the step (d1) of extending the current frame, the current frame can be extended by extracting keypoints from each of the similar frame and the current frame and matching both images based on the extracted keypoints. .

Furthermore, the step (d) of generating an image of the unsecured region may include (d3) a Generative Adversarial Network (GAN) or an autoencoder when the image of the unsecured region is not generated through the frame expansion of the step (d1). The method may further include supplementing an image of the unsecured area using an autoencoder.

In the image processing method according to an embodiment, the step (e) of outputting a video having a second screen standard includes (e1) the original video and the unsecured video for all frames included in the video having the first screen standard. It may include combining the images generated for the region and converting them into images having the second screen standard.

In addition, in the (e) step of outputting a video having a second screen standard, (e2) when the quality difference between the video generated for the unsecured area and the original video is greater than the acceptance limit, the resolution of the inferior video or Compensating for a bit rate from the similar frame; may further include.

Furthermore, in the step (e) of outputting a video having the second screen standard, (e3) smoothing the boundary between the video generated for the unsecured area and the original video included in the video of the first screen standard. It may further include;

Meanwhile, hereinafter, a computer-readable recording medium in which a program for executing the above-described image processing methods in a computer is recorded is provided.

In order to solve the above technical problem, an image processing device according to an embodiment of the present invention includes an input unit for receiving a video having a first screen standard; a memory for storing a program for converting the input video having the first screen standard into a video having a second screen standard defined differently from the first screen standard; and a processor executing a program stored in the memory, wherein the program stored in the memory generates an image corresponding to a difference between the first screen standard and a second screen standard defined differently from the first screen standard. An area is set as an unsecured area, similar frames are detected for each frame included in the input video of the first screen standard, and set as a similar frame group, and included in the input video of the first screen standard For each frame, an image for the unsecured area is generated by referring to the similar frame group, and the second screen standard is obtained from the original image included in the video of the first screen standard and the image generated for the unsecured area. Includes a command to output a video with

In the image processing device according to an embodiment, the program stored in the memory includes at least one of an aspect ratio, a resolution, and an angle of view targeting the second screen standard. Due to the difference in screen standards, an image area not possessed by the first screen standard may be set as an unsecured area.

In the image processing device according to an embodiment, the program stored in the memory determines the similarity of one reference frame included in the moving picture of the first screen standard with neighboring frames temporally preceding or following the reference frame. If the similarity calculated from the adjacent frames is equal to or higher than the threshold value, the adjacent frame equal to or higher than the threshold value is set as a new reference frame, the similarity value with each adjacent frame is calculated again, and whether the newly calculated similarity value is equal to or higher than the threshold value is checked. and setting only frames whose similarity is equal to or higher than a threshold value as similar frame groups for the first reference frame by serially repeating the process, and calculating the similarity for all frames included in the video of the first screen standard. Similar frame groups may be derived for each frame by performing a process of setting similar frame groups.

In the image processing device according to an embodiment, the program stored in the memory sequentially calculates a similarity between adjacent frames for all frames included in the moving picture of the first screen standard, and the calculated similarity is a value equal to or greater than a threshold value. With respect to the time-sequentially continuous frame sections, a similar frame group including only frames having similarities greater than or equal to a threshold value for each section may be set.

In the image processing device according to an exemplary embodiment, the program stored in the memory uses a geometric relationship between the current frame and the similar frame included in the similar frame group with respect to the current frame included in the input moving picture of the first screen standard. Keypoints are extracted from each of the similar frames and the current frame, and the current frame is expanded by matching both images based on the extracted keypoints, and all frames included in the input video of the first screen standard An image of the unsecured area may be generated by performing a process of extending the current frame for .

In addition, the program stored in the memory, when the image of the unsecured area is not generated through the process of extending the current frame, uses a Generative Adversarial Network (GAN) or an autoencoder to generate an image of the unsecured area. can be supplemented.

In the image processing device according to an embodiment, the program stored in the memory combines an original image and an image generated for the unsecured area for all frames included in the video of the first screen standard, respectively, to obtain the second screen. It can be converted into an image having a standard.

Embodiments of the present invention can actively and effectively respond to changes in the screen standard by expanding an image based on information in an adjacent frame in a video, but supplementing the image for an unextended area through artificial intelligence. It is possible to prevent letterbox or pillarbox, which is displayed in black because there is no information to be displayed in some areas in the video according to the change of , and as a result, the quality satisfaction experienced by the user is improved by sufficiently utilizing the resources of the latest video playback device. can make it

1 is an exemplary diagram for explaining the basic idea of the present invention for generating an image with a changed screen standard by acquiring information outside an image area.
2 is a flowchart illustrating an image processing method for generating information outside an image area according to an embodiment of the present invention.
3 is a diagram for explaining a process of detecting similar frames in an image processing method according to an embodiment of the present invention.
4 is a diagram for explaining a process of setting similar frame groups in an image processing method according to an embodiment of the present invention.
5 and 6 are diagrams illustrating a process of generating an image of an unsecured area in an image processing method according to an embodiment of the present invention and program codes for the same.
7 and 8 are exemplary diagrams and block diagrams for explaining a process of supplementing blank areas in an image using a deep neural network, respectively.
9 is a block diagram illustrating an image processing device generating information outside an image area according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, detailed descriptions of well-known functions or configurations that may obscure the gist of the present invention will be omitted in the following description and accompanying drawings. In addition, throughout the specification, 'including' a certain component means that other components may be further included, not excluding other components unless otherwise stated.

Also, terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms may be used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention.

Terms used in the present invention are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "comprise" or "comprise" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, but that one or more other features or It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Unless specifically defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in this application, they are not interpreted in an ideal or excessively formal meaning. .

1 is an exemplary diagram for explaining the basic idea of the present invention for generating an image with a changed screen standard by acquiring information outside an image area.

FIG. 1(A) illustrates some frames of the Barcelona Olympics marathon final video, and a video having a wide aspect ratio is intended to be created by changing the current aspect ratio. To this end, it was noted that information outside the field of view (angle of view) of an image being reproduced at a given playback time point is contained in images before/after the playback time point in many cases. That is, the field of view (angle of view) of the image being reproduced at the specific reproduction time point can be expanded by utilizing information included in frames before/after the specific play time point.

Referring to (B) of FIG. 1 , it is shown that a larger image is generated by combining information obtained from frames before/after a specific viewpoint illustrated in (A) of FIG. 1 with original frames of a specific viewpoint. Through this method, information outside the video area at a specific point in time can be obtained, and when the search for adjacent previous/next frames is repeatedly expanded, a larger image can be created.

Furthermore, it is possible to obtain information of an area that cannot be obtained from adjacent previous/next frames by utilizing artificial intelligence. Referring to (C) of FIG. 1, as illustrated through (B) of FIG. 1 above, a result of generating a larger image than a large image generated from information obtained from frames before and after a given frame is illustrated. The image of FIG. 1 (C) was obtained by estimating an area outside the angle of view using artificial intelligence and newly generating it.

In other words, it is possible to actively and effectively respond to changes in screen specifications by utilizing the information in adjacent frames in the video and the supplement of the video through artificial intelligence. In the embodiments of the present invention presented below, specific technical means for implementing these ideas will be described with reference to the drawings.

2 is a flowchart illustrating an image processing method for generating information outside an image area according to an embodiment of the present invention.

In step S210, the image processing device receives a video having a first screen standard. Here, a video may be understood as aggregate data formed by continuously connecting a plurality of still image frames. Therefore, each frame has a time-sequential relationship, and for convenience of description, a frame that precedes a frame selected at a specific point in time is referred to as a preceding frame, and a frame that chronologically lags behind is referred to as a following frame.

In step S220, the image processing device sets an image area corresponding to a difference between the first screen standard and a second screen standard defined differently as an unsecured area based on the first screen standard. In this process, the first screen standard is held due to a difference in screen standards including at least one of aspect ratio, resolution, and angle of view, aiming at the second screen standard. An unsecured image area may be set as an unsecured area. Through this process, it is specified which region the information that the original image does not have corresponds to.

In step S230, the image processing device detects similar frames for each frame included in the input video of the first screen standard and sets them as similar frame groups. That is, a previous frame or a subsequent frame adjacent to one reference frame is searched to check whether frames similar to the reference frame exist, and as a result of the check, if the similarity value satisfies a preset criterion, the corresponding frame is regarded as a similar frame as one group. tied together and managed. By performing this process on all frames in the video, a similar frame group having a high similarity to a frame at a specific point in time is derived. In this process, a similarity value may be calculated by comparing two adjacent frames using at least one of feature matching, template matching, and histogram comparison.

Referring to FIG. 3 for describing a process of detecting similar frames in an image processing method according to an embodiment of the present invention, five frames are illustrated along the time axis. For example, after selecting frames f _t-1 and f _t+ 1 at times t-1 and t+1 adjacent to the frame f _t corresponding to time t, respectively, and comparing them with the reference frame f _t , the degree of similarity is determined. yield If the calculated similarity is equal to or greater than the preset threshold, similarity evaluation is performed again for frames f _t-2 and f _t+2 at time t-2 or t+2 adjacent to the adjacent frame. In this way, adjacent frames similar to the reference frame f _t are searched for, and only temporally continuous similar frames are included in the similar frame group. Here, the similar frame group means a set of candidates that can be used to obtain information about an area outside the screen standard in the reference frame f _t .

Meanwhile, in order to detect similar frames for each frame included in the video of the first screen standard and set them as similar frame groups, various embodiments may be utilized in terms of implementation.

As a first embodiment, first, for one reference frame included in a video of the first screen standard, similarities with adjacent frames temporally preceding or following the reference frame are calculated. Then, if the similarity calculated from the adjacent frames is greater than or equal to the threshold, setting the adjacent frames that are greater than or equal to the threshold as new reference frames, calculating similarities with the adjacent frames again, and checking whether the newly calculated similarity is greater than or equal to the threshold. By repeating the process in chain, only frames whose similarity is equal to or greater than the threshold are set as similar frame groups for the first reference frame. Now, for all frames included in the video of the first screen standard, similar frame groups can be derived for each frame by performing the process of calculating the degree of similarity with the adjacent frames and the process of setting them as similar frame groups. .

In order to reduce the repetition of operations and increase the efficiency of data reuse, the similarity value between two adjacent frames is matched with the identifier pair of the corresponding frame and stored in a look-up table. When calculating the similarity between the two, the lookup table is first inquired, and if there is a pre-stored similarity value, the stored similarity value is read and used, and only when the pre-stored similarity value does not exist, the similarity is calculated and stored in the look-up table It is desirable to do

4 is a diagram for explaining a process of setting a similar frame group in an image processing method according to an embodiment of the present invention, and illustrates a lookup table. Referring to FIG. 4 , for consecutive frames, two pairs of frame identifiers are set as an index, and a similarity value corresponding thereto is stored. As the similarity calculation process is repeated, the similarity value is recorded and a test is performed to form a similar frame group. For example, assuming that the threshold for similarity determination is set to '0.7', the successive sections of (f ₁ ,f ₂ ), (f ₂ ,f ₃ ), and (f ₃ ,f ₄ ) are 'similarity frames'. It can be confirmed that group #1' was formed. However, the similarity corresponding to (f ₄ ,f ₅ ) did not satisfy the threshold criterion, and the consecutive sections of (f ₅ ,f ₆ ) and (f ₆ ,f ₇ ) are 'Similar frame group #2'. was formed. Therefore, by referring to the lookup table generated in this way, it is possible to quickly determine which adjacent frame is similar to a frame at a specific point in time, and when the screen standard of an image is changed thereafter, it can be used as candidate data for obtaining information outside the region from the original image. .

As a second embodiment for detecting similar frames for each frame included in the video of the first screen standard and setting them as similar frame groups, a method of pre-calculating the similarity between frames according to time order can be utilized. First, similarities between adjacent frames are sequentially calculated for all frames included in a video of the first screen standard. Then, with respect to frame sections in which the calculated similarity value is equal to or greater than the threshold value, a similar frame group including only frames having a similarity value equal to or greater than the threshold value for each section may be set as a similar frame group. That is, while the first embodiment described above searches for previous/next adjacent frames from one reference frame, the second embodiment operates in a way of calculating the degree of similarity between adjacent frames in one direction. Of course, in this second embodiment, the lookup table introduced above can be utilized to manage the calculated similarity.

Returning to FIG. 2 again, in step S240, the image processing device refers to the similar frame group for each frame included in the input video of the first screen standard to generate an image for the unsecured area. In this process, with respect to the current frame included in the input moving picture of the first screen standard, the current frame is extended by using a geometric relationship between the current frame and the similar frame included in the similar frame group. Then, the video for the unsecured area may be generated by performing the process of expanding the current frame with respect to all frames included in the input video of the first screen standard.

Assume that a similar frame group including a plurality of similar frames having a high similarity to the current frame is set when extending the current frame (a basic frame serving as a reference). In a situation where the current frame is expanded by selecting one frame from the similar frame group, even if the contents of the current frame and the similar frame are the same (for example, a frame including a scene of a person running in an image), the frame is captured. The characteristics of the frames appearing may be different from each other according to the angle, perspective, illumination, etc. to which the camera is directed at the moment when the frame is displayed. Therefore, when extending the current frame, it is necessary to match the characteristics of the frame. For example, when a cube exists, the shape of the cube may be seen as a square, trapezoid, or arbitrary square depending on the angle at which the camera is directed.

In order to merge the current frame and similar frames into one frame, various corrections for characteristic elements including a camera angle and perspective between the two frames are required. From the implementation point of view, mathematical conversion such as enlarging or reducing an image is required for correction, and correction can be effectively performed using a series of processing procedures exemplified below.

First, an artificial neural network is constructed. Then, the basic three-dimensional model (cuboid, cube, cylinder, sphere, etc.) and its combined configuration model are artificially created using a computer using images viewed from various angles and perspectives, and are trained through an artificial neural network. Then, the artificial neural network learns the concept of angle and perspective, and when an image of an arbitrary angle and perspective is received as an input, it is in a state in which it can output an image of a specific angle and perspective. Now, an artificial neural network that takes two images as input and outputs the difference in angle and perspective between them is trained, and by using this, a similar frame is converted to match the angle and perspective characteristics with the basic frame to merge the two frames into one. can

5 and 6 are diagrams illustrating a process of generating an image of an unsecured area in an image processing method according to an embodiment of the present invention and program codes for the same.

Previously, in step S240 of FIG. 2 , it has been described that an image for an unsecured area can be generated by referring to a similar frame group for each frame included in the input video of the first screen standard. To this end, the current frame may be extended by extracting keypoints from each of the similar frames and the current frame and matching both images based on the extracted keypoints. Referring to FIG. 5, a process of generating an image larger than the original image from information obtained from preceding and succeeding frames adjacent to a given reference frame is illustrated, and FIG. 6 illustrates a corresponding frame by extending adjacent preceding and succeeding frames from the image of FIG. It shows that a larger image was created using the information obtained from the field.

Meanwhile, in FIG. 6 , it can be confirmed that blank areas in which information in the image is not secured appear in the lower left and upper right portions. In the following, technical means to compensate for this will be introduced.

7 and 8 are exemplary diagrams and block diagrams for explaining a process of supplementing blank areas in an image using a deep neural network, respectively.

In the embodiments of the present invention, an image of an unsecured area is created by referring to a group of similar frames. Nevertheless, information on the unsecured area may not be obtained from the similar frame group for the required screen size. That is, with respect to the current frame included in the input video of the first screen standard, the current frame is extended using the geometric relationship between the current frame and the similar frame included in the similar frame group, but the image for the unsecured area is not generated. In this case, the image for the unsecured area may be supplemented using a Generative Adversarial Network (GAN) or an autoencoder.

Referring to FIG. 7 , when compared with FIG. 6 , it can be seen that the area 710 at the upper left and the area 720 at the lower right are supplemented. Referring to FIG. 8 , first, a plurality of adjacent images including an image with empty information are input as an input image 810 . Then, the deep neural network 820 receives clue information 830 for the vacant image with empty information, so that an image capable of supplementing the information-empty area is newly generated and then output as an output image 840. print out In this case, the deep neural network 820 can be implemented using a GAN or an autoencoder.

Finally, in step S250 of FIG. 2 , the image processing device outputs a video having the second screen standard from the original video included in the video of the first screen standard and the video generated for the unsecured area. Here, with respect to all frames included in the video of the first screen standard, each original image and an image generated for the unsecured area may be combined and converted into an image having the second screen standard.

In the case where the frame is determined to have a high similarity above, three types may be considered. The first type is when the image is captured while the recording means (camera) is fixed, the second type is when the recording means is moved up/down/left/right, and the third type is when the recording means is used to zoom in (zoom-in)/zoom-out (zoom-out) to take a picture.

The first case can be taken as an example of a scene in which a presenter speaks in a news broadcast in a studio. When comparing the similarity, pixels with a large difference between frames are discarded, and resolution can be improved by using only pixels with a small difference. For example, if it is determined that four similar frames exist in a temporally successive section of a reference frame, 1 by 1 image information can be enhanced to 4 by 4 image information by using a similar frame group composed of these similar frames. .

In the second or third case, the reference frame can be expanded based on preceding and succeeding frames adjacent to the reference frame. For example, an image having an aspect ratio of 16:9 may be created by connecting images having an aspect ratio of 4:3.

If the quality of a video is improved using the video generation method proposed above, problems such as inconsistent resolution or unintended color (for example, black) appearing in some areas of the supplemented video may occur. there is. Therefore, image post-processing or filtering is required to resolve this sense of heterogeneity. From an implementation point of view, when the quality difference between the image generated for the unsecured area and the original image is greater than or equal to the acceptance limit, the resolution or bit rate of the inferior image may be supplemented from the similar frame. In addition, the boundary between the video generated for the unsecured area and the original video included in the video of the first screen standard may be smoothed.

FIG. 9 is a block diagram illustrating an image processing device 920 generating information outside an image area according to an embodiment of the present invention. Corresponding to the flowchart of FIG. function was shown. Therefore, in order to avoid duplication of description, only the general configuration is outlined here.

The input unit 921 is a means for receiving the video 910 having the first screen standard. The memory 923 stores a program for converting the video 910 having the first screen standard input through the input unit 921 into a video 930 having a second screen standard defined differently from the first screen standard. . The processor 925 is a subject that executes a program stored in the memory 923 . Here, the program stored in the memory 923 sets an image area corresponding to a difference between the first screen standard and a second screen standard defined differently from the first screen standard as an unsecured area, and Each similar frame is detected for each frame included in the video of the first screen standard and set as a similar frame group, and the similar frame group is set for each frame included in the input video of the first screen standard. and a command for generating an image of the unsecured area with reference to, and outputting a video having the second screen standard from an original video included in the video of the first screen standard and an image generated for the unsecured area.

The program stored in the memory 923 aims at the second screen standard, and due to a difference in screen standard including at least one of aspect ratio, resolution, and angle of view, the first An image area not possessed by the 1 screen standard can be set as an unsecured area.

A program stored in the memory 923 calculates a similarity between one reference frame included in the moving picture of the first screen standard and adjacent frames temporally preceding or following the reference frame, and comparing the adjacent frames with each other. If the similarity calculated from is greater than or equal to the threshold value, setting adjacent frames equal to or greater than the threshold value as a new reference frame, calculating similarities with adjacent frames, and checking whether the newly calculated similarity value is greater than or equal to the threshold value are repeated in chain. The process of setting only the frames whose similarity is equal to or higher than the threshold value as a similar frame group for the first reference frame, calculating the similarity for all frames included in the video of the first screen standard, and setting the similar frame group as the similar frame group. By performing this, similar frame groups can be derived for each frame.

The program stored in the memory 923 sequentially calculates the similarity between adjacent frames for all frames included in the video of the first screen standard, and the frame intervals in which the calculated similarity is greater than or equal to a threshold are consecutive in time series. For each section, it can be set as a similar frame group that includes only frames whose degree of similarity is equal to or higher than the threshold value.

The program stored in the memory 923 uses a geometric relationship between the current frame and the similar frame included in the similar frame group with respect to the current frame included in the input moving picture of the first screen standard, and uses the similar frame and the current frame. Expanding the current frame by extracting keypoints from each and matching both images based on the extracted keypoints, and extending the current frame for all frames included in the input video of the first screen standard A process may be performed to generate an image of the unsecured area. In addition, the program stored in the memory 923, when the image of the unsecured area is not generated through the process of extending the current frame, uses a Generative Adversarial Network (GAN) or an autoencoder to capture the unsecured area. You can supplement the video about it.

A program stored in the memory 923 may combine original images and images generated for the unsecured area for all frames included in the video of the first screen standard, respectively, and convert them into images having the second screen standard. there is.

According to the embodiments of the present invention, it is possible to actively and effectively respond to changes in screen specifications by expanding an image based on information in an adjacent frame in a video, but supplementing the image for an unexpanded area through artificial intelligence, As the screen size changes, it is possible to prevent letterboxing or pillarboxing, which is displayed in black because there is no information to be displayed in some areas of the video, and as a result, users experience quality satisfaction by fully utilizing the resources of the latest video playback device. can improve

Meanwhile, the embodiments of the present invention can be implemented as computer readable codes in a computer readable recording medium. The computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored.

Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. In addition, the computer-readable recording medium may be distributed to computer systems connected through a network, so that computer-readable codes may be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the present invention can be easily inferred by programmers in the technical field to which the present invention belongs.

In the above, the present invention was examined focusing on various embodiments thereof. Those of ordinary skill in the art pertaining to the present invention will be able to understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from an illustrative rather than a limiting point of view. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the equivalent scope will be construed as being included in the present invention.

810: input image
820: deep neural network 830: vacancy image clue information
840: output image
910 First standard video
920: image processing device 921: input unit
923: memory 925: processor
930: Second standard video

Claims (24)

(a) receiving, by an image processing device, a video having a first screen standard;
(b) setting, by the image processing device, an image area corresponding to a difference between the first screen standard and a second screen standard defined differently from the first screen standard as an unsecured area;
(c) detecting similar frames for each frame included in the input moving picture of the first screen standard by the image processing device and setting them as a similar frame group;
(d) generating, by the image processing device, an image for the unsecured area by referring to the similar frame group for each frame included in the input video of the first screen standard; and
(e) outputting, by the image processing device, a video having the second screen standard from an original image included in the video of the first screen standard and an image generated for the unsecured area; . According to claim 1,
In step (b),
An image that is not possessed by the first screen standard due to a difference in screen standards including at least one of aspect ratio, resolution, and angle of view targeting the second screen standard An image processing method of setting an area as an unsecured area. According to claim 1,
In step (c),
An image processing method of calculating a similarity value by comparing two adjacent frames using at least one of feature matching, template matching, and histogram comparison. According to claim 1,
In step (c),
(c1) calculating a similarity between one reference frame included in the video of the first screen standard and adjacent frames temporally preceding or following the reference frame;
(c2) If the similarity calculated from the adjacent frames is equal to or greater than the threshold, setting adjacent frames equal to or greater than the threshold as a new reference frame, calculating the similarity with each adjacent frame again, and checking whether the newly calculated similarity is equal to or greater than the threshold setting only frames whose degree of similarity is equal to or higher than a threshold value as a group of similar frames with respect to the first reference frame by repeating the process serially; and
(c3) deriving a similar frame group for each frame by performing steps (c1) and (c2) for all frames included in the video of the first screen standard. According to claim 4,
In step (c),
The similarity value between two adjacent frames is matched with an identifier pair of the corresponding frame and stored in a look-up table,
When calculating the similarity between two adjacent frames, first, the lookup table is searched, and if there is a pre-stored similarity value, the stored similarity value is read and used, and the similarity is calculated only when the pre-stored similarity value does not exist and storing them in the lookup table. According to claim 1,
In step (c),
(c4) sequentially calculating similarities between adjacent frames for all frames included in the video of the first screen standard; and
(c5) setting a similar frame group including only frames having a similarity equal to or greater than the threshold value for each frame interval in which the calculated similarity value is equal to or greater than the threshold value in time-sequential succession; According to claim 1,
In step (d),
(d1) extending the current frame with respect to the current frame included in the input moving picture of the first screen standard by using a geometric relationship between the current frame and the similar frame included in the similar frame group; and
(d2) generating an image for the unsecured area by performing the frame extension of step (d1) on all frames included in the input video of the first screen standard. According to claim 7,
In the step (d1),
The image processing method of extending the current frame by extracting a keypoint from each of the similar frame and the current frame and matching both images based on the extracted keypoint. According to claim 7,
(d3) supplementing an image of the unsecured region using a Generative Adversarial Network (GAN) or an autoencoder when the image of the unsecured region is not generated through frame extension in step (d1); Further comprising a, image processing method. According to claim 1,
In step (e),
(e1) converting an image having the second screen standard by combining an original image and an image generated for the unsecured area for all frames included in the video of the first screen standard; processing method. According to claim 10,
(e2) supplementing the resolution or bit rate of the inferior image from the similar frame when the quality difference between the image generated for the unsecured area and the original image is greater than the acceptance limit; , image processing method. According to claim 10,
(e3) smoothing a boundary between the image generated for the unsecured area and the original image included in the video of the first screen standard; A computer-readable recording medium recording a program for executing the method of any one of claims 1 to 12 in a computer. an input unit for receiving a video having a first screen standard;
a memory for storing a program for converting the input video having the first screen standard into a video having a second screen standard defined differently from the first screen standard; and
A processor that executes a program stored in the memory; includes,
Programs stored in the memory,
Based on the first screen standard, an image area corresponding to a difference between the first screen standard and a second screen standard defined differently is set as an unsecured area, and a frame included in the input video of the first screen standard ( Detect similar frames for each frame, set them as a similar frame group, and create an image for the unsecured area by referring to the similar frame group for each frame included in the input moving picture of the first screen standard; and a command for outputting a video having the second screen standard from an original video included in the video of the first screen standard and an image generated for the unsecured area. 15. The method of claim 14,
Programs stored in the memory,
An image that is not possessed by the first screen standard due to a difference in screen standards including at least one of aspect ratio, resolution, and angle of view targeting the second screen standard An image processing device that sets an area as an unsecured area. 15. The method of claim 14,
Programs stored in the memory,
For one reference frame included in the video of the first screen standard, a similarity with adjacent frames temporally preceding or following the reference frame is calculated,
If the similarity calculated from the adjacent frames is greater than or equal to the threshold value, the adjacent frames that are greater than or equal to the threshold value are set as new reference frames, the similarities with the adjacent frames are calculated again, and the process of checking whether the newly calculated similarity value is greater than or equal to the threshold value is chained. By repeatedly repeating, only frames whose similarity is equal to or higher than the threshold are set as similar frame groups for the first reference frame.
The image processing apparatus of claim 1 , wherein a similar frame group is derived for each frame by performing a process of calculating the degree of similarity and a process of setting the similar frame group for all frames included in the video of the first screen standard. 15. The method of claim 14,
Programs stored in the memory,
For all frames included in the video of the first screen standard, similarities between adjacent frames are sequentially calculated, respectively;
An image processing device configured to set a similar frame group including only frames having a similarity equal to or greater than a threshold value for each section of frame sections in which the calculated similarity value is equal to or greater than the threshold value. 15. The method of claim 14,
Programs stored in the memory,
With respect to the current frame included in the input video of the first screen standard, a keypoint is extracted from each of the similar frame and the current frame by using a geometric relationship between the similar frame included in the similar frame group and the current frame; Expanding the current frame by matching both images based on the extracted feature points;
An image processing device that generates an image for the unsecured area by performing a process of extending the current frame for all frames included in the input video of the first screen standard. According to claim 18,
Programs stored in the memory,
When the image of the unsecured region is not generated through the process of extending the current frame, supplementing the image of the unsecured region using a Generative Adversarial Network (GAN) or an autoencoder. Image processing device. 15. The method of claim 14,
Programs stored in the memory,
An image processing device for converting an image having the second screen standard by combining an original image and an image generated for the unsecured area, respectively, for all frames included in the video of the first screen standard. According to claim 1,
In step (d),
For each frame included in the input video of the first screen standard, the similar frame group is referred to, and the angle and perspective characteristics of the similar frame are determined by using an artificial neural network that has learned the angle and perspective of the object. An image processing method of generating an image of the unsecured area by merging the current frame and the similar frame into one frame by correcting frame characteristics to match . According to claim 21,
In step (d),
(d1) With respect to the current frame included in the input video of the first screen standard, the current frame is expanded by using the geometric relationship between the similar frame included in the similar frame group and the current frame, and the object is viewed from various angles and perspectives. correcting a camera angle and perspective between the current frame and the similar frame using the artificial neural network that outputs a difference in angle and perspective between input images by pre-learning an image viewed from the camera; and
(d2) generating an image for the unsecured area by performing the frame expansion and correction of step (d1) on all frames included in the input video of the first screen standard; . 15. The method of claim 14,
Programs stored in the memory,
For each frame included in the input video of the first screen standard, the similar frame group is referred to, but the angle and perspective characteristics of the similar frame are applied to the current frame using an artificial neural network that has learned the angle and perspective of the object. and generating an image of the unsecured area by merging the current frame and the similar frame into one frame by correcting frame characteristics to match. 24. The method of claim 23,
Programs stored in the memory,
With respect to the current frame included in the input video of the first screen standard, a keypoint is extracted from each of the similar frame and the current frame by using a geometric relationship between the similar frame included in the similar frame group and the current frame; Expanding the current frame by matching both images based on the extracted feature points, using the artificial neural network that outputs the angle and perspective differences between input images by pre-learning images viewed from various angles and perspectives Correct camera angle and perspective between the current frame and the similar frame;
An image processing device that generates an image for the unsecured area by performing a process of extending and correcting the current frame for all frames included in the input video of the first screen standard.

Images