KR102640653B1 - Apparatus and method for image processing - Google Patents

Abstract

The present invention relates to an image processing device and method. The disclosed image processing device and method are capable of editing or modifying an image acquired from an arbitrary camera while maintaining such pattern and characteristic information, and re-mosaicing the de-mosaiced input image. It includes a mosaic unit that returns the image to the state obtained at the sensor end, an interpolator that interpolates empty information of the mosaiced image, and a generator that learns to have the same demosaicing pattern as the input image, and the generator Provides a depth image processing device and method, characterized in that the learning involves dividing an input image into a plurality of patches and learning such that characteristics within the generated patch are similar to characteristics of other patches of the input image. According to the present invention, there is an advantage of providing an image processing device and method that can edit or modify an input image while preserving as much as possible the patterns and characteristics that occur during the image acquisition process.

Description

Image processing apparatus and method {APPARATUS AND METHOD FOR IMAGE PROCESSING}

The present invention relates to an image processing device and method, and more specifically, to an image processing device and method that can edit or modify an image randomly captured without any special information while maintaining the demosaicing pattern of the image. .

This study was funded by Korea Aerospace University's basic research project, "Research on a counter-antiforensic system capable of responding to generative models using the characteristics of AI-based video compression" (research period: 2022.3.1. ~ 2025.2.28). Supported by (host organization: Korea Aerospace University).

The content described in this section simply provides background information for this embodiment and does not constitute prior art.

During the digital image acquisition process, there is a process in which the image sensor acquires color information such as R, G, and B at all sensing positions. However, if there is no special device, there is a problem in that only one color information is obtained from one sensor. In a typical camera, interpolation technology is used to predict information about the current location from surrounding information to fill in the information that was not obtained, and this interpolation technology is called demosaicing. There is no specific standard for demosaicing technology, and each manufacturer applies its own algorithm for interpolation, so by analyzing the demosaicing pattern, it is possible to find out the type of camera that captured the video. In other words, such a pattern can be used as important information containing image information and the authenticity of the image.

Meanwhile, the number of cases of arbitrarily manipulating and distributing given videos for specific purposes has recently been increasing. In order to prevent such indiscriminate manipulation of images, various video forensic technologies have been developed to find manipulated images. In particular, as an example of forensic technology, technology was introduced to find traces of manipulation by analyzing demosaicing patterns in the aforementioned video.

On the other hand, when you want to modify or edit a video for various reasons, there is a need for the modified video to maintain the demosaicing pattern of the original video. The easiest way is to analyze and model the characteristics of the captured camera and reproduce them during the demosaicing process of the modified video. In the case of widely known cameras, technology has been introduced to reproduce demosaicing patterns like this as long as there is enough data.

However, with the technology for reproducing the demosaicing pattern according to the prior art, if only an image is provided without camera information, it becomes difficult to edit the image while maintaining the demosaicing pattern based on this. Although it is possible to track the camera model by analyzing the video, it is generally unrealistic to analyze each model for all cameras actually used and create a database.

Accordingly, the present invention proposes an image processing device and method that can modify/edit an image while maintaining the inherent pattern of the image compared to the prior art.

Korean Patent Publication No. 10-2022-0084236, published on June 21, 2022 (Name: Improved detection system and method for manipulated images) Korean Patent Publication No. 10-18181181, published on June 22, 2017 (name: Digital Forensic Image Verification System) Korean Patent Publication No. 10-1687989, published on December 20, 2016 (Name: Digital forensic image verification system and imaging and image storage devices used therein)

(Non-patent Document 1) [1] Chen, Chen, Xinwei Zhao, and Matthew C. Stamm. “Mislgan: An anti-forensic camera model falsification framework using a generative adversarial network.”?2018 25th IEEE International Conference on Image Processing (ICIP). IEEE, 2018. (Non-patent Document 2) Andrews, Jerone T.A., Yidan Zhang, and Lewis D. Griffin. “Conditional Adversarial Camera Model Anonymization.” European Conference on Computer Vision. Springer, Cham, 2020.

The present invention was proposed to solve the problems of the prior art described above. When editing or modifying an input image, the present invention provides image processing that allows editing or modification while preserving as much as possible the patterns and characteristics that occur during the image acquisition process. The main purpose is to provide devices and methods.

In addition, another object of the present invention is to provide an image processing device and method that can determine the authenticity of the image and the camera model by identifying patterns and characteristics in the image.

The problems to be solved by the present invention are not limited to those mentioned above, and other problems to be solved that are not mentioned will be clearly understood by those skilled in the art to which the present invention pertains from the following description.

One aspect of the present invention for achieving the above-described object is an image processing device capable of editing or modifying an image acquired from an arbitrary camera while maintaining such pattern and characteristic information, wherein the demosaicing A mosaicing unit that mosaics the input image again and returns it to the state obtained at the sensor end of the image; an interpolation unit that interpolates empty information of the mosaiced image; and a generator that learns to have the same demosaicing pattern as the input image, wherein the generator's learning divides the input image into a plurality of patches so that characteristics within the generated patch are similar to characteristics of other patches of the input image. Provided is a depth image processing device characterized in that it learns to become similar.

Another aspect of the present invention is an image processing method that can edit or modify an image acquired from an arbitrary camera while maintaining the pattern and characteristic information, and includes demosaicing the image given to the image processing device. Entering; Mosaicing the input image again to return it to the state obtained at the sensor end of the image; filling empty information of the input image using a commonly used interpolation method; And in order for the generator to learn to have the same demosaicing pattern as the input image, the input image is divided into a plurality of patches and the characteristics of the generated patch are learned to be similar to the characteristics of other patches of the input image. Provides a depth image processing method comprising the following steps:

Another aspect of the present invention provides a computer-readable recording medium on which a program for executing an image processing method is recorded.

According to the image processing device and method of the present invention, when editing or modifying an input image, it is possible to provide an image processing device and method that can perform editing or modification while preserving as much as possible the patterns and characteristics that occur during the image acquisition process. There is an effect that it can be done.

In addition, it is possible to provide an image processing device and method that can determine the authenticity of the image and the camera model by identifying patterns and characteristics in the image.

The effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. .

The accompanying drawings, which are included as part of the detailed description to aid understanding of the present invention, provide embodiments of the present invention, and together with the detailed description, explain technical features of the present invention.
Figure 1 is a diagram illustrating the configuration of an image processing device according to an embodiment of the present invention.
Figure 2 is a diagram illustrating an image processing method according to an embodiment of the present invention.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the attached drawings. The detailed description set forth below in conjunction with the accompanying drawings is intended to illustrate exemplary embodiments of the invention and is not intended to represent the only embodiments in which the invention may be practiced. The following detailed description includes specific details to provide a thorough understanding of the invention. However, those skilled in the art will recognize that the present invention may be practiced without these specific details.

In some cases, in order to avoid ambiguity of the concept of the present invention, well-known structures and devices may be omitted or may be shown in block diagram form focusing on the core functions of each structure and device.

Throughout the specification, when a part is said to "comprise or include" a certain element, this means that it may further include other elements rather than excluding other elements unless specifically stated to the contrary. do. In addition, terms such as “… unit”, “… unit”, “module”, etc. used in the specification refer to a unit that processes at least one function or operation, which may be implemented as hardware, software, or a combination of hardware and software. there is. In addition, the terms “a or an,” “one,” “the,” and similar related terms are used in the context of describing the present invention (particularly in the context of the claims below) as used herein. It may be used in both singular and plural terms, unless indicated otherwise or clearly contradicted by context.

In describing embodiments of the present invention, if a detailed description of a known function or configuration is judged to unnecessarily obscure the gist of the present invention, the detailed description will be omitted. The terms described below are terms defined in consideration of functions in the embodiments of the present invention, and may vary depending on the intention or custom of the user or operator. Therefore, the definition should be made based on the contents throughout this specification.

Each component in the drawings of the present invention is shown independently to represent different characteristic functions in the image processing device and method, and does not mean that each component is comprised of separate hardware or a single software unit. That is, each component is listed and included as a separate component for convenience of explanation, and at least two of each component can be combined to form one component, or one component can be divided into a plurality of components to perform a function, and each of these components can be divided into a plurality of components. Integrated embodiments and separate embodiments of the constituent parts are also included in the scope of the present invention as long as they do not deviate from the essence of the present invention.

Additionally, some components may not be essential components that perform essential functions in the present invention, but may simply be optional components to improve performance. The present invention can be implemented by including only essential components for implementing the essence of the present invention excluding components used only to improve performance, and a structure including only essential components excluding optional components used only to improve performance. is also included in the scope of rights of the present invention.

Hereinafter, embodiments of the present invention will be described with reference to the attached drawings.

Figure 1 is a diagram illustrating the configuration of an image processing device according to an embodiment of the present invention.

In one embodiment of the present invention, a method of editing the video is used while maintaining the demosaicing pattern of the input video. In detail, it mainly uses the deep learning-based Generative adversarial network (GAN) structure, which is mainly used in the field of image generation recently.

Deep Learning is also called deep learning, as it performs machine learning using an artificial neural network (ANN) with multiple layers.

GAN stands for 'Generative Adversarial Network' and refers to a model (Network) in which generators and identifiers compete with each other (Adversarial) to generate data (Generative). If a portrait is generated using a GAN, the person who creates the portrait is called a generator, and the person who evaluates the created portrait is called a discriminator. The main concept of GAN is that the generator and separator are in opposition to each other, and learning progresses in a way that gradually improves each other's performance. Machine learning is largely classified into three concepts: supervised learning/reinforcement learning/unsupervised learning, and GAN corresponds to 'unsupervised learning'.

In one embodiment of the present invention, the mosaiced image given by the mosaic unit 110 is first mosaiced again to return it to the state actually obtained at the sensor stage.

Afterwards, the interpolation unit 130 first fills in all empty information in the image using a commonly used interpolation method.

Afterwards, the image is restored using the proposed generative adversarial network (GAN) method. In order to train the generator (150) of the GAN system to have the same demosaicing pattern as the original image, another image of the same image is used as shown in Figure 1. We propose a method using patches.

For example, in order to generate a demosaicing pattern for the x0 patch of the original image, patches at different positions such as x1 of the original image are compared with each other to induce the generation of the same demosaicing pattern within the image processing device 100. .

At this time, in order to minimize the relative distance between the original pattern and the generated pattern, the internal distance (intra-distance) of the patterns generated by the proposed generator 150 is the distance (inter-distance) between the patterns generated by other devices. ) to induce learning to be relatively smaller than ). To this end, the proposed GAN system includes a pre-trained generator (170) that models the demosaicing generation method used in existing devices.

The embedder 190 extracts a measurable feature vector through the pattern generated by the generator 150 and the pre-trained generator 170.

Embedding refers to the value or process of converting a word, sentence, or document into a vector. It preserves necessary information while converting high-dimensional information to low-dimensional information.

The following four loss functions L1, L2, L3, and L4 are generated through the feature vector extracted from the embedder 190, and learning is performed based on these.

Deep learning progresses learning by reducing the error between predictions and results, and the function that represents the error is the loss function.

The loss function serves to measure how well the Neural Network's predictions are correct. The loss value obtained from the loss function serves as an indicator to check how well the Neural Network has been trained during the training process.

The loss function of the image processing device 100 of the present invention is as follows, and various elements can be added or subtracted.

L1: Loss function that distinguishes between m different pre-trained generators (170, pre-trained generator)

L2: Loss function that classifies the patches of the original video and the output patches of the proposed generator (150, generator)

L3: Loss function that measures the distance between features of the outputs of proposal generators (150, generators)

L4: Loss function that measures the distance between features between the proposed generator (150, generator) and the pre-trained generator (170, pre-trained generator)

Figure 2 is a diagram illustrating an image processing method according to an embodiment of the present invention.

First, the de-mosaiced image given to the image processing device 100 is input (S201), and mosaicing is performed again to return it to the state actually obtained at the sensor stage (S203).

Afterwards, all empty information in the image is first filled using a commonly used interpolation method (S203).

Afterwards, the image is restored using the proposed generative adversarial network (GAN) method, and the generator of the GAN system is trained to have the same demosaicing pattern as the original image (S205).

Specifically, different patches of the same image are used. In order to generate a demosaicing pattern for the x0 patch of the original image, patches at different positions such as x1 of the original image are compared with each other, and the image processing device 100 It induces the creation of the same demosaicing pattern within.

At this time, in order to minimize the relative distance between the original pattern and the generated pattern, the internal distance (intra-distance) of the patterns generated by the proposed generator 150 is the distance (inter-distance) between the patterns generated by other devices. ) to induce learning to be relatively smaller than ).

To this end, a pre-trained generator that models the demosaicing generation method used in existing devices is secured (S207) and included in the proposed GAN system.

Embedding, that is, a measurable feature vector, is extracted through the pattern generated by the generator and the pre-trained generator (S209).

Through the extracted feature vectors, the following four loss functions L1, L2, L3, and L4 are generated (S211), and learning is performed based on them.

The loss function is as follows, and various elements can be added or subtracted.

L1: Loss function that distinguishes m different pre-trained generators.

L2: Loss function that classifies the patches of the original video and the output patches of the proposal generator.

L3: Loss function that measures the distance between features of the outputs of proposal generators.

L4: Loss function that measures the distance between features between the proposed generator and the pre-trained generator.

In Figure 2, steps S201 to S211 are described as being sequentially executed, but this is merely an illustrative explanation of the technical idea of this embodiment, and those skilled in the art in the technical field to which this embodiment belongs will understand the steps of this embodiment. Since it is possible to apply various modifications and modifications by changing the order shown in FIG. 2 or executing one or more of steps S201 to S211 in parallel without departing from the essential characteristics, FIG. 2 is shown in a time-serial order. It is not limited.

Combinations of each block of the block diagram and each step of the flow diagram attached to this specification may be performed by computer program instructions. Since these computer program instructions can be mounted on the processor of a general-purpose computer, special-purpose computer, or other programmable data processing equipment, the instructions performed through the processor of the computer or other programmable data processing equipment are shown in each block of the block diagram or flow diagram. Each step creates the means to perform the functions described. These computer program instructions may also be stored in computer-usable or computer-readable memory that can be directed to a computer or other programmable data processing equipment to implement a function in a particular manner, so that the computer-usable or computer-readable memory The instructions stored in can also produce manufactured items containing instruction means that perform the functions described in each block of the block diagram or each step of the flow diagram. Computer program instructions can also be mounted on a computer or other programmable data processing equipment, so that a series of operational steps are performed on the computer or other programmable data processing equipment to create a process that is executed by the computer, thereby generating a process that is executed by the computer or other programmable data processing equipment. Instructions that perform processing equipment may also provide steps for executing functions described in each block of the block diagram and each step of the flow diagram.

Additionally, each block or each step may represent a module, segment, or portion of code that includes one or more executable instructions for executing specified logical function(s). Additionally, it should be noted that in some alternative embodiments it is possible for the functions mentioned in the blocks or steps to occur out of order. For example, two blocks or steps shown in succession may in fact be performed substantially simultaneously, or the blocks or steps may sometimes be performed in reverse order depending on the corresponding function.

The above description is merely an illustrative explanation of the technical idea of the present invention, and various modifications and variations will be possible to those skilled in the art without departing from the essential characteristics of the present invention. Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but are for illustrative purposes, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention shall be interpreted in accordance with the claims below, and all technical ideas within the equivalent scope shall be construed as being included in the scope of rights of the present invention.

According to the image processing device and method of the present invention, when editing or modifying an input image, it is possible to provide an image processing device and method that can edit or modify the input image while preserving as much as possible the patterns and characteristics that occur during the image acquisition process. In that it can be used as a viable solution, it overcomes the limitations of existing technology, and not only has the potential to market or sell the applied device, not only the use of the related technology, but also the degree to which it can be clearly implemented in reality, making it industrially viable. It is an invention that has potential for use.

100: Image processing device 110: Mosaic unit 130: Interpolation unit
150: Generator 170: Pretrained constructor
190: Embedder

Claims (19)

In an image processing device that can edit or modify an image acquired from an arbitrary camera while maintaining pattern and characteristic information,
A mosaicing unit that re-mosaics the de-mosaiced input image and returns it to the state obtained at the sensor end of the image;
an interpolation unit that interpolates empty information of the mosaiced image; and
Includes a generator that learns to have the same demosaicing pattern as the input image,
The learning of the generator divides the input image into a plurality of patches, and compares second patches at different positions of the input image to generate a demosaicing pattern for the first patch of the input image, so that the same A depth image processing device characterized in that it induces the creation of a demosaicing pattern. According to paragraph 1,
The constructor is:
A depth image processing device characterized by performing learning with a deep learning-based generative adversarial network (GAN) structure. delete According to paragraph 1,
Inducing the generation of the same demosaicing pattern is,
In order to minimize the relative distance between the original pattern and the pattern generated by the generator, the internal distance (intra-distance) of the generated patterns is relatively greater than the distance (inter-distance) between patterns generated by other devices. A depth image processing device characterized by inducing learning to become smaller. According to paragraph 4,
In order to minimize the relative distance,
A pre-trained generator that models the demosaicing generation method used in existing devices; and
A depth image processing device further comprising an embedder that extracts a measurable feature vector through the pattern generated by the generator and the pre-trained generator. According to clause 5,
A depth image processing device that generates a loss function using the feature vector extracted from the embedder and performs learning using the loss function. According to clause 6,
The loss function is,
A first loss function that distinguishes between m different pretrained generators;
a second loss function that classifies the patch of the input image and the output patch of the generator;
a third loss function that measures distances between features of outputs of the generator; and
A fourth loss function that measures the distance between features between the generator and the pre-trained generator.
A depth image processing device comprising one or more of the following. In an image processing method that can edit or modify an image acquired from an arbitrary camera while maintaining pattern and characteristic information,
Inputting a given de-mosaiced image into an image processing device;
Mosaicing the input image again to return it to the state obtained at the sensor end of the image;
Filling in empty information of the input image using an interpolation method; and
In order for the generator to learn to have the same demosaicing pattern as the input image, the input image is divided into a plurality of patches, and in order to generate a demosaicing pattern for the first patch of the input image, the input image is divided into multiple patches. A depth image processing method comprising the step of comparing second patches at different positions to generate the same demosaicing pattern.
delete According to clause 8,
Inducing the generation of the same demosaicing pattern is,
In order to minimize the relative distance between the original pattern and the pattern generated by the generator, the internal distance (intra-distance) of the generated patterns is relatively greater than the distance (inter-distance) between patterns generated by other devices. A depth image processing method characterized by inducing learning to become smaller. According to clause 10,
In order to minimize the relative distance,
Securing a pre-trained generator that models the demosaicing generation method used in existing devices; and
A depth image processing method further comprising an embedding step of extracting a measurable feature vector through the pattern generated by the generator and the pre-trained generator. According to clause 11,
A depth image processing method characterized by generating a loss function through the feature vector extracted from the embedder and performing learning using the loss function. According to clause 12,
The loss function is,
A first loss function that distinguishes between m different pretrained generators;
a second loss function that classifies the patch of the input image and the output patch of the generator;
a third loss function that measures distances between features of outputs of the generator; and
A fourth loss function that measures the distance between features between the generator and the pre-trained generator.
A depth image processing method comprising one or more of the following. In an image processing method that can edit or modify an image acquired from an arbitrary camera while maintaining pattern and characteristic information,
Inputting a given de-mosaiced image into an image processing device;
Mosaicing the input image again to return it to the state obtained at the sensor end of the image;
Filling in empty information of the input image using an interpolation method; and
In order for the generator to learn to have the same demosaicing pattern as the input image, the input image is divided into a plurality of patches , and to generate a demosaicing pattern for the first patch of the input image, the input image is divided into multiple patches. A computer-readable recording medium recording a program for executing a depth image processing method , comprising the step of comparing second patches at different positions in the image to generate the same demosaicing pattern .
delete delete delete delete delete