KR102364822B1 - Method and apparatus for recovering occluded area - Google Patents

Abstract

The present invention relates to a method and apparatus for recovering an occluded area to generate a restoration image suitable for an occluded area. According to one embodiment of the present invention, a recovery method is performed by an electronic device which recovers an object (occluded object) having an occluded area in an image to an object (restoration object) in which the occluded area is restored, wherein restoration is performed by using a restorer which is a model learned with a machine learning technique, to output a restoration object when receiving an occluded object and additional information related to the occluded object. The recovery method comprises the following steps: analyzing the type of an object occluded in the image; checking whether a pre-learned restorer exists based on the analyzed type of object; selecting a corresponding restorer when the restorer exists in the checking step; and determining whether re-learning is required for the selected restorer according to additional information related to the analyzed type of the object and using the selected restorer according to the determined necessity or using a restorer retrained from the selected restorer on the basis of the additional information to generate a restored object.

Description

METHOD AND APPARATUS FOR RECOVERING OCCLUDED AREA

The present invention relates to a method and apparatus for reconstructing an obscured area, and more particularly, to a method and apparatus for reconstructing a specific object such as a person in an image captured by a camera when it is obscured.

The restoration problem for the occluded region in the image can be defined as “the problem of estimating the occluded image region using the image region that is not occluded in the image”.

Conventionally, in order to restore an area covered by a person or an object in an image, a method using spatial correlation and feature information existing between pixels (pixels, pixels, picture elements) of an image is conventionally used.

For example, to estimate the occluded pixel using spatial correlation, interpolation is used. That is, an arbitrary pixel in an unoccluded area is received, the distribution of surrounding pixels is assumed to be a function expression such as a straight line, a curved line, or a flat or curved surface, and the hidden pixels are interpolated to estimate. Although this prior art is widely used because it is simple and easy to implement, if the occluded region has poor correlation with the surrounding region or the occluded region is very large, the possibility of erroneous estimation of the restoration result increases. In particular, it is not possible to know the exact form of a function formula (eg, a straight line, a curve, a plane, a curved surface, etc.) only with a known pixel, and it can be predicted only through a predetermined assumption, and the detailed setting value is regarded as a kind of hyper parameter. Therefore, there was a difficulty in estimating this additionally.

The latest approach to improve this problem is to use an artificial neural network. That is, it is a method of utilizing spatial correlation or feature information of an image by learning an artificial neural network using the acquired training data set. However, in the conventional artificial neural network technology, a problem in which a learned artificial neural network is overfitted to a learning data set is excessively overfitted to a learning data set.

In particular, side information about an object hidden in the restoration process may be additionally provided during the restoration process. However, in the conventional artificial neural network technology, since such additional information is not inferred during the learning process, and the artificial neural network is learned in a state in which the corresponding additional information is not reflected, the corresponding additional information cannot be used in the restoration process, so the utilization and accuracy of the restoration object are reduced. There was a problem with falling.

KR 10-2006-0112666 A

In order to solve the problems of the prior art as described above, it is an object of the present invention to provide a method and apparatus for reconstructing an obscured area using side information in a process of reconstructing an obscured area in an image.

However, the problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned can be clearly understood by those of ordinary skill in the art to which the present invention belongs from the description below. There will be.

In a restoration method according to an embodiment of the present invention for solving the above problems, an object having an obscured area (occluded object) in an image is restored to an object (restored object) in which the obscured area is restored, but the obscured object and its object A method performed by an electronic device that restores using a restorer, which is a model learned by machine learning, to output a restored object when receiving additional information related to step; checking whether there is a previously learned reconstructor based on the analyzed type of object; selecting a corresponding restorer if there is a restorer in the checking step; and determining whether re-learning of the selected reconstructor is necessary according to additional information related to the analyzed type of object, and uses the selected reconstructor according to the determined necessity or re-learns the selected reconstructor based on the additional information and generating a restoration object by using the learned restoration unit.

In the restoration method according to an embodiment of the present invention, when a restoration device is absent in the selecting step, the analyzed type of object is collected, and the restoration is newly learned based on the collected object and additional information related to the object. It may further include the step of creating a restoration object using the group.

The restoration method according to an embodiment of the present invention may further include setting a region including the occluded object in the image, and the analyzing may include performing the analysis on the occluded object in the restoration region. there is.

The relearned restorer may be relearned using a first generator learned by machine learning techniques to output an unobscured object and a hidden object related to the additional information when additional information is input.

The generating may include, when re-learning of the selected reconstructor is unnecessary, generating a reconstructed object by inputting an object hidden in the selected reconstructor and additional information thereof; When re-learning of the selected reconstructor is required, an image related to the analyzed type of unoccluded object is collected, and the unoccluded object of the collected image and its additional information are input to the first generator, and After generating a occluded object, re-learning the selected reconstructor using the occluded object of the collected image generated by the first generator and its additional information, the occluded object and its additional information are input into the re-learned reconstructor to restore the object may include; generating

The reconstructor includes a second generator for generating a reconstructed object from the obscured object, and a discriminator for discriminating the reconstructed object generated by the second generator, respectively, wherein the second generator and the discriminator are adversarially learned based on a Generative Adversarial Network (GAN). can

The second generator may be implemented based on an auto-encoder, a variant auto-encoder, a U-Net generator, a ResNET generator, or a generative adversarial network.

The additional information may be provided through analysis of a region other than the occluded region of the occluded object.

The additional information may be provided through analysis of other images captured at the same location where the image was captured.

The additional information may be input or selected by a user and provided.

The additional information may include attribute information on the analyzed type of object, information on a plurality of detailed objects included in the analyzed type of object, or 3D information on the analyzed type of object.

A restoration apparatus according to an embodiment of the present invention is an apparatus for restoring an object (occluded object) having an occluded area in an image into an object (restored object) in which the occluded region is restored, inputting the occluded object and additional information related to the object. A memory for storing a restorer that is a model trained by machine learning techniques to output a restored object when received; and a control unit that controls to create a restoration object by using the restoration unit stored in the memory.

A restoration apparatus according to an embodiment of the present invention is an apparatus for restoring an object (occluded object) having an occluded area in an image into an object (restored object) in which the occluded region is restored, inputting the occluded object and additional information related to the object. a communication unit for receiving a restorer that is a model trained by a machine learning technique to output a restored object when received; and a control unit controlling to generate a restoration object using the restoration unit received from the communication unit.

The control unit analyzes the type of the object obscured from the image, confirms the existence of a pre-learned reconstructor based on the analyzed type of object, and if there is a reconstructor, selects the reconstructor, Determining whether or not re-learning for the selected reconstructor is necessary according to additional information related to the type of object, and using the selected reconstructor according to the determined necessity or re-learning the selected reconstructor based on the additional information By using it, you can control to create a restoration object.

The present invention configured as described above has an advantage in that a reconstructed image more suitable for the occluded region can be generated by restoring the occluded region of the image using side information at the restoration time.

In addition, the present invention can augment the training data using the available additional information, and at the same time modify the input data to retrain the pre-learned model, so that the restoration performance can be improved.

In addition, since the present invention generates a restored image according to additional information at the time of restoration, its utility is increased, such as CCTV control site, whether a person matches, image restoration after an incident, biometric information-based authentication in a mobile terminal, building entrance, etc. In application fields such as access control, there is an advantage that can be widely used to restore a hidden area of an object when it exists.

The effects obtainable in the present invention are not limited to the above-mentioned effects, and other effects not mentioned may be clearly understood by those of ordinary skill in the art to which the present invention belongs from the following description. will be.

1 shows a conceptual diagram for restoration of an area covered by the present invention.
FIG. 2 shows an example of reconstructing a hidden object using ideal additional information.
3 illustrates an example of reconstructing a hidden object using realistic additional information.
4 is a block diagram illustrating a restoration apparatus 100 according to an embodiment of the present invention.
5 is a block diagram of the control unit 150 in the restoration apparatus 100 according to an embodiment of the present invention.
6 is a flowchart of a restoration method according to an embodiment of the present invention.
7 shows a specific example of FIG. 6 (ie, an example of a case in which re-learning is required for a previously learned reconstructor).
8 shows a conceptual diagram for learning a reconstructor for each of various objects.
9 is a diagram illustrating a process of restoring an obscured object in terms of input and output.
10 shows an example of a specific learning process for a reconstructor.
11 shows an example of providing additional information.
12 shows an example of a learning process of a second generator and a discriminator.

The above object and means of the present invention and its effects will become more apparent through the following detailed description in relation to the accompanying drawings, and accordingly, those of ordinary skill in the art to which the present invention pertains can easily understand the technical idea of the present invention. will be able to carry out In addition, in describing the present invention, if it is determined that a detailed description of a known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

The terminology used herein is for the purpose of describing the embodiments, and is not intended to limit the present invention. In the present specification, the singular form also includes the plural form as the case may be, unless otherwise specified in the phrase. In this specification, terms such as “include”, “provide”, “provide” or “have” do not exclude the presence or addition of one or more other components other than the mentioned components.

In this specification, terms such as “or” and “at least one” may indicate one of the words listed together, or a combination of two or more. For example, “or B” and “at least one of B” may include only one of A or B, or both A and B.

In the present specification, descriptions according to “for example” and the like may not exactly match the information presented, such as recited properties, variables, or values, tolerances, measurement errors, limits of measurement accuracy, and other commonly known factors The embodiments of the present invention according to various embodiments of the present invention should not be limited by effects such as modifications including .

In this specification, when it is described that a certain element is 'connected' or 'connected' to another element, it may be directly connected or connected to the other element, but other elements may exist in between. It should be understood that there may be On the other hand, when it is mentioned that a certain element is 'directly connected' or 'directly connected' to another element, it should be understood that the other element does not exist in the middle.

In this specification, when it is described that a certain element is 'on' or 'in contact with' another element, it may be directly in contact with or connected to the other element, but another element may exist in the middle. It should be understood that On the other hand, when it is described that a certain element is 'directly on' or 'directly' of another element, it may be understood that another element does not exist in the middle. Other expressions describing the relationship between the elements, for example, 'between' and 'directly between', etc. may be interpreted similarly.

In this specification, terms such as 'first' and 'second' may be used to describe various components, but the components should not be limited by the above terms. In addition, the above terms should not be construed as limiting the order of each component, and may be used for the purpose of distinguishing one component from another. For example, a 'first component' may be referred to as a 'second component', and similarly, a 'second component' may also be referred to as a 'first component'.

Unless otherwise defined, all terms used herein may be used with meanings commonly understood by those of ordinary skill in the art to which the present invention pertains. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly.

Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the accompanying drawings.

1 shows a conceptual diagram for restoration of an area covered by the present invention.

As shown in Fig. 1, the present invention relates to a technique for restoring an object having an obscured area (i.e., an obscured object) in an image captured by a camera into an object in which the obscured area is restored (i.e., a restored object). , video surveillance systems, etc. In this case, the reconstructor (ie, the obscured object reconstructor) is a learning model learned by a machine learning technique, and may restore the obscured object to the reconstructed object using additional information.

The video surveillance system may be a system for protecting life and property, such as crime prevention and disaster monitoring, based on analysis of images collected from cameras such as CCTV. For example, the video surveillance system recognizes objects to be monitored, such as people and vehicles, from the collected images, analyzes their behavior patterns, and when abnormal events (eg, intrusion, fire, violence, theft, speculation, etc.) occur , it may be a system that transmits information on this to a supervisor through various warning devices, but is not limited thereto.

In FIG. 1 , if a restorer is used for an image of an occluded object, an image in which the occluded portion is restored (removed) may be obtained. In this case, more desirable restoration is possible by using the available additional information. For example, in the example of FIG. 1 , information on the contour of the face and the positions of the eyes, nose, and mouth is input as additional information, so that the hidden area can be accurately restored.

In relation to the restoration of the obscured object, the reason for the need for additional information will be described in terms of the example of FIG. 1 as follows. That is, about 60 to 90 main feature points of the face can be derived, and when the face is covered with a mask or sunglasses, the specific feature points disappear. In the case of restoring these missing feature points, the hidden part may be restored using surrounding feature points, but this case generally corresponds to a case in which the number of variables is greater than the number of equations. This problem occurs most of the time even when trying to restore an obscured object other than a face, and is also referred to as an “ill-posed problem”.

The above-described problem is very difficult in terms of solving. In addition, the application of restoration techniques to specific applications requires careful consideration. For example, if artificial neural network technology (hereinafter, referred to as “general artificial neural network technology”) is used, it is possible to use multiple faces for learning to restore an obscured face or a completely new face. However, the face generated in this way is a restoration of a “general human face”, not a restoration of a “specific human face”.

Accordingly, the present invention proposes a method of using additional information to solve this problem. That is, in order to restore the occluded object, additional information of a specific object is provided to restore the occluded area while maintaining the characteristics of the specific object.

FIG. 2 shows an example of restoring an obscured object using ideal additional information, and FIG. 3 shows an example of restoring an obscured object using realistic additional information.

Referring to FIGS. 2 and 3 , a restorer that restores an obscured object (ie, an obscured object restorer) may generate a restored object by using various additional information. For example, the additional information may be information such as a characteristic such as an outline of an object or a color or material of a surface.

2 shows a case in which the quality of additional information is very ideal, and such information can usually be acquired only when there is original data or ground truth data. On the other hand, as shown in FIG. 3 , in reality, the provided additional information may be different from the ideal one. That is, FIG. 3 shows a case where the surface material and shape of the restored object are different. Nevertheless, in terms of utilizing the restored result, the restoration result may be affected by referring to the additional information provided for restoration. Knowing the facts, the result may be more useful than the prior art (that is, when a general artificial neural network is used, generating the result without an explicit use case) when using the result in a specific application.

On the other hand, difficulties that occur when restoring an object hidden using additional information are that there are various types of objects to be restored and that the types of additional information that can be used are not definitive and may vary depending on circumstances. In order to solve this problem, in the present invention, by configuring a restorer for restoring the obscured object at the time of restoration (that is, at the restoration time), and new learning or re-learning, restoration of the input occluded object is performed. can That is, in the present invention, at the time of restoration, if there is a previously-learned restoration unit related to the corresponding additional information, the restoration device is used as it is or after re-learning based on the additional information is used. Based on the corresponding additional information, a new restorer can be learned and used.

Accordingly, the two features of the present invention are summarized as follows. (1) Additional information related to the obscured object can be used at the time of restoration of the obscured area. (2) The number of training data can be augmented by using the available additional information, and the restoration performance can be improved by re-learning the previously learned restoration unit by correcting the input data in the training data.

4 is a block diagram illustrating a restoration apparatus 100 according to an embodiment of the present invention.

The restoration apparatus 100 according to an embodiment of the present invention is an apparatus (ie, an apparatus for restoring an obscured object to a restored object) in an image captured by a camera such as an image monitoring system, and is performed by computing. This may be an electronic device or a computing network capable of this.

For example, the electronic device includes a desktop personal computer (PC), a laptop personal computer (PC), a tablet personal computer (PC), a netbook computer, a workstation, and a personal digital assistant (PDA). , a smartphone (smartphone), a smart pad (smartpad), or a mobile phone (mobile phone), etc., but is not limited thereto.

The restoration apparatus 100 may include an input unit 110 , a communication unit 120 , a display 130 , a memory 140 , and a control unit 150 , as shown in FIG. 4 .

The input unit 110 may generate input data in response to input of various users (monitor, etc.), and may include various input means. For example, the input unit 110 may include a keyboard, a keypad, a dome switch, a touch panel, a touch key, a touch pad, and a mouse. (mouse), a menu button (menu button) and the like may be included, but is not limited thereto.

The communication unit 120 is configured to communicate with other devices. For example, the communication unit 120 is 5th generation communication (5G), long term evolution-advanced (LTE-A), long term evolution (LTE), Bluetooth, bluetooth low energe (BLE), near field communication (NFC), Wireless communication such as Wi-Fi communication may be performed or wired communication such as cable communication may be performed, but is not limited thereto. For example, the communication unit 120 may receive an image, a generator, a restorer, etc. from another device, and may transmit a restoration result (image), a generator, a restorer, etc. to the other device. In this case, the case where the generator or the restorer is received from the other device may be a case in which learning of the generator or the restorer is performed in the other device in a restoration method to be described later.

The display 130 displays various image data on a screen, and may be configured as a non-emission panel or a light emitting panel. For example, the display 130 may include a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, and a micro electromechanical system (MEMS). mechanical systems) display, or an electronic paper display, etc., but is not limited thereto. In addition, the display 130 may be implemented as a touch screen or the like in combination with the input unit 110 .

The memory 140 stores various types of information necessary for the operation of the restoration apparatus 100 . The storage information may include, but is not limited to, an image, a generator, a restoration device, and program information related to a restoration method to be described later. For example, the memory 140 may be a hard disk type, a magnetic media type, a compact disc read only memory (CD-ROM), or an optical media type depending on the type. ), a Sagneto-optical media type, a multimedia card micro type, a flash memory type, a read only memory type, or a random access memory type), but is not limited thereto. In addition, the memory 140 may be a cache, a buffer, a main memory, an auxiliary memory, or a separately provided storage system according to its purpose/location, but is not limited thereto.

The controller 150 may perform various control operations of the restoration apparatus 100 . That is, the controller 150 may control the execution of the restoration method to be described later, and the rest of the configuration of the restoration apparatus 100 , that is, the input unit 110 , the communication unit 120 , the display 130 , the memory 140 , etc. You can control the action. For example, the control unit 150 may include a processor that is hardware or a process that is software that is executed in a corresponding processor, but is not limited thereto.

5 is a block diagram of the control unit 150 in the restoration apparatus 100 according to an embodiment of the present invention.

The controller 150 controls the execution of the restoration method according to an embodiment of the present invention, and as shown in FIG. 4 , the image analyzer 151 , the restorer selector 152 , the learner 153 , It may include an additional information providing unit 154 and a restoration unit 155 . For example, the image analysis unit 151 , the restorer selection unit 152 , the learning unit 153 , the additional information providing unit 154 , and the restoration unit 155 are hardware components of the control unit 150 or the control unit ( 150) may be a process that is software performed, but is not limited thereto.

Meanwhile, the restorer may be a model trained according to a machine learning technique through training data of an input data and an output data pair (data set). That is, the reconstructor includes a plurality of layers and has a function of the relationship between input data and output data. That is, when input data is input to the restorer, output data according to a corresponding function may be output.

In this case, the input data may include an image of the obscured object and additional information related to the object, and the output data may include an image of the reconstructed object. Accordingly, the reconstructor may be a model trained by machine learning to output a reconstructed object when receiving an input of an obscured object and additional information related to the object.

For example, the machine learning technique may include, but is not limited to, a deep learning technique. That is, the restorer may be a deep learning model learned according to a deep learning technique through the above-described learning data. In this case, the reconstructor expresses the relationship between the input data and the output data in a plurality of layers (ie, layers), and the plurality of expression layers is also referred to as an “artificial neural network”. Each layer in the neural network consists of at least one filter, and each filter has a matrix of weights. That is, each element (pixel) in the matrix of the corresponding filter may correspond to a weight value.

For example, deep learning techniques include Deep Neural Network (DNN), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Deep Q-Networks, Autoencoder (Auto-encoder), and may include a variational auto-encoder (Variational Auto-encoder), but is not limited thereto.

Hereinafter, the restoration method according to the present invention will be described in more detail.

6 shows a flowchart of a restoration method according to an embodiment of the present invention, and FIG. 7 shows a specific example of FIG.

The restoration method according to an embodiment of the present invention sets an area to be restored, and if it is an object that has already been learned by analyzing the object, the object is restored by selecting a corresponding occluded object restorer. However, if the object is not already learned, image data associated with the obscured object is additionally collected, new learning is performed on the collected object, and when the learning is completed, the newly learned restorer is stored (registered). In addition, with respect to the reconstructor selected from the previously learned reconstructors, if restoration is possible using the provided additional information, the object hidden by the selected reconstructor is restored, and if the provided additional information cannot be used, a reconstructor selected based on the provided additional information is used. By re-learning, more accurate restoration is possible.

That is, referring to FIG. 6 , the restoration method according to an embodiment of the present invention may include steps S201 to S212.

First, the image analyzer 151 sets a region (ie, a region to be restored) including an obscured object in the image (S201), and analyzes the type (ie, the type of the occluded object) of the occluded object in the restoration region (S201). do (S202).

In S201, the setting of the area to be restored may be processed manually or automatically. That is, when manually setting an area to be restored, the user may select an area requiring restoration (ie, an area including a hidden object) through the input unit 110 . On the other hand, when an area to be automatically restored is set, various object detection techniques may be utilized. For example, object detection technologies include Haar detector, Histogram of Oriented Gradient (HOG), Scale Invariant Feature Transform (SIFT), Local Binary Pattern (LBP), Modified Census Transform (MCT), Support Vector Machine (SVM), Adaboost, etc. may include, and a method based on deep learning technology such as YOLO may be used, but is not limited thereto.

Meanwhile, although it is indicated that the region to be restored is set in a rectangular shape in FIG. 7 , the present invention is not limited thereto and may be selected as an arbitrary polygon. However, for restoration, it is necessary to input an image that includes a region covered by a restoration device such as an artificial neural network. can be For example, since the artificial neural network preferably forms a matrix with different row and column sizes and a tensor form that is a set of these matrices, when the area to be restored is selected as a rectangle, its size is changed or the area is cut to fit the artificial neural network. etc. may be performed. In addition, when the region to be restored is selected as another polygon, a bounding box including the corresponding polygon may be covered, and a size change or region cropping may be performed.

In S202, the region set through setting the region to be restored and the related image fragment are analyzed, and the type of object that is hidden within the region is analyzed.

In S203 , the restorer selector 152 checks whether a previously learned restorer exists based on the analyzed type of object. If there is a pre-learned reconstructor, S207 is performed, and if there is no pre-learned reconstructor, S204 is performed. That is, if the analyzed obscured object corresponds to a pre-learned object, the model for restoring the object, the restorer, is an internal memory 140, an external database, or an external file system (hereinafter, “restorer storage DB”). ”) is stored (registered).

Specifically, if there is no reconstructor previously learned based on the analyzed type of object, the learning unit 153 collects an image related to the analyzed type of object in S202 (S204), and the collected object image and the object A new reconstructor is learned based on additional information related to ( S205 ) (hereinafter, the corresponding reconstructor is referred to as a “newly learned reconstructor”). In particular, in S205, the learning unit 153 stores the newly learned restorer in the restorer storage DB.

Thereafter, the restoration unit 155 may input an image corresponding to the restoration area of S201 (ie, an image including an obscured object) and additional information thereof to the newly learned restoration unit to restore the occluded object ( S206 ). . That is, a restoration object can be created. In this case, the additional information may be separately provided through the additional information providing unit 154 .

On the other hand, if there is a pre-learned reconstructor based on the analyzed type of object, the reconstructor selection unit 152 selects the pre-learned reconstructor (S207) (hereinafter referred to as a “selected reconstructor”). referred to), the additional information is provided through the additional information providing unit 154 (S208), and whether the provided additional information is additional information related to the selected restorer is checked to determine whether re-learning is required for the selected restorer (S209).

That is, when the provided additional information matches the additional information included in the input data of the learning data used during the pre-learning of the selected reconstructor (that is, when re-learning of the selected reconstructor is unnecessary), the reconstructor 155 is An image corresponding to the restoration area of S201 (ie, an image including an obscured object) and additional information thereof (ie, additional information provided in S208) may be input to the selected restoration unit to restore the occluded object (S210). That is, if the object hidden by the selected restorer can be restored in consideration of the provided additional information, the object hidden by the selected restorer is restored without any additional procedures.

On the other hand, when the provided additional information does not match the additional information included in the input data of the training data used during pre-learning of the selected reconstructor (that is, when re-learning of the selected reconstructor is required), the learning unit 153 relearns the selected reconstructor based on the additional information provided in S208 (S211) (hereinafter referred to as a “relearned reconstructor”). In particular, in S211, the learning unit 153 stores the re-learned restorer in the restorer storage DB. Thereafter, the restoration unit 155 inputs an image corresponding to the restoration area of S201 (that is, an image including an obscured object) and its additional information (additional information provided in S208) to the retrained restoration unit to restore the occluded object. can be restored

For example, in 211, the learning unit 153 may include an object image collected when the selected reconstructor is newly learned, and the additional information provided in S208 (that is, updated additional information, and the conventional additional information used when the selected reconstructor is newly learned). It is possible to retrain the selected reconstructor based on the information). As a result, in S212 , the relearned restorer may generate a restoration object that better reflects the updated additional information.

Referring to FIG. 5 (an example of a case in which re-learning of the selected reconstructor is required), in an image including a plurality of objects, after setting the area including the object type 1 including the hidden area as the area to be restored, Analyze the type of the object type 1. Thereafter, while selecting a previously learned reconstructor for object type 1, additional information about the object type 1 is provided. Thereafter, after re-learning a corresponding restorer based on the provided additional information, a restored object for object type 1 may be generated using the re-learned restorer.

Meanwhile, the additional information may be different for each hidden object, and only some information may exist in the same object. Desirable additional information may be divided into attributes, characteristics (detailed objects), and 3D information. That is, the feature may be information on a smaller detailed object included in the object, and the attribute may be information for distinguishing the object except for the feature of the detailed object. Also, the 3D information may be depth information corresponding to each pixel of the 2D image.

For example, if the hidden object is a “human face,” information such as skin color, eye color, hair color, gender, age, height, weight, etc. may correspond to attribute information. Also, main points representing a human face, such as eyes, nose, lips, ears, forehead, and jaw lines, may correspond to feature information. In addition, distance information from the camera to a specific position of a person's face may correspond to 3D information.

As another example, if the object is “car”, color, vehicle type (eg, sedan, truck, bus), manufacturer, and brand name may correspond to attribute information. Also, information such as a set of points representing an outline, a position of a headlamp, a position of a side mirror, etc. may correspond to the characteristic information. In addition, distance information from the camera to a specific location of the vehicle may correspond to 3D information.

For example, the present invention can analyze the type of obscured objects such as “human face”, “human body”, or “car”, and through this, when each object is obscured, it is restored by loading a restorer that restores it. can be performed. In the present invention, the size or shape of the object is not particularly limited. Therefore, as an example, it is of course possible to create a new restoration device that professionally restores only the pupils, lips, and cheeks in more detail with respect to a “human face”. That is, according to the present invention, if an object is to be restored, it is an object that has already been learned. It is possible to take the corresponding model and use it for restoration as it is or by re-learning, and if it is not an object that has already been learned, a new restoration model for the object may be newly trained through a predetermined method.

8 shows a conceptual diagram for learning a reconstructor for each of various objects.

Referring to FIG. 8 , a reconstructor for each object may be learned by composing training data through images related to various objects that have been previously secured or obtained through a gradual image securing method such as web crawling. For example, images of various objects may be collected and classified from images obtained by using an object detector, and a reconstructor for each of various objects may be newly learned or re-learned using the classified object images. Such learning may be performed when branching to S204 because it is not a pre-learned object in S203, or when branching to S211 because re-learning is required in S209.

Here, the difference between the terms “new learning” and “re-learning” is described as follows. In the present invention, since the reconstructor must be capable of classification (Discrimination, Classification) and data generation (Generation) through learning, a method using an artificial neural network may be preferable. Therefore, “new learning” means learning from the beginning using the given learning data in a state where no learning has been done before. On the other hand, the meaning of “re-learning” is that the artificial neural network has already been trained in advance, and a model describing the artificial neural network has been secured through this, and the artificial neural network that has been previously learned through additional data (ie, provided additional information, etc.) It means to further improve (update) the internal state.

9 is a diagram illustrating a process of restoring an obscured object in terms of input and output.

Referring to FIG. 9 , the restorer may receive an image of an obscured object (for restoration) and additional information provided from the additional information providing unit 154 to output a restored object. This is because learning is performed using learning data including input data of the object image and its additional information hidden by the restoration device, and result data of the restored image, respectively. In particular, as described above, in the present invention, a process of selecting a model suitable for restoration by checking an object of an image of an obscured object (for restoration) is added from the restoration storage DB.

In FIG. 9 , the additional information providing unit 154 provides additional information to the restorer. As shown in FIG. 9 , the additional information is extracted from an image of an obscured object (for restoration), or provided through a reference DB or a user. can be

Specifically, since all of the hidden objects are not normally completely covered, additional information can be obtained with only partial information of the hidden objects. In this case, the additional information may be extracted from the obscured object (for restoration) image. That is, the additional information may be provided through analysis of a region other than the occluded region of the occluded object.

For example, assuming that the nose and mouth are covered by a mask on a human face, the cascade boosting (a method of finding feature points gradually) or artificial neural networks are used to find the feature points, or use the skin color of the part that is not covered. Thus, the skin color of the covered area can be estimated, and the age or gender can also be estimated by referring to the uncovered skin features or hair shape.

On the other hand, when additional information is provided from the reference DB, additional information is provided through analysis of other images stored in various databases. That is, additional information may be provided through analysis of other images captured at the same location where the image of the obscured object was captured.

For example, if an obscured object occurs in an image captured by a certain camera in the same place, in the reference DB in which other images previously captured by the corresponding camera are stored, objects such as brightness, color, shape, etc. are stored from the other stored images. Additional information can be obtained.

In addition, when additional information is provided through input or selection by the user, additional information is provided based on the user's knowledge or hypothesis. It may be a case in which the user confirms the provided additional information and selectively corrects it.

10 shows an example of a specific learning process for the reconstructor.

Referring to FIG. 10 , the learning unit 153 may play a role of learning the first generator (ie, the obscured object generator) in addition to the restorer (ie, the obscured object generator). In this case, the first generator is a model trained by a machine learning technique to output an obscured object related to the additional information when an unoccluded object image and the additional information are input. That is, the first generator serves to generate an obscured object (for learning) by receiving an unoccluded object (for learning). That is, in the training data for learning the first generator, input data may include an unoccluded object image and additional information thereof, and output data may include a hidden object related to the additional information.

In FIG. 10 , the additional information providing unit 154 may utilize the additional information providing unit 154 used in FIG. 9 as it is. That is, they are interchangeable.

As an example, when the object that is not covered is a “human face,” the first generator arbitrarily covers the “human face” with objects such as “hat”, “sunglasses”, “mask”, “mustache”, “shawl”, etc. A hidden object (for learning) image can be created. In addition, the first generator generates an obscured object (for learning) image that arbitrarily obscures the corresponding “car” with “another car”, “tree”, “block” or “arbitrary polygonal shape” if the unoccluded object is “car” You may. Alternatively, the first generator may generate a random pattern such as Gaussian noise, salt and pepper noise, and uniform noise for an area to be covered.

11 shows an example of providing additional information.

An important feature of the first generator (ie, the hidden object generator) according to the present invention is that additional information can be additionally provided through the additional information providing unit 154 in generating the hidden object. 11 is an example of this, if the additional information providing unit 154 provides mask information that covers a person's face as additional information to the first generator, the first generator converts an unmasked object into a masked object. can be synthesized with Through this process, when the restorer is trained, the restorer can be retrained to effectively restore it when it is obscured by an object that has not been learned at all.

That is, the learning unit 153 may learn the reconstructor (ie, the hidden object reconstructor) using the first generator. In this case, the learning unit 1530 provides an unoccluded object (learning) image input to the first generator, an obscured object (learning) image output from the first generator, and the related additional information providing unit 154 . Using the additional information (ie, additional information provided to the first generator), the restorer may be trained, and when the learning is completed, the restorer may be stored in the restorer storage DB. This process may be performed in addition to the new learning of the restorer, particularly during re-learning.

For example, if there is no pre-learned reconstructor for the occluded object, in S204, an image related to the unoccluded object analyzed in S202 is collected, and the non-occluded object image of the collected image and its additional information are first An occluded object image of the collected image is generated by input to the generator, and the selected reconstructor may be newly trained using the occluded object of the collected image generated by the first generator and additional information thereof.

In addition, when re-learning for the selected reconstructor is required, in S211, an image related to the unoccluded object analyzed in S202 is collected, and the unoccluded object image of the collected image and its additional information are transmitted to the first generator. An occluded object image of the collected image is generated by inputting the image, and the selected reconstructor may be re-learned using the occluded object of the collected image generated by the first generator and additional information thereof.

As described above, it may be desirable to use an artificial neural network for the configuration of the restorer. The artificial neural network may preferably have a structure including a second generator and a discriminator internally.

In this case, the second generator may preferably have a structure including an auto-encoder or a variant auto-encoder, a U-Net generator (generator), or a ResNET generator (generator). In addition, the second generator may be preferably implemented by extracting the generator part learned through various modifications of the generative adversarial network. For example, Pix2Pix, CycleGAN, U-Net, ResNet, WGAN, ProgressivGAN, World Models, Self-Attention GAN, BigGAN, BERT, StyleGAN, etc. are well-known GAN variants and can be used by additionally transforming the generator part in these structures. there will be

In generative adversarial neural networks and their variants, it is common for the generator and discriminator to feed back each other's results to proceed with learning, but they are distinguished by modifying the feedback structure or optimization method. The discriminator of the generative adversarial network usually performs binary judgments such as “real” and “fake”, but it can also distinguish a number of classes, and it is interpreted as the reliability of the output result rather than a definite value such as 0 or 1 It is also possible to output possible values. The generator of the generative adversarial neural network may use an autoencoder or a variant autoencoder, but variants of the autoencoder such as a U-net generator or a ResNet generator may be used. U-net or ResNet creators can restore hidden parts through configurations such as downsampling, upsampling, and skip connection, and can perform resolution improvement, noise removal, and style conversion.

In addition to the autoencoder, the principle of restoring the area covered by the U-net generator or ResNet generator is additionally described as follows. The role of downsampling in the U-net generator or ResNet generator can be implemented by simultaneously reducing the spatial dimension and the channel, or extending the channel instead of reducing the spatial dimension. This is a method used when constructing a general classifier. The spatially reduced hidden layers can detect what an image is, but lose location information about where it is. It is a phenomenon that can be interpreted in terms of the uncertainty principle of information found in the Fourier transform. On the other hand, the role of upsampling is to restore spatial information lost during downsampling. The skip connection simply compensates for the shortcomings of sequential upsampling, and enables to effectively mix and output various levels of latent information of each hidden layer obtained in the downsampling process. To summarize, generators such as U-net generators and ResNet generators also disappear in the encoding or downsampling process, and special phenomena that are not common in a given training set, such as noise or occlusion, disappear in the encoding or downsampling process, just like the original autoencoder does. If you perform sampling, you get a result that has been masked or de-noised.

A method of learning to use the additional information in the second generator and using it for restoration will be further described as follows. The input data of the training data may include an obscured object image, and the output data may include an unoccluded object image (reconstructed image). In the case of learning in this way, encoding between the hidden object (input) and the non-occluded object (output) may be implemented in the hidden layer of the artificial neural network. That is, the hidden layer learned by reflecting the additional information (ie, information on the hidden area) can be implemented.

For example, in the input for learning, the hidden objects are R, G, and B channels, and as additional information, an additional information channel means an outline, an additional information channel means depth information, an additional information channel means a color for each area, It is assumed that additional information channels indicating a pattern of an image are additionally input individually or in combination. Of course, it is also possible not to add the additional information channel. At this time, the output data includes an unobscured object image called ground truth, and learning is performed.

As described above, various situations in which the model is actually restored from the learned state will be described as follows.

(1) When there is no additional information other than R, G, and B color information in reconstructing an obscured object, a model learned without additional information can be used. Even in this case, a model trained only with R, G, and B color information can restore sporadic noise, clustered spots, and occluded parts. Even if specific color information does not exist in the input, it can be restored to some extent. This is due to the characteristics of the autoencoder, variant autoencoder, GAN, and various variants of GAN.

(2) In restoring the obscured object, when additional information channels representing contour lines are input in addition to the R, G, and B color information, a corresponding model is used. As long as the contour information is maintained, the restoration object is restored in consideration of the contour. In a similar case, when facial feature points are used as additional information, even if the nose or mouth is covered with a mask, etc., information on the eyes and jaw lines is extracted from the hidden R, G, and B color information itself to determine the positions of the hidden nose and mouth. By estimating and using this as additional information for facial feature points, it is possible to effectively restore the face shape.

(3) In restoring the obscured object, when the additional depth information channel is input in addition to the R, G, and B color information, a corresponding model is used. Effective restoration is possible by estimating depth information on its own or using previously known information. As an example of using known information in advance, in the case of known car types, products, buildings, etc., three-dimensional shape information can be obtained through multiple paths. B, it can be added to the color channel. To simplify the discussion, if the depth map is equal to the size of the R, G, and B color image, each pixel of the image corresponds to each point of the depth map one-to-one. In this case, even if the image of car A is obscured, the model of car A can be known through the area that is not covered during restoration. It is possible to restore the part that is obscured in a direction that reflects the 3D information. In this case, an interesting feature is that even if the vehicle and the vehicle depth map are used for learning, as long as the size and specification of the image and the depth map are the same, the depth map can be used as additional information to restore a building or other product. .

(4) The additional information channel meaning color for each region and the additional information channel meaning the image pattern can be restored in a similar manner to the cases (1) to (3).

(5) It is also possible to learn by combining single additional information channels as in (1) to (4) above and use them for restoration. Of course, in this case, as the data for learning increases and the dimension increases, the time required for learning increases and the required operator resources can also increase significantly. important characteristics to be present.

The second generator may be said to be a device that internally implements the generative model. In this case, the generative model may be defined as a model that outputs data corresponding to the input of a predetermined parameter value in a specific probability distribution.

For example, a generative model having a one-dimensional Gaussian probability distribution may generate data having a bell-shaped occurrence distribution centered on an average value. In addition, if there is a generative model expressed as the sum of a plurality of two-dimensional Gaussian functions, the generated data will show a distribution having several peaks.

An autoencoder used as an important technical element in a generative adversarial neural network is a kind of generative model, and the output result may be an image depending on the learning method. Autoencoders typically have a structure in which dimensionality reduction is performed through encoding and then expanded to the original dimension size by decoding it. Therefore, it is necessary to take X as input and learn to output X. The learned autoencoder has the characteristic that it is possible to output almost similar to X even when a value of X + delta is input. Mathematically, it can be expressed that arbitrary multi-dimensional information is projected onto a multi-dimensional manifold through an autoencoder and the result is output. That is, X + delta means that noise, occlusion area, and speckle exist in the image. If you input this, you can get a clean output X from which these are removed.

These features are also important features expected in the present invention. That is, a generative model such as an autoencoder outputs an unoccluded object image (that is, a reconstructed object image) when a occluded object image is input. It can be interpreted because

12 shows an example of a learning process of a second generator and a discriminator.

On the other hand, the discriminator receives an unoccluded object image output as the occluded object image is input to the second generator, and determines how faithfully the generator restores the occluded object (that is, whether the restored object matches). carry out For example, referring to FIG. 12 , the occluded object image may be synthesized from the non-occluded object image (ie, the first image) using a first generator or the like. Thereafter, when the synthesized occluded object image is input to the second generator, the second generator outputs an unoccluded object image (ie, the second image). In this case, the discriminator compares the first image with the second image and outputs a result of determining how faithfully the second image is restored by the second generator (that is, the comparison result of how similar the second image is to the first image) do. Also, when the synthesized occluded object image and the additional information are input together into the second generator, the second generator outputs an unoccluded object image (ie, the second image) in consideration of the additional information. In this case, the discriminator compares the first image with the second image and outputs a result of determining how faithfully the second image is restored by the second generator (that is, the comparison result of how similar the second image is to the first image) do.

As described above, the second generator is a conventional autoencoder, variant autoencoder, generative adversarial neural network (GAN) or its variants Pix2Pix, CycleGAN, U-Net, ResNet, WGAN, ProgressivGAN, World Models, Self-Attention GAN, It can be configured by extracting the generator part from BigGAN, BERT, StyleGAN, etc. and transforming it. Therefore, from the viewpoint of re-learning, the learning time can be reduced by learning the generator in the conventional way and configuring the learned generator model as the initial value of the second generator.

As another re-learning method of the second generator, when the third additional information is additionally generated in the previously-learned generator and this is reflected in learning, the previously-learned model is initially used instead of randomly using the weight values of the artificial neural network. When used as a value, it is possible to reduce the resources and time required for learning.

That is, the second generator and the discriminator are adversarially (complementarily) learning based on a Generative Adversarial Network (GAN). That is, the second generator is learned to make the second image more similar to the first image by reflecting the determination result of the discriminator, and the discriminator is also learned to reflect the result of the second generator to improve the accuracy of the discrimination result. . Accordingly, the second generator improves the performance of the discriminator, and the discriminator evolves with each other in a manner that increases the performance of the second generator. Thereafter, the second generator that has evolved over a certain period may be used as the above-described newly learned or re-learned restorer.

However, in FIG. 12 , the additional information input to the first generator and the additional information input to the second generator are not the same and may be different additional information. That is, the case in which additional information is also input to the second generator may be more common for restoration. For example, an outline channel or a depth map channel corresponding to an additional information channel may be simultaneously used as an input to the second generator together with R, G, and B color images. In this case, the additional information channel may not match the additional information input to the first generator>. For example, in consideration of the form of the artificial neural network, the size, form, or data type of additional information may be changed.

The present invention configured as described above has an advantage in that a reconstructed image more suitable for the occluded region can be generated by restoring the occluded region of the image using side information at the restoration time. In addition, the present invention can augment the training data using the available additional information, and at the same time, by modifying the input data to retrain the pre-trained model, there is an advantage that can improve the restoration performance. In addition, since the present invention generates a restored image according to additional information at the time of restoration, its utility is increased, such as CCTV control site, person matching, image restoration after an incident, biometric information-based authentication in a mobile terminal, building entrance, etc. In application fields such as access control, there is an advantage that can be widely used to restore a hidden area of an object when it exists.

In the detailed description of the present invention, although specific embodiments have been described, various modifications are possible without departing from the scope of the present invention. Therefore, the scope of the present invention is not limited to the described embodiments, and should be defined by the following claims and their equivalents.

100: restoration device 110: input unit
120: communication unit 130: display unit
140: memory 150: control unit
151: image analysis unit 152: restorer selection unit
153: learning unit 154: additional information providing unit
155: restoration unit

Claims (14)

Machine learning (machine learning) to restore an object (occluded object) having an obscured area in an image to an object in which the obscured area is restored (restored object), but to output a reconstructed object when the obscured object and additional information related to the object are input ) method, which is a method performed by an electronic device that restores using a restorer that is a model learned by the method,
analyzing a type of an object that is obscured from the image;
checking whether there is a previously learned reconstructor based on the analyzed type of object;
selecting a corresponding restorer if a restorer exists in the checking step; and
It is determined whether re-learning of the selected reconstructor is necessary according to additional information related to the analyzed type of object, and uses the selected reconstructor according to the determined necessity or relearns the selected re-learner based on the additional information Including; creating a restoration object by using the restored restorer;
The relearned restorer is a relearned method using a first generator learned by machine learning techniques to output an unobscured object and a hidden object related to the additional information when additional information is input.
According to claim 1,
If there is no restorer in the selecting step, collecting the analyzed type of object, and generating a restored object using the newly learned restorer based on the collected object and additional information related to the object How to.
According to claim 1,
Further comprising the step of setting a region including the object covered in the image,
The analyzing is a method of performing the analysis on the occluded object in the restoration area.
delete According to claim 1,
The generating step is
generating a restored object by inputting an object hidden in the selected restorer and additional information thereof when re-learning of the selected restorer is unnecessary; and
When re-learning of the selected reconstructor is required, an image related to the analyzed type of unobscured object is collected, and the unoccluded object of the collected image and its additional information are input to the first generator, and After generating an obscured object, re-learning the selected reconstructor using the obscured object of the collected image generated by the first generator and its additional information, the occluded object and its additional information are input into the re-learned reconstructor to restore the object creating a;
How to include.
According to claim 1,
The restorer includes a second generator for generating a restored object from the obscured object, and a discriminator for determining the restored object generated by the second generator, respectively, wherein the second generator and the discriminator are adversarially learned based on a GAN (Generative Adversarial Network). method.
7. The method of claim 6,
The second generator is an auto-encoder, a variant auto-encoder, a U-Net generator, a ResNET generator, or a method implemented based on a generative adversarial network.
According to claim 1,
The additional information is provided through analysis of a region other than the occluded region of the occluded object.
According to claim 1,
The additional information is provided through analysis of other images captured at the same location where the image was captured.
According to claim 1,
The additional information is provided by being input or selected by a user.
According to claim 1,
The additional information includes attribute information on the analyzed type of object, information on a plurality of detailed objects included in the analyzed type of object, or 3D information on the analyzed type of object.
A device for restoring an object (occluded object) having an obscured area in an image to an object (restored object) in which the obscured area is restored,
a memory for storing a reconstructor, which is a model learned by a machine learning technique, to output a reconstructed object when an obscured object and additional information related to the object are input; and
Includes; a control unit for controlling to create a restoration object using the restoration stored in the memory;
The control unit is
Analyzes the type of object obscured from the image, and confirms the existence of a previously learned reconstructor based on the analyzed type of object,
If there is a restorer, select the corresponding restorer, determine whether re-learning is necessary for the selected restorer according to additional information related to the analyzed type of object, and use the selected restorer according to the determined necessity, or Controlling to create a restoration object by using the restoration that re-learned the selected restoration unit based on the additional information,
The relearned restorer is a relearned device using a first generator learned by machine learning techniques to output an unobscured object and a hidden object related to the additional information when additional information is input.
A device for restoring an object (occluded object) having an obscured area in an image to an object (restored object) in which the obscured area is restored,
a communication unit for receiving a reconstructor, which is a model learned by a machine learning technique, to output a reconstructed object when receiving an input of an obscured object and additional information related to the object; and
It includes; a control unit for controlling to create a restored object using the restorer received from the communication unit;
The control unit is
Analyzes the type of object obscured from the image, and confirms the existence of a previously learned reconstructor based on the analyzed type of object,
If there is a restorer, select the corresponding restorer, determine whether re-learning is necessary for the selected restorer according to additional information related to the analyzed type of object, and use the selected restorer according to the determined necessity, or Controlling to create a restoration object by using the restoration that re-learned the selected restoration unit based on the additional information,
The relearned restorer is a relearned device using a first generator learned by machine learning techniques to output an unobscured object and a hidden object related to the additional information when additional information is input.
delete

Images