WO2022080681A1 - Image inpainting method and device considering peripheral region - Google Patents
Image inpainting method and device considering peripheral region Download PDFInfo
- Publication number
- WO2022080681A1 WO2022080681A1 PCT/KR2021/012505 KR2021012505W WO2022080681A1 WO 2022080681 A1 WO2022080681 A1 WO 2022080681A1 KR 2021012505 W KR2021012505 W KR 2021012505W WO 2022080681 A1 WO2022080681 A1 WO 2022080681A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- processor
- pixels
- information
- neural network
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 40
- 230000002093 peripheral effect Effects 0.000 title description 4
- 238000013528 artificial neural network Methods 0.000 claims abstract description 72
- 238000011017 operating method Methods 0.000 claims description 2
- 238000012545 processing Methods 0.000 description 16
- 230000008569 process Effects 0.000 description 11
- 238000010586 diagram Methods 0.000 description 10
- 238000013473 artificial intelligence Methods 0.000 description 6
- 238000012937 correction Methods 0.000 description 6
- 238000007781 pre-processing Methods 0.000 description 6
- 238000003702 image correction Methods 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 3
- 238000001914 filtration Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 239000003086 colorant Substances 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000010422 painting Methods 0.000 description 2
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 229910044991 metal oxide Inorganic materials 0.000 description 1
- 150000004706 metal oxides Chemical class 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000010008 shearing Methods 0.000 description 1
- 230000002194 synthesizing effect Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/77—Retouching; Inpainting; Scratch removal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/60—Image enhancement or restoration using machine learning, e.g. neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Definitions
- the present invention relates to an image inpainting method and device in consideration of a surrounding area, and more particularly, to an image inpainting method using an artificial neural network.
- Image inpainting technology is a technology that started with the restoration of damaged works of art, etc.
- image inpainting technology has various uses, such as restoring damaged parts of images, such as paintings, photos, and videos, or removing parts from images. is used as
- the image inpainting technique corrects pixels of the background image region adjacent to the boundary of the region to be restored (pixels in the peripheral region) of the image, and uses the corrected pixels to restore the region to be restored technology to fill
- the background image area may mean an area other than the area to be restored from the original image.
- the image inpainting technique is performed by a professional image editor, and even for a professional image editor, the editing operation for image inpainting may be a task that requires a high degree of concentration. Therefore, there is a problem in that it is difficult to use image inpainting technology for ordinary people who need image inpainting of images taken in their daily life.
- An object of the present invention to solve the above problems is to provide a learning method of an artificial neural network for image inpainting by updating an artificial neural network and a weight vector of a device performing image inpainting.
- Another object of the present invention to solve the above problems is to provide a method of restoring an original image using an object in a peripheral area of the removed image.
- an operating method for learning an image inpainting model includes the steps of adding a separate object to an original image to generate a first image; Using an artificial neural network to classify objects included in the first image based on information on each pixel of the first image, removing the separate object from the first image, and the first image
- the step of setting a boundary between a second object and a third object in the first area corresponding to the removed separate object using a second artificial neural network for image restoration, the first area is divided into the second Restoring based on pixel information of an object and the third object, generating a restored image, calculating a difference value between pixels of the original image and the restored image, and the original image and the restored image
- the method may include updating a weight vector of the second artificial neural network for image reconstruction based on the difference value of the pixels.
- the step of setting a boundary between the second object and the third object in the first area comprises:
- It may be set based on at least one of pixel information of each of the second object and the third object, and attribute information of each of the second object and the third object.
- the method may include overlapping the pattern of the second object in at least a partial area.
- the method may include obtaining information of one pixel group and overlapping the first pixel group in the first area of the first image.
- the step of compensating the first pixel group may include, and the overlapping of the pixels may include overlapping the corrected pixels in the first area.
- the second image tracking the first object located in a second area, obtaining information on a second pixel group that is a set of pixels corresponding to the second area in the first image and overlapping the second pixel group on the second area of the second image.
- the method further includes updating a weight vector of a first artificial neural network for classifying the object based on a difference value between pixels of the original image and the reconstructed image, wherein the weight vector of the first artificial neural network includes:
- the information may be updated based on a difference value between objects divided based on information on preset objects of the first image and information on each pixel of the first image.
- the weight vector of the second artificial neural network is a difference between information on pixels in a region corresponding to the preset first object of the first image and information on pixels in a region corresponding to the restored first object. It can be updated based on the value.
- an image inpainting service can be more easily provided to a user by performing an operation of restoring an original image using an object in a peripheral area of the removed image.
- FIG. 1 is a block diagram showing a first embodiment of the structure of a terminal.
- FIG. 2 is a flowchart illustrating an embodiment of an operation of a device for image inpainting.
- FIG. 3 is a flowchart illustrating a first embodiment of an operation of a processor for image restoration among operations for image inpainting.
- FIG. 4 is a flowchart illustrating a second embodiment of an operation of a processor for image restoration among operations for image inpainting.
- FIG. 5 is a conceptual diagram illustrating an embodiment of an image correction process according to each operation process for image inpainting.
- FIG. 6 is a flowchart illustrating an embodiment of an operation of a device for inpainting an image of a moving picture.
- FIG. 7 is a conceptual diagram illustrating an embodiment of an image correction process according to each operation process for image inpainting of a moving picture.
- FIG. 8 is a flowchart illustrating an embodiment of an artificial neural network learning operation of a device for image inpainting.
- FIG. 9 is a conceptual diagram illustrating an embodiment of a process of performing an artificial neural network learning operation of a device for image inpainting.
- 10 is a flowchart illustrating an embodiment of an operation based on an artificial neural network among operations for image inpainting.
- 11 is a conceptual diagram illustrating an embodiment of an object tracking operation among operations for image inpainting.
- first, second, etc. may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. and/or includes a combination of a plurality of related listed items or any of a plurality of related listed items.
- FIG. 1 is a block diagram showing a first embodiment of the structure of a terminal.
- the terminal may include a camera 110 , a processor 120 , a display 130 , and a memory 140 .
- the camera 110 may include an image sensor 111 , a buffer 112 , a preprocessing module 113 , a resizer 114 , and a controller 115 .
- the camera 110 may acquire an image of the external area.
- the camera 110 may store the raw data generated by the image sensor 111 in the buffer 112 of the camera.
- the raw data may be processed by the controller 115 or the processor 120 in the camera 110 .
- the processed data may be transmitted to the display 140 or the encoder 123 .
- the raw data may be processed and then stored in the buffer 112 , and may be transferred from the buffer 112 to the display 140 or encoder 174 .
- the image acquired by the camera 110 may be a portion of an equirectangular (ERP) image, a panoramic image, a circular fisheye image, a spherical image, or a three-dimensional image.
- ERP equirectangular
- the image sensor 111 may collect raw data by detecting light incident from the outside.
- the image sensor 111 may include, for example, at least one of a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS) image sensor, and an infrared (IR) optical sensor 150 .
- CMOS complementary metal oxide semiconductor
- IR infrared
- the image sensor 111 may be controlled by the controller 115 .
- the pre-processing module 113 may convert the raw data obtained by the image sensor 111 into a color space form.
- the color space may be one of a YUV color space, a red green blue (RGB) color space, and a red green blue alpha (RGBA) color space.
- the preprocessing module 113 may transmit the data converted into the color space form to the buffer 112 or the processor 120 .
- the pre-processing module 113 may correct an error or distortion of an image included in the received raw data. In addition, the pre-processing module 113 may adjust the color or size of the image included in the raw data.
- the preprocessing module 113 is, for example, bad pixel correction (BPC), lens shading (LS), demosaicing, white balance (WB), gamma correction, color space conversion (CSC) , HSC (hue, saturation, contrast) improvement, size conversion, filtering, and at least one of image analysis may be performed.
- the processor 120 of the terminal may include a management module 121 , an image processing module 122 , and an encoder 123 .
- the management module 121 , the image processing module 122 , and the encoder 123 may be hardware modules included in the processor 120 , or may be software modules that may be executed by the processor 290 . Referring to FIG. 1 , the management module 121 , the image processing module 122 , and the encoder 123 are illustrated as being included in the processor 120 , but may not be limited thereto. Some of the management module 121 , the image processing module 122 , and the encoder 123 may be implemented as a module separate from the processor 120 .
- the management module 121 may control the camera 110 included in the terminal.
- the management module 121 may control initialization of the camera 110 , a power input mode of the camera 110 , and an operation of the camera 110 .
- the management module 121 may control an image processing operation of the buffer 112 included in the camera 110 , captured image processing, image size, and the like.
- the management module 121 controls the first electronic device 100 so that the first electronic device 100 adjusts auto focus, auto exposure, resolution, bit rate, frame rate, camera power mode, VBI, zoom, gamma, or white balance. ) can be controlled.
- the management module 121 may transmit the acquired image to the image processing module 122 and control the image processing module 122 to perform processing.
- the management module 121 may transmit the acquired image to the encoder 123 .
- the management module 121 may control the encoder 123 to encode the obtained image.
- the image processing module 122 may acquire an image from the management module 121 .
- the image processing module 122 may perform a processing operation on the acquired image. Specifically, the image processing module 122 may perform noise reduction, filtering, image synthesize, color correction, color conversion, image transformation, and/or filtering of the acquired image. 3D modeling, image drawing, augmented reality (AR)/virtual reality (VR) processing, dynamic range adjusting, perspective adjustment, shearing, resizing, edge extraction, region of interest (ROI) Judgment, image matching, and/or image segmentation may be performed.
- the image processing module 122 may perform processing such as synthesizing a plurality of images, generating a stereoscopic image, or generating a depth-based panoramic image.
- FIG. 2 is a flowchart illustrating an embodiment of an operation of a device for image inpainting.
- the processor of the device may perform operations for image inpainting.
- the processor may include an artificial neural network (ANN), and may perform operations for image inpainting based on artificial intelligence using the artificial neural network.
- ANN artificial neural network
- the processor may classify objects included in the first image.
- the processor may classify objects included in the first image by using an artificial neural network. For example, the processor may classify a plurality of objects included in the first image based on information on pixels included in the first image.
- the processor may reflect color information of pixels, etc. in classifying objects included in the first image. When the difference between the color information values of different adjacent pixels is less than or equal to a preset value, the processor determines that different adjacent pixels are pixels obtained from images of the same and/or similar object in classifying objects included in the first image can do.
- the processor may determine a first object that is a correction target object (or a separate object) from among the objects of the first image.
- the processor may analyze each of the objects based on color information and shape information of the objects included in the first image, and may determine the first object to be corrected based on the analysis result of each of the objects.
- the processor may determine a first object to be corrected from among the objects of the first image based on input information from the user.
- the processor may remove the first object determined as the object to be corrected from the first image.
- the processor may set a boundary between the second object and the third object in a region corresponding to the first object removed from the first image.
- the processor may set a boundary between the second object and the third object in a region corresponding to the first object removed from the first image by using the artificial neural network.
- the processor may set a boundary between the second object and the third object in the area corresponding to the first object based on pixel information of each of the second object and the third object.
- the processor may set a boundary between the second object and the third object in the area corresponding to the first object based on at least one piece of attribute information of each of the second object and the third object. That is, the processor may divide the region corresponding to the first object of the first image into an (a) region associated with the second object and a (b) region associated with the third object, (a) region and (b) You can set boundaries between regions.
- the processor may restore an area corresponding to the first object based on the boundary based on pixel information of the second object and the third object.
- the processor may reconstruct a region corresponding to the first object based on pixel information of the second object and the third object by using the artificial neural network. That is, the processor may reconstruct region (a) of the first image based on pixel information of the second object, and may reconstruct region (b) of the first image based on pixel information of the third object. Accordingly, as a result of image inpainting by the processor, the processor may acquire a restored first image.
- FIG. 3 is a flowchart illustrating a first embodiment of an operation of a processor for image restoration among operations for image inpainting.
- the processor of the device may perform operations for image inpainting.
- the processor may include an artificial neural network, and may perform operations for image inpainting based on artificial intelligence using the artificial neural network.
- the processor may be in a state in which the operations of steps S210 to S240 of FIG. 2 are performed.
- the processor may analyze the pattern information of the second object.
- the processor may analyze the pattern information of the second object by using the artificial neural network.
- the processor may analyze the second object based on color information of pixels included in the second object, shape information of the second object, and the like, and may set an additional boundary line inside the second object based on the analysis result.
- the processor may acquire pattern information inside the second object based on the information on the additional boundary line inside the set second object.
- the processor may analyze attribute information of the second object.
- the processor may analyze attribute information of the second object by using the artificial neural network.
- the processor may analyze the second object based on color information and shape information of the second object, and may determine attribute information of the second object based on the analysis result.
- the processor may label the attribute information on the second object.
- the processor may additionally label the attribute information in detailed areas of the second object.
- the processor may overlap the pattern of the second object on an area corresponding to the first object of the first image.
- the processor may overlap the pattern of the second object in an area corresponding to the first object by using the artificial neural network. Accordingly, as a result of image inpainting by the processor, a restored first image may be obtained.
- FIG. 4 is a flowchart illustrating a second embodiment of an operation of a processor for image restoration among operations for image inpainting.
- the processor of the device may perform operations for image inpainting.
- the processor may include an artificial neural network, and may perform operations for image inpainting based on artificial intelligence using the artificial neural network.
- the processor may be in a state in which the operations of steps S210 to S240 of FIG. 2 are performed.
- the processor may acquire a second image.
- the processor may acquire a second image similar to the first image, which is an image inpainting target, by using the artificial neural network.
- the processor may analyze the separate images based on the information on the pixels of the first image to obtain a second image having a similarity greater than or equal to a preset value among the separate images.
- the processor may acquire a second image similar to the first image from among the separate images stored in the memory of the device.
- the processor may acquire a second image similar to the first image from among separate images stored in an external database.
- the processor may acquire a first pixel group corresponding to an area corresponding to the first object in the second image.
- the processor may obtain a first pixel group corresponding to a region corresponding to the first object of the first image in the second image by using the artificial neural network.
- the processor may calculate location information of a region corresponding to the first object in the first image, and apply the calculated location information to the second image to generate a first pixel group from the second image. can be obtained
- the processor may calculate information on pixels adjacent to an area corresponding to the first object in the first image, and obtain a first pixel group from the second image based on the calculated information on the pixels.
- the processor may correct the pixels of the first pixel group.
- the processor may optimize the pixels of the first pixel group to the first image by correcting the pixels of the first pixel group using the artificial neural network.
- the processor may compare color information of pixels of the first group with color information of pixels adjacent to the first object of the first image.
- the processor is configured to: Colors of pixels of the first group may be corrected based on .
- the processor may compare the shape information of the first group with the shape information of the first object of the first image. When the shape information of the first group does not match the shape of the first object of the first image, the processor may correct the shape of the first group based on the shape of the first object of the first image.
- the processor may overlap the pixels of the first pixel group in an area corresponding to the first object.
- the processor may use an artificial neural network to correct the first image by overlapping the pixels of the reconstructed first pixel group on the first image obtained from the second image. Accordingly, as a result of image inpainting by the processor, a restored first image may be obtained.
- FIG. 5 is a conceptual diagram illustrating an embodiment of an image correction process according to each operation process for image inpainting.
- the processor of the device may perform operations for image inpainting.
- the processor applies an image inpainting technique to the first image of FIG. 5( a ), as shown in FIG. 5( c ).
- a restored first image may be acquired.
- Specific operations in which the processor acquires a reconstructed image by applying the image inpainting technique may be as follows.
- the processor may distinguish objects 510 , 520 , and 530 included in the first image.
- the processor may reflect color information of pixels, and the like.
- the processor may determine a first object 510 that is a correction target object from among the objects of the first image.
- the processor may remove the first object 510 determined as the object to be corrected from the first image.
- the image from which the first object is removed from the first image may be as shown in FIG. 5(b) .
- the processor may set a boundary between the second object and the third object in a region corresponding to the first object removed from the first image. Accordingly, an area corresponding to the removed first object may be divided by a boundary to correspond to the object. Specifically, the processor may set a boundary between the second object 520 and the third object 530 in an area corresponding to the first object based on pixel information of each of the second object and the third object.
- the processor may reconstruct an area corresponding to the first object 510 based on the pixel information of the second object and the third object based on the boundary.
- the processor may reconstruct a partial region 512 from among the regions corresponding to the first object 510 based on the pixel information of the second object, and may reconstruct the region 512 corresponding to the first object 510 based on the pixel information of the third object.
- the remaining area 513 may be restored among the areas to be used.
- the processor obtains the first pixel group for the partial region from the second image, which is an image separate from the first image
- the processor places the pixels of the first pixel group into the partial region 512 of the region corresponding to the first object. can overlap.
- the processor may overlap the pattern of the third object in a partial region 513 of the regions corresponding to the first object.
- the restored first image may be as shown in FIG. 5( c ).
- FIG. 6 is a flowchart illustrating an embodiment of an operation of a device for inpainting an image of a moving picture.
- the processor of the device may perform operations for image inpainting of a moving picture including a first image and a second image.
- the processor may include an artificial neural network, and may perform operations for image inpainting of a moving image based on artificial intelligence using the artificial neural network.
- the processor may be in a state in which the operations of steps S210 to S240 of FIG. 2 are performed.
- the processor may track the first object in the second image.
- the processor may track the first object in the second image by using the artificial neural network.
- the processor may determine, among the objects of the second image, an object having a degree of similarity to the first object in the first image equal to or greater than a preset value as the first object. That is, the processor may track an object having a similar color and shape in the second image by using the color and shape information of the first object.
- the processor may acquire, from the first image, a second pixel group corresponding to an area corresponding to the first object included in the second image.
- the processor may acquire a second pixel group corresponding to an area corresponding to the first object included in the second image by using the artificial neural network.
- the processor may calculate location information of a region corresponding to the first object in the second image, and apply the calculated location information to the second image to generate a first pixel group from the second image. can be obtained
- the processor may calculate information on pixels adjacent to an area corresponding to the first object in the second image, and obtain a second pixel group from the first image based on the calculated information on the pixels.
- the processor may correct the pixels of the second pixel group.
- the processor may optimize the pixels of the second pixel group to the second image by correcting the pixels of the second pixel group using the artificial neural network.
- the processor may compare color information of pixels of the second group with color information of pixels adjacent to the first object of the second image.
- the processor receives the color information of pixels adjacent to the first object of the second image Colors of pixels of the second group may be corrected as a reference.
- the processor may compare the shape information of the second group with the shape information of the first object of the second image. When the shape information of the second group does not match the shape of the first object of the second image, the processor may correct the shape of the second group based on the shape of the first object of the second image.
- the processor may overlap the pixels of the second pixel group in an area corresponding to the first object.
- the processor may correct the second image by overlapping the pixels of the reconstructed second pixel group on the second image obtained from the first image using the artificial neural network. Accordingly, as a result of image inpainting by the processor, a moving picture including the restored first image and the restored second image may be obtained.
- FIG. 7 is a conceptual diagram illustrating an embodiment of an image correction process according to each operation process for image inpainting of a moving picture.
- the processor of the device applies an image inpainting technique to a moving picture including the first image 710 and the second image 720 , and includes the restored first image 730 and the restored second image 740 .
- Operations for generating a video may be performed.
- An operation for generating a moving picture including the restored images may be as follows.
- the processor may distinguish objects included in the first image 710 .
- the processor may determine a first object 711 that is a correction target object from among the objects of the first image 710 .
- the processor may track the first object 721 in the second image 720 . Specifically, the processor may determine, among the objects of the second image 720 , as the first object 721 , an object having a degree of similarity with the first object 711 in the first image 710 equal to or greater than a preset value. . The processor may identify the objects included in the second image 720 while tracking the first object 721 in the second image 720 .
- the processor may obtain, from the second image 720 , a first pixel group 722 corresponding to an area corresponding to the first object 711 included in the first image 710 .
- the processor may obtain, from the first image 710 , a second pixel group 712 corresponding to an area corresponding to the first object 721 included in the second image 720 .
- the processor may further correct pixels of the first pixel group and the second pixel group. For example, the processor based on the comparison result of 720 of the first image 710 and the second image, based on the degree of change of each of the objects of the first image 710 and the second image 720 . Pixels of the first pixel group and the second pixel group may be further corrected.
- the processor may overlap the pixels of the first pixel group 722 in an area corresponding to the first object 711 in the first image 710 .
- the processor may overlap the pixels of the first pixel group 722 in an area corresponding to the first object 711 in the first image 710 .
- the processor may generate a moving picture including the restored first image 730 and the restored second image 740 .
- FIG. 8 is a flowchart illustrating an embodiment of an artificial neural network learning operation of a device for image inpainting.
- a processor performing operations for image inpainting of a moving picture may include an artificial neural network and may perform operations for learning of the artificial neural network.
- the processor may be in a state in which the operations of steps S210 to S240 of FIG. 2 are performed.
- the processor may insert a first object into an existing first image and perform an operation of reconstructing the first image into which the first object is inserted at least once or more.
- the processor may perform an operation for learning the artificial neural network based on a comparison result between the existing first image and the restored first image obtained as a result of the restoration operation, and specific operations may be as follows.
- the processor may compare the first image with the first image restored as a result of image inpainting.
- the processor may compare information on objects included in the first image with information on objects included in the restored first image.
- the processor may compare information on each of the pixels of the first image with information on each of the pixels of the reconstructed first image.
- the information of the pixels compared by the processor may include color information of the pixel and the like.
- the processor may calculate a difference value between the first image and the object classification result included in the restored first image.
- the processor may calculate a difference value between the objects included in the first image and the objects included in the restored first image, and the calculated difference value may be a value such as a difference between ranges of the objects.
- the processor may calculate a difference value between the first image and the first region restoration result of the restored first image.
- the processor may calculate a difference value between the restoration result of the region corresponding to the first object and the first image.
- the calculated difference value may be a difference value between information on pixels included in the region corresponding to the first object and information on a boundary line for distinguishing other objects in the region corresponding to the first object.
- the processor may update the weight vector of the artificial neural network based on the difference value between the first image and the reconstructed first image.
- the processor may update the weight vector of the artificial neural network for object classification based on the difference value between the first image and the corrected first image.
- the processor may update a weight vector for image restoration based on a difference value between the first image and the corrected first image.
- the processor may update the weight vector of the artificial neural network by applying the backpropagation method. Specifically, the processor may backpropagate a difference value between the objects included in the first image and the objects included in the restored first image to the artificial neural network to update the weight vector of the artificial neural network for object classification. In addition, the processor may backpropagate a difference value between the first image and the first region restoration result of the restored first image to the artificial neural network, and update the weight vector of the artificial neural network for image restoration.
- FIG. 9 is a conceptual diagram illustrating an embodiment of a process of performing an artificial neural network learning operation of a device for image inpainting.
- a processor performing operations for image inpainting may include an artificial neural network network, and may perform operations for learning the artificial neural network.
- the processor inserts the first object 921 into the existing first image 910 to perform operations for learning the artificial neural network, and inserts the image into the first image 920 into which the first object 910 is inserted.
- the operation of applying the inpainting technique may be performed at least once.
- the processor may perform an operation for learning the artificial neural network based on a comparison result between the existing first image 910 and the restored first image 930 obtained as a result of the restoration operation.
- the first image may be an image previously stored in a database or memory.
- the processor may compare the first image 910 with the reconstructed first image 930 .
- the processor may compare information on objects included in the first image 910 with information on objects included in the restored first image 930 .
- the processor may compare information on each of the pixels of the first image 910 with information on each of the pixels of the reconstructed first image 930 .
- the processor may calculate a difference value of a result of classification between the first image 910 and objects included in the restored first image 930 .
- the processor may calculate a difference value between the first region restoration result of the first image 910 and the restored first image 930 .
- the processor may update the weight vector of the artificial neural network based on a difference value between the first image 910 and the reconstructed first image 930 .
- the processor backpropagates the difference value between the objects included in the first image 910 and the objects included in the restored first image 930 to the artificial neural network to update the weight vector of the artificial neural network for object classification.
- the processor back-propagates the difference value between the first image 910 and the restoration result of the first region 921 of the restored first image 930 to the artificial neural network to update the weight vector of the artificial neural network for image restoration.
- 10 is a flowchart illustrating an embodiment of an operation based on an artificial neural network among operations for painting an image.
- first artificial neural network processes in the first artificial neural network and the second artificial neural network are respectively shown.
- objects included in the first image may be classified based on information on each pixel of the first image.
- a second artificial neural network may be used to restore a corresponding area (the first area).
- the first region may be restored based on pixel information of the second object and the third object.
- the second object and the third object may correspond to surrounding areas based on the first area.
- 11 is a conceptual diagram illustrating an embodiment of an object tracking operation among operations for image inpainting.
- two-way tracking and one-way tracking are possible. For example, as shown in the upper figure of FIG. 11 , tracking from the left and tracking from the right may be indicated. In addition, in the case of one-way tracking, it will be performed simultaneously with the segmentation task, thereby improving the learning efficiency of the deep learning network.
- the boundary between the second object and the third object used to restore the first area may be divided according to the area occupied by the second object and the third object in the entire image. This is for simple convenience and may be used when restoring an area having a size smaller than a predetermined size.
- the image area occupied by the second object in the entire image is larger than that of the third object.
- the image area corresponding to the second object in the restored first area may be larger than the image area corresponding to the third object.
- the ratio of the area occupied by the second object to the area occupied by the third object in the entire image and the ratio of the image area corresponding to the second object to the image area corresponding to the third object in the restored first area may be the same.
- the methods according to the present invention may be implemented in the form of program instructions that can be executed by various computer means and recorded in a computer-readable medium.
- the computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination.
- the program instructions recorded on the computer-readable medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software.
- Examples of computer-readable media include hardware devices specially configured to store and carry out program instructions, such as ROM, RAM, flash memory, and the like.
- Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.
- the hardware device described above may be configured to operate as at least one software module to perform the operations of the present invention, and vice versa.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Image Analysis (AREA)
- Computer Vision & Pattern Recognition (AREA)
Abstract
Description
Claims (7)
- 이미지 인페인팅 모델의 학습을 위한 동작 방법에 있어, In an operation method for learning an image inpainting model,원본 이미지에 별도의 객체를 추가하여, 제1 이미지를 생성하는 단계;generating a first image by adding a separate object to the original image;객체 구분을 위한 제1 인공 신경망을 이용하여, 상기 제1 이미지에 포함된 객체들을 상기 제1 이미지의 픽셀들 각각의 정보에 기초하여 구분하는 단계;classifying objects included in the first image based on information on each pixel of the first image using a first artificial neural network for object classification;상기 제1 이미지에서 상기 별도의 객체를 제거하는 단계;removing the separate object from the first image;상기 제1 이미지에서, 상기 제거된 별도의 객체에 상응하는 제1 영역에서의 제2 객체와 제3 객체 간의 경계를 설정하는 단계;setting a boundary between a second object and a third object in a first area corresponding to the removed separate object in the first image;이미지 복원을 위한 제2 인공 신경망을 이용하여, 상기 제1 영역을 상기 제2 객체 및 상기 제3 객체의 픽셀 정보를 기초로 복원하고, 복원된 이미지를 생성하는 단계; restoring the first region based on pixel information of the second object and the third object using a second artificial neural network for image restoration, and generating a restored image;상기 원본 이미지 및 상기 복원된 이미지의 픽셀들의 차이값을 산출하는 단계; 및 calculating a difference value between pixels of the original image and the reconstructed image; and상기 원본 이미지 및 상기 복원된 이미지의 픽셀들의 차이값을 기초로 상기 이미지 복원을 위한 상기 제2 인공 신경망의 가중치 벡터를 업데이트하는 단계를 포함하고,updating a weight vector of the second artificial neural network for image restoration based on a difference value between pixels of the original image and the restored image,상기 제2 인공 신경망의 가중치 벡터는, 상기 제1 이미지의 미리 설정된 상기 제1 객체에 상응하는 영역에서의 픽셀들의 정보 및 복원된 상기 제1 객체에 상응하는 영역에서의 픽셀들의 정보 간의 차이값을 기초로 업데이트되는, 이미지 인페인팅 모델의 학습을 위한 동작 방법.The weight vector of the second artificial neural network is a difference value between information on pixels in a region corresponding to the preset first object of the first image and information on pixels in a region corresponding to the restored first object. An operational method for training of an image inpainting model, updated on a basis.
- 청구항 1에 있어, The method according to claim 1,상기 제1 영역에서의 제2 객체와 제3 객체 간의 경계를 설정하는 단계는, The step of setting a boundary between the second object and the third object in the first area comprises:상기 제2 객체 및 상기 제3 객체 각각의 픽셀 정보 및 상기 제2 객체 및 상기 제3 객체 각각의 속성 정보 중 적어도 하나의 정보를 기초로 설정하는, 이미지 인페인팅을 위한 동작 방법.An operating method for image inpainting, setting based on at least one of pixel information of each of the second object and the third object, and attribute information of each of the second and third objects.
- 청구항 1에 있어, The method according to claim 1,상기 복원된 이미지를 생성하는 단계 이후, After generating the restored image,상기 제2 객체의 적어도 일부의 영역의 픽셀 정보를 기초로 상기 제2 객체의 패턴 정보를 분석하는 단계; 및 analyzing pattern information of the second object based on pixel information of at least a portion of the second object; and상기 경계에 의해 구분되는 상기 제1 영역 중 적어도 일부의 영역에 상기 제2 객체의 패턴을 오버랩하는 단계를 포함하는, 이미지 인페인팅을 위한 동작 방법.and overlapping the pattern of the second object on at least a partial area of the first area divided by the boundary.
- 청구항 1에 있어, The method according to claim 1,상기 복원된 이미지를 생성하는 단계 이후, After generating the restored image,상기 제1 이미지를 포함하는 동영상에서, 별도의 제2 이미지를 획득하는 단계; obtaining a separate second image from the moving picture including the first image;상기 제2 이미지에서, 상기 제1 영역에 대응되는 픽셀들의 집합인 제1 픽셀 그룹의 정보를 획득하는 단계; 및obtaining information on a first pixel group that is a set of pixels corresponding to the first area from the second image; and상기 제1 이미지의 상기 제1 영역에 상기 제1 픽셀 그룹을 오버랩하는 단계를 포함하는, 이미지 인페인팅을 위한 동작 방법.and overlapping the first group of pixels in the first area of the first image.
- 청구항 4에 있어, The method according to claim 4,상기 제1 픽셀 그룹을 보정하는 단계를 포함하고, calibrating the first group of pixels;상기 픽셀들을 오버랩하는 단계는, The overlapping of the pixels comprises:상기 보정된 픽셀들을 상기 제1 영역에 오버랩하는, 이미지 인페인팅을 위한 동작 방법.and overlapping the corrected pixels in the first area.
- 청구항 4에 있어, The method according to claim 4,상기 제2 이미지에서, 제2 영역에 위치하는 상기 제1 객체를 트래킹하는 단계; tracking the first object located in a second area in the second image;상기 제1 이미지에서, 상기 제2 영역에 대응되는 픽셀들의 집합인 제2 픽셀 그룹의 정보를 획득하는 단계; 및 obtaining information on a second pixel group that is a set of pixels corresponding to the second area in the first image; and상기 제2 이미지의 상기 제2 영역에 상기 제2 픽셀 그룹을 오버랩하는 단계를 포함하는, 이미지 인페인팅을 위한 동작 방법.and overlapping the second group of pixels in the second region of the second image.
- 청구항 1에 있어, The method according to claim 1,상기 원본 이미지 및 상기 복원된 이미지의 픽셀들의 차이값을 기초로 상기 객체를 구분하기 위한 제1 인공 신경망의 가중치 벡터를 업데이트하는 단계를 더 포함하고,The method further comprising: updating a weight vector of a first artificial neural network for classifying the object based on a difference value between pixels of the original image and the reconstructed image;상기 제1 인공 신경망의 가중치 벡터는, 상기 제1 이미지의 미리 설정된 객체들의 정보 및 상기 제1 이미지의 픽셀들 각각의 정보를 기초로 구분된 객체들 간의 차이값을 기초로 업데이트되는, 이미지 인페인팅을 위한 동작 방법.The weight vector of the first artificial neural network is updated based on a difference value between objects classified based on information on preset objects of the first image and information on each pixel of the first image. how it works for you.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
KR1020200134397A KR102389304B1 (en) | 2020-10-16 | 2020-10-16 | Method and device for image inpainting considering the surrounding information |
KR10-2020-0134397 | 2020-10-16 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2022080681A1 true WO2022080681A1 (en) | 2022-04-21 |
Family
ID=81208271
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/KR2021/012505 WO2022080681A1 (en) | 2020-10-16 | 2021-09-14 | Image inpainting method and device considering peripheral region |
Country Status (2)
Country | Link |
---|---|
KR (1) | KR102389304B1 (en) |
WO (1) | WO2022080681A1 (en) |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20110103644A1 (en) * | 2009-10-30 | 2011-05-05 | Zoran Corporation | Method and apparatus for image detection with undesired object removal |
KR20150009184A (en) * | 2013-07-16 | 2015-01-26 | 삼성전자주식회사 | Apparatus and method for processing an image having a camera device |
KR20190078543A (en) * | 2017-12-26 | 2019-07-04 | 삼성전자주식회사 | Image acqusition device and controlling method thereof |
-
2020
- 2020-10-16 KR KR1020200134397A patent/KR102389304B1/en active IP Right Grant
-
2021
- 2021-09-14 WO PCT/KR2021/012505 patent/WO2022080681A1/en active Application Filing
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20110103644A1 (en) * | 2009-10-30 | 2011-05-05 | Zoran Corporation | Method and apparatus for image detection with undesired object removal |
KR20150009184A (en) * | 2013-07-16 | 2015-01-26 | 삼성전자주식회사 | Apparatus and method for processing an image having a camera device |
KR20190078543A (en) * | 2017-12-26 | 2019-07-04 | 삼성전자주식회사 | Image acqusition device and controlling method thereof |
Non-Patent Citations (2)
Title |
---|
AHN, WOO JIN : "A Study on inpainting background images", PROCEEDINGS OF DOMESTIC CONFERENCE OF INSTITUTE OF CONTROL, ROBOTICS AND SYSTEMS (ICROS 2019); MAY 16-18, 2019, 1 May 2019 (2019-05-01) - 18 May 2019 (2019-05-18), Korea, pages 381 - 382, XP009535836 * |
YANG CHAO; LU XIN; LIN ZHE; SHECHTMAN ELI; WANG OLIVER; LI HAO: "High-Resolution Image Inpainting Using Multi-scale Neural Patch Synthesis", 2017 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), IEEE COMPUTER SOCIETY, US, 21 July 2017 (2017-07-21), US , pages 4076 - 4084, XP033249760, ISSN: 1063-6919, DOI: 10.1109/CVPR.2017.434 * |
Also Published As
Publication number | Publication date |
---|---|
KR102389304B1 (en) | 2022-04-21 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2022108061A1 (en) | Method and device for joint denoising and demosaicing using neural network | |
WO2010038941A2 (en) | Apparatus and method of obtaining high resolution image | |
WO2017007096A1 (en) | Image capturing apparatus and method of operating the same | |
WO2013103184A1 (en) | Apparatus and method for improving image using color channels | |
WO2010101434A2 (en) | Apparatus and method of generating panoramic image and computer-readable recording medium storing program for executing the method | |
US20130242127A1 (en) | Image creating device and image creating method | |
WO2020111426A1 (en) | Method and system of presenting moving images or videos corresponding to still images | |
WO2009151292A2 (en) | Image conversion method and apparatus | |
WO2020085874A1 (en) | Method and apparatus for dynamic image capturing based on motion information in image | |
WO2015102203A1 (en) | Image processing apparatus, image processing method, and computer-readable recording medium | |
EP3776377A1 (en) | Method and system for dnn based imaging | |
US5644359A (en) | Image input apparatus having a white balance correction means and a method of inputting an image using the image input apparatus | |
WO2013118955A1 (en) | Apparatus and method for depth map correction, and apparatus and method for stereoscopic image conversion using same | |
WO2021241804A1 (en) | Multi-flow-based frame interpolation device and method | |
WO2020040521A1 (en) | Method for synthesizing intermediate view of light field, system for synthesizing intermediate view of light field, and method for compressing light field | |
WO2019035551A1 (en) | Apparatus for composing objects using depth map and method for the same | |
WO2022080680A1 (en) | Artificial-intelligence-based image inpainting method and device | |
EP3164992A1 (en) | Method and apparatus for image capturing and simultaneous depth extraction | |
WO2022080681A1 (en) | Image inpainting method and device considering peripheral region | |
WO2021137555A1 (en) | Electronic device comprising image sensor and method of operation thereof | |
WO2020055196A1 (en) | Apparatus and methods for generating high dynamic range media, based on multi-stage compensation of motion | |
WO2020189953A1 (en) | Camera analyzing images on basis of artificial intelligence, and operating method therefor | |
WO2023219466A1 (en) | Methods and systems for enhancing low light frame in a multi camera system | |
WO2019151808A1 (en) | Electronic device for compressing image by using compression attribute generated in image acquisition procedure using image sensor, and operating method thereof | |
WO2023080667A1 (en) | Surveillance camera wdr image processing through ai-based object recognition |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21880329 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 21880329 Country of ref document: EP Kind code of ref document: A1 |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 21880329 Country of ref document: EP Kind code of ref document: A1 |
|
32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 20.10.2023) |