KR102546070B1 - Method for removing noise from time lapse video - Google Patents

Abstract

The present invention relates to a method for removing noise from a time-lapse video.
A method for denoising a time-lapse image according to an example of the present invention includes the steps of generating a codebook including at least one codeword for a plurality of pixels arranged in time-series by position of each image frame included in the image; Correcting the codebook by calculating a continuity factor among at least one codeword, extracting a noise pixel from among the plurality of pixels included in each image frame by using the continuity factor, and including it in each image frame. Generating a super pixel by grouping pixels having the same color of the plurality of pixels having a predetermined threshold value or less, and when a ratio occupied by the noise pixel in the super pixel is equal to or greater than a predetermined threshold value, the super pixel is generated. and determining the pixel as a noise region.

Description

Method for removing noise from time lapse video {Method for removing noise from time lapse video}

The present invention relates to a method for removing noise from a time-lapse image, and more particularly, to a method for removing noise from a time-lapse image captured at preset time intervals using a codebook method and a super pixel generation method.

A time-lapse video is a video that collects a list of images photographed at a long time interval, reduces them into a short time, and plays them back. There are two main ways to create a time-lapse video.

The first method is a method of editing an existing video to increase the playback speed, and the second method is a method of combining the captured images after photographing the same subject at a long time interval with a still image camera. If the purpose of shooting is to create a time-lapse video, most people use the second method.

Time-lapse video has the advantage of being able to show changes over a long period of time in a short period of time, so it is used in a variety of applications.

One example is the creation of educational content that shows the growth of plants. In the case of the ‘Life Cycle of a Plant’ unit in the 3rd and 4th grade science subjects of elementary school, students grow plants and observe the life cycle process to learn that plants have a lifetime.

A time-lapse video can be created by continuously filming the growth process of a plant from a fixed angle and collecting the videos.

Even in the case of physics experiments, in the case of experiments that require a long time, time-lapse video can be used to create content that can effectively convey the contents of the experiment to students.

Another example can be applied to tourism. Showing the changes of a specific area in a short period of time to tourists visiting other tourist destinations with the scenery of the four seasons can effectively convey the ecological information of the area to tourists.

Studies to improve the quality of such time-lapse images are also being conducted. In particular, when noise pixels are separated from the background using an insufficient number of time-lapse images, it is difficult to express the background as a statistical model.

A Gaussian Mixture Model (GMM) method and a codebook method for modeling a background by quantizing a plurality of pixel values have been used as a method of removing unwanted noise pixels from a conventional time-lapse image.

However, both the GMM method and the codeword method are widely used for background modeling, but basically have a problem in that a step of learning using sufficient images is required, and there is a limit to removing noise from the background when the given image is not sufficient. .

KR 10-1215987B1

The problem to be solved by the present invention relates to a method for removing noise from a time-lapse image, and more particularly, removes noise from a time-lapse image captured at a preset time interval using a codebook method and a method for generating a super pixel. It's about how to do it.

A method for denoising a time-lapse image according to an example of the present invention includes at least one codeword for a plurality of pixels arranged in time-sequential order by position of each image frame included in an image captured at a preset time interval. Generating a codebook, correcting the codebook by calculating a continuity factor representing the maximum number of pixels in which the same codeword among the at least one codeword is time-sequentially consecutive, and modifying the codebook by using the continuity factor. extracting a noise pixel from among the plurality of pixels included in each image frame, generating a super pixel by grouping pixels having the same color below a preset threshold for the plurality of pixels included in each image frame. and determining the super pixel as a noise region when a ratio occupied by the noise pixel in the super pixel is equal to or greater than a preset threshold.

Each of the at least one codeword may include a color factor representing minimum and maximum values of at least one color component included in each pixel, a pixel number factor representing the number of pixels constituting the same codeword, and the continuity factor. there is.

In the step of generating the codebook, the codebook is generated for each position of each image frame, and the at least one codeword is stored as a group of information about pixels arranged time-sequentially at the same position of each image frame. It can be.

The color factor and the pixel number factor excluding the continuity factor may be generated in the at least one codeword generated in the step of generating the codebook.

In the generating of the codebook, a codeword is generated based on the color value of the first pixel for each position of the first frame among the image frames listed in time series, and then any one of all pixels after the first If the color value of the current pixel corresponds to the pre-generated first codeword, the color value of the current pixel is updated to the pre-generated first codeword, or the color value of the current pixel is different from the first codeword. The first and second codewords may be merged when they correspond to different second codewords, or a new codeword may be added when there is no pre-generated codeword corresponding to the color value of the current pixel.

When the color value of the current pixel is within the threshold value range of the color factor of the first codeword, it is determined that the current pixel corresponds to the first codeword, and the color value of the current pixel is set to the first codeword. The first codeword may be updated by reflecting the color factor of 1 codeword and increasing the number of pixels factor of the first codeword.

When the current pixel corresponds to the first and second codewords, the current pixel is merged into any one of the first and second codewords, but the other codeword is invalidated and included in the other codeword. The color factor value can be reflected in any one of the above codewords.

The at least one codeword generated in the step of generating the codebook is plural, and in the step of modifying the codebook, if the codeword of the current pixel matches the previous codeword for the previous pixel, the same codeword is consecutive. A current contiguous length defined as an attribute is increased, and when the current contiguous length is greater than a continuity factor of the previous codeword, the continuity factor of the previous codeword may be updated to the current contiguous length.

In the step of extracting the noise pixel, the current pixel may be extracted as noise when the pixel number factor and the continuity factor of the codeword for the current pixel are equal to or less than a predetermined threshold value.

In the step of extracting the noise pixels, when the pixel number factor of the codeword for the current pixel is less than or equal to a preset minimum threshold value for the pixel number factor, the current pixel is detected as noise, and the number of pixels for the current pixel is detected. If the number of pixels factor of the codeword is greater than the preset maximum threshold for the number of pixels factor, the current pixel is classified as a background, and the continuity factor of the codeword for the current pixel classified as a background is set to the preset number of pixels factor. If it is less than the threshold for the continuity factor, it may be detected as noise, and if it is greater than the threshold for the continuity factor, it may be detected as the background.

After determining the noise area, the method may further include replacing the superpixel determined as the noise area with a color of a superpixel located in the same part of a time-sequentially adjacent image frame.

In the step of inpainting the noise area, super pixels located in the same part of the time-sequentially adjacent image frames may be super pixels determined as the background area.

The present invention applies a codebook method to a time-lapse image, generates a codeword having a new configuration factor, and generates noise pixels from a plurality of pixels arranged in time series using a pixel number factor and a continuity factor of the generated codeword. By detecting it, it is possible to efficiently remove noise even in a time-lapse image having a relatively small image frame.

1 is a diagram for explaining an example of a noise removal system for removing noise from a time-lapse image according to an embodiment of the present invention.
2 is a flowchart for explaining an example of a method of removing noise from a time-lapse image according to an embodiment of the present invention.
3 is a diagram for explaining the concept of a codeword for a pixel for each position of a time-lapse video.
FIG. 4 is a flow chart for explaining an example of a method of generating a codebook in FIG. 2 .
FIG. 5 is an example of an algorithm for generating a codebook according to the flow chart shown in FIG. 4 .
6(a) is an example of an algorithm for creating a new codeword in FIG. 5, and (b) of FIG. 6 is an example of an algorithm for updating a pre-generated codeword in FIG.
FIG. 7 is an example of an algorithm for merging codewords previously generated in FIG. 5 .
8 is an example of an algorithm for calculating a continuity factor by modifying the codebook in FIG. 2 .
FIG. 9 is a diagram for explaining an example of a method of extracting a noise pixel in FIG. 2 .
FIG. 10 is an example of an algorithm for extracting noise pixels according to FIG. 9 .
11 is an example of a time-lapse video to be subjected to noise extraction.
FIG. 12 illustrates an example of noise pixels extracted from the t11 image frame of FIG. 11 .
13(a) shows an example in which super pixels are generated for the t11 image frame of FIG. 11, and FIG. 13(b) shows a noise area among a plurality of super pixels shown in FIG. An example of the determined super pixel is shown.
FIG. 14 illustrates an example in which noise is removed from the t11 image frame of FIG. 11 .

Hereinafter, the embodiments disclosed in this specification will be described in detail with reference to the accompanying drawings, but the same or similar elements are given the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. In addition, in describing the embodiments disclosed in this specification, if it is determined that a detailed description of a related known technology may obscure the gist of the embodiment disclosed in this specification, the detailed description thereof will be omitted.

Terms including ordinal numbers, such as first and second, may be used to describe various components, but the components are not limited by the terms. These terms are only used for the purpose of distinguishing one component from another.

Singular expressions include plural expressions unless the context clearly dictates otherwise.

In this application, each step described can be performed regardless of the listed order, except for the case where it must be performed in the listed order due to a special causal relationship.

In this application, terms such as "comprise" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Hereinafter, the present invention will be described with reference to the accompanying drawings.

A method for removing noise in a time-lapse image according to an embodiment of the present invention may be effective in removing noise that suddenly appears in a time-lapse image using images captured at long time intervals for correlation observation.

When an image that is not sufficient to train a general background model is given, the present invention newly defines the function of the codeword to effectively model the background and apply it to a small number of images based on the codebook method.

In the present invention, in order to remove noise from a background image, a codebook is first generated, a noise region is determined and removed using the generated codebook, and then the removed noise region can be filled in time-sequentially using neighboring images.

1 is a diagram for explaining an example of a noise removal system for removing noise from a time-lapse image according to an embodiment of the present invention.

A noise removal system according to an example of the present invention may remove noise from a time-lapse video.

The noise canceling system may be a fixed terminal implemented as a computer device or a mobile terminal. As an example, the noise removal system may include a smart phone, a mobile phone, a tablet PC, a navigation device, a computer, a laptop computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), and the like.

Such a noise removal system may include a processor 100 , a memory 200 , an input unit 300 and an output unit 400 .

The processor 100 may be configured to process commands of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to processor 100 by memory 200 . For example, the processor 100 may be configured to execute received instructions according to program codes stored in a recording device such as the memory 200 .

For example, the processor 100 may generate or modify a codebook for time-lapse images captured at preset time intervals according to a preloaded program, extract noise pixels, and target each image frame of the time-lapse images. After generating a super pixel to determine the noise area, it is possible to perform a function of inpainting the noise area with the background color.

The memory 200 is a computer-readable recording medium, and may include a permanent mass storage device such as a random access memory (RAM), a read only memory (ROM), and a disk drive, and the Internet ( A form of web storage performing a storage function of the memory 200 on the internet may also be included.

Also, an operating system or at least one program code may be stored in the memory 200 . These software components may be loaded from a computer-readable recording medium separate from the memory 200 .

The separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD/CD-ROM drive, and a memory 200 card.

In such a memory 200, a time-lapse image input by a user may be stored, information on a codebook processed by the processor 100, information on noise pixels, information on super pixels, and the like may be stored.

The input unit 300 may include a keyboard, mouse, USB input terminal, and the like, and the output unit 400 may include a display device on which an image is displayed.

2 is a flowchart for explaining an example of a method of removing noise from a time-lapse image according to an embodiment of the present invention, and FIG. 3 is a flow chart for explaining the concept of a codeword for each pixel for each location of a time-lapse image. 4 is a flow chart for explaining an example of a method for generating a codebook in FIG. 2, and FIG. 5 is an example of an algorithm for generating a codebook according to the flow chart shown in FIG.

As shown in FIG. 2, a method for removing noise from a time-lapse image according to an embodiment of the present invention includes generating a codebook (S100), modifying the codebook (S200), and extracting noise pixels. (S300), generating a super pixel (S400), determining a noise area (S500), and inpainting the noise area (S600).

In order to generate a codebook, first, a time-lapse image taken from the outside at a preset time interval may be input.

Specifically, the time-lapse video may be a set of image frames IF1 to IFN captured at preset time intervals (eg, 5 minutes, 10 minutes, 1 hour, etc.) as shown in (a) of FIG. 2 .

In the step of generating a codebook (S100), a codebook including at least one codeword is generated for a plurality of pixels arranged in time-series for each position of each image frame included in the time-lapse video captured at a preset time interval (S100). It can be.

Here, the codebook may be generated for a plurality of pixels arranged in time series at each position of each image frame in the type-lapse video.

For example, as shown in (a) of FIG. 3, when a time-lapse video is composed of N image frames, the number of vertical pixels is an, and the number of horizontal pixels is bn in each image frame, the codebook (CB) is each It may be generated for a plurality of pixels that are sequentially listed from positions (eg, a1 and b1) to t1 to tN.

The codebook (CB) for each position may include at least one codeword (C1, C2, Ci, CL) as shown in (b) of FIG.

The codebook (CB) generated in the step of generating the codebook (S100) may be generated including at least one codeword for each position of each image frame, and each of the at least one codeword is at the same position of each image frame. Information on pixels listed in time series may be grouped and stored.

For example, as shown in (b) of FIG. 3, in each of the N image frames, there may be N pixels arranged in time series at specific positions (a1, b1), and each of at least one codeword (eg, C1, C2, Ci, CL) may store information about pixels having the same color within a threshold range among N pixels by grouping them.

In (b) of FIG. 3, the case in which the codebook (CB) is generated at the position (a1, b1) of each image frame is shown as an example, but the codebook may be generated for each position of each image frame.

)은 다음의 수학식 1과 같을 수 있으며, 복수의 픽셀 각각은 수학식 2와 같이, 적어도 하나의 컬러값을 가질 수 있다.When a time-lapse image of N image frames is given, a plurality of pixels configured for each position (

) may be the same as Equation 1 below, and each of a plurality of pixels may have at least one color value as shown in Equation 2.

Here, Xi denotes a pixel of the i-th image and xj denotes a j-th color component among three color components. Here, the three color components may include the first color (x1), the second color (x2), and the third color (x3), and for example, the three color components are red (R) and green ( G) and blue (B), but the present invention is not limited thereto.

)은 적어도 하나의 코드워드(예를 들어, L개)로 구성된 코드북 C로 모델링될 수 있으며, 일례로, 다음의 수학식 3과 같이 표현될 수 있다.Each set of pixels like this (

) may be modeled as a codebook C composed of at least one codeword (eg, L), and may be expressed as, for example, Equation 3 below.

Each of the at least one codeword is a color factor representing minimum values (min1, min2, min3) and maximum values (max1, max2, max3) of at least one color component included in each pixel, and the number of pixels constituting the same codeword. and a continuity factor (G) representing the maximum number of consecutive pixels in time series.

In Equation 4, Ci means the i-th codeword among a plurality of codewords constituting the codebook. Each codeword may consist of 8 attributes, and min j and max j may be color factors representing the minimum and maximum values of the j th color component among color components, respectively. In the case of Equation 4, each codeword may include a maximum value and a minimum value for three color components.

M may be a pixel count factor (M) representing the number of pixels constituting the same codeword, and G may be a continuity factor (G) representing the maximum number of pixels that are consecutive in time series among pixels corresponding to the same codeword. can

In the conventional codebook method, a codeword uses a value of brightness or an average value of pixels, but in the codebook method of the present invention, the maximum value and minimum value for each color component can be used.

First, a method for generating a codebook using N consecutive pixels is an example, as shown in FIG.

As shown in FIG. 4, generating a codebook includes generating a codeword for a first pixel (S110), updating the first codeword (S120), and merging the first and second codewords. It may include step S130 and adding a new codeword (S140).

First, as shown in FIG. 4, in the step of generating a codeword for the first pixel (S110), the codeword is based on the color value of the first pixel among a plurality of pixels listed in time series at each position of a plurality of image frames. Can be generated (S110).

Then, in the step of updating the first codeword (S120), if the color value of the current pixel, which is any one of all pixels after the first one, corresponds to the pre-generated first codeword, the color value of the current pixel is based on The generated first codeword may be updated (S120).

For example, if a first codeword for a first pixel is generated and a color value of a second pixel is equal to a color factor of the first codeword within a threshold range, the value of the second pixel will be updated to the first codeword. can

Similarly, when the color value of the n-th pixel is the same as that of another pre-generated codeword within a critical range, the value of the n-th pixel may be updated to the other pre-generated codeword.

Thereafter, in the step of merging the first and second codewords (S130), when the color value of the current pixel corresponds to a second codeword other than the first codeword, the first and second codewords are merged with each other (S130) It can be.

For example, if the current pixel corresponds to the pre-generated first and second codewords, the first codeword and the second codeword may be merged with each other after the current pixel is updated to the first codeword.

Next, in the step of adding a new codeword (S140), if there is no pre-generated codeword corresponding to the color value of the current pixel of the current pixel, the current pixel may be generated as a new codeword and added (S140).

As such, the implementation method for generating the codebook (S100) may be implemented as Algorithm 1 as shown in FIG. 5, for example.

In Algorithm 1 of FIG. 5, Ci(j) may represent the j attribute of the i-th codeword. Like Algorithm 1, generating a codebook (S100) may start with adding a first pixel to a codeword (S110).

The first for statement may mean creating a codebook using all pixels after the first one.

The second for statement measures the similarity with the codeword generated for the current pixel, and if the color value for the current pixel is within the critical range of the color factor of the generated codeword, the generated codeword is updated ( S120) or merging (S130).

Also, when the color value of the current pixel exceeds a critical range of a color factor of the previously generated codeword and is not similar to the previously generated codeword, the value of the current pixel may be added as a new codeword.

The present invention determines that there is no pre-generated code word corresponding to the current pixel when at least one color value among a plurality of color values for the current pixel is out of the critical range of the color factor value of the pre-generated code word, New codewords can be added. A detailed description of Algorithm 1 of FIG. 5 is the same as that of FIG. 4 and thus omitted.

A case of updating the color value of the current pixel to a pre-generated codeword, merging two codewords, or adding a new codeword will be described in more detail with reference to FIGS. 6 and 7 below.

In the step of generating the codebook of FIG. 2 (S100), the codebook is generated for each position of each image frame, and at least one codeword included in the codebook is time-sequentially located at the same position of each image frame as described above. Information on the listed pixels may be grouped and stored.

In addition, as described above in Equation 4, each codeword generated in the step of generating a codebook (S100) has continuity among six color factors, a pixel number factor (M), and a continuity factor (G) included in the codeword. Six color factors and a pixel count factor (M) excluding factor (G) can be created.

Then, in FIG. 2, in the step of modifying the codebook (S200), the codebook may be modified by calculating a continuity factor (G) representing the maximum number of pixels in which the same codeword among at least one codeword is time-sequentially consecutive.

After six color factors and a pixel number factor (M) for each codeword are generated in the step of generating the codebook (S100), the continuity of the codeword for each codeword in the step of modifying the codebook (S200) The codebook can be modified for the entire image frame once more to obtain the continuity factor (G) representing

A method for obtaining the continuity factor G will be described in more detail with reference to Algorithm 5 shown in FIG. 8 later. The continuity factor (G) represents the number of pixels in which the same codeword appears consecutively, and only the largest value of the continuity factor (G) can be stored in each codeword included in the same codebook.

In this way, after the codebook is completed, in the step of extracting the noise pixels of FIG. 2 (S300), the codebook and each image frame are input, and the plurality of images included in each image frame are used by using the continuity factor (G) included in the codebook. A noise pixel may be extracted from among the pixels of .

In the step of extracting noise pixels (S300), each image frame may be analyzed using the generated codebook. At this time, a codeword corresponding to each pixel listed in time-sequential order at each position of the image frame is found, and a continuity factor (G) indicating the number of consecutive groups and a factor of the number of pixels constituting the codeword (M) are used. A noise pixel may be detected from among a plurality of pixels listed as .

A more detailed description of the step of extracting the noise pixels (S300) will be described with reference to FIGS. 9 and 10 later.

Next, in the step of generating a super pixel in FIG. 2 (S400), apart from the step of extracting a noise pixel (S300), each image frame is received and a plurality of pixels included in each image frame are below a predetermined threshold value. A super pixel can be created by grouping pixels having the same color.

More specifically, in the step of generating a super pixel ( S400 ), pixels having similar colors in each image frame may be grouped and set as a super pixel.

A method of generating a super pixel may use the SLIC algorithm. In general, the SLIC algorithm performs clustering using the k-mean method, but in the present invention, the k-mean method is not performed for all pixels, but clustering is performed only for a predetermined seed pixel, so that it is more accurate and faster. pixels can be created.

For example, in the step of generating a super pixel (S400), after setting the noise pixel extracted in the step of extracting a noise pixel (S300) as a seed pixel, clustering may be performed only on the seed pixel. However, the present invention is not necessarily limited thereto.

Since the codebook generated according to an example of the present invention is an information value for pixels located in each part of each image frame and arranged in time series, there may be limitations in accurately detecting a region where noise is generated.

However, since a noise pixel is detected from a plurality of pixels arranged in time series in each part of an image frame, after setting the super pixel area for each image frame and reflecting the ratio occupied by the noise pixel in each super pixel, the corresponding super pixel area is set. It is easy to detect whether a pixel is a noise region or not.

Next, in determining the noise area of FIG. 2 ( S500 ), the super pixel may be determined as the noise area when the ratio occupied by the noise pixel in the super pixel is equal to or greater than a preset threshold value.

More specifically, in determining the noise area (S500), assuming that the number of pixels in the area forming the superpixel is Ns and the number of noise pixels in the superpixel is Nn, the value of Nn/Ns is When the value of Nn/Ns is greater than or equal to a preset threshold value, the corresponding super pixel may be determined as the noise region, and when the value of Nn/Ns is less than the threshold value, the corresponding super pixel may be determined as the background region.

In this way, after the noise area is determined for each image frame, the color of the area of the super pixel determined as the noise area may be removed.

Then, in the step of inpainting the noise area of FIG. 2 ( S600 ), the superpixel determined as the noise area may be replaced with the color of a superpixel located in the same part of an adjacent image frame in time series.

More specifically, in the step of inpainting the noise area (S600), the superpixel area is inked using the color of the superpixel located in the same part of the previous image frame (or subsequent image frame) than the superpixel determined as the noise area (S600). By painting, the color of the super pixel determined as the noise area can be replaced.

In this case, super pixels located in the same part of time-sequentially adjacent image frames may be super pixels determined as the background area.

Thereafter, an image as a result of noise filtering may be output.

Hereinafter, an example of generating a new codeword, updating an existing codeword, or merging two codewords in the above-described step of generating a codebook in FIG. 2 (S100) will be described in detail.

6(a) is an example of an algorithm for creating a new codeword in FIG. 5, FIG. 6(b) is an example of an algorithm for updating a pre-generated codeword in FIG. 5, and FIG. This is an example of an algorithm that merges pre-generated codewords.

The method of adding a new codeword in FIGS. 4 and 5 may be implemented as Algorithm 2 shown in (a) of FIG. 6, for example.

When a new codeword (CL) is added for the current pixel (X), as shown in FIG. It is composed of the maximum and minimum values of , and since there is no pre-generated codeword corresponding to the current pixel, the pixel number factor (M) and the continuity factor (G) may each be configured as 1. This newly generated codeword may be updated in the codebook.

In the step of updating the first codeword of FIGS. 4 and 5 (S120), the color values (x1, x2, x3) of the current pixel (X) are within the critical range of the color factor of the first codeword (Ci). If included, as shown in (a) of FIG. 6, it is determined that the current pixel (X) corresponds to the first codeword (Ci), and the color value of the current pixel is assigned to the color factor of the first codeword (Ci). The first codeword may be updated by reflecting and increasing the pixel number factor (M) of the first codeword.

Here, the critical range of the color factor of the first codeword may be larger than the maximum value (Ci(maxj)) of the color factor of the first codeword by a preset value and smaller than the minimum value (Ci(minj)) by a preset value. there is.

Therefore, even if the color values (x1, x2, x3) of the current pixel (X) are smaller than the minimum value (Ci(minj)) or greater than the maximum value (Ci(maxj)) of the color factor of the previously generated codeword, The current pixel may be matched to the pre-generated codeword.

In this way, when the color value of the current pixel exceeds the range of the minimum value to the maximum value of the parasitic codeword, the color values (x1, x2, x3) of the current pixel X are changed when the generated codeword is updated. This may be reflected in the color factors (Ci(minj), Ci(maxj)) of 1 codeword (Ci).

In FIGS. 4 and 5 , a method of updating a current pixel to a pre-generated codeword (S120) may be implemented as Algorithm 3 shown in (b) of FIG. 6, for example.

If there is a pre-generated codeword Ci that matches the current pixel, the property of the pre-generated codeword Ci may be updated by reflecting the current pixel value, as shown in FIG.

When the value of the current pixel (X) is updated to the previously generated codeword (Ci), the number of pixels factor (M) is increased by one in the previously generated codeword (Ci), and the color value of the current pixel (X) is When the minimum or maximum value of the color factor of the pre-generated codeword Ci is smaller than or greater than the maximum value, the minimum or maximum value of the color factor of the pre-generated codeword may be changed to the color value of the current pixel.

In the step of merging the first and second codewords of FIGS. 4 and 5 (S130), when the current pixel corresponds to the first and second codewords Ci and Cj, the first and second codewords Ci, Cj), the other codeword may be invalidated, and the color factor value included in the other codeword may be reflected in any one codeword.

To this end, the range of the codeword can be extended using the current pixel value that is basically input. Specifically, when the first and second codewords Ci and Cj are extended, overlapping occurs. In this case, the first and second codewords Ci and Cj may be merged into one codeword.

Codeword merging according to the present invention may occur when the color values of the current pixel fall within the critical ranges of the color factors of the first and second codewords. At this time, among the first and second codewords, each codeword may be merged into a codeword having a small index, and a codeword having a large index may be invalidated by setting a value to 0. However, merging into a codeword having a small index is an example, and the present invention is not limited thereto, and conversely, merging into a codeword having a large index value is also possible.

In this way, the merging of codewords can be implemented according to Algorithm 4 shown in FIG. 7, for example. Such codeword merging may be performed according to Algorithm 1 of FIG. 5 after the value of the current pixel is updated to the first codeword (codeword index i).

As shown in FIG. 7 , a value having a smaller index value (i, j) among the first and second codewords (Ci, Cj) may be designated as k, and a larger value may be designated as l. If the index i of the first codeword is smaller than the index j of the second codeword, the k value may be designated as i and the l value may be designated as j.

Thereafter, a codeword having a codeword index value of Ck may be designated by summing the value of the factor M of the number of pixels of the first and second codewords Ci and Cj. In this case, the codeword index value of Ck may be the index value of the first codeword Ci.

In the present invention, when merging the first and second codewords Ci and Cj, the first codeword Ci having a smaller index value among the index values i and j of the first and second codewords Ci and Cj ) has been described as an example of merging the second codeword Cj, but the present invention is not limited thereto, and the first codeword has a relatively large index value among the first and second codewords. It is also possible to merge words.

When merging the second codeword (Cj) with the first codeword (Ci), the smaller of the minimum values (Ci(mint), Cj (mint)) of the color factors in each of the first and second codewords is used as the first codeword. It is changed to the minimum value (Ck(mint)) of the color factor of the word (Ci), and the largest of the maximum values (Ci(maxt), Cj (maxt)) of the color factors in each of the first and second codewords (Ci, Cj). The value may be changed to the maximum value (Ck(maxt)) of the color factor of the first codeword.

Hereinafter, an example of the step (S200) of modifying the codebook will be described in detail with reference to FIG. 8.

FIG. 8 is an example of an algorithm for calculating a continuity factor (G) by modifying the codebook in FIG. 2 .

In the step of modifying the codebook (S200), the codebook may be modified while calculating a continuity factor (G) indicating the maximum number of pixels that are sequentially sequential among pixels corresponding to the same codeword among at least one codeword.

In this way, the step of modifying the codebook (S200) may be implemented as Algorithm 5, such as Algorithm 5 shown in FIG. 8, for example.

The step of modifying the codebook (S200) may be performed on codewords for a plurality of pixels located in the same part of each image frame and arranged in time series.

As in Algorithm 5, 1 may be set as an initial value for the continuity factor (G) of the codeword for the first pixel (x1) among the plurality of pixels.

A codeword for a previous pixel is defined as a previous codeword (prev), and the current pixel (X) may be a pixel sequentially progressing in time series from the second pixel.

If the codeword (i) for the current pixel (X) is the same as the previous codeword (prev) for the previous pixel, the same codeword may increase the adjacent length (adjcentLength) defined as a contiguous attribute.

That is, if the color factor of the codeword i for the current pixel X is equal within the critical range of the color factor of the previous codeword, the codeword i for the current pixel X is the same as the previous codeword prev. It is determined that it is the same by matching with , and the value of the adjacent length (adjcentLength) may be increased by 1.

In this way, while increasing the value of the adjacent length (adjcentLength), when the value of the adjacency length is greater than the value of the continuity factor (G) of the previous codeword (prev), the continuity factor (G) of the previous codeword (prev) can be updated with the value of the adjacent length (adjcentLength).

In addition, when the color factor of the codeword i for the current pixel X is out of the critical range of the color factor of the previous codeword prev, the continuity factor of the codeword i for the current pixel X ( G) may be set to 1.

In the step of modifying the codebook (S200), the codebook can be modified by repeating this process up to the Nth image frame.

Accordingly, a codebook is generated and modified by position for each image frame listed in time series, and at least one color factor, a pixel number factor (M) and a continuity factor (G) are determined for at least one codeword included in each codebook. ) can be created.

Hereinafter, a method of extracting a noise pixel for each position of each image frame using the codebook generated as described above and each image frame will be described with reference to FIGS. 9 and 10 .

FIG. 9 is a diagram for explaining an example of a concept of extracting a noise pixel in FIG. 2 , and FIG. 10 is an example of an algorithm for extracting a noise pixel according to FIG. 9 .

In the present invention, in determining the noise area (S500), the super pixel may be determined as the noise area when the ratio occupied by the noise pixel in the super pixel is equal to or greater than a preset threshold value.

That is, in the step of extracting noise pixels ( S300 ), if the pixel number factor (M) and the continuity factor (G) of the codeword for the current pixel are equal to or less than a preset threshold value, the current pixel may be extracted as noise.

For example, in the step of extracting a noise pixel (S300), if the pixel number factor (M) of the codeword for the current pixel is equal to or less than a preset minimum threshold value for the pixel number factor (M), the current pixel is detected as noise. can do.

Conversely, if the pixel number factor (M) of the codeword for the current pixel is greater than the maximum threshold for the preset pixel number factor (M), the current pixel may be classified as a background.

In addition, if the continuity factor (G) of the codeword for the current pixel as the background is smaller than the preset threshold for the continuity factor (G), it is detected as noise, and if it is greater than the threshold for the continuity factor (G), it is returned to the background. can be detected.

For example, in FIG. 9 , a horizontal axis represents a time axis arranged in time series, and pulses displayed on the horizontal axis represent the same codeword.

9(a) may be a case in which code words a(Ca) having the same property are temporally discontinuous with respect to the time axis. Such a code word a(Ca) may have the same maximum and minimum values of the color factor, and illustrates a case where the pixel number factor (M) is 2 and the continuity factor (G) is 1.

In the graph shown in (a) of FIG. 9, the value of the number of pixels (M) forming the same code word a (Ca) is 2 and the value of the continuity factor (G) is 1.

In this case, it can be interpreted that the number of pixels having color values within the same threshold range for the code word a(Ca) is two in total, and there are no consecutive pixels. As shown in (a) of FIG. 9, for a pixel corresponding to the code word a(Ca), the value of the factor M of the number of pixels of the code word a(Ca) is smaller than the minimum threshold value, so that it can be detected as a noise pixel. there is.

As shown in (b) of FIG. 9, when the number of pixels constituting the code word b(Cb) is 11 and the continuity factor G is 6, pixels having color values within the same threshold range for the code word b(Cb) It can be interpreted that the number of is a total of 11, and the maximum number of consecutive pixels is 6.

As shown in (b) of FIG. 9, for a pixel corresponding to code word b(Cb), the value of the factor M of the number of pixels of code word b is greater than the maximum threshold and the value of the continuity factor G is the continuity factor Since it is greater than the threshold for (G), it can be detected as background.

As shown in (c) of FIG. 9, when the number of pixels constituting the code word c(Cc) is 5 and the continuity factor G is 4, pixels having color values within the same threshold range for the code word c(Cc) It can be interpreted that the number of is a total of 5, and the maximum number of consecutive pixels is 4.

As shown in (c) of FIG. 9, for a pixel corresponding to the code word c(Cc), the value of the factor M of the number of pixels of the code word c(Cc) is greater than the maximum threshold value and the value of the continuity factor G Since it is greater than this threshold, it can be detected as background.

However, as shown in (d) of FIG. 9, when the number of pixels constituting the code word d(Cd) is 5 and the continuity factor G is 1, a color value within the same threshold range for the code word d(Cd) The total number of pixels is 5, and the maximum number of consecutive pixels is 1, which can be interpreted as no consecutive pixels.

As shown in (d) of FIG. 9, for a pixel corresponding to code word d(Cd), the factor M of the number of pixels of code word d is greater than the maximum threshold value, but the continuity factor G is less than the threshold value. , can be detected as noise.

As such, in the present invention, the step of extracting a noise pixel (S300) may be implemented as Algorithm 6 shown in FIG. 10 .

10, Thmin represents the minimum threshold for the pixel number factor (M) of the codeword, Thmax represents the maximum threshold value for the pixel number factor (M) of the codeword, and ThG represents the continuity factor of the codeword ( Indicates the threshold for G).

In this way, noise can be extracted for each pixel at each position in each image frame.

Hereinafter, an example of extracting noise using an actual time-lapse image will be described with reference to FIGS. 11 to 13 .

FIG. 11 is an example of a time-lapse video subject to noise extraction, and FIG. 12 illustrates an example of noise pixels extracted from the t11 image frame of FIG. 11 .

13(a) shows an example in which super pixels are generated for the t11 image frame of FIG. 11, and FIG. 13(b) shows a noise area among a plurality of super pixels shown in FIG. An example of the determined super pixel is shown, and FIG. 14 shows an example in which noise is removed from the t11 image frame of FIG. 11 .

In FIG. 11, image frames for t=1, 11, 21, 31, and 41, which are part of time-lapse images related to plant observation having a total of 50 image frames, are shown. Here, as an example, a case in which an insect is photographed with noise in an image frame at a time point t11 is illustrated.

The present invention detects noise, such as insects that suddenly appear and disappear in a plant observation image, deletes an insect photographed portion, and inpaints the deleted portion using time-sequentially adjacent image frames.

To this end, a codebook can be generated and modified using 50 image frames, and noise pixels can be detected from each image frame by applying the generated codebook to each image frame.

As shown in (a) of FIG. 12, in the step of extracting the noise pixels (S300), the generated codebook may be applied to the image frame at time t11 to extract the noise pixels.

As can be seen in (a) of FIG. 12, it can be seen that the insect area in the lower right corner is well detected. In addition, it can be seen that the petal part is also detected as a noise pixel, because the position of the edge part of the petal is changed by wind or the like with the passage of time.

It can be seen that small parts detected as noise pixels also appear inside the flower. Small-sized noise pixels can be removed by applying a morphology operation, and small noise pixels can be removed by applying an opening operation, as shown in FIG. 11(b).

13(a) is an example in which super pixels are generated for an image frame at time t11. In the present invention, in the step of generating a super pixel ( S400 ), as shown in (a) of FIG. 13 , the SLIC algorithm may be applied to each image frame to generate a super pixel like the image frame at time t11.

In addition, in the step of determining the noise area (S500), as shown in FIG. 13(b), when the ratio occupied by the noise pixel in the super pixel is equal to or greater than a preset threshold value, the corresponding super pixel may be determined as the noise area.

Thereafter, the color of the superpixel determined as the noise area is removed, and the noise shown in FIG. 14(a) is removed, and the color of the image frame adjacent to it in time-sequentially is inpainted to remove the noise as shown in FIG. 14(b). image frames may be created.

The present invention applies a codebook method to a time-lapse image, generates a codeword having a new configuration factor, and uses a pixel number factor (M) and a continuity factor (G) of the generated codeword to chronologically arrange a plurality of codewords. By detecting noise pixels from the pixels of , noise can be efficiently removed even in a time-lapse image having a relatively small image frame.

The technical features disclosed in each embodiment of the present invention are not limited to the corresponding embodiment, and unless incompatible with each other, the technical features disclosed in each embodiment may be merged and applied to other embodiments.

Therefore, in each embodiment, each technical feature is mainly described, but each technical feature may be merged and applied to each other unless incompatible with each other.

The present invention is not limited to the above-described embodiments and accompanying drawings, and various modifications and variations will be possible from the viewpoint of those skilled in the art to which the present invention belongs. Therefore, the scope of the present invention should be defined by not only the claims of this specification but also those equivalent to these claims.

Claims (12)

generating a codebook including at least one codeword for a plurality of pixels arranged in chronological order according to positions of each image frame included in an image captured at preset time intervals;
modifying the codebook by calculating a continuity factor indicating a maximum number of pixels in which the same codeword among the at least one codeword is time-sequentially consecutive;
extracting, as noise pixels, pixels having the continuity factor equal to or less than a set threshold value among the plurality of pixels included in each image frame;
generating a super pixel by grouping pixels having the same color with a predetermined threshold value or less for the plurality of pixels included in each image frame;
determining the super pixel as a noise area when a ratio occupied by the noise pixel in the super pixel is equal to or greater than a predetermined threshold value; and
Replacing the superpixel determined as the noise area with the color of a superpixel located in the same part of an adjacent image frame in time series;
Each of the at least one codeword includes a color factor representing minimum and maximum values of at least one color component included in each pixel, a pixel number factor representing the number of pixels constituting the same codeword, and the continuity factor. How to remove noise from lapse video. delete According to claim 1,
In the step of generating the codebook,
The codebook is generated for each position of each image frame,
Wherein the at least one codeword is a method of removing noise of a time-lapse image in which information on pixels sequentially listed in time series at the same location of each image frame is grouped and stored. According to claim 1,
The method of removing noise from a time-lapse image, wherein the color factor and the pixel number factor excluding the continuity factor are generated in the at least one codeword generated in the step of generating the codebook. According to claim 1,
The step of generating the codebook is
After generating a codeword based on the color value of the first pixel for each position of the first frame among the image frames listed in time series,
If the color value of the current pixel, which is any one of all pixels after the first one, corresponds to the previously generated first codeword, update the color value of the current pixel to the previously generated first codeword;
When the color value of the current pixel corresponds to a second codeword other than the first codeword, the first and second codewords are merged, or
A method of removing noise from a time-lapse image in which a new codeword is added when there is no pre-generated codeword corresponding to the color value of the current pixel. According to claim 5,
When the color value of the current pixel is included within the threshold value range of the color factor of the first codeword,
It is determined that the current pixel corresponds to the first codeword, the color value of the current pixel is reflected in the color factor of the first codeword, and the number of pixels factor of the first codeword is increased to obtain the first codeword. 1 A method for removing noise from time-lapse video updating codewords. According to claim 5,
When the current pixel corresponds to the first and second codewords,
Merge into any one codeword of the first and second codewords, but invalidate the other codeword,
A method of removing noise from a time-lapse image in which a color factor value included in the other codeword is reflected in the one codeword. According to claim 1,
The at least one codeword generated in the step of generating the codebook is plural,
In the step of modifying the codebook,
If the codeword of the current pixel matches the previous codeword for the previous pixel, then increment the current contiguous length defined by the property that the same codeword is contiguous;
When the current contiguous length is greater than the continuity factor of the previous codeword, updating the continuity factor of the previous codeword with the current contiguous length. According to claim 1,
In the step of extracting the noise pixel,
A method for removing noise from a time-lapse image in which the current pixel is extracted as noise when the pixel number factor and the continuity factor of a codeword for the current pixel are equal to or less than a preset threshold. According to claim 9,
In the step of extracting the noise pixel,
If the pixel number factor of the codeword for a current pixel is less than or equal to a preset minimum threshold for the pixel number factor, detecting the current pixel as noise;
classifying the current pixel as a background when the pixel count factor of the codeword for the current pixel is greater than a preset maximum threshold for the pixel count factor;
When the continuity factor of the codeword for the current pixel classified as background is smaller than a preset continuity factor threshold, it is detected as noise, and when it is greater than the continuity factor threshold, time-lapse video is detected as background. of noise removal method. delete According to claim 1,
In the step of replacing with the color of the super pixel,
The method of removing noise from a time-lapse image wherein the superpixels located in the same part of the time-sequentially adjacent image frames are superpixels determined as a background area.

Images