KR102206490B1 - Image process system by applying selective filtering and image process method therefor - Google Patents

Abstract

An image processing system comprises: a frequency conversion unit converting an input image in units of patches into a frequency to output a frequency coefficient matrix; a filter unit including at least one filter capable of filtering the input image in units of patches using the frequency coefficient matrix; and a control unit generating a control signal for controlling the filter unit. Therefore, the contradictory relationship of trade-offs between filters can be resolved.

Description

An image processing system applying selective filtering and its image processing method {IMAGE PROCESS SYSTEM BY APPLYING SELECTIVE FILTERING AND IMAGE PROCESS METHOD THEREFOR}

The present invention relates to an image processing system to which selective filtering is applied, and to an image processing method thereof, and more particularly, to an image processing system capable of performing different filtering processing for each area of an image of one frame, and an image processing method thereof. About.

In image processing, most of the denoising methods perform filtering in a way to reduce the difference from neighboring pixels using information of neighboring pixels to remove noise. As a result of this, the resolution is reduced at the boundary. Falling blurring is common.

In addition, most of the boundary line sharpening methods improve the resolution by performing filtering by increasing the difference with the neighboring pixels using information of the neighboring pixels, and a noising phenomenon that increases noise due to adverse effects. This usually appears.

Therefore, since noise removal and boundary enhancement methods have a contradictory relationship of trade off with each other in their results and effects, they operate through separate modules or processes, respectively, and when controlling the camera, external illumination, camera exposure, and The degree of noise removal and the degree of border reinforcement are adjusted by reflecting sensitivity, etc.

The present invention is an invention aimed at solving the technical problem as described above, and makes it possible to select a plurality of filters having various different effects according to the need in the image area, so that the filters are traded off with each other. An object of the present invention is to provide an image processing system and an image processing method to which selective filtering can be applied to solve the contradictory relationship of ).

An image processing system of the present invention includes: a frequency converter configured to output a frequency coefficient matrix by converting a frequency of an input image in units of patches; A filter unit including at least one filter capable of filtering the input image in a patch unit using the frequency coefficient matrix; And a control unit that generates a control signal for controlling the filter unit.

Specifically, the filter unit may include a noise removal filter unit configured to remove noise by using the frequency coefficient matrix; A boundary line enhancement and signal amplification unit for enhancing a boundary line and amplifying a signal using the frequency coefficient matrix; And a blurring and masking filter unit that performs blurring and masking by using the frequency coefficient matrix. It is preferable to include at least one of. In addition, the control unit, the noise removal filter unit; The boundary line enhancement and signal amplification unit; And the blurring and masking filter unit. Generating a control signal for controlling at least one of or not to perform at least one of, and the noise removal filter unit; The boundary line enhancement and signal amplification unit; And the blurring and masking filter unit. When performing two or more of them, each weight is set.

In addition, the filter unit, the noise removal filter unit by the control unit; The boundary line enhancement and signal amplification unit; And the blurring and masking filter unit. When two or more of them are implemented, the noise removal filter unit; The boundary line enhancement and signal amplification unit; And the blurring and masking filter unit. And a multi-filter summing unit that multiplies and adds each weight to the output performed in the output.

Preferably, the noise removal filter unit is characterized in that noise is removed by comparing the frequency coefficient matrix and a threshold matrix. The control unit may set a value of a threshold matrix of the noise removal filter unit.

In addition, the boundary line enhancement and signal amplification unit may perform a Hadamard product operation on the weight matrix and the frequency coefficient matrix to strengthen the boundary line, and multiply the signal amplification gain to amplify the boundary line. The controller may set a value of a weight matrix of the boundary line enhancement and signal amplification unit and a value of a signal amplification gain.

In addition, the blurring and masking filter unit adjusts filtering processing intervals in vertical and horizontal directions, and performs blur filtering by selecting only DC component values from the frequency coefficient matrix. The control unit is characterized in that it is possible to set the filtering processing interval in the vertical and horizontal directions of the blurring and masking filter unit.

Preferably, the image processing system of the present invention further comprises an image information extracting unit that extracts information on quality of the input image in pixel units using the frequency coefficient matrix.

Specifically, the image information extracting unit may include an image noise level evaluation unit for evaluating a noise level of the input image using the frequency coefficient matrix; And an image sharpness evaluation unit that evaluates the sharpness of the input image by using the frequency coefficient matrix. In addition, the image noise level evaluation unit evaluates the first noise level using the energy of the high-frequency components of the input image, and evaluates the second noise level using the difference between the input image and the noise-removed image. . In addition, the image sharpness evaluation unit evaluates the sharpness of the input image by using a product of the energy of the frequency coefficient matrix and an intermediate frequency selection and weight matrix taking an intermediate frequency band of the frequency coefficient matrix.

In addition, the image processing system of the present invention further comprises a frequency inverse transform unit configured to output an image by performing inverse frequency transformation on the output of the filter unit. The frequency converter includes: a first transformation matrix; The frequency coefficient matrix; And a transpose matrix of the second transformation matrix; Preferably, the frequency inverse transform unit comprises: a transpose matrix of the first transform matrix; An output of the filter unit; And the second transformation matrix.

In addition, the image processing system of the present invention changes the patch for the same pixel position, and performs the frequency conversion unit, the filter unit, and the frequency inverse transform unit a plurality of times, and uses statistical values of a plurality of output values. It further includes an image synthesizing unit for synthesizing. In addition, the image processing system of the present invention, the input image; A composite image of the image synthesis unit; And user setting screen; It is preferable to further include an image output unit that outputs at least one of or outputs an image using at least one.

In addition, the controller may display at least some image regions in a specific color according to the image sharpness value evaluated by the image sharpness evaluation unit; Alternatively, the image output unit may be controlled to output an image by displaying a bitmap or a black and white grayscale image. In addition, the control unit may control the image output unit to output by adding a plurality of grid types at a predetermined interval to the input image, and the image noise level evaluation unit or the image using user input information for selecting a grid. The area to be evaluated by the sharpness evaluation unit can be set.

Preferably, the control unit may control the input image to be output by the image output unit, and the image noise level evaluation unit or the image noise level evaluation unit using user input information for setting a region of interest of an arbitrary size among the input images The area to be evaluated by the image sharpness evaluation unit can be set.

In addition, the control unit may control the input image to be output by the image output unit, and may include the noise removal filter unit by using user input information of the input image; The boundary line enhancement and signal amplification unit; And the blurring and masking filter unit. You can select one to be implemented or set an area to be implemented. In addition, the control unit, the noise removal filter unit; The boundary line enhancement and signal amplification unit; And the blurring and masking filter unit. The image output unit may be controlled to output a parameter setting screen so that a user may set a parameter for the selected one.

According to the present invention, it is possible to select a plurality of filters having various different effects according to need in an image area, thereby solving the contradictory relationship of trade-offs between filters.

1 is a block diagram of an image processing system to which selective filtering is applied according to a preferred embodiment of the present invention.
2 is an exemplary diagram of an output of a noise removal filter unit and a boundary line enhancement filter unit.
3 is an exemplary diagram of a result of performing a blurring and masking filter unit.
4 is an exemplary view of a result of estimating sharpness by an image sharpness evaluation unit.
5 is an exemplary view of the image processing system of the present invention.

Hereinafter, an image processing system to which selective filtering is applied and an image processing method thereof according to embodiments of the present invention will be described in detail with reference to the accompanying drawings.

It goes without saying that the following examples of the present invention are intended to embody the present invention, and do not limit or limit the scope of the present invention. What can be easily inferred by experts in the technical field to which the present invention pertains from the detailed description and examples of the present invention is interpreted as belonging to the scope of the present invention.

The overall operation of the image processing system 1000 of the present invention will be described as follows.

The image processing system 1000 of the present invention performs filtering by sequentially taking patches X of M×N size included in the input image I of H×W size, and in this process, the execution position is set to the entire image and a partial area. Or the like, or the execution interval s, and during execution, by appropriately selecting and selecting a plurality of filters according to the control signal of the control unit 700, various filtering effects can be obtained within one filtering framework. Is composed. For reference, the image processing system 1000 of the present invention inputs the starting position of the M×N patch X to the frequency converter 100 while moving one pixel at a time. Accordingly, the frequency converting unit 100, the filter unit 200, and the frequency inverse transform unit 300 are each performed at most M×N times for one pixel.

를 영상 합성부(400)에서, 최종 결과 영상 J를 생성하게 된다. 주파수 변환부(100)를 거친 주파수 출력 정보는 영상 정보 추출부(500)에 입력되어 잡음도와 선명도를 평가한 후 그 결과가 제어부(700)로 입력되고, 제어부(700)에서는 필터부(200)와 영상 정보 추출부(500)의 파라미터들을 조정하게 된다. 필터부(200)는 잡음 제거, 경계선 강화 및 신호 증폭, 그리고 흐릿화 및 마스킹 필터부(230)가 포함되어 동시에 신호 처리를 수행한 후 결과를 출력한다. The resulting M×N sized input image patch X is subjected to frequency transformation, multiple filtering processes, and frequency inverse transformation.

The image synthesizing unit 400 generates a final result image J. The frequency output information that has passed through the frequency conversion unit 100 is input to the image information extracting unit 500 to evaluate noise and clarity, and the result is input to the control unit 700, and the control unit 700 includes the filter unit 200 And parameters of the image information extraction unit 500 are adjusted. The filter unit 200 includes a noise removal, boundary line enhancement and signal amplification, and a blurring and masking filter unit 230 to simultaneously perform signal processing and output a result.

1 is a block diagram of an image processing system 1000 to which selective filtering is applied according to an exemplary embodiment of the present invention.

As can be seen from FIG. 1, the image processing system 1000 to which the selective filtering according to the preferred embodiment of the present invention is applied includes a frequency converter 100, a filter 200, an inverse frequency transform 300, an image It includes a synthesis unit 400, an image information extraction unit 500, an image output unit 600, and a control unit 700.

Each component of the image processing system 1000 of the present invention may use at least part of a processor such as an MCU, a CPU, or a DSP.

The frequency converter 100 serves to output a frequency coefficient matrix Y by converting a frequency of an input image in units of a patch. The filter unit 200 includes at least one filter capable of filtering the input image in units of patches using the frequency coefficient matrix Y.

In addition, the frequency inverse transform unit 300 performs inverse frequency transform on the output of the filter unit 200 and outputs the image in a patch unit. The image synthesizing unit 400 changes the patch for the same pixel location, and performs the frequency conversion unit 100, the filter unit 200, and the frequency inverse transform unit 300 a plurality of times to obtain statistics of a plurality of output values output. It plays a role of synthesizing images using values.

In addition, the image information extracting unit 500 extracts information on quality of the input image in pixel units, using the frequency coefficient matrix Y.

The image output unit 600 serves to output an image.

The control unit 700 includes information extracted from the image information extraction unit 500; User input information; And an object recognizer for recognizing an object included in the image. Using at least one of them, it serves to generate a control signal for controlling the filter unit 200 and the image information extraction unit 500. For reference, information extracted from the image information extraction unit 500 by the control unit 700; User input information; And an object recognizer for recognizing an object included in an image; information from, may be used complementarily.

Hereinafter, each configuration of the image processing system 1000 of the present invention will be described in detail.

Frequency conversion unit 100 and frequency inverse conversion unit 300

The frequency converter 100 includes: a first transformation matrix; Frequency coefficient matrix Y; And a transpose matrix of the second transformation matrix; In addition, the frequency inverse transform unit 300 may include a transpose matrix of the first transform matrix; The output of the filter unit 200; And a second transformation matrix;

Specifically, the frequency converter 100 generates a frequency coefficient matrix Y having the same size as shown in [Equation 1] below through a frequency conversion process for the input image X of the M×N patch unit.

Here, D and C denote a first transformation matrix and a second transformation matrix for frequency space transformation, respectively. In addition, D and C have M×N and N×N sizes, respectively. As a frequency transformation method, general methods such as Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), and Discrete Fourier Transform (DFT) can be applied as they are.

을, 주파수 변환부(100)에서 사용한 동일한 주파수 공간 변환 행렬들을 사용하여 출력 영상 패치

로 다음의 [수학식 2]와 같이 역변환한다.The frequency inverse transform unit 300 is a frequency coefficient matrix passed through the filter unit 200 with respect to the input frequency coefficient matrix Y

The output image patch using the same frequency space transform matrices used in the frequency converter 100

Inversely transform as shown in [Equation 2] below.

Here, matrices D and C used for frequency transformation and inverse transformation are expressed as orthogonal matrices.

Filter part 200

The filter unit 200 includes: a noise removal filter unit 210 for removing noise by receiving a frequency coefficient matrix Y; A boundary line enhancement and signal amplification unit 220 for receiving the frequency coefficient matrix Y and enhancing the boundary line and amplifying the signal; And a blurring and masking filter unit 230 that receives the frequency coefficient matrix Y and performs blurring and masking. It may be configured to include at least one of.

In addition, the filter unit 200 is controlled so that at least two of the noise removal filter unit 210, the boundary line enhancement and signal amplification unit 220, and the blurring and masking filter unit 230 are implemented by the control unit 700. In this case, a multiple filter summing unit 240 that multiplies and sums the outputs of the noise removal filter unit 210, the boundary line enhancement and signal amplification unit 220, and the blurring and masking filter unit 230 by each weight is further added. It is preferable to include.

Noise removal filter unit 210

The noise removal filter unit 210 performs denoising by comparing the frequency coefficient matrix Y and the threshold matrix.

Specifically, the noise removal filter unit 210 includes a filter that reduces noise of an image and performs noise removal in a manner that reduces a difference with neighboring pixels in an image space, and a high-frequency component in which noise is mainly distributed in a frequency space. Perform in the direction of removing them.

The frequency coefficient matrix Y that has passed through the frequency converter 100 has a direct current (DC) value as an average value, and alternating current (AC) value coefficients corresponding to a low frequency to a high frequency. Since the noise generated in the camera does not have a certain rule and is generally expressed in the form of a Poisson distribution, they appear in the form of increasing the value of the high frequency component without a rule in the input image. Accordingly, the noise removal method can be expressed as the following [Equation 3] in a manner that appropriately takes the DC value and the low frequency components as the average value.

Here, f _d (·) is expressed as a threshold function as follows, and Q _d is a threshold matrix of size M×N. The value Q _d (i, j) of the threshold matrix is set larger as the high frequency component is, so that only a meaningful part of the image can be taken as shown in [Equation 4] below.

In the present invention, the threshold matrix Q _d may be configured to generate and set a control signal in the control unit 700 through user setting through prior knowledge of the image or noise level evaluation by the image information extraction unit 500. Accordingly, the user can perform noise removal by selecting an appropriate value according to the camera input image.

Boundary line enhancement and signal amplification unit 220

The boundary line enhancement and signal amplification unit 220 multiplies the boundary line enhancement filter (not shown) for enhancing the boundary line by performing a Hadamard product operation on the weight matrix and the frequency coefficient matrix Y and amplifies the signal by multiplying the signal amplification gain. Includes an amplifier (not shown). Boundary line enhancement and a weight matrix of the signal amplification unit 220 and a value of the signal amplification gain may be set by the control unit 700.

The sharpening filter is a filter that emphasizes the boundary of an image. In the image space, the resolution is improved by increasing the difference with the neighboring pixels by using information of neighboring pixels, but this adverse effect causes the noise reduction phenomenon that increases noise. Usually appear. Accordingly, a method of removing noise by appropriately using a noise removal filter in a trade-off relationship with the boundary line enhancement filter and enhancing the boundary line is mainly taken.

In the present invention, in the frequency coefficient matrix Y that has passed through the frequency converter 100, by taking a method of appropriately increasing the values of the low-frequency AC components, which are components that increase the contrast that mainly affect the human vision, noise is mainly Boundary line enhancement is performed as shown in [Equation 5] below to obtain an effect of enhancing the boundary line of an image without using distributed high-frequency components.

는 아다마르 곱을 나타낸다. Here, f _s (·) is a weight function expressed as [Equation 5], W _s is a weight matrix of size M×N, and operator

Represents the Hadamard product.

The signal amplifier amplifies the level of the image in order to increase visibility in the case of a low illuminance image shot in a low light environment, and is usually a method of amplifying an analog or digital signal from a sensor or in a camera processor. A digital signal amplification method or the like is used. While the digital signal amplification method is simple, it has the disadvantage of degrading the quality of the image by generating a step phenomenon, and the analog amplification method has the disadvantage that it is supported only by the sensor, but is difficult to implement in the digital signal processing camera processor .

In the present invention, by utilizing the above boundary line enhancement filter processing, it can be used as it is for signal amplification. When the value of the frequency coefficient matrix Y as a whole is increased, it serves to increase the level of the overall image. In this method, since the amplified signals enter the image synthesizing unit 400 to be described later and take M×N result values to generate the final result, it is possible to generate an image without stepping phenomenon, like an analog amplification method. There is an advantage.

Specifically, the signal amplifier performs signal amplification as shown in [Equation 6] below.

는 M×N 크기의 신호 증폭 및 가중치 행렬이다. Where w _g is the signal amplification gain,

Is an M×N-sized signal amplification and weighting matrix.

2 shows an exemplary diagram of the output of the noise removal filter unit 210 and the boundary line enhancement filter unit.

In FIG. 2, regions R ₁ , R ₂ , and R ₃ show the results of the noise removal filter unit 210, and R ₄ , R ₅ , and R ₆ show the results of the boundary line enhancement and the signal amplification unit 220. The boundary line enhancement of the regions R ₄ , R ₅ , and R ₆ and the performance of the signal amplification unit 220 are compared with the results of the noise removal filter unit 210 of the regions R ₁ , R ₂ , and R ₃ . And it can be confirmed that the visibility is improved by increasing the contrast difference.

In the present invention, as shown in FIG. 2, the threshold matrix Q _d can be set differently for a plurality of regions. The setting of the threshold matrix Q _d is also performed by the control unit 700.

Blur and mask filter unit 230

The blurring and masking filter unit 230 adjusts filtering processing intervals in vertical and horizontal directions, and performs blurring filtering by selecting only DC component values from the frequency coefficient matrix Y.

Specifically, the blurring filter can be used for reducing the size of data in an image compression codec or to make a user more interested in a necessary part by removing unnecessary information from an image. The masking filter is similar to this, but it is a function that makes it difficult to check in the video because it is related to privacy.

In the present invention, the masking function is implemented by modifying the blurring function by applying a frequency conversion process.

,

를 늘림으로써 생성할 수 있다. 이를 위해 영상 좌표(x,y)에 위치한 입력 패치 X의 경우, 흐릿화와 마스킹 필터링의 결과 Yb는 수행 간격의 배수에서만 주파수 계수 행렬 Y의 직류 성분 값을 갖게 된다. 참고로, 수직 및 수평 방향의 필터링 처리 간격은 제어부(700)에 의해 설정될 수 있다.The blurring filter generates a blurring effect by taking only the DC component value from the input frequency coefficient matrix Y, and the masking filter is a filtering process interval (stride) in the vertical and horizontal directions, respectively.

,

It can be created by increasing. To this end, in the case of the input patch X located at the image coordinate (x,y), as a result of the blurring and masking filtering, Yb has a DC component value of the frequency coefficient matrix Y only at a multiple of the execution interval. For reference, the vertical and horizontal filtering intervals may be set by the controller 700.

The output of the blurring and masking filter unit 230 can be expressed as the following [Equation 7].

3 is an exemplary diagram of a result of performing the blurring and masking filter unit 230.

로 설정한 흐릿화 수행 결과, R₃는

로 설정한 마스킹 수행 결과, R₄는

로 설정한 마스킹 수행 결과이다. 본 발명에서는, 제어부(700)에 의해 각 영역별로

,

를 다르게 설정할 수 있다.In Figure 3, R ₁ is a result of performing noise removal, R ₂ is

As a result of performing the blurring set to, R ₃ is

As a result of performing masking set to, R ₄ is

This is the result of performing the masking set to. In the present invention, by the control unit 700 for each area

,

Can be set differently.

Multiple filter summing unit 240

를 생성한다.The multiple filter summing unit 240 includes a plurality of filter units for the noise removal filter unit 210, the boundary line enhancement and signal amplification unit 220, the blurring and masking filter unit 230, and the like for the input frequency coefficient matrix Y. After processing the signal, collect the results obtained from each, and output frequency coefficient matrix by weighted summation as shown in [Equation 8] below.

Create

는 잡음 제거 필터부(210)를 거친 결과와 제 1 가중치를, Y_s 및

는 경계선 강화 및 신호 증폭부(220)를 거친 결과와 제 2 가중치를, Y_b 및

는 흐릿화 및 마스킹 필터부(230)를 거친 결과와 제 3 가중치를 각각 나타낸다. Where Y _d and

Is the result of passing through the noise removal filter unit 210 and the first weight, Y _s and

Is the boundary line enhancement and signal amplification unit 220 and the second weight, Y _b and

Denotes a result of passing through the blurring and masking filter unit 230 and a third weight, respectively.

In the present invention, the first weight, the second weight, and the third weight include information extracted from the image information extraction unit 500 according to the characteristics of the image and the purpose of the camera; User input information; And an object recognizer for recognizing an object included in the image. The control unit 700 may be set using at least one of the images, and may be set for each region of interest (ROI) or an entire image.

Image synthesizing unit 400

을 사용하여 출력 영상의 최종 결과값을 다음의 [수학식 9]와 같이 생성함으로써, 잡음 제거 성능을 더욱 향상하게 된다.The image synthesizing unit 400 passes through the frequency converter 100, the filter unit 200, and the frequency inverse transform unit 300 M×N times, and the maximum M×N processing results obtained at the pixel position of the same output in the image

By generating the final result value of the output image as shown in [Equation 9] below, noise removal performance is further improved.

Here, w _k is information extracted from the image information extraction unit 500; User input information; And an object recognizer for recognizing an object included in the image. By using at least one of them, it can be defined in various ways by the control unit 700.

으로 표현할 수 있고, 출력 패치의 통계를 활용하여 각 출력 패치의 분산

에 반비례하도록

로 표현하여, 출력 영상의 변화가 적도록 생성할 수도 있으며, 비슷한 결과를 얻기 위한 좀더 간단한 방법으로, 역변환에 사용되는 주파수 성분

의 개수에 반비례하도록 구현할 수도 있다. When taking the average value

Can be expressed as, and the distribution of each output patch by utilizing the statistics of the output patch

In inverse proportion to

By expressing as, it can be generated so that the change in the output image is small, and as a simpler method to obtain similar results, the frequency component used for inverse transformation

It can also be implemented in inverse proportion to the number of.

Image information extraction unit 500

The image information extracting unit 500 receives a frequency coefficient matrix Y, an image noise level evaluation unit 510 for evaluating the noise level of an input image, and an image for receiving the frequency coefficient matrix Y, and evaluates the sharpness of the input image. It is configured to include a sharpness evaluation unit 520.

Image noise level evaluation unit 510

The image noise level evaluation unit 510 includes: a first noise level evaluator (not shown) for evaluating a first level noise level using energy of high-frequency components of an input image; And a second-order noise level evaluator (not shown) for evaluating a second-order noise level by using a difference between the input image and the noise-removed image.

If the noise level of the image can be evaluated, it can be mechanically used as an important factor in determining the aperture aperture of the lens and the exposure time of the sensor. During the image processing process, the parameters of the noise removal filter of the input image are used. Can be used to determine.

In the present invention, in the frequency coefficient matrix Y passed through the frequency converter 100, the primary noise level E ₁ is evaluated by using the energy of the high-frequency AC components mainly distributed by the primary noise level evaluator, and the secondary noise The second-order noise level E ₂ is evaluated by taking the difference between the input image and the image that has undergone noise removal by a degree evaluator. The noise level obtained through this is reflected in the threshold matrix Q _d of the noise removal filter unit 210 by the control unit 700, and when the noise is high, the Q _d value is increased, and when the noise is small, the Q _d value is decreased. , It is used to remove noise appropriately according to the input image.

The following [Equation 10] shows an equation for evaluating the first-order noise level E ₁ .

Here, E ₁ represents the first-order noise level at the image coordinate (x,y) and is expressed as an energy average value of high-frequency components, and W _N is a noise frequency selection matrix of size M×N, and high frequency components are Is set to choose.

As can be seen from [Equation 10], the first-order noise level evaluator displays the input image as an average value of energy of high-frequency components for each pixel. Specifically, the square value of the value of the frequency coefficient matrix for each pixel is multiplied by the noise frequency selection matrix, and the average value of the maximum M×N processing results obtained at the position of the same output pixel in the image is taken.

The following [Equation 11] shows an equation for evaluating the second-order noise level E ₁₂ .

Here, E ₂ denotes a second-order noise degree at the image coordinate (x,y), and is expressed as an average value of the difference between the input image and the image passed through the noise removal filter unit 210.

That is, as can be seen from [Equation 11], the second-order noise level evaluator performs frequency inverse conversion after filtering by the noise removal filter unit 210 and the corresponding pixel before frequency conversion for each pixel. After calculating the difference value with the pixel, it is squared, and the average value of the maximum M×N processing results obtained at the position of the same output pixel in the image is taken.

Image sharpness evaluation unit 520

The image sharpness evaluation unit 520 evaluates the sharpness of the input image by using a product of the energy of the frequency coefficient matrix Y and an intermediate frequency selection and weight matrix taking an intermediate frequency band of the frequency coefficient matrix Y.

The sharpness and blurness of the image can be used as a reference for controlling the camera focus or evaluating the resolution of the image. The image-based focus measure method is mainly used to obtain the amount of boundary lines using a differential filter in an image space.

In the present invention, focus evaluation is performed using information of an intermediate frequency band in a frequency space corresponding to a differential filter of an image space. That is, the image sharpness evaluation unit 520 collects an intermediate frequency selection taking an intermediate frequency band of the frequency coefficient matrix Y passed through the frequency conversion unit 100 and the product of the energy of the weight matrix W _f and the frequency coefficient matrix Y, Sharpness evaluation is performed as in [Equation 12].

Here, S represents the sharpness at the image coordinate (x,y), and is expressed as a weighted sum of the intermediate frequency band.

That is, as can be seen from [Equation 12], the image sharpness evaluation unit 520 multiplies the square value of the value of the frequency coefficient matrix for each pixel by the intermediate frequency selection and weight matrix, and the position of the same output pixel in the image The average value of the maximum M×N processing results obtained in is taken.

4 is an exemplary diagram of a result of estimating sharpness by the image sharpness evaluation unit 520.

In FIG. 4, the region of interest R ₁ is a red color superimposed on the original image, and the region of interest R ₂ indicates only the sharpness. Information extracted from the image information extraction unit 500; User input information; And an object recognizer for recognizing an object included in the image. Using at least one of them, the control unit 700 may be configured to set a display method for each of a plurality of regions of interest.

For example, a plurality of regions of interest may be set to be evenly distributed over the entire image, or may be set to be limited to some regions.

Video output unit 600

The image output unit 600 includes an input image; A composite image of the image synthesis unit 400; And user setting screen; It serves to output at least one of them or to display an image using at least one of them on a display. With respect to the image information output to the image output unit 600, a control signal of the controller 700 may be generated by a user inputting setting information.

Control unit 700

The control unit 700 includes at least one of the noise removal filter unit 210, the boundary line enhancement and signal amplification unit 220, and the blurring and masking filter unit 230 according to region information set for the input image of one frame. It is possible to generate a control signal for controlling to execute or not to perform any one. When two or more of the noise removal filter unit 210, the boundary line enhancement and signal amplification unit 220, and the blurring and masking filter unit 230 are performed, each weight may be set by the control unit 700, The multiple filter summing unit 240 may use the set weight for summing.

In addition, the controller 700 may generate a control signal for controlling the filter unit 200 by using information extracted from the image information extraction unit 500 by artificial intelligence.

In addition, the controller 700 may display at least a partial area in a specific color as shown in FIG. 4 according to the image sharpness value evaluated by the image sharpness evaluation unit 520; or, display a bitmap or a black and white grayscale image; Thus, the image output unit 600 may be controlled to output an image. That is, the image output unit 600 emphasizes the corresponding part by the color of the image according to the image sharpness value evaluated by the image sharpness evaluation unit 520 on the synthesized image of the image synthesis unit 400, or a bitmap or It can be expressed as a black and white grayscale image and output in a format that displays sharpness. For example, when a corresponding part is emphasized by color, when the sharpness value is greater than or equal to a predetermined value, the color may be displayed in red as shown in FIG. In addition, in the case of displaying a bitmap, when the sharpness value is greater than or equal to a predetermined value, it may be displayed in white, and the rest may be displayed in black.

The control unit 700 can control the image output unit 600 to output by adding a plurality of grids at regular intervals to the entire input image, and image noise level by using user input information for selecting a plurality of grids. An area to be evaluated by the evaluation unit 510 or the image sharpness evaluation unit 520 may be set.

In addition, the control unit 700 may control the image output unit 600 to output an input image, and the image noise level evaluation unit 510 using user input information for setting an ROI of an arbitrary size among the input images. ) Or, an area to be evaluated by the image sharpness evaluation unit 520 may be set. In addition, the control unit 700 may control the input image to be output by the image output unit 600, and by using user input information for setting a region of interest of an arbitrary size among the input images, the noise removal filter unit 210 ), the boundary line enhancement and signal amplification unit 220 and the blurring and masking filter unit 230 may be selected or a region to be implemented may be set.

Preferably, the control unit 700 includes a threshold matrix, a weight matrix, and a signal amplification for a selected one of the noise removal filter unit 210, the boundary line enhancement and signal amplification unit 220, and the blurring and masking filter unit 230. The image output unit 600 may be controlled to output a parameter setting screen so that a user may set parameters including a gain and a filtering interval in the vertical and horizontal directions.

Application field

Based on the information of the image noise level evaluation unit 510 and the image sharpness evaluation unit 520 and information from a recognition engine such as an automatic object recognizer, the control unit 700 includes the entire image within one filtering framework. Alternatively, different filtering effects can be applied to some areas.

The image sharpness evaluation unit 520 provides the focus positions in the image and the sharpness information of each position to the control unit 700, so that the user can control the camera lens by selecting one of several focus positions. .

The image noise evaluation unit 510 may control a value of the threshold matrix of the noise removal filter by providing noise information from the image to the controller 700. In addition, when storing an image, the threshold of the noise removal filter may be adjusted according to the storage size set by the user.

5 shows an example of application of the image processing system 1000 of the present invention.

Specifically, FIG. 5 shows an example of being used in combination with a recognizer, which is a recognition engine. In the recognizer, object recognition information such as a vehicle, a license plate, a human face, and a tree is input to the controller 700. For the entire area indicated as R ₈ for image quality enhancement applied to a noise removal filter, apply the fuzzy screen filter for unnecessary parts of non-interest region, such as R ₇ in order to reduce the storage capacity of the image, it displayed on the R ₁ If you need to check detailed information, such as a license plate, you can apply a border enhancement filter. In addition, when a human face such as R ₃ to R ₆ is detected, a masking function may be applied so that an image is not stored for privacy protection.

Unlike most camera processors that must implement at least one filter individually, in the present invention, filtering processes by separate filters can be simplified into one filtering framework, thereby increasing the efficiency of camera system configuration and control. .

Hereinafter, an image processing method to which selective filtering is applied according to an exemplary embodiment of the present invention will be described. Since the image processing method of the present invention uses the image processing system 1000 described above, it goes without saying that all the features of the image processing system 1000 are included even if there is no separate description.

In addition, the image processing method of the present invention may be implemented in the form of a computer program executed by a processor.

The image processing method of the present invention includes a frequency conversion step; Filtering step; Frequency inverse transform step; Image synthesis step; Extracting image information; An image output step; And a control step.

In the frequency conversion step, a frequency coefficient matrix Y is output by converting the frequency of the input image in units of patches. Specifically, the frequency conversion step includes: a first conversion matrix; Frequency coefficient matrix Y; And a transpose matrix of the second transformation matrix;

In the filtering step, the input image is filtered in units of patches by selectively applying at least one filter using the frequency coefficient matrix Y. In addition, in the control step, a control signal for controlling the filtering step is generated.

Specifically, the filtering may include a noise removal filtering step of receiving a frequency coefficient matrix Y and removing noise; A boundary line enhancement and signal amplification step of receiving a frequency coefficient matrix Y to enhance a boundary line and amplify a signal; And a blurring and masking filtering step of receiving the frequency coefficient matrix Y and performing blurring and masking. It is preferable to include at least one of.

In addition, the filtering step may include a noise removal filter step by a control step; Border line enhancement and signal amplification steps; And a blurring and masking filter step. When two or more of the above are performed, a multi-filter summing step of multiplying and summing outputs performed among the noise removal filter step, the boundary line enhancement and signal amplification step, and the blurring and masking filter steps by respective weights.

The noise removal filtering step removes noise by comparing the frequency coefficient matrix Y and the threshold matrix. In the control step, a value of the threshold matrix of the noise removal filter step may be set. In addition, the step of strengthening the boundary line and amplifying the signal is characterized in that the weight matrix and the frequency coefficient matrix Y are subjected to a Hadamard product operation to strengthen the boundary line, and multiply the signal amplification gain to amplify the boundary line. In the control step, a value of a weight matrix and a value of a signal amplification gain in the boundary line enhancement and signal amplification steps may be set. Further, in the blurring and masking filtering step, the filtering process interval is adjusted in the vertical and horizontal directions, and blur filtering is performed by selecting only DC component values from the frequency coefficient matrix Y. In the control step, a filtering process interval of the blurring and masking filter steps may be set.

In addition, in the frequency inverse transform step, frequency inverse transform is performed on the output of the filtering step, and the image is output in a patch unit. Specifically, the inverse frequency transformation step includes: a transpose matrix of the first transformation matrix; The output of the filtering step; And a second transformation matrix.

In the image synthesizing step, the patch is changed for the same pixel position, and the frequency conversion step, the filtering step, and the frequency inverse transform step are performed a plurality of times, and an image is synthesized using statistical values of a plurality of output values.

In the image information extraction step, quality information is extracted from the input image in pixel units using the frequency coefficient matrix Y.

Specifically, the step of extracting image information includes: an image noise level evaluation step of receiving a frequency coefficient matrix Y and evaluating a noise level of the input image; And an image sharpness evaluation step of receiving the frequency coefficient matrix Y and evaluating the sharpness of the input image.

The image noise level evaluation step includes: a first order noise level evaluation step of evaluating a first order noise level using energy of high-frequency components of the input image; And a second-order noise level evaluation step of evaluating a second-order noise level by using a difference between the input image and the noise-removed image.

In the image sharpness evaluation step, the energy of the frequency coefficient matrix Y; and an intermediate frequency selection and weight matrix taking an intermediate frequency band of the frequency coefficient matrix Y are used to evaluate the sharpness of the input image.

The image output step serves to output an image.

The image output step may include an input image; A composite image in the image synthesis step; And user setting screen; It serves to output at least one of them or to display an image using at least one of them on a display. With respect to the image information output in the image output step, a control signal may be generated in the control step by the user inputting setting information.

The control step may include information extracted from the image information extraction step; User input information; And an object recognizer for recognizing an object included in the image. By using at least one of, a control signal for controlling the filtering step is generated. Noise removal filter step; Strengthening the boundary line and amplifying the signal; And a blurring and masking filter step. When two or more of them are performed, each weight can be set.

For example, the controlling step may include a noise removal filtering step; Border line enhancement and signal amplification steps; And blurring and masking filtering. A control signal for controlling at least one of or not to be performed is generated.

In addition, the control step may include displaying at least a partial area in a specific color according to the image sharpness value evaluated by the image sharpness evaluation step; or, displaying a bitmap or a black and white grayscale image; so that the image output step outputs an image. Can be controlled. In addition, in the control step, a plurality of grid types having a predetermined interval may be added to the input image and controlled to be output in the image output step, and the image noise level evaluation step or the image sharpness evaluation step using user input information for selecting a grid. You can set the area to be evaluated by.

Preferably, the control step may control the input image to be output by the image output step, and an image noise level evaluation step or an image sharpness evaluation step using user input information for setting a region of interest of an arbitrary size among the input images. You can set the area to be evaluated by.

In addition, the controlling step may include controlling the input image to be output by the image output step, and using the user input information of the input image, a noise removal filter step; Border line enhancement and signal amplification steps; And a blurring and masking filter step. You can select one to be implemented or set an area to be implemented. In addition, the control step may include a noise removal filter step; Border line enhancement and signal amplification steps; And a blurring and masking filter step. For the selected one, the image output step may control to output a parameter setting screen so that a user can set parameters including a threshold matrix, a weight matrix, a signal amplification gain, and a filtering processing interval in the vertical and horizontal directions.

According to the image processing system 1000 to which the selective filtering of the present invention is applied and the image processing method thereof, a plurality of filters having various different effects can be selected according to necessity in the image area, so that the filters are mutually traded off. It can be seen that contradictory relationships can be resolved.

1000: image processing system
100: frequency conversion unit 200: filter unit
300: frequency inverse transform unit 400: image synthesis unit
500: image information extraction unit 600: image output unit
700: control unit
210: noise removal filter unit 220: boundary line enhancement and signal amplification unit
230: blurring and masking filter unit 240: multiple filter summing unit
510: image noise evaluation unit 520: image sharpness evaluation unit

Claims (48)

In the image processing system,
A frequency converter configured to convert an input image into a frequency in a patch unit and output a frequency coefficient matrix;
A filter unit including at least one filter capable of filtering the input image in a patch unit using the frequency coefficient matrix;
A control unit generating a control signal for controlling the filter unit; And
Including, by using the frequency coefficient matrix, an image information extracting unit for extracting information on the quality of the input image pixel unit,
The image information extraction unit,
Including; an image noise degree evaluation unit for evaluating the noise degree of the input image using the frequency coefficient matrix,
The image noise degree evaluation unit,
The image processing system according to claim 1, wherein the first-order noise level is evaluated using the energy of the high-frequency components of the input image, and the second-order noise level is evaluated using a difference between the input image and a noise-removed image. The method of claim 1,
The filter unit,
A noise removal filter to remove noise by using the frequency coefficient matrix;
A boundary line enhancement and signal amplification unit for enhancing a boundary line and amplifying a signal using the frequency coefficient matrix; And
A blurring and masking filter unit that performs blurring and masking using the frequency coefficient matrix; Image processing system comprising at least one of. The method of claim 2,
The control unit,
The noise removal filter unit; The boundary line enhancement and signal amplification unit; And the blurring and masking filter unit. Generating a control signal for controlling at least one of or not to perform at least one of, and the noise removal filter unit; The boundary line enhancement and signal amplification unit; And the blurring and masking filter unit. When two or more of them are performed, each weight is set. The method of claim 3,
The filter unit,
The noise removal filter unit by the control unit; The boundary line enhancement and signal amplification unit; And the blurring and masking filter unit. When two or more of them are implemented, the noise removal filter unit; The boundary line enhancement and signal amplification unit; And the blurring and masking filter unit. And a multi-filter summing unit for multiplying and summing the outputs performed in the output by each weight. The method of claim 2,
The noise removal filter unit,
And removing noise by comparing the frequency coefficient matrix and the threshold matrix. The method of claim 5,
The control unit,
The image processing system, characterized in that it is possible to set a value of the threshold matrix of the noise removal filter unit. The method of claim 2,
The boundary line enhancement and signal amplification unit,
An image processing system characterized in that the weight matrix and the frequency coefficient matrix are subjected to a Hadamard product operation to enhance a boundary line, and amplify by multiplying a signal amplification gain. The method of claim 7,
The control unit,
And setting a value of a weight matrix of the boundary line enhancement and signal amplification unit and a value of a signal amplification gain. The method of claim 2,
The blurring and masking filter unit,
An image processing system, comprising: adjusting filtering intervals in vertical and horizontal directions, and performing blur filtering by selecting only DC component values from the frequency coefficient matrix. The method of claim 9,
The control unit,
The image processing system according to claim 1, wherein the blurring and masking filter unit is capable of setting filtering processing intervals in the vertical and horizontal directions. delete delete delete The method of claim 2,
The image information extraction unit,
An image processing system comprising: an image sharpness evaluation unit for evaluating the sharpness of the input image by using the frequency coefficient matrix. The method of claim 14,
The image sharpness evaluation unit,
And a product of the energy of the frequency coefficient matrix; and an intermediate frequency selection and weighting matrix taking an intermediate frequency band of the frequency coefficient matrix to evaluate the sharpness of the input image. The method of claim 14,
The image processing system,
And a frequency inverse transform unit configured to perform inverse frequency transformation on the output of the filter unit and output an image as an image. The method of claim 16,
The frequency conversion unit,
A first transformation matrix; The frequency coefficient matrix; And a transpose matrix of the second transformation matrix; multiply and output,
The frequency inverse transform unit,
A transpose matrix of the first transformation matrix; An output of the filter unit; And multiplying and outputting the second transformation matrix. The method of claim 16,
The image processing system,
An image synthesizing unit that changes a patch for the same pixel location and synthesizes an image using statistical values of a plurality of output values output by performing the frequency conversion unit, the filter unit, and the frequency inverse transform unit a plurality of times; An image processing system, characterized in that. The method of claim 18,
The image processing system,
The input image; A composite image of the image synthesis unit; And user setting screen; An image processing system comprising: an image output unit that outputs at least one of or outputs an image using at least one. The method of claim 19,
The control unit,
For at least some image areas according to the image sharpness value evaluated by the image sharpness evaluation unit,
Marked in a specific color; or,
Displaying as a bitmap or a black and white grayscale image; and controlling the image output unit to output an image. The method of claim 19,
The control unit,
It is possible to control the image output unit to output by adding a plurality of grid types at regular intervals to the input image, and evaluate by the image noise level evaluation unit or the image sharpness evaluation unit using user input information for selecting a grid. An image processing system, characterized in that it is possible to set an area to be used. The method of claim 19,
The control unit,
The input image may be controlled to be output by the image output unit, and may be evaluated by the image noise level evaluation unit or the image sharpness evaluation unit using user input information for setting a region of interest of an arbitrary size among the input images. An image processing system, characterized in that the area can be set. The method of claim 19,
The control unit,
A noise removal filter unit capable of controlling the input image to be output by the image output unit, and using user input information of the input image; The boundary line enhancement and signal amplification unit; And the blurring and masking filter unit. An image processing system, characterized in that it is possible to select one to be performed or set a region to be performed. The method of claim 23,
The control unit,
The noise removal filter unit; The boundary line enhancement and signal amplification unit; And the blurring and masking filter unit. The image processing system, characterized in that the image processing system is capable of controlling the image output unit to output a parameter setting screen so that a user can set a parameter for the selected one. In the image processing method,
A frequency conversion step of outputting a frequency coefficient matrix by converting a frequency of the input image in units of patches;
Filtering the input image in units of patches by selectively applying at least one filter using the frequency coefficient matrix;
A control step of generating a control signal for controlling the filtering step; And
Including, by using the frequency coefficient matrix, extracting the information on the quality of the input image pixel unit;
The image information extraction step,
An image noise degree evaluation step of evaluating a noise degree of the input image using the frequency coefficient matrix; Including,
The image noise level evaluation step,
And evaluating a first-order noise level using the energy of the high-frequency components of the input image, and evaluating a second-order noise level using a difference between the input image and a noise-removed image. The method of claim 25,
The filtering step,
A noise removal filter for removing noise by using the frequency coefficient matrix;
A boundary line enhancement and signal amplification step of enhancing a boundary line and amplifying a signal using the frequency coefficient matrix; And
A blurring and masking filter step of performing blurring and masking using the frequency coefficient matrix; Image processing method comprising at least one of. The method of claim 26,
The control step,
The noise removal filter step; Strengthening the boundary line and amplifying the signal; And the blurring and masking filter step. Generating a control signal for controlling to perform at least one or not to perform at least one of, and the noise removal filter step; Strengthening the boundary line and amplifying the signal; And the blurring and masking filter step. When two or more of them are performed, each weight is set. The method of claim 27,
The filtering step,
The noise removal filter step by the control step; Strengthening the boundary line and amplifying the signal; And the blurring and masking filter step. When two or more of them are performed, the noise removal filter step; Strengthening the boundary line and amplifying the signal; And the blurring and masking filter step. And a multi-filter summing step of multiplying and summing the output performed during the multiplying by each weight. The method of claim 26,
The noise removal filter step,
And removing noise by comparing the frequency coefficient matrix and the threshold matrix. The method of claim 29,
The control step,
And setting a value of a threshold matrix of the noise removal filter step. The method of claim 26,
The boundary line enhancement and signal amplification step,
An image processing method, comprising: performing a Hadamard product operation on a weight matrix and the frequency coefficient matrix to enhance a boundary line, and multiplying a signal amplification gain to amplify it. The method of claim 31,
The control step,
And setting a value of a weight matrix and a signal amplification gain in the boundary line enhancement and signal amplification steps. The method of claim 26,
The blurring and masking filter step,
The image processing method according to claim 1, wherein filtering intervals are adjusted in vertical and horizontal directions, and blur filtering is performed by selecting only DC component values from the frequency coefficient matrix. The method of claim 33,
The control step,
The image processing method, characterized in that it is possible to set filtering processing intervals in the vertical and horizontal directions of the blurring and masking filtering step. delete delete delete The method of claim 26,
The image information extraction step,
And an image sharpness evaluation step of evaluating the sharpness of the input image by using the frequency coefficient matrix. The method of claim 38,
The image clarity evaluation step,
The image processing method, characterized in that the clarity of the input image is evaluated by using a product of the energy of the frequency coefficient matrix; and an intermediate frequency selection and weight matrix taking an intermediate frequency band of the frequency coefficient matrix. The method of claim 38,
The image processing method,
And performing inverse frequency transformation on the output of the filtering step and outputting an image as an image. The method of claim 40,
The frequency conversion step,
A first transformation matrix; The frequency coefficient matrix; And a transpose matrix of the second transformation matrix; multiply and output,
The frequency inverse transform step,
A transpose matrix of the first transformation matrix; The output of the filtering step; And multiplying and outputting the second transformation matrix. The method of claim 40,
The image processing method,
An image synthesizing step of synthesizing an image using statistical values of a plurality of output values output by performing the frequency conversion step, the filtering step, and the frequency inverse transform step a plurality of times while changing a patch for the same pixel position; Image processing method comprising a. The method of claim 42,
The image processing method,
The input image; A composite image of the image synthesis step; And user setting screen; An image processing method comprising: an image output step of outputting at least one of or outputting an image using at least one. The method of claim 43,
The control step,
For at least some image areas according to the image sharpness value evaluated by the image sharpness evaluation step,
Marked in a specific color; or,
Displaying as a bitmap or black-and-white grayscale image; and controlling the image output step to output an image. The method of claim 43,
The control step,
The image output step can be controlled to output by adding a plurality of grid shapes at a predetermined interval to the input image, and by using the user input information for selecting a grid, the image noise level evaluation step or the image sharpness evaluation step An image processing method, characterized in that it is possible to set an area to be evaluated. The method of claim 43,
The control step,
The input image can be controlled to be output by the image output step, and evaluated by the image noise level evaluation step or the image sharpness evaluation step using user input information for setting a region of interest of an arbitrary size among the input images An image processing method, characterized in that it is possible to set an area to be used. The method of claim 43,
The control step,
A noise removal filter step capable of controlling the input image to be output by the image output step, and using user input information of the input image; Strengthening the boundary line and amplifying the signal; And the blurring and masking filter step. An image processing method characterized in that it is possible to select one to be performed or set an area to be performed. The method of claim 47,
The control step,
The noise removal filter step; Strengthening the boundary line and amplifying the signal; And the blurring and masking filter step. An image processing method, characterized in that the image output step can control to output a parameter setting screen so that a user can set a parameter for the selected one.

Images