KR102663964B1 - Entropy filter implementation method and hardware device for implementing the same - Google Patents

Abstract

The present invention relates to a method of implementing an entropy filter for real-time processing of an entropy filter, which is widely used in the image processing field, and a hardware device for implementing the same. More specifically, it relates to an entropy filter (entropy filter) that is widely used in the field of image processing. This relates to a method of implementing an entropy filter to calculate entropy using pixel alignment and differences between adjacent pixels by implementing a hardware device differently from existing methods and a hardware device for implementing the same.

Description

Entropy filter implementation method and hardware device for implementing the same}

The present invention relates to a method of implementing an entropy filter for real-time processing of an entropy filter, which is widely used in the image processing field, and a hardware device for implementing the same. More specifically, it relates to an entropy filter (entropy filter) that is widely used in the field of image processing. This relates to a method of implementing an entropy filter to calculate entropy using pixel alignment and differences between adjacent pixels by implementing a hardware device differently from existing methods and a hardware device for implementing the same.

In the field of image processing, entropy is a value that measures the average randomness of an image from the normalized histogram of the image. These entropy values are used in various image processing algorithms such as detail emphasis and image separation. In addition, real-time processing of entropy filters is becoming very important these days, as heterogeneous computing systems are popular as well as being used in systems that include cutting-edge technologies such as intelligent surveillance cameras or self-driving cars.

However, to date, there is only a software implementation of the entropy filter and no hardware implementation yet exists.

in 2008 Patent Document 1, a registered US patent, introduced the concept of an entropy filter and showed an example of smoothing an image using an entropy filter. Patent Document 2, a U.S. published patent published in 2012, proposed an image separation method based on an entropy filter, which was helpful in object recognition.

In addition, Patent Document 3, a U.S. registered patent registered in 2014, introduced a method of emphasizing detailed information in an image using an entropy filter, and Patent Document 4, a Chinese public patent published in 2015, introduced an entropy filter. An algorithm was proposed to make small objects in noisy infrared images appear clearer using an entropy filter. What patent documents 1 to 4 introduced above have in common is that they use a software implementation of an entropy filter in a homogeneous computing system. In homogeneous computing systems, it is difficult to utilize the latest technologies in advanced systems such as intelligent surveillance cameras or self-driving cars.

Considering the hardware implementation of an entropy filter, there are several difficulties. First, to generate a normalized histogram of an image, the methods developed to date generate a normalized histogram according to the size of the image data.

For example, assuming 8-bit image data, there are 256 pixel intensity levels, so 256 registers or 256-sized memory must be used. In addition, because the number of occurrences of the pixel intensity level must be counted, the entropy information of a video frame is not the actual entropy information of that frame, but the entropy information of the previous frame. This is not a big problem, but it may reduce the accuracy of the system somewhat.

Second, calculating entropy requires accumulating sum. However, considering the above normalized histogram generation method, the cumulative sum operation is inefficient. This is because, even considering the basic 8-bit video data, 255 additions must be used. As the size of the image data increases, the number of additions increases rapidly.

Due to the above two problems, it is impossible to implement an entropy filter in hardware using the above method.

In addition, Non-Patent Document 1 proposes a hardware structure for calculating a cumulative histogram. Although an entropy filter can be implemented in hardware using this structure, the entire histogram can be calculated depending on the size of the input data. must be calculated. As a result, there is a problem in that the size of the hardware increases significantly when calculating entropy, and it is difficult to implement an entropy filter in hardware because it uses a structure that calculates a cumulative histogram.

US Patent Publication US7460702B2 (registered on December 2, 2008) Entropy filter, and area extracting method using the filter Entropy based image separation US20120075440A1 (published March 29, 2012) US Patent Publication US8655096B2 (registered on February 18, 2014) Automatic image sharpening using entropy-based blur radius Chinese Patent Publication CN104268844A (published on 2015.01.07) Small target infrared image processing method based on weighting local image entropy Korean Patent Publication KR10-2039653B1 (2019.11.01 notice) Edge-preserving smoothing value detection method applying biter sorting algorithm and comparison switching system

Novel FPGA-based implementation of median and weighted median filters for image processing, Suhaib A., Fahmy, Peter Y. K. Cheung, Wayne Luk, International Conference on Field Programmable Logic and Applications (2005) 142-147

The present invention is intended to solve the above problems, by introducing an efficient entropy calculation method to implement an entropy filter in hardware, a method of implementing an entropy filter for a hardware structure accordingly, and a hardware device for implementing the same. It is intended to provide.

In addition, the present invention is intended to provide a method of implementing an entropy filter that utilizes differences between adjacent pixels by sorting pixels in a filtering window to generate a normalized histogram, unlike the basic method, and a hardware device for implementing the same. .

In addition, the newly proposed entropy calculation method is generalized regardless of the size of the filtering window, and the purpose is to provide a method of implementing an entropy filter and a hardware device for implementing it so that the corresponding hardware structure has features that can be reconfigured according to the window size. .

In addition, the proposed entropy calculation method has a simple pipeline structure, and the purpose of the present invention is to provide a method of implementing an entropy filter to enable real-time processing and a hardware device for implementing the same.

However, the objects of the present invention are not limited to the objects mentioned above, and other objects not mentioned will be clearly understood by those skilled in the art from the description below.

To achieve the above object, a method of implementing an entropy filter according to an embodiment of the present invention includes a first step of extracting pixels in a window using a line memory and a buffer register according to the size of the filtering window; A second step of sorting the extracted pixels using a Batcher's sorting network; a third step of calculating adjacent differences of aligned pixels; A fourth step of performing a reduction OR step to efficiently express adjacent differences on the difference value calculated in the third step; and a fifth step of performing a pipelined entropy calculation step; It is characterized by including.

In the first step, when input image data is input together with timing signals (horizontal active video and vertical active video) to extract pixels in the window, it is assumed that the size of the filtering window is N-by-N, and all of the pixels in the window are It is characterized by first using (N-1) line memories 101 to extract (N×N) (N is an odd number of natural numbers greater than 1 and is 3 to 15) pixels.

And in order to achieve the above object, a hardware device for implementing an entropy filter according to another embodiment of the present invention extracts pixels in the window using a line memory and a buffer register according to the size of the filtering window. An extraction module 100 that performs the steps of: A sorting module 200 that sorts the extracted pixels using a Batcher's sorting network; a difference calculation module 300 that performs the step of calculating adjacent differences of aligned pixels; an adjacent difference expression module 400 that performs a reduction OR step to efficiently express the adjacent difference with respect to the difference value calculated in the difference calculation module 300; and an entropy calculation module 500 that performs a pipelined entropy calculation step; It is characterized by including.

The method of implementing an entropy filter and the hardware device for implementing the same according to an embodiment of the present invention can be efficiently designed with hardware of an entropy filter, thereby saving hardware resources and achieving fast processing speed, In line with this, it provides the effect of enabling image processing algorithms that frequently use the latest filtering technologies to be implemented in hardware.

In addition, the method of implementing an entropy filter and the hardware device for implementing the same according to another embodiment of the present invention provide the possibility of reconfiguration of the hardware structure because the hardware structure of the proposed entropy filter is generalized regardless of the filtering window size. (reconfigurability) provides the effect of having a feature.

1 is a diagram showing a method of implementing an entropy filter using components of a hardware device 1 for implementing an entropy filter according to an embodiment of the present invention.
Figure 2 is a diagram showing a specific example of a method of implementing an entropy filter according to an embodiment of the present invention.
Figure 3 is a diagram showing the necessity of hardware implementation of an entropy filter according to an embodiment of the present invention.
FIG. 4 is a diagram showing a process performed by the extraction module 110 corresponding to the first step of FIG. 1 according to an embodiment of the present invention.
Figure 5 is a diagram showing the Batcher's sorting network (registration number: 10-2039653) of Patent Document 5, for which the present applicant has received domestic patent registration.
FIG. 6 is a diagram for explaining the entropy calculation method in FIG. 5.
FIG. 7 is a diagram illustrating adjacent difference calculation by receiving outputs of a Batcher's sorting network by the difference calculation module 300 according to an embodiment of the present invention.
FIG. 8 is a diagram showing how the adjacent difference expression module 400 according to an embodiment of the present invention receives the difference between adjacent pixels and passes it through a reduction OR gate to generate a 1-bit signal for determining the presence or absence of a difference. am.
Figure 9 is a diagram showing the final step of pipelined entropy calculation by the entropy calculation module 500 according to an embodiment of the present invention.
Figure 10 is a diagram showing the hardware structure of the final stage of pipelined entropy calculation according to an embodiment of the present invention.

Hereinafter, a detailed description of preferred embodiments of the present invention will be described with reference to the attached drawings. In the following description of the present invention, if a detailed description of a related known function or configuration is judged to unnecessarily obscure the gist of the present invention, the detailed description will be omitted.

In this specification, when one component 'transmits' data or a signal to another component, the component may transmit the data or signal directly to the other component, and may transmit the data or signal through at least one other component. This means that data or signals can be transmitted to other components.

1 is a diagram illustrating a method of implementing an entropy filter using components of a hardware device 1 for implementing an entropy filter according to an embodiment of the present invention. Referring to FIG. 1, the hardware device 1 for implementing an entropy filter, which is a hardware device of an entropy filter according to the present invention, includes a line memory and a buffer register according to the filtering window size. ), an extraction module 100 that performs the step of extracting pixels in the window, a sorting module 200 that performs the step of sorting the extracted pixels using a Batcher's sorting network, and the adjacent differences of the sorted pixels. A difference calculation module 300 that performs a calculation step, an adjacent difference expression module 400 that performs a reduction OR step to efficiently express adjacent differences, and an entropy calculation module that performs a pipelined entropy calculation step. It consists of (500).

The function of the present invention is to design the first hardware structure of an entropy filter, which is widely used in the image processing field. In order to compensate for the problems still faced with the hardware implementation of entropy filters, we propose a new entropy calculation method and introduce an efficient hardware structure accordingly. The newly proposed method relies on pixel alignment and differences between adjacent pixels. The hardware structure of the entropy filter of the present invention enables real-time processing, which is of great help to complex image processing algorithms belonging to smart surveillance cameras or autonomous vehicles.

[Example]

For example, we would like to design the hardware structure of a 5×5 entropy filter. Since the size of the window is (N=5), 4 line memories corresponding to N-1 and 20 buffer registers are needed to extract 25 pixels in the window according to Figure 4. .

12 22 0 4 9 0 200 0 144 9 One 7 25 91 51 69 136 87 15 100 77 13 8 2 251

First, [Table 1] relates to pixel data of a window, and Figure 2 is a diagram showing a specific example of a method of implementing an entropy filter according to an embodiment of the present invention, and is a diagram showing an implementation calculation for a window. It represents the process.

1. Step of extracting filtering window pixels by the extraction module 100

[12, 22, 0, 4, 9, 0, 200, 0, 144, 9, 1, 7, 25, 91, 51, 69, 136, 87, 15, 100, 77, 13, 8, 2, 251 ]

2. Pixel sorting step using Batcher’s sorting network by the sorting module 200

[0, 0, 0, 1, 2, 4, 7, 8, 9, 9, 12, 13, 15, 22, 25, 51, 69, 77, 87, 91, 100, 136, 144, 200, 251 ]

3. Difference calculation step of adjacent pixels by the difference calculation module 300

[0, 0, 1, 1, 2, 3, 1, 1, 0, 3, 1, 2, 7, 3, 26, 18, 8, 10, 4, 9, 36, 8, 56, 51]

4. Reduction OR performance step by the adjacent difference expression module 400

[0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]

5. Pipelined entropy calculation by the adjacent difference expression module 400

This step proceeds according to Figure 9. Assume that the variable storing the entropy value is H. Additionally, the variables used in the calculation process include addr and cnt. So, if the signal output by the previous step is [s ₁ , s ₂ , ..., s ₂₄ ], the results of each pipeline step are as follows.

6.1. s ₁ = 0 → H = 0, addr = 2, cnt = 2

6.2. s ₂ = 0 → H = 0, addr = 3, cnt = 3

6.3. s ₃ = 1 → H = 0.5528, addr = 0, cnt = 4

6.4. s ₄ = 1 → H = 0.7386, addr = 0, cnt = 5

6.5. s ₅ = 1 → H = 0.9243, addr = 0, cnt = 6

6.6. s ₆ = 1 → H = 1.1101, addr = 0, cnt = 7

6.7. s ₇ = 1 → H = 1.2958, addr = 0, cnt = 8

6.8. s ₈ = 1 → H = 1.4816, addr = 0, cnt = 9

6.9. s ₉ = 0 → H = 1.4816, addr = 2, cnt = 11

6.10. s ₁₀ = 1 → H = 1.9589, addr = 0, cnt = 12

6.11. s ₁₁ = 1 → H = 2.1446, addr = 0, cnt = 13

6.12. s ₁₂ = 1 → H = 2.3304, addr = 0, cnt = 14

6.13. s ₁₃ = 1 → H = 2.5161, addr = 0, cnt = 15

6.14. s ₁₄ = 1 → H = 2.7019, addr = 0, cnt = 16

6.15. s ₁₅ = 1 → H = 2.8876, addr = 0, cnt = 17

6.16. s ₁₆ = 1 → H = 3.0734, addr = 0, cnt = 18

6.17. s ₁₇ = 1 → H = 3.2591, addr = 0, cnt = 19

6.18. s ₁₈ = 1 → H = 3.4449, addr = 0, cnt = 20

6.19. s ₁₉ = 1 → H = 3.6306, addr = 0, cnt = 21

6.20. s ₂₀ = 1 → H = 3.8164, addr = 0, cnt = 22

6.21. s ₂₁ = 1 → H = 4.0022, addr = 0, cnt = 23

6.22. s ₂₂ = 1 → H = 4.1879, addr = 0, cnt = 24

6.23. s ₂₃ = 1 → H = 4.3737, addr = 0, cnt = 25

6.24. s ₂₄ = 1 → H = 4.5594, addr = 0, cnt = 26

6.25. H = 4.5594 + 0 - (26 - 25)×0.1858 = 4.3737

And to examine real-time processing, an entropy filter was implemented in hardware, and the hardware synthesis results are shown in [Table 2] below. According to [Table 2] below, the hardware of the entropy filter achieved a high operating frequency without consuming a lot of hardware resources on the FPGA device. For this reason, the hardware structure of the entropy filter proposed in the present invention can be easily integrated into other image processing systems.

Xilinx Vivado v2019.1 Device xczu7ev-2ffvc1156 Design 5×5 entropy filter Slice Logic Utilization Available Used Utilization register (#) 460,800 5374 1.17% LUTs (#) 230,400 5925 2.57% RAM36X1S (#) 312 4 1.28% Minimum Period (ns) 1.493 Maximum Frequency (MHz) 669.792

Hereinafter, we will look in detail at the method of implementing an entropy filter and the hardware device for implementing it with reference to each drawing for entropy calculation according to the above-described embodiment.

Referring again to FIG. 1, FIG. 1 is a block diagram showing a hardware implementation of an entropy filter according to an embodiment of the present invention. The block diagram showing the hardware implementation of FIG. 1 is divided into five steps and implemented in the same way as the implementation flowchart of the entropy filter of FIG. 2.

Next, referring to FIG. 3, FIG. 3 is a diagram showing the necessity of hardware implementation of an entropy filter. Existing Chinese and US registered patents proposed the concept of entropy filter and showed its application to image processing algorithms belonging to autonomous vehicles or smart surveillance cameras, which have recently attracted attention. However, to date, only software implementations that operate on homogeneous computing systems have been implemented, and hardware implementations of entropy filters have not yet been developed. By integrating FPGA (field programmable gate array) into the heterogeneous computing system (heterogeneous computing system), which has recently received a lot of attention from industry and academia, complex operations can be implemented on FPGA, making it possible to implement entropy filters. It can be said that hardware implementation is very necessary. This is because implementing the hardware of an entropy filter can rapidly accelerate the previously mentioned autonomous vehicles or smart surveillance cameras. Therefore, the present invention proposes the first hardware structure of an entropy filter.

FIG. 4 is a diagram showing the process performed by the extraction module 110 corresponding to the first step in FIG. 1. Input video data is input together with timing signals (horizontal active video and vertical active video). Therefore, it is assumed that the size of the filtering window is N-by-N, and the extraction module 110 extracts all (N 7, 9, 11, 13, 15) pixels To extract, first (N-1) line memories 101 are used. The extraction module 110 can extract N lines of the filtering window through these (N-1) line memories. Next, the extraction module 110 uses (N-1) registers 102 to extract line data of each image of the window and the corresponding N columns. Therefore, the extraction module 110 can extract a total of (N×N) pixels using a total of (N-1) line memories and N×(N-1) registers.

Lastly, the extraction module 110 utilizes a memory controller 103, which is a circuit that operates line memory using timing signals.

The operation mode of line memory can be divided into data writing and data reading. In the case of writing, write enable, write address, write data, etc. must be created. In the case of reading, read enable (reverse of write enable) and read address must be created. Such signals can be generated from timing signals.

Next, Figure 5 is a diagram showing the Batcher's sorting network (registration number: 10-2039653) of Patent Document 5, for which the present applicant has registered a domestic patent. The Batcher's sorting network sorts all (N This is to calculate the normalized histogram used when calculating. In the case of an entropy filter, only the pixels in the filtering window need to be considered instead of the entire image.

Therefore, after the alignment module 200 aligns the pixels in the window, the difference calculation module 300 calculates the difference between adjacent pixels. By these pixel-to-pixel differences, we not only know how many intensity levels there are for a pixel in the window, but also know the number of occurrences of that pixel's intensity level. In addition, since the adjacent difference expression module 400 requires only a 1-bit signal to represent the difference between pixels, it performs a reduction OR step with a simple operation. Lastly, the entropy calculation module 500 uses a pipelined entropy calculation step to calculate the entropy value using 1-bit signals. This new entropy calculation process is an original method of the present invention, and thus enables efficient hardware implementation.

Figure 6 is a diagram showing the entropy calculation method described above in Figure 5. In the case of existing methods, when creating a histogram, it depends on the size of the input data. For example, considering 8-bit image data as shown in FIG. 6, a 256-size histogram must be generated. Therefore, we extract 9 pixels from a 3×3 window, separate them into 5 pixel intensity levels, and then count the number of occurrences of each intensity level. This is called a histogram, and to calculate the normalized histogram, divide the histogram by the number of pixels in the window. Therefore, the entropy value can be calculated using the formula shown in Figure 3. It is very difficult to implement these existing entropy calculation methods in hardware. To solve this problem, the present invention proposed a new entropy calculation method.

First, the window pixels are extracted, and then the pixels are sorted in ascending order through the Batcher’s sorting network. Afterwards, the human difference of the sorted pixels is calculated to know the number of pixel intensity levels and the corresponding normalized histogram value. This signal is efficiently expressed to generate 1-bit signals through a reduction OR gate. In Figure 6, the first four pixels (10, 10, 51, 75) are considered after batcher's sorting. The corresponding 1-bit signals are (0, 1, 1). So, depending on the 1-bit signals, we know how many intensity levels there are. In addition, depending on the values of 0 and 1, the corresponding normalized histogram value is also known. Using this information, the entropy value can later be calculated through the pipelined entropy calculation method shown in FIG. 9.

FIG. 7 is a diagram illustrating adjacent difference calculation by receiving outputs of a Batcher's sorting network by the difference calculation module 300. At this stage, the difference calculation module 300 only needs to perform a subtraction operation to calculate the difference between pixels. In addition, the number of input signals to the difference calculation module 300 is (N×N), but the number of output signals on the difference calculation module 300 is (N×N-1).

FIG. 8 is a diagram showing how the adjacent difference expression module 400 receives the difference between adjacent pixels, passes it through a reduction OR gate, and generates a signal of one bit size to determine whether there is a difference. The operation of the reduction OR gate on the adjacent difference expression module 400 is to receive a signal according to the size of the input data and create a 1-bit signal. As with the OR operation, if the input is 0, the output is 0. If the input is not 0, the output is 1. Also, at this stage, the adjacent difference expression module 400 receives (N×N-1) input signals and creates (N×N-1) output signals.

FIG. 9 is a diagram showing the final step of pipelined entropy calculation by the entropy calculation module 500. The detailed hardware structure of the entropy calculation module 500 is shown in Figure 9, and Figure 9 explains the calculation flow. The final step performed by the entropy calculation module 500 includes (N×N) pipeline steps and can be broadly divided into three types. The first step (501) is to initialize a register called H, which stores the entropy value, and registers called addr and cnt, which help calculate entropy. From the second step to the (N The last (N×N)-th step 503 is a step of calculating the final entropy value. Therefore, unlike existing methods, the entropy calculation method proposed in the present invention not only can process real-time image data, but the newly proposed entropy calculation method can also be generalized according to the window size (N×N).

Figure 10 is a diagram showing the hardware structure of the final stage of pipelined entropy calculation. The operation of this step is illustrated in Figure 9, which shows the detailed hardware structure. According to this, the green blocks represent pipeline registers, and the white blocks represent combinational logic circuits. The calculation shown in Figure 9 has the advantage of a simple hardware structure because it consists only of register, multiplexer, adder, look-up table (LUT), and multiplier operations. can be provided.

The present invention can also be implemented as computer-readable code on a computer-readable recording medium. Computer-readable recording media include all types of recording devices that store data that can be read by a computer system.

Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices, and are also implemented in the form of a carrier wave (e.g., transmission via the Internet). It also includes

Additionally, computer-readable recording media can be distributed across computer systems connected to a network, so that computer-readable code can be stored and executed in a distributed manner. And functional programs, codes, and code segments for implementing the present invention can be easily deduced by programmers in the technical field to which the present invention belongs.

As described above, the specification and drawings disclose preferred embodiments of the present invention, and although specific terms are used, they are used only in a general sense to easily explain the technical content of the present invention and aid understanding of the invention. , it is not intended to limit the scope of the present invention. It is obvious to those skilled in the art that in addition to the embodiments disclosed herein, other modifications based on the technical idea of the present invention can be implemented.

1: Hardware device for implementing entropy filter
100: Extraction module
200: Alignment module
300: difference calculation module
400: Adjacent difference expression module
500: entropy calculation module

Claims (3)

A first step of extracting pixels in the window using a line memory and a buffer register according to the filtering window size;
A second step of sorting the extracted pixels using a Batcher's sorting network;
a third step of calculating adjacent differences of aligned pixels;
A fourth step of performing a reduction OR step to express the difference calculated in the third step as 1 bit; and
A fifth step of performing a pipelined entropy calculation step; A method of implementing an entropy filter comprising:
The method of claim 1, wherein the first step is,
When input video data is input together with timing signals (horizontal active video and vertical active video), all (N A method of implementing an entropy filter, characterized in that (N-1) line memories 101 are first used to extract ) pixels.
An extraction module 100 that extracts pixels in a window using a line memory and a buffer register according to the size of the filtering window;
A sorting module 200 that sorts the extracted pixels using a Batcher's sorting network;
a difference calculation module 300 that performs the step of calculating adjacent differences of aligned pixels;
an adjacent difference expression module 400 that performs a reduction OR step to express the difference value calculated in the difference calculation module 300 in 1 bit; and
an entropy calculation module 500 that performs a pipelined entropy calculation step; A hardware device for implementing an entropy filter, comprising: