WO2020057062A1 - Method and apparatus for implementing bilateral image filtering in fpga, and fpga - Google Patents

Abstract

A method and apparatus for implementing bilateral image filtering in an FPGA, and an FPGA, for use in solving the problem in the prior art of being unable to meet real-time requirements when filtering image data by using a bilateral filtering algorithm on an FPGA. The method comprises: in the process that an FPGA receives image data from an imaging device, after receiving valid data in the image data, determining one piece of received pixel data as target pixel data, and determining all pixel data in a filtering window where the target pixel data is located as influence pixel data (102); sequentially searching for and determining a domain weight value corresponding to each influence pixel data from a stored domain weight table and searching for and determining a range weight value corresponding to each influence pixel data from a stored range weight table (104); and filtering the target pixel data by means of a bilateral filter according to the found domain weight values and range weight values of all the influence pixel data of the target pixel data, to obtain filtered pixel data (106).

Description

Method and device for implementing bilateral image filtering in FPGA, FPGA

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on September 19, 2018, with the application number 201811096288.1, and the invention name is "Method and Apparatus for Implementing Bilateral Filtering of Images in FPGAs, FPGAs". Citations are incorporated in this application.

Technical field

The present invention relates to the field of image processing, and in particular, to a method and device for realizing bilateral filtering of an image in a Field-Programmable Gate Array (FPGA), an FPGA, a computer program, and an imaging device.

Background technique

At present, in some application scenarios, image processing is performed on a video image acquired by an imaging device to obtain required image data.

These image processing techniques include filtering. Image filtering is to suppress the noise of the target image while preserving image feature information as much as possible. It is an indispensable operation in image preprocessing. The quality of its processing effect will directly affect the effectiveness and reliability of subsequent image processing and analysis. Sex. There are many methods for image filtering, such as median filtering, non-linear filtering, morphological filtering, etc. Each filtering method has different advantages and uses, and can be personalized according to customer needs.

Bilateral filter (Bilateral filter) is a non-linear filtering method. It considers spatial information and gray similarity at the same time, and combines the spatial proximity of the image and the gray value similarity of the pixel to filter the image. At the same time, the edge (ie, detail features) information of the image is well preserved. In comparison, general filters (such as Wiener filter, Gaussian filter, etc.) will have obvious edge blurring, and the protection effect on high-frequency edge details is not good. Therefore, bilateral filtering is widely used in applications that require high image edge details. It comprehensively considers the correlation between spatial and value domains to process target data. It is a compromise processing method with simple, non-iterative, and local Features.

In the discrete field of image processing, the definition of bilateral filtering can be expressed as:

among them,

Among them, I ′ is the image pixel after bilateral filtering, I is the input image pixel of the filter, I (i, j) is the gray value of the current filtered target pixel, and (i, j) is the coordinate of the target pixel , Ω is the set of all pixels contained in the filtering window (filtering is performed in a certain window area), I (k, l) is the gray value of any pixel within the filtering window Ω, (k, l ) Is the coordinates of the pixel. g _s is a filter function in the spatial domain, and f _r is a filter function in the value domain. K is the weight, which represents the proximity in the airspace and the similarity in the range.

For any point I (i, j) on the image, I (k, l) is any pixel point in the area window Ω (for example, a window of 3 * 3, 5 * 5, or 7 * 7) where the point is located, Then the weight function g _s , which characterizes the spatial proximity of two pixels, can be expressed by a Gaussian kernel:

Similarly, the weight function f _r representing the similarity of the range of two pixels is:

Among them, σ _s and σ _r represent the smoothness parameters of the space domain and the range, respectively, and σ _s is defined as the spatial degree proximity factor. The larger the value, the greater the influence of surrounding pixels on the target pixel, and the more blurred the image; otherwise, the image is clearer, but if its value is too small, the filtering effect of noise may be limited.

In some application scenarios, after the video imaging data is obtained by the image imaging device, the video image data is processed by the FPGA using a bilateral filtering algorithm to obtain filtered image data. Generally, IP (Intellectual Property) is called in FPGA to check each pixel in the image data and calculate the above formula.

However, in the process of using bilateral filtering algorithms to filter image data in FPGAs, a large amount of logical resources are consumed, and the image data acquired by the imaging equipment cannot be processed in real time, so the acquired image data is usually cached to Locally, filtering is performed.

It can be seen that, in the method of filtering the image data by using the bilateral filtering algorithm on the FPGA of the existing imaging equipment, there is a problem that the real-time requirement cannot be met.

Summary of the Invention

In view of the above problems, the present application provides a method and device for implementing bilateral filtering of an image in an FPGA, an FPGA, a computer program, and an imaging device. , Can not meet the real-time requirements.

According to an aspect of the present application, a method for implementing bilateral filtering of an image in an FPGA is provided, including:

In the process of FPGA receiving image data from the image sensor, after receiving valid data in the image data, the received pixel data is determined as the target pixel data, and all the pixel data in the filtering window where the target pixel data is located is determined. To affect pixel data;

Look up and determine the airspace weight value corresponding to each affected pixel data from the stored airspace weight table in turn, and look up and determine the range weight value corresponding to each affected pixel data from the stored range weight table;

The target pixel data is filtered by using a bilateral filter to obtain the filtered pixel data according to all the found spatial weight values and range weight values of the affected pixel data.

According to an aspect of the present application, an apparatus for implementing bilateral filtering of an image in an FPGA is provided, including:

A receiving unit for receiving image data;

The searching unit is configured to determine, after receiving the valid data in the image data, the received pixel data as target pixel data, and determine that all the pixel data in the filtering window where the target pixel data is located is the affected pixel data; Look up and determine the airspace weight value corresponding to each affected pixel data from the stored airspace weight table, and find and determine the value of the domain weight corresponding to each affected pixel data from the stored value domain weight table;

The filtering unit is configured to filter the target pixel data by using a bilateral filter according to the spatial weight value and the range weight value of all the affected pixel data of the found target pixel data, to obtain filtered pixel data.

According to an aspect of the present application, an FPGA is provided, and the FPGA includes the foregoing device for implementing bilateral filtering of an image. According to one aspect of the present application, a computer program is provided, the computer program having a code segment configured to perform an image bilateral filtering process, the image bilateral filtering process includes:

In the process of receiving image data, after receiving valid data in the image data, the received one pixel data is determined as the target pixel data, and all the pixel data in the filtering window where the target pixel data is located is determined as the affected pixel data;

Look up and determine the spatial weight value corresponding to each affected pixel data from the stored spatial weight table in turn, and look up and determine the corresponding domain weight value from the stored value domain weight table;

The target pixel data is filtered by using a bilateral filter to obtain the filtered pixel data according to all the found spatial weight values and range weight values of the affected pixel data.

According to an aspect of the present application, there is provided an imaging device including an image sensor and an FPGA;

An image sensor is used to obtain image data;

In the process of receiving image data from the image sensor, the FPGA determines the received pixel data as target pixel data after receiving valid data in the image data, and determines all pixels in the filtering window where the target pixel data is located. Data is the data of the affected pixels;

Look up and determine the spatial weight value corresponding to each affected pixel data from the stored spatial weight table in turn, and look up and determine the corresponding domain weight value from the stored value domain weight table;

The target pixel data is filtered by using a bilateral filter to obtain the filtered pixel data according to all the found spatial weight values and range weight values of the affected pixel data.

According to the method for implementing bilateral filtering of an image in an FPGA provided by an embodiment of the present application, in the process of receiving image data from an image sensor by the FPGA, for any pixel data received, all The pixel data is the influential pixel data. Look up and determine the corresponding airspace weight value from the stored airspace weight table, and look up and determine the corresponding value domain weight value from the stored value domain weight table. The spatial weight value and range weight value of all influencing pixels are filtered by a bilateral filter to obtain the filtered pixels. Among them, the complex spatial weight calculation and range weight calculation are changed from real-time calculation to query. The table operation and table lookup operation speed is much faster than the real-time calculation speed, which can speed up the processing speed of the bilateral filtering of the image and improve the real-time processing. Therefore, it can solve the problem of filtering the image data by using the bilateral filtering algorithm on the FPGA in the prior art. Handle problems that cannot meet real-time requirements.

Other features and advantages of the present invention will be explained in the following description, and partly become apparent from the description, or be understood by implementing the present invention. The objects and other advantages of the present invention can be achieved and obtained through the structures specifically pointed out in the written description, claims, and drawings.

The technical solutions of the present invention will be described in further detail below with reference to the drawings and embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention, and do not constitute a limitation on the present invention. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts. In the drawings:

1 is a processing flowchart of a method for implementing bilateral filtering of an image in an FPGA according to an embodiment of the present application;

FIG. 2 is another processing flowchart of a method for implementing bilateral filtering of an image in an FPGA according to an embodiment of the present invention; FIG.

FIG. 3 is an example of position labels in the filtering window;

FIG. 4 is an example of an airspace weight table;

FIG. 5 is an example of a range weight table;

FIG. 6 is another processing flowchart of a method for implementing bilateral filtering of an image in an FPGA according to an embodiment of the present invention; FIG.

7 is a schematic structural diagram of an apparatus for implementing bilateral filtering of an image in an FPGA according to an embodiment of the present invention;

FIG. 8 is another schematic structural diagram of an apparatus for implementing bilateral filtering of an image in an FPGA provided by an embodiment of the present invention.

detailed description

In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described The embodiments are only a part of the embodiments of the present invention, but not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by a person of ordinary skill in the art without creative efforts should fall within the protection scope of the present invention.

Aiming at the problem that the bilateral filtering algorithm is used to filter image data on the FPGA in the prior art and cannot meet the real-time requirement, the embodiment of the present application provides a solution for implementing bilateral filtering of an image in an FPGA to solve the problem. . In this solution, in the process of receiving image data from the image sensor by the FPGA, for any pixel data received, it is determined that all the pixel data in the filtering window where the pixel data is located is the affected pixel data, and the weight from the stored airspace is determined. Look up and determine the spatial weight value corresponding to each affected pixel in the table, find and determine the value range weight value corresponding to each affected pixel from the stored range weight table, and according to the found space weight value and range weight of all affected pixels Value, the bilateral target filter is used to filter the target pixel to obtain the filtered pixel. Among them, the complex spatial weight calculation and range weight calculation are changed from real-time calculation to table lookup operation, which is much faster than real-time. The speed of calculation can speed up the processing speed of bilateral filtering of the image and improve the real-time performance of the processing, which can solve the problem of using the bilateral filtering algorithm on the FPGA to filter the image data in the prior art, which cannot meet the real-time requirements.

The above is the core idea of the present invention. In order for those skilled in the art to better understand the technical solutions in the embodiments of the present invention, and to make the above-mentioned objects, features, and advantages of the embodiments of the present invention more obvious and understandable, the following is combined with the attached The drawings further illustrate the technical solutions in the embodiments of the present invention in detail.

An embodiment of the present application provides a method for implementing bilateral filtering of an image in an FPGA. As shown in FIG. 1, the method includes:

Step 102: In the process of the FPGA receiving the image data from the image sensor, after receiving the valid data in the image data, determine the received pixel data as the target pixel data, and determine the target pixel data in the filter window. All pixel data is the affected pixel data;

Step 104: Look up and determine the airspace weight value corresponding to each affected pixel data from the stored airspace weight table in turn, and look up and determine the value of the domain weight corresponding to each affected pixel data from the stored range weight table.

Step 106: Filter the target pixel data by using a bilateral filter to obtain filtered pixel data according to the spatial weight values and range weight values of all the affected pixel data that are found by the target pixel data.

Through the processing process shown in FIG. 1, a table lookup operation is performed on all the affected pixel data of each received pixel data in the filtering window to find the spatial weight value and range weight value of all affected pixel data, and according to The found spatial weight value and range weight value, and the target pixel data is filtered by using a bilateral filter, which can improve the operation speed of determining the spatial weight value and the range weight value through a fast table lookup operation. It can improve the processing speed and real-time performance of image bilateral filtering processing on FPGA, and can solve the problem of using the bilateral filtering algorithm on FPGA to filter image data in the prior art, which cannot meet the real-time requirements.

In some embodiments, as shown in FIG. 2, before the processing shown in FIG. 1, the airspace weight table and the range weight table may be determined and stored according to the following steps:

Step 100: During the FPGA receiving the first frame of image data, during the blanking period of the first frame of image data, according to a preset spatial smoothing parameter σ _s , a range smoothing parameter σ _r , a radius r of the filtering window Ω, and The bit width n of the gray value data determines the airspace weight value included in the airspace weight table and the range weight value included in the range weight table;

Among them, the airspace weight table includes M airspace weight values. One airspace weight value is an airspace neighboring weight of the target pixel data in the filter window that affects the pixel data to the target pixel data. The filter window includes M influence pixel data, M = (2r + 1) ² , r is the window radius of the filtering window Ω; the range weight table includes N range weight values, and a range weight value is a range proximity weight that affects pixel data to target pixel data. , N = 2 ⁿ , n is the bit width of the gray value data.

In some application scenarios, the airspace weight table and range weight table can be saved as a look-up-table (LUT).

Specifically, an airspace weight value that affects pixel data is determined according to the following formula (3); a range weight value is determined according to the formula (4);

Among them, g _s is a filtering function in the spatial domain, f _r is a filtering function in the value domain, I (i, j) is the gray value of the target pixel data, (i, j) is the coordinates of the target pixel data, and I ( k, l) is the gray value that affects the pixel data, (k, l) is the coordinate that affects the pixel data, σ _s is a preset spatial smoothness parameter, and σ _r is a preset range smooth parameter.

In a specific application scenario, σ _s and σ _r are usually preset parameters, and are set by an operator according to specific application needs.

Step 101: Store the determined airspace weight table and range weight table.

The operation of storing the determined spatial weight table may include: storing the spatial weight value corresponding to the affected pixel data in the spatial weight table according to the position of the affected pixel data in the filtering window.

For example, taking a 3 * 3 filter window Ω as an example, the radius of the window Ω is r = 1, and when a target pixel data is filtered by the filter window, the filter window includes (2r + 1) ² = 9 Affected pixel data can be labeled with the position of the affected pixel in the filter window, as shown in Figure 3; and according to the position of the affected pixel data in the filter window, the spatial weight value corresponding to the affected pixel data is stored in the spatial weight table, as shown in the figure. 4 shown. In the filter window shown in FIG. 3, the pixel data with the position number (1, 1) is the target pixel data, and all the data in the filter window Ω = 3 × 3 are the affected pixel data, that is, as shown in FIG. 3 below. The position numbers shown are (0,0), (0,1), (0,2), (1,0), (1,1), (1,2), (2,0), (2, 1), (2, 2) pixel data. In the airspace weight table shown in FIG. 4, the index is a position number that affects pixel data, and each position number corresponds to an airspace weight value, including g1 to g9.

The operation of storing the determined range weight table may include: storing the range weight values corresponding to the values 0 to N in the range weight table in the order of the values from 0 to N in ascending or descending order. ;

In the above formula (4), the variables in the window Ω include the smoothing parameter σ _r , the gray value I (i, j) of the target pixel, and the gray value I (k, l) of the affected pixel, where the preset Σ _{r is} constant, then the variable only contains | I (i, j) -I (k, l) | (that is, the difference between the target pixel data and the gray value of the affected pixel data). For example, for image data with a bit width of n = 8 bits of gray value data, I (i, j) ∈ [0,255], I (k, l) ∈ [0,255], so | I (i, j) -I (k, l) | ∈ [0,255]. Because I (i, j) and I (k, l) are the gray values of pixels at a point on the image I, the values are non-negative integers. At this time, | I (i, j) -I (k, l) | can be used as an index, and the values are 0, 1, 2, ..., 255, and the corresponding formula (4) is stored in the LUT In Figure 5, the index is 0 to 255, and n0 to n255 represent the value of formula (4) when | I (i, j) -I (k, l) | ∈ [0,255], that is, Range weight value.

According to the operations of storing the airspace weight table and the range weight table, correspondingly, the operation of finding and determining an airspace weight value corresponding to the pixel data from the stored airspace weight table in step 104 includes:

According to the position of the affected pixel data in the filtering window, a spatial weight value corresponding to the position of the affected pixel data is determined from the stored spatial weight table.

For example, as shown in FIG. 3 and FIG. 4, the position of the affected pixel data in the filtering window may be determined first, and the spatial weight value corresponding to the position may be found in the spatial weight table.

The operation of searching for and determining an affected range weight value corresponding to pixel data from the stored range weight table in the above step 104 includes:

A difference between the gray value of the target pixel data and the gray value of the affected pixel data is determined, and a range weight value corresponding to the difference value is searched and determined in the range weight table according to the difference value.

For example, after determining the difference between the gray value of the target pixel data and a gray value that affects the pixel data, use the value of the difference as an index. In the range weight table shown in FIG. 5, Find the range weight value corresponding to the difference value.

Through the above processing, the spatial weight value and range weight value of the affected pixels can be found.

In the above step 106, using a bilateral filter to filter the target pixel data includes:

Filter the target pixel data according to the following formula (1) and formula (2):

Among them, I ′ is the image pixel data after bilateral filtering, I is the input image pixel data of the filter, I (i, j) is the gray value of the target pixel data, and (i, j) represents the coordinates of the target pixel data point , Ω is the set of all pixel data points contained in the filter window, I (k, l) is an influential pixel data in the filter window Ω, (k, l) is the coordinate that affects the pixel data; g _s is in the spatial domain Filter function, f _r is the filter function on the value range; K is the weight.

In the above step 104, the value of g _{s and} the value of f _r have been found, and the searched value is substituted into the above formulas (1) and (2), and the filtered pixel data is calculated and obtained.

The process of bilateral filtering the pixel data has been described above.

In the above embodiment, the spatial smoothing parameter σ _s , the range smoothing parameter σ _r , the filtering window radius r, and the gray value data bit width n are preset and fixed. In other embodiments, according to the needs of the application scenario, it is allowed to modify and update any one of the parameters of the spatial smoothing parameter σ _s , the range smoothing parameter σ _r , the filtering window radius r, and the gray value data bit width n. Then, the process shown in FIG. 1 needs to update the airspace weight table and the range weight table according to the updated parameters.

FIG. 6 shows a processing flow for updating the airspace weight table and the range weight table. The processing before the processing flow shown in FIG. 1 includes:

Step 100 ': In the process of receiving subsequent image data, during the frame blanking period of the received image data, detect any of the spatial smoothing parameters, range smoothing parameters, filter window size, and gray value data bit width of the bilateral filter. Whether a parameter has been updated;

Step 101 ': When it is determined that the parameters are not updated, valid data in the image data is received, and the process proceeds to step 102;

Step 102 ': When it is determined that the parameters are updated, determine the airspace weight values included in the airspace weight table according to the updated airspace smoothing parameters, range smoothing parameters, filter window size, and gray value data bit width, and / The range weight value included in the OR range weight table; stores the updated airspace weight table and / or range weight table; and receives valid data in the image data, and the process proceeds to step 102.

The process of determining the airspace weight value included in the airspace weight table and determining the range weight value included in the range weight table is as described above, and is not repeated here.

Through the process shown in FIG. 6, the airspace weight table and the range weight table can be updated according to the updated parameters. Further combined with the processing shown in FIG. 1, the airspace weight table and the range weight table can be updated according to the processing shown in FIG. 1. To filter the pixel data.

According to the method provided in the embodiment of the present application, during the process of receiving image data from the image sensor by the FPGA, for the received pixel data, it is determined that all the pixel data in the filtering window where the pixel data is located is affected pixel data. Look up the airspace weight table to determine the airspace weight value corresponding to each influencing pixel, look up and determine the value range weight value corresponding to each influencing pixel from the stored value range weight table, and according to the airspace weight values and values of all the influencing pixels found Domain weight value, the bilateral target filter is used to filter the target pixel to obtain the filtered pixel; among them, the complex spatial weight calculation and range weight calculation are changed from real-time calculation to table lookup operation, and the speed of table lookup operation is much faster The speed of real-time calculation can speed up the processing speed of bilateral filtering of the image and improve the real-time performance of the processing, so that it can solve the problem of using the bilateral filtering algorithm on the FPGA to filter the image data in the prior art, which cannot meet the real-time requirements.

Based on the same inventive concept, an embodiment of the present application further provides a device for implementing bilateral filtering of an image in an FPGA.

FIG. 7 shows a structure of an apparatus for implementing bilateral filtering of an image in an FPGA provided by an embodiment of the present application. The apparatus includes:

A receiving unit 71, configured to receive image data from an image sensor;

The searching unit 72 is configured to determine, after the receiving unit 71 receives valid data in the image data, one pixel data received as the target pixel data, and determine that all the pixel data in the filtering window where the target pixel data is located is the affected pixel data. ; Search and determine the spatial weight value corresponding to each affected pixel data in turn from the stored spatial weight table, and find and determine the corresponding domain weight value from each stored pixel weight table from the stored range weight table;

The filtering unit 73 is configured to filter the target pixel data by using a bilateral filter to obtain the filtered pixel data according to the spatial weight value and the range weight value of all the affected pixel data of the found target pixel data.

In some embodiments, as shown in FIG. 8, the apparatus shown in FIG. 7 may further include a determining unit 70.

A determining unit 70 is configured to: during the process of receiving the first frame of image data by the receiving unit, during the frame blanking period of the first frame of image data, according to a preset spatial smoothing parameter σ _s , a range smoothing parameter σ _r , and a filtering window The radius r and the bit width n of the gray value data determine the airspace weight value included in the airspace weight table and the value of the range weight included in the value range weight table; the determined airspace weight table and range weight table are stored;

Among them, the airspace weight table includes M airspace weight values. One airspace weight value is an airspace neighboring weight of the target pixel data in the filter window that affects the pixel data to the target pixel data. The filter window includes M influence pixel data, M = (2r + 1) ² , r is the window radius of the window Ω; the range weight table includes N range weight values. A range weight value is a range weight that affects pixel data to target pixel data. N = 2 ⁿ , n is the gray-scale data bit width.

In some embodiments, the determining unit 70 stores the determined spatial weight table, which includes: storing the spatial weight values corresponding to the affected pixel data in the spatial weight table according to the position order of the affected pixel data in the filtering window;

Then, the searching unit 72 searches and determines an airspace weight value corresponding to the affected pixel data from the stored airspace weight table, including: according to the position of the affected pixel data in the filtering window, searching and determining from the stored airspace weight table to determine the affected pixel. The spatial weight value corresponding to the position of the data.

In some embodiments, the determining unit stores the determined range weight table, which includes: storing values corresponding to the values 0 to N in the range weight table according to the order from 0 to N values from small to large or from large to small. Domain proximity weight value;

Then, the searching unit 72 searches and determines from the stored range weight table a range weight value corresponding to the affected pixel data, including: determining a difference between a gray value of the target pixel data and a gray value of the affected pixel data. Value, according to the difference value, look up in the range weight table to determine the range weight value corresponding to the difference value.

Further, in some embodiments, the determining unit 70 is further configured to: in the process of receiving the subsequent image data by the receiving unit 71, during the frame blanking period of the received image data, detect the spatial smoothness parameters and smooth value range of the bilateral filter Whether any one of the parameters, filter window size and gray value data bit width is updated; if it is determined that the parameter is not updated, valid data in the image data is received; if it is determined that the parameter is updated, according to the update The airspace smoothing parameter, range smoothing parameter, filter window size, and gray value data bit width, determine the airspace weight values included in the airspace weight table, and / or the range weight values included in the range weight table; after the update is stored, The airspace weight table and / or range weight table; and receive valid data from the image data.

In some embodiments, the determining unit determines an airspace weight value included in the airspace weight table, and determines a range weight value included in the range weight table, including:

According to formula

Determine a spatial weight value that affects pixel data;

According to formula

Determine a range weight value;

Among them, g _s is a filtering function in the spatial domain, f _r is a filtering function in the value domain, I (i, j) is the gray value of the target pixel data, (i, j) is the coordinates of the target pixel data, and I ( k, l) is the gray value that affects the pixel data, (k, l) is the coordinate that affects the pixel data, σ _s is the spatial smoothing parameter, and σ _r is the smoothing parameter of the range.

In some embodiments, the filtering unit 73 uses a bilateral filter to filter the target pixel data, including: filtering the target pixel data according to the following formula,

among them,

I ′ is the image pixel data after bilateral filtering, I is the input image pixel data of the filter, I (i, j) is the gray value of the target pixel data, (i, j) represents the coordinates of the target pixel data point, Ω Is the set of all pixel data points contained in the filter window, I (k, l) is a gray value that affects the pixel data in the filter window Ω, (k, l) is the coordinates that affect the pixel data; g _s is the spatial domain Filter function, f _r is the filter function on the value range; K is the weight.

In a specific application scenario, the device shown in FIG. 7 or 8 can be encapsulated into an IP core, and the method shown in FIG. 1 or FIG. 2 is implemented by calling the IP core.

Based on the same inventive concept, an embodiment of the present application further provides a computer program having a code segment configured to perform an image bilateral filtering process, where the image bilateral filtering process may be as shown in FIG. 1 or FIG. 2 Processing.

Based on the same inventive concept, an embodiment of the present application further provides an FPGA. In some embodiments, the FPGA includes a device as shown in FIG. 7 or FIG. 8. In other embodiments, the FPGA may further include a computer program as described above. The FPGA provided in the embodiment of the present application can speed up the processing speed of bilateral filtering of an image and improve the real-time performance of processing.

Based on the same inventive concept, an embodiment of the present application further provides an imaging device. The imaging device includes an image sensor and an FPGA as described above. The imaging device provided in the embodiment of the present application can speed up the processing speed of bilateral filtering of an image and improve the real-time performance of processing. The imaging device provided in the embodiment of the present application may be a specific device such as a camera.

In summary, according to the technical solution provided in the embodiment of the present application, during the process of receiving image data from the image sensor by the FPGA, for the received pixel data, it is determined that all the pixel data in the filtering window where the pixel data is located is the affected pixel data , From the stored airspace weight table to find and determine the corresponding airspace weight value of each affected pixel, from the stored value domain weight table to find and determine the value of the corresponding domain weight value of each affected pixel, according to the found all affected pixels of the airspace Weight value and range weight value. The target pixel is filtered by using a bilateral filter to obtain filtered pixels. The complex spatial weight calculation and range weight calculation are changed from real-time calculation to table lookup operation and table lookup operation. The speed is much faster than the real-time calculation speed, which can speed up the processing speed of bilateral filtering of the image and improve the real-time performance of the processing, which can solve the problem of filtering the image data by using the bilateral filtering algorithm on the FPGA in the prior art. Asked questions.

The basic principles of the present invention have been described above in conjunction with specific embodiments. However, it should be pointed out that those skilled in the art can understand all or any steps or components of the methods and devices of the present invention on any computing device (including Processors, storage media, etc.) or computing devices in a network, implemented with hardware firmware, software, or a combination of them. This is how a person of ordinary skill in the art would use their basic programming skills after reading the description of the present invention. Achievable.

A person of ordinary skill in the art may understand that all or part of the steps carried by the methods in the foregoing embodiments may be implemented by a program instructing related hardware. The program may be stored in a computer-readable storage medium. When the program is executed, Including one or a combination of steps of a method embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing module, or each unit may exist separately physically, or two or more units may be integrated into one module. The above integrated modules may be implemented in the form of hardware or software functional modules. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

Those skilled in the art should understand that the embodiments of the present invention may be provided as a method, a system, or a computer program product. Therefore, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, magnetic disk memory, optical memory, etc.) containing computer-usable program code therein.

The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, and combinations of processes and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, so that the instructions generated by the processor of the computer or other programmable data processing device are used to generate instructions Means for implementing the functions specified in one or more flowcharts and / or one or more blocks of the block diagrams.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to work in a particular manner such that the instructions stored in the computer-readable memory produce a manufactured article including an instruction device, the instructions The device implements the functions specified in one or more flowcharts and / or one or more blocks of the block diagram.

These computer program instructions can also be loaded on a computer or other programmable data processing device, so that a series of steps can be performed on the computer or other programmable device to produce a computer-implemented process, which can be executed on the computer or other programmable device. The instructions provide steps for implementing the functions specified in one or more flowcharts and / or one or more blocks of the block diagrams.

Although the above embodiments of the present invention have been described, those skilled in the art can make other changes and modifications to these embodiments once they know the basic inventive concepts. Therefore, the appended claims are intended to be construed to include the embodiments described above and all changes and modifications that fall within the scope of the invention.

Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. In this way, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention also intends to include these modifications and variations.

Claims (18)

1. A method for implementing bilateral filtering of an image in an FPGA is characterized in that it includes:

   In the process of FPGA receiving image data from the image sensor, after receiving valid data in the image data, the received pixel data is determined as the target pixel data, and all the pixel data in the filtering window where the target pixel data is located is determined. To affect pixel data;

   Look up and determine the spatial weight value corresponding to each affected pixel data from the stored spatial weight table in turn, and look up and determine the corresponding domain weight value from the stored value domain weight table;

   The target pixel data is filtered by using a bilateral filter to obtain the filtered pixel data according to all the found spatial weight values and range weight values of the affected pixel data.
2. The method according to claim 1, further comprising:

   During the FPGA receiving the first frame of image data, during the frame blanking period of the first frame of image data, according to the preset spatial smoothing parameter σ _s , range smoothing parameter σ _r , filtering window radius r, and gray value data The bit width n determines the airspace weight value included in the airspace weight table and the range weight value included in the range weight table; stores the determined airspace weight table and range weight table;

   Among them, the airspace weight table includes M airspace weight values. One airspace weight value is an airspace neighboring weight of the target pixel data in the filter window that affects the pixel data to the target pixel data. The filter window includes M influence pixel data, M = (2r + 1) ² , r is the window radius of the filtering window Ω; the range weight table includes N range weight values, and a range weight value is a range weight that affects pixel data to target pixel data. , N = 2 ⁿ , n is the bit width of the gray value data.
3. The method according to claim 2, wherein storing the determined airspace weight table comprises:

   According to the position order of the affected pixel data in the filtering window, the spatial weight value corresponding to the affected pixel data is stored in the spatial weight table; then,

   Look up and determine an airspace weight value corresponding to the affected pixel data from the stored airspace weight table, including:

   According to the position of the affected pixel data in the filtering window, a spatial weight value corresponding to the position of the affected pixel data is determined from the stored spatial weight table.
4. The method according to claim 2, wherein storing the determined range weight table comprises:

   According to the order from 0 to N values from small to large or from big to small, the range weights corresponding to the values 0 to N are correspondingly stored in the range weight table; then,

   Look up and determine a range weight value corresponding to the affected pixel data from the stored range weight table, including:

   A difference between the gray value of the target pixel data and the gray value of the affected pixel data is determined, and a range weight value corresponding to the difference value is searched and determined in the range weight table according to the difference value.
5. The method according to claim 2, further comprising:

   In the process of receiving subsequent image data, during the frame blanking period of the received image data, it is detected whether any one of the parameters of the spatial smoothing parameter, the range smoothing parameter, the filtering window size, and the gray value data bit width of the bilateral filter is present. Update

   Receive valid data in image data if it is determined that the parameters have not been updated;

   When it is determined that the parameters are updated, the airspace weight values and / or range weights included in the airspace weight table are determined according to the updated airspace smoothing parameters, range smoothing parameters, filter window size, and gray value data bit width. The range weight values included in the table; store the updated airspace weight table and / or range weight table; and receive valid data in the image data.
6. The method according to claim 2 or 5, wherein determining an airspace weight value included in the airspace weight table, and determining a range weight value included in the range weight table includes:

   According to formula

   

   Determine a spatial weight value that affects pixel data;

   According to formula

   

   Determine a range weight value;

   Among them, g _s is a filtering function in the spatial domain, f _r is a filtering function in the value domain, I (i, j) is the gray value of the target pixel data, (i, j) is the coordinates of the target pixel data, and I ( k, l) is the gray value that affects the pixel data, (k, l) is the coordinate that affects the pixel data, σ _s is the spatial smoothing parameter, and σ _r is the smoothing parameter of the range.
7. The method according to claim 1, wherein filtering the target pixel data by using a bilateral filter comprises:

   Filter the target pixel data according to the following formula,

   

   among them,

   

   I ′ is the image pixel data after bilateral filtering, I is the input image pixel data of the filter, I (i, j) is the gray value of the target pixel data, (i, j) represents the coordinates of the target pixel data point, Ω Is the set of all pixel data points contained in the filter window, I (k, l) is a gray value that affects the pixel data in the filter window Ω, (k, l) is the coordinates that affect the pixel data; g _s is the spatial domain Filter function, f _r is the filter function on the value range; K is the weight.
8. The method according to claim 1, wherein the airspace weight table and the range weight table are stored as a display lookup table.
9. A device for implementing bilateral filtering of an image in an FPGA is characterized in that it includes:

   A receiving unit for receiving image data;

   The searching unit is configured to determine, after receiving the valid data in the image data, the received pixel data as target pixel data, and determine that all the pixel data in the filtering window where the target pixel data is located is the affected pixel data; Look up and determine the airspace weight value corresponding to each affected pixel data from the stored airspace weight table, and find and determine the value of the domain weight corresponding to each affected pixel data from the stored value domain weight table;

   The filtering unit is configured to filter the target pixel data by using a bilateral filter according to the spatial weight value and the range weight value of all the affected pixel data of the found target pixel data, to obtain filtered pixel data.
10. The apparatus according to claim 9, further comprising:

    A determining unit, configured to: during the receiving of the first frame of image data by the receiving unit, during the frame blanking period of the first frame of image data, according to a preset spatial smoothing parameter σ _s , a range smoothing parameter σ _r , and a filtering window radius r and gray value data bit width n, determine the airspace weight value included in the airspace weight table, and the range weight value included in the range weight table; store the determined airspace weight table and range weight table;

    Among them, the airspace weight table includes M airspace weight values. One airspace weight value is an airspace neighboring weight of the target pixel data in the filter window that affects the pixel data to the target pixel data. The filter window includes M influence pixel data, M = (2r + 1) ² , r is the window radius of the filtering window Ω; the range weight table includes N range weight values, and a range weight value is a range weight that affects pixel data to target pixel data. , N = 2 ⁿ , n is the bit width of the gray value data.
11. The apparatus according to claim 10, wherein the determining unit stores the determined airspace weight table, comprising:

    According to the position order of the affected pixel data in the filtering window, the spatial weight value corresponding to the affected pixel data is stored in the spatial weight table; then,

    The search unit determines from the stored airspace weight table an airspace weight value corresponding to the affected pixel data, including:

    According to the position of the affected pixel data in the filtering window, a spatial weight value corresponding to the position of the affected pixel data is determined from the stored spatial weight table.
12. The apparatus according to claim 10, wherein the determining unit stores the determined range weight table, comprising:

    According to the order from 0 to N values from small to large or from big to small, the range weights corresponding to the values 0 to N are correspondingly stored in the range weight table; then,

    The search unit searches the stored range weight table to determine a range weight value corresponding to the affected pixel data, including:

    A difference between the gray value of the target pixel data and the gray value of the affected pixel data is determined, and a range weight value corresponding to the difference value is searched and determined in the range weight table according to the difference value.
13. The apparatus according to claim 10, wherein the determining unit is further configured to:

    During the process of receiving the subsequent image data by the receiving unit, during the frame blanking period of the received image data, any one of the parameters of the spatial smoothing parameter, the range smoothing parameter, the filtering window size and the gray value data width of the bilateral filter is detected Whether there is an update;

    Receive valid data in image data if it is determined that the parameters have not been updated;

    When it is determined that the parameters are updated, the airspace weight values and / or range weights included in the airspace weight table are determined according to the updated airspace smoothing parameters, range smoothing parameters, filter window size, and gray value data bit width. The range weight values included in the table; store the updated airspace weight table and / or range weight table; and receive valid data in the image data.
14. The apparatus according to claim 10 or 13, wherein the determining unit determines an airspace weight value included in the airspace weight table, and determines a range weight value included in the range weight table, comprising:

    According to formula

    

    Determine a spatial weight value that affects pixel data;

    According to formula

    

    Determine a range weight value;

    Among them, g _s is a filtering function in the spatial domain, f _r is a filtering function in the value domain, I (i, j) is the gray value of the target pixel data, (i, j) is the coordinates of the target pixel data, and I ( k, l) is the gray value that affects the pixel data, (k, l) is the coordinate that affects the pixel data, σ _s is the spatial smoothing parameter, and σ _r is the smoothing parameter of the range.
15. The device according to claim 9, wherein the filtering unit filters the target pixel data using a bilateral filter, comprising:

    Filter the target pixel data according to the following formula,

    

    among them,

    

    I ′ is the image pixel data after bilateral filtering, I is the input image pixel data of the filter, I (i, j) is the gray value of the target pixel data, (i, j) represents the coordinates of the target pixel data point, Ω Is the set of all pixel data points contained in the filter window, I (k, l) is a gray value that affects the pixel data in the filter window Ω, (k, l) is the coordinates that affect the pixel data; g _s is the spatial domain Filter function, f _r is the filter function on the value range; K is the weight.
16. An FPGA is characterized in that it includes a device for implementing bilateral filtering of an image, and the device includes:

    A receiving unit for receiving image data from an image sensor;

    The searching unit is configured to determine, after receiving the valid data in the image data, the received pixel data as target pixel data, and determine that all the pixel data in the filtering window where the target pixel data is located is the affected pixel data; Look up and determine the airspace weight value corresponding to each affected pixel data from the stored airspace weight table, and find and determine the value of the domain weight corresponding to each affected pixel data from the stored value domain weight table;

    The filtering unit is configured to filter the target pixel data by using a bilateral filter according to the spatial weight value and the range weight value of all the affected pixel data of the found target pixel data, to obtain filtered pixel data.
17. A computer program, characterized in that the computer program has a code segment configured to perform bilateral image filtering processing, and the image bilateral filtering processing includes:

    In the process of receiving image data, after receiving valid data in the image data, the received one pixel data is determined as the target pixel data, and all the pixel data in the filtering window where the target pixel data is located is determined as the affected pixel data;

    Look up and determine the spatial weight value corresponding to each affected pixel data from the stored spatial weight table in turn, and look up and determine the corresponding domain weight value from the stored value domain weight table;

    The target pixel data is filtered by using a bilateral filter to obtain the filtered pixel data according to all the found spatial weight values and range weight values of the affected pixel data.
18. An imaging device, comprising an image sensor and an FPGA; the image sensor is used to acquire image data;

    In the process of receiving image data from the image sensor, the FPGA determines the received pixel data as target pixel data after receiving valid data in the image data, and determines all pixels in the filtering window where the target pixel data is located. Data is the data of the affected pixels;

    Look up and determine the spatial weight value corresponding to each affected pixel data from the stored spatial weight table in turn, and look up and determine the corresponding domain weight value from the stored value domain weight table;

    The target pixel data is filtered by using a bilateral filter to obtain the filtered pixel data according to all the found spatial weight values and range weight values of the affected pixel data.

Images