CN114040156B - FPGA system and method for video format and image quality processing - Google Patents
FPGA system and method for video format and image quality processing Download PDFInfo
- Publication number
- CN114040156B CN114040156B CN202111295001.XA CN202111295001A CN114040156B CN 114040156 B CN114040156 B CN 114040156B CN 202111295001 A CN202111295001 A CN 202111295001A CN 114040156 B CN114040156 B CN 114040156B
- Authority
- CN
- China
- Prior art keywords
- video
- value
- window
- module
- image quality
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/30—Transforming light or analogous information into electric information
- H04N5/33—Transforming infrared radiation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/76—Television signal recording
- H04N5/765—Interface circuits between an apparatus for recording and another apparatus
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
- H04N7/0125—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level one of the standards being a high definition standard
Abstract
An FPGA system and method for video format and image quality processing, which performs format conversion on video image signals; carrying out image quality improvement on the converted video stream signal; and inputting the video stream with the improved image quality into the SD card, or converting the video stream with the improved image quality into a video stream conforming to the video display standard of the HDMI and outputting the video stream. The invention carries out image quality processing on the medium wave and short wave infrared video signals in a targeted manner, realizes the improvement of the display image quality of the infrared images, ensures the real-time processing of the infrared detector videos of different wave bands, and can realize the output of display signals of different formats to display the processed medium wave and short wave infrared video images according to the characteristics of the medium wave and short wave video signals. Compared with the prior art, the method has better real-time performance, and can support multiband input processing and different format output display.
Description
Technical Field
The invention belongs to the technical field of infrared detector video format and image processing, and particularly relates to an FPGA system and an FPGA method for video format and image quality processing.
Background
Infrared radiation is ubiquitous in nature and any object with a temperature above absolute zero constantly emits infrared light. The infrared focal plane detector is very sensitive to infrared radiation of a specific waveband, so that the infrared focal plane detector can be used for imaging. Compared with visible light imaging, the infrared imaging technology has the unique advantages of long detection distance, strong penetrating power, capability of working at night and in severe environment and the like. Therefore, the infrared imaging technology is widely applied to the fields of industry, security, medicine and the like.
However, due to the limitations of semiconductor materials and manufacturing processes, imaging of the infrared focal plane detector has the characteristics of low resolution, low contrast, few details, high noise, indefinite video format, and the like, and therefore, in an infrared detector imaging system, it is important to realize video format conversion and image quality improvement of infrared detectors with different wave bands.
Disclosure of Invention
In order to overcome the defect of video imaging of a detector, the invention aims to provide an FPGA system and an FPGA method for processing video format and image quality.
In order to achieve the purpose, the invention adopts the technical scheme that:
an FPGA system for video format and image quality processing comprises an input module, an image quality improving module and an output processing module;
the input module is used for carrying out format conversion on the received video image signal of the infrared detector and inputting the converted video stream signal into the image quality improving module;
the image quality improving module is used for outputting the video stream after the image quality of the received video image is improved to the output processing module;
and the output processing module is used for inputting the received video stream into the SD card or converting the received video stream into a video stream conforming to the video display standard of the HDMI and then outputting the video stream to the display.
Further, the input processing module comprises an asynchronous FIFO submodule, a polling reading submodule and a line field generating submodule;
the asynchronous FIFO submodule is used for receiving four paths of parallel video streams under 160M clocks of the infrared detector, converting the four paths of parallel video data streams under 160M clocks into one path of serial infrared video data stream under 40M clocks, and transmitting the one path of serial infrared video data stream under 40M clocks to the image quality improving module;
the polling reading sub-module is used for receiving four paths of parallel video streams under 160M clock of the infrared detector, converting four paths of parallel video data under 160M clock into one path of serial video data in a polling mode by adopting polling signals, and transmitting the converted one path of serial video data to the image quality improving module;
and the line field generation submodule is used for performing clock domain crossing processing on the received line synchronization signal and field synchronization signal of the input video under the clock of 160M, generating a new line field of the clock of 40M through the rising edge count of the line signal and the field signal, and inputting the new line field to the image quality improvement module.
Further, a 160M clock down-line synchronizing signal and a field synchronizing signal input by an external detector are processed across clock domains, and then a new line field of 40M clock is generated by counting rising edges of the line synchronizing signal and the field synchronizing signal.
Further, the polling signal is a counter with a bit width of 2, when the counter value is 0, the first path of parallel video stream is read, when the counter value is 1, the second path of parallel video stream is read, when the counter value is 2, the third path of parallel video stream is read, and when the counter value is 3, the fourth path of parallel video stream is read.
Further, the image quality improving module comprises a non-uniformity correction sub-module, a bad element replacing sub-module, a median filtering sub-module, an unsharp mask sub-module, a dynamic range conversion sub-module, a Sobel edge enhancement sub-module and a Gamma correction sub-module;
the non-uniformity correction submodule is used for calculating 10 frames of data in one path of serial infrared video data stream under the 40M clock by adopting MATLAB to obtain correction parameters, correcting data after 10 frames in one path of serial infrared video data stream under the 40M clock to obtain corrected video image data and inputting the corrected video image data into the bad element replacement submodule;
the bad element replacement submodule is used for caching every two lines of the received corrected video image data by using FIFO (first in first out) to realize window operation, and outputting a median value of a 3 x 3 window through a 6-stage comparison circuit according to the result of the window operation; detecting a median value of the window, replacing the value of the window center point with the median value of the window if the difference between the value of the window center point and the mean value of the rest points in the window is greater than a first threshold value, and outputting the median value of the window to an unsharp mask module; if the difference between the value of the center point of the window and the mean value of the rest points in the window is less than or equal to a first threshold value, outputting the median value of the window to an unsharp mask module;
the unsharp mask submodule is used for caching four rows of the median value of a window, realizing 5-by-5 window operation, calculating through an accumulator to obtain a window convolution sum, obtaining a Gaussian filtering result by using a divider according to the window convolution sum, subtracting the Gaussian filtering result from the median value of the window to obtain a high-frequency image, multiplying the high-frequency image by a coefficient to obtain a video data value after the unsharp mask operation, and inputting the video data value after the unsharp mask operation into the dynamic range conversion submodule;
the dynamic range conversion submodule is used for counting the occurrence frequency of pixel points of the video data value after the unsharp mask operation, storing a counting result in four BRAMs, traversing the four BRAMs in a horizontal field blanking period respectively, finding out the maximum value of pixels in the video data value and the minimum value of pixels in the video data value under a second threshold value, obtaining an 8-bit output value according to the maximum value of pixels in the video data value and the minimum value of pixels in the video data value, and inputting the 8-bit output value into the CLAHE submodule;
the CLAHE submodule is used for calculating the 8-bit output value by using a CLAHE algorithm to obtain the brightness value of the image and inputting the brightness value of the image into the edge enhancement submodule;
the edge enhancement submodule is used for carrying out Sobel operator processing on the brightness value of the image to obtain a brightness value after edge enhancement, and inputting the brightness value after edge enhancement into the Gamma correction submodule;
and the Gamma correction submodule is used for adjusting the brightness value after the edge enhancement to meet the requirement and transmitting the adjusted brightness value to the output processing module.
A method for video format and image quality processing includes the following steps:
performing format conversion on the video image signal;
carrying out image quality improvement on the converted video stream signal;
and inputting the video stream with the improved image quality into the SD card, or converting the video stream with the improved image quality into a video stream conforming to the video display standard of the HDMI and outputting the video stream.
Further, the format conversion of the video image signal includes the steps of:
converting four paths of parallel video data streams under 160M clocks into one path of serial infrared video data stream under 40M clocks;
four paths of parallel video data under 160M clock are converted into one path of serial video data in a polling mode by adopting polling signals;
the horizontal synchronizing signal and the field synchronizing signal of the input video under 160M clock are processed by crossing clock domains, and then a new horizontal field of 40M clock is generated by counting the rising edges of the horizontal signal and the field signal.
Further, the method for improving the image quality of the converted video stream signal comprises the following steps:
MATLAB calculation is carried out on 10 frames of data in one path of serial infrared video data stream under the 40M clock to obtain correction parameters, and data after 10 frames in one path of serial infrared video data stream under the 40M clock are corrected to obtain corrected video image data;
caching every two lines of corrected video image data by using FIFO (first in first out) to realize window operation, and outputting a median value of 3 x 3 windows by using a 6-stage comparison circuit according to a window operation result; detecting the median value of the window, and replacing the value of the window center point with the median value of the window if the difference between the value of the window center point and the mean value of the rest points in the window is greater than a first threshold value;
caching four rows of median values of a window to realize 5-by-5 window operation, then calculating through an accumulator to obtain a window convolution sum, then utilizing a divider to obtain a Gaussian filtering result according to the window convolution sum, subtracting the Gaussian filtering result from the median value of the window to obtain a high-frequency image, and multiplying the high-frequency image by a coefficient to obtain a video data value after the unsharp mask operation;
counting the occurrence times of pixel points of the video data value after the unsharp masking operation, storing the counting result in four BRAMs, traversing the four BRAMs in a line field blanking period respectively, finding out the maximum value of pixels in the video data value and the minimum value of pixels in the video data value under a second threshold value, and obtaining an 8-bit output value according to the maximum value of pixels in the video data value and the minimum value of pixels in the video data value;
calculating the 8-bit output value by using a CLAHE algorithm to obtain the brightness value of the image;
and carrying out Sobel operator processing on the brightness value of the image to obtain the brightness value after edge enhancement.
Further, the method includes the following steps:
inputting the video stream with the improved image quality into an SD card; or the video stream with the improved image quality is subjected to frequency reduction to 50 or 60 frames, and then the video stream subjected to frequency reduction is subjected to video line, field and data signal time sequence adjustment to be output after being subjected to video stream meeting the video display standard of the HDMI.
Compared with the prior art, the invention has the following beneficial effects: firstly, the invention can simultaneously meet the video format input of the infrared detectors of medium wave and short wave, and has better expansibility and compatibility compared with most infrared detectors supporting a single wave band; on the other hand, the invention carries out image quality processing on the medium wave and short wave infrared video signals in a targeted manner, thereby not only realizing the improvement of the display image quality of the infrared images, but also ensuring the real-time processing of the infrared detector videos of different wave bands, and realizing the display of the processed medium wave and short wave infrared video images by outputting display signals of different formats according to the characteristics of the medium wave and short wave video signals. Compared with the prior art, the method has better real-time performance, and can support multiband input processing and different format output display.
Furthermore, the image quality improvement processing module comprises a bad element replacement sub-module, a non-uniformity correction sub-module, an unsharp mask sub-module, a dynamic range conversion sub-module, a CLAHE sub-module, an edge enhancement sub-module and a gamma correction sub-module which support medium wave and short wave infrared signals. The method runs and verifies on the Xersis FPGA, can adapt to different customized infrared detectors, and obviously improves the quality of the output infrared image.
Drawings
FIG. 1 is an overall architecture diagram of the present invention.
FIG. 2 shows a format of a raw input video data stream of an infrared detector.
Fig. 3 is a block diagram of the raw video stream input processing of the detector.
Fig. 4 is a diagram of a video stream format after processing the original video stream format of the detector.
Fig. 5 is a flowchart of an image quality improvement hardware circuit module.
FIG. 6 is a diagram illustrating a parameter configuration module.
Fig. 7 is a schematic diagram of an SPI transmission data format.
Fig. 8 is a schematic diagram of an output processing module.
Fig. 9 is an effect diagram of the image quality improving module (of the output signal of the medium wave detector) verified by the FPGA. The method comprises the following steps of (a) obtaining an original picture (containing noise points, vertical bars and dead points), (b) obtaining a dynamic range conversion module output graph after dead element replacement and nonuniformity correction, (c) obtaining an edge enhancement sub-module output graph, (d) obtaining a CLAHE sub-module output graph, (e) obtaining an edge enhancement module output graph, and (f) obtaining a Gamma correction sub-module output graph.
Fig. 10 is an effect diagram of the image quality improving module (of the output signal of the short wave detector) verified by the FPGA. The method comprises the following steps of (a) obtaining an original picture (containing noise points, vertical bars and dead points), (b) obtaining a dynamic range conversion module output graph after dead element replacement and nonuniformity correction, (c) obtaining an edge enhancement sub-module output graph, (d) obtaining a CLAHE sub-module output graph, (e) obtaining an edge enhancement module output graph, and (f) obtaining a Gamma correction sub-module output graph.
Detailed Description
The present invention will be described in detail with reference to the accompanying drawings.
Referring to fig. 1, the invention provides an infrared video stream real-time processing system structure based on short-wave and medium-wave detectors, and an FPGA system for video format and image quality processing of the infrared detectors comprises an input module, an image quality improving module, a parameter configuration module and an output processing module. The multi-band infrared detector collects video image signals, the multi-band infrared detector is connected with the input module, video image signal streams of the multi-band infrared detector are input into the input module, and after format conversion is carried out on the video image signals by the input module, the converted video stream signals are input into the image quality improving module; the image quality improving module comprises sub-modules of non-uniformity correction, bad element replacement, unsharp masking, dynamic range conversion, CLAHE, edge enhancement, Gamma correction and the like, and configuration parameters required by the video image processing modules are output by the parameter configuration module; the parameters of the parameter configuration module are input by a user through an external upper computer; the image quality improving module outputs the video stream after the video image is improved to the output processing module, the output processing module comprises two output modes, one mode is that the video stream is directly input to the SD card for data storage, and the other mode is that the video stream is converted by modules such as frequency reduction, line field generation and the like to be in accordance with the video display standard of the HDMI and finally output to a display with an HDMI interface outside, so that the display function is completed.
The original video stream of the multiband infrared detector output by the multiband infrared detector is input into an input module for format processing, the output processed video signal formats are 7, specifically, as shown in fig. 2, the format interface of the video stream of the infrared detector outputs 7 paths of differential signals in total. And inputs the differential signal into the input processing module. Wherein, CLK is a clock signal of a pixel in a video; VS is the field sync signal for video, which contains 512 complete lines, with high level being field active; HS is a video line synchronizing signal, the high level of the HS indicates that the data is valid data, and the low level indicates that the data is blanking data, namely, an invalid data area; LVDS1, LVDS2, LVDS3 and LVDS4 are pixel data of a video stream, data transitions and clock rising edges are aligned, and each clock transfers 1-bit data, and 16-bit data constitutes one pixel data. Typical parameters of the short and medium wave infrared detector outputs are shown in table 1.
TABLE 1 short-and medium-wave Infrared Detector output parameters
As shown in fig. 3, the input processing module performs the following processing on the input video stream signal of the infrared detector:
the input processing module comprises three sub-modules, namely an asynchronous FIFO sub-module, a polling reading sub-module and a line field generating sub-module, and the specific description is as follows:
(1) asynchronous FIFO submodule: the 7 paths of video image signals of the medium wave/short wave infrared detector comprise: the clock signal of the pixels in the video, the field synchronous signal of the video, the line synchronous signal of the video and the pixel data of the four-path video stream are converted into a 4-path video signal format, and the 4-path video signal format comprises the clock signal of the pixels in the video, the field synchronous signal of the video, the line synchronous signal of the video and the pixel data of the 1-path video stream. Specifically, the sub-module is configured to receive four video streams of an external infrared detector under 160M clock, convert four parallel video data streams under 160M clock (such as LVDS1, LVDS2, LVDS3, LVDS4 in fig. 2) into one serial infrared video data stream under 40M clock, and transmit the one serial infrared video data stream under 40M clock to the image quality improving module.
As shown in fig. 4, the basic format of one serial ir video Data stream under 40M clock includes a new clock CLK, a field sync signal VS, a line sync signal HS and one Data.
The input and output data of the asynchronous FIFO submodule are both 16bit wide, the data is temporarily stored at the input end through a 16-bit shift register, and after a complete 16-bit pixel data is stored, a writing mark signal is generated and the data is stored in the asynchronous FIFO submodule. This ensures that the asynchronous FIFO sub-modules operate at a relatively low clock frequency, ensuring stability of the transmission.
(2) A polling reading sub-module: the submodule is used for converting four paths of parallel video data under 160M clock in an asynchronous FIFO submodule into one path of serial video data in a polling mode by adopting a polling signal, wherein the polling signal is a counter with the bit width of 2, when the counter value is 0, a first path of parallel video stream is read, when the counter value is 1, a second path of parallel video stream is read, when the counter value is 2, a third path of parallel video stream is read, when the counter value is 3, a fourth path of parallel video stream is read, and the like, the data is read in sequence. And transmitting the converted one path of serial video data to the image quality improving module.
(3) A line field generation submodule: the system is used for generating a new line field of 40M clock and inputting the new line field to the image quality improving module. Specifically, a 160M clock down-line synchronization signal and a field synchronization signal (such as HS and VS in fig. 2) input by an external detector are subjected to clock domain crossing processing, and then a new line field of 40M clock is generated by counting rising edges of the line synchronization signal and the field synchronization signal.
Only the rising edge of the line field is captured, so that the stability of the target clock down-field signal can be ensured, and the robustness is enhanced.
In the invention, after the format of an original video data stream of a multiband infrared detector is converted by an input module, the converted video stream enters an image quality improving module, the image quality improving module comprises 7 hardware circuit sub-modules (non-uniformity correction, bad element replacement, median filtering, unsharp masking, dynamic range conversion, Sobel edge enhancement and Gamma correction), and the image quality of an infrared video image is improved by the video stream sequentially through the 7 sub-modules, as shown in figure 5. The details of these 7 sub-modules are as follows:
(1) non-uniformity syndrome module: the non-uniformity of the infrared image is represented as non-uniform vertical bars during imaging, and the non-uniformity is mainly caused by the non-uniformity of the response characteristics of each array element in the detector, 1/f noise, the influence of the scanning linearity of an optical system and a scanning motor, the non-uniformity of an electric signal transmission and an amplification channel and the like. It is not practical to completely eliminate the non-uniformity of the infrared image at the time of production, and only to improve its performance by non-uniformity correction. The non-uniformity correction adopted by the invention is two-point correction and single-point correction proposed by the predecessor. As the imaging principle of the short-wave detector is close to that of visible light imaging, the short-wave detector is insensitive to temperature, and therefore single-point correction is adopted by the short-wave detector. And the medium wave infrared detector is thermal imaging, so the medium wave detector adopts two-point calibration correction.
The non-uniformity correction sub-module is used for calculating 10 frames of data in one path of serial infrared video data stream under the 40M clock by using MATLAB to obtain correction parameters, correcting the data after 10 frames in one path of serial infrared video data stream under the 40M clock by using the existing correction formula (as follows), obtaining corrected video image data and inputting the corrected video image data into the bad element replacement sub-module.
S i，j (φ)＝g i，j (φ)φ+o i，j (φ)
Wherein s is i，j (phi) is the data coordinate, g i，j (phi) is a gain factor,
as intensity of concern in the data, o i，j (φ) is a bias weight parameter.
(2) Bad element replacement submodule: the array process in the image sensor has defects, or errors occur in the process of converting optical signals, so that the information of some pixels on the image is wrong, and the necrosis phenomenon exists in individual pixels, which is represented as excessively bright or excessively dark pixel points, thereby affecting the whole imaging effect. The invention adopts a median filtering mode to eliminate bad elements.
And the bad element replacement submodule is used for caching every two lines of the received corrected video image data by using FIFO (first in first out) to realize window operation, and outputting a median value of a 3 x 3 window through a 6-stage comparison circuit according to the window operation structure.
And then detecting the median of the 3-by-3 window, replacing the value of the window center point with the median of the window when detecting that the difference between the value of the window center point and the mean of the rest points in the window is greater than a first threshold (which can be set according to the actual situation), and outputting the median of the window to the unsharp mask module. And if the difference between the value of the center point of the window and the average value of the rest points in the window is not detected to be larger than the first threshold value, outputting the median value of the 3-by-3 window to the unsharp mask module.
(3) The unsharp mask submodule: the purpose of the unsharp mask module is to enhance the details of the picture. Gaussian filtering can smooth, i.e. low pass filter, the image. The high frequency information of the image is obtained by subtracting the pixel value of the video stream after the Gaussian smoothing filtering from the pixel value of the original video stream, and the details of the original image can be enhanced by multiplying the high frequency information by a specific coefficient and adding the result back to the original video stream.
And the unsharp masking module is used for caching the median of the window every four rows by using FIFO (first in first out) to realize 5 x 5 window operation, then calculating by using an accumulator to obtain a window convolution sum, and then obtaining a Gaussian filtering result by using a divider according to the window convolution sum. And subtracting the Gaussian filtering result from the median value of the window to obtain a high-frequency image, multiplying the high-frequency image by a coefficient (which can be 0.1, 1.0 or 1.2 according to the actual situation) to obtain a video data value after the unsharp mask operation, and inputting the video data value after the unsharp mask operation into the dynamic range conversion sub-module.
(4) Dynamic range transformation module: in general, a typical display device displays an 8-bit image, so dynamic range conversion is required. The purpose of this module is to convert a 16-bit image to an 8-bit image for display on a display. The module uses a dual-threshold linear transformation based on histogram statistics, and the threshold can be configured parametrically.
The dynamic range conversion submodule is used for counting the occurrence frequency of pixel points of a video data value after unsharp mask operation, because the 16-bit image range is too large, the dynamic range conversion submodule is divided into four parts according to the size of the pixel points for counting, the counting result is stored in four BRAMs, then the four BRAMs are traversed respectively in a line field blanking period, the maximum value and the minimum value under a given second threshold value (which can be set according to the actual situation) are found, an 8-bit output value is obtained through the existing dynamic range conversion formula (the formula is as follows) according to the maximum value and the minimum value, and the 8-bit output value is input into the CLAHE submodule.
Wherein D is i，j (phi) is the converted 8-bit output value, D i，j (φ) is a 16-bit value, max is the maximum value of a pixel in the video data values, and min is the minimum value of a pixel in the video data values.
(5) CLAHE submodule: CLAHE is known as adaptive histogram equalization with limited contrast. The invention performs contrast limiting processing for preventing image overexposure on the basis of block equalization, and the image enhancement effect is obvious.
And the CLAHE submodule is used for calculating the 8-bit output value by adopting a CLAHE algorithm to obtain the brightness value of the image and inputting the brightness value of the image into the edge enhancement submodule.
(6) Edge enhancer module: according to the method, the edge information of the image is obtained through the sobel operator, and then the positioned edge is enhanced, so that the purpose of enhancing the edge is achieved.
And the edge enhancement submodule is used for carrying out Sobel operator processing on the brightness value of the image to obtain the brightness value after edge enhancement. And the edge-enhanced luminance value is input to the Gamma correction submodule.
(7) Gamma syndrome module: the Gamma correction algorithm can enhance the bright or dark portions of the infrared image, thereby highlighting more bright or dark detail. In order to save hardware resources and reduce the complexity of design, the module is realized by a lookup table mode, and the manual adjustment of parameters can be realized by a reserved parameter transmission interface.
And the Gamma correction submodule is used for adjusting the brightness value after the edge enhancement until the brightness value meets the requirement, and transmitting the adjusted brightness value to the output processing module.
(5) A parameter configuration module: the parameter configuration module is used for completing the parameter configuration of the first threshold and the second threshold in the image quality improving module, and the configuration module can complete whether all the image quality improving modules are enabled or directly connected or not and complete the parameter dynamic configuration of the image quality improving module. As shown in fig. 6, the SPI transmission control interface in the parameter configuration module communicates with the parameter configuration module through the single chip or the upper computer, and the SPI protocol is selected as the communication protocol, which is simple to operate and high in transmission efficiency. When the parameters are configured, one byte is transmitted each time, and the configuration and reading of the parameters are realized. Each byte includes control instructions (6-7 bits) and transmission content (0-5 bits), as shown in fig. 7. Bit 7 of the control instruction is a 1 to indicate writing and a 0 to indicate reading. The 6 th bit of the control instruction is 0, which indicates that the subsequent 0 th-5 th bits are the parameter register address, referring to table 2, the 6 th bit of the control instruction is 1, which indicates that the subsequent 0 th-5 th bits are the value of the parameter.
TABLE 2 parameter register Address distribution map
An output processing module: the output processing module provided by the invention comprises an SD card storage submodule, a frequency reduction submodule and a video time sequence adjusting submodule, and finally stores or outputs a video stream, and the framework of the module is shown in figure 8. The three sub-modules included in the output processing module are specifically described as follows:
the output processing module is used for outputting the received adjusted brightness value, the converted one-path serial video data transmission and a new line field, and the output forms are two, the first one is: directly storing the signal into an SD card so as to check the original signal; and the second method comprises the following steps: and displaying in real time through an HDMI display protocol.
Since the timing sequence of the input video stream is not a standard video timing sequence, in order to normally display the non-standard video timing sequence, corresponding frequency conversion and resolution adjustment are required to be performed so as to display the non-standard video timing sequence on the standard HDMI.
The converted one path of serial video data transmission and the new line field are transmitted to the output processing module only through the image quality improving module without being processed in the image quality improving module.
(1) The SD card storage submodule comprises: as shown in the overall architecture diagram of fig. 1, the SD card storage sub-module is used for storing the received adjusted luminance value, the converted one-channel serial video data transmission, and a new line field.
(2) A frequency reduction submodule:
short-wave and medium-wave detectors cannot display such a high frame rate, so that corresponding frequency reduction processing is required.
The down-conversion sub-module is used for down-converting one path of serial video data transmission and a new line field, and is generally characterized in that for a short-wave detector with a frame rate of 225, the down-conversion sub-module is used for down-converting one path of serial video data transmission and the new line field to 60 frames; for a medium wave infrared detector with a frame rate of 100, a path of serial video data transmission and a new line field frequency after conversion are reduced to 50 frames. And outputting the data after frequency reduction.
The specific realization method of the frequency-reducing submodule uses a ping-pong RAM to achieve read-write separation. Taking a short-wave detector as an example, the original data is 225 frames per second, the target frame rate is 60 frames, and 225/60 is 3.75 frames, which means that each frame of the target frame rate (60 frames) corresponds to 3.75 frames of the original frame rate (225 frames), so a frame counter is created to determine the writing time, the frame counter is cleared whenever a frame of the target video stream starts, the number of the original video frames is counted by using the frame counter, and when the number of the 3 rd frames reaches, a write RAM signal is generated and the frame is completely written into the RAM. Similarly, the medium wave detector writes the video stream into the RAM when the frame counter counts the 1 st frame, and writes one frame every other frame in terms of angle, so as to achieve the purpose of reducing the frequency to 50 frames. In addition, the video stream written into the ping-pong RAM is read by using the target clock and the line field signal of the target resolution, so that the purpose of reading the video stream is achieved.
(3) The video time sequence adjusting submodule: and the video frequency conversion sub-module is used for further carrying out video line, field and data signal time sequence adjustment on the video stream output of the frequency conversion sub-module, so that the adjusted line, field and data signal line conform to the HDMI standard, and further, the HDMI is utilized for displaying. The method adopted by the invention is that the other two ping-pong RAMs are used for buffering, the ping-pong RAM _1 is responsible for inputting and storing the line, field and data signals before adjustment, the ping-pong RAM _2 is responsible for outputting the line, field and data signals which accord with the HDMI standard and outputting the line, field and data signals to an external FMC _ LPC interface, and the external side of the FMC _ LPC is connected with an HDMI display, so that the adjusted video data which accord with the HDMI standard can be displayed.
The effect of the FPGA system of the present invention is verified on the sailing certification FPGA, and the result is shown in fig. 9 and fig. 10, where fig. 9 is an experimental result of the input of the output signal of the medium wave detector to the image quality improvement module of the FPGA verification system of the present invention, and fig. 10 is an experimental result of the input of the output signal of the medium wave detector to the image quality improvement module of the FPGA verification system of the present invention. As can be seen from fig. 9 and 10, the image quality is significantly improved.
Claims (7)
1. An FPGA system for video format and image quality processing is characterized by comprising an input module, an image quality improving module and an output processing module;
the input module is used for carrying out format conversion on the received video image signal of the infrared detector and inputting the converted video stream signal into the image quality improving module;
the image quality improving module is used for outputting the video stream after the image quality of the received video image is improved to the output processing module; the input processing module comprises an asynchronous FIFO submodule, a polling reading submodule and a line field generating submodule;
the asynchronous FIFO submodule is used for receiving four paths of parallel video streams under 160M clocks of the infrared detector, converting the four paths of parallel video data streams under 160M clocks into one path of serial infrared video data stream under 40M clocks, and transmitting the one path of serial infrared video data stream under 40M clocks to the image quality improving module;
the polling reading sub-module is used for receiving four paths of parallel video streams under 160M clock of the infrared detector, converting four paths of parallel video data under 160M clock into one path of serial video data in a polling mode by adopting polling signals, and transmitting the converted one path of serial video data to the image quality improving module;
the line field generation submodule is used for performing clock domain crossing processing on a received line synchronization signal and a received field synchronization signal of an input video under 160M clock, generating a new line field of 40M clock through the rising edge count of the line signal and the field signal, and inputting the new line field to the image quality improvement module;
the output processing module is used for inputting the received video stream into the SD card or converting the received video stream into a video stream which conforms to the video display standard of the HDMI and then outputting the video stream to the display;
the image quality improving module comprises a non-uniformity correction sub-module, a bad element replacing sub-module, a median filtering sub-module, an unsharp masking sub-module, a dynamic range conversion sub-module, a Sobel edge enhancement sub-module and a Gamma correction sub-module;
the non-uniformity correction submodule is used for calculating 10 frames of data in one path of serial infrared video data stream under the 40M clock by adopting MATLAB to obtain correction parameters, correcting data after 10 frames in one path of serial infrared video data stream under the 40M clock to obtain corrected video image data and inputting the corrected video image data into the bad element replacement submodule;
the bad element replacement submodule is used for caching every two lines of the received corrected video image data by using FIFO (first in first out) to realize window operation, and outputting a median value of a 3 x 3 window through a 6-stage comparison circuit according to the result of the window operation; detecting a median value of the window, replacing the value of the window center point with the median value of the window if the difference between the value of the window center point and the mean value of the rest points in the window is greater than a first threshold value, and outputting the median value of the window to an unsharp mask module; if the difference between the value of the center point of the window and the mean value of the rest points in the window is less than or equal to a first threshold value, outputting the median value of the window to an unsharp mask module;
the unsharp mask submodule is used for caching four rows of the median value of a window, realizing 5-by-5 window operation, calculating through an accumulator to obtain a window convolution sum, obtaining a Gaussian filtering result by using a divider according to the window convolution sum, subtracting the Gaussian filtering result from the median value of the window to obtain a high-frequency image, multiplying the high-frequency image by a coefficient to obtain a video data value after the unsharp mask operation, and inputting the video data value after the unsharp mask operation into the dynamic range conversion submodule;
the dynamic range conversion submodule is used for counting the occurrence frequency of pixels of the video data value after the unsharp masking operation is carried out, storing a counting result in four BRAMs, respectively traversing the four BRAMs in a horizontal field blanking period, finding out the maximum value of pixels in the video data value and the minimum value of pixels in the video data value under a second threshold value, obtaining an 8-bit output value according to the maximum value of pixels in the video data value and the minimum value of pixels in the video data value, and inputting the 8-bit output value into the CLAHE submodule;
the CLAHE submodule is used for calculating the 8-bit output value by adopting a CLAHE algorithm to obtain the brightness value of the image and inputting the brightness value of the image into the edge enhancement submodule;
the edge enhancement submodule is used for carrying out Sobel operator processing on the brightness value of the image to obtain a brightness value after edge enhancement, and inputting the brightness value after edge enhancement into the Gamma correction submodule;
and the Gamma correction submodule is used for adjusting the brightness value after the edge enhancement to meet the requirement and transmitting the adjusted brightness value to the output processing module.
2. The FPGA system according to claim 1, wherein a 160M clock down-line sync signal and a field sync signal inputted from an external detector are processed across clock domains, and a new line field of 40M clock is generated by counting rising edges of the line sync signal and the field sync signal.
3. The FPGA system of claim 1, wherein the polling signal is a counter with a bit width of 2, and the first parallel video stream is read when the counter value is 0, the second parallel video stream is read when the counter value is 1, the third parallel video stream is read when the counter value is 2, and the fourth parallel video stream is read when the counter value is 3.
4. A method for video format and image quality processing using the system of any one of claims 1-3, comprising the steps of:
performing format conversion on the video image signal;
carrying out image quality improvement on the converted video stream signal;
and inputting the video stream with the improved image quality into the SD card, or converting the video stream with the improved image quality into a video stream conforming to the video display standard of the HDMI and outputting the video stream.
5. The method of claim 4, wherein converting the format of the video image signal comprises:
converting four paths of parallel video data streams under 160M clocks into one path of serial infrared video data stream under 40M clocks;
four paths of parallel video data under 160M clock are converted into one path of serial video data in a polling mode by adopting polling signals;
the horizontal synchronizing signal and the field synchronizing signal of the input video under 160M clock are processed by crossing clock domains, and then a new horizontal field of 40M clock is generated by counting the rising edges of the horizontal signal and the field signal.
6. The method of claim 4, wherein the step of improving the quality of the converted video stream signal comprises the steps of:
MATLAB calculation is carried out on 10 frames of data in one path of serial infrared video data stream under the 40M clock to obtain correction parameters, and data after 10 frames in one path of serial infrared video data stream under the 40M clock are corrected to obtain corrected video image data;
caching every two lines of corrected video image data by using FIFO (first in first out) to realize window operation, and outputting a median value of 3 x 3 windows by using a 6-stage comparison circuit according to a window operation result; detecting the median of the window, and replacing the value of the window center point with the median of the window if the difference between the value of the window center point and the mean of the rest points in the window is greater than a first threshold value;
caching four rows of median values of a window to realize 5-by-5 window operation, then calculating through an accumulator to obtain a window convolution sum, then utilizing a divider to obtain a Gaussian filtering result according to the window convolution sum, subtracting the Gaussian filtering result from the median value of the window to obtain a high-frequency image, and multiplying the high-frequency image by a coefficient to obtain a video data value after the unsharp mask operation;
counting the occurrence times of pixel points of the video data value after the unsharp masking operation, storing the counting result in four BRAMs, traversing the four BRAMs in a line field blanking period respectively, finding out the maximum value of pixels in the video data value and the minimum value of pixels in the video data value under a second threshold value, and obtaining an 8-bit output value according to the maximum value of pixels in the video data value and the minimum value of pixels in the video data value;
calculating the 8-bit output value by using a CLAHE algorithm to obtain the brightness value of the image;
and carrying out Sobel operator processing on the brightness value of the image to obtain the brightness value after edge enhancement.
7. The method according to claim 4, wherein inputting the video stream with improved image quality into an SD card, or converting the video stream with improved image quality into a video stream conforming to the video display standard of HDMI and outputting the video stream to a display, comprises the following steps:
inputting the video stream with the improved image quality into an SD card; or the video stream with the improved image quality is subjected to frequency reduction to 50 or 60 frames, and then the video stream subjected to frequency reduction is subjected to video line, field and data signal time sequence adjustment to be output after being subjected to video stream meeting the video display standard of the HDMI.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111295001.XA CN114040156B (en) | 2021-11-03 | 2021-11-03 | FPGA system and method for video format and image quality processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111295001.XA CN114040156B (en) | 2021-11-03 | 2021-11-03 | FPGA system and method for video format and image quality processing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114040156A CN114040156A (en) | 2022-02-11 |
| CN114040156B true CN114040156B (en) | 2022-08-16 |
Family
ID=80142746
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111295001.XA Active CN114040156B (en) | 2021-11-03 | 2021-11-03 | FPGA system and method for video format and image quality processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114040156B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102811354A (en) * | 2011-05-30 | 2012-12-05 | 深圳市快播科技有限公司 | Video image quality enhanced playing method and on-demand terminal |
| EP2803963A1 (en) * | 2012-01-12 | 2014-11-19 | Mission Infrared Electro Optics Technology Co. Ltd | Thermal imagery device and normalized thermal imagery shooting method |
| CN105827999A (en) * | 2016-05-04 | 2016-08-03 | 吴礼刚 | FPGA based infrared image acquisition and processing system |
| EP3410688A1 (en) * | 2017-06-01 | 2018-12-05 | Axis AB | Method for improving image quality in images acquired by a near-infrared sensitive video camera and such a camera |
| CN111030670A (en) * | 2019-12-20 | 2020-04-17 | 中国科学院长春光学精密机械与物理研究所 | Infrared imaging processing circuit and infrared detection system |
| KR20200076100A (en) * | 2018-12-19 | 2020-06-29 | (주)인성 | Apparatus for processing short-wave infrared images |
| CN111402165A (en) * | 2020-03-18 | 2020-07-10 | 展讯通信（上海）有限公司 | Image processing method, device, equipment and storage medium |
-
2021
- 2021-11-03 CN CN202111295001.XA patent/CN114040156B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102811354A (en) * | 2011-05-30 | 2012-12-05 | 深圳市快播科技有限公司 | Video image quality enhanced playing method and on-demand terminal |
| EP2803963A1 (en) * | 2012-01-12 | 2014-11-19 | Mission Infrared Electro Optics Technology Co. Ltd | Thermal imagery device and normalized thermal imagery shooting method |
| CN105827999A (en) * | 2016-05-04 | 2016-08-03 | 吴礼刚 | FPGA based infrared image acquisition and processing system |
| EP3410688A1 (en) * | 2017-06-01 | 2018-12-05 | Axis AB | Method for improving image quality in images acquired by a near-infrared sensitive video camera and such a camera |
| KR20200076100A (en) * | 2018-12-19 | 2020-06-29 | (주)인성 | Apparatus for processing short-wave infrared images |
| CN111030670A (en) * | 2019-12-20 | 2020-04-17 | 中国科学院长春光学精密机械与物理研究所 | Infrared imaging processing circuit and infrared detection system |
| CN111402165A (en) * | 2020-03-18 | 2020-07-10 | 展讯通信（上海）有限公司 | Image processing method, device, equipment and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| Quan Lu等.An Effective Low-Contrast SF₆Gas Leakage Detection Method for Infrared Imaging.《IEEE Transactions on Instrumentation and Measurement》.2021,第70卷(第99期), * |
| 面向格式转换的数字视频处理方法及其硬件实现;张光烈等;《中国工程科学》;20010630;第3卷(第6期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114040156A (en) | 2022-02-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10825426B2 (en) | Merging multiple exposures to generate a high dynamic range image | |
| US20220217271A1 (en) | Global Tone Mapping | |
| US8639050B2 (en) | Dynamic adjustment of noise filter strengths for use with dynamic range enhancement of images | |
| US5848181A (en) | Image processing method, image processing apparatus, noise removing method, and noise removing apparatus | |
| US8363131B2 (en) | Apparatus and method for local contrast enhanced tone mapping | |
| US7366347B2 (en) | Edge detecting method | |
| CN102265595B (en) | Method and apparatus for image processing and on-vehicle camera apparatus | |
| US8106986B2 (en) | Image sensor, data output method, image pickup device, and camera | |
| CN107862672B (en) | Image defogging method and device | |
| WO2019104047A1 (en) | Global tone mapping | |
| CN110686783A (en) | Blind pixel detection method and image processing method for InGaAs short-wave infrared imager | |
| US7983454B2 (en) | Image processing apparatus and image processing method for processing a flesh-colored area | |
| CN114040156B (en) | FPGA system and method for video format and image quality processing | |
| TWI650998B (en) | Format conversion circuit and related method applied to set-top box | |
| CN106686363B (en) | Signal mixed processing method and device in display terminal | |
| US10178359B2 (en) | Macropixel processing system, method and article | |
| US7352397B2 (en) | Circuit and method for contour enhancement | |
| US8072516B2 (en) | Dynamic range enhancement method and apparatus | |
| CN105787883A (en) | Wide dynamic image device and algorithm | |
| CN106454274A (en) | Surveillance camera system adapting to special environment | |
| CN203219431U (en) | Image processing module | |
| US8355073B2 (en) | Image processing apparatus and image processing method | |
| CN106899813A (en) | NEXT series of products SOPC and FPGA picture systems | |
| KR100747729B1 (en) | Image processor, device for compensating of lens shading and the same method | |
| JP2002251171A (en) | Device for improving gray scale of image |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |