WO2023213002A1

WO2023213002A1 - Video data processing system based on fpga

Info

Publication number: WO2023213002A1
Application number: PCT/CN2022/103811
Authority: WO
Inventors: 张田
Original assignee: 张田
Priority date: 2022-05-06
Filing date: 2022-07-05
Publication date: 2023-11-09
Also published as: CN117082195A

Abstract

The present invention relates to the technical field of video data processing, and disclosed is a video data processing system based on an FPGA, comprising an hdmi receiving module, an hdmi transmitting module, and an LED driving module. An input end of the hdmi receiving module is in signal connection with an output end of an external hdmi input terminal; an input end of the hdmi transmitting module is in signal connection with an output end of an external hdmi output terminal; an input end of the LED driving module is in signal connection with an input end of an external LED lamp strip interface; and an output end of the hdmi receiving module is in signal connection with the input end of the hdmi transmitting module. The video data processing system based on the FPGA has the advantages of parallel processing of mass data, low latency, high processing speed, and no need of debugging in configuration of a user, and solves the problems that existing systems are low in integration level, low in reliability, high in cost, serious in system latency, and prone to heating.

Description

A video data processing system based on FPGA

Technical field

The invention relates to the technical field of video data processing, specifically a video data processing system based on FPGA.

Background technique

FPGA is a field programmable gate array. In layman's terms, it is a blank digital logic circuit that can be programmed arbitrarily to achieve different functions by modifying the internal program. Due to its strong ability to process big data, it is widely used in high-speed data processing fields, such as high-definition video data streams.

In the existing technology, traditional video atmosphere lights have the following solutions:

1) Installed on the PC, it is used to record the screen in real time on the computer, then analyze the edge data of the screen, calculate the weighted average, and send it to the MCU through the USB to serial port. The MCU receives the RGB data of the light strip and controls the light strip. RGB brightness.

2) Through the way of 1 in and 2 out of the HDMI splitter, one output channel is directly looped out, and the other channel is sent to the arm processor through HDMI to USB. After the arm processor receives the RGB data, it performs video processing internally, and finally The RGB brightness value of the edge is calculated and sent to the LED strip.

The above two solutions have the following shortcomings:

Option 1) Although the cost is low and the price is low, it can only run on the Windows platform. Currently, Linux, Mac, and Android TV are not available, and cannot be used on multiple platforms such as TVs or Android systems; due to the need to record the screen in real time for video processing, It takes up a lot of computer resources; the delay is large and the frame rate is low. Since the data needs to be forwarded multiple times, the delay on the light strip is larger than the screen, causing out-of-synchronization problems.

Option 2) Although it solves the cross-platform problem, it requires HDMI2.0 splitter, HDMI high-definition conversion chip and high-performance arm processing chip, which causes high cost. Moreover, HDMI2.0 has massive data and relies on CPU to calculate. It will cause overload and large delay problems.

In order to solve the above problems, we proposed a video data processing system based on FPGA, which has the advantages of parallel processing of massive data, low cost, low latency, fast processing speed, and user-free debugging configuration.

Technical solutions

In view of the shortcomings of the existing technology, the present invention provides a video data processing system based on FPGA to solve the problems raised in the above background technology.

In order to achieve the above purpose, the present invention provides the following technical solution: a video data processing system based on FPGA, including an HDMI receiving module, an HDMI transmitting module and an LED driving module. The input end of the HDMI receiving module is connected to the external HDMI input terminal. The output end is connected to the signal, the input end of the HDMI transmitting module is connected to the output end of the external HDMI output terminal, and the input end of the LED drive module is connected to the input end of the external LED light strip interface, so The output end of the HDMI receiving module is connected to the input end of the HDMI transmitting module for signal connection, and is used to restore the TMDS data of the HDMI cable to the original video data;

The FPGA-based video data processing system also includes a decryption module. The output signal of the decryption module is connected to the input end of the color space conversion module. The output signal of the color space conversion module is connected to the resolution scaling module. At the input end, the output end signal of the resolution scaling module is connected to the input end of the frame extraction module, the output end signal of the frame extraction module is connected to the input end of the HDR decoding module, and the output end signal of the HDR decoding module is connected There is an input end of the data buffer module, the output end signal of the data buffer module is connected to the input end of the memory module, the output end signal of the memory module is connected to the input end of the pixel average calculation module, and the pixel average calculation module The output signal is connected to the input end of the frame delay module, and the output signal of the frame delay module is connected to the input end of the LED driver module.

To further optimize this technical solution, the FPGA-based video data processing system is also provided with a parameter module, which is used to calculate each lamp bead by detecting the input resolution and the number of detected chips of the LED strip. For the corresponding rectangular area, the chip model of the LED light strip is ws2812 chip.

To further optimize this technical solution, the input signal of the memory module is connected to a memory read and write address module. The memory read and write module is an address counter of RAM. When writing data, it is written line by line; when reading At this time, the data is randomly taken out according to the address required by the light strip. The random address is calculated based on the address mapping of the RAM coordinates according to the position of the light strip.

To further optimize this technical solution, the output end of the LED drive module is also signal-connected to the input end of the LED drive quantity detection module. The LED drive quantity detection module is used to detect external voltages through the current increment method when the system is turned on. The number of chips on the LED strip.

To further optimize this technical solution, the color space conversion module is used to convert YCbCr to RGB color space. RGB grayscale values in different standard formats can be calculated through a series of matrix operations; when the input HDMI signal source is After formatting YCbCr, the color space conversion module will automatically perform color space conversion and convert the YCbCr color space to RGB color space.

To further optimize this technical solution, the resolution scaling module is used to convert 4K signals to the resolution of 1080P format. Under the bandwidth condition of HDMI2.018G, the purpose of reducing the bandwidth of the pipeline is achieved by reducing the resolution of the input signal. When the system bandwidth exceeds 165M*24bit, the system enables the resolution reduction module. At this time, when the input signal is 3840*2160, the actual output resolution is 1920*1080, and the FPGA system bandwidth is reduced to 1/4 of the original.

To further optimize this technical solution, the frame extraction module is used to convert signals higher than 60Hz into signals not higher than 60Hz. The frame extraction module detects the frame rate of the input signal in real time. When it exceeds 60Hz, the frame rate is Divide by 2. If it still exceeds 60Hz, continue dividing by 2 until it is less than 60Hz to ensure that the load points of the back-end ws2812 chip are not affected.

To further optimize this technical solution, the HDR decoding module is used to determine whether the input signal is in HDR format by detecting the header of the HDR data packet, identify the HDR data, and perform a remapping calculation on the grayscale of the data. Avoid the problem of RGB data desaturation.

To further optimize this technical solution, the data buffer module is used to buffer data and isolate the internal and external clock domains of the FPGA; the signal from HDMI is sent line by line, written to the buffer according to the control of the DE signal, and FIFO is used to match The speed difference is used to match the internal speed of the FPGA.

To further optimize this technical solution, the parameter module is used to calculate the rectangular pixel area of the TV set corresponding to each LED lamp bead by detecting the resolution of the HDMI signal and the number of lamp beads of the LED lamp strip, thereby obtaining The average value of the RGB in the rectangular area is used as the RGB gray value; at the same time, the parameter module can configure various parameters of the FPGA and define the parameters of the FPGA as a table.

beneficial effects

Compared with the existing technology, the present invention provides a video data processing system based on FPGA, which has the following beneficial effects:

This FPGA-based video data processing system is used for TV background atmosphere lights, controls LED lamp beads and detects the number of driver chips to change colors synchronously with the TV periphery to enhance the atmosphere. It has the ability to process massive data in parallel and is low-cost. , low latency, fast processing speed, and user-free debugging configuration, etc., it solves the problems of low integration, low reliability, high cost, serious system delay and easy heat generation of the current system.

Description of the drawings

Figure 1 is a schematic structural diagram of an FPGA-based video data processing system proposed by the present invention;

Figure 2 is a schematic diagram of an LED drive quantity detection module of an FPGA-based video data processing system proposed by the present invention;

Figure 3 is a schematic structural diagram of a pixel mean calculation module of an FPGA-based video data processing system proposed by the present invention;

Figure 4 is a rendering of screen-light synchronization of an FPGA-based video data processing system proposed by the present invention;

FIG. 5 is a schematic diagram of the driving method of the LED driving module of the FPGA-based video data processing system proposed by the present invention.

Embodiments of the invention

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

Example 1:

Please refer to Figure 1. A video data processing system based on FPGA includes an HDMI receiving module, an HDMI transmitting module and an LED driver module. The input end of the HDMI receiving module is signal connected to the output end of the external HDMI input terminal, so The input end of the HDMI transmitting module is connected to the output end of the external HDMI output terminal, the input end of the LED drive module is connected to the input end of the external LED light strip interface, and the output end of the HDMI receiving module is connected to the input end of the LED light strip interface. Connect the signal to the input end of the HDMI transmitter module to restore the TMDS data of the HDMI cable to the original video data.

Among them, the HDMI receiving module converts 3 sets of HDMI differential data and 1 set of differential clock into raw data; the HDMI transmitting module converts raw data into 3 sets of HDMI differential data and 1 set of differential clock.

The FPGA-based video data processing system also includes a decryption module, which is used for HDMI data decryption.

The output signal of the decryption module is connected to the input end of the color space conversion module. The color space conversion module includes the conversion of YCbCr to RGB color space. RGB grayscales in different standard formats can be calculated through a series of matrix operations. value.

The output signal of the color space conversion module is connected to the input end of the resolution scaling module, and the 4K signal is converted into a resolution of 1080P format.

The output signal of the resolution scaling module is connected to the input end of the frame extraction module, and the frame extraction module converts a signal higher than 60 Hz (for example, 240 Hz) into a signal not higher than 60 Hz.

The output signal of the frame extraction module is connected to the input end of the HDR decoding module. The HDR decoding module identifies the HDR data by detecting the header of the HDR data packet, thereby avoiding the problem of RGB data saturation decrease.

The output signal of the HDR decoding module is connected to the input end of the data buffer module. The data buffer module is an HDMI data buffer and is used to buffer data and isolate the internal and external clock domains of the FPGA.

The output signal of the data buffer module is connected to the input end of the memory module. The memory module caches one or more frames of images to prepare data for the subsequent averaging module to ensure that there will be no more errors when the data is processed. Reading and writing changes.

The output signal of the memory module is connected to the input end of the pixel average calculation module. Since each LED lamp bead corresponds to the rectangular area of the a*b area on the screen, the pixel average calculation module calculates the The average value of RGB.

The output signal of the pixel average calculation module is connected to the input end of the frame delay module, and the frame delay module is used to cache one frame to multiple frames.

The output signal of the frame delay module is connected to the input end of the LED driver module. The LED driver module is a driver module of the ws2812 chip and converts 24-bit RGB data into single-line serial RGB. NRZ format, used to drive LED light strips.

Furthermore, the FPGA-based video data processing system is also provided with a parameter module, which is used to calculate the number of chips corresponding to each lamp bead by detecting the input resolution and the number of detected LED lamp strip chips. The rectangular area of the LED light strip is a ws2812 chip. The parameter module is used to calculate the TV set corresponding to each LED light bead by detecting the resolution of the HDMI signal and the number of LED light beads. rectangular pixel area, thereby obtaining the average value of RGB in the rectangular area, and the average value is used as the RGB gray value; at the same time, the parameter module can configure various parameters of the FPGA and define the parameters of the FPGA as a table to achieve .

Furthermore, the input signal of the memory module is connected to a memory read and write address module. The memory read and write module is an address counter of the RAM. When writing data, it writes line by line; when reading, According to the address required by the light strip, the data is randomly taken out. The random address is based on the position of the light strip and the coordinates of the RAM are calculated through address mapping.

Furthermore, the output end of the LED drive module is also connected to the input end of the LED drive quantity detection module. The LED drive quantity detection module is used to detect external LED lights through the current increment method when the system is turned on. Bring the number of chips above.

Example 2:

Based on the FPGA-based video data processing system described in Embodiment 1, the implementation process between each module is further explained.

The HDMI receiving module and HDMI transmitting module can be implemented with either a dedicated chip or an FPGA transceiver to restore the TMDS data of the HDMI cable to the original video data.

The decryption module adopts high-bandwidth digital content protection technology. The era of HDTV (High Definition Television) is coming. In order to ensure that high-definition signals transmitted by HDMI or DVI will not be illegally recorded, HDCP technology has emerged. Only after entering the authorized key can the HDMI signal be correctly decrypted. If there is no HDCP signal, the module will automatically bypass it.

The color space conversion module, when the input HDMI signal source is in YCbCr format, the module will automatically perform color space conversion. Since the back-end LED lamp beads adopt RGB format, the YCbCr color space needs to be converted into RGB color. space. Conversion of RGB and YCbCr color spaces. Commonly used color gamut spaces are BT.601 (SDTV, standard definition TV), BT.709 (HDTV, high definition TV), BT.2020 (UHDTV, ultra high definition TV). In different color gamuts In space, the transformation matrix is inconsistent. Therefore, you only need to configure different coefficients to calculate and adapt to different color spaces.

In the format of YCbCr:

Y: Brightness (Luminance or Luma), which is the grayscale value. "Brightness" is established from the RGB input signal by adding specific parts of the RGB signal together.

Cb: reflects the difference between the blue part of the RGB input signal and the brightness value of the RGB signal.

Cr: reflects the difference between the red part of the RGB input signal and the brightness value of the RGB signal.

The value range of each component of RGB and YCbCr is 0-255, and there is a linear transformation relationship between YCbCr and RGB.

The resolution scaling module, because the throughput of the FPGA's processing data volume is directly related to the cost, in order to reduce the cost of the FPGA and the system bandwidth, it can be achieved by reducing the resolution of the input signal under the bandwidth condition of HDMI2.018G. For the bandwidth purpose of the pipeline, when the system bandwidth exceeds 165M*24bit, the system will automatically enable the resolution scaling module. At this time, when the input signal is 3840*2160, the actual output resolution is 1920*1080. The system bandwidth of the FPGA Reduced to 1/4 of the original, but the frame rate does not change in any way.

The frame extraction module, in the specifications of HDMI2.0, 1080P can go up to 240Hz and the timing is within this range. Then there is a problem. In the datasheet of the ws2812 chip, when the number of loading points is 512 points, the maximum frame The frame rate is 60Hz. If it exceeds this frame rate, the system frame rate must be reduced. The solution is to detect the frame rate of the input signal in real time. When it exceeds 60Hz, divide the frame rate by 2. If it still exceeds, continue dividing by 2 until it is less than 60Hz. This will not affect the load points of the back-end ws2812 chip.

The data buffer module is usually used as a data buffer. For example, the signal from HDMI is sent line by line and written to the buffer according to the control of the DE signal. In order to match the speed with the inside of the FPGA, a FIFO needs to be used to match the speed difference. On the other hand, since HDMI data travels across clock domains, in order to solve this problem, FIFO is used to solve the cross-clock domain problem.

The memory module needs to cache one frame of data in order to facilitate back-end data processing. At this time, a large-capacity RAM is selected to store the HDMI RGB data. This RAM can be implemented using an independent chip or a high-speed internal FPGA. RAM to implement caching. In order to realize the pipeline operation mode, we define RAM as ping-pong operation mode, that is, when port A is written, port B can be read. When port A is reading, port B can be written. At this time, the maximum ram capacity we need is: 2*1920*1080*24bit.

The memory read and write address module, that is, the address counter of ram, since there will be read data and write data at the same time, we need a set of read and write address modules to let ram know which location of data we are accessing, and when writing When entering, we write it line by line. When reading out, we randomly take out the data we need according to the address required by the light strip. The random address is based on the position of the light strip and the coordinates of the RAM are calculated through address mapping.

The LED driver quantity detection module uses the current increment method to detect the number of chips on the lamp strip when it is turned on. Since each customer's TV size is different, each customer's light strip length is different, so users do not need to configure their own number of LED lamp beads to achieve a debugging-free effect. As shown in Figure 2, the schematic diagram of the LED driver quantity detection module, we take 4 driver chips as an example to illustrate the detection of the number of driver chips through software:

1) Power on and initialize, set all WS2812 grayscales to 0, and record the current on VDD as the I0 initial current;

2) The software lights U1, U2U3U4 goes off, the recorded current is I1, and the incremental current = I1-I0;

3) The software lights U2, U1U3U4 goes off, the recorded current is I2, and the incremental current = 0;

4) The software lights U3, U1U2U4 goes off, the recorded current is I3, and the incremental current = 0;

5) The software lights U4, U1U2U3 goes off, the recorded current is I4, and the incremental current = 0;

6) The software lights up U5 and U1U2U3U4 goes off. In fact, the software does not know that U5 does not actually exist. The recorded current is I5. The incremental current can detect a value close to: -(I1-I0) at this time, then it is judged that the value at this time is The number of chips is 4. In the same way, this method can be used in other situations where current change detection is required to determine the number of working modules.

The parameter module module can calculate the rectangular pixel area of the TV corresponding to each LED lamp bead by detecting the resolution of the HDMI signal and the number of LED lamp beads, thereby obtaining the average RGB in the rectangular area. value, and the average value is used as the RGB grayscale value. The second function of this module is to configure various parameters of the FPGA. Since there are various transformations in the input parameters, and the FPGA is not good at floating point operations, we need to define the parameters of the FPGA as tables to achieve this.

The pixel average calculation module is shown in Figure 3. For example, we have N horizontal LEDs, M vertical LEDs, and the input resolution of the TV is X0*Y0. In this way, we can calculate the pixel value of each LED through the formula Corresponding TV pixels.

A=X0/N;

B=Y0/M;

Therefore, we only need to find the average value of RGB in the area A*B, which is the RGB value corresponding to the LED lamp bead.

The frame delay module, as shown in Figure 4, because MEMC requires the internal buffering of many frames in the TV, causes the light strip to be displayed earlier than the TV picture. On some TVs with MEMC functions, the delay of the TV is For time matching, the signal on the light strip needs to be partially delayed to achieve the effect of screen light synchronization.

The LED driver module is shown in Figure 5. The driving method is: convert the 24-bit RGB data inside the system from parallel to serial 24-bit and store it in the buffer, and then follow the instructions of G7-G0, R7-R0, and B7-B0. In sequence, it is converted into NRZ square waves corresponding to 1 and 0 through the PWM module, thereby realizing single-line transmission of LED light. After receiving the signal, the driving signal will be forwarded to the last chip step by step.

The beneficial effects of the present invention are:

In the description of this specification, reference to the terms "one embodiment," "some embodiments," "an example," "specific examples," or "some examples" or the like means that specific features are described in connection with the embodiment or example. , structures, materials or features are included in at least one embodiment or example of the invention. In this specification, the schematic expressions of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the specific features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and combine different embodiments or examples and features of different embodiments or examples described in this specification unless they are inconsistent with each other.

Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art will understand that various changes, modifications, and substitutions can be made to these embodiments without departing from the principles and spirit of the invention. and modifications, the scope of the invention is defined by the appended claims and their equivalents.

Claims

A video data processing system based on FPGA, which is characterized in that it includes an HDMI receiving module, an HDMI transmitting module and an LED driving module. The input end of the HDMI receiving module is signal-connected to the output end of the external HDMI input terminal, and the The input end of the HDMI transmitting module is connected to the output end of the external HDMI output terminal. The input end of the LED drive module is connected to the input end of the external LED light strip interface. The output end of the HDMI receiving module is connected to the input end of the external HDMI output terminal. The input end of the HDMI transmitter module is connected to the signal, which is used to restore the TMDS data of the HDMI cable to the original video data;

The FPGA-based video data processing system also includes a decryption module. The output signal of the decryption module is connected to the input end of the color space conversion module. The output signal of the color space conversion module is connected to the resolution scaling module. At the input end, the output end signal of the resolution scaling module is connected to the input end of the frame extraction module, the output end signal of the frame extraction module is connected to the input end of the HDR decoding module, and the output end signal of the HDR decoding module is connected There is an input end of the data buffer module, the output end signal of the data buffer module is connected to the input end of the memory module, the output end signal of the memory module is connected to the input end of the pixel average calculation module, and the pixel average calculation module The output signal is connected to the input end of the frame delay module, and the output signal of the frame delay module is connected to the input end of the LED driver module.
An FPGA-based video data processing system according to claim 1, characterized in that the FPGA-based video data processing system is also provided with a parameter module, and the parameter module is used to detect input resolution and detection The number of chips in the LED light strip is obtained to calculate the rectangular area corresponding to each lamp bead. The chip model of the LED light strip is ws2812 chip.
A video data processing system based on FPGA according to claim 1, characterized in that the input signal of the memory module is connected to a memory read and write address module, and the memory read and write module is an address counter of RAM, and the write When data is input, it is written line by line; when read, data is randomly taken out according to the address required by the light strip. The random address is calculated based on the address mapping of the RAM coordinates according to the position of the light strip.
A video data processing system based on FPGA according to claim 1, characterized in that the output end of the LED driving module is also signal-connected to the input end of the LED driving quantity detection module, and the LED driving quantity detection module uses When the system is turned on, the number of chips on the external LED strip is detected through the current increment method.
A video data processing system based on FPGA according to claim 1, characterized in that the color space conversion module is used to convert YCbCr to RGB color space, and different standards can be calculated through a series of matrix operations. format RGB grayscale value; when the input HDMI signal source is in YCbCr format, the color space conversion module will automatically perform color space conversion and convert the YCbCr color space to RGB color space.
An FPGA-based video data processing system according to claim 1, characterized in that the resolution scaling module is used to convert 4K signals to a resolution of 1080P format, under the bandwidth condition of HDMI2.018G, By reducing the resolution of the input signal, the bandwidth of the pipeline is reduced. When the system bandwidth exceeds 165M*24bit, the system enables the resolution reduction module. At this time, when the input signal is 3840*2160, the actual output resolution is 1920 *1080, the system bandwidth of FPGA is reduced to 1/4 of the original.
A video data processing system based on FPGA according to claim 1, characterized in that the frame extraction module is used to convert signals higher than 60 Hz into signals not higher than 60 Hz, and the frame extraction module real-time Detect the frame rate of the input signal. When it exceeds 60Hz, divide the frame rate by 2. If it still exceeds 60Hz, continue dividing by 2 until it is less than 60Hz, ensuring that it does not affect the load points of the back-end ws2812 chip.
An FPGA-based video data processing system according to claim 1, characterized in that the HDR decoding module is used to determine whether the input signal is in HDR format by detecting the header of the HDR data packet, and to identify the HDR data, and perform a remapping calculation on the gray scale of the data to avoid the problem of RGB data desaturation.
A video data processing system based on FPGA according to claim 1, characterized in that the data buffer module is used to buffer data and isolate the internal and external clock domains of the FPGA; the signal from HDMI is sent line by line, Write to the buffer according to the control of the DE signal, and use FIFO to match the speed difference, which is used to match the internal speed of the FPGA.
A video data processing system based on FPGA according to claim 2, characterized in that the parameter module is used to calculate each LED by detecting the resolution of the HDMI signal and the number of lamp beads of the LED lamp strip. The rectangular pixel area of the TV corresponding to the lamp bead is used to obtain the average value of the RGB in the rectangular area, and the average value is used as the RGB gray value; at the same time, the parameter module can configure various parameters of the FPGA, and the parameters of the FPGA Defined as a table to achieve.