WO2023169060A1 - 图像数据处理方法及装置、存储介质 - Google Patents

图像数据处理方法及装置、存储介质 Download PDF

Info

Publication number
WO2023169060A1
WO2023169060A1 PCT/CN2022/142611 CN2022142611W WO2023169060A1 WO 2023169060 A1 WO2023169060 A1 WO 2023169060A1 CN 2022142611 W CN2022142611 W CN 2022142611W WO 2023169060 A1 WO2023169060 A1 WO 2023169060A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
image
frame
row
line
Prior art date
Application number
PCT/CN2022/142611
Other languages
English (en)
French (fr)
Inventor
马龙飞
张鹏国
原育光
Original Assignee
浙江宇视科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 浙江宇视科技有限公司 filed Critical 浙江宇视科技有限公司
Publication of WO2023169060A1 publication Critical patent/WO2023169060A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
    • G06F13/14—Handling requests for interconnection or transfer
    • G06F13/16—Handling requests for interconnection or transfer for access to memory bus
    • G06F13/1668—Details of memory controller
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02—Addressing or allocation; Relocation
    • G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
    • G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
    • G06F12/0806—Multiuser, multiprocessor or multiprocessing cache systems
    • G06F12/0815—Cache consistency protocols
    • G06F12/0831—Cache consistency protocols using a bus scheme, e.g. with bus monitoring or watching means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02—Addressing or allocation; Relocation
    • G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
    • G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
    • G06F12/0893—Caches characterised by their organisation or structure
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
    • G06F13/14—Handling requests for interconnection or transfer
    • G06F13/16—Handling requests for interconnection or transfer for access to memory bus
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/32—Circuits or arrangements for control or supervision between transmitter and receiver or between image input and image output device, e.g. between a still-image camera and its memory or between a still-image camera and a printer device
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/32—Circuits or arrangements for control or supervision between transmitter and receiver or between image input and image output device, e.g. between a still-image camera and its memory or between a still-image camera and a printer device
    • H04N1/32358—Circuits or arrangements for control or supervision between transmitter and receiver or between image input and image output device, e.g. between a still-image camera and its memory or between a still-image camera and a printer device using picture signal storage, e.g. at transmitter
    • H04N1/32363—Circuits or arrangements for control or supervision between transmitter and receiver or between image input and image output device, e.g. between a still-image camera and its memory or between a still-image camera and a printer device using picture signal storage, e.g. at transmitter at the transmitter or at the receiver
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2213/00—Indexing scheme relating to interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
    • G06F2213/0026—PCI express

Definitions

  • the present application relates to image processing technology, for example, to an image data processing method and device, and a storage medium.
  • the PCIe device based on the Peripheral Component Interconnect express (PCIe) architecture is shown in Figure 1 and can include a central processing unit (CPU), a switch (Switch), an input board and an output board, etc.
  • the CPU is connected to the Root Complex (RC)
  • the RC is connected to the switch (Switch) and memory (memory)
  • the switch (Switch) is connected to the input board and output board.
  • the input board and output board can be implemented using Field Programmable Gate Array (FPGA).
  • FPGA Field Programmable Gate Array
  • the CPU implements the initialization and configuration of PCIe devices (switches, input boards and output boards, etc.) through RC.
  • the CPU and input board, output board, and input board and output board can be connected through the transaction layer package (Transaction Layer Packet , TLP) to communicate.
  • TLP Transaction Layer Packet
  • the input board realizes the functions of image collection, caching, image reduction and image cutting, and sends the collected image data to the output board through PCIe.
  • the output board realizes the functions of image caching, amplification, stacking and display, and the CPU completes the processing of the input board. and output board business management and scheduling.
  • the input board collection, caching, scaling and image sending are all processed on a line basis, and the output board image reception, caching, scaling and display are also processed on a line basis.
  • the advantage is that it is easy to cut, cache and zoom images. and image splicing processing, but the shortcomings are equally obvious. Since multiple modules are processed in rows, the bandwidth utilization of modules such as caching, image sending, and image receiving will inevitably be reduced. For example, the more fragmented the image is, the lower the bandwidth utilization will be. The lower the rate, which will affect the performance of the product.
  • This application provides an image data processing method, device, and storage medium, which can improve bandwidth utilization.
  • This application provides an image data processing method, including:
  • the data of the image cached in frames is composed into data packets in frames and then sent.
  • This application provides an image data processing method, including:
  • the images processed by frames are split by rows to generate images processed by rows; and the images processed by rows generated by splitting are processed into images. Splicing.
  • the present application provides an image data processing device, which includes a memory and a processor.
  • the memory stores a program.
  • the program is read and executed by the processor, the above image data processing method is implemented.
  • the present application provides a computer-readable storage medium that stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above image data processing. method.
  • Figure 1 is a schematic diagram of a PCIe architecture spelling control system provided by an embodiment of the present application
  • Figure 2 is a schematic diagram of row-by-row processing provided by an embodiment of the present application.
  • Figure 3 is a schematic diagram of frame-by-frame processing provided by an embodiment of the present application.
  • Figure 4 is a flow chart of an image data processing method provided by an embodiment of the present application.
  • Figure 5 is a flow chart of another image data processing method provided by an embodiment of the present application.
  • Figure 6 is a flow chart of another image data processing method provided by an embodiment of the present application.
  • Figure 7 is a schematic diagram of a line-based cache provided by an embodiment of the present application.
  • Figure 8 is a schematic diagram of converting an image processed by lines into an image processed by frames according to an embodiment of the present application
  • Figure 9 is a schematic diagram of frame-by-frame caching provided by an embodiment of the present application.
  • Figure 10 is a schematic diagram of a row-based TLP provided by an embodiment of the present application.
  • FIG 11 is a schematic diagram of a frame-based TLP provided by an embodiment of the present application.
  • Figure 12 is a flow chart of another image data processing method provided by an embodiment of the present application.
  • Figure 13 is a schematic diagram of converting an image processed by frames into an image processed by lines according to an embodiment of the present application
  • Figure 14 is a schematic diagram of an image data processing device provided by an embodiment of the present application.
  • DDR SDRAM Double Data Rate Synchronous Dynamic Random Access Memory
  • PCIe PCIe
  • DDR SDRAM Double Data Rate Synchronous Dynamic Random Access Memory
  • PCIe PCIe
  • An image data processing method which processes images on a frame-by-frame basis without cutting or splicing, thereby improving bandwidth utilization.
  • FIG. 2 is a schematic diagram of row-by-row processing provided by an embodiment of the present application.
  • one frame of the image includes N lines: Line 0 to LineN-1.
  • Each line includes the line head (sol), line end (eol) and the data of the line.
  • the line head of the first line (Line 0) It is also the frame header (sof), and the end of the last line (Line N-1) is also the frame end (eof).
  • FIG. 3 is a schematic diagram of frame-by-frame processing provided by an embodiment of the present application.
  • the data of the next row is spliced to the end of the data of the current row, taking one frame as a unit.
  • the frame includes the content of N rows, only the frame header (sof) and the frame end (eof), not including Head of line (sol) and end of line (eol).
  • the data to be spliced in each line does not include the above-mentioned line header (sol) and line end (eol).
  • FIG 4 is a flow chart of an image data processing method provided by an embodiment of the present application. As shown in Figure 4, the image data processing method provided by this embodiment includes:
  • Step 401 obtain the image.
  • Step 402 When the image does not need to be cut, cache the image by frame.
  • Step 403 The data of the image cached by frames is composed into data packets by frames and then sent.
  • the solution provided by this embodiment processes image data on a frame-by-frame basis. Compared with processing data on a per-row basis, when processing on a per-frame basis, the data of one frame can be processed centrally, thereby improving bandwidth utilization.
  • caching by frame includes: sequentially writing the data of each frame into the cache with the maximum burst length, and when the remaining data length is less than the maximum burst length, writing the data of each frame with the actual length of the remaining data.
  • the length is written to the cache. That is, for a frame of data, the data with the maximum burst length is written into the cache each time until all the data of the frame is written into the cache. The last data written may be less than the maximum burst length, so the last remaining data is used.
  • the actual length of the data is written to the cache.
  • the solution provided by this embodiment is compared with caching by row.
  • each row may have a burst with a length smaller than the maximum burst length, so the bandwidth utilization is higher when processing by frames.
  • one frame of the acquired image includes multiple lines of data.
  • the image can be processed first to splice the multiple lines of data into one frame of data, that is: for each frame of data, from From the second row to the last row, the data of each row is spliced to the end of the data of the previous row, and the invalid data has been removed from the data (including two situations, when there is invalid data in the data, the invalid data is removed, if the data If there is no invalid data in it, there is no need to remove invalid data).
  • the actual number of pixels in each row can be determined based on preconfigured system parameters. When the number of pixels in each row of data exceeds the above-mentioned actual number of pixels, the excess pixels are invalid data.
  • the data of each line does not include the beginning and end of the line.
  • the invalid data in each row may not be removed, and the data in each row may be directly spliced to the end of the data in the previous row. Removing invalid data can save bandwidth resources and improve bandwidth utilization.
  • composing data packets by frames includes: sequentially composing data packets with the maximum packet length supported by the transmission interface for each frame of data. When the length of the remaining data is less than the maximum packet length, configuring the data packets with the maximum packet length supported by the transmission interface. The actual length of the remaining data makes up the packet. Compared with the solution of processing by rows, the solution provided by this embodiment has more data packets with the maximum packet length when processing one frame of data (only the packet length of the last data packet may be smaller than the maximum packet length). During processing, there may be data packets with a packet length smaller than the maximum packet length in each row. Therefore, bandwidth utilization is higher when processing by frames.
  • the transmission interface is, for example, PCIe, and the data packet may be TLP, but the embodiment of the present application is not limited thereto, and may be other transmission interfaces and other types of data packets.
  • the method further includes: when the image needs to be cut, caching the image by rows; and composing the data of the image cached by rows into data packets by rows. send.
  • the solution provided by this embodiment can cache images that need to be cut by rows to facilitate cutting.
  • the method further includes, after caching the image, carrying instruction information in the data of the image, the instruction information indicating a processing method of the image, and the processing method includes : Process by frame or by line.
  • the instruction information may be directly carried instruction information (for example, carrying different flags to indicate different processing methods, the flags may be preset flag bits, and different values of the preset flag bits may indicate different processing methods), or may be implicit Instruction information (for example, carrying the preset field indicates one processing method, not carrying the preset field indicates another processing method, etc.).
  • the above image data processing method can be applied to the input board of the spelling control system, but is not limited thereto.
  • FIG. 5 is a flow chart of another image data processing method provided by an embodiment of the present application. As shown in Figure 5, the image data processing method provided by the embodiment of the present application includes:
  • Step 501 Receive images.
  • Step 502 When the images need to be spliced and the images are processed by frames, the images processed by frames are split by rows to generate images processed by rows.
  • Step 503 perform image splicing on the line-processed images generated by splitting.
  • the solution provided by this embodiment can improve bandwidth utilization by receiving images processed on a frame-by-frame basis.
  • splitting the frame-processed image by rows includes:
  • the pixels corresponding to the last clock of the current row include pixels of the next row, and the pixels of the last clock belonging to the next row are split to the front end of the next row.
  • the actual number of pixels per row can be determined based on preconfigured system parameters.
  • the method further includes: determining to process the image by frames or by rows according to the instruction information carried in the image.
  • the indication information can be obtained from a frame header packet.
  • the embodiments of the present application are not limited to this. You can determine whether to process by frame or by line based on the received data packet.
  • the method further includes, when the images need to be spliced and the images are processed in rows, performing image splicing on the images. That is, the images processed by rows can be directly spliced.
  • the solution provided by this embodiment implements line-by-line and frame-by-frame hybrid processing, facilitates image splicing, and can improve bandwidth utilization.
  • the images can be sent to the display module for display after being spliced.
  • the above image data processing method can be applied to the output board of the spelling control system, but is not limited to this.
  • FIG. 6 is a flow chart of another image data processing method provided by an embodiment of the present application.
  • external interfaces such as DDR SDRAM, PCIe, etc.
  • the image data processing Methods include:
  • Step 601 obtain the image.
  • Step 602 Determine whether the image needs to be cut. When cutting is required, step 603 is executed. When cutting is not required, step 604 is executed.
  • Step 603 Cache the image by row and execute step 605.
  • DDR SDRAM can be used for caching.
  • the caching by row includes: in row units, the cached data is written sequentially with the maximum burst length (Max Burst Length) in a row. When the remaining data is less than the maximum burst length, the cached data is written with the actual length, that is, the last data in a row is written. The length of a burst is the length of the remaining data, and the length of the burst before the last burst is the maximum burst length.
  • the data of Line 0 may include two bursts 0-0 and 0-1, where the data length of burst 0-0 is the maximum burst length, and the burst length is 0-1 is the last burst, which is the actual length of the remaining data after Line 0 data minus burst 0-0 data.
  • the subsequent Line 1 to Line N-1 are similar and will not be described again. What is shown in Figure 7 is only an example, and the embodiment of the present application is not limited thereto.
  • One row may include multiple bursts whose length is the maximum burst length.
  • Step 604 The DDR cache caches the image by frame, and step 605 is executed.
  • the DDR cache caching the image frame by frame includes:
  • each clock For a video with an image resolution, each clock must process 4 pixels (this is only an example, the solution provided by the embodiment of the present application can be applied to a solution in which each clock processes other numbers of pixels), the image resolution It is not necessarily a multiple of 4, so there may be invalid pixels in the 4 pixels at the end of each line of the image. In order to be compatible with the image resolution, one clock is used to process 4 pixels.
  • the processing flow is as follows:
  • the spliced data into the cache including: taking the frame as a unit, writing the data in one frame to the cache sequentially with the maximum burst length (Max Burst Length).
  • Maximum Burst Length When the remaining data in one frame is less than the maximum burst length, The actual length is written to the cache. That is, the length of the last burst in a frame is the length of the remaining data, and the length of the burst before the last burst is the maximum burst length.
  • the length of a frame of data is an integer multiple of the maximum burst length, the length of the last burst is the maximum burst length.
  • the data of one frame includes bursts 0 to N-2 whose length is the maximum burst length, and burst N-1 whose length is the remaining data length in the frame.
  • the number of bursts here is N may be the same as the number of rows N contained in the aforementioned image, but the embodiment of the present application is not limited thereto, and the number of bursts may be other values.
  • Step 605 Perform marking processing according to the caching mode, where the marking processing indicates the caching mode of the image; the caching mode includes: line-by-line processing or frame-by-frame processing. For example, when the image is cached by rows, a first tag indicating that the image is cached by rows is added, and when the image is cached by frames, a second tag indicating that the image is cached by frames is added; In an exemplary embodiment, a tag may be added when caching on a per-line basis, and no tag will be carried when caching on a frame-by-frame basis (that is, no tag means caching on a per-frame basis), or a tag will be carried when caching on a per-frame basis, and a tag will be carried when caching on a per-line basis.
  • the mark can be carried in a frame header packet (not the aforementioned frame header), and each frame of data corresponds to a frame header packet.
  • the data of each frame can be one frame of data processed by frame, or one frame of data processed by row. Multiple rows of data.
  • Step 606 The image data is composed into a TLP, and the TLP is sent to the output board.
  • the TLP may be sent to the output board via PCIe.
  • composing the image data into a TLP may include:
  • TLP When the image data is processed by rows, TLP is generated in row units, and each row is packed sequentially with the maximum TLP packet length. When the remaining data in a row is less than the maximum TLP packet length, the TLP is packetized with the actual data length. Therefore, each row The length of the last burst may not be the maximum TLP packet length. In an exemplary embodiment, when the image is small and the data length of each line is small, there may be only one TLP per line, and the packet length of the TLP may be much smaller than the maximum TLP packet length.
  • FIG 10 is a schematic diagram of a row-based TLP provided by an embodiment of the present application.
  • each row includes two TLPs, one packet length is the maximum TLP packet length, and the other packet length is the row data length minus the maximum TLP packet length.
  • the embodiments of the present application are not limited to this.
  • each row may include more TLPs.
  • TLP When the image data is processed by frames, TLP is generated in units of frames.
  • the data of each frame is packaged sequentially with the maximum TLP packet length.
  • the actual data is The last TLP of the length group, and sequentially send TLP 0, TLP 1, TLP 2...TLP M-4, TLP M-3, TLP M-2, TLP M-1 to the output board, M represents the number of TLPs in a frame Number, this way of grouping TLP by frame only the length of the last TLP in a frame may not be the maximum TLP packet length, other TLPs are grouped according to the maximum TLP packet length, as shown in Figure 11, TLP 0 to TLP M
  • the length of -2 is the maximum TLP packet length
  • the length of TLP M-1 is the actual length of the remaining data in a frame.
  • the embodiments of the present application are not limited to this.
  • the solution provided by this embodiment can process the image on a frame-by-frame basis when the image does not need to be cut, thereby making full use of bandwidth, improving bandwidth utilization, and improving product performance.
  • the image needs to be cut it can be processed on a line-by-line basis, making it easy to cut.
  • the solution provided in this embodiment is applied to the splicing control system, the mixed processing of rows and frames not only has the advantage of easy image cutting and image splicing, but also improves the bandwidth utilization of the external interface.
  • FIG 12 is a flow chart of another data processing method provided by an embodiment of the present application. As shown in Figure 12, the data processing method provided by this embodiment can be applied to the output board, including:
  • Step 1201 Receive image data sent by the input pad.
  • Step 1202 Determine whether the image needs to be spliced. If the image needs to be spliced, perform step 1203. If the image does not need to be spliced, perform step 1206.
  • Step 1203 Determine whether to process by lines or by frames based on the marks in the image data. If the processing is by lines, execute step 1205; if the processing is by frames, execute step 1204.
  • Step 1204 convert the data processed by frames into processed by rows
  • converting data processed by frames into processing by rows includes:
  • Step 1205 perform image stitching.
  • the received row-processed images may be image spliced, or the row-processed images converted in step 1204 may be spliced.
  • the splicing method is performed according to the pre-configured parameters of the system.
  • Step 1206 Send the images that do not need to be spliced or the spliced images to the display module for display.
  • the solution provided by this embodiment can improve the bandwidth utilization efficiency of external interfaces (such as DDR SDRAM, PCIe, etc.), improve image fluency, and enhance product competitiveness.
  • external interfaces such as DDR SDRAM, PCIe, etc.
  • this embodiment of the present application provides an image data processing device 140, which includes a memory 1410 and a processor 1420.
  • the memory 1410 stores a program.
  • the program is read and executed by the processor 1420, Implement the image data processing method described in any of the above embodiments.
  • Embodiments of the present application provide a computer-readable storage medium that stores one or more programs, and the one or more programs can be executed by one or more processors to implement any of the above.
  • Computer storage media includes volatile and nonvolatile media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. removable, removable and non-removable media.
  • Computer storage media includes, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Flash Memory or other memory technologies, Compact Disk-Read Only Memory (CD-ROM), Digital Video Disk (DVD) or other optical disk storage, magnetic cassettes, tapes, disk storage or other magnetic storage devices , or any other medium that can be used to store the desired information and can be accessed by a computer.
  • communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media .

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Memory System Of A Hierarchy Structure (AREA)

Abstract

一种图像数据处理方法及装置、存储介质。所述图像数据处理方法包括：获取图像，当所述图像不需要切割时，将所述图像按帧进行缓存；将所述按帧进行缓存的图像的数据按帧组成数据包后发送。

Description

图像数据处理方法及装置、存储介质
本申请要求在2022年03月11日提交中国专利局、申请号为202210239330.0的中国专利申请的优先权，该申请的全部内容通过引用结合在本申请中。
技术领域
本申请涉及图像处理技术，例如涉及一种图像数据处理方法及装置、存储介质。
背景技术
基于快捷外围部件互联(Peripheral Component Interconnect express，PCIe)架构的拼控设备如图1所示，可以包括中央处理器(Central Processing Unit，CPU)、交换器(Switch)、输入板和输出板等，CPU连接到根联合体(Root Complex，RC)，RC连接到交换器(Switch)和存储器(memory)，交换器(Switch)连接到输入板和输出板。输入板和输出板可以使用现场可编程逻辑门阵列(Field Programmable Gate Array，FPGA)实现。其中，CPU通过RC实现对PCIe设备(交换器、输入板和输出板等)的初始化及配置，CPU和输入板、输出板，以及输入板和输出板之间可以通过事务层包(Transaction Layer Packet，TLP)进行通信。输入板实现图像的采集、缓存、图像缩小及图像切割功能，并通过PCIe将采集的图像数据发送给输出板，输出板实现图像的缓存、放大、叠层及送显功能，CPU完成对输入板和输出板的业务管理及调度。
传统拼控系统中，输入板采集、缓存、缩放及图像发送都是按行处理的，输出板图像接收、缓存、缩放及送显也是按行处理的，优点是易于图像的切割、缓存、缩放及图像的拼接处理，但缺点也同样明显，由于多个模块都是按行处理，势必会造成缓存、图像发送、图像接收等模块的带宽利用率降低，例如，图像切割的越碎，带宽利用率越低，进而影响到产品的性能。
发明内容
本申请提供了一种图像数据处理方法及装置、存储介质，可以提高带宽利用率。
本申请提供了一种图像数据处理方法，包括：
获取图像；
当所述图像不需要切割时，将所述图像按帧进行缓存；
将所述按帧进行缓存的图像的数据按帧组成数据包后发送。
本申请提供一种图像数据处理方法，包括：
接收图像；
当所述图像需要拼接且所述图像是按帧处理时，将所述按帧处理的图像按行进行拆分生成按行处理的图像；将拆分生成的所述按行处理的图像进行图像拼接。
本申请提供一种图像数据处理装置，包括存储器和处理器，所述存储器存储有程序，所述程序在被所述处理器读取执行时，实现上述的图像数据处理方法。
本申请提供一种计算机可读存储介质，所述计算机可读存储介质存储有一个或者多个程序，所述一个或者多个程序可被一个或者多个处理器执行，以实现上述的图像数据处理方法。
附图说明
图1为本申请实施例提供的一种PCIe架构的拼控系统的示意图；
图2为本申请实施例提供的一种按行处理的示意图；
图3为本申请实施例提供的一种按帧处理的示意图；
图4为本申请实施例提供的一种图像数据处理方法的流程图；
图5为本申请实施例提供的另一种图像数据处理方法的流程图；
图6为本申请实施例提供的另一种图像数据处理方法的流程图；
图7为本申请实施例提供的一种按行缓存的示意图；
图8为本申请实施例提供的一种将按行处理的图像转换为按帧处理的图像的示意图；
图9为本申请实施例提供的一种按帧进行缓存的示意图；
图10为本申请实施例提供的一种按行组TLP的示意图；
图11为本申请实施例提供的一种按帧组TLP的示意图；
图12为本申请实施例提供的另一种图像数据处理方法的流程图；
图13为本申请实施例提供的一种将按帧处理的图像转换为按行处理的图像的示意图；
图14为本申请实施例提供的一种图像数据处理装置示意图。
具体实施方式
本申请描述了多个实施例，但是该描述是示例性的。
此外，在描述具有代表性的实施例时，说明书可能已经将方法和/或过程呈现为特定的步骤序列。然而，在该方法或过程不依赖于本文所述步骤的特定顺序的程度上，该方法或过程不应限于所述的特定顺序的步骤。如本领域普通技术人员将理解的，其它的步骤顺序也是可能的。
对于外部接口，如双倍速率同步动态随机存储器(Double Data Rate Synchronous Dynamic Random Access Memory，DDR SDRAM)和PCIe等，发送接收数据的突发长度越大，带宽利用率越高，本申请实施例中提出了一种图像数据处理方法，对于无需切割、拼接的图像按帧进行处理，从而提高带宽利用率。
图2为本申请实施例提供的一种按行处理的示意图。如图2所示，图像的一帧包括N行：Line 0至LineN-1，每一行都包括行头(sol)、行尾(eol)和该行的数据，第一行(Line 0)的行头也是帧头(sof)，最后一行(Line N-1)的行尾也是帧尾(eof)。
图3为本申请实施例提供的一种按帧处理的示意图。如图3所示，将下一行的数据拼接到当前行的数据的结尾处，以一帧为单位，该帧包括N行的内容，只有帧头(sof)和帧尾(eof)，不包括行头(sol)和行尾(eol)。每行进行拼接的数据不包括上述行头(sol)和行尾(eol)。
图4为本申请实施例提供的一种图像数据处理方法的流程图。如图4所示，本实施例提供的图像数据处理方法包括：
步骤401，获取图像。
步骤402，当所述图像不需要切割时，将所述图像按帧进行缓存。
步骤403，将所述按帧进行缓存的图像的数据按帧组成数据包后发送。
本实施例提供的方案，按帧对图像数据进行处理，相比按行对数据进行处理，按帧处理时，可以集中对一帧的数据进行处理，提高带宽利用率。
在一示例性实施例中，所述按帧进行缓存包括：对每帧的数据，以最大突发长度依次写入缓存，当剩余数据长度小于所述最大突发长度时，以剩余数据的实际长度写入缓存。即对一帧的数据，每次取最大突发长度的数据写入缓存，直到将该帧的数据全部写入缓存，最后一次写入的数据可能长度不足最大突发长度，则以最后剩余的数据的实际长度写入缓存。本实施例提供的方案，相比按行进行缓存，对一帧数据，按帧处理时，长度为最大突发长度的突发更多(仅 最后一个突发的长度可能小于最大突发长度)，按行处理时，每行都可能存在长度小于最大突发长度的突发，因此，按帧处理时带宽利用率更高。
在一示例性实施例中，获取的图像的一帧包括多行数据，在按帧进行缓存前，可以对图像先进行处理，将多行数据拼接成一帧数据，即：对每帧数据，从第二行至最后一行，依次将每行的数据拼接至前一行的数据结尾处，且所述数据中已去除无效数据(包括两种情况，数据中存在无效数据时，去除无效数据，如果数据中不存在无效数据，则无需进行去除无效数据的操作)。其中，可以根据预先配置的系统参数确定每行实际的像素数量，每行数据中的像素数量超出上述实际的像素数量时，超出的像素即为无效数据。所述每行的数据不包括该行的行头和行尾。
在另一示例性实施例中，多行数据拼接成一帧数据时，可以不去除每行的无效数据，直接将每行的数据拼接至前一行的数据的结尾处。去除无效数据可以节省带宽资源，提高带宽利用率。
在一示例性实施例中，所述按帧组成数据包包括：对每帧的数据，以传输接口支持的最大包长依次组成数据包，当剩余数据的长度小于所述最大包长时，以剩余数据的实际长度组成数据包。本实施例提供的方案，相比按行处理的方案，处理一帧数据时，包长为最大包长的数据包更多(仅最后一个数据包的包长可能小于最大包长)，按行处理时，可能每行均存在包长小于最大包长的数据包，因此，按帧处理时带宽利用率更高。传输接口比如为PCIe，所述数据包可以是TLP，但本申请实施例不限于此，可以是其他传输接口和其他类型的数据包。
在一示例性实施例中，所述方法还包括：当所述图像需要切割时，将所述图像按行进行缓存；以及，将所述按行进行缓存的图像的数据按行组成数据包后发送。本实施例提供的方案，可以对需要切割的图像按行进行缓存，便于实现切割。
在一示例性实施例中，所述方法还包括，将所述图像进行缓存后，在所述图像的数据中携带指示信息，所述指示信息指示所述图像的处理方式，所述处理方式包括：按帧处理、或按行处理。所述指示信息可以是直接携带的指示信息(比如，携带不同标记指示不同的处理方式，标记比如为预设标志位，预设标志位不同的值表示不同的处理方式)，或者为隐含的指示信息(比如，携带预设字段表示一种处理方式，不携带该预设字段表示另一种处理方式，等等)。
上述图像数据处理方法可以应用于拼控系统的输入板，但不限于此。
图5为本申请实施例提供的另一种图像数据处理方法的流程图。如图5所 示，本申请实施例提供的图像数据处理方法包括：
步骤501，接收图像。
步骤502，当所述图像需要拼接且所述图像是按帧处理时，将所述按帧处理的图像按行进行拆分生成按行处理的图像。
步骤503，将拆分生成的所述按行处理的图像进行图像拼接。
本实施例提供的方案，接收按帧处理的图像，可以提高带宽利用率。
在一示例性实施例中，所述将所述按帧处理的图像按行进行拆分包括：
将所述图像的一帧的数据按行拆分，且对任一行数据，当当前行包括下一行的数据时，将当前行中属于下一行的数据拆分到下一行的最前端；为每行数据添加行头和行尾。
比如，当前行的最后一个时钟对应的像素中包括下一行的像素，将最后一个时钟中属于下一行的像素拆分到下一行的最前端。
可以根据预先配置的系统参数确定每行实际的像素数量。
在一示例性实施例中，所述方法还包括：根据所述图像中携带的指示信息确定图像按帧处理或按行处理。比如，可以从帧头包中获取所述指示信息。但本申请实施例不限于此。可以根据接收到的数据包判断是按帧处理还是按行处理。
在一示例性实施例中，所述方法还包括，当所述图像需要拼接且所述图像是按行处理时，将所述图像进行图像拼接。即按行处理的图像直接进行拼接即可。本实施例提供的方案，实现按行，按帧的混合处理，易于图像拼接，且可以提高带宽利用率。
在一示例性实施例中，可以在进行图像拼接后发送至显示模块进行显示。
上述图像数据处理方法可以应用于拼控系统的输出板，但不限于此。
图6为本申请实施例提供的另一种图像数据处理方法流程图。如图6所示，本实施例中，使用外部接口，如DDR SDRAM、PCIe等实现对图像数据的缓存、发送，但本申请实施例不限于此，可以是其他外部接口，所述图像数据处理方法包括：
步骤601，获取图像。
步骤602，判断所述图像是否需要切割，当需要切割时，执行步骤603，当不需要切割时，执行步骤604。
步骤603，将所述图像按行进行缓存，执行步骤605。
比如，可以使用DDR SDRAM进行缓存。
所述按行进行缓存包括：以行为单位，一行中以最大突发长度(Max Burst Length)依次写缓存数据，剩余数据不足最大突发长度时，以实际长度写入缓存数据，即一行中最后一个突发的长度为剩余数据的长度，最后一个突发之前的突发的长度为最大突发长度。
在一示例性实施例中，如图7所示，Line 0的数据可以包括两个突发0-0和0-1，其中，突发0-0的数据长度为最大突发长度，突发0-1为最后一个突发，为Line 0数据减去突发0-0的数据后剩余数据的实际长度。后续Line 1至Line N-1类似，不再赘述。图7所示仅为示例，本申请实施例不限于此，一行可能包括多个长度为最大突发长度的突发。
步骤604，DDR缓存将所述图像按帧进行缓存，执行步骤605。
所述DDR缓存将所述图像按帧进行缓存包括：
将所述图像的数据进行拼接处理，将拼接好的处理写入缓存。
针对一图像分辨率的视频，每个时钟要处理4个像素(此处仅为示例，本申请实施例提供的方案可以应用到每个时钟处理其他数量的像素的方案中)，该图像分辨率不一定是4的倍数，所以在每行图像的行尾4像素中可能存在无效像素，为了兼容该图像分辨率，统一采用1个时钟处理4个像素，处理流程如下：
判断是否需要剔除行尾的无效像素，若需要剔除行尾的无效像素，则根据需要剔除的行尾的无效像素个数进行剔除，若不需要剔除行尾的无效像素，不进行剔除；从第二行开始，将每行的数据拼接到前一行的数据的结尾处，如图8所示，直至最后一行，生成拼接好的数据。
将拼接好的数据写入缓存中，包括：以帧为单位，将一帧中的数据以最大突发长度(Max Burst Length)依次写缓存，一帧中剩余数据不足最大突发长度时，以实际长度写入缓存。即一帧中最后一个突发的长度为剩余数据的长度，最后一个突发之前的突发的长度为最大突发长度。当一帧的数据的长度为最突发长度的整数倍时，最后一个突发的长度为最大突发长度。如图9所示，一帧的数据包括长度为最大突发长度的突发0至突发N-2，以及长度为一帧中剩余数据长度的突发N-1，此处的突发数N可能与前述图像包含的行数N相同，但本申请实施例不限于此，突发数可以是其他值。
步骤605，根据缓存方式进行标记处理，所述标记处理指示对所述图像的缓存方式；所述缓存方式包括：按行处理、或按帧处理。比如，当对图像按行进行缓存时，添加指示对所述图像按行进行缓存的第一标记，当对图像按帧进行 缓存时，添加指示对所述图像按帧进行缓存的第二标记；在一示例性实施例中，可以是按行进行缓存时添加标记，按帧缓存时不携带标记(即无标记时表示是按帧进行缓存)，或者，按帧进行缓存时携带标记，按行进行缓存时不携带标记(即无标记时表示是按行进行缓存)等等。所述标记可以携带在帧头包中(非前述帧头)，每帧数据对应一个帧头包，所述每帧数据可以是按帧处理的一帧数据，或者是按行处理的一帧的多行数据。
步骤606，将所述图像的数据组成TLP，将所述TLP发送给输出板。
在一示例性实施例中，可以通过PCIe将所述TLP发送给输出板。
在一示例性实施例中，将所述图像的数据组成TLP可以包括：
当所述图像的数据是按行处理时，以行为单位生成TLP，每行以最大TLP包长依次组包，一行最后剩余数据不足最大TLP包长时按实际数据长度组包，因此，每行的最后一次突发的长度有可能不是最大TLP包长。在一示例性实施例中，当图像较小，每行数据长度较小时，可能每行只有一个TLP，且该TLP的包长可能远远小于最大TLP包长。
图10为本申请实施例提供的一种按行组TLP的示意图。如图10所示，本实施例中，每行包括两个TLP，一个包长为最大TLP包长，另一个包长为该行数据长度减去最大TLP包长。但本申请实施例不限于此，当每行数据较多时，每行可能包括更多TLP。
当所述图像的数据是按帧处理时，以帧为单位生成TLP，对每帧的数据，以最大TLP包长依次组包，一帧最后剩余的数据不足最大TLP包长时，以实际数据长度组最后一个TLP，并依次将TLP 0、TLP 1、TLP 2……TLP M-4、TLP M-3、TLP M-2、TLP M-1发送给输出板，M表示一帧中TLP的个数，这种按帧组TLP的方式只在一帧的最后一个TLP的长度可能不是最大TLP包长，其他TLP均按最大TLP包长组包，如图11所示，TLP 0至TLP M-2的长度均为最大TLP包长，TLP M-1的长度为一帧剩余数据的实际长度。但本申请实施例不限于此，当一帧的数据的长度为最大TLP包长的整数倍时，最后一个TLP的包长为最大TLP包长。
本实施例提供的方案，可以在图像无需切割时，按帧进行处理，从而可以充分利用带宽，提高带宽利用率，提高产品性能，且在图像需要切割时，按行处理，易于切割。将本实施例提供的方案应用于拼控系统时，按行、按帧混合处理，既有易于图像切割、图像拼接的优势又能提高外部接口带宽利用率。
图12为本申请实施例提供的另一种数据处理方法的流程图。如图12所示，本实施例提供的数据处理方法，可以应用于输出板，包括：
步骤1201，接收输入板发送的图像数据。
比如，通过PCIe接收输入板发送的数据。
步骤1202，判断所述图像是否需要拼接，如果图像需要拼接，执行步骤1203，如果图像不需要拼接，执行步骤1206。
比如，可以根据系统预先配置的参数确定图像是否需要拼接。
步骤1203，根据图像数据中的标记判断是按行处理还是按帧处理，如果是按行处理，则执行步骤1205；如果是按帧处理，执行步骤1204。
步骤1204，将按帧处理的数据转换成按行处理；
在一示例性实施例中，按帧处理的数据转换成按行处理包括：
将一帧的数据按行进行拆分，并为每行添加行头、行尾；拆分时判断是否因前一行的最后一个时钟的像素不足4像素而拼接了当前行的像素，若是因前一行的最后一个时钟的像素不足4像素而拼接了当前行的像素，则根据要拆分的像素个数进行像素拆分，若不是因前一行的最后一个时钟的像素不足4像素而拼接了当前行的像素，则不需要进行像素拆分。如图13所示，Line 0的最后3个像素属于Line 1，进行拆分时，将该3个像素拆分到Line 1的最前端。
步骤1205，进行图像拼接。
此处可能是将接收到的按行处理的图像进行图像拼接，或者，将步骤1204中转换得到的按行处理的图像进行图像拼接。
拼接方式按照系统预先配置的参数进行。
步骤1206，将无需拼接的图像或者拼接后的图像发送到显示模块进行显示。
本实施例提供的方案，可以提高外部接口(比如DDR SDRAM、PCIe等)的带宽利用效率，提高图像流畅度，提升产品竞争力。
本申请实施例提供的方案，可以应用到拼控系统中，但不限于此，可以是其他系统。
如图14所示，本申请实施例提供一种图像数据处理装置140，包括存储器1410和处理器1420，所述存储器1410存储有程序，所述程序在被所述处理器1420读取执行时，实现上述任一实施例所述的图像数据处理方法。
本申请实施例提供一种计算机可读存储介质，所述计算机可读存储介质存储有一个或者多个程序，所述一个或者多个程序可被一个或者多个处理器执行，以实现上述任一实施例所述的图像数据处理方法。
本领域普通技术人员可以理解，上文中所公开方法中的全部或一些步骤、 系统、装置中的功能模块/单元可以被实施为软件、固件、硬件及其适当的组合。在硬件实施方式中，在以上描述中提及的功能模块/单元之间的划分不一定对应于物理组件的划分；例如，一个物理组件可以具有多个功能，或者一个功能或步骤可以由多个物理组件合作执行。一些组件或所有组件可以被实施为由处理器，如数字信号处理器或微处理器执行的软件，或者被实施为硬件，或者被实施为集成电路，如专用集成电路。这样的软件可以分布在计算机可读介质上，计算机可读介质可以包括计算机存储介质(或非暂时性介质)和通信介质(或暂时性介质)。如本领域普通技术人员公知的，术语计算机存储介质包括在用于存储信息(诸如计算机可读指令、数据结构、程序模块或其他数据)的任何方法或技术中实施的易失性和非易失性、可移除和不可移除介质。计算机存储介质包括但不限于随机存取存储器(Random Access Memory，RAM)、只读存储器(Read Only Memory，ROM)、带电可擦可编程只读存储(Electrically Erasable Programmable Read Only Memory，EEPROM)、闪存或其他存储器技术、光盘只读存储器(Compact Disk-Read Only Memory，CD-ROM)、数字多功能盘(Digital Video Disk，DVD)或其他光盘存储、磁盒、磁带、磁盘存储或其他磁存储装置、或者可以用于存储期望的信息并且可以被计算机访问的任何其他的介质。此外，本领域普通技术人员公知的是，通信介质通常包含计算机可读指令、数据结构、程序模块或者诸如载波或其他传输机制之类的调制数据信号中的其他数据，并且可包括任何信息递送介质。

Claims (10)

  1. 一种图像数据处理方法，包括：
    获取图像；
    在所述图像不需要切割的情况下，将所述图像按帧进行缓存；
    将按帧进行缓存的图像的数据按帧组成数据包后发送。
  2. 根据权利要求1所述的图像数据处理方法，其中，所述按帧进行缓存，包括：
    对每帧的数据，以最大突发长度依次写入缓存，在剩余数据长度小于所述最大突发长度的情况下，以剩余数据的实际长度写入缓存。
  3. 根据权利要求2所述的图像数据处理方法，其中，所述图像的一帧包括多行数据，在所述按帧进行缓存前，还包括：
    对每帧的多行数据，从第二行至最后一行，将每行的数据拼接至前一行的数据结尾处，且所述数据中已去除无效数据，且所述每行的数据不包括所述每行的行头和行尾。
  4. 根据权利要求1所述的图像数据处理方法，其中，所述按帧组成数据包，包括：
    对每帧的数据，以传输接口支持的最大包长依次组成数据包，在剩余数据长度小于所述最大包长的情况下，以剩余数据的实际长度组成数据包。
  5. 根据权利要求1至4任一所述的图像数据处理方法，还包括：
    在所述图像需要切割的情况下，将所述图像按行进行缓存；
    将按行进行缓存的图像的数据按行组成数据包后发送；
    其中，在缓存的图像的数据中携带指示信息，所述指示信息指示所述图像的处理方式，所述处理方式包括：按帧处理、或按行处理。
  6. 一种图像数据处理方法，包括：
    接收图像；
    在所述图像需要拼接且所述图像是按帧处理的情况下，将按帧处理的图像按行进行拆分生成按行处理的图像；
    将拆分生成的所述按行处理的图像进行图像拼接。
  7. 根据权利要求6所述的图像数据处理方法，其中，所述将按帧处理的图像按行进行拆分，包括：
    将所述图像的一帧的数据按行拆分，且对每行数据，在当前行包括下一行的数据的情况下，将当前行中属于下一行的数据拆分到下一行的最前端；
    为每行数据添加行头和行尾。
  8. 根据权利要求6所述的图像数据处理方法，还包括：
    根据所述图像中携带的指示信息确定图像按帧处理或按行处理；
    在所述图像需要拼接且所述图像是按行处理的情况下，将所述图像进行图像拼接。
  9. 一种图像数据处理装置，包括存储器和处理器，所述存储器存储有程序，所述程序在被所述处理器读取执行时，实现如权利要求1至8任一所述的图像数据处理方法。
  10. 一种计算机可读存储介质，所述计算机可读存储介质存储有至少一个程序，所述至少一个程序可被至少一个处理器执行，以实现如权利要求1至8任一所述的图像数据处理方法。
PCT/CN2022/142611 2022-03-11 2022-12-28 图像数据处理方法及装置、存储介质 WO2023169060A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210239330.0A CN116775517A (zh) 2022-03-11 2022-03-11 一种图像数据处理方法及装置、存储介质
CN202210239330.0 2022-03-11

Publications (1)

Publication Number Publication Date
WO2023169060A1 true WO2023169060A1 (zh) 2023-09-14

Family

ID=87937110

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/142611 WO2023169060A1 (zh) 2022-03-11 2022-12-28 图像数据处理方法及装置、存储介质

Country Status (2)

Country Link
CN (1) CN116775517A (zh)
WO (1) WO2023169060A1 (zh)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106973188A (zh) * 2017-04-11 2017-07-21 北京图森未来科技有限公司 一种图像传输装置和方法
CN110073653A (zh) * 2018-09-07 2019-07-30 深圳鲲云信息科技有限公司 视频图像数据传输方法、系统、存储介质及程序产品
JP2020160828A (ja) * 2019-03-27 2020-10-01 池上通信機株式会社 ２次元画像をアフィン変換するための画像データ処理装置

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106973188A (zh) * 2017-04-11 2017-07-21 北京图森未来科技有限公司 一种图像传输装置和方法
CN110073653A (zh) * 2018-09-07 2019-07-30 深圳鲲云信息科技有限公司 视频图像数据传输方法、系统、存储介质及程序产品
JP2020160828A (ja) * 2019-03-27 2020-10-01 池上通信機株式会社 ２次元画像をアフィン変換するための画像データ処理装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
ANONYMOUS: "FPGA design experience: image processing", 22 April 2020 (2020-04-22), XP093090904, Retrieved from the Internet <URL:https://blog.csdn.net/qq_40310273/article/details/105678708> *

Also Published As

Publication number Publication date
CN116775517A (zh) 2023-09-19

Similar Documents

Publication Publication Date Title
CN107766270B (zh) 用于PCIe设备的数据读取管理方法及装置
CN111126589B (zh) 神经网络数据处理装置、方法和电子设备
CN112100119B (zh) 一种基于fpga的高速以太网帧重构系统
CN102566958B (zh) 一种基于sgdma的图像分割处理装置
EP3166269B1 (en) Queue management method and apparatus
WO2020047843A1 (zh) 视频图像数据传输方法、系统、存储介质及程序产品
US20030012223A1 (en) System and method for processing bandwidth allocation messages
CN105573922B (zh) 一种实现数据格式转换的方法和装置
CN112511839A (zh) 一种视频传输控制方法、装置、设备及可读存储介质
CN114286035B (zh) 图像采集卡、图像采集方法及图像采集系统
WO2023169060A1 (zh) 图像数据处理方法及装置、存储介质
CN109800195A (zh) 一种基于fpga的光纤通道适配器及数据传输方法
US9137167B2 (en) Host ethernet adapter frame forwarding
CN110012367B (zh) 用于gpon olt的omci组帧装置及组帧方法
US20160085683A1 (en) Data receiving device and data receiving method
US20160315883A1 (en) Packet control method, transmission apparatus, and storage medium
CN110489353A (zh) 一种提高固态硬盘带宽读取性能方法及装置
CN114302087B (zh) 一种mipi数据传输模式转换方法、装置及电子设备
CN112637602B (zh) 一种jpeg接口及数字图像处理系统
CN114584782A (zh) 一种图像发送方法、装置及电子设备
US20200193135A1 (en) Image processing method and device, and unmanned aerial vehicle
CN114302089A (zh) 一种基于fpga的多路视频信号缓存控制方法及系统
CN109213710B (zh) 高速串行接口装置与其数据传输方法
JP2010118058A (ja) トリプル・バッファリングを使用するデータを処理する方法
JP2005303787A (ja) パケット生成装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22930679

Country of ref document: EP

Kind code of ref document: A1