CN114449245A - Real-time two-way video processing system and method based on programmable chip - Google Patents

Real-time two-way video processing system and method based on programmable chip Download PDF

Info

Publication number
CN114449245A
CN114449245A CN202210108244.6A CN202210108244A CN114449245A CN 114449245 A CN114449245 A CN 114449245A CN 202210108244 A CN202210108244 A CN 202210108244A CN 114449245 A CN114449245 A CN 114449245A
Authority
CN
China
Prior art keywords
image
target image
pixel
module
calculation
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202210108244.6A
Other languages
Chinese (zh)
Other versions
CN114449245B (en
Inventor
于非
赵征
高岩
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Tong Guan Intelligent Technology Co ltd
Original Assignee
Shanghai Tong Guan Intelligent Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Tong Guan Intelligent Technology Co ltd filed Critical Shanghai Tong Guan Intelligent Technology Co ltd
Priority to CN202210108244.6A priority Critical patent/CN114449245B/en
Publication of CN114449245A publication Critical patent/CN114449245A/en
Application granted granted Critical
Publication of CN114449245B publication Critical patent/CN114449245B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F15/00—Digital computers in general; Data processing equipment in general
    • G06F15/76—Architectures of general purpose stored program computers
    • G06F15/78—Architectures of general purpose stored program computers comprising a single central processing unit
    • G06F15/7807—System on chip, i.e. computer system on a single chip; System in package, i.e. computer system on one or more chips in a single package
    • G06F15/7825—Globally asynchronous, locally synchronous, e.g. network on chip
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/139—Format conversion, e.g. of frame-rate or size
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

The invention provides a real-time two-channel video processing system and a method thereof based on a programmable chip, wherein the system comprises an off-chip memory DDR, an intermediate parameter calculation module and a remapping module, wherein the intermediate parameter calculation module is used for acquiring the calculation relationship of the absolute position and the relative position of a pixel point of a target image and the calculation relationship of the pixel value of the pixel point of the target image according to an image correction parameter; the remapping module is used for sequentially reading out the pixel values currently required by the pixels of the source image from the corresponding positions in the DDR of the off-chip memory, calculating the pixel values of the pixels of the target image according to the calculation relation of the pixel values of the pixels of the target image, correcting the error of the two paths of source images, eliminating the error existing between the two paths of original images, enabling the time delay to be millisecond-level, directly obtaining safe and high-quality images to be directly watched, omitting a preposed motor mechanical correction link or a subsequent error processing link, simplifying the system and reducing the cost.

Description

Real-time two-way video processing system and method based on programmable chip
Technical Field
The invention relates to the technical field of two-way image acquisition and processing, in particular to a real-time two-way image processing system and method based on a programmable chip.
Background
The double-path image acquisition technology is widely applied to the fields of three-dimensional image acquisition, VR image acquisition and the like. The images acquired during the two-way image acquisition need to be subjected to image processing so as to eliminate errors between the images during the image acquisition.
In the three-dimensional image acquisition process, two cameras are used for acquiring images through two-way image acquisition, and the two cameras are arranged vertically or parallelly for presenting different degrees of opening angles, but due to mechanical reasons such as position deviation between the two cameras and optical reasons, errors such as translation, rotation, scaling and the like of the images often exist between the two images. In order to eliminate the error between two images, a manual adjustment or an automatic adjustment mode is usually adopted at present, the manual adjustment can calibrate two cameras for three-dimensional image acquisition in advance, and the manual adjustment is carried out on site to eliminate the error between the images to the maximum extent. The whole process consumes more time, and has higher requirements on the level of field workers. Moreover, when the scene changes, the above manner needs to be repeated to readjust, the whole process is time-consuming and labor-consuming, and the requirements of the dynamic scene cannot be met. The automatic adjustment can be carried out by firstly calculating the error amount between two images through an algorithm and then carrying out automatic adjustment according to the error amount. Automatic adjustment currently generally includes two methods, electric adjustment and post-software adjustment. During electric adjustment, the motors are used for respectively controlling and adjusting the physical positions of the two cameras so as to correct errors between graphs. The advantage of the motorized adjustment is that the original image can be output, and the disadvantage is that additional mechanical structure is required, which increases the cost and volume of the device. And when the software is adjusted in the later stage, processing the video files acquired by the two cameras, and performing operations such as rotation, translation, cutting, zooming and the like on the images. The later software adjustment has the advantages that an additional physical adjustment part is not needed, and the defects that video file processing is needed after shooting is completed, the method is not suitable for real-time occasions such as live broadcast and the like, and certain influence on image quality and a visual range can be possibly generated.
In VR image acquisition field, the two fisheye schemes that use at present, its cost is lower and realize simply. However, due to mechanical installation, each camera of the front camera and the rear camera during two-way image acquisition may not reach a perfect 180 degrees, so that an error may exist, and a camera lens may also not reach a manufacturer nominal value, which may cause a fracture at a seam of a panorama image spliced by pictures shot by the two cameras, and a sense of incongruity may occur during viewing. In order to solve the problems, at present, a double-path fisheye image is generally directly output, and then image processing and correction are performed through a computer, but the processing increases the complexity of the system and cannot meet the requirement of real-time performance.
Disclosure of Invention
The invention aims to provide a real-time two-way video processing system and a real-time two-way video processing method based on a programmable chip, which can eliminate errors between acquired images after two-way images are acquired and can output the images in real time.
In order to achieve the purpose, the invention provides a real-time two-way video processing system based on a programmable chip, which comprises an off-chip memory DDR, an intermediate parameter calculation module and a remapping module, wherein the off-chip memory DDR is used for storing source images input by at least two parallel video input sources; the intermediate parameter calculation module is used for acquiring the calculation relationship of the absolute positions of the pixel points of the target image, the calculation relationship of the relative positions of the pixel points of the target image and the calculation relationship of the pixel values of the pixel points of the target image according to the image correction parameters; the remapping module is used for sequentially reading out the pixel values currently required by the pixels of the source image from corresponding positions in an off-chip memory DDR according to the calculation relationship of the absolute positions of the pixels of the target image and the calculation relationship of the relative positions of the pixels of the target image, and calculating the pixel values of the pixels of the target image one by one according to the calculation relationship of the pixel values of the pixels of the target image.
Optionally, the video image processing system further comprises a synchronous detection module, at least two input format conversion modules and an output format conversion module, wherein the synchronous detection module is used for receiving source images provided by at least two parallel video input sources and detecting the quality of all the source images; each input format conversion module is used for receiving a source image provided by one video input source, performing format conversion on image data of the source image, and storing the image data of the source image after format conversion into the off-chip memory DDR; the output format conversion module is used for converting the format of the target image calculated by the remapping module and outputting the target image after format conversion.
The device further comprises an image writing control module, an image reading control module and an on-chip memory, wherein the image writing control module is used for sequentially writing all the source images subjected to format conversion by the input format conversion module into the off-chip memory DDR; the image reading control module is used for reading out the cache data which is required to be read out from the off-chip memory DDR once and contains the pixel values required by the pixel points of the target image according to the calculation relation of the absolute positions of the pixel points of the target image, which is provided by the intermediate parameter calculation module, and storing the cache data containing the pixel values of the pixel points required by the target image into the on-chip memory so as to be used by the remapping module.
The image correction system further comprises a parameter selection module, wherein the parameter selection module is used for selecting required image correction parameters and supplying the required image correction parameters to the intermediate parameter calculation module for use.
The image correction system further comprises an image sequence reading control module and an image correction parameter calculation module, wherein the image sequence reading control module is used for reading image data from the off-chip memory DDR, the image correction parameter calculation module is used for acquiring a first image correction parameter according to the image data read by the image sequence reading control module, and the parameter selection module is used for selecting a required image correction parameter from a second image correction parameter and/or the first image correction parameter input from the outside and supplying the required image correction parameter to the intermediate parameter calculation module for use.
Optionally, the image writing control module includes a pixel fusion module and an image longitude and latitude expansion module,
when the real-time two-way video processing system is used for processing three-dimensional image acquisition, the image write-in control module performs pixel fusion on image data of all the source images through the pixel fusion module and stores the image data subjected to pixel fusion into the off-chip memory DDR;
when the real-time two-way video processing system is used for processing VR image acquisition, the image write-in control module is used for writing all image data of the source images into the DDR, the image data of the source images are respectively stored into the DDR after being longitudinally and longitudinally expanded according to the field angle parameters and each angle of yaw of the source images, which are input from the outside, through the image longitudinal and latitudinal expansion module.
On the other hand, the invention provides a real-time two-way video processing method based on a programmable chip, which comprises the following steps:
an off-chip memory DDR stores source images input by at least two parallel video input sources;
the intermediate parameter calculation module acquires the calculation relationship of the absolute positions of the pixel points of the target image, the calculation relationship of the relative positions of the pixel points of the target image and the calculation relationship of the pixel values of the pixel points of the target image according to the image correction parameters;
and the remapping module reads out the pixel values currently required by the pixel points of the source image from corresponding positions in an off-chip memory DDR in sequence according to the calculation relationship of the absolute positions of the pixel points of the target image and the calculation relationship of the relative positions of the pixel points of the target image, and calculates the pixel values of the pixel points of the target image one by one according to the calculation relationship of the pixel values of the pixel points of the target image so as to obtain the internal image data of the target pixel, thereby forming the target image.
Optionally, the processing flow for three-dimensional image acquisition specifically includes the following steps:
the method comprises the steps that a synchronous detection module receives source images input by at least two parallel video input sources and detects the synchronism of all the source images;
an input format conversion module receives a source image input by the video input source and performs format conversion on the source image;
the image writing control module performs pixel fusion on all the source images after format conversion to obtain fused image data, and stores the fused image data into an off-chip memory DDR;
the intermediate parameter calculation module calculates the calculation relation of the absolute positions of the pixel points of the target image, the calculation relation of the relative positions of the pixel points of the target image and the calculation relation of the pixel values of the pixel points of the target image according to the image correction parameters;
the image reading control module reads out cache data which is required to be read out from an off-chip memory DDR once and contains pixel values required by the pixel points of the target image according to the calculation relation of the absolute positions of the pixel points of the target image, and stores the cache data into the on-chip memory;
the remapping module calculates the pixel value of the pixel point of the target image according to the pixel point output sequence of the target image, the calculation relation of the relative position of the pixel point of the target image and the calculation relation of the pixel value of the pixel point of the target image so as to obtain the internal image data of the target pixel, thereby forming the target image; and
and the output format conversion module converts the format of the target image and outputs the target image after format conversion.
Optionally, the processing flow for VR image acquisition specifically includes the following steps:
the method comprises the steps that a synchronous detection module receives source images input by at least two parallel video input sources and detects the synchronism of all the source images;
an input format conversion module receives a source image input by the video input source and performs format conversion on the source image;
the image writing control module performs longitude and latitude expansion on all the source images after format conversion according to externally input field angle parameters and the yaw angle of each source image and then respectively stores the source images into an off-chip memory DDR;
the intermediate parameter calculation module calculates the calculation relation of the absolute positions of the pixel points of the target image, the calculation relation of the relative positions of the pixel points of the target image and the calculation relation of the pixel values of the pixel points of the target image according to the image correction parameters;
the image reading control module reads out cache data which is required to be read out from an off-chip memory DDR once and contains pixel values required by the pixel points of the target image according to the calculation relation of the absolute positions of the pixel points of the target image, and stores the cache data into the on-chip memory;
the remapping module sequentially reads out pixel values required by pixel points of a source image from corresponding positions in an on-chip memory according to the calculation relationship of the relative positions of the pixel points of a target image, and calculates the pixel values of the pixel points of the target image one by one according to the calculation relationship of the pixel values of the pixel points of the target image so as to obtain internal image data of the target pixel and form the target image; and
and the output format conversion module converts the format of the target image and outputs the target image after format conversion.
Furthermore, the pixel fusion module of the image writing control module fuses the image data of all the source images into image data of left and right half side-by-side, and stores the fused image data of the left and right half side-by-side into the off-chip memory according to a 3D format.
Further, the calculating relationship of the absolute position of the pixel point of the target image, the calculating relationship of the relative position of the pixel point of the target image and the calculating relationship of the pixel value of the pixel point of the target image by the intermediate parameter calculating module according to the image correction parameter includes:
the parameter selection module selects the required image correction parameters;
the intermediate parameter calculation module establishes an affine matrix according to the image correction parameters provided by the parameter selection module and generates a remapping relation so as to obtain a calculation relation of absolute positions of pixel points of the target image;
and according to the data reading and writing condition in the on-chip memory, the intermediate parameter calculation module calculates the calculation relationship of the relative positions of the pixel points of the target image according to the calculation relationship of the absolute positions of the pixel points of the target image, and simultaneously calculates the calculation relationship of the pixel values of the pixel points of the target image according to the absolute positions of the pixel points of the current target image and the space alternation condition of the on-chip memory.
Further, the remapping module reads pixel values required by the pixel points of the source image from corresponding positions in the on-chip memory in sequence according to the pixel point output sequence of the target image and the calculation relationship of the relative positions of the pixel points of the target image and the address of the pixel points of the current target pixel in the on-chip memory provided by the on-chip memory, and performs interpolation operation on the pixel values of the pixel points of the target image according to the calculation relationship of the pixel values of the pixel points of the target image to obtain the pixel values of the pixel points of the target image one by one so as to obtain the internal image data of the target pixel, thereby forming the target image.
Further, the calculating relationship of the absolute position of the pixel point of the target image, the calculating relationship of the relative position of the pixel point of the target image and the calculating relationship of the pixel value of the pixel point of the target image by the intermediate parameter calculating module according to the image correction parameter includes:
the parameter selection module selects the required image correction parameters;
the intermediate parameter calculation module performs transmission transformation to generate a remapping relation according to the image correction parameter so as to obtain a calculation relation of absolute positions of pixel points of the target image;
and the intermediate parameter calculation module calculates the calculation relation of the relative positions of the pixel points of the target image and the calculation relation of the pixel values of the pixel points of the target image according to the calculation relation of the absolute positions of the pixel points of the target image, and provides the calculation relations to the remapping module.
Further, the remapping module reads the required pixel values of the pixel points of the source image from the corresponding positions in the on-chip memory in sequence according to the pixel point output sequence of the target image and the calculation relationship of the relative positions of the pixel points of the target image, and also according to the address of the pixel points of the current target pixel provided by the on-chip memory in the on-chip memory, and performs interpolation operation on the pixel values of the pixel points of the target image according to the calculation relationship of the pixel values of the pixel points of the target image, so as to calculate the pixel values of the pixel points of the target image without overlapping regions of all the source images one by one, and performs superposition calculation on the overlapping regions after the interpolation operation when all the source images have overlapping regions, so as to obtain the pixel values of the pixel points of the target image one by one, so as to obtain the internal image data of the target pixel, thereby forming a target image.
Optionally, there is an image overlap region between source images input by at least two parallel video input sources.
Compared with the prior art, the invention has at least the following beneficial effects:
the invention provides a real-time two-way video processing system and a method thereof based on a programmable chip, wherein the real-time two-way video processing system comprises an off-chip memory DDR, an intermediate parameter calculation module and a remapping module, wherein the off-chip memory DDR is used for storing source images input by at least two parallel video input sources; the intermediate parameter calculation module is used for acquiring the calculation relationship of the absolute positions of the pixel points of the target image, the calculation relationship of the relative positions of the pixel points of the target image and the calculation relationship of the pixel values of the pixel points of the target image according to the image correction parameters; the remapping module is used for reading out the pixel values currently required by the pixels of the source image from the corresponding positions in the off-chip memory DDR in sequence according to the calculation relationship of the absolute positions of the pixels of the target image and the calculation relationship of the relative positions of the pixels of the target image, and calculating the pixel values of the pixel points of the target image one by one according to the pixel values of the pixel points of the target image for the pixel values currently required by the pixel points of the source image, can correct the error of the two paths of source images, eliminates the error existing between the two paths of original images, has the time delay of millisecond level, ensures that the time delay can be ignored, therefore, safe and high-quality images can be directly obtained for direct viewing, a motor mechanical correction link or a subsequent error processing link in the front can be omitted, the system is further simplified, and the cost is reduced.
Drawings
Fig. 1 is a schematic structural diagram of a real-time two-way video processing system based on a programmable chip according to a first embodiment of the present invention;
fig. 2 is a schematic flowchart of a real-time two-way video processing method based on a programmable chip according to a first embodiment of the present invention;
fig. 3 is a schematic structural diagram of a real-time two-way video processing system based on a programmable chip according to a second embodiment of the present invention;
fig. 4 is a flowchart illustrating a real-time two-way video processing method based on a programmable chip according to a second embodiment of the present invention.
Detailed Description
The core idea of the invention is to provide a real-time two-way video processing system based on a programmable chip, which comprises an off-chip memory DDR, an intermediate parameter calculation module and a remapping module, wherein the off-chip memory DDR is used for storing source images input by at least two parallel video input sources; the intermediate parameter calculation module is used for acquiring the calculation relationship of the absolute positions of the pixel points of the target image, the calculation relationship of the relative positions of the pixel points of the target image and the calculation relationship of the pixel values of the pixel points of the target image according to the image correction parameters; the remapping module is used for sequentially reading out the pixel values currently required by the pixels of the source image from corresponding positions in an off-chip memory DDR according to the calculation relationship of the absolute positions of the pixels of the target image and the calculation relationship of the relative positions of the pixels of the target image, and calculating the pixel values of the pixels of the target image one by one according to the calculation relationship of the pixel values of the pixels of the target image.
The real-time two-way video processing method of the programmable chip comprises the following steps:
an off-chip memory DDR stores source images input by at least two parallel video input sources;
the intermediate parameter calculation module acquires the calculation relationship of the absolute positions of the pixel points of the target image, the calculation relationship of the relative positions of the pixel points of the target image and the calculation relationship of the pixel values of the pixel points of the target image according to the image correction parameters;
and the remapping module reads out the pixel values currently required by the pixel points of the source image from corresponding positions in an off-chip memory DDR in sequence according to the calculation relationship of the absolute positions of the pixel points of the target image and the calculation relationship of the relative positions of the pixel points of the target image, and calculates the pixel values of the pixel points of the target image one by one according to the calculation relationship of the pixel values of the pixel points of the target image so as to obtain the internal image data of the target pixel, thereby forming the target image.
The present invention will be described in further detail below with reference to a real-time two-way video processing system based on a programmable chip and a method thereof. The present invention will now be described in more detail with reference to the accompanying drawings, in which preferred embodiments of the invention are shown, it being understood that one skilled in the art may modify the invention herein described while still achieving the advantageous effects of the invention. Accordingly, the following description should be construed as broadly as possible to those skilled in the art and not as limiting the invention.
In the interest of clarity, not all features of an actual implementation are described. In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail. It will of course be appreciated that in the development of any such actual embodiment, numerous implementation-specific details must be set forth in order to achieve the developer's specific goals, such as compliance with system-related and business-related constraints, which will vary from one implementation to another. Moreover, it will be appreciated that such a development effort might be complex and time-consuming, but would nevertheless be a routine undertaking for those of ordinary skill in the art.
In order to make the objects and features of the present invention more comprehensible, embodiments of the present invention are described in detail below with reference to the accompanying drawings. It is to be noted that the drawings are in a very simplified form and are all used in a non-precise ratio for the purpose of facilitating and distinctly aiding in the description of the embodiments of the invention.
Fig. 1 is a schematic structural diagram of a real-time two-way video processing system based on a programmable chip according to this embodiment. As shown in fig. 1, the present embodiment provides a real-time two-way video processing system based on a programmable chip. The real-time double-channel video processing system can be applied to the fields of 3D shooting, a 3D microscope, heterogeneous image processing, three-dimensional reconstruction, large-pixel image splicing, VR splicing and the like. The real-time two-way video processing system comprises a synchronous detection module 10, a format conversion module, an image writing control module 41, an off-chip memory DDR50, an image reading control module 42, an intermediate parameter calculation module 70, an on-chip memory 80 and a remapping module 30.
The synchronous detection module 10 is configured to receive source images provided by at least two parallel video input sources, detect the quality of all the source images to determine whether all the source images are synchronous, whether a video is interrupted, and the like, and further perform frame rate detection and provide a signal prompt for a synchronous detection abnormal result of all the source images, so as to facilitate processing by a field operator. In this embodiment, the real-time two-way video processing system faces processing during three-dimensional image acquisition, and the synchronous detection module 10 receives source images provided by two parallel video input sources, that is, a first source image and a second source image, where the two video input sources are both obtained by shooting with an SDI camera, for example, and an image overlapping area is provided between the two source images, that is, a shooting overlapping area is provided during shooting with the SDI camera. The format of the source image is, for example, SDI format, and the pixels of both the source images are the same, for example 1920x 1080.
The format conversion module comprises at least two input format conversion modules 21 and 22, the number of the input format conversion modules is the same as that of the video input sources, each input format conversion module is used for receiving a source image provided by one video input source and performing format conversion on image data of the source image to obtain image data of at least two source images in a uniform format, for example, clocks of the source images are uniformly converted into 300M to realize the uniformity of clock domains in the system, and the uniform clock frequency is larger than that of the source images provided by the video input sources, which is beneficial to the pixel fusion when a subsequent image is written into the control module 41; and the video stream format is uniformly converted into axi-stream, and the color space is uniformly converted into the encoding format of RGB, so that the image data processing in the real-time double-path video processing system is facilitated.
An FIFO is arranged between the image write-in control module 41 and the format conversion module, and the FIFO is used for buffering the image data of all source images after format conversion by the format conversion module and sending the pixel data in the image data to be processed by the image write-in control module 41 to the image write-in control module 41. In this embodiment, FIFOs may be provided between other modules, for example, between the intermediate parameter calculation module and the image reading control module, and between the intermediate parameter calculation module and the remapping module.
The image write control module 41 is configured to sequentially write all the source images (i.e., source images with a uniform format) with the format converted into the off-chip memory DDR50, and if necessary, sequentially process each pixel point of all the source images and rearrange the storage location of each pixel point according to the actual processing requirement. In this embodiment, the image writing control module 41 includes a pixel fusion module 411, and the pixel fusion module 411 may perform pixel fusion on the image data of at least two source images to obtain a fused image data.
Specifically, the image write control module 41 is configured to receive image data of all source images after format conversion, and the pixel fusion module 411 performs pixel fusion on the image data of the two source images according to the side-by-side format. Taking a left half side-by-side (left and right are combined and then transversely halved), and taking a color space as RGB as an example, firstly processing a line of pixel data of a first source image, namely grouping a line of pixel points of the first source image, dividing every two adjacent pixel points into a group, taking an average value of pixel values of the two pixel points as a pixel value of each group of pixel points to obtain a half-line of pixel values, and sequentially storing the processed half-line of pixel data into an off-chip memory DDR 50; and then processing a line of pixel data (corresponding to the line just processed by the first source image) of the second source image, namely grouping a line of pixel points of the second source image, dividing every two adjacent pixel points into a group, taking the average value of the pixel values of the two pixel points for the pixel value of each group of pixel points to obtain a half-line of pixel values, sequentially storing the processed half-line of pixel data into the off-chip memory DDR50 to obtain a line of pixel data after pixel fusion, and performing pixel fusion on the pixel data of all the lines of the two source images according to the method to obtain a frame of half side-by-side video (namely, fusing the image data of the two source images to obtain a fused image data).
In order to facilitate reading and obtain the address of the pixel point of the source image in the off-chip memory DDR50 from the row and column sequence number more easily, the inner space of the off-chip memory DDR50 is distributed according to an array, and each row, each column and each frame in the off-chip memory DDR50 occupy the inner space of the off-chip memory DDR50 by taking the power of 2 as a unit, which is to allow a certain redundant space in the off-chip memory DDR 50.
Taking as an example that one frame of pixel fused image data with a pixel size of 1920x1080 and a pixel size of 4Byte is stored in the off-chip memory DDR 50: the pixel value of each pixel point of one frame of image data occupies 4 pixel internal Byte addresses, so that [ bit1: bit0] in each address in the off-chip memory DDR50 represents the pixel internal Byte address; each row of 1920 pixels occupies the space required by 2048 pixels in the internal space of the off-chip memory DDR50, so that [ bit 12: bit2 represents the column sequence number of the pixel point; the image data of each frame occupies space required by 2048 lines of pixel points in the internal space of the off-chip memory DDR50, so that [ bit23: bit13] in each address in the off-chip memory DDR50 represents the line sequence number of the pixel point. According to the above planning of the address space, the pixel data of the frame of image data transmitted from the pixel fusion module 411 is sequentially written into the off-chip memory DDR 50.
The intermediate parameter calculating module 70 is configured to calculate, according to the image correction parameter, a calculation relationship of absolute positions of pixel points of the target image (i.e., a position of a pixel point of the source image corresponding to each pixel point of the target image), and a calculation relationship of absolute positions of pixel points of the target image, calculate a calculation relationship of relative positions of pixel points of the target image, and a calculation relationship of pixel values of pixel points of the target image, and provide the position of the pixel point of the source image corresponding to each pixel point of the target image to the image reading control module 42, so that the image reading control module 42 reads out, from the off-chip memory DDR50, the cache data including the pixel values required by the pixel points of the target image, and further sends the calculation relationship of the relative positions of the pixel points of the target image, and the calculation relationship of pixel values of the pixel points of the target image to the remapping module 30, so as to obtain image data of the target image. The absolute position of the pixel point of the target image refers to the position of the pixel point of the target image read from the off-chip memory DDR50, that is, the position of the coordinate of the pixel point of the target image in the source image. The relative position of the pixel point of the target image refers to an address of writing the pixel data read from the off-chip memory DDR50 into the RAM, and an address of the pixel point of the target image read from the RAM.
The on-chip memory 80 is, for example, a RAM, which serves as a buffer space that can accommodate several rows of pixel data for direct random access by the remapping module 30. The buffer space replaces the stored pixel data in turn in a first-in first-out mode, and a counter is arranged to record the number of lines of reading/writing of each frame of image data and the reading/writing of the RAM, so that the RAM is guaranteed to be not empty or not full.
The image modification parameters may include a first image modification parameter obtained via a real-time two-way video processing system internal calculation and/or a second image modification parameter obtained by a real-time two-way video processing system external input. In order to select the image correction parameter source, the real-time two-way video processing system further comprises a parameter selection module 63, and when the image correction parameters comprise a first image correction parameter and a second image correction parameter, the parameter selection module 63 preferentially selects the first image correction parameter.
The real-time two-way video processing system further comprises an image sequence reading control module 61 and an image correction parameter calculation module 62, which can provide a first image correction parameter to the intermediate parameter calculation module 70, and can also output the first image correction parameter for prompting a user. The image sequence reading control module 61 is configured to read the image data after pixel fusion of each frame, and is used by the image correction parameter calculation module 62. The image correction parameter calculation module 62 is configured to obtain a first image correction parameter by calculating errors such as rotation, translation, and scaling between the first source image and the second source image according to the image data provided by the image sequence reading control module, provide the first image correction parameter to the intermediate parameter calculation module 70 through the parameter selection module 63, and output the first image correction parameter for prompting a user. In the present embodiment, FIFOs are provided between the parameter selection module 63 and the intermediate parameter calculation module, and between the image sequence reading control module 61 and the image correction parameter calculation module 62.
The image reading control module reads out the cache data containing the pixel values required by the pixel points of the target image, which needs to be read out from the off-chip memory DDR50 at a time, and stores the cache data containing the pixel values of the pixel points required by the target image into the on-chip memory 80 for use by the remapping module 30.
The remapping module 30 is configured to read out pixel values required by the pixel points of the source image from the corresponding positions in the on-chip memory 80 in sequence according to the output sequence of the pixel points of the target image and the calculation relationship of the relative positions of the pixel points of the target image provided by the intermediate parameter calculation module 70, and also according to the address of the pixel points of the current target pixel provided by the on-chip memory 80 in the on-chip memory 80, and obtain the pixel values of the pixel points of the target image one by one through interpolation operation (specifically, bilinear interpolation operation, for example) to obtain the internal image data of the target pixel, thereby forming the target image.
The format conversion module further includes an output format conversion module 23, and the output format conversion module 23 is configured to convert the format of the target image into a format (such as axi-stream) required by output, and output the target image after format conversion.
The embodiment is based on a Xilinx Ultrascale + platform and adopts a Xilinx ZCU106 development board, wherein a programmable chip adopts Xilinx XCZU7 EV; the qpsi Flash uses the MT25QU512 of Micron as a program memory; the off-chip memory DDR50 adopts Micron MT40A256M16 GE-075E; SI570 on the programmable chip provides the master clock signal.
Fig. 2 is a schematic flowchart of a real-time two-way video processing method based on a programmable chip according to this embodiment. As shown in fig. 2, this embodiment further provides a real-time two-way video processing method based on a programmable chip, which is oriented to a processing flow during three-dimensional image acquisition to eliminate an error between a first source image and a second source image, and because the processing flow oriented to three-dimensional image acquisition is the same as the processing flow during large pixel image stitching, the processing flow oriented to three-dimensional image acquisition is taken as an example for description, and the processing flow oriented to large pixel image stitching is to be able to stitch the first source image and the second source image together. The real-time two-way video processing method comprises the following steps:
step S11: the synchronous detection module 10 receives source images input by at least two parallel video input sources and detects the synchronism of all the source images;
step S12: an input format conversion module receives a source image input by the video input source and performs format conversion on the source image;
step S13: the image write control module 41 performs pixel fusion on all the source images after format conversion to obtain fused image data, and stores the fused image data into an off-chip memory DDR 50;
step S14: the intermediate parameter calculation module 70 calculates the calculation relationship of the absolute positions of the pixels of the target image, the calculation relationship of the relative positions of the pixels of the target image, and the calculation relationship of the pixel values of the pixels of the target image according to the image correction parameters;
step S15: the image reading control module 42 reads out, according to the calculation relationship of the absolute positions of the pixel points of the target image, the cache data which is required to be read out from the off-chip memory DDR50 once and contains the pixel values required by the pixel points of the target image, and stores the cache data into the on-chip memory 80;
step S16: the remapping module 30 calculates the pixel values of the pixels of the target image according to the pixel output sequence of the target image, the calculation relationship of the relative positions of the pixels of the target image and the calculation relationship of the pixel values of the pixels of the target image to obtain the internal image data of the target pixel, thereby forming the target image; and
step S17: the output format conversion module 23 converts the format of the target image and outputs the format-converted target image.
In step S11, the synchronization detection module 10 determines whether the source images provided by the at least two parallel video input sources are synchronized, whether the video is interrupted, and the like, and further performs frame rate detection, and provides a prompt signal for an abnormal detection result, so as to facilitate the processing of the field operator. In this embodiment, the synchronous detection module 10 receives input source images provided by two parallel video input sources, namely a first source image and a second source image, and an image overlap region is formed between the first source image and the second source image.
In step S12, when the synchronization detection module 10 detects that all source images are synchronized, a video is not interrupted, and frame rate detection is abnormal, each input format conversion module receives a source image input by the video input source and performs format conversion on the source image to unify formats of all the source images. In this embodiment, the input format conversion module unifies the formats of the clock domain, the video stream format, the encoding format, and the like of all the source images, so as to facilitate the internal calculation of the real-time two-way video processing system.
In step S13, the pixel fusion module 411 of the image write control module 41 fuses the image data of all the source images into left and right half side-by-side image data, and stores the fused left and right half side-by-side image data into the off-chip memory DDR50 according to the 3D format.
Step S14 specifically includes:
firstly, the image correction parameters include a first image correction parameter and/or a second image correction parameter, when the image correction parameters at least include the first image correction parameter, the parameter selection module 63 preferentially selects the first image correction parameter, and selects the second image correction parameter when there is no first image correction parameter, and sends the selected image correction parameter to the intermediate parameter calculation module 70; the first image correction parameter is image data obtained by reading the complete image data fusion of one frame from the off-chip memory DDR50 through the image sequence reading control module 61, and is supplied to the image correction parameter calculation module 62, and the image correction parameter calculation module 62 calculates errors such as rotation, translation, scaling and the like between the first source image and the second source image to obtain the first image correction parameter.
Next, the intermediate parameter calculation module 70 establishes an affine matrix according to the image correction parameters (i.e. the first image correction parameters or the second image correction parameters) provided by the parameter selection module 63, and generates a remapping relationship, so as to obtain a calculation relationship of the absolute positions of the pixel points of the target image. Specifically, according to the coordinate sequence of the pixel points of the target image, the position of the pixel point of the source image corresponding to each output pixel point is calculated one by one, that is, the coordinates (Xs, Ys) of the pixel point of each target image and the pixel point of the source image corresponding thereto are calculated one by one from the coordinates (0, 0) of the pixel point of the target image, and are provided to the image reading control module 42, that is, the position of the pixel point of the source image corresponding to each pixel point of the target image. The absolute position of the pixel point of the target image refers to the position of the pixel point of the target image read from the off-chip memory DDR50, that is, the coordinate of the pixel point of the target image in the source image.
Due to the limited space of the on-chip memory 80, the strategy of reading the entire row of the on-chip memory 80 makes the number of rows that the on-chip memory 80 can store limited, and makes the data in the on-chip memory 80 follow the first-in first-out management. Therefore, next, according to the data reading and writing condition in the on-chip memory 80, the intermediate parameter calculating module 70 calculates the calculation relationship (i.e. the current writable address) of the relative position of the pixel point of the target image according to the position of the pixel point of the source image corresponding to each pixel point of each target image, and provides the calculated relationship to the remapping module 30; meanwhile, according to the absolute position of the pixel point of the current target image and the space alternation condition of the on-chip memory 80, the calculation relation of the pixel value of the pixel point of the target image is calculated and provided to the remapping module 30. In this embodiment, for the case that the position of the pixel point of the target image does not coincide with the position of any pixel point in the source image, an interpolation operation needs to be performed, and the pixel value relationship of the pixel point of the target image calculates the pixel full weight of the pixel point of the target image according to the distance between the position of the pixel point of the target image and the pixel points of 4 source images, and for the case that the position of the pixel point of the target image coincides with the position of any pixel point in the source image, the interpolation operation is not needed, and the pixel value of the pixel point of the source image is directly copied and provided to the remapping module 30.
Step S16 specifically includes: the remapping module 30 sequentially reads pixel values required by pixel points of the source image from corresponding positions in the on-chip memory 80 according to the pixel point output sequence of the target image and the calculation relationship of the relative positions of the pixel points of the target image and the address of the pixel point of the current target pixel provided by the on-chip memory 80 in the on-chip memory 80, so as to find out 4 pixel values around the target pixel, and perform interpolation operation (specifically, for example, bilinear interpolation operation) on the pixel values of the pixel points of the target image according to the calculation relationship of the pixel values of the pixel points of the target image, so as to obtain the pixel values of the pixel points of the target image one by one, so as to obtain the internal image data of the target pixel, thereby forming the target image.
Example two
Compared with the first embodiment, the difference of the present embodiment is that fig. 3 is a schematic structural diagram of the real-time two-way video processing system based on the programmable chip of the present embodiment. As shown in fig. 3, the real-time two-channel video processing system based on the programmable chip is applied to the fields of VR splicing after VR image acquisition and the like. The image writing control module 41 of the real-time two-way video processing system further includes an image longitude and latitude unfolding module, and the image longitude and latitude unfolding module is configured to perform longitude and latitude unfolding according to an externally input field angle parameter and a yaw angle (e.g., yaw angles of 0 ° and 180 °) of each of the source images. The image write control module 41 is configured to receive all source images after format conversion, perform warp and weft expansion on the source images after format conversion through the image warp and weft expansion module according to an externally input field angle parameter and a yaw angle (e.g., yaw angles of 0 ° and 180 °) of each of the source images, and write the source images into the off-chip memory DDR50, where the source images do not pass through the pixel fusion module 411, that is, the image data of all the source images are not subjected to pixel fusion, but the image data of all the source images are stored in the off-chip memory DDR 50.
Fig. 4 is a schematic flowchart of a real-time two-way video processing method based on a programmable chip according to this embodiment. As shown in fig. 4, the real-time two-way video processing method of the embodiment is oriented to a processing flow during VR image acquisition. The real-time two-way video processing method comprises the following steps:
step S21: the synchronous detection module 10 receives source images input by at least two parallel video input sources and detects the synchronism of all the source images;
step S22: an input format conversion module receives a source image input by the video input source and performs format conversion on the source image;
step S23: the image write control module 41 respectively expands all the source images after format conversion according to externally input field angle parameters and the yaw angle of each source image and stores the expanded source images into an off-chip memory DDR 50;
step S24: the intermediate parameter calculating module 70 calculates the calculation relationship of the absolute positions of the pixels of the target image, the calculation relationship of the relative positions of the pixels of the target image and the calculation relationship of the pixel values of the pixels of the target image according to the image correction parameters;
step S25: the image reading control module 42 reads out, according to the calculation relationship of the absolute positions of the pixel points of the target image, the cache data which is required to be read out from the off-chip memory DDR50 once and contains the pixel values required by the pixel points of the target image, and stores the cache data into the on-chip memory 80;
step S26: the remapping module 30 sequentially reads out the pixel values required by the pixel points of the source image from the corresponding positions in the on-chip memory 80 according to the calculation relationship of the relative positions of the pixel points of the target image, and calculates the pixel values of the pixel points of the target image one by one according to the calculation relationship of the pixel values of the pixel points of the target image so as to obtain the internal image data of the target image, thereby forming the target image; and
step S27: the output format conversion module 23 converts the format of the target image and outputs the format-converted target image.
In step S23, the image writing control module 41 does not fuse the image data of the format-converted source images, but directly stores the image data of each source image, which is subjected to warp and weft expansion according to the externally input field angle parameter and the yaw angle of each source image, into the DDR respectively.
Step S24 includes:
firstly, the image correction parameters include a first image correction parameter and/or a second image correction parameter, when the image correction parameters at least include the first image correction parameter, the parameter selection module 63 preferentially selects the first image correction parameter, and selects the second image correction parameter when there is no first image correction parameter, and sends the selected image correction parameter to the intermediate parameter calculation module 70; the first image correction parameter is obtained by reading two image data of the complete double fisheyes after longitude and latitude expansion from the off-chip memory DDR50 through the image sequence reading control module 61 and supplying the two image data to the image correction parameter calculation module 62, and the image correction parameter calculation module 62 extracts feature points and calculates relative positions through the overlapped part of the first image after longitude and latitude expansion and the second image after longitude and latitude expansion, so as to obtain the first image correction parameter.
Next, the intermediate parameter calculating module 70 performs transmission transformation according to the image correction parameter to generate a remapping relationship, so as to obtain a calculation relationship of the absolute positions of the pixels of the target image. Specifically, the intermediate parameter calculation module 70 performs transmission transformation on the panoramic image according to the image correction parameter to eliminate the position error of each pixel, so as to generate a remapping relationship, so as to obtain a calculation relationship of the absolute position of the pixel of the target image. The matrix used for the transmission transformation can be obtained through external input or through calculation in a system.
Next, the intermediate parameter calculating module 70 calculates the calculation relationship of the relative positions of the pixels of the target image and the calculation relationship of the pixel values of the pixels of the target image according to the calculation relationship of the absolute positions of the pixels of the target image, and provides the calculation relationships to the remapping module 30.
In step S26, the remapping module 30 reads the required pixel values of the pixel points of the source image from the corresponding positions in the on-chip memory 80 in sequence according to the pixel point output sequence of the target image and the calculation relationship of the relative positions of the pixel points of the target image, and also according to the address of the pixel point of the current target pixel provided by the on-chip memory 80 in the on-chip memory 80, so as to find out the 4 pixel values around the target pixel, and performs interpolation operation (specifically, for example, bilinear interpolation operation) on the pixel values of the pixel points of the target image according to the calculation relationship of the pixel values of the pixel points of the target image, so as to calculate the pixel values of the pixel points of the target image in which all the source images do not overlap, and perform superposition calculation on the superposition region after interpolation operation when all the source images have overlap regions, so as to obtain the pixel values of the pixel points of the target image one by one, to obtain internal image data of the target pixel, thereby forming a target image.
In summary, the real-time two-way video processing system and method based on the programmable chip of the invention can correct the error of two-way source images, eliminate the error between two-way original images, and the time delay is millisecond level, so that the time delay can be ignored, thereby directly obtaining safe and high-quality images for direct viewing, and also can save a previous motor mechanical correction link or a subsequent error processing link, further simplify the system and reduce the cost.
It is to be understood that while the present invention has been described in conjunction with the preferred embodiments thereof, it is not intended to limit the invention to those embodiments. It will be apparent to those skilled in the art from this disclosure that many changes and modifications can be made, or equivalents modified, in the embodiments of the invention without departing from the scope of the invention. Therefore, any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention are still within the scope of the protection of the technical solution of the present invention, unless the contents of the technical solution of the present invention are departed.

Claims (15)

1. A real-time two-way video processing system based on a programmable chip is characterized by comprising an off-chip memory DDR, an intermediate parameter calculation module and a remapping module, wherein the off-chip memory DDR is used for storing source images input by at least two parallel video input sources; the intermediate parameter calculation module is used for acquiring the calculation relationship of the absolute positions of the pixel points of the target image, the calculation relationship of the relative positions of the pixel points of the target image and the calculation relationship of the pixel values of the pixel points of the target image according to the image correction parameters; the remapping module is used for sequentially reading out the pixel values currently required by the pixels of the source image from corresponding positions in an off-chip memory DDR according to the calculation relationship of the absolute positions of the pixels of the target image and the calculation relationship of the relative positions of the pixels of the target image, and calculating the pixel values of the pixels of the target image one by one according to the calculation relationship of the pixel values of the pixels of the target image.
2. The real-time two-way video processing system according to claim 1, further comprising a synchronization detection module, at least two input format conversion modules and an output format conversion module, said synchronization detection module for receiving source images provided by at least two parallel video input sources and detecting the quality of all said source images; each input format conversion module is used for receiving a source image provided by one video input source, performing format conversion on image data of the source image, and storing the image data of the source image after format conversion into the off-chip memory DDR; the output format conversion module is used for converting the format of the target image calculated by the remapping module and outputting the target image after format conversion.
3. The real-time two-way video processing system according to claim 2, further comprising an image writing control module, an image reading control module and an on-chip memory, wherein the image writing control module is configured to sequentially write all the source images after format conversion by the input format conversion module into the off-chip memory DDR; the image reading control module is used for reading out the cache data which is required to be read out from the off-chip memory DDR once and contains the pixel values required by the pixel points of the target image according to the calculation relation of the absolute positions of the pixel points of the target image, which is provided by the intermediate parameter calculation module, and storing the cache data containing the pixel values of the pixel points required by the target image into the on-chip memory so as to be used by the remapping module.
4. The real-time two-way video processing system of claim 3, further comprising a parameter selection module for selecting desired image modification parameters for use by the intermediate parameter calculation module.
5. The real-time two-way video processing system according to claim 4, further comprising an image sequence reading control module and an image correction parameter calculation module, wherein the image sequence reading control module is configured to read image data from the off-chip memory DDR, the image correction parameter calculation module is configured to obtain a first image correction parameter according to the image data read by the image sequence reading control module, and the parameter selection module is configured to select a required image correction parameter from a second image correction parameter and/or the first image correction parameter input from the outside and supply the selected image correction parameter to the intermediate parameter calculation module for use.
6. The real-time two-way video processing system according to any one of claims 1 to 5, wherein the image writing control module comprises a pixel fusion module and an image longitude and latitude expansion module,
when the real-time two-way video processing system is used for processing three-dimensional image acquisition, the image writing control module performs pixel fusion on image data of all the source images through the pixel fusion module and stores the image data subjected to pixel fusion into the off-chip memory DDR;
when the real-time two-way video processing system is used for processing VR image acquisition, the image write-in control module is used for writing all image data of the source images into the DDR, the image data of the source images are respectively stored into the DDR after being longitudinally and longitudinally expanded according to the field angle parameters and each angle of yaw of the source images, which are input from the outside, through the image longitudinal and latitudinal expansion module.
7. A real-time two-way video processing method based on a programmable chip, comprising the real-time two-way video processing system of any one of claims 1 to 6, characterized by comprising the following steps:
an off-chip memory DDR stores source images input by at least two parallel video input sources;
the intermediate parameter calculation module acquires the calculation relationship of the absolute positions of the pixel points of the target image, the calculation relationship of the relative positions of the pixel points of the target image and the calculation relationship of the pixel values of the pixel points of the target image according to the image correction parameters;
and the remapping module sequentially reads out the pixel values currently required by the pixel points of the source image from the corresponding positions in the off-chip memory DDR according to the calculation relationship of the absolute positions of the pixel points of the target image and the calculation relationship of the relative positions of the pixel points of the target image, and calculates the pixel values of the pixel points of the target image one by one according to the calculation relationship of the pixel values of the pixel points of the target image so as to obtain the internal image data of the target pixel, thereby forming the target image.
8. The real-time two-way video processing method of claim 7, which is oriented to a processing flow during three-dimensional image acquisition, and specifically comprises the following steps:
the method comprises the steps that a synchronous detection module receives source images input by at least two parallel video input sources and detects the synchronism of all the source images;
an input format conversion module receives a source image input by the video input source and performs format conversion on the source image;
the image writing control module performs pixel fusion on all the source images after format conversion to obtain fused image data, and stores the fused image data into an off-chip memory DDR;
the intermediate parameter calculation module calculates the calculation relation of the absolute positions of the pixel points of the target image, the calculation relation of the relative positions of the pixel points of the target image and the calculation relation of the pixel values of the pixel points of the target image according to the image correction parameters;
the image reading control module reads out cache data which is required to be read out from a DDR (double data rate) of an off-chip memory and contains pixel values required by the pixel points of the target image at a time according to the calculation relation of the absolute positions of the pixel points of the target image, and stores the cache data into the on-chip memory;
the remapping module calculates the pixel value of the pixel point of the target image according to the pixel point output sequence of the target image, the calculation relation of the relative position of the pixel point of the target image and the calculation relation of the pixel value of the pixel point of the target image so as to obtain the internal image data of the target pixel, thereby forming the target image; and
and the output format conversion module converts the format of the target image and outputs the target image after format conversion.
9. The real-time two-way video processing method of claim 7, which is directed to a processing flow during VR image acquisition, and specifically comprises the following steps:
the method comprises the steps that a synchronous detection module receives source images input by at least two parallel video input sources and detects the synchronism of all the source images;
an input format conversion module receives a source image input by the video input source and performs format conversion on the source image;
the image writing control module respectively expands all the source images after format conversion according to externally input field angle parameters and the yaw angle of each source image and stores the expanded source images into an off-chip memory DDR;
the intermediate parameter calculation module calculates the calculation relation of the absolute positions of the pixel points of the target image, the calculation relation of the relative positions of the pixel points of the target image and the calculation relation of the pixel values of the pixel points of the target image according to the image correction parameters;
the image reading control module reads out cache data which is required to be read out from an off-chip memory DDR once and contains pixel values required by the pixel points of the target image according to the calculation relation of the absolute positions of the pixel points of the target image, and stores the cache data into the on-chip memory;
the remapping module sequentially reads out pixel values required by pixel points of a source image from corresponding positions in an on-chip memory according to the calculation relationship of the relative positions of the pixel points of a target image, and calculates the pixel values of the pixel points of the target image one by one according to the calculation relationship of the pixel values of the pixel points of the target image so as to obtain internal image data of the target pixel and form the target image; and
and the output format conversion module converts the format of the target image and outputs the target image after format conversion.
10. The real-time two-way video processing method of claim 8,
and the pixel fusion module of the image write-in control module fuses the image data of all the source images into image data of left and right half side-by-side, and stores the fused image data of the left and right half side-by-side into the off-chip memory according to a 3D format.
11. The real-time two-way video processing method according to claim 8, wherein the intermediate parameter calculating module calculates the calculation relationship of the absolute position of the pixel point of the target image, the calculation relationship of the relative position of the pixel point of the target image, and the calculation relationship of the pixel value of the pixel point of the target image according to the image modification parameters includes:
the parameter selection module selects the required image correction parameters;
the intermediate parameter calculation module establishes an affine matrix according to the image correction parameters provided by the parameter selection module and generates a remapping relation so as to obtain a calculation relation of absolute positions of pixel points of the target image;
and according to the data reading and writing condition in the on-chip memory, the intermediate parameter calculation module calculates the calculation relationship of the relative positions of the pixel points of the target image according to the calculation relationship of the absolute positions of the pixel points of the target image, and simultaneously calculates the calculation relationship of the pixel values of the pixel points of the target image according to the absolute positions of the pixel points of the current target image and the space alternation condition of the on-chip memory.
12. The real-time two-way video processing method according to claim 8, wherein the remapping module sequentially reads pixel values required by the pixels of the source image from corresponding positions in the on-chip memory according to the pixel output order of the target image and the calculation relationship of the relative positions of the pixels of the target image and the address of the pixel of the current target pixel in the on-chip memory provided by the on-chip memory, and performs interpolation operation on the pixel values of the pixels of the target image according to the calculation relationship of the pixel values of the pixels of the target image to obtain the pixel values of the pixels of the target image one by one to obtain the internal image data of the target image, thereby forming the target image.
13. The real-time two-way video processing method as claimed in claim 9, wherein the calculating of the absolute position of the pixel point of the target image, the calculating of the relative position of the pixel point of the target image and the calculating of the pixel value of the pixel point of the target image by the intermediate parameter calculating module according to the image modification parameters comprises:
the parameter selection module selects the required image correction parameters;
the intermediate parameter calculation module performs transmission transformation to generate a remapping relation according to the image correction parameter so as to obtain a calculation relation of absolute positions of pixel points of the target image;
and the intermediate parameter calculation module calculates the calculation relation of the relative positions of the pixel points of the target image and the calculation relation of the pixel values of the pixel points of the target image according to the calculation relation of the absolute positions of the pixel points of the target image, and provides the calculation relations to the remapping module.
14. The real-time two-way video processing method according to claim 9, wherein the remapping module sequentially reads the required pixel values of the pixel points of the source image from the corresponding positions in the on-chip memory according to the pixel point output order of the target image and the calculation relationship of the relative positions of the pixel points of the target image, and performs interpolation operation on the pixel values of the pixel points of the target image according to the calculation relationship of the pixel values of the pixel points of the target image, so as to calculate the pixel values of the pixel points of the target image of all the source images without overlapping regions one by one, and performs superposition calculation on the overlapping regions after interpolation operation when all the source images have overlapping regions, and obtaining pixel values of pixel points of the target image one by one to obtain internal image data of the target pixel, thereby forming the target image.
15. The real-time two-way video processing method of claim 7, wherein there is an image overlap region between source images of at least two parallel video input source inputs.
CN202210108244.6A 2022-01-28 2022-01-28 Real-time two-way video processing system and method based on programmable chip Active CN114449245B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210108244.6A CN114449245B (en) 2022-01-28 2022-01-28 Real-time two-way video processing system and method based on programmable chip

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210108244.6A CN114449245B (en) 2022-01-28 2022-01-28 Real-time two-way video processing system and method based on programmable chip

Publications (2)

Publication Number Publication Date
CN114449245A true CN114449245A (en) 2022-05-06
CN114449245B CN114449245B (en) 2024-04-05

Family

ID=81372088

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210108244.6A Active CN114449245B (en) 2022-01-28 2022-01-28 Real-time two-way video processing system and method based on programmable chip

Country Status (1)

Country Link
CN (1) CN114449245B (en)

Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140010479A1 (en) * 2012-07-09 2014-01-09 Samsung Electro-Mechanics Co., Ltd. Bilinear interpolation circuit for image and method thereof
CN105635602A (en) * 2015-12-31 2016-06-01 天津大学 System for mosaicing videos by adopting brightness and color cast between two videos and adjustment method thereof
CN107948544A (en) * 2017-11-28 2018-04-20 长沙全度影像科技有限公司 A kind of multi-channel video splicing system and method based on FPGA
CN108055472A (en) * 2017-12-21 2018-05-18 长沙全度影像科技有限公司 A kind of real time panoramic live broadcast system and method
CN109961401A (en) * 2017-12-14 2019-07-02 西安全志科技有限公司 A kind of method for correcting image and storage medium of binocular camera
CN110246081A (en) * 2018-11-07 2019-09-17 浙江大华技术股份有限公司 A kind of image split-joint method, device and readable storage medium storing program for executing
US20200228796A1 (en) * 2019-01-16 2020-07-16 Qualcomm Incorporated Local illumination compensation in video coding
CN112399095A (en) * 2019-08-15 2021-02-23 西安诺瓦星云科技股份有限公司 Video processing method, device and system
CN112511896A (en) * 2020-11-05 2021-03-16 浙江大华技术股份有限公司 Video rendering method and device
CN112541972A (en) * 2019-09-23 2021-03-23 华为技术有限公司 Viewpoint image processing method and related equipment
CN113421184A (en) * 2021-06-21 2021-09-21 上海爱观视觉科技有限公司 Real-time image processing method based on FPGA

Patent Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140010479A1 (en) * 2012-07-09 2014-01-09 Samsung Electro-Mechanics Co., Ltd. Bilinear interpolation circuit for image and method thereof
CN105635602A (en) * 2015-12-31 2016-06-01 天津大学 System for mosaicing videos by adopting brightness and color cast between two videos and adjustment method thereof
CN107948544A (en) * 2017-11-28 2018-04-20 长沙全度影像科技有限公司 A kind of multi-channel video splicing system and method based on FPGA
CN109961401A (en) * 2017-12-14 2019-07-02 西安全志科技有限公司 A kind of method for correcting image and storage medium of binocular camera
CN108055472A (en) * 2017-12-21 2018-05-18 长沙全度影像科技有限公司 A kind of real time panoramic live broadcast system and method
CN110246081A (en) * 2018-11-07 2019-09-17 浙江大华技术股份有限公司 A kind of image split-joint method, device and readable storage medium storing program for executing
US20200228796A1 (en) * 2019-01-16 2020-07-16 Qualcomm Incorporated Local illumination compensation in video coding
CN112399095A (en) * 2019-08-15 2021-02-23 西安诺瓦星云科技股份有限公司 Video processing method, device and system
CN112541972A (en) * 2019-09-23 2021-03-23 华为技术有限公司 Viewpoint image processing method and related equipment
CN112511896A (en) * 2020-11-05 2021-03-16 浙江大华技术股份有限公司 Video rendering method and device
CN113421184A (en) * 2021-06-21 2021-09-21 上海爱观视觉科技有限公司 Real-time image processing method based on FPGA

Also Published As

Publication number Publication date
CN114449245B (en) 2024-04-05

Similar Documents

Publication Publication Date Title
US8049786B2 (en) Method for improving image stitching accuracy with lens distortion correction and device for implementing the same
CN101064780B (en) Method and apparatus for improving image joint accuracy using lens distortion correction
CN104506826B (en) A kind of real-time splicing apparatus of fixed point orientation video without effective overlapping region
KR101915729B1 (en) Apparatus and Method for Generating 360 degree omni-directional view
US8326023B2 (en) Photographing field angle calculation apparatus
US20130235149A1 (en) Image capturing apparatus
WO1997015150A1 (en) Method and device for forming three-dimensional image
CN109996013B (en) A kind of low delay video-splicing methods, devices and systems
CN101489053B (en) Video image rotating system
US20070252905A1 (en) Image processing apparatus
JP5735227B2 (en) Image conversion apparatus and image conversion system
KR20180106549A (en) 360 degree panorama multimedia contents processing apparatus and control method thereof
CN102325261A (en) Synchronization method for eliminating jitter of inter-video video data of stereo video collecting and synthetizing system
KR20180056540A (en) Frame grabber, image processing system including the same, and image processing method using the frame grabber
CN114449245A (en) Real-time two-way video processing system and method based on programmable chip
CN102625086B (en) DDR2 (Double Data Rate 2) storage method and system for high-definition digital matrix
US20130100246A1 (en) Apparatus for synchronizing stereocamera, stereocamera, and method of synchronizing stereocamera
RU2497195C2 (en) Processor of signals, method of signal processing, display device and software product
US10701286B2 (en) Image processing device, image processing system, and non-transitory storage medium
US20210097660A1 (en) Multi-eye camera system, multi-eye photographing camera head, image processing device, multi-eye photographing program and multi-eye photographing method
JPH08251626A (en) Zooming position control device for stereoscopic television camera
JP2018005778A (en) Image display device, image display system, and image display method
JP5297899B2 (en) Stereoscopic image adjustment device, stereoscopic image adjustment system, and program
JP2001148806A (en) Video image synthesis arithmetic processing unit, its method and system
JP2017085207A (en) Stereoscopic video device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant