CN114449245B - Real-time two-way video processing system and method based on programmable chip - Google Patents

Real-time two-way video processing system and method based on programmable chip Download PDF

Info

Publication number
CN114449245B
CN114449245B CN202210108244.6A CN202210108244A CN114449245B CN 114449245 B CN114449245 B CN 114449245B CN 202210108244 A CN202210108244 A CN 202210108244A CN 114449245 B CN114449245 B CN 114449245B
Authority
CN
China
Prior art keywords
image
pixel
target image
module
chip memory
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202210108244.6A
Other languages
Chinese (zh)
Other versions
CN114449245A (en
Inventor
于非
赵征
高岩
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Tong Guan Intelligent Technology Co ltd
Original Assignee
Shanghai Tong Guan Intelligent Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Tong Guan Intelligent Technology Co ltd filed Critical Shanghai Tong Guan Intelligent Technology Co ltd
Priority to CN202210108244.6A priority Critical patent/CN114449245B/en
Publication of CN114449245A publication Critical patent/CN114449245A/en
Application granted granted Critical
Publication of CN114449245B publication Critical patent/CN114449245B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F15/00—Digital computers in general; Data processing equipment in general
    • G06F15/76—Architectures of general purpose stored program computers
    • G06F15/78—Architectures of general purpose stored program computers comprising a single central processing unit
    • G06F15/7807—System on chip, i.e. computer system on a single chip; System in package, i.e. computer system on one or more chips in a single package
    • G06F15/7825—Globally asynchronous, locally synchronous, e.g. network on chip
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/139—Format conversion, e.g. of frame-rate or size
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computing Systems (AREA)
  • Microelectronics & Electronic Packaging (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Image Processing (AREA)

Abstract

The invention provides a real-time two-way video processing system based on a programmable chip and a method thereof, wherein the system comprises an off-chip memory DDR, an intermediate parameter calculation module and a remapping module, wherein the intermediate parameter calculation module is used for acquiring the calculation relation of the absolute position and the relative position of a pixel point of a target image and the calculation relation of the pixel value of the pixel point of the target image according to an image correction parameter; the remapping module is used for sequentially reading out the pixel values currently required by the pixel points of the source image from the corresponding positions in the off-chip memory DDR, and calculating the pixel values of the pixel points of the target image according to the calculation relation of the pixel values of the pixel points of the target image so as to correct the errors of the two paths of source images, eliminate the errors between the two paths of original images, and the time delay is millisecond level, so that the safe high-quality image can be directly obtained for direct watching, a front motor mechanical correction link or a subsequent error processing link is omitted, the system is simplified, and the cost is reduced.

Description

Real-time two-way video processing system and method based on programmable chip
Technical Field
The invention relates to the technical field of double-path image acquisition and processing, in particular to a real-time double-path image processing system and method based on a programmable chip.
Background
The double-path image acquisition technology is widely applied to the fields of three-dimensional image acquisition, VR image acquisition and the like. The images acquired during the two-way image acquisition need to be subjected to image processing so as to eliminate errors among the images during the image acquisition.
In the process of three-dimensional image acquisition, two cameras are used for acquiring images through two-way image acquisition, and in order to form different degrees of open angles, the two cameras are mutually perpendicular or parallel, but due to mechanical reasons such as position deviation and optical reasons between the two cameras, errors such as translation, rotation, scaling and the like of the images often exist between the two images. In order to eliminate the error between two images, currently, two cameras for three-dimensional image acquisition can be calibrated in advance through manual adjustment or automatic adjustment, and the two cameras are manually adjusted on site to eliminate the error between the images to the greatest extent. The whole process is time-consuming and has high requirements on the level of on-site staff. And when the scene changes, the mode is required to be repeated for readjustment, the whole process is time-consuming and labor-consuming, and the requirement of the dynamic scene cannot be met. The automatic adjustment can be performed by calculating the error amount between the two images through an algorithm and then automatically adjusting according to the error amount. Automatic adjustment generally comprises two methods, namely electric adjustment and later-stage software adjustment. During electric adjustment, motors are used for respectively controlling and adjusting the physical positions of the two cameras so as to further correct errors between the patterns. The electric adjustment has the advantage of outputting the original image, and has the disadvantage of requiring an additional mechanical structure, which increases the cost and volume of the apparatus. And when the later software is adjusted, processing video files acquired by the two cameras, and performing operations such as rotation, translation, cutting, scaling and the like on the images. The post software adjustment has the advantages that no additional physical adjustment part is needed, and the disadvantage that video file processing is needed after shooting is completed, is not suitable for occasions such as live broadcasting in real time, and can have certain influence on image quality and visual range.
In VR image acquisition field, the double-fisheye scheme that uses at present, its cost is lower and the realization is simple. However, due to mechanical installation reasons, each of the front and rear cameras during two-way image acquisition may not reach a perfect 180 degrees, so that errors may exist, camera lenses may not reach a manufacturer nominal value, and this may cause cracking of a joint of a panorama spliced by pictures shot by the two cameras, and discomfort occurs during viewing. In order to solve the above problems, at present, a two-way fisheye image is generally output directly, and then an image processing correction is performed by a computer, but such processing not only increases the complexity of the system, but also cannot meet the requirement of real-time.
Disclosure of Invention
The invention aims to provide a real-time double-path video processing system and a method based on a programmable chip, which can eliminate errors between acquired images after the acquisition of double-path images and can output real-time images.
In order to achieve the above objective, the present invention provides a real-time two-way video processing system based on a programmable chip, which includes an off-chip memory DDR, an intermediate parameter calculation module, and a remapping module, where the off-chip memory DDR is used to store source images input by at least two parallel video input sources; the intermediate parameter calculation module is used for acquiring the calculation relation of the absolute positions of the pixel points of the target image, the calculation relation of the relative positions of the pixel points of the target image and the calculation relation of the pixel values of the pixel points of the target image according to the image correction parameters; the remapping module is used for sequentially reading out the pixel value currently required by the pixel point of the source image from the corresponding position in the off-chip memory DDR according to the calculation relation of the absolute position of the pixel point of the target image and the calculation relation of the relative position of the pixel point of the target image, and calculating the pixel value of the pixel point of the target image one by one according to the calculation relation of the pixel value currently required by the pixel point of the source image.
Optionally, the system further comprises a synchronous detection module, at least two input format conversion modules and an output format conversion module, wherein the synchronous detection module is used for receiving source images provided by at least two parallel video input sources and detecting the quality of all the source images; each input format conversion module is used for receiving a source image provided by one video input source, carrying out format conversion on image data of the source image, and storing the image data of the source image after format conversion into the off-chip memory DDR; the output format conversion module is used for converting the format of the target image calculated by the remapping module and outputting the target image after format conversion.
Further, the system also comprises an image writing control module, an image reading control module and an on-chip memory, wherein the image writing control module is used for writing all the source images subjected to format conversion by the input format conversion module into the on-chip memory DDR in sequence; the image reading control module is used for reading out the cache data containing the pixel values needed by the pixel points of the target image, which are needed to be read out from the off-chip memory DDR at one time, according to the calculation relation of the absolute positions of the pixel points of the target image provided by the intermediate parameter calculation module, and storing the cache data containing the pixel values needed by the pixel points of the target image into the on-chip memory for being used by the remapping module.
Further, the image correction device also comprises a parameter selection module, wherein the parameter selection module is used for selecting required image correction parameters and supplying the required image correction parameters to the intermediate parameter calculation module.
Further, the device also comprises an image sequence reading control module and an image correction parameter calculation module, wherein the image sequence reading control module is used for reading image data from the off-chip memory DDR, the image correction parameter calculation module is used for acquiring a first image correction parameter according to the image data read by the image sequence reading control module, and the parameter selection module is used for selecting a required image correction parameter from second image correction parameters and/or the first image correction parameters which are input from the outside and supplying the required image correction parameter to the intermediate parameter calculation module for use.
Optionally, the image writing control module comprises a pixel fusion module and an image longitude and latitude unfolding module,
when the real-time two-way video processing system is oriented to processing when three-dimensional images are acquired, the image writing control module performs pixel fusion on image data of all source images through the pixel fusion module, and stores the image data subjected to the pixel fusion into the off-chip memory DDR;
When the real-time two-way video processing system is oriented to processing when VR images are acquired, the image writing control module expands the image data of all the source images through the image longitude and latitude expansion module according to externally input field angle parameters and the yaw angle of each source image, and stores the expanded image data into the off-chip memory DDR.
On the other hand, the invention provides a real-time two-way video processing method based on a programmable chip, which comprises the following steps of:
the off-chip memory DDR stores at least two parallel source images input by the video input source;
the intermediate parameter calculation module obtains the calculation relation of the absolute positions of the pixel points of the target image, the calculation relation of the relative positions of the pixel points of the target image and the calculation relation of the pixel values of the pixel points of the target image according to the image correction parameters;
and the remapping module reads out the pixel value currently required by the pixel point of the source image from the corresponding position in the off-chip memory DDR according to the calculation relation of the absolute position of the pixel point of the target image and the calculation relation of the relative position of the pixel point of the target image, and calculates the pixel value of the pixel point of the target image one by one according to the calculation relation of the pixel value of the pixel point of the target image for the pixel value currently required by the pixel point of the source image so as to obtain the internal image data of the target pixel, thereby forming the target image.
Optionally, the processing flow for three-dimensional image acquisition specifically includes the following steps:
the synchronization detection module receives source images input by at least two parallel video input sources and detects the synchronicity of all the source images;
the input format conversion module receives a source image input by the video input source and performs format conversion on the source image;
the image writing control module performs pixel fusion on all source images after format conversion to obtain fused image data, and stores the fused image data into an off-chip memory DDR;
the intermediate parameter calculation module calculates the calculation relation of the absolute positions of the pixel points of the target image, the calculation relation of the relative positions of the pixel points of the target image and the calculation relation of the pixel values of the pixel points of the target image according to the image correction parameters;
the image reading control module reads out the cache data which is required to be read out from the off-chip memory DDR and contains the pixel values required by the pixel points of the target image at one time according to the calculation relation of the absolute positions of the pixel points of the target image, and stores the cache data into the on-chip memory;
the remapping module calculates the pixel value of the pixel point of the target image according to the pixel point output sequence of the target image, the calculation relation of the relative position of the pixel point of the target image and the calculation relation of the pixel value of the pixel point of the target image so as to obtain the internal image data of the target pixel, thereby forming the target image; and
The output format conversion module converts the format of the target image and outputs the target image after format conversion.
Optionally, the processing flow for VR image acquisition specifically includes the following steps:
the synchronization detection module receives source images input by at least two parallel video input sources and detects the synchronicity of all the source images;
the input format conversion module receives a source image input by the video input source and performs format conversion on the source image;
the image writing control module expands the longitude and latitude of all the source images after format conversion according to externally input field angle parameters and the yaw angle of each source image, and then stores the expanded source images into an off-chip memory DDR;
the intermediate parameter calculation module calculates the calculation relation of the absolute positions of the pixel points of the target image, the calculation relation of the relative positions of the pixel points of the target image and the calculation relation of the pixel values of the pixel points of the target image according to the image correction parameters;
the image reading control module reads out the cache data which is required to be read out from the off-chip memory DDR and contains the pixel values required by the pixel points of the target image at one time according to the calculation relation of the absolute positions of the pixel points of the target image, and stores the cache data into the on-chip memory;
The remapping module reads out pixel values required by the pixel points of the source image from corresponding positions in the on-chip memory in sequence according to the calculation relation of the relative positions of the pixel points of the target image, and calculates the pixel values of the pixel points of the target image one by one according to the calculation relation of the pixel values of the pixel points of the target image so as to obtain the internal image data of the target pixel, thereby forming the target image; and
the output format conversion module converts the format of the target image and outputs the target image after format conversion.
Further, the pixel fusion module of the image writing control module fuses the image data of all the source images into left and right half side-by-side image data, and stores the fused left and right half side-by-side image data into the off-chip memory according to a 3D format.
Further, the calculating relationship of the absolute position of the pixel point of the target image, the calculating relationship of the relative position of the pixel point of the target image and the calculating relationship of the pixel value of the pixel point of the target image according to the image correction parameters by the intermediate parameter calculating module comprises:
The parameter selection module selects required image correction parameters;
the intermediate parameter calculation module establishes an affine matrix according to the image correction parameters provided by the parameter selection module, and generates a remapping relation to acquire the calculation relation of the absolute positions of the pixel points of the target image;
according to the condition of data reading and writing in the on-chip memory, the intermediate parameter calculation module calculates the calculation relation of the relative positions of the pixel points of the target image according to the calculation relation of the absolute positions of the pixel points of the target image, and simultaneously calculates the calculation relation of the pixel values of the pixel points of the target image according to the absolute positions of the pixel points of the current target image and the space alternation condition of the on-chip memory.
Further, the remapping module reads the pixel values required by the pixel points of the source image from the corresponding positions in the on-chip memory in sequence according to the pixel point output sequence of the target image and the calculation relation of the relative positions of the pixel points of the target image, and also according to the address of the pixel points of the current target pixel provided by the on-chip memory in the on-chip memory, and carries out interpolation operation on the pixel values of the pixel points of the target image according to the calculation relation of the pixel values of the pixel points of the target image, so as to obtain the pixel values of the pixel points of the target image one by one, and obtain the internal image data of the target pixel, thereby forming the target image.
Further, the calculating relationship of the absolute position of the pixel point of the target image, the calculating relationship of the relative position of the pixel point of the target image and the calculating relationship of the pixel value of the pixel point of the target image according to the image correction parameters by the intermediate parameter calculating module comprises:
the parameter selection module selects required image correction parameters;
the intermediate parameter calculation module performs transmission transformation according to the image correction parameters to generate a remapping relation so as to obtain the calculation relation of the absolute positions of the pixel points of the target image;
and the intermediate parameter calculation module calculates the calculation relation of the relative positions of the pixel points of the target image and the calculation relation of the pixel values of the pixel points of the target image according to the calculation relation of the absolute positions of the pixel points of the target image, and provides the calculated relation of the relative positions of the pixel points of the target image and the calculation relation of the pixel values of the pixel points of the target image to the remapping module.
Further, the remapping module reads the required pixel value of the pixel of the source image from the corresponding position in the on-chip memory according to the pixel output sequence of the target image and the calculation relation of the relative positions of the pixels of the target image, and further according to the address of the pixel of the current target pixel provided by the on-chip memory in the on-chip memory, sequentially reads the required pixel value of the pixel of the source image from the corresponding position in the on-chip memory, and carries out interpolation operation on the pixel value of the pixel of the target image according to the calculation relation of the pixel value of the pixel of the target image, so as to calculate the pixel values of the pixels of the target image one by one, wherein all the source images have no overlapping area, and when all the source images have overlapping areas, carries out superposition calculation on the overlapping areas one by one, so as to obtain the pixel value of the pixel of the target image, and internal image data of the target image is obtained, so that the target image is formed.
Optionally, there is an image overlap region between source images input by at least two parallel video input sources.
Compared with the prior art, the invention has at least the following beneficial effects:
the invention provides a real-time two-way video processing system and a method thereof based on a programmable chip, wherein the real-time two-way video processing system comprises an off-chip memory DDR, an intermediate parameter calculation module and a remapping module, wherein the off-chip memory DDR is used for storing source images input by at least two parallel video input sources; the intermediate parameter calculation module is used for acquiring the calculation relation of the absolute positions of the pixel points of the target image, the calculation relation of the relative positions of the pixel points of the target image and the calculation relation of the pixel values of the pixel points of the target image according to the image correction parameters; the remapping module is used for sequentially reading out the pixel value currently required by the pixel point of the source image from the corresponding position in the off-chip memory DDR according to the calculation relation of the absolute position of the pixel point of the target image and the calculation relation of the relative position of the pixel point of the target image, calculating the pixel value of the pixel point of the target image one by one according to the pixel value currently required by the pixel point of the source image through the calculation relation of the pixel value of the pixel point of the target image, correcting the error of the two paths of source images, eliminating the error existing between the two paths of original images, and the time delay is in the millisecond level, so that the time delay can be ignored, and the safe and high-quality image can be directly obtained for direct viewing, and a front motor mechanical correction link or a subsequent error processing link can be omitted, so that the system is further simplified, and the cost is reduced.
Drawings
Fig. 1 is a schematic structural diagram of a real-time two-way video processing system based on a programmable chip according to a first embodiment of the present invention;
fig. 2 is a flow chart of a real-time two-way video processing method based on a programmable chip according to a first embodiment of the invention;
fig. 3 is a schematic structural diagram of a real-time two-way video processing system based on a programmable chip according to a second embodiment of the present invention;
fig. 4 is a flow chart of a real-time two-way video processing method based on a programmable chip according to a second embodiment of the invention.
Detailed Description
The invention provides a real-time two-way video processing system based on a programmable chip, which comprises an off-chip memory DDR, an intermediate parameter calculation module and a remapping module, wherein the off-chip memory DDR is used for storing source images input by at least two parallel video input sources; the intermediate parameter calculation module is used for acquiring the calculation relation of the absolute positions of the pixel points of the target image, the calculation relation of the relative positions of the pixel points of the target image and the calculation relation of the pixel values of the pixel points of the target image according to the image correction parameters; the remapping module is used for sequentially reading out the pixel value currently required by the pixel point of the source image from the corresponding position in the off-chip memory DDR according to the calculation relation of the absolute position of the pixel point of the target image and the calculation relation of the relative position of the pixel point of the target image, and calculating the pixel value of the pixel point of the target image one by one according to the calculation relation of the pixel value currently required by the pixel point of the source image.
The real-time two-way video processing method of the programmable chip comprises the following steps:
the off-chip memory DDR stores at least two parallel source images input by the video input source;
the intermediate parameter calculation module obtains the calculation relation of the absolute positions of the pixel points of the target image, the calculation relation of the relative positions of the pixel points of the target image and the calculation relation of the pixel values of the pixel points of the target image according to the image correction parameters;
and the remapping module reads out the pixel value currently required by the pixel point of the source image from the corresponding position in the off-chip memory DDR according to the calculation relation of the absolute position of the pixel point of the target image and the calculation relation of the relative position of the pixel point of the target image, and calculates the pixel value of the pixel point of the target image one by one according to the calculation relation of the pixel value of the pixel point of the target image for the pixel value currently required by the pixel point of the source image so as to obtain the internal image data of the target pixel, thereby forming the target image.
The following describes the real-time two-way video processing system and method based on the programmable chip in further detail. The present invention will be described in more detail below with reference to the attached drawings, in which preferred embodiments of the present invention are shown, it being understood that one skilled in the art can modify the present invention described herein while still achieving the advantageous effects of the present invention. Accordingly, the following description is to be construed as broadly known to those skilled in the art and not as limiting the invention.
In the interest of clarity, not all features of an actual implementation are described. In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail. It should be appreciated that in the development of any such actual embodiment, numerous implementation details must be made to achieve the developer's specific goals, such as compliance with system-related or business-related constraints, which will vary from one implementation to another. In addition, it will be appreciated that such a development effort might be complex and time-consuming, but would nevertheless be a routine undertaking for those of ordinary skill in the art.
In order to make the objects and features of the present invention more comprehensible, embodiments accompanied with figures are described in detail below. It is noted that the drawings are in a very simplified form and utilize non-precise ratios, and are intended to facilitate a convenient, clear, description of the embodiments of the invention.
Fig. 1 is a schematic structural diagram of a real-time dual-channel video processing system based on a programmable chip according to the present embodiment. As shown in fig. 1, the present embodiment provides a real-time two-way video processing system based on a programmable chip. The real-time two-way video processing system can be applied to the fields of 3D shooting, 3D microscope, heterologous image processing, three-dimensional reconstruction, large-pixel image stitching, VR stitching and the like. The real-time two-way video processing system comprises a synchronous detection module 10, a format conversion module, an image writing control module 41, an off-chip memory DDR50, an image reading control module 42, an intermediate parameter calculation module 70, an on-chip memory 80 and a remapping module 30.
The synchronization detection module 10 is configured to receive source images provided by at least two parallel video input sources, detect the quality of all the source images, determine whether all the source images are synchronized, whether the video is interrupted, and the like, and perform frame rate detection, and provide signal prompts for abnormal results of synchronization detection of all the source images, so as to facilitate processing by field operators. In this embodiment, the real-time two-way video processing system is oriented to processing during three-dimensional image acquisition, and the synchronous detection module 10 receives source images provided by two parallel video input sources, i.e., a first source image and a second source image, where the two video input sources are obtained by, for example, SDI camera shooting, and an image overlapping area is formed between the two source images, i.e., a shooting overlapping area is formed when the two source images are shot by the SDI camera. The format of the source image is, for example, an SDI format, and pixels of the two source images are the same, for example, 1920×1080.
The format conversion modules include at least two input format conversion modules 21 and 22, the number of the input format conversion modules is the same as that of the video input sources, and each input format conversion module is configured to receive a source image provided by one video input source, and perform format conversion on image data of the source image to obtain image data of at least two source images in a unified format, for example, the clocks of the source images are unified into 300M, so as to realize the unification of clock domains in the system, and the clock frequency after unification is greater than that of the source images provided by the video input source, which is beneficial to pixel fusion when the subsequent images are written into the control module 41; the video stream format is uniformly converted into axi-stream, and the color space is uniformly converted into RGB coding format, so that the image data processing in the real-time two-way video processing system is facilitated.
A FIFO is disposed between the image writing control module 41 and the format conversion module, and the FIFO is configured to buffer the image data of all source images after format conversion by the format conversion module, and send pixel data in the image data that needs to be processed by the image writing control module 41 to the image writing control module 41. In the present embodiment, FIFOs may be provided between other modules, for example, between the intermediate parameter calculation module and the image reading control module, and between the intermediate parameter calculation module and the remapping module.
The image writing control module 41 is configured to sequentially write all the source images after format conversion (i.e. source images after unified format) into the off-chip memory DDR50, and sequentially process each pixel point of all the source images and rearrange the storage locations of each pixel point according to the actual processing requirements if necessary. In this embodiment, the image writing control module 41 includes a pixel fusion module 411, and the pixel fusion module 411 may perform pixel fusion on the image data of at least two source images to obtain fused image data.
Specifically, the image writing control module 41 is configured to receive image data of all source images after format conversion, and the pixel fusion module 411 fuses the image data of two paths of source images according to a side-by-side format. Taking left and right half side-by-side (left and right combined and then transversely halved), taking a color space as RGB as an example, firstly processing one row of pixel data of a first source image, namely grouping one row of pixel points of the first source image, dividing each two adjacent pixel points into one group, taking an average value of pixel values of the two pixel points as a pixel value of each group of pixel points to obtain a half row of pixel values, and sequentially storing the half row of processed pixel data into an off-chip memory DDR 50; and then processing one row of pixel data of the second source image (the row corresponding to the row just processed by the first source image), namely grouping one row of pixel points of the second source image, dividing each two adjacent pixel points into a group, taking the average value of the pixel values of the two pixel points of each group of pixel points to obtain a half row of pixel values, sequentially storing the processed half row of pixel data into an off-chip memory DDR50 to obtain one row of pixel data after pixel fusion, and carrying out pixel fusion on the pixel data of all rows of the two source images according to the method to obtain one frame of half side-by-side video (namely fusing the image data of the two source images to obtain one fused image data).
In order to facilitate reading and to obtain the address of the pixel point of the source image from the row and column sequence number more easily in the off-chip memory DDR50, the internal space of the off-chip memory DDR50 is distributed in an array, and each row, each column and each frame in the off-chip memory DDR50 occupies the internal space of the off-chip memory DDR50 in units of a power number of 2, which is to allow a certain redundancy space to exist in the off-chip memory DDR 50.
Taking the example of storing image data after one frame of pixel fusion with pixels 1920×1080 and pixel size 4Byte in the off-chip memory DDR 50: the pixel value of each pixel point of one frame of image data occupies 4 pixel internal Byte addresses, so that [ bit1:bit0] in each address in the off-chip memory DDR50 represents the pixel internal Byte address; each row of 1920 pixels occupies 2048 pixels in the internal space of the off-chip memory DDR50, so that [ bit12 ] in each address in the off-chip memory DDR 50: bit2 represents the column number of the pixel point; each frame of image data occupies the space required by 2048 rows of pixels in the internal space of the off-chip memory DDR50, so that [ bit23:bit13] in each address in the off-chip memory DDR50 represents the row number where the pixel is located. According to the above address space planning, each pixel data of a frame of image data transmitted from the pixel fusion module 411 is written into the off-chip memory DDR50 in sequence.
The intermediate parameter calculation module 70 is configured to calculate a calculation relationship of an absolute position of a pixel of the target image (i.e., a position of a pixel of the source image corresponding to each pixel of the target image), a calculation relationship of a relative position of a pixel of the target image, and a calculation relationship of a pixel value of a pixel of the target image according to the image correction parameter, and provide the position of the pixel of the source image corresponding to each pixel of the target image to the image reading control module 42, so that the image reading control module 42 reads out cache data including a pixel value required for the pixel of the target image from the off-chip memory DDR50, and sends the calculation relationship of the relative position of the pixel of the target image and the calculation relationship of the pixel value of the pixel of the target image to the remapping module 30, so as to obtain the image data of the target image. The absolute position of the pixel point of the target image refers to the position of the pixel point of the target image read from the off-chip memory DDR50, that is, the position of the coordinate of the pixel point of the target image in the coordinate of the source image. The relative position of the pixel point of the target image refers to the address where the pixel data read out from the off-chip memory DDR50 is written into the RAM, and the address of the pixel point of the target image read out from the RAM.
The on-chip memory 80 is, for example, RAM, which serves as a buffer space that can hold several lines of pixel data for direct random reading by the remapping module 30. The buffer space is used for alternately replacing and storing pixel data in a first-in first-out mode, and a counter is arranged to record the number of lines of reading/writing of each frame of image data, and the reading/writing of the RAM is ensured to be neither empty nor full.
The image correction parameters may include a first image correction parameter obtained via internal calculation of the real-time two-way video processing system and/or a second image correction parameter obtained from external input of the real-time two-way video processing system. In order to select the source of the image correction parameters, the real-time two-way video processing system further includes a parameter selection module 63, and when the image correction parameters include a first image correction parameter and a second image correction parameter, the parameter selection module 63 preferentially selects the first image correction parameter.
The real-time two-way video processing system further comprises an image sequence reading control module 61 and an image correction parameter calculation module 62, and can provide a first image correction parameter to the intermediate parameter calculation module 70 and output the first image correction parameter to prompt a user. The image sequence reading control module 61 is configured to read the image data of each frame after pixel fusion, and is configured to be used by the image correction parameter calculation module 62. The image correction parameter calculation module 62 is configured to obtain a first image correction parameter by calculating errors such as rotation, translation, scaling, etc. between the first source image and the second source image according to the image data provided by the image sequential reading control module, and provide the first image correction parameter to the intermediate parameter calculation module 70 through the parameter selection module 63, and output the first image correction parameter to prompt a user. In the present embodiment, FIFOs are provided between the parameter selection module 63 and the intermediate parameter calculation module and between the image sequential reading control module 61 and the image correction parameter calculation module 62.
The image reading control module reads out the buffered data containing the pixel values of the pixel points of the target image, which need to be read out from the off-chip memory DDR50 at a time, and stores the buffered data containing the pixel values of the pixel points of the target image into the on-chip memory 80 for the remapping module 30.
The remapping module 30 is configured to, according to the output order of the pixels of the target image and the calculation relation of the relative positions of the pixels of the target image provided by the intermediate parameter calculating module 70, further according to the address of the pixels of the current target pixel provided by the on-chip memory 80 in the on-chip memory 80 and sequentially read out the pixel values required by the pixels of the source image from the corresponding positions in the on-chip memory 80, and obtain the pixel values of the pixels of the target image one by one through interpolation operation (specifically, for example, bilinear interpolation operation), so as to obtain the internal image data of the target pixel, thereby forming the target image.
The format conversion module further includes an output format conversion module 23, where the output format conversion module 23 is configured to convert the format of the target image into a format (such as axi-stream) required for output, and output the target image after the format conversion.
The embodiment is based on a Xilinx Ultrascale + platform and adopts a ZCU development board of Xilinx, wherein the programmable chip adopts XCZU7EV of the Xilinx; qpsi Flash uses MT25QU512 of Micron as program memory; the DDR50 off-chip memory adopts MT40A256M16GE-075E of micro; SI570 on the programmable chip provides the master clock signal.
Fig. 2 is a flow chart of a real-time two-way video processing method based on a programmable chip in the present embodiment. As shown in fig. 2, the present embodiment further provides a real-time two-way video processing method based on a programmable chip, which is oriented to a processing flow during three-dimensional image acquisition to eliminate errors between a first source image and a second source image, and because the processing flow during three-dimensional image acquisition is the same as the processing flow during large-pixel image stitching, the processing flow during three-dimensional image acquisition is taken as an example for explanation, and the processing flow during large-pixel image stitching is used for stitching the first source image and the second source image. The real-time two-way video processing method comprises the following steps:
step S11: the synchronization detection module 10 receives source images input by at least two parallel video input sources and detects the synchronicity of all the source images;
Step S12: the input format conversion module receives a source image input by the video input source and performs format conversion on the source image;
step S13: the image writing control module 41 performs pixel fusion on all the source images after format conversion to obtain fused image data, and stores the fused image data into the off-chip memory DDR 50;
step S14: the intermediate parameter calculation module 70 calculates a calculation relationship of the absolute positions of the pixels of the target image, a calculation relationship of the relative positions of the pixels of the target image, and a calculation relationship of the pixel values of the pixels of the target image according to the image correction parameters;
step S15: the image reading control module 42 reads out the cache data containing the pixel values required by the pixel points of the target image, which need to be read out from the off-chip memory DDR50 at a time, according to the calculation relation of the absolute positions of the pixel points of the target image, and stores the cache data in the on-chip memory 80;
step S16: the remapping module 30 calculates the pixel value of the pixel point of the target image according to the pixel point output sequence of the target image, the calculation relation of the relative position of the pixel point of the target image and the calculation relation of the pixel value of the pixel point of the target image, so as to obtain the internal image data of the target pixel, thereby forming the target image; and
Step S17: the output format conversion module 23 converts the format of the target image and outputs the format-converted target image.
In step S11, the synchronization detection module 10 determines whether source images provided by at least two parallel video input sources are synchronized, whether video is interrupted, and the like, and may also perform frame rate detection, and provide a prompt signal for an abnormality of the detection result, so as to facilitate processing by field operators. In this embodiment, the synchronization detection module 10 receives input source images provided by two parallel video input sources, i.e. a first source image and a second source image, and there is an image overlapping area between the first source image and the second source image.
In step S12, when the synchronization detection module 10 detects that all source images are synchronized, video is not interrupted, and frame rate detection is not abnormal, each of the input format conversion modules receives a source image input by the video input source and performs format conversion on the source image to unify the formats of all the source images. In this embodiment, the input format conversion module performs format unification on the clock domains, the video stream formats, the encoding formats, and the like of all source images, so as to facilitate internal calculation of the real-time two-way video processing system.
In step S13, the pixel fusion module 411 of the image writing control module 41 fuses the image data of all the source images into left and right half side-by-side image data, and stores the fused left and right half side-by-side image data in the off-chip memory DDR50 according to the 3D format.
The step S14 specifically includes:
first, the image correction parameters include a first image correction parameter and/or a second image correction parameter, when the image correction parameters include at least a first image correction parameter, the parameter selection module 63 preferentially selects the first image correction parameter, and selects the second image correction parameter when there is no first image correction parameter, and sends the selected image correction parameter to the intermediate parameter calculation module 70; the first image correction parameter is obtained by reading, by the image sequence reading control module 61, image data obtained by fusing complete image data of one frame from the off-chip memory DDR50, and supplying the image data to the image correction parameter calculation module 62, where the image correction parameter calculation module 62 calculates errors such as rotation, translation, and scaling between the first source image and the second source image, so as to obtain the first image correction parameter.
Then, the intermediate parameter calculation module 70 establishes an affine matrix according to the image correction parameters (i.e. the first image correction parameters or the second image correction parameters) provided by the parameter selection module 63, and generates a remapping relationship to obtain a calculation relationship of the absolute positions of the pixels of the target image. Specifically, the position of the pixel of the source image corresponding to each output pixel is calculated one by one according to the coordinate sequence of the pixel of the target image, that is, the coordinates (Xs, ys) of the pixel of each target image and the pixel of the source image corresponding to the pixel of each target image are calculated one by one from the coordinates (0, 0) of the pixel of the target image, and are provided to the image reading control module 42, that is, the position of the pixel of the source image corresponding to each pixel of the target image. The absolute position of the pixel point of the target image refers to the position of the pixel point of the target image read from the off-chip memory DDR50, that is, the coordinate of the pixel point of the target image in the source image.
Due to the limited space of the on-chip memory 80, the policy of the entire row read of the on-chip memory 80 makes the number of rows that the on-chip memory 80 can store limited and makes the data in the on-chip memory 80 follow first-in-first-out management. Therefore, next, according to the condition of data read-write in the on-chip memory 80, the intermediate parameter calculation module 70 calculates the calculation relation (i.e. the address which can be written currently) of the relative positions of the pixels of the target image according to the positions of the pixels of the source image corresponding to each pixel of each target image, and provides the calculated relation to the remapping module 30; meanwhile, according to the absolute position of the pixel point of the current target image and the space alternation condition of the on-chip memory 80, the calculation relation of the pixel value of the pixel point of the target image is calculated and provided for the remapping module 30. In this embodiment, for the case where the position of the pixel point of the target image does not overlap with the position of any pixel point in the source image, interpolation operation is required, and the pixel value relationship of the pixel point of the target image is that the pixel total weight of the pixel point of the target image is calculated according to the distance between the position of the pixel point of the target image and the pixel points of the 4 source images, and for the case where the position of the pixel point of the target image overlaps with the position of any pixel point in the source image, the pixel value of the pixel point of the source image is directly copied without interpolation operation and provided to the remapping module 30.
The step S16 specifically includes: the remapping module 30 reads the pixel values required by the pixel points of the source image from the corresponding positions in the on-chip memory 80 in turn according to the pixel point output sequence of the target image and the calculation relation of the relative positions of the pixel points of the target image, and the address of the pixel point of the current target pixel provided by the on-chip memory 80 in the on-chip memory 80, so as to find out 4 pixel values around the target pixel, and performs interpolation operation (specifically, for example, bilinear interpolation operation) on the pixel values of the pixel points of the target image according to the calculation relation of the pixel values of the pixel points of the target image, so as to obtain the pixel values of the pixel points of the target image one by one, so as to obtain the internal image data of the target pixel, thereby forming the target image.
Example two
Compared with the first embodiment, the difference between the present embodiment is that fig. 3 is a schematic structural diagram of the real-time dual-channel video processing system based on the programmable chip of the present embodiment. As shown in fig. 3, the real-time two-way video processing system based on the programmable chip is applied to the fields of VR splicing after VR image acquisition and the like. The image writing control module 41 of the real-time two-way video processing system further includes an image longitude and latitude expansion module for performing longitude and latitude expansion according to externally input field angle parameters and the yaw angle (e.g., the yaw angles of 0 ° and 180 °) of each source image. The image writing control module 41 is configured to receive all the source images after format conversion, and perform, by using the image longitude and latitude expansion module, writing operation after longitude and latitude expansion on the off-chip memory DDR50 according to an externally input view angle parameter and a yaw angle (for example, a yaw angle of 0 ° and 180 °) of each of the source images, respectively, where image data of all the source images are not subjected to pixel fusion by the pixel fusion module 411, but are stored in the off-chip memory DDR 50.
Fig. 4 is a flow chart of a real-time two-way video processing method based on a programmable chip in the present embodiment. As shown in fig. 4, the real-time two-way video processing method of the present embodiment is oriented to the processing flow during VR image acquisition. The real-time two-way video processing method comprises the following steps:
step S21: the synchronization detection module 10 receives source images input by at least two parallel video input sources and detects the synchronicity of all the source images;
step S22: the input format conversion module receives a source image input by the video input source and performs format conversion on the source image;
step S23: the image writing control module 41 expands all the source images after format conversion according to externally input field angle parameters and the yaw angle of each source image, and stores the expanded source images into the off-chip memory DDR 50;
step S24: the intermediate parameter calculation module 70 calculates a calculation relationship of the absolute positions of the pixels of the target image, a calculation relationship of the relative positions of the pixels of the target image, and a calculation relationship of the pixel values of the pixels of the target image according to the image correction parameters;
step S25: the image reading control module 42 reads out the cache data containing the pixel values required by the pixel points of the target image, which need to be read out from the off-chip memory DDR50 at a time, according to the calculation relation of the absolute positions of the pixel points of the target image, and stores the cache data in the on-chip memory 80;
Step S26: the remapping module 30 sequentially reads out the pixel values required by the pixels of the source image from the corresponding positions in the on-chip memory 80 according to the calculation relation of the relative positions of the pixels of the target image, and calculates the pixel values of the pixels of the target image one by one according to the calculation relation of the pixel values of the pixels of the target image, so as to obtain the internal image data of the target pixel, thereby forming the target image; and
step S27: the output format conversion module 23 converts the format of the target image and outputs the format-converted target image.
In step S23, the image writing control module 41 does not fuse the image data of the source images after format conversion, but directly stores the image data of each source image after longitude and latitude expansion according to the externally input view angle parameter and the yaw angle of each source image in the DDR.
Step S24 includes:
first, the image correction parameters include a first image correction parameter and/or a second image correction parameter, when the image correction parameters include at least a first image correction parameter, the parameter selection module 63 preferentially selects the first image correction parameter, and selects the second image correction parameter when there is no first image correction parameter, and sends the selected image correction parameter to the intermediate parameter calculation module 70; the first image correction parameter is two image data of the complete double fish eyes after the warp and weft expansion are read from the off-chip memory DDR50 by the image sequence reading control module 61, and are supplied to the image correction parameter calculation module 62, and the image correction parameter calculation module 62 extracts feature points and calculates relative positions of the overlapping portion of the first image after the warp and weft expansion and the second image after the warp and weft expansion, so as to obtain the first image correction parameter.
Then, the intermediate parameter calculation module 70 performs transmission transformation according to the image correction parameters to generate a remapping relationship so as to obtain a calculation relationship of the absolute positions of the pixels of the target image. Specifically, the intermediate parameter calculation module 70 performs transmission transformation on the panoramic image according to the image correction parameters to eliminate the position error of each pixel, thereby generating a remapping relationship to obtain the calculation relationship of the absolute positions of the pixels of the target image. The matrix used for the transmission transformation can be obtained through external input or through intra-system calculation.
Next, the intermediate parameter calculation module 70 calculates a calculation relationship of the relative positions of the pixels of the target image and a calculation relationship of the pixel values of the pixels of the target image according to the calculation relationship of the absolute positions of the pixels of the target image, and provides the calculated relationship to the remapping module 30.
In step S26, the remapping module 30 reads the required pixel values of the pixel points of the source image from the corresponding positions in the on-chip memory 80 in turn according to the pixel output sequence of the target image and the calculation relation of the relative positions of the pixel points of the target image, and the address of the pixel points of the current target pixel provided by the on-chip memory 80 in the on-chip memory 80, so as to find out 4 pixel values around the target pixel, and perform interpolation operation (specifically, bilinear interpolation operation) on the pixel values of the pixel points of the target image according to the calculation relation of the pixel values of the pixel points of the target image, so as to calculate the pixel values of the pixel points of the target image without overlapping areas of all the source images one by one, and when all the source images have overlapping areas, the overlapping areas need to perform overlapping calculation again after the interpolation operation, so as to obtain the pixel values of the pixel points of the target image one by one, so as to obtain the internal image data of the target image.
In summary, the invention provides a real-time dual-path video processing system based on a programmable chip and a method thereof, which can correct errors of dual-path source images, eliminate errors between two paths of original images, and the time delay is in millisecond level, so that the time delay can be ignored, thereby directly obtaining safe and high-quality images for direct viewing, further omitting a front motor mechanical correction link or a subsequent error processing link, further simplifying the system and reducing the cost.
It will be appreciated that although the invention has been described above in terms of preferred embodiments, the above embodiments are not intended to limit the invention. Many possible variations and modifications of the disclosed technology can be made by anyone skilled in the art without departing from the scope of the technology, or the technology can be modified to be equivalent. Therefore, any simple modification, equivalent variation and modification of the above embodiments according to the technical substance of the present invention still fall within the scope of the technical solution of the present invention.

Claims (13)

1. The real-time two-way video processing system based on the programmable chip is characterized by comprising an off-chip memory DDR, an intermediate parameter calculation module, a remapping module, a parameter selection module, an image sequence reading control module and an image correction parameter calculation module, wherein the off-chip memory DDR is used for storing source images input by at least two parallel video input sources; the intermediate parameter calculation module is used for acquiring the calculation relation of the absolute positions of the pixel points of the target image, the calculation relation of the relative positions of the pixel points of the target image and the calculation relation of the pixel values of the pixel points of the target image according to the image correction parameters; the remapping module is used for sequentially reading out the pixel value currently required by the pixel point of the source image from the corresponding position in the off-chip memory DDR according to the calculation relation of the absolute position of the pixel point of the target image and the calculation relation of the relative position of the pixel point of the target image, and calculating the pixel value of the pixel point of the target image one by one according to the calculation relation of the pixel value currently required by the pixel point of the source image;
the image sequence reading control module is used for reading image data from the off-chip memory DDR, the image correction parameter calculation module is used for obtaining a first image correction parameter according to the image data read by the image sequence reading control module, and the parameter selection module is used for selecting a required image correction parameter from second image correction parameters and/or the first image correction parameters which are input from the outside and supplying the required image correction parameter to the intermediate parameter calculation module for use.
2. The real-time two-way video processing system of claim 1, further comprising a synchronization detection module, at least two input format conversion modules, and an output format conversion module, wherein the synchronization detection module is configured to receive source images provided by at least two parallel video input sources and detect quality of all the source images; each input format conversion module is used for receiving a source image provided by one video input source, carrying out format conversion on image data of the source image, and storing the image data of the source image after format conversion into the off-chip memory DDR; the output format conversion module is used for converting the format of the target image calculated by the remapping module and outputting the target image after format conversion.
3. The real-time two-way video processing system according to claim 2, further comprising an image writing control module, an image reading control module and an on-chip memory, wherein the image writing control module is used for writing all the source images after format conversion by the input format conversion module into the on-chip memory DDR in sequence; the image reading control module is used for reading out the cache data containing the pixel values needed by the pixel points of the target image, which are needed to be read out from the off-chip memory DDR at one time, according to the calculation relation of the absolute positions of the pixel points of the target image provided by the intermediate parameter calculation module, and storing the cache data containing the pixel values needed by the pixel points of the target image into the on-chip memory for being used by the remapping module.
4. The real-time two-way video processing system as recited in any one of claims 1-3, wherein the image writing control module comprises a pixel fusion module and an image longitude and latitude expansion module,
when the real-time two-way video processing system is oriented to processing when three-dimensional images are acquired, the image writing control module performs pixel fusion on image data of all source images through the pixel fusion module, and stores the image data subjected to the pixel fusion into the off-chip memory DDR;
when the real-time two-way video processing system is oriented to processing when VR images are acquired, the image writing control module expands the image data of all the source images through the image longitude and latitude expansion module according to externally input field angle parameters and the yaw angle of each source image, and stores the expanded image data into the off-chip memory DDR.
5. A real-time two-way video processing method based on a programmable chip, comprising the real-time two-way video processing system according to any one of claims 1 to 4, characterized by comprising the steps of:
the off-chip memory DDR stores at least two parallel source images input by the video input source;
The image sequence reading control module reads image data from the off-chip memory DDR, the image correction parameter calculation module acquires a first image correction parameter according to the image data read by the image sequence reading control module, and the parameter selection module selects a required image correction parameter from second image correction parameters and/or the first image correction parameters which are input from the outside and supplies the required image correction parameter to the intermediate parameter calculation module;
the intermediate parameter calculation module obtains the calculation relation of the absolute positions of the pixel points of the target image, the calculation relation of the relative positions of the pixel points of the target image and the calculation relation of the pixel values of the pixel points of the target image according to the image correction parameters;
and the remapping module reads out the pixel value currently required by the pixel point of the source image from the corresponding position in the off-chip memory DDR according to the calculation relation of the absolute position of the pixel point of the target image and the calculation relation of the relative position of the pixel point of the target image, and calculates the pixel value of the pixel point of the target image one by one according to the calculation relation of the pixel value of the pixel point of the target image for the pixel value currently required by the pixel point of the source image so as to obtain the internal image data of the target pixel, thereby forming the target image.
6. The real-time two-way video processing method according to claim 5, which is a processing flow for three-dimensional image acquisition, and is characterized by comprising the following steps:
the synchronization detection module receives source images input by at least two parallel video input sources and detects the synchronicity of all the source images;
the input format conversion module receives a source image input by the video input source and performs format conversion on the source image;
the image writing control module performs pixel fusion on all source images after format conversion to obtain fused image data, and stores the fused image data into an off-chip memory DDR;
the intermediate parameter calculation module calculates the calculation relation of the absolute positions of the pixel points of the target image, the calculation relation of the relative positions of the pixel points of the target image and the calculation relation of the pixel values of the pixel points of the target image according to the image correction parameters;
the image reading control module reads out the cache data which is required to be read out from the off-chip memory DDR and contains the pixel values required by the pixel points of the target image at one time according to the calculation relation of the absolute positions of the pixel points of the target image, and stores the cache data into the on-chip memory;
The remapping module calculates the pixel value of the pixel point of the target image according to the pixel point output sequence of the target image, the calculation relation of the relative position of the pixel point of the target image and the calculation relation of the pixel value of the pixel point of the target image so as to obtain the internal image data of the target pixel, thereby forming the target image; and
the output format conversion module converts the format of the target image and outputs the target image after format conversion.
7. The real-time two-way video processing method according to claim 5, which is directed to a processing flow during VR image acquisition, and is characterized by comprising the following steps:
the synchronization detection module receives source images input by at least two parallel video input sources and detects the synchronicity of all the source images;
the input format conversion module receives a source image input by the video input source and performs format conversion on the source image;
the image writing control module expands the longitude and latitude of all the source images after format conversion according to externally input field angle parameters and the yaw angle of each source image respectively and stores the expanded source images into an off-chip memory DDR;
the intermediate parameter calculation module calculates the calculation relation of the absolute positions of the pixel points of the target image, the calculation relation of the relative positions of the pixel points of the target image and the calculation relation of the pixel values of the pixel points of the target image according to the image correction parameters;
The image reading control module reads out the cache data which is required to be read out from the off-chip memory DDR and contains the pixel values required by the pixel points of the target image at one time according to the calculation relation of the absolute positions of the pixel points of the target image, and stores the cache data into the on-chip memory;
the remapping module reads out pixel values required by the pixel points of the source image from corresponding positions in the on-chip memory in sequence according to the calculation relation of the relative positions of the pixel points of the target image, and calculates the pixel values of the pixel points of the target image one by one according to the calculation relation of the pixel values of the pixel points of the target image so as to obtain the internal image data of the target pixel, thereby forming the target image; and
the output format conversion module converts the format of the target image and outputs the target image after format conversion.
8. The method for real-time two-way video processing according to claim 6, wherein,
and the pixel fusion module of the image writing control module fuses the image data of all the source images into left and right half side-by-side image data, and stores the fused left and right half side-by-side image data into the off-chip memory according to a 3D format.
9. The method of real-time two-way video processing according to claim 6, wherein the calculating of the absolute position of the pixel of the target image, the calculating of the relative position of the pixel of the target image, and the calculating of the pixel value of the pixel of the target image according to the image correction parameters includes:
the parameter selection module selects required image correction parameters;
the intermediate parameter calculation module establishes an affine matrix according to the image correction parameters provided by the parameter selection module, and generates a remapping relation to acquire the calculation relation of the absolute positions of the pixel points of the target image;
according to the condition of data reading and writing in the on-chip memory, the intermediate parameter calculation module calculates the calculation relation of the relative positions of the pixel points of the target image according to the calculation relation of the absolute positions of the pixel points of the target image, and simultaneously calculates the calculation relation of the pixel values of the pixel points of the target image according to the absolute positions of the pixel points of the current target image and the space alternation condition of the on-chip memory.
10. The method according to claim 6, wherein the remapping module reads the pixel values required by the pixel points of the source image from the corresponding positions in the on-chip memory in turn according to the pixel point output sequence of the target image and the calculation relation of the relative positions of the pixel points of the target image, and further according to the address of the pixel point of the current target pixel provided by the on-chip memory in the on-chip memory, performs interpolation operation on the pixel values of the pixel points of the target image according to the calculation relation of the pixel values of the pixel points of the target image, and obtains the pixel values of the pixel points of the target image one by one to obtain the internal image data of the target pixel, thereby forming the target image.
11. The method of real-time two-way video processing according to claim 7, wherein the calculating of the absolute position of the pixel of the target image, the calculating of the relative position of the pixel of the target image, and the calculating of the pixel value of the pixel of the target image according to the image correction parameters includes:
the parameter selection module selects required image correction parameters;
the intermediate parameter calculation module performs transmission transformation according to the image correction parameters to generate a remapping relation so as to obtain the calculation relation of the absolute positions of the pixel points of the target image;
and the intermediate parameter calculation module calculates the calculation relation of the relative positions of the pixel points of the target image and the calculation relation of the pixel values of the pixel points of the target image according to the calculation relation of the absolute positions of the pixel points of the target image, and provides the calculated relation of the relative positions of the pixel points of the target image and the calculation relation of the pixel values of the pixel points of the target image to the remapping module.
12. The method according to claim 7, wherein the remapping module reads the required pixel value of the pixel of the source image from the corresponding position in the on-chip memory in turn according to the pixel output sequence of the target image and the calculation relation of the relative positions of the pixels of the target image, and further according to the address of the pixel of the current target pixel in the on-chip memory provided by the on-chip memory, and according to the calculation relation of the pixel value of the pixel of the target image, performs interpolation operation on the pixel value of the pixel of the target image to calculate the pixel values of the pixels of the target image one by one, and when all the source images have overlapping areas, performs one-time superposition calculation on the overlapping areas again to obtain the pixel value of the pixel of the target image one by one, so as to obtain the internal image data of the target pixel, thereby forming the target image.
13. The method of real-time two-way video processing according to claim 5, wherein there is an image overlap region between source images input from at least two parallel video input sources.
CN202210108244.6A 2022-01-28 2022-01-28 Real-time two-way video processing system and method based on programmable chip Active CN114449245B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210108244.6A CN114449245B (en) 2022-01-28 2022-01-28 Real-time two-way video processing system and method based on programmable chip

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210108244.6A CN114449245B (en) 2022-01-28 2022-01-28 Real-time two-way video processing system and method based on programmable chip

Publications (2)

Publication Number Publication Date
CN114449245A CN114449245A (en) 2022-05-06
CN114449245B true CN114449245B (en) 2024-04-05

Family

ID=81372088

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210108244.6A Active CN114449245B (en) 2022-01-28 2022-01-28 Real-time two-way video processing system and method based on programmable chip

Country Status (1)

Country Link
CN (1) CN114449245B (en)

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105635602A (en) * 2015-12-31 2016-06-01 天津大学 System for mosaicing videos by adopting brightness and color cast between two videos and adjustment method thereof
CN107948544A (en) * 2017-11-28 2018-04-20 长沙全度影像科技有限公司 A kind of multi-channel video splicing system and method based on FPGA
CN108055472A (en) * 2017-12-21 2018-05-18 长沙全度影像科技有限公司 A kind of real time panoramic live broadcast system and method
CN109961401A (en) * 2017-12-14 2019-07-02 西安全志科技有限公司 A kind of method for correcting image and storage medium of binocular camera
CN110246081A (en) * 2018-11-07 2019-09-17 浙江大华技术股份有限公司 A kind of image split-joint method, device and readable storage medium storing program for executing
CN112399095A (en) * 2019-08-15 2021-02-23 西安诺瓦星云科技股份有限公司 Video processing method, device and system
CN112511896A (en) * 2020-11-05 2021-03-16 浙江大华技术股份有限公司 Video rendering method and device
CN112541972A (en) * 2019-09-23 2021-03-23 华为技术有限公司 Viewpoint image processing method and related equipment
CN113421184A (en) * 2021-06-21 2021-09-21 上海爱观视觉科技有限公司 Real-time image processing method based on FPGA

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140010479A1 (en) * 2012-07-09 2014-01-09 Samsung Electro-Mechanics Co., Ltd. Bilinear interpolation circuit for image and method thereof
US11206396B2 (en) * 2019-01-16 2021-12-21 Qualcomm Incorporated Local illumination compensation in video coding

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105635602A (en) * 2015-12-31 2016-06-01 天津大学 System for mosaicing videos by adopting brightness and color cast between two videos and adjustment method thereof
CN107948544A (en) * 2017-11-28 2018-04-20 长沙全度影像科技有限公司 A kind of multi-channel video splicing system and method based on FPGA
CN109961401A (en) * 2017-12-14 2019-07-02 西安全志科技有限公司 A kind of method for correcting image and storage medium of binocular camera
CN108055472A (en) * 2017-12-21 2018-05-18 长沙全度影像科技有限公司 A kind of real time panoramic live broadcast system and method
CN110246081A (en) * 2018-11-07 2019-09-17 浙江大华技术股份有限公司 A kind of image split-joint method, device and readable storage medium storing program for executing
CN112399095A (en) * 2019-08-15 2021-02-23 西安诺瓦星云科技股份有限公司 Video processing method, device and system
CN112541972A (en) * 2019-09-23 2021-03-23 华为技术有限公司 Viewpoint image processing method and related equipment
CN112511896A (en) * 2020-11-05 2021-03-16 浙江大华技术股份有限公司 Video rendering method and device
CN113421184A (en) * 2021-06-21 2021-09-21 上海爱观视觉科技有限公司 Real-time image processing method based on FPGA

Also Published As

Publication number Publication date
CN114449245A (en) 2022-05-06

Similar Documents

Publication Publication Date Title
US5586231A (en) Method and device for processing an image in order to construct from a source image a target image with charge of perspective
US5594845A (en) Method and device for processing an image in order to construct a target image from a plurality of contiguous source images
KR101915729B1 (en) Apparatus and Method for Generating 360 degree omni-directional view
US6191809B1 (en) Method and apparatus for aligning stereo images
US9699438B2 (en) 3D graphic insertion for live action stereoscopic video
CN104506826B (en) A kind of real-time splicing apparatus of fixed point orientation video without effective overlapping region
WO2017173735A1 (en) Video see-through-based smart eyeglasses system and see-through method thereof
KR101259559B1 (en) Method and apparatus for automatic controlling of multi cameras
JP4995092B2 (en) Stereo camera image distortion correction apparatus and method
US8326023B2 (en) Photographing field angle calculation apparatus
US20130235149A1 (en) Image capturing apparatus
CN107395985B (en) A kind of mobile terminal image composition method, device and mobile terminal
JPH10155104A (en) Compound eye image pickup method and device and storage medium
CN105744137A (en) System for real-time correction and splicing of image formed by fish-eye lens through using hardware circuit
WO2003036565A2 (en) System and method for obtaining video of multiple moving fixation points within a dynamic scene
CN114449245B (en) Real-time two-way video processing system and method based on programmable chip
KR101704362B1 (en) System for real time making of panoramic video base on lookup table and Method for using the same
JP2004056779A (en) Image inputting device
CN205647708U (en) System for use hardware circuit to correct and concatenation image that fisheye lens becomes is real -time
JPH0865715A (en) Method and device for stereoscopic video display
US11137582B2 (en) Omnidirectional catadioptric lens with odd aspheric contour or multi-lens
CN108900734B (en) Wide-angle lens distortion automatic correction device and method
JPH08251626A (en) Zooming position control device for stereoscopic television camera
JP5297899B2 (en) Stereoscopic image adjustment device, stereoscopic image adjustment system, and program
CN102033414B (en) Stereoscopic digital imaging converging device and method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant