CN107959805A - Light field video imaging system and method for processing video frequency based on Hybrid camera array - Google Patents
Light field video imaging system and method for processing video frequency based on Hybrid camera array Download PDFInfo
- Publication number
- CN107959805A CN107959805A CN201711262173.0A CN201711262173A CN107959805A CN 107959805 A CN107959805 A CN 107959805A CN 201711262173 A CN201711262173 A CN 201711262173A CN 107959805 A CN107959805 A CN 107959805A
- Authority
- CN
- China
- Prior art keywords
- image
- global
- light field
- resolution
- local
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012545 processing Methods 0.000 title claims abstract description 32
- 238000003384 imaging method Methods 0.000 title claims abstract description 13
- 238000000034 method Methods 0.000 title claims abstract description 11
- 230000009466 transformation Effects 0.000 claims description 52
- 238000001228 spectrum Methods 0.000 claims description 30
- 239000011159 matrix material Substances 0.000 claims description 22
- 238000001914 filtration Methods 0.000 claims description 18
- 230000001360 synchronised effect Effects 0.000 claims description 12
- 238000003672 processing method Methods 0.000 claims description 10
- 230000004927 fusion Effects 0.000 claims description 7
- 230000002123 temporal effect Effects 0.000 claims description 7
- 230000003287 optical effect Effects 0.000 claims description 6
- 238000005070 sampling Methods 0.000 claims description 6
- 230000001131 transforming effect Effects 0.000 claims description 6
- 238000011084 recovery Methods 0.000 claims description 5
- 238000007499 fusion processing Methods 0.000 claims description 4
- 238000012937 correction Methods 0.000 claims description 3
- 238000009499 grossing Methods 0.000 claims description 3
- 230000003068 static effect Effects 0.000 claims description 2
- 230000000007 visual effect Effects 0.000 claims 1
- 238000012876 topography Methods 0.000 abstract 2
- 238000000605 extraction Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 3
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000002194 synthesizing effect Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/265—Mixing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/268—Signal distribution or switching
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
The invention discloses light field video imaging system and method for processing video frequency based on Hybrid camera array, including control holder, image processing unit, storage unit and Hybrid camera array；Hybrid camera array includes global camera, global light-field camera and multiple local cameras, synchronously gathers global low resolution reference video, global light field video and the multiple local HD videos of Same Scene；Image processing unit carries out following processing to each group of reference picture shot in the same time, global light field image and multiple high-definition images：Each topography and reference picture are subjected to Block- matching, topography is transformed in the corresponding blocks of reference picture based on matching result, obtains global high-definition picture；Again based on global high-definition picture, super-resolution is carried out to global light field image, to obtain global high-resolution light field image；All frame overall situation high-resolution light field images are synthesized into global high-resolution light field video.
Description
Technical Field
The invention relates to the field of computer vision and video processing, in particular to a light field video imaging system based on a hybrid camera array and a related video processing method.
Background
Since ren.ng et al proposed a handheld model of a light field camera in 2005, the light field camera has been greatly developed and gradually moved out of the laboratory as a civilian product. However, in the current era of pursuing high resolution, the disadvantage of low spatial resolution of the light field camera is continuously amplified and even becomes an obstacle to further development of the light field camera. However, the characteristics of arbitrary zooming and multi-view stereoscopic display of light field video are much pursued, and therefore how to obtain high-pixel light field video is a problem that many people are continuously exploring.
The above background disclosure is only for the purpose of assisting understanding of the inventive concept and technical solutions of the present invention, and does not necessarily belong to the prior art of the present patent application, and should not be used for evaluating the novelty and inventive step of the present application in the case that there is no clear evidence that the above content is disclosed at the filing date of the present patent application.
Disclosure of Invention
Because angular resolution and spatial resolution have been a pair of irreconcilable contradictions for a light field camera, increasing the angular resolution necessarily sacrifices the spatial resolution, and increasing the spatial resolution necessarily reduces the angular resolution. The invention provides a light field video imaging system based on a hybrid camera array, which is characterized in that a global camera, a plurality of local high-definition cameras and a global light field camera are used for respectively and simultaneously acquiring global common images, local high-definition images and global light field images of the same wide view field, and then transformation, fusion and super-resolution are carried out to obtain the wide view field high-definition light field images, so that the defects of the light field camera are overcome, and billion-pixel-level wide view field light field videos can be obtained.
In order to achieve the above object, one embodiment of the present invention provides the following technical solutions:
a light field video imaging system based on a hybrid camera array comprises a control holder, an image processing unit, a storage unit and the hybrid camera array arranged on the control holder; the hybrid camera array comprises at least one global camera, at least one global light field camera, and a plurality of local cameras; the global camera, the global light field camera and the local cameras synchronously acquire a global low-resolution reference video, a global low-resolution light field video and a plurality of different local high-resolution videos of a preset scene respectively and store the videos in the storage unit; the image processing unit is connected with the storage unit and is used for carrying out image processing on a global low-resolution reference video, a global low-resolution light field video and each group of synchronous frame images of a plurality of high-resolution videos so as to obtain a global high-resolution light field video; wherein a set of synchronized frame images includes a global reference image from the reference video, a global light field image from the light field video, and local images from the high resolution videos, the image processing including the steps of S1 to S4:
s1, block matching: finding a corresponding block of each local image in the global reference image by adopting a block matching algorithm;
s2, transformation and registration: transforming each local image onto a corresponding block in the global reference image and performing image registration on each local image and its corresponding block to obtain a global high resolution image of the predetermined scene;
s3, super-resolution of images: performing up-sampling on the global light field image with low resolution by a set magnification factor to obtain a sampled low-resolution light field image, performing Fourier transform on the sampled low-resolution light field image to obtain a first frequency spectrum image, and performing low-pass filtering on the first frequency spectrum image; carrying out high-pass filtering on the global high-resolution image to obtain a second frequency spectrum image;
s4, fusion recovery: and performing inverse Fourier transform after the first frequency spectrum image subjected to low-pass filtering and the second frequency spectrum image are linearly added to obtain a global high-resolution light field image.
The light field video imaging system provided by the invention synchronously acquires global low-pixel video, global light field video and different local high-definition video of a long-distance wide view field by adopting a hybrid camera array consisting of a low-pixel global industrial camera, a low-pixel global light field camera and a plurality of high-definition local cameras, then performs image processing on the acquired videos by adopting a series of image processing algorithms, performs image matching transformation on images of the global low-pixel video and the local high-definition video to obtain global high-resolution images, performs image super-resolution on the images of the global light field video by combining the obtained global high-resolution images, compensates for the low spatial resolution of the high-resolution light field images of the global high-resolution images to obtain high-resolution light field images, and combines each frame of high-resolution light field images obtained by the super-resolution, the high-resolution light field video can be obtained, so that the technical contradiction that the spatial resolution and the angular resolution of the light field video can not be considered at the same time is overcome, and if a billion-pixel-level local camera is adopted in a matching manner, the wide-field billion-pixel-level light field video can be processed.
Another embodiment of the present invention provides a video processing method, configured to perform a fusion process on a global low-resolution reference video and a global low-resolution light field video of the same scene, which are acquired synchronously, and a plurality of local high-resolution videos of different viewing angles of the scene to obtain a global high-resolution light field video of the scene; the fusion process includes performing the following steps S1-S4 on each set of synchronized frame images of the reference video, the light field video, and the plurality of the local high resolution videos:
wherein a set of sync frame images includes a global reference image from the reference video, a global light field image from the light field video, and local images from the high resolution videos; the steps S1 to S4 are:
s1, block matching: finding a corresponding block of each local image in the global reference image by adopting a block matching algorithm;
s2, transformation and registration: transforming each local image onto a corresponding block in the global reference image and registering to obtain a global high-resolution image of the scene;
s3, super-resolution of images: performing up-sampling on the global light field image with low resolution by a set magnification factor to obtain a sampled low-resolution light field image, performing Fourier transform on the sampled low-resolution light field image to obtain a first frequency spectrum image, and performing low-pass filtering on the first frequency spectrum image; carrying out high-pass filtering on the global high-resolution image to obtain a second frequency spectrum image;
s4, fusion recovery: and performing inverse Fourier transform after the first frequency spectrum image subjected to low-pass filtering and the second frequency spectrum image are linearly added to obtain a global high-resolution light field image.
Drawings
FIG. 1 is a workflow diagram for obtaining a global high resolution light field video of a large scene using the light field video imaging system of the present invention;
FIG. 2 is a schematic diagram of a comparison of a global reference image and a local image;
fig. 3 is a process diagram of image processing.
Detailed Description
The invention is further described with reference to the following figures and detailed description of embodiments.
The specific implementation mode of the invention provides a light field video imaging system capable of generating billion-pixel light field videos, which is realized based on a hybrid camera array and a special video image fusion algorithm and specifically comprises a control holder, an image processing unit, a storage unit and the hybrid camera array arranged on the control holder. The hybrid camera array comprises at least one global camera, at least one global light field camera, and a plurality of local cameras; the global camera, the global light field camera and the local cameras synchronously acquire a global low-resolution reference video, a global low-resolution light field video and a plurality of different local high-resolution videos of a preset scene respectively and store the videos in the storage unit; referring to fig. 1, the image processing unit is connected to the storage unit, and is configured to perform image processing on a global low-resolution reference video, a global low-resolution light field video, and each group of synchronous frame images of a plurality of the high-resolution videos, so as to obtain a global high-resolution light field video; wherein a set of synchronized frame images includes a global reference image from the reference video, a global light field image from the light field video, and local images from the high resolution videos, referring to fig. 1, the image processing includes the following steps S1 to S4:
s1, block matching: finding a corresponding block of each local image in the global reference image by adopting a block matching algorithm;
s2, transformation and registration: transforming each local image onto a corresponding block in the global reference image and performing image registration on each local image and its corresponding block to obtain a global high resolution image of the predetermined scene;
s3, super-resolution of images: performing up-sampling on the global light field image with low resolution by a set magnification factor to obtain a sampled low-resolution light field image, performing Fourier transform on the sampled low-resolution light field image to obtain a first frequency spectrum image, and performing low-pass filtering on the first frequency spectrum image; carrying out high-pass filtering on the global high-resolution image to obtain a second frequency spectrum image;
s4, fusion recovery: and performing inverse Fourier transform after the first frequency spectrum image subjected to low-pass filtering and the second frequency spectrum image are linearly added to obtain a global high-resolution light field image.
In the hybrid camera array, the global camera is a common camera, the resolution of the global camera is much lower than that of the local camera, the global camera is used for shooting a global video with a wide field of view and is used as a reference video in subsequent video processing, and the global light field video acquired by the global light field camera has the traditional characteristics of low spatial resolution and high angular resolution. As shown in fig. 2, the image 10 is a global reference image of a frame captured by a global camera, the local image 100 'is a high-definition image of the local scene 100 in the image 10 at the same time captured by a local camera, and the image 1000' is an enlarged view of a local area in the local image 100 ', on which words on the guideboard are clearly displayed, and it can be seen that the pixels of the local image 100' are very high. The focal length of the local camera is much larger than that of the global camera, and in one embodiment, the focal length of the local camera is more than 8 times of the focal length of the global camera, and the focal length of the local camera may be, for example, 32-200 mm, and the focal length of the global camera may be, for example, 4-25 mm. Preferably, the hybrid camera array comprises one global camera, one global light field camera and a series (e.g. more than 100) of super-high pixel local cameras. The local camera can be fixedly arranged on the control cloud platform, and can also be rotatably arranged on the control cloud platform.
The global camera, the global light field camera and the local cameras realize image acquisition of a large scene at the same time based on a synchronous acquisition module of the cameras, and acquired image data and intermediate image data generated by the image processing unit in the image processing process can be stored in the storage unit.
In a specific embodiment, the image processing unit performs image processing in units of frames, that is, an algorithm flow of performing the image processing each time is performed, and processes a global reference image (captured by a global camera), a global light field image (captured by the global light field camera), and a plurality of different local images (captured by a plurality of local cameras, respectively) captured at the same time to obtain a global high-resolution light field image at the time. And executing the image processing algorithm on each group of synchronous frame images to obtain a plurality of continuous frames of global high-resolution light field images so as to synthesize a global high-resolution light field video. That is to say, the present invention can fuse, through the image processing algorithms of the foregoing steps S1 to S4, the global low-resolution reference video, the global low-resolution light field video, and the local high-resolution videos of the scene from different perspectives, which are acquired synchronously, of the same scene to obtain the global high-resolution light field video of the scene.
In a preferred embodiment, referring to fig. 1, in step S1, a zero-mean normalized cross-correlation block matching algorithm (abbreviated as "ZNCC algorithm") is used to perform block matching, preferably two ZNCC iterations are performed to find a corresponding block (also referred to as "corresponding region" or "reference block") of each local image in the global reference image, so as to obtain a pixel matching relationship between each local image and the global reference image. As shown in fig. 3, after passing through ZNCC block matching process 001, the corresponding reference block 101 of local image 101' in global reference image 10 can be found.
As in fig. 3, after finding the reference block of each local image in the global reference image, performed from path 002 onward, a transformation and registration will take place, which is roughly divided into three steps: global transforms, mesh-based transforms, and temporal, spatial smoothing transforms.
Firstly, integral transformation is carried out: each local image and its reference block are used as a pair, and then the extraction and matching of feature points are performed on each pair of local images and their corresponding blocks by adopting the ZNCC algorithm, so as to extract the corresponding (matched) feature point pairs in each local image and its corresponding block. In general, a certain number of pairs of feature points may be extracted in one iteration, and based on these pairs of feature points, a homography matrix between the local image and its corresponding block may be preliminarily calculated, and based on the homography matrix, image transformation may be performed, i.e., the local image is transformed onto the corresponding block in the global reference image. The extraction of the feature point pairs may be performed repeatedly, for example, a preliminary homography matrix is calculated after the first iterative extraction, more feature point pairs are extracted based on the preliminary homography matrix in the second iterative extraction, and the homography matrix may be recalculated based on the feature point pairs extracted twice for the next iterative extraction, and so on. In a preferred embodiment of the present invention, two ZNCC iterations are performed to extract the feature point pairs and calculate the homography matrix, and the process of the two iterations can be represented by the following iteration formula:
wherein,representing matching blocks between a local image and its corresponding blocks (characterizing the matching relationship between the two), Il、IrRespectively representing a local image and its corresponding block in a global reference image, plAnd prAre respectively partial images IlAnd its corresponding block IrCorresponding feature point of, i.e. plAnd prIs a characteristic point pair; ZNCC () represents an energy function for calculating a local image and a corresponding block thereof by using a ZNCC algorithm; h denotes a homography matrix, at initialization, firstInitializing into an identity matrix;is plPi () represents the central projection and the inverse normalization function; w is the size of the partial image (the partial image is a square, w represents the side length of the square), and epsilon is the search width. When feature point extraction and matching are performed in the first iteration, w and epsilon are both set to 256, H in the constraint condition (s.t. represents the constraint condition) is an identity matrix, as shown in fig. 3, based on the identity matrix, the iteration formula is used for performing the first iteration 003 to establish a matching correspondence, and the successfully-matched feature points are displayed in the local image 101' a, that is, a certain number of feature point pairs are extracted, and a homography matrix H1 is calculated (H in the constraint condition is replaced in the second iteration); and performing a second iteration 005 to improve matching, setting w and epsilon as 128, extracting more feature point pairs and recalculating the homography matrix to obtain an updated homography matrix H2, wherein feature points successfully matched after two iterations are displayed in the local image 101' b. In the preferred embodiment, the local image is transformed onto the corresponding reference block over two iterations 003 and 005. And performing ZNCC iteration on all local images to extract characteristic point pairs and calculate corresponding homography matrixes respectively so as to transform all local images to corresponding reference blocks respectively, thereby completing the overall transformation and obtaining a primary global transformation image.
The global transformation image obtained by the above-mentioned overall transformation has some regions with poor transformation effect due to the lack of feature point pairs, especially when the feature point pairs are located at the edge of the image. Therefore a mesh-based transformation is then performed: on the basis of the preliminary global transformation image obtained above, the feature point pairs extracted in the overall transformation process are subjected to grid-based transformation (H3 in fig. 3 indicates that transformation is performed based on grids) by using an ASAP transformation frame (most similar transformation frame), and then subjected to optical flow-based transformation (H4 in fig. 3 indicates that transformation is performed based on optical flows) on the result of the grid transformation, so as to optimize the pixel matching relationship, obtain more reliable feature point pairs, and obtain feature points showing more successful matching in the local image 101' c at this time and changed optical flows. The distortion of the optical flow transformation is combined with the stability of the local image, the homography matrix is recalculated, the grid-based transformation and the optical flow-based transformation are completed, and the transformation result 101'd is obtained. The transformed and registered partial images 101 "are then color calibrated 012. The flow shown in fig. 3 is performed for each local image, and all local images are transformed onto the reference image, thereby obtaining an optimized global transformed image.
The optimized global transformation image is a high-resolution global image synthesized by local images, and in order to ensure the temporal stability of the overall video after all the subsequent frames are combined together, the optimized global transformation image can be subjected to temporal and spatial smooth transformation.
In a specific embodiment, a time and space smooth transformation is performed by introducing a time stability constraint, and the energy function of the smooth transformation is
E(V)＝λrEr(V)+λtEt(V)+λsEs(V)
Where V represents a homography matrix of the transformation that depends on the vertices of the mesh, Er(V) is the sum of the distances of each pair of feature points between the local images in the global transform image and the global reference image, Et(V) is a temporal stability constraint; es(V) is a spatial smoothing term defined as the spatial deformation between adjacent vertices; lambda [ alpha ]r、λtAnd λsAre all constants greater than 0; wherein:
αplis plBilinear interpolation weights of (1);
pl' is a feature point p in a local imagelCorresponding characteristic points in the time prior graph; b is an indication function for checking the pixel point plWhether or not on a static background, when B (p)l) 0 denotes a pixel plOn a moving background; s is the global transformation function between the local image and its temporal prior map.
After the series of transformation and registration, a global high-resolution image is obtained, wherein in consideration of the problem of inconsistent color of each local image in the global high-resolution image caused by different color illumination of the local camera, color correction can be performed on each local image until the local image is consistent with the global reference image, so that the global high-resolution image has a uniform color style as a whole. In addition, such an optimization can also be made for a global high resolution image: and removing the overlapped part between the local images on the transformation by adopting a graph cutting method so as to minimize the error of image registration.
After obtaining the global high-resolution image, the global light field image needs to be subjected to image super-resolution to improve the problem of low spatial resolution, and the specific method is as follows: performing up-sampling on a global light field image with low resolution (referring to spatial resolution) by a set magnification factor to obtain a sampled low-resolution light field image, performing Fourier transform on the sampled low-resolution light field image to obtain a first frequency spectrum image, and performing low-pass filtering on the first frequency spectrum image; and carrying out high-pass filtering on the global high-resolution image to obtain a second frequency spectrum image. And performing inverse Fourier transform after the first frequency spectrum image subjected to low-pass filtering and the second frequency spectrum image are linearly added to obtain a global high-resolution light field image. Wherein the set multiple is fh/fl，fhAnd flThe focal lengths of the local camera and the global camera, respectively.
And performing the above-mentioned series of image processing on each group of synchronous frame images of the global reference video, the global light field video and all local videos, so that each group of synchronous frame images finally obtains a corresponding frame of global high-resolution light field image. And finally, synthesizing all the frame global high-resolution light field images into the global high-resolution light field video required by us. Using a local camera at the gigapixel level in the system to acquire a local image at the gigapixel level, a global gigapixel light field video of a wide field of view can be finally obtained.
The foregoing is a more detailed description of the invention in connection with specific preferred embodiments and it is not intended that the invention be limited to these specific details. For those skilled in the art to which the invention pertains, several equivalent substitutions or obvious modifications can be made without departing from the spirit of the invention, and all the properties or uses are considered to be within the scope of the invention.
Claims (11)
1. A light field video imaging system based on a hybrid camera array, characterized by: the system comprises a control holder, an image processing unit, a storage unit and a hybrid camera array arranged on the control holder;
the hybrid camera array comprises at least one global camera, at least one global light field camera, and a plurality of local cameras; the global camera, the global light field camera and the local cameras synchronously acquire a global low-resolution reference video, a global low-resolution light field video and a plurality of different local high-resolution videos of a preset scene respectively and store the videos in the storage unit;
the image processing unit is connected with the storage unit and is used for carrying out image processing on a global low-resolution reference video, a global low-resolution light field video and each group of synchronous frame images of a plurality of high-resolution videos so as to obtain a global high-resolution light field video; wherein a set of synchronized frame images includes a global reference image from the reference video, a global light field image from the light field video, and local images from the high resolution videos, the image processing including the steps of S1 to S4:
s1, block matching: finding a corresponding block of each local image in the global reference image by adopting a block matching algorithm;
s2, transformation and registration: transforming each local image onto a corresponding block in the global reference image and performing image registration on each local image and its corresponding block to obtain a global high resolution image of the predetermined scene;
s3, super-resolution of images: performing up-sampling on the global light field image with low resolution by a set magnification factor to obtain a sampled low-resolution light field image, performing Fourier transform on the sampled low-resolution light field image to obtain a first frequency spectrum image, and performing low-pass filtering on the first frequency spectrum image; carrying out high-pass filtering on the global high-resolution image to obtain a second frequency spectrum image;
s4, fusion recovery: and performing inverse Fourier transform after the first frequency spectrum image subjected to low-pass filtering and the second frequency spectrum image are linearly added to obtain a global high-resolution light field image.
2. The light field video imaging system of claim 1, wherein: step S2 further includes performing color correction on each local image to keep the local image color consistent with the global reference image.
3. The light field video imaging system of claim 1, wherein: the global camera and the global light field camera are fixedly arranged on the control cloud deck, and the local camera is rotatably arranged on the control cloud deck.
4. The light field video imaging system of claim 1, wherein: the focal length of the local camera is more than 8 times of the focal length of the global camera.
5. A video processing method, characterized by: the system comprises a video acquisition module, a video processing module and a video processing module, wherein the video acquisition module is used for carrying out fusion processing on a global low-resolution reference video and a global low-resolution light field video of the same scene which are synchronously acquired and local high-resolution videos of a plurality of different visual angles of the scene to obtain the global high-resolution light field video of the scene;
the fusion process includes performing the following steps S1-S4 on each set of synchronized frame images of the reference video, the light field video, and the plurality of the local high resolution videos:
wherein a set of sync frame images includes a global reference image from the reference video, a global light field image from the light field video, and local images from the high resolution videos; the steps S1 to S4 are:
s1, block matching: finding a corresponding block of each local image in the global reference image by adopting a block matching algorithm;
s2, transformation and registration: transforming each local image onto a corresponding block in the global reference image and registering to obtain a global high-resolution image of the scene;
s3, super-resolution of images: performing up-sampling on the global light field image with low resolution by a set magnification factor to obtain a sampled low-resolution light field image, performing Fourier transform on the sampled low-resolution light field image to obtain a first frequency spectrum image, and performing low-pass filtering on the first frequency spectrum image; carrying out high-pass filtering on the global high-resolution image to obtain a second frequency spectrum image;
s4, fusion recovery: and performing inverse Fourier transform after the first frequency spectrum image subjected to low-pass filtering and the second frequency spectrum image are linearly added to obtain a global high-resolution light field image.
6. The video processing method of claim 5, wherein: step S2 further includes performing color correction on each local image to keep the local image color consistent with the global reference image.
7. The video processing method of claim 5, wherein: in step S1, a zero-mean normalized cross-correlation block matching algorithm is used to perform block matching to find a corresponding block of each local image in the global reference image, so as to obtain a pixel matching relationship between each local image and the global reference image.
8. The video processing method of claim 7, wherein: step S2 specifically includes S21 to S25:
s21, extracting and matching feature points of the local image and the corresponding block thereof by adopting a zero-mean normalized cross-correlation block matching algorithm to extract corresponding feature point pairs of the local image and the corresponding block thereof; calculating a homography matrix between the local image and the corresponding block of the local image based on the extracted characteristic point pairs, and transforming the local image to the corresponding block of the local image according to the homography matrix;
s22, executing the step S21 on each local image to transform all local images to each corresponding block in the global reference image to obtain a preliminary global transformation image;
s23, on the basis of the preliminary global transformation image, carrying out grid-based transformation on the feature point pairs extracted in the step S21 by using an ASAP transformation frame, and then carrying out optical flow-based transformation on the result of the grid transformation to optimize the pixel matching relationship and obtain more feature point pairs;
s24, recalculating the homography matrix between the local image and the corresponding block thereof based on all the characteristic point pairs acquired in the step S23 to perform second transformation to obtain an optimized global transformation image;
and S25, performing smooth transformation of time and space on the optimized global transformation image to obtain the global high-resolution image.
9. The video processing method of claim 8, wherein: in step S21, performing two iterations by using a zero-mean normalized cross-correlation block matching algorithm to extract feature point pairs, wherein in the first iteration, a first group of feature point pairs between a local image and a corresponding block thereof are extracted, and an estimated homography matrix is calculated based on the first group of feature point pairs; during the second iteration, extracting more characteristic point pairs based on the estimation homography matrix; and calculating a homography matrix between the local image and the corresponding block of the local image based on all the characteristic point pairs obtained by the two iterations.
10. The video processing method of claim 9, wherein: performing a smooth transformation in time and space by introducing a time stability constraint condition, wherein the energy function of the smooth transformation is
E(V)＝λrEr(V)+λtEt(V)+λsEs(V)
Where V represents a homography matrix of the transformation that depends on the vertices of the mesh, Er(V) is the sum of the distances of each pair of feature points between the local images in the global transform image and the global reference image, Et(V) is a temporal stability constraint, Es(V) is a spatial smoothing term, λr、λtAnd λsAre all constants greater than 0; wherein:
<mrow> <msub> <mi>E</mi> <mi>r</mi> </msub> <mrow> <mo>(</mo> <mi>V</mi> <mo>)</mo> </mrow> <mo>=</mo> <mi>&Sigma;</mi> <mo>|</mo> <mo>|</mo> <msub> <mi>&alpha;</mi> <mrow> <mi>p</mi> <mi>l</mi> </mrow> </msub> <mi>V</mi> <mo>-</mo> <msub> <mi>p</mi> <mi>r</mi> </msub> <mo>|</mo> <msubsup> <mo>|</mo> <mn>2</mn> <mn>2</mn> </msubsup> <mo>;</mo> </mrow>
<mrow> <msub> <mi>E</mi> <mi>t</mi> </msub> <mrow> <mo>(</mo> <mi>V</mi> <mo>)</mo> </mrow> <mo>=</mo> <mi>&Sigma;</mi> <mi>B</mi> <mrow> <mo>(</mo> <msub> <mi>p</mi> <mi>l</mi> </msub> <mo>)</mo> </mrow> <mo>|</mo> <mo>|</mo> <msub> <mi>&alpha;</mi> <mrow> <mi>p</mi> <mi>l</mi> </mrow> </msub> <mi>V</mi> <mo>-</mo> <mrow> <mo>(</mo> <msup> <msub> <mi>p</mi> <mi>l</mi> </msub> <mo>&prime;</mo> </msup> <mo>+</mo> <mi>s</mi> <mo>)</mo> </mrow> <mo>|</mo> <msubsup> <mo>|</mo> <mn>2</mn> <mn>2</mn> </msubsup> <mo>;</mo> </mrow>
Es(V) is the spatial deformation between adjacent vertices;
wherein p islAnd prcorresponding feature points, α, of corresponding blocks in the local image and the global reference image, respectivelyplIs plBilinear interpolation weights of pl' is a feature point p in a local imagelCorresponding characteristic points in the time prior graph; b is an indication function for checking the pixel point plWhether or not on a static background, when B (p)l) 0 denotes a pixel plOn a moving background; s is the global transformation function between the local image and its temporal prior map.
11. The video processing method of claim 5, wherein: step S2 further includes using a graph cutting method to remove the overlapped parts between the transformed local images for the global high resolution image, so as to minimize the error of image registration.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711262173.0A CN107959805B (en) | 2017-12-04 | 2017-12-04 | Light field video imaging system and method for processing video frequency based on Hybrid camera array |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711262173.0A CN107959805B (en) | 2017-12-04 | 2017-12-04 | Light field video imaging system and method for processing video frequency based on Hybrid camera array |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107959805A true CN107959805A (en) | 2018-04-24 |
| CN107959805B CN107959805B (en) | 2019-09-13 |
Family
ID=61963130
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711262173.0A Active CN107959805B (en) | 2017-12-04 | 2017-12-04 | Light field video imaging system and method for processing video frequency based on Hybrid camera array |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107959805B (en) |
Cited By (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109151334A (en) * | 2018-09-21 | 2019-01-04 | 中国计量大学 | A kind of unmanned vehicle camera system |
| CN109167924A (en) * | 2018-10-24 | 2019-01-08 | 清华-伯克利深圳学院筹备办公室 | Video imaging method, system, equipment and storage medium based on Hybrid camera |
| CN109285136A (en) * | 2018-08-31 | 2019-01-29 | 清华-伯克利深圳学院筹备办公室 | A kind of Multiscale Fusion method, apparatus, storage medium and the terminal of image |
| CN110044291A (en) * | 2019-05-16 | 2019-07-23 | 苏州汇才土水工程科技有限公司 | A kind of method of camera battle array measurement local deformation |
| CN110057654A (en) * | 2019-05-16 | 2019-07-26 | 苏州汇才土水工程科技有限公司 | A kind of New universal material test machine |
| CN110086994A (en) * | 2019-05-14 | 2019-08-02 | 宁夏融媒科技有限公司 | A kind of integrated system of the panorama light field based on camera array |
| CN110430445A (en) * | 2019-06-26 | 2019-11-08 | 北京拙河科技有限公司 | A kind of audio video synchronization playback method, device, equipment and medium |
| CN111640065A (en) * | 2020-05-29 | 2020-09-08 | 北京拙河科技有限公司 | Image stitching method and imaging device based on camera array |
| CN111882589A (en) * | 2020-06-23 | 2020-11-03 | 广州万维创新科技有限公司 | Image-based monocular vision SLAM initialization method |
| CN111932594A (en) * | 2020-09-18 | 2020-11-13 | 西安拙河安见信息科技有限公司 | Billion pixel video alignment method and device based on optical flow and medium |
| CN111935417A (en) * | 2020-07-02 | 2020-11-13 | 清华大学 | Hierarchical video splicing method and device based on multi-scale camera array |
| CN112040140A (en) * | 2020-09-02 | 2020-12-04 | 衢州光明电力投资集团有限公司赋腾科技分公司 | Wide-view-field high-resolution hybrid imaging device based on light field |
| CN112381894A (en) * | 2021-01-15 | 2021-02-19 | 清华大学 | Adaptive light field imaging calibration method, device and storage medium |
| CN113014871A (en) * | 2021-02-20 | 2021-06-22 | 青岛小鸟看看科技有限公司 | Endoscope image display method, device and endoscope operation auxiliary system |
| CN113079325A (en) * | 2021-03-18 | 2021-07-06 | 北京拙河科技有限公司 | Method, apparatus, medium, and device for imaging billions of pixels under dim light conditions |
| WO2022007840A1 (en) * | 2020-07-08 | 2022-01-13 | 杭州萤石软件有限公司 | Image fusion method and device |
| CN114170567A (en) * | 2021-12-01 | 2022-03-11 | 北京拙河科技有限公司 | Light field camera AI analysis system and method for monitoring hundred million level pixels of area |
| CN114373153A (en) * | 2022-01-12 | 2022-04-19 | 北京拙河科技有限公司 | Video imaging optimization system and method based on multi-scale array camera |
| CN114612613A (en) * | 2022-03-07 | 2022-06-10 | 北京拙河科技有限公司 | Dynamic light field reconstruction method and system |
| CN115103125A (en) * | 2022-07-13 | 2022-09-23 | 北京拙河科技有限公司 | Broadcasting directing method and device |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080180754A1 (en) * | 2007-01-25 | 2008-07-31 | Mark Eric Miller | Image Illumination And Capture In A Scanning Device |
| CN101720027A (en) * | 2009-11-27 | 2010-06-02 | 西安电子科技大学 | Method for cooperative acquisition of multi-target videos under different resolutions by variable-focus array camera |
| CN102148999A (en) * | 2011-01-10 | 2011-08-10 | 南京理工大学 | Wide-waveband large-span tuned filter single-channel colorful low-level-light night vision method and device |
| CN102656878A (en) * | 2009-11-27 | 2012-09-05 | 三星电子株式会社 | Image processing apparatus and method |
| CN103442234A (en) * | 2013-08-31 | 2013-12-11 | 西安电子科技大学 | Infrared super-resolution imaging method based on time varying encoding measurement |
| CN105844589A (en) * | 2016-03-21 | 2016-08-10 | 深圳市未来媒体技术研究院 | Method for realizing light field image super-resolution based on mixed imaging system |
| CN105913070A (en) * | 2016-04-29 | 2016-08-31 | 合肥工业大学 | Multi-thread significance method based on light field camera |
| CN206450959U (en) * | 2017-01-24 | 2017-08-29 | 叠境数字科技（上海）有限公司 | Light field panorama camera |
-
2017
- 2017-12-04 CN CN201711262173.0A patent/CN107959805B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080180754A1 (en) * | 2007-01-25 | 2008-07-31 | Mark Eric Miller | Image Illumination And Capture In A Scanning Device |
| CN101720027A (en) * | 2009-11-27 | 2010-06-02 | 西安电子科技大学 | Method for cooperative acquisition of multi-target videos under different resolutions by variable-focus array camera |
| CN102656878A (en) * | 2009-11-27 | 2012-09-05 | 三星电子株式会社 | Image processing apparatus and method |
| CN102148999A (en) * | 2011-01-10 | 2011-08-10 | 南京理工大学 | Wide-waveband large-span tuned filter single-channel colorful low-level-light night vision method and device |
| CN103442234A (en) * | 2013-08-31 | 2013-12-11 | 西安电子科技大学 | Infrared super-resolution imaging method based on time varying encoding measurement |
| CN105844589A (en) * | 2016-03-21 | 2016-08-10 | 深圳市未来媒体技术研究院 | Method for realizing light field image super-resolution based on mixed imaging system |
| CN105913070A (en) * | 2016-04-29 | 2016-08-31 | 合肥工业大学 | Multi-thread significance method based on light field camera |
| CN206450959U (en) * | 2017-01-24 | 2017-08-29 | 叠境数字科技（上海）有限公司 | Light field panorama camera |
Cited By (32)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109285136B (en) * | 2018-08-31 | 2021-06-08 | 清华-伯克利深圳学院筹备办公室 | Multi-scale fusion method and device for images, storage medium and terminal |
| CN109285136A (en) * | 2018-08-31 | 2019-01-29 | 清华-伯克利深圳学院筹备办公室 | A kind of Multiscale Fusion method, apparatus, storage medium and the terminal of image |
| WO2020043155A1 (en) * | 2018-08-31 | 2020-03-05 | 清华-伯克利深圳学院筹备办公室 | Multiple scale image fusion method and device, storage medium, and terminal |
| CN109151334B (en) * | 2018-09-21 | 2020-12-22 | 中国计量大学 | Unmanned vehicle camera system |
| CN109151334A (en) * | 2018-09-21 | 2019-01-04 | 中国计量大学 | A kind of unmanned vehicle camera system |
| CN109167924A (en) * | 2018-10-24 | 2019-01-08 | 清华-伯克利深圳学院筹备办公室 | Video imaging method, system, equipment and storage medium based on Hybrid camera |
| WO2020082722A1 (en) * | 2018-10-24 | 2020-04-30 | 清华-伯克利深圳学院筹备办公室 | Mixed camera-based video imaging method, system, device, and storage medium |
| CN110086994A (en) * | 2019-05-14 | 2019-08-02 | 宁夏融媒科技有限公司 | A kind of integrated system of the panorama light field based on camera array |
| CN110044291A (en) * | 2019-05-16 | 2019-07-23 | 苏州汇才土水工程科技有限公司 | A kind of method of camera battle array measurement local deformation |
| CN110057654A (en) * | 2019-05-16 | 2019-07-26 | 苏州汇才土水工程科技有限公司 | A kind of New universal material test machine |
| CN110430445A (en) * | 2019-06-26 | 2019-11-08 | 北京拙河科技有限公司 | A kind of audio video synchronization playback method, device, equipment and medium |
| CN111640065B (en) * | 2020-05-29 | 2023-06-23 | 深圳拙河科技有限公司 | Image stitching method and imaging device based on camera array |
| CN111640065A (en) * | 2020-05-29 | 2020-09-08 | 北京拙河科技有限公司 | Image stitching method and imaging device based on camera array |
| CN111882589A (en) * | 2020-06-23 | 2020-11-03 | 广州万维创新科技有限公司 | Image-based monocular vision SLAM initialization method |
| CN111935417A (en) * | 2020-07-02 | 2020-11-13 | 清华大学 | Hierarchical video splicing method and device based on multi-scale camera array |
| CN111935417B (en) * | 2020-07-02 | 2021-12-14 | 清华大学 | Hierarchical video splicing method and device based on multi-scale camera array |
| WO2022007840A1 (en) * | 2020-07-08 | 2022-01-13 | 杭州萤石软件有限公司 | Image fusion method and device |
| CN112040140A (en) * | 2020-09-02 | 2020-12-04 | 衢州光明电力投资集团有限公司赋腾科技分公司 | Wide-view-field high-resolution hybrid imaging device based on light field |
| CN111932594B (en) * | 2020-09-18 | 2023-12-19 | 西安拙河安见信息科技有限公司 | Billion pixel video alignment method and device based on optical flow and medium |
| CN111932594A (en) * | 2020-09-18 | 2020-11-13 | 西安拙河安见信息科技有限公司 | Billion pixel video alignment method and device based on optical flow and medium |
| CN112381894B (en) * | 2021-01-15 | 2021-04-20 | 清华大学 | Adaptive light field imaging calibration method, device and storage medium |
| CN112381894A (en) * | 2021-01-15 | 2021-02-19 | 清华大学 | Adaptive light field imaging calibration method, device and storage medium |
| CN113014871A (en) * | 2021-02-20 | 2021-06-22 | 青岛小鸟看看科技有限公司 | Endoscope image display method, device and endoscope operation auxiliary system |
| CN113014871B (en) * | 2021-02-20 | 2023-11-10 | 青岛小鸟看看科技有限公司 | Endoscopic image display method and device and endoscopic surgery auxiliary system |
| CN113079325A (en) * | 2021-03-18 | 2021-07-06 | 北京拙河科技有限公司 | Method, apparatus, medium, and device for imaging billions of pixels under dim light conditions |
| CN114170567A (en) * | 2021-12-01 | 2022-03-11 | 北京拙河科技有限公司 | Light field camera AI analysis system and method for monitoring hundred million level pixels of area |
| CN114170567B (en) * | 2021-12-01 | 2022-07-12 | 北京拙河科技有限公司 | Light field camera AI analysis system and method for monitoring hundred million level pixels of area |
| CN114373153A (en) * | 2022-01-12 | 2022-04-19 | 北京拙河科技有限公司 | Video imaging optimization system and method based on multi-scale array camera |
| CN114373153B (en) * | 2022-01-12 | 2022-12-27 | 北京拙河科技有限公司 | Video imaging optimization system and method based on multi-scale array camera |
| CN114612613A (en) * | 2022-03-07 | 2022-06-10 | 北京拙河科技有限公司 | Dynamic light field reconstruction method and system |
| CN114612613B (en) * | 2022-03-07 | 2022-11-29 | 北京拙河科技有限公司 | Dynamic light field reconstruction method and system |
| CN115103125A (en) * | 2022-07-13 | 2022-09-23 | 北京拙河科技有限公司 | Broadcasting directing method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107959805B (en) | 2019-09-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107959805B (en) | Light field video imaging system and method for processing video frequency based on Hybrid camera array | |
| JP6273163B2 (en) | Stereoscopic panorama | |
| CN101276465B (en) | Method for automatically split-jointing wide-angle image | |
| CN103971352A (en) | Rapid image splicing method based on wide-angle lenses | |
| CN103826033B (en) | Image processing method, image processing equipment, image pick up equipment and storage medium | |
| KR102013978B1 (en) | Method and apparatus for fusion of images | |
| CN107767339B (en) | Binocular stereo image splicing method | |
| CN101872491A (en) | Free view angle relighting method and system based on photometric stereo | |
| EP2823461A1 (en) | Method and apparatus for performing super-resolution | |
| WO2010051147A2 (en) | Method and apparatus for transforming a non-linear lens-distorted image | |
| WO2002045003A1 (en) | Techniques and systems for developing high-resolution imagery | |
| JP2009037460A (en) | Image processing method, image processor, and electronic equipment equipped with image processor | |
| CN107564091A (en) | A kind of three-dimensional rebuilding method and device based on quick corresponding point search | |
| TW201246124A (en) | Image generation apparatus | |
| Wee et al. | A novel fast fractal super resolution technique | |
| CN111553841B (en) | Real-time video splicing method based on optimal suture line updating | |
| US10600154B2 (en) | System and method for generating high-resolution stereo image and depth map | |
| CN105005964A (en) | Video sequence image based method for rapidly generating panorama of geographic scene | |
| Jeong et al. | Multi-frame example-based super-resolution using locally directional self-similarity | |
| CN104796623B (en) | Splicing video based on pyramid Block-matching and functional optimization goes structural deviation method | |
| CN110784642B (en) | Image processing apparatus, control method thereof, storage medium, and imaging apparatus | |
| CN107330856A (en) | A kind of method for panoramic imaging based on projective transformation and thin plate spline | |
| CN104933692B (en) | A kind of method for reconstructing and device of human face super-resolution | |
| JP2013069012A (en) | Multi-lens imaging apparatus | |
| Fujiyama et al. | Multiple view geometries for mirrors and cameras |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP01 | Change in the name or title of a patent holder |
Address after: 518000 Guangdong city in Shenzhen Province, Nanshan District City Xili University Tsinghua Park F305B Co-patentee after: Tsinghua Shenzhen International Graduate School Patentee after: SHENZHEN WEILAI MEDIA TECHNOLOGY Research Institute Address before: 518000 Guangdong city in Shenzhen Province, Nanshan District City Xili University Tsinghua Park F305B Co-patentee before: Graduate School at Shenzhen, Tsinghua University Patentee before: SHENZHEN WEILAI MEDIA TECHNOLOGY RESEARCH INSTITUTE |
|
| CP01 | Change in the name or title of a patent holder | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20191101 Address after: 100083 No. 1 Wangzhuang Road, Haidian District, Beijing Patentee after: Beijing Zhuohe Technology Co.,Ltd. Address before: 518000 Guangdong city in Shenzhen Province, Nanshan District City Xili University Tsinghua Park F305B Co-patentee before: Tsinghua Shenzhen International Graduate School Patentee before: SHENZHEN WEILAI MEDIA TECHNOLOGY RESEARCH INSTITUTE |
|
| TR01 | Transfer of patent right |