CN104410789A - Staring super-resolution imaging device and method - Google Patents
Staring super-resolution imaging device and method Download PDFInfo
- Publication number
- CN104410789A CN104410789A CN201410746361.0A CN201410746361A CN104410789A CN 104410789 A CN104410789 A CN 104410789A CN 201410746361 A CN201410746361 A CN 201410746361A CN 104410789 A CN104410789 A CN 104410789A
- Authority
- CN
- China
- Prior art keywords
- detector
- resolution
- image
- super
- imaging device
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003384 imaging method Methods 0.000 title claims abstract description 53
- 238000000034 method Methods 0.000 title claims abstract description 29
- 230000003287 optical effect Effects 0.000 claims abstract description 26
- 238000003860 storage Methods 0.000 claims abstract description 23
- 238000012545 processing Methods 0.000 claims abstract description 18
- 239000011159 matrix material Substances 0.000 claims description 12
- 230000000694 effects Effects 0.000 claims description 11
- 230000008859 change Effects 0.000 claims description 5
- 238000004364 calculation method Methods 0.000 claims description 4
- 230000000295 complement effect Effects 0.000 claims description 4
- 230000006872 improvement Effects 0.000 claims description 4
- 239000000919 ceramic Substances 0.000 claims description 3
- 238000002347 injection Methods 0.000 claims description 3
- 239000007924 injection Substances 0.000 claims description 3
- 229910044991 metal oxide Inorganic materials 0.000 claims description 3
- 150000004706 metal oxides Chemical class 0.000 claims description 3
- 239000004065 semiconductor Substances 0.000 claims description 3
- 238000002945 steepest descent method Methods 0.000 claims description 3
- 238000006073 displacement reaction Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 7
- 230000008569 process Effects 0.000 description 7
- 238000009826 distribution Methods 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 239000000243 solution Substances 0.000 description 2
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011549 displacement method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012634 optical imaging Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 238000003466 welding Methods 0.000 description 1
Landscapes
- Transforming Light Signals Into Electric Signals (AREA)
Abstract
本发明提供了一种凝视型超分辨率成像装置及方法,用于解决现有成像装置分辨率低的问题。该装置包括成像装置镜头组(1)、探测器(2)、探测器驱动平台(3)、驱动器(4)、存储单元(5)、图像处理电路(6)和图像显示电路(7);探测器(2)设置于成像装置镜头组(1)光路上的焦平面内;驱动器(4)的驱动部分带动探测器驱动平台(3)产生大小和方向均为随机值的移动。实现成像方法的步骤:设置变动次数指令;驱动器(4)驱动探测器(4)作相应次数的随机抖动;光信号经成像装置镜头组(1)在探测器(4)上成像,获得与变动次数相应帧数的低分辨率图像;利用变分贝叶斯算法对低分辨率图像重建。适用于视频重建和卫星拍摄。
The present invention provides a staring super-resolution imaging device and method, which are used to solve the problem of low resolution of existing imaging devices. The device comprises an imaging device lens group (1), a detector (2), a detector driving platform (3), a driver (4), a storage unit (5), an image processing circuit (6) and an image display circuit (7); The detector (2) is arranged in the focal plane on the optical path of the lens group (1) of the imaging device; the driving part of the driver (4) drives the detector driving platform (3) to generate movements of random values in size and direction. The steps of realizing the imaging method: setting the command of the number of changes; the driver (4) drives the detector (4) to perform random shaking of the corresponding number of times; the optical signal is imaged on the detector (4) through the lens group (1) of the imaging device, and obtained and changed Low-resolution images corresponding to the number of frames; use variational Bayesian algorithm to reconstruct low-resolution images. Suitable for video reconstruction and satellite capture.
Description
技术领域technical field
本发明属于超分辨率成像技术领域,具体涉及一种多帧图像获取和重建的凝视型超分辨率成像装置及成像方法,可用于视频重建和卫星拍摄。The invention belongs to the technical field of super-resolution imaging, and in particular relates to a staring super-resolution imaging device and imaging method for multi-frame image acquisition and reconstruction, which can be used for video reconstruction and satellite shooting.
背景技术Background technique
提高光学成像系统的分辨率是开展图像科学研究及其工程应用的不懈追求,除了可以通过提高成像系统相关部件,如光学系统焦距、孔径和探测器等的性能以提高分辨率外,还可以选择在成像系统中加入合适的光学器件以提高分辨率,但是调整成像系统结构在一定程度上会导致系统的复杂化和任务的大幅增加,因而如何基于现有成像设备恢复或重构出超分辨率图像成为当今众多图像应用领域的迫切需求。在科研中,人们将目光逐渐投入到应用现代光学和数字图像处理技术来提高成像系统的分辨率上来。Improving the resolution of the optical imaging system is the relentless pursuit of image science research and its engineering applications. In addition to improving the resolution by improving the performance of imaging system-related components, such as optical system focal length, aperture and detectors, you can also choose Adding suitable optical devices to the imaging system can improve the resolution, but adjusting the structure of the imaging system will lead to the complexity of the system and a large increase in tasks to a certain extent, so how to restore or reconstruct super-resolution based on existing imaging equipment Image has become an urgent need in many image application fields today. In scientific research, people gradually focus on applying modern optics and digital image processing technology to improve the resolution of the imaging system.
提高成像装置分辨率可以通过单帧图像或多帧图像的重构实现,由于单帧图像包含的信息量有限,重建过程中缺乏新的信息,因而分辨率提高的效果不是很理想。基于多帧图像的超分辨率重建算法,即利用同一场景的多帧低分辨率图像来获取该场景的一帧超分辨率图像,因为多帧图像包含的信息量要大于单帧图像,图像序列间彼此包含有类似但又不完全相同的互补信息及一定的先验信息,因而为恢复出真实有效的超分辨率图像提供了可能。Improving the resolution of an imaging device can be achieved by reconstructing a single-frame image or multiple-frame images. Due to the limited amount of information contained in a single-frame image and the lack of new information in the reconstruction process, the effect of improving the resolution is not ideal. The super-resolution reconstruction algorithm based on multi-frame images uses multiple low-resolution images of the same scene to obtain a frame of super-resolution images of the scene, because the amount of information contained in multi-frame images is greater than that of single-frame images, image sequences They contain similar but not identical complementary information and certain prior information, which makes it possible to restore real and effective super-resolution images.
目前在多帧图像的获取方式上,可以采用三种方式实现:第一种是通过镜头的移动获取多帧低分辨率图像。深圳雅图数字视频技术有限公司在专利申请号为201010505956.9的申请文件中,公开了一种通过透镜平移法获取多帧低分辨率图像的系统,该系统控制简单,位移精确可调,但驱动电路较复杂,光学设计受到微位移机构限制,通用性较差;第二种是给光路中加入平行平板,通过摆动或旋转平板位置实现多帧图像的获取,北京航空航天大学在申请号201210451785.5的专利申请中,公开了一种在光路中加入平行平板以获取多帧图像的方法,该方法可实现动态场景超分辨率实时成像的效果,这是比较实用的方式。但是其对平行平板的加工精度要求很高,同时光路的改变会导致系统的复杂化和任务量的大幅度加大,因而在实际应用中成本过高,难度过大,且预先设置了采集到同一场景的帧数,限制了分辨率的提高;第三种是通过移动探测器的方式获得多帧图像,清华大学在申请号为200810056002.7的专利申请中,公开了一种通过采用移动探测器的方法来获得多帧图像的装置,通过控制转像机构或使光电探测器阵列微旋转得到一系列有相对微旋转的低分辨率图像。该装置与第一种方式相比,实用性高;与第二种方式相比,因为其没有引入新的光学元件不会导致系统的复杂化,制造成本较低。但是第三种方式中精确控制位移量,对硬件要求高,而且事先设定微旋转角度限定了获取的多帧图像的范围以及采集到同一场景的图像的帧数,会造成重建效果不理想。At present, there are three ways to acquire multi-frame images: the first one is to acquire multi-frame low-resolution images by moving the lens. Shenzhen Yatu Digital Video Technology Co., Ltd. disclosed a system for obtaining multi-frame low-resolution images through lens translation in the application document of patent application number 201010505956.9. It is more complex, and the optical design is limited by the micro-displacement mechanism, so the versatility is poor; the second is to add a parallel plate to the optical path, and obtain multiple frames of images by swinging or rotating the position of the plate, the patent of Beihang University with application number 201210451785.5 In the application, a method of adding parallel plates in the optical path to obtain multi-frame images is disclosed. This method can achieve the effect of super-resolution real-time imaging of dynamic scenes, which is a relatively practical way. However, it requires high processing precision for parallel plates, and at the same time, the change of the optical path will lead to the complexity of the system and a large increase in the workload. Therefore, in practical applications, the cost is too high and the difficulty is too great. The number of frames of the same scene limits the improvement of resolution; the third is to obtain multi-frame images by means of moving detectors. Tsinghua University discloses a method of The method is a device for obtaining multi-frame images, and obtains a series of low-resolution images with relative micro-rotations by controlling the transfer mechanism or micro-rotating the photodetector array. Compared with the first method, the device has high practicability; compared with the second method, because no new optical elements are introduced, the system will not be complicated, and the manufacturing cost is lower. However, the precise control of displacement in the third method requires high hardware requirements, and the pre-set micro-rotation angle limits the range of acquired multi-frame images and the number of frames of images collected from the same scene, resulting in unsatisfactory reconstruction results.
发明内容Contents of the invention
本发明的目的在于克服现有技术存在的通用性差、工艺复杂、重建效果差的缺陷,提供了一种凝视型超分辨率成像装置及方法,通过随机移动探测器以获得多帧图像,用于解决现有成像装置分辨率低的问题。The purpose of the present invention is to overcome the defects of poor versatility, complex process, and poor reconstruction effect in the prior art, and provides a staring super-resolution imaging device and method, which obtains multiple frames of images by randomly moving the detector, and is used for The problem of low resolution of the existing imaging device is solved.
为了实现上述目的,本发明的技术方案包括:In order to achieve the above object, technical solutions of the present invention include:
成像装置镜头组1,用于采集光信号,得到采集到的光信号;The lens group 1 of the imaging device is used to collect the optical signal and obtain the collected optical signal;
探测器2,用于接收采集到的光信号,在其上成像,并将所成图像依次输出给存储单元5、图像处理电路6和图像显示电路7;The detector 2 is used to receive the collected optical signal, form an image on it, and output the formed image to the storage unit 5, the image processing circuit 6 and the image display circuit 7 in sequence;
探测器驱动平台3,用于装载探测器2并带动其移动;The detector driving platform 3 is used to load the detector 2 and drive it to move;
驱动器4,用于带动探测器驱动平台3移动,并在探测器2上成多帧图像;The driver 4 is used to drive the detector to drive the platform 3 to move, and form multiple frames of images on the detector 2;
存储单元5,用于实时存储探测器2移动获得的多帧图像;The storage unit 5 is used to store multiple frames of images obtained by moving the detector 2 in real time;
图像处理电路6,用于重构存储单元5中的多帧图像,得到超分辨率图像;The image processing circuit 6 is used to reconstruct the multi-frame images in the storage unit 5 to obtain super-resolution images;
图像显示电路7,用于输出重构的超分辨率图像;Image display circuit 7, for outputting a reconstructed super-resolution image;
所述探测器2设置于成像装置镜头组1的焦平面上,且固定在探测器驱动平台3面向成像装置镜头组1的一侧;所述驱动器4与探测器驱动平台3之间采用刚性连接,以使两者在工作时的移动保持一致,且移动的大小和方向是随机值。The detector 2 is arranged on the focal plane of the imaging device lens group 1, and is fixed on the side of the detector driving platform 3 facing the imaging device lens group 1; a rigid connection is adopted between the driver 4 and the detector driving platform 3 , so that the movement of the two during work is consistent, and the size and direction of the movement are random values.
所述探测器2采用电荷耦合器件CCD或互补金属氧化物半导体CMOS或电荷注入器件CID。The detector 2 adopts a charge coupled device CCD or a complementary metal oxide semiconductor CMOS or a charge injection device CID.
所述探测器驱动平台3的移动,是在与采集光线垂直的平面内所进行水平方向和竖直方向的随机抖动,且其在水平方向上抖动范围小于探测器宽度尺寸的0.23%,在竖直方向上的抖动范围小于探测器长度尺寸的0.23%。The movement of the detector driving platform 3 is random shaking in the horizontal direction and vertical direction in the plane perpendicular to the collection light, and its shaking range in the horizontal direction is less than 0.23% of the width of the detector. The jitter range in the vertical direction is less than 0.23% of the length of the detector.
所述探测器驱动平台3的随机抖动次数与所述探测器2得到的图像帧数相同。The number of random shakes of the detector driving platform 3 is the same as the number of image frames obtained by the detector 2 .
所述驱动器4采用压电陶瓷驱动器。The driver 4 is a piezoelectric ceramic driver.
实现凝视型超分辨率成像的成像方法,包括如下步骤:The imaging method for realizing staring type super-resolution imaging comprises the following steps:
步骤1:根据所要求的成像效果,设置驱动器驱动部分的变动次数指令;Step 1: According to the required imaging effect, set the change frequency command of the driving part of the driver;
步骤2:利用成像镜头组1采集光信号,得到采集到的光信号;Step 2: Using the imaging lens group 1 to collect optical signals to obtain the collected optical signals;
步骤3:根据设置的变动次数指令,驱动器4驱动探测器2分别在水平和竖直方向上作相应次数的随机抖动,抖动范围小于探测器2长和宽的0.23%,采集到的光信号在探测器2上成像,获得帧数与变动次数相应的多帧低分辨率图像;Step 3: According to the command of the number of changes set, the driver 4 drives the detector 2 to perform random jitters of corresponding times in the horizontal and vertical directions respectively. The jitter range is less than 0.23% of the length and width of the detector 2. The collected optical signal is Imaging on the detector 2 to obtain multi-frame low-resolution images corresponding to the number of frames and the number of changes;
步骤4:利用存储单元5实时存储上述获得的多帧低分辨率图像;Step 4: Utilize the storage unit 5 to store the multi-frame low-resolution images obtained above in real time;
步骤5:利用图像处理电路6中的变分贝叶斯算法,重建存储单元5中的多帧低分辨率图像,得到相应的超分辨率图像;Step 5: Utilize the variational Bayesian algorithm in the image processing circuit 6 to reconstruct the multi-frame low-resolution images in the storage unit 5 to obtain corresponding super-resolution images;
步骤6:利用图像显示电路7输出上述经过计算和重建得到的超分辨率图像。Step 6: Utilize the image display circuit 7 to output the above-mentioned super-resolution image obtained through calculation and reconstruction.
上述步骤五采用的变分贝叶斯算法包括如下步骤:The variational Bayesian algorithm adopted in the fifth step above includes the following steps:
步骤5.1建立重建约束模型来求取后验概率函数P(x/yk):Step 5.1 Establish a reconstruction constraint model to obtain the posterior probability function P(x/y k ):
重建约束模型yk=DHkC(sk)x+nk=Bk(sk)x+nk,其中x表示场景的高分辨率图像,其像素数为PN;yk表示获得的低分辨率图像,其像素数为N;P表示算法对图像空间分辨率的提高;k表示获得的不大于低分辨率图像总帧数的图像帧数;D是N×PN的降采样矩阵,Hk为PN×PN的运动矩阵,C(sk)是运动向量sk产生的PN×PN的运动矩阵,nk是N×1的噪声,成像系统的降采样、模糊和变形效应可以组合成一个N×PN的系统矩阵Bk(sk);每帧低分辨率图像的nk、Hk和sk可以不相同,根据重建约束模型可求出P(x/yk),进而求得其负对数-lnP(yk/x);Reconstruction constraint model y k =DH k C(s k )x+n k =B k (s k )x+ nk , where x represents the high-resolution image of the scene, and its number of pixels is PN; y k represents the obtained Low-resolution image, the number of pixels is N; P represents the improvement of the spatial resolution of the image by the algorithm; k represents the number of image frames obtained that is not greater than the total number of frames of the low-resolution image; D is the downsampling matrix of N×PN, H k is the PN×PN motion matrix, C(s k ) is the PN×PN motion matrix generated by the motion vector s k , n k is the N×1 noise, and the downsampling, blurring and deformation effects of the imaging system can be combined form an N×PN system matrix B k (s k ); nk , H k and s k of each low-resolution image can be different, and P(x/y k ) can be obtained according to the reconstruction constraint model, and then Find its negative logarithm -lnP(y k /x);
步骤5.2建立先验约束模型来求取先验概率密度函数P(x):Step 5.2 Establish a priori constraint model to obtain the prior probability density function P(x):
建立由拉普拉斯金字塔和高斯金字塔的二阶偏导数构成的包含图像空间尺度信息和方向性信息的特征空间,以拉普拉斯金字塔预测为初始估计值,通过加速块匹配方法获得包含图像高频信息的估计梯度先验知识,即求出P(x),进而求得其负对数-lnP(x);Establish a feature space containing image spatial scale information and directional information composed of the second-order partial derivatives of Laplacian pyramid and Gaussian pyramid, and use the prediction of Laplacian pyramid as the initial estimate, and obtain the image containing image by accelerating block matching method The estimated gradient prior knowledge of high-frequency information, that is, to obtain P(x), and then obtain its negative logarithm -lnP(x);
步骤5.3根据贝叶斯最大后验概率估计理论,为了求得最优高分辨率图像估计值x,需要先求得使后验概率P(x/yk)取最大值的x的数值;因为yk为获得的已知低分辨率图像,所以P(yk)为常数,因而高分辨率图像最大后验概率为:
本发明与现有技术相比,具有如下优点:Compared with the prior art, the present invention has the following advantages:
1、本发明采用多帧图像重建的方法得到输出图像,相对普通相机直接成像的方法而言,在使用相同探测器的情况下,可以获得更高分辨率的图像。1. The present invention adopts the method of multi-frame image reconstruction to obtain the output image. Compared with the method of direct imaging with ordinary cameras, images with higher resolution can be obtained under the condition of using the same detector.
2、本发明的成像装置是通过移动探测器的方式以获得多帧图像的,相对于现有技术中通过移动镜头或者在光路中加入光学元件的方式而言,工艺简单、造价低且能达到凝视效果,提高了装置的通用性。2. The imaging device of the present invention obtains multi-frame images by moving the detector. Compared with the method of moving the lens or adding optical elements in the optical path in the prior art, the process is simple, the cost is low, and it can achieve The gaze effect increases the versatility of the device.
3、本发明中探测器抖动位移的大小和方向是随机的,相对现有技术中探测器固定移动而言,可在运动参数未知的情况下,获得超分辨率图像,适用于自身产生随机位移的拍摄装置,如卫星。3. The magnitude and direction of the detector shaking displacement in the present invention are random. Compared with the fixed movement of the detector in the prior art, the super-resolution image can be obtained when the motion parameters are unknown, and it is suitable for generating random displacement by itself. shooting devices, such as satellites.
4、本发明中探测器随机抖动的次数可以通过变动次数指令设定,相对现有技术中探测器的移动次数是一个固定值而言,变动次数越多,得到的低分辨率图像也相应增多,通过后期的重建,最后输出的图像的分辨率就更高。4. The number of random vibrations of the detector in the present invention can be set by the command of the number of changes. Compared with the number of moves of the detector in the prior art is a fixed value, the more the number of changes, the corresponding increase in the obtained low-resolution images , through later reconstruction, the resolution of the final output image is higher.
5、本发明成像方法中,在低分辨率图像重建时应用了变分贝叶斯算法,该算法引入了分布估计,增强了算法对噪声的稳定性,有效地提高了图像分辨率。5. In the imaging method of the present invention, the variational Bayesian algorithm is applied in low-resolution image reconstruction, which introduces distribution estimation, enhances the stability of the algorithm to noise, and effectively improves the image resolution.
附图说明Description of drawings
图1是本发明的结构示意图。Fig. 1 is a structural schematic diagram of the present invention.
图2是本发明中一种简单的探测器位移方式示意图。Fig. 2 is a schematic diagram of a simple detector displacement method in the present invention.
图3是本发明中探测器随机移动四次,位移为半个像素时,可以选择的一种获取图像的示意图。Fig. 3 is a schematic diagram of an optional image acquisition when the detector moves randomly four times and the displacement is half a pixel in the present invention.
图4是本发明中的低分辨率图像为四帧的一种超分辨率重构示意图。Fig. 4 is a schematic diagram of a super-resolution reconstruction in which the low-resolution image is four frames in the present invention.
具体实施方式detailed description
为了使本发明的目的、所解决的技术问题和技术方案更加清晰明了,以下结合附图对本发明作进一步详细描述。In order to make the purpose of the present invention, the technical problem to be solved and the technical solution clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings.
参照图1,本发明包括成像装置镜头组1、探测器2、探测器驱动平台3、驱动器4、存储单元5、图像处理电路6和图像显示电路7;为了保证探测器2能够成功地接收图像并与探测器驱动平台3同步运动,探测器2位于成像装置镜头组1的光路上的且在其焦平面内,该探测器2采用电荷耦合器件CCD或互补金属氧化物半导体CMOS或电荷注入器件CID,固定于探测器驱动平台3之前;驱动器4可采用压电陶瓷驱动器,设置于探测器驱动平台3的一侧,外壳与机体固定连接,保证其在驱动过程中本身不产生相对移动;该驱动器4与探测器平台3刚性连接,如杆连接、焊接或螺栓连接,用以保证探测器平台3和探测器2的同步运动;探测器2与存储单元5、图像处理电路6和图像显示电路7依次形成电路连接,用于传递和处理图像。With reference to Fig. 1, the present invention comprises imaging device lens group 1, detector 2, detector driving platform 3, driver 4, storage unit 5, image processing circuit 6 and image display circuit 7; In order to ensure that detector 2 can successfully receive images And move synchronously with the detector driving platform 3, the detector 2 is located on the optical path of the lens group 1 of the imaging device and in its focal plane, the detector 2 adopts a charge-coupled device CCD or a complementary metal oxide semiconductor CMOS or a charge injection device The CID is fixed before the detector driving platform 3; the driver 4 can be a piezoelectric ceramic driver, which is arranged on one side of the detector driving platform 3, and the shell is fixedly connected with the body to ensure that it does not move relative to itself during the driving process; The driver 4 is rigidly connected to the detector platform 3, such as rod connection, welding or bolt connection, to ensure the synchronous movement of the detector platform 3 and the detector 2; the detector 2 is connected to the storage unit 5, the image processing circuit 6 and the image display circuit 7 in turn form circuit connections for transferring and processing images.
本发明的成像原理是:根据所要求的成像效果,设置驱动器驱动部分的变动次数指令;利用成像镜头组1对准场景目标进行光信号的采集,得到采集到的光信号;成像装置开始工作时,根据设定的变动次数指令,驱动器4的驱动部分促使探测器驱动平台3运动,从而控制探测器2在镜头组的焦平面内的随机抖动,探测器2在水平和垂直方向上的位移分别小于探测器长和宽的0.23%,该位移的大小为随机值;探测器2获得帧数与设定的变动次数相应的低分辨率图像,帧数用L表示,获取到的每帧低分辨率图像实时存储到相机的存储单元5当中,直至存储单元5中存储的图像帧数达到与设定的变动次数指令相应的L帧;针对一个场景拍摄完L帧图像之后将图像数字信息传输到图像处理电路6上对其进行重构,获得超分辨率图像,最后由图像显示电路7对重建的超分辨率图像进行输出显示。The imaging principle of the present invention is: according to the required imaging effect, set the change frequency command of the drive part of the driver; use the imaging lens group 1 to aim at the scene target to collect the optical signal, and obtain the collected optical signal; when the imaging device starts to work , according to the set number of changes, the driving part of the driver 4 drives the detector driving platform 3 to move, thereby controlling the random vibration of the detector 2 in the focal plane of the lens group, and the displacement of the detector 2 in the horizontal and vertical directions respectively Less than 0.23% of the length and width of the detector, the magnitude of the displacement is a random value; detector 2 obtains low-resolution images corresponding to the number of frames and the set number of changes, the number of frames is represented by L, and the obtained low-resolution images of each frame The high-rate image is stored in the middle of the storage unit 5 of the camera in real time, until the number of image frames stored in the storage unit 5 reaches the L frames corresponding to the set number of times of change instruction; after shooting L frames of images for a scene, the image digital information is transmitted to The image processing circuit 6 reconstructs it to obtain a super-resolution image, and finally the image display circuit 7 outputs and displays the reconstructed super-resolution image.
上述图像处理电路6中对L帧图像重建的原理如下:The principle of L frame image reconstruction in the above-mentioned image processing circuit 6 is as follows:
在图像处理电路6中采用的算法为变分贝叶斯算法,变分贝叶斯重建算法的核心,是利用未知概率密度的形式和未知参数的取值范围等先验知识来自训练样本本身的信息,计算后验概率P(yk/x)。The algorithm used in the image processing circuit 6 is the variational Bayesian algorithm. The core of the variational Bayesian reconstruction algorithm is to use the prior knowledge of the form of the unknown probability density and the value range of the unknown parameter from the training sample itself. Information, calculate the posterior probability P(y k /x).
重建约束模型:Rebuild constraint model:
yk=DHkC(sk)x+nk=Bk(sk)x+nk (1)y k =DH k C(s k )x+n k =B k (s k )x+n k (1)
其中D是N×PN的降采样矩阵,Hk为PN×PN的运动矩阵,C(sk)是运动向量sk产生的PN×PN的运动矩阵,nk是N×1的噪声,成像系统的降采样、模糊和变形效应可以组合成一个N×PN的系统矩阵Bk(sk)。每帧低分辨率图像的nk、Hk和sk可以不相同。其中x表示场景的高分辨率图像,其像素数为PN;yk表示获得的低分辨率图像,其像素数为N;P表示算法对图像空间分辨率的提高;Among them, D is the downsampling matrix of N×PN, H k is the motion matrix of PN×PN, C(s k ) is the motion matrix of PN×PN generated by the motion vector s k , n k is the noise of N×1, and the imaging The system's downsampling, blurring and deformation effects can be combined into an N×PN system matrix B k (s k ). nk , H k and s k of each frame of low-resolution images may be different. Among them, x represents the high-resolution image of the scene, and its number of pixels is PN; y k represents the obtained low-resolution image, and its number of pixels is N; P represents the improvement of the spatial resolution of the image by the algorithm;
在已知yk的条件下,x的后验概率可以写成:Under the condition of known y k , the posterior probability of x can be written as:
其中,P(yk/x)是x已知的情况下,观测yk的条件概率密度函数;P(x)和P(yk)分别表示x和yk的先验概率。根据贝叶斯最大后验概率估计理论,为了求得最优的估计超分辨率图像,必须找到使后验概率P(x/yk)取最大值的x。因为yk已知,所以P(yk)是常数,又对数函数是单调递增函数,所以最大后验概率估计可以用公式表示如下:Among them, P(y k /x) is the conditional probability density function of observing y k when x is known; P(x) and P(y k ) represent the prior probability of x and y k respectively. According to Bayesian maximum a posteriori probability estimation theory, in order to obtain the optimal estimated super-resolution image, it is necessary to find the x that maximizes the posterior probability P(x/y k ). Because y k is known, P(y k ) is a constant, and the logarithmic function is a monotonically increasing function, so the maximum a posteriori probability estimate can be expressed as follows:
表示最优估计的超分辨率图像。 Denotes the best estimated super-resolution image.
由公式(3)可知,上述图像处理电路6中对L帧图像重建的步骤如下:As can be seen from formula (3), the steps of L frame image reconstruction in the above-mentioned image processing circuit 6 are as follows:
步骤1:建立重建约束模型来求取-lnP(yk/x),重建约束模型如式(1)所示;Step 1: Establish a reconstruction constraint model to obtain -lnP(y k /x), and the reconstruction constraint model is shown in formula (1);
步骤2:建立先验约束模型来求取-lnP(x)。建立由拉普拉斯金字塔和高斯金字塔的二阶偏导数构成的包含图像空间尺度信息和方向性信息的特征空间,以拉普拉斯金字塔预测为初始估计值,通过加速块匹配方法获得包含着图像高频信息的估计梯度先验知识,即求出-lnP(x);Step 2: Establish a priori constraint model to obtain -lnP(x). A feature space containing image spatial scale information and directional information composed of the second-order partial derivatives of the Laplacian pyramid and the Gaussian pyramid is established, and the Laplacian pyramid prediction is used as the initial estimate, and the accelerated block matching method is used to obtain the feature space containing The estimated gradient prior knowledge of image high-frequency information, that is, to find -lnP(x);
步骤3:将上述步骤1和步骤2所求结果集成到最大后验概率框架中,并使用最速下降法求得最优估计超分辨图像x,即超分辨率图像的最终结果。Step 3: Integrate the results obtained in the above steps 1 and 2 into the maximum a posteriori probability framework, and use the steepest descent method to obtain the optimal estimated super-resolution image x, which is the final result of the super-resolution image.
变分贝叶斯算法把未知的超分辨率图像和运动参数放在同一框架下进行联合估计,通过未知的超分辨率图像和运动参数概率分布的方差,给估计过程引入了一定的不确定性,增强了算法对噪声的稳定性;而且,算法中采用贝叶斯估计方法对未知参数进行分布估计而不是传统算法的点估计,有效抑制了算法中估计误差的扩大,使图像的分辨率得到提高。The variational Bayesian algorithm puts the unknown super-resolution image and motion parameters in the same framework for joint estimation, and introduces certain uncertainty into the estimation process through the variance of the probability distribution of the unknown super-resolution image and motion parameters. , which enhances the stability of the algorithm to noise; moreover, the Bayesian estimation method is used in the algorithm to estimate the distribution of unknown parameters instead of the point estimation of the traditional algorithm, which effectively suppresses the expansion of the estimation error in the algorithm and improves the resolution of the image. improve.
参照图2,是本发明中一种简单的探测器位移方式示意图,驱动器4的驱动部分促使探测器驱动平台3运动,使得处于相机成像镜头组焦平面处的探测器2在焦平面内做随机抖动,假设第一次抖动时探测器2中心点位置坐标为(0,0),获得第一帧图像y1,接下来的运动方式可以分为以下几个步骤:Referring to Fig. 2, it is a schematic diagram of a simple detector displacement mode in the present invention, the driving part of the driver 4 impels the detector driving platform 3 to move, so that the detector 2 at the focal plane of the imaging lens group of the camera is randomly moved in the focal plane. Shaking, assuming that the coordinates of the center point of the detector 2 are (0, 0) during the first shaking, the first frame of image y 1 is obtained, and the next movement method can be divided into the following steps:
1)以第一帧图像为参考,水平移动探测器2距离c2,垂直移动探测器2距离d2,获得第二帧低分辨率图像y2,其中心位置处坐标为(c2,d2),将该图像传输到存储单元5中进行暂时保存。1) Taking the first frame of image as a reference, move the detector 2 horizontally for a distance c 2 , and vertically move the detector 2 for a distance d 2 to obtain the second frame of low-resolution image y 2 . The coordinates of its center are (c 2 , d 2 ), the image is transferred to the storage unit 5 for temporary storage.
2)以第一帧图像为参考,水平移动探测器2距离c3,垂直移动探测器2距离d3,获得第三帧低分辨率图像y3,其中心位置处坐标为(c3,d3),将该图像传输到存储单元5中进行暂时保存。2) Taking the first frame of image as a reference, move the detector 2 horizontally for a distance c 3 and vertically move the detector 2 for a distance d 3 to obtain the third frame of low-resolution image y 3 , and the coordinates of its center are (c 3 , d 3 ), the image is transferred to the storage unit 5 for temporary storage.
3)以此类推,探测器2从第四次到第L次的抖动,可获得L‐3帧图像,结合上述获得的三帧图像,累计获取L帧图像,依次实时存储到存储单元5中。3) By analogy, the detector 2 can obtain L-3 frames of images from the fourth to the L-th shaking, combined with the above-mentioned three frames of images, a total of L frames of images are obtained, which are sequentially stored in the storage unit 5 in real time .
4)将上述步骤中获得并存储在存储单元5里的L帧图像,一起传输到图像处理电路6中。4) The L frames of images obtained in the above steps and stored in the storage unit 5 are transmitted to the image processing circuit 6 together.
为了使以L帧低分辨率图像为数据源的重建算法达到超分辨率的重建效果,需要保证选择的每帧图像之间水平和垂直方向上的位移为亚像素量级,即移动的距离ck和dk不是整像素的大小。通过控制探测器2的随机抖动可以使L帧低分辨率图像分别包含原场景不同部分的信息,但是各低分辨率图像彼此之间存在信息冗余,因此在重叠区域可以通过图像处理电路6进行重建,得到超分辨率图像,最后通过图像显示电路7输出。In order to make the reconstruction algorithm using L frames of low-resolution images as the data source achieve the super-resolution reconstruction effect, it is necessary to ensure that the displacement in the horizontal and vertical directions between the selected images of each frame is at the sub-pixel level, that is, the moving distance c k and d k are not the size of an integer pixel. By controlling the random jitter of the detector 2, the L frames of low-resolution images can respectively contain the information of different parts of the original scene, but there is information redundancy between the low-resolution images, so the overlapping area can be processed by the image processing circuit 6 Reconstruct to obtain a super-resolution image, and finally output it through the image display circuit 7.
参照图3,是本发明中探测器随机移动四次,位移为半个像素时,可能的一种获取图像的示意图,图3(a)表示探测器第一次抖动的位置,图3(b)表示探测器相对第一次抖动的位置向左平移了半个像素,图3(c)表示探测器在图3(b)的基础上向上平移了半个像素,图3(d)表示探测器在图3(c)的基础上向右平移了半个像素。对得到的四帧低分辨率图像进行超分辨率重建过程参照图4,是本发明中的低分辨率图像为四帧的一种超分辨率重构示意图,重建过程中对图4(a)每个低分辨率图像的像素进行重新配准排列,得到图4(b)显示的一帧分辨率显著提高的超分辨率图像。Referring to Fig. 3, it is a schematic diagram of a possible image acquisition when the detector moves four times at random in the present invention when the displacement is half a pixel. Fig. 3(a) shows the position where the detector shakes for the first time, and Fig. 3(b ) means that the detector has shifted half a pixel to the left relative to the position of the first jitter. Figure 3(c) shows that the detector has shifted up by half a pixel on the basis of Figure 3(b), and Figure 3(d) shows that the detection On the basis of Figure 3(c), the device is shifted by half a pixel to the right. Carry out super-resolution reconstruction process to the obtained four-frame low-resolution image with reference to Fig. 4, be the low-resolution image among the present invention is a kind of super-resolution reconstruction schematic diagram of four frames, in the reconstruction process Fig. 4 (a) The pixels of each low-resolution image are re-registered to obtain a super-resolution image with significantly improved resolution as shown in Figure 4(b).
Claims (8)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201410746361.0A CN104410789A (en) | 2014-12-08 | 2014-12-08 | Staring super-resolution imaging device and method |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201410746361.0A CN104410789A (en) | 2014-12-08 | 2014-12-08 | Staring super-resolution imaging device and method |
Publications (1)
Publication Number | Publication Date |
---|---|
CN104410789A true CN104410789A (en) | 2015-03-11 |
Family
ID=52648377
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201410746361.0A Pending CN104410789A (en) | 2014-12-08 | 2014-12-08 | Staring super-resolution imaging device and method |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN104410789A (en) |
Cited By (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104833307A (en) * | 2015-04-14 | 2015-08-12 | 中国电子科技集团公司第三十八研究所 | Method of high-frame-rate motioning object three-dimensional measurement |
CN106525238A (en) * | 2016-10-27 | 2017-03-22 | 中国科学院光电研究院 | Spaceborne multispectral imaging system design method based on super-resolution reconstruction |
CN106960416A (en) * | 2017-03-20 | 2017-07-18 | 武汉大学 | A kind of video satellite compression image super-resolution method of content complexity self adaptation |
CN106982315A (en) * | 2017-04-01 | 2017-07-25 | 中国科学院上海应用物理研究所 | A kind of high-resolution imaging system |
CN107194874A (en) * | 2017-05-26 | 2017-09-22 | 上海微小卫星工程中心 | Super-resolution imaging system and method based on bias image stabilization |
CN110764085A (en) * | 2019-09-29 | 2020-02-07 | 西安电子科技大学 | Variational Bayesian Radar Correlation Imaging Method for Joint Minimum Mean Square Error Estimation |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20050019000A1 (en) * | 2003-06-27 | 2005-01-27 | In-Keon Lim | Method of restoring and reconstructing super-resolution image from low-resolution compressed image |
CN101217625A (en) * | 2008-01-11 | 2008-07-09 | 清华大学 | Apparatus and method for super-resolution imaging |
CN101707670A (en) * | 2009-05-13 | 2010-05-12 | 西安电子科技大学 | Motion random exposure based super-resolution imaging system and method |
CN101980291A (en) * | 2010-11-03 | 2011-02-23 | 天津大学 | Super-resolution image reconstruction method based on random micro-displacement |
CN102006477A (en) * | 2010-11-25 | 2011-04-06 | 中兴通讯股份有限公司 | Image transmission method and system |
US20120105690A1 (en) * | 2010-11-03 | 2012-05-03 | Sony Corporation | Camera system and imaging method using multiple lens and aperture units |
CN102980664A (en) * | 2012-11-12 | 2013-03-20 | 北京航空航天大学 | Superpixel micro-scanning method and corresponding infrared super-resolution real-time imaging device |
-
2014
- 2014-12-08 CN CN201410746361.0A patent/CN104410789A/en active Pending
Patent Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20050019000A1 (en) * | 2003-06-27 | 2005-01-27 | In-Keon Lim | Method of restoring and reconstructing super-resolution image from low-resolution compressed image |
CN101217625A (en) * | 2008-01-11 | 2008-07-09 | 清华大学 | Apparatus and method for super-resolution imaging |
CN101707670A (en) * | 2009-05-13 | 2010-05-12 | 西安电子科技大学 | Motion random exposure based super-resolution imaging system and method |
CN101980291A (en) * | 2010-11-03 | 2011-02-23 | 天津大学 | Super-resolution image reconstruction method based on random micro-displacement |
US20120105690A1 (en) * | 2010-11-03 | 2012-05-03 | Sony Corporation | Camera system and imaging method using multiple lens and aperture units |
CN102006477A (en) * | 2010-11-25 | 2011-04-06 | 中兴通讯股份有限公司 | Image transmission method and system |
CN102980664A (en) * | 2012-11-12 | 2013-03-20 | 北京航空航天大学 | Superpixel micro-scanning method and corresponding infrared super-resolution real-time imaging device |
Cited By (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104833307A (en) * | 2015-04-14 | 2015-08-12 | 中国电子科技集团公司第三十八研究所 | Method of high-frame-rate motioning object three-dimensional measurement |
CN106525238A (en) * | 2016-10-27 | 2017-03-22 | 中国科学院光电研究院 | Spaceborne multispectral imaging system design method based on super-resolution reconstruction |
CN106960416A (en) * | 2017-03-20 | 2017-07-18 | 武汉大学 | A kind of video satellite compression image super-resolution method of content complexity self adaptation |
CN106960416B (en) * | 2017-03-20 | 2019-05-10 | 武汉大学 | A Content Complexity Adaptive Super-Resolution Method for Video Satellite Compressed Images |
CN106982315A (en) * | 2017-04-01 | 2017-07-25 | 中国科学院上海应用物理研究所 | A kind of high-resolution imaging system |
CN107194874A (en) * | 2017-05-26 | 2017-09-22 | 上海微小卫星工程中心 | Super-resolution imaging system and method based on bias image stabilization |
CN110764085A (en) * | 2019-09-29 | 2020-02-07 | 西安电子科技大学 | Variational Bayesian Radar Correlation Imaging Method for Joint Minimum Mean Square Error Estimation |
CN110764085B (en) * | 2019-09-29 | 2023-04-18 | 西安电子科技大学 | Variational Bayes radar correlation imaging method combined with minimum mean square error estimation |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN109087243B (en) | A Video Super-Resolution Generation Method Based on Deep Convolutional Generative Adversarial Networks | |
CN104410789A (en) | Staring super-resolution imaging device and method | |
US12148123B2 (en) | Multi-stage multi-reference bootstrapping for video super-resolution | |
Zhong et al. | Towards rolling shutter correction and deblurring in dynamic scenes | |
CN101710224B (en) | On-orbit self-adaptive focusing method for space optical camera | |
Zhao et al. | Super resolve dynamic scene from continuous spike streams | |
Abuolaim et al. | Improving single-image defocus deblurring: How dual-pixel images help through multi-task learning | |
JP2019535167A (en) | Method for achieving clear and accurate optical zoom mechanism using wide-angle image sensor and long-focus image sensor | |
CN105069748B (en) | A kind of method that high-definition picture is obtained based on microsatellite object space scanning technique | |
Liang et al. | Coherent event guided low-light video enhancement | |
CN106331480A (en) | Video Stabilization Method Based on Image Stitching | |
CN110596885A (en) | Scanning Light Field Imaging System | |
CN112837245A (en) | A dynamic scene deblurring method based on multimodal fusion | |
CN103348667A (en) | Image-capturing device, image-capturing method, and program | |
CN108965736A (en) | A kind of sub-pixel super-resolution imaging method, device, equipment, system and medium | |
CN103595922A (en) | Imaging method based on electronic shutter exposure coding | |
CN103595923A (en) | Double-camera high-speed imaging system and method based on code exposure | |
Xu et al. | Hdrflow: Real-time hdr video reconstruction with large motions | |
CN103985129B (en) | Image motion self-registration method for TDI imaging | |
JP6513941B2 (en) | Image processing method, image processing apparatus and program | |
Xiao et al. | Asymmetric Event-Guided Video Super-Resolution | |
CN112651911A (en) | High dynamic range imaging generation method based on polarization image | |
US20140049661A1 (en) | Higher Resolution Still Image Generation from Lower Resolution Video Camera Frames | |
Lin et al. | Event-guided frame interpolation and dynamic range expansion of single rolling shutter image | |
Zhang et al. | Computational super-resolution imaging with a sparse rotational camera array |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
C06 | Publication | ||
PB01 | Publication | ||
C10 | Entry into substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
WD01 | Invention patent application deemed withdrawn after publication | ||
WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20150311 |