CN103942807A - Real-time processing fast image block fusion system and fusion method - Google Patents

Abstract

The invention discloses a real-time processing fast image block fusion system and fusion method. The real-time processing fast image block fusion system comprises an exposure control module, a camera and other similar gathering tools, an image buffer module, and an image fusion module and so on. The exposure control module controls the camera to output a plurality of exposed images in sequence according to the exposure time through interacting with the camera. The image buffer module is used for storing images shot by the camera, and several images with different exposure values are stored for a scene at each time. The image fusion module is responsible for reading exposed images from a cache and integrating the exposed images. The fusion method mainly optimizes the image through optimizing an image sequence, optimizing entropy calculation and optimizing gaussian weighted fusion, the memory use is greatly reduced, the calculation steps are simplified, the calculation time is shortened, and the algorithm running efficiency is improved.

Description

A kind of rapid image segment fusion system and fusion method of real-time processing

Technical field

The present invention relates to image processing field, relate to more specifically a kind of rapid image segment fusion system and fusion method of real-time processing.

Background technology

Very broad brightness range has been contained in the real world, such as, noon, the brightness of daylight can reach a thousands of times of the moon in late into the night luminance brightness.The brightness range of actual natural scene can be up to 14 orders of magnitude, and ordinary camera can only collect very narrow brightness range, be about 4 orders of magnitude, this is a very large challenge for image acquisition, even if be equipped with automatic exposure module accurately, also inevitably there will be overexposure and under-exposed phenomenon, especially for the dynamic scene of height (HDR).

In order to solve overexposure and under-exposed problem, image co-registration (Image Fusion) technology is introduced into, and obtains all information of high dynamic scene.Image fusion technology refers to, first collects the one group of image that carries out different exposures for Same Scene, then this group image is processed by blending algorithm, is finally combined into the image that a pair almost comprises all information in scene, as shown in Figure 1.

In current image fusion technology, some method can be by obtaining a secondary high dynamic image after one group of image sequence is merged, but such high-dynamics image can not normally show in display, need to carry out height dynamically to low dynamic mapping, can see effect, this process is too complicated, and the effect merging is closely bound up with the mapping in later stage processing.

T.Mertens has been used image pyramid to carry out layering to image; and use the parameter of picture quality as weight, successively merge, because this method has been used image pyramid; so calculated amount is very large, and often there will be the vestige of some artificial treatment.

A.Goshtasby has described a kind of method of segment fusion.In this method, first need image block, and block-by-block calculating weight, again each pixel is weighted and obtains final fused images afterwards.This method can obtain good syncretizing effect, but same calculated amount is larger.

Above two kinds of methods have important using value in the fusion of still image, still, because himself many disadvantages is with not enough, all cannot meet the requirement of the real-time processing that we will realize.Therefore, need to provide a kind of better image processing method.

Summary of the invention

For above the deficiencies in the prior art, the invention provides a kind of rapid image segment fusion system and fusion method of real-time processing.By the optimization to image block blending algorithm, can be intended to significantly shorten the used time of this algorithm, to improve its execution efficiency, can be applied to well real time image processing system.

For solving the problems of the technologies described above, the present invention adopts following technical proposals.

A kind of rapid image segment fusion method of real-time processing comprises the steps:

1. obtain the image I (x, y) for one group of different exposure value of Same Scene.Use very few image, cannot obtain enough dynamic ranges, and use too much image can reduce treatment effeciency.Therefore the present invention is optimized for 3 width left and right by the picture number obtaining.Fig. 2, Fig. 3 illustrate this situation.

2. every image is carried out to piecemeal, every is d x d pixel, and every image is divided into n _r* n _cpiece, wherein, n _r, n _crepresent respectively laterally, block count longitudinally.

3. calculate the entropy of each piece.The entropy computing formula of using in the present invention is:

$E=\underset{i=0}{\overset{255}{\mathrm{\Σ}}}-p_i\log \left(p_i\right)={\log }_2\left(N\right)-\frac{1}{N}\·\mathrm{Cpr}$

Wherein, Cpr operator representation is:

$\mathrm{Cpr}=\underset{i}{\mathrm{\Σ}}{n}_i\·{\log }_2\left(n_i\right)$

Visible, Cpr operator is larger, and entropy E is just less.

In the optimized algorithm proposing in the present invention, by using Cpr operator rather than original entropy E to be used as judgment criteria, can reduce the division outside the computing of Cpr operator, subtraction, logarithm operation, effectively improve operation efficiency.

4. select the piece of entropy maximum in corresponding a certain region in different images as optimum piece.Because previous step has been calculated the Cpr operator of each piece, in this step, only need compare the Cpr operator of the piece on the same position of multiple former figure, choose minimum piece corresponding to Cpr operator wherein, as optimum piece.

5. use and using Rational Gaussian that the center of each image block being selected is axis of symmetry as weighted value, image sequence is carried out to linear weighted function.Fusion function is:

$O(x,y)=\underset{j=1}{\overset{n_r}{\mathrm{\Σ}}}\underset{k=1}{\overset{n_c}{\mathrm{\Σ}}}{W}_{\mathrm{jk}}(x,y)I_{\mathrm{jk}}(x,y)$

In this calculating process, need to carry out (n _r* n _c) inferior multiplication and (n _r* n _c-1) sub-addition.

The present invention improves its computing flow process, rewrites fusion function to be:

$O(x,y)=\underset{i=2}{\overset{n_r}{\mathrm{\Σ}}}\underset{j=2}{\overset{n_c}{\mathrm{\Σ}}}{w}_{\mathrm{ij}}(x,y)\·I_{\mathrm{ij}}(x,y)=\underset{k=2}{\overset{M}{\mathrm{\Σ}}}{I}_k{w}_k,w_k=\underset{{\mathrm{Opt}}_{\mathrm{ij}}\∈I_k}{\mathrm{\Σ}}{w}_{\mathrm{ij}}$

Wherein M represents the picture number merging, and refers to the optimum piece in (i, j) piece position, if represent that optimum piece belongs to k width image, so just this weight is added up.

By this conversion, by first taking advantage of to add afterwards, become the calculating of taking advantage of after now adding, first merge the weighted value that belongs to same sub-picture, do again afterwards multiplication, final calculated amount becomes (n _r* n _c-1) sub-addition and M multiplication, efficiency of algorithm rises suddenly.

6. the image O (x, y) after being merged.

In the process being optimized at the algorithm flow to above, mainly contain the optimization of image sequence, the optimization that entropy calculates, the optimization of Gauss's Weighted Fusion.By the optimization of these three aspects, can significantly reduce the use of internal memory, the step of simplified operation, significantly reduces operation time, significantly improves the efficiency of algorithm operation, make it more appropriate to real-time system.

A rapid image segment fusion system for real-time processing, this system comprises:

Exposure control module, exports many group exposure images for controlling camera successively according to the difference of time shutter;

Image buffers module, gets off for the Image Saving that camera is taken, and stores the image of the different exposure values of several width of a scene at every turn;

Image co-registration module, for image buffers module being read to many group exposure images, and carries out fusion treatment to this exposure image.

Beneficial effect of the present invention is as follows:

The efficiency that piece merges is about 2.5 times of left and right in the past.Higher efficiency can make the real-time system based on this algorithm receive the image data stream of higher frame per second, is applied in the occasion of higher demand, and the effect that image is processed is outstanding.

Accompanying drawing explanation

Below in conjunction with accompanying drawing, the specific embodiment of the present invention is described in further detail.

Fig. 1. (A) (B) is (C) image sequence collecting under different exposure values.

Fig. 1. (D) be the image after the fusion finally obtaining.

Fig. 2. (A) (B) (C) (D) (E) (F) from http://www.imagefusion.org/ website, obtain " an office room " image sequence, 6 width figure altogether, size is 1024x768.Wherein the time shutter from (A) to (F) increases successively.

Fig. 3. be (A) to use Fig. 2. (A) (B) (C) (D) result that (E) (F) this 6 width picture merges.

Fig. 3. be (B) to use Fig. 2. (A) (D) result that (F) this 3 width picture merges.

Fig. 3. be (C) to use Fig. 2. the result that (A) (F) this 2 width picture merges.

Fig. 3. be (D) to use Fig. 2. the result that (B) (E) this 2 width picture merges.

Fig. 4. 3 former figure that (A) (B) (C) selects while being concrete implementation algorithm flow process.

Fig. 4. the optimum piece while (D) being concrete implementation algorithm flow process is chosen result.

Fig. 4. the final fusion results while (E) being concrete implementation algorithm flow process.

Fig. 5 is real time integrating method process flow diagram.

Fig. 6 is the design sketch of real-time image fusion system.

Fig. 7 is the design sketch of real-time image fusion system.

Embodiment

In order to be illustrated more clearly in the present invention, below in conjunction with preferred embodiments and drawings, the present invention is described further.

An object of the present invention is to provide a kind of rapid image segment fusion system of real-time processing.Based on this system, the image interfusion method of above-mentioned optimization has obtained application.

Described image block emerging system comprises exposure control module, the first-class sampling instrument of making a video recording, image buffers module, image co-registration module etc.

Described exposure control module is exported many exposure images for controlling camera successively according to the difference of time shutter.First, by automatic exposure module, obtain optimum exposure parameter, and real-time update.Suppose that the optimum time shutter is Tn (n is frame number), reducing afterwards the time shutter is the image that Tn-t obtains low exposure, again be increased to the image that Tn+t obtains overexposure the time shutter, so just obtained 3 width and merged needed image, three width images are merged and obtain required result.The image after but processing in real time need to constantly be merged, so the present invention completes by the structure of frame iteration.

Described image buffers module is for adopting the mode of " go here and there and change ", and the Image Saving that camera is taken gets off, and stores the image of the different exposure values of several width of a scene at every turn.Mutual by with camera of exposure control module, controls camera successively according to time shutter T0, T0-t, T0+t, T1, T1-t, T1+t ... Tn, Tn-t, the Sequential output of Tn+t, T1 wherein, T2 ... Tn represents respectively the automatic exposure time of constantly updating.Owing to only need to preserving 3 width images, therefore, the time shutter is T0, T1, T2 ... the image of Tn deposits image buffer 1 in, and the time shutter is T0-t, T1-t, T2-t ... the image of Tn-t deposits buffer zone 2 in, T0+t, T1+t, T2+t ... the image of Tn+t deposits buffer zone 3 in, and whenever new collection one sub-picture just covers the image in respective image impact damper, make only to preserve in buffer zone up-to-date image sequence.

More described image co-registration module is for reading image buffers module to organize exposure images, and this exposure image is carried out to fusion treatment.In this real-time system, first image co-registration needs to obtain the image of different exposures, then exposure control module is combined with image co-registration module.Use exposure control module to control exposure parameter, obtain the image under different exposure values.

In real-time image fusion system, every renewal piece image just merges once, so the image sequence merging not is under same automatic exposure value, may there is the combination of three kinds of different automatic exposure values: (Tn, Tn-t, Tn+t), or (Tn+1, Tn-t, Tn+t), (Tn+1, Tn+1-t, Tn+t), Tn represents n automatic exposure value constantly, and Tn+1 is n+1 exposure value constantly.First group of three width image for unified automatic exposure value, latter two is the combination of different automatic exposure values, but because the gap of scene brightness between twice automatic exposure test is very little, so can not bring to system unstable.

In test practice, adopt the camera of 30 frames per second to carry out data acquisition, resolution is 640X480, because fusion ratio is more consuming time, per secondly can only carry out 30 times and merge, so image acquisition and image co-registration are divided in two threads and carried out, image acquisition is only responsible for providing endlessly the images of different exposures, and image co-registration part is only responsible for 3 width images to merge, can guarantee like this frame frequency of real time fusion system.

The operational effect of real-time system proposed by the invention as shown in Figure 6, Figure 7.

As shown in Figure 5, another object of the present invention is to provide a kind of optimization method based on image block blending algorithm, has effectively improved the efficiency of algorithm of image co-registration, and its method flow comprises:

1. obtain the image for the different exposure values of 3 width of Same Scene.As Fig. 4. (A) (B) 3 former figure as shown in (C).

2. these 3 images are carried out to piecemeal.For example, every is 160x160 pixel, is divided into 4x3 piece.

3. calculate 3 images totally 36 pieces entropy separately.In concrete computation process, only calculate Cpr operator:

$\mathrm{Cpr}=\underset{i}{\mathrm{\Σ}}{n}_i\·{\log }_2\left(n_i\right)$

N wherein _iin a gray level for gray level image, the number of times that the pixel that gray scale is i occurs.According to the relation of Cpr operator and entropy E, can obtain, when the value of Cpr operator is less, entropy E is just larger.

The Cpr operator of 4. calculating according to previous step, image block on the same position of 3 images is contrasted, select the piece of Cpr operator minimum in 3 image blocks as optimum piece, this selects piece process to carry out 12 times, and optimum is chosen result soon as Fig. 4. (D).

5. use and using Rational Gaussian that the center of each image block being selected is axis of symmetry as weighted value, image sequence is carried out to linear weighted function.In optimized algorithm, weighting formula is:

$O(x,y)=\underset{k=1}{\overset{M}{\mathrm{\Σ}}}{I}_k{w}_k,w_k=\underset{{\mathrm{Opt}}_{\mathrm{ij}}\∈I_k}{\mathrm{\Σ}}{w}_{\mathrm{ij}}$

In concrete operation process, first merge the weighted value that belongs to same sub-picture, do afterwards multiplication again, final calculated amount becomes (4x3 – 1=11) sub-addition and 3 multiplication.

6. the image O (x, y) after being merged.As Fig. 4. (E).

On the basis of above optimized algorithm, the present invention proposes a kind of real-time system of image co-registration, its system chart is as shown in Figure 5.

By the optimization to blending algorithm, obtained Fast Block fusion method.Several groups of image sequences are tested, compared the processing speed between Fast Block blending algorithm and former algorithm, data are relatively in Table 1.

Table 1 Fast Block blending algorithm and former algorithm speed comparison

From test data, can find out, through optimizing, the efficiency that piece merges is about 2.5 times of left and right in the past.Higher efficiency can make the real-time system based on this algorithm receive the image data stream of higher frame per second, is applied in the occasion of higher demand, and the effect that image is processed is outstanding.

Obviously; the above embodiment of the present invention is only for example of the present invention is clearly described; and be not the restriction to embodiments of the present invention; for those of ordinary skill in the field; can also make other changes in different forms on the basis of the above description; here cannot give all embodiments exhaustive, every still row in protection scope of the present invention of apparent variation that technical scheme of the present invention extends out or change that belong to.

Claims (4)

1. a rapid image segment fusion method of processing in real time, is characterized in that, this fusion method comprises the steps:

1) obtain the image I (x, y) for one group of different exposure value of Same Scene;

2) every image is carried out to piecemeal, every is d x d pixel, and every image is divided into n _r* n _cpiece, wherein, n _r, n _crepresent respectively laterally, block count longitudinally;

3) calculate respectively entropy and the Cpr value of each piece, described entropy is:

$E=\underset{i=0}{\overset{255}{\mathrm{\Σ}}}-p_i\log \left(p_i\right)={\log }_2\left(N\right)-\frac{1}{N}\·\mathrm{Cpr}$

Wherein, p _irefer to that pixel value is the number n of the pixel of i _iaccount for the ratio of total pixel N, and Cpr operator representation is:

$\mathrm{Cpr}=\underset{i}{\mathrm{\Σ}}{n}_i\·{\log }_2\left(n_i\right);$

4) piece of selecting entropy maximum is as optimum piece;

5) use and to using Rational Gaussian that each optimum image Kuai center being selected is axis of symmetry as weighted value, image sequence is carried out to linear weighted function, merge, the image O (x, y) after being merged,

$O(x,y)=\underset{i=2}{\overset{n_r}{\mathrm{\Σ}}}\underset{j=2}{\overset{n_c}{\mathrm{\Σ}}}{w}_{\mathrm{ij}}(x,y)\·I_{\mathrm{ij}}(x,y)=\underset{k=2}{\overset{M}{\mathrm{\Σ}}}{I}_k{w}_k,w_k=\underset{{\mathrm{Opt}}_{\mathrm{ij}}\∈I_k}{\mathrm{\Σ}}{w}_{\mathrm{ij}}$

Wherein M represents the picture number merging, and refers to the optimum piece in (i, j) piece position, if optimum piece belongs to k width image, so just this weight is added up.

2. a rapid image segment fusion system of processing in real time, is characterized in that, this system comprises:

Exposure control module, exports many group exposure images for controlling camera successively according to the difference of time shutter;

Image buffers module, gets off for the Image Saving that camera is taken, and stores the image of the different exposure values of several width of a scene at every turn;

Image co-registration module, for image buffers module being read to many group exposure images, and carries out fusion treatment to this exposure image.

3. the rapid image segment fusion system of a kind of real-time processing according to claim 2, is characterized in that, described many group exposure images are Tn frame, Tn-t frame, Tn+t frame, wherein: the group number that n is image, t is that the time shutter is poor, and Tn+t, Tn, Tn-t are 3 different time shutter.

4. the rapid image segment fusion system of a kind of real-time processing according to claim 2, is characterized in that, the method that described image co-registration is processed is:

1) obtain the image I (x, y) for one group of different exposure value of Same Scene;

2) every image is carried out to piecemeal, every is d x d pixel, and every image is divided into n _r* n _cpiece, wherein, n _r, n _crepresent respectively laterally, block count longitudinally;

3) calculate respectively entropy and the Cpr value of each piece, described entropy is:

$E=\underset{i=0}{\overset{255}{\mathrm{\Σ}}}-p_i\log \left(p_i\right)={\log }_2\left(N\right)-\frac{1}{N}\·\mathrm{Cpr}$

Wherein Cpr operator representation is:

$\mathrm{Cpr}=\underset{i}{\mathrm{\Σ}}{n}_i\·{\log }_2\left(n_i\right);$

4) piece of entropy maximum of selecting corresponding a certain region in different images is as optimum piece;

5) use and to using Rational Gaussian that the center of each image block being selected is axis of symmetry as weighted value, image sequence is carried out to linear weighted function, merge, the image O (x, y) after being merged,

$O(x,y)=\underset{i=2}{\overset{n_r}{\mathrm{\Σ}}}\underset{j=2}{\overset{n_c}{\mathrm{\Σ}}}{w}_{\mathrm{ij}}(x,y)\·I_{\mathrm{ij}}(x,y)=\underset{k=1}{\overset{M}{\mathrm{\Σ}}}{I}_k{w}_k,w_k=\underset{{\mathrm{Opt}}_{\mathrm{ij}}\∈I_k}{\mathrm{\Σ}}{w}_{\mathrm{ij}}$

Wherein M represents the picture number merging, and refers to the optimum piece in (i, j) piece position, if optimum piece belongs to k width image, so just this weight is added up.