CN104469372B - Method and system for the light field image for compressing microlens array acquisition - Google Patents

Abstract

The present invention provides a kind of method and system for compressing the light field image of microlens array acquisition, the method includes：Several micro- images are decomposited from light field image, wherein the pixel of same position forms in all macro block of pixels of the every micro- image by forming the light field image；Several described micro- images are ranked up and synthesize lossless video file in the order；And the video file is compressed using the method based on Video coding.The present invention is suitable for compressing the light field image obtained by microlens array, can reach higher compression ratio, and have preferable compression performance.

Description

Method and system for compressing light field images acquired by microlens arrays

technical field

The invention relates to a processing technology for light field images, in particular to a method and system for compressing light field images collected by a microlens array.

Background technique

The light field is the sum of light rays in any direction in any position in space, and the light field image can be acquired in the following two ways: through a common camera array or through a light field camera (which has a microlens array). A two-dimensional array with two-dimensional images as elements can be obtained through an ordinary camera array, where each image represents light information under different viewing angles; while a light-field camera obtains a large-scale image with a macro-pixel block as a unit Different pixels in the same macro pixel block represent the light information of the same object point under different viewing angles.

Due to the large size of the light field image, in order to save storage space, the light field image is usually compressed. There are six types of existing light field image compression methods: methods based on vector quantization, methods based on transform, methods based on video coding, methods based on difference compensation, and methods based on multi-focus images. However, these light field image compression methods deal with the two-dimensional matrix with two-dimensional image as the unit collected by the ordinary camera array, and most of them cannot be directly used to compress the light field image collected by the microlens array. Among them, the method based on video coding achieves the purpose of compression by removing the correlation within the same viewing angle and between different viewing angles, and directly arranges the light field images collected by multiple light field cameras into multiple frames of video in a certain order, and then uses Video encoding method for compression. However, the compression ratio and compression quality using this method is low.

With the popularization of light field cameras, how to compress the light field images acquired by the microlens array and achieve a higher compression rate and better compression performance is a problem that needs to be solved at present.

Contents of the invention

In view of the above problems, the present invention provides a method for compressing a light field image collected by a microlens array. The method includes:

Step 1), decomposing a plurality of micro-images from the light field image; wherein, each micro-image is composed of pixels at the same position in all macro pixel blocks forming the light field image;

Step 2), sorting the multiple micro-images and synthesizing them into video files in this order;

Step 3), using a method based on video coding to compress the video file.

In the above method, step 1) includes performing the following steps for each pixel position of all macro pixel blocks forming the light field image:

The pixel points at the position in all macro pixel blocks are taken out to form a micro image, so that the position of the extracted pixel point in the micro image is consistent with the position of the macro pixel block to which the pixel point belongs in the light field image.

In the above-mentioned method, step 2) comprises:

Step 21), sorting the multiple micro-images decomposed from the light field image according to the correlation between the micro-images;

Step 22), according to this sequence, multiple micro-images are synthesized into a video file.

In one embodiment, in step 21), for each pixel position in the macro pixel block, sort clockwise from the center point from the inside to the outside; according to the order of each pixel position, obtain the The sequence of micro-images composed of pixels.

In another embodiment, in step 21), for each pixel position in the macro-pixel block, it is sorted counterclockwise from the center point from inside to outside; according to the order of each pixel position, it is obtained from the position The sequence of micro-images composed of pixels.

In yet another embodiment, in step 21), for each pixel position in the macro pixel block, sort from left to right and from top to bottom starting from the upper left corner; according to the order of each pixel position, Get the sequence of microimages composed of pixels from this location.

In the above method, the video file is a lossless video file.

According to an embodiment of the present invention, there is also provided a method for compressing a light field image collected by a microlens array, comprising:

Step A), for each light field image in a plurality of light field images, decompose a plurality of micro images from the light field image, wherein each micro image consists of the same position in all macro pixel blocks that make up the light field image The pixel composition;

Step B), for each light field image in the plurality of light field images, sort the multiple micro images decomposed from the light field image and synthesize them into a video corresponding to the light field image in this order document;

Step C), connecting multiple video files corresponding to the multiple light field images into one video file;

Step D), using a method based on video coding to compress the video file obtained in the previous step.

In the above method, in step C), the video files corresponding to each light field image are concatenated into one video file according to the time sequence of light field image acquisition.

According to an embodiment of the present invention, there is also provided a system for compressing a light field image collected by a microlens array, including:

A device for decomposing a plurality of micro-images from a light field image; wherein, each micro-image is composed of pixels at the same position in all macro pixel blocks forming the light field image;

A device for sorting the multiple micro-images and synthesizing them into a video file in this order; a device for compressing the video file by adopting a method based on video coding.

The present invention is suitable for compressing the light field image obtained by the microlens array, and by decomposing the light field image into multiple micro images and sorting and merging them, the correlation between the sorted adjacent micro images is relatively large, which is convenient for subsequent use. It is compressed based on the method of video coding. The invention can achieve higher compression ratio and has better compression performance.

Description of drawings

1 is a flowchart of a method for compressing a light field image collected by a microlens array according to an embodiment of the present invention;

Fig. 2 (a) is a light field image collected by microlens array according to one embodiment of the present invention;

Figure 2(b) is the 121 micro-images obtained by decomposing the light field image shown in Figure 2(a);

Fig. 3 is a comparison diagram of the compression effect of compressing multiple light field images collected by the microlens array using the method provided by the present invention and the existing method respectively.

Detailed ways

The present invention will be described below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

According to an embodiment of the present invention, a method for compressing a light field image collected by a microlens array is provided.

Referring to Fig. 1 and generally speaking, the method includes: decomposing the light field image collected by the microlens array to obtain multiple microimages, wherein each microimage is composed of pixels at the same position in all macropixel blocks; will obtain Sorting the multiple micro-images, and synthesizing the micro-images into a video file according to the sequence; and compressing the video file by using a method based on video coding.

Each step of the compression method is described below for a light field image.

Step 1: Decompose the light field image collected by the microlens array to obtain multiple microimages.

As mentioned above, what the microlens array collects is a large-sized image in units of macro-pixel blocks, and a light field image is composed of multiple macro-pixel blocks. For example, for a light field image with a size of 4147x4147 composed of 11x11 (unit: pixel) macropixel blocks, the image includes 377x377 macropixel blocks.

In one embodiment, the implementation process of this step is as follows:

For all the macro pixel blocks forming a light field image, the pixels in the same row and column (ie the same position) in all the macro pixel blocks are taken out to form a small image (ie micro image). In a preferred embodiment, the position of the extracted pixel in the micro image should be consistent with the position of the macro pixel block to which the pixel belongs in the light field image. In other embodiments, the position of the extracted pixel in the micro image may not be consistent with the position of the macro pixel block to which the pixel belongs in the light field image, but the pixel points from the same macro pixel block in different micro images The location needs to be consistent. Using this method, a micro image corresponding to the position of each pixel in the macro pixel block can be obtained.

According to an embodiment of the present invention, for all macro pixel blocks, pixel points may be obtained in order from left to right and from top to bottom. For example, first take out the pixel points at the first row and the first column in each macro pixel block to obtain the micro image corresponding to the first row and the first column of the macro pixel block; then, take out the first row and the first column in each macro pixel block The pixels at the 2nd column are used to obtain the micro image corresponding to the 1st row and 2nd column of the macro pixel block; and so on.

Step 2: Sorting the obtained multiple micro-images, and synthesizing the micro-images into a video file according to the sequence.

This step includes the following two sub-steps:

Step 1. Sorting the multiple micro-images corresponding to the light field image according to the correlation between the micro-images, so that the correlation between adjacent micro-images is relatively large, so as to facilitate subsequent compression using a method based on video coding. Sorting methods can include the following:

(1), rotation sort

Since each micro image corresponds to a pixel position in the macro pixel block (that is, the pixel point is taken from this position), for the convenience of expression, this paper sorts the position of each pixel point in the macro pixel block to obtain the The sequence between the micro-images corresponding to pixel positions.

Rotation sorting refers to sorting the position of each pixel point from the inside to the outside starting from the center pixel of the macro pixel block. For example, starting from the central pixel, traverse all pixel positions in the macro pixel block clockwise or counterclockwise from inside to outside. According to the order of the pixel positions, the order of the micro-images corresponding to each pixel position can be obtained.

(2), horizontal sorting

Horizontal sorting refers to starting from the leftmost end of a row of a macro pixel block, traversing the pixel positions of the row from left to right, and traversing the pixel positions of each row of the macro pixel block in this way.

Preferably, the order of the position of each pixel can be obtained from the upper left corner of the macro pixel block, from left to right, and from top to bottom, so as to obtain the order of the corresponding micro image.

(3), vertical sorting

Vertical sorting refers to starting from the top of a certain column of the macro pixel block, traversing the pixel positions of the column from top to bottom, and traversing the pixel positions of each column of the macro pixel block in this way.

Preferably, the order of the position of each pixel can be obtained from the upper left corner of the macro pixel block, from top to bottom, and from left to right, so as to obtain the order of the corresponding micro image.

(4), custom sorting

It is also possible to arrange the micro-images corresponding to a light field image according to a predefined sequence.

Step 2. After sorting, multiple micro-images corresponding to the light field images are merged in this order as frames in the video to obtain a lossless video file, such as a YUV file.

Step 3: Compress the video file using a method based on video coding.

Those skilled in the art should understand that existing video file compression methods can be used for compression. In one embodiment, the video file is compressed using x264 coding method, and in another embodiment of the present invention, coding method such as x265 is used for coding, so as to obtain a compressed code stream file.

The method for compressing a light field image collected by a microlens array is described above by taking a light field image as an example. In an embodiment where multiple light field images need to be compressed, the video files corresponding to each light field image can be obtained first, and then these video files are synthesized into a total video file, and then the total video file is encoded based on video method to compress. Specifically, the following steps are included:

Step 1: Decompose each light field image in the plurality of light field images to obtain multiple microimages corresponding to each light field image. The process of decomposition has been described above, and will not be repeated here.

Step 2: For each light field image, sort the corresponding multiple micro images, and synthesize the multiple micro images into a video file according to the order, so as to obtain a video file corresponding to each light field image. Wherein, the sorting method may include rotation sorting, horizontal sorting, vertical sorting and the like.

The third step: connect all the obtained video files to synthesize a total video file.

Wherein, these video files can be connected in any order, and in a preferred embodiment, the video files corresponding to each light field image can be connected according to the time sequence of collecting these light field images, and a plurality of light field images can be obtained. The total video file obtained by merging field images.

Step 4: Compress the total video file obtained in the previous step by using a method based on video coding.

Taking 19 light field images of three villains collected by a light field camera as an example, the process of compressing these light field images according to the method provided by the present invention is described in detail, and Fig. 2(a) shows one of them Light field image. Wherein, the format of the light field image is BMP, the size of each light field image is 4147x4147 and the image size is 49.2MB, and the size of the macro pixel block is 11x11.

1. Since each light field image has 377x377 11x11 macro pixel blocks, each light field image can be decomposed into 11x11 micro-images with a size of 377x377.

The specific decomposition process includes:

For 377x377 macropixel blocks of 11x11, take out the pixel points at the i-th row and jth column in each macropixel block (i=1,...11; j=1,...,11), then there are 377x377 pixels in total Points; each pixel in the 377x377 pixels is arranged according to the position of the macro pixel block to which the pixel belongs in the original light field image to form a 377x377 micro image, and the file size of the micro image is 417KB.

Based on the above decomposition method, traversing the macro pixel block from left to right and from top to bottom can obtain 121 (11x11) micro images, each of which corresponds to a position in the macro pixel block. When i is 6 and j is 6, the extracted micro-image is shown in the center image of Fig. 2(b).

2a. For each light field image, the decomposed 121 micro-images are reordered in a clockwise rotation sorting manner, and the rotation sorting mapping table is shown in Table 1. In Table 1, the element in row i and column j indicates that the pixel point is from the micro image in row i and column j of the macro pixel block, and the element value indicates the serial number of the micro image after rotation and sorting. Taking a micro-image with a size of 377x377 composed of pixels from the fourth row and sixth column in the macro pixel block as an example, its sequence number after rotation and sorting is 15. Similarly, taking a micro-image with a size of 377x377 composed of pixels from the fifth row and fifth column in the macro pixel block as an example, the serial number after rotation and sorting is 3.

Table 1

3. Synthesize the micro-images decomposed from each light field image into YUV files according to the arranged order, then each YUV file contains 121 frames of images, and the size of each frame of images is 377x377.

4. Connect the YUV files corresponding to the 19 light field images according to the order of light field image acquisition time to form a YUV file containing 2299 (121x19) frames of images. The size of the YUV file is 487535136 Bytes (464MB).

5. Use the x264 encoder to encode according to the following parameter settings.

The total number of frames participating in encoding: 2299 frames;

Frame rate: 25frames/s;

Quantization factor: 32;

GOPSize: 25;

Number of consecutive B-frames allowed: 1.

The resulting encoding results are as follows:

The compressed code rate is 73028.34kb/s; the PSNR is 29.8885dB; the compressed code stream file size is 6937642Bytes (6.6MB), and the compression rate is: 487535136bytes/6937642Bytes=70.2739.

You can also use another way to change the above step 2a into:

2b. For each light field image, the decomposed 121 micro-images are reordered according to the horizontal sorting method from top to bottom and from left to right. The horizontal sorting mapping table is shown in Table 2. In Table 2, the element in row i and column j indicates that the pixel point is from the micro image in row i and column j of the macro pixel block, and the element value indicates the serial number of the micro image after horizontal sorting. Taking a micro-image with a size of 377x377 composed of pixels from the fourth row and sixth column in the macro pixel block as an example, its serial number after horizontal sorting is 39. Similarly, taking a micro-image with a size of 377x377 composed of pixels from the fifth row and fifth column in the macro pixel block as an example, its sequence number after horizontal sorting is 49.

Table 2

Keeping other steps unchanged, the obtained encoding result is as follows:

The compressed code rate is 80675.54kb/s; the PSNR is 29.8881dB; the compressed code stream file size is 7664165Bytes (6.6MB), and the compression rate is: 487535136bytes/7664165Bytes=63.6123.

According to an embodiment of the present invention, there is also provided a system for compressing a light field image collected by a microlens array, including the following equipment:

A device for decomposing a plurality of micro-images from a light-field image; wherein each micro-image is composed of pixels at the same position in all macro-pixel blocks constituting the light-field image.

A device for sequencing multiple microimages and compositing them into a video file in that order.

A device for compressing video files using methods based on video encoding.

In order to verify the validity of the method and system for compressing the light field image collected by the microlens array provided by the present invention, the inventors carried out experiments using the existing compression method based on video coding and the method provided by the present invention respectively. The material is multiple light field images with slight differences in the same scene collected by the light field camera.

Represent the existing method with method 1, this method directly synthesizes multiple light field images into YUV files according to the time sequence of shooting, and adopts x264 coding technology to compress; Represent the method provided by the present invention with method 2, wherein adopt clockwise Rotation sorting and compression using x264 coding technology; method 3 represents the method provided by the present invention, wherein horizontal sorting from top to bottom and left to right is used and x264 coding technology is used for compression.

Table 3 shows the experimental results, according to the set different quantization coefficients (QP), gives the information in three aspects of bit rate, PSNR and compression rate.

table 3

Fig. 3 shows schematic diagrams of the compression quality obtained by using the above three methods respectively. As can be seen from Table 3 and Fig. 3, compared with the existing compression methods, the method provided by the present invention is to decompose each light field image into multiple micro-images, sort the micro-images and synthesize YUV files, and then Combining the YUV files corresponding to each light field image into a total YUV file for compression can achieve better results in terms of compression rate and compression quality (PSNR-bit rate).

The inventor also conducted experiments on different sorting methods, and the experimental results show that the compression performance of clockwise rotation sorting and counterclockwise rotation sorting is similar, and the compression performance of horizontal sorting and vertical sorting is similar, and compared with the existing technology , the compression ratio and compression performance of these sorting methods are greatly improved.

It should be noted and understood that various modifications and improvements can be made to the invention described in detail above without departing from the spirit and scope of the invention as claimed in the appended claims. Accordingly, the scope of the claimed technical solution is not limited by any particular exemplary teaching given.

Claims (9)

1. A method for compressing a light field image collected by a microlens array, comprising: Step 1), decompose a plurality of micro-images from the light field image collected by the microlens array; wherein, each micro-image is composed of pixels at the same position in all macro pixel blocks forming the light field image, and comes from the same The positions of the pixels of the macro-pixel block in different micro-images are consistent; Step 2), sorting the multiple micro-images and synthesizing them into video files in this order; Step 3), using a method based on video coding to compress the video file. 2. The method according to claim 1, wherein step 2) comprises: Step 21), sorting the multiple micro-images decomposed from the light field image according to the correlation between the micro-images; Step 22), according to this sequence, multiple micro-images are synthesized into a video file. 3. The method according to claim 2, in step 21), For each pixel position in the macro pixel block, it is sorted clockwise from the center point from the inside to the outside; According to the order of the position of each pixel, the order of the micro-image composed of the pixels from this position is obtained. 4. The method according to claim 2, in step 21), For each pixel position in the macro pixel block, it is sorted counterclockwise from the center point from the inside to the outside; According to the order of the position of each pixel, the order of the micro-image composed of the pixels from this position is obtained. 5. The method according to claim 2, in step 21), For each pixel position in the macro pixel block, sort from left to right and from top to bottom starting from the upper left corner; According to the order of the position of each pixel, the order of the micro-image composed of the pixels from this position is obtained. 6. The method of claim 1, wherein the video file is a lossless video file. 7. A method for compressing a light field image collected by a microlens array, comprising: Step A), for each light field image in the multiple light field images collected by the microlens array, decompose multiple micro images from the light field image, wherein each micro image is made up of all the light field images The pixels in the same position in the macro pixel block are composed, and the positions of the pixels from the same macro pixel block in different micro images are consistent; Step B), for each light field image in the plurality of light field images, sort the multiple micro images decomposed from the light field image and synthesize them into a video corresponding to the light field image in this order document; Step C), connecting multiple video files corresponding to the multiple light field images into one video file; Step D), using a method based on video coding to compress the video file obtained in the previous step. 8. The method according to claim 7, wherein, in step C), The video files corresponding to each light field image are connected into one video file according to the acquisition time sequence of the light field image. 9. A system for compressing a light field image captured by a microlens array, comprising: A device for decomposing a plurality of microimages from a light field image collected by a microlens array; wherein each microimage is composed of pixels at the same position in all macropixel blocks constituting the light field image, and comes from the same The positions of the pixels of the macro-pixel block in different micro-images are consistent; A device for ordering said plurality of microimages and compositing them into a video file in that order; A device for compressing said video file using a method based on video coding.