CN117853351A

CN117853351A - Shooting fusion method and device based on camera array

Info

Publication number: CN117853351A
Application number: CN202311445420.6A
Authority: CN
Inventors: 何觉清; 刘瑜; 李东荣
Original assignee: Guangzhou Lijiahe Electronic Technology Co ltd
Current assignee: Guangzhou Lijiahe Electronic Technology Co ltd
Priority date: 2023-11-01
Filing date: 2023-11-01
Publication date: 2024-04-09

Abstract

The invention relates to the technical field of picture processing, and discloses a shooting fusion method and device based on a camera array, wherein the shooting fusion method comprises the following steps of: the method comprises the steps of simultaneously shooting an object to be shot by using a plurality of cameras to obtain a preferred shooting set, selecting a basic shooting image from the preferred shooting set, setting other preferred shooting images as weight shooting images, obtaining a basic correction shooting image and a weight correction shooting image through correction, calculating local standard deviations of pixels of the basic correction shooting image, dividing the basic correction shooting image into local fusion areas and global fusion areas based on the local standard deviations of the pixels, fusing pixel information of the weight correction shooting image in the local fusion areas to obtain local fused areas, processing the global fusion areas by taking the weight correction shooting image as parameters of histogram equalization to obtain global fused areas, and combining the local fused areas and the global fused areas to obtain imaging images of shooting equipment. The invention mainly aims to complete high-quality imaging on the premise of occupying less calculation resources.

Description

Shooting fusion method and device based on camera array

Technical Field

The invention relates to a shooting fusion method and device based on a camera array, and belongs to the technical field of picture processing.

Background

The photos can be used to convey information, stories, and emotions. High quality pictures more clearly convey this information, whether for artistic, scientific, or personal purposes, they help convey information, convey emotion, and create visual appeal. At present, the high-quality imaging is not difficult to finish based on the improvement of deep learning and computational power. However, there is still a problem that the deep learning needs to occupy too much computing resources, and it is not easy to improve the computing power of each photographing device, so how to complete high-quality imaging on the premise of occupying less computing resources is a technical problem that needs to be solved urgently.

Disclosure of Invention

The invention provides a photographing fusion method, a photographing fusion device and a computer-readable storage medium based on a camera array, and mainly aims to complete high-quality imaging on the premise of occupying fewer computing resources.

In order to achieve the above object, the present invention provides a photographing fusion method based on a camera array, including:

receiving a photographing instruction, and connecting photographing equipment according to the photographing instruction, wherein the photographing equipment consists of a plurality of cameras which are distributed in a matrix form and are all positioned in the same plane;

Confirming a focal length value of photographing equipment, and simultaneously photographing an object to be photographed by utilizing a plurality of cameras according to the focal length value to obtain a plurality of photographing pictures, wherein the number of the photographing pictures is a multiple of the number of the cameras;

according to different shooting time, performing time classification on the plurality of shooting pictures to obtain a plurality of groups of time shooting sets, wherein the number of the time shooting pictures in each group of time shooting sets is the same as that of cameras;

extracting a specified number of time shooting blocks with n x n size from each time shooting image in each time shooting set;

calculating pixel information entropy and pixel gradient values of each time shooting block;

selecting one of the time shooting sets from the plurality of groups of time shooting sets according to the pixel information entropy and the pixel gradient value to obtain a preferable shooting set;

selecting a basic correction shooting picture from the preferred shooting set, and setting other preferred shooting pictures as weight correction shooting pictures;

sequentially extracting basic pixel points from the basic shooting diagram, identifying basic pixel points and pixel values of the basic pixel points, and selecting pixel points to be weighted from the weight shooting diagram according to the basic pixel points;

acquiring shooting distance parameters of the basic pixel points, and judging whether related weight pixel points corresponding to the basic pixel points exist in the weight shooting diagram according to the shooting distance parameters and preset camera array parameters;

If the related weight pixel point corresponding to the basic pixel point does not exist in the weight shooting diagram, setting the pixel value of the pixel point with the weight to be determined as the pixel value of the basic pixel point;

if the related weight pixel points corresponding to the basic pixel points exist in the weight shooting image, acquiring pixel values of the related weight pixel points, setting the pixel values of the to-be-determined weight pixel points as the pixel values of the related weight pixel points until the basic pixel points are extracted, and obtaining a basic correction shooting image and a weight correction shooting image;

calculating a pixel local standard deviation of the basic correction shooting image, and dividing the basic correction shooting image into a local fusion area and a global fusion area based on the pixel local standard deviation;

the pixel information of the weight correction shooting image is fused into the local fusion area to obtain a local fused area, and the weight correction shooting image is used as a parameter for histogram equalization to process the global fusion area to obtain the global fused area;

and merging the local fused area and the global fused area to obtain an imaging diagram of the photographing device.

Optionally, the extracting a specified number of time shooting blocks with n×n sizes in each time shooting chart in each time shooting set includes:

Acquiring an image specification of each time shooting picture, wherein the image specification is M;

confirming a photographing block specification of a time photographing block based on the image specification,wherein the shooting block specification is n, and the value of n is smaller than or equal to

Confirming the appointed number of time shooting blocks with the size of n x n, wherein the appointed number is required to be more than or equal to 5;

and selecting time shooting blocks with the size of n x n at the designated positions of the time shooting diagram according to the designated number, wherein the designated positions at least comprise an upper left corner, a lower left corner, an upper right corner, a lower right corner and a middle part.

Optionally, the calculating the pixel information entropy and the pixel gradient value of each time shooting block includes:

counting the pixel sum of each time shooting block to obtain a pixel total value;

and calculating according to the pixel total value to obtain a pixel information entropy, wherein the calculation of the pixel information entropy comprises the following steps:

calculating to obtain pixel information entropy according to the following formula:

wherein E is _i,j,s Entropy of pixel information representing the s-th time shooting block of the j-th time shooting diagram in the i-th time shooting set, p _s Representing each pixel value, p, in the s-th temporal shooting block _u A pixel value representing a u-th pixel in the s-th time photographing block;

generating 2 groups of gradient matrixes, wherein the scales of the 2 groups of gradient matrixes are 3*3, and the concrete steps are as follows:

And performing convolution operation by using the 2 groups of gradient matrixes and the time shooting block to obtain pixel gradient values.

Optionally, the performing a convolution operation with the time photographing block using 2 sets of gradient matrices to obtain pixel gradient values includes:

according to the principle from left to right, performing convolution operation on the 2 groups of gradient matrixes and the time shooting block respectively to obtain a convolution matrix, wherein the convolution matrix is as follows:

wherein M is _s，t Convolution pixels representing the t-th convolution operation performed by the s-th time capturing block of the j-th time capturing image in the I-th group of time capturing sets, |represents an absolute value symbol, I _3*3 A 3*3-dimensional matrix block in the s-th time shooting block;

summing the plurality of convolution matrices to obtain pixel gradient values, wherein the pixel gradient values are calculated as follows:

wherein T is _i，j，s The pixel gradient value representing the s-th time photographing block, U is the number of matrix pixels of the s-th time photographing block.

Optionally, selecting one of the plurality of sets of time shooting sets according to the pixel information entropy and the pixel gradient value to obtain a preferred shooting set includes:

calculating an image pixel deviation value for each time-captured image according to the following equation:

wherein P is _i，j Representing the image pixel deviation value of the jth time shooting picture in the ith group of time shooting sets, wherein C is the appointed number of time shooting blocks included in the jth time shooting picture;

Sequentially calculating the difference value of the image pixel deviation values among 2 groups of time shooting pictures in each group of time shooting sets to obtain an image pixel group difference value;

judging whether 2 groups of time shooting pictures with the difference value of the image pixel groups being larger than the image pixel threshold value exist or not, and if 2 groups of time shooting pictures with the difference value of the image pixel groups being larger than the image pixel threshold value exist, eliminating the corresponding time shooting set;

and if 2 groups of time shooting pictures with the image pixel group differences larger than the image pixel threshold value do not exist, screening a time shooting set corresponding to the minimum average value of the image pixel group differences, and confirming the time shooting set as a preferable shooting set.

Optionally, the selecting a basic correction shot map from the preferred shot set and setting other preferred shot maps as weight correction shot maps includes:

acquiring an image pixel group difference value of each preferable shooting picture and other Zhang Youxuan shooting pictures in the preferable shooting set to obtain a pixel group difference value set;

calculating a pixel group difference distribution coefficient of each preferable shooting image according to the pixel group difference set, wherein the pixel group difference distribution coefficient consists of an average value of pixel group differences, a median value of the pixel group differences and a maximum value and a minimum value of the pixel group differences;

Performing average value calculation on the average value, the median value, the maximum value and the minimum value of the pixel group difference values to obtain difference value coefficient values;

the preferred shot map with the smallest difference coefficient value is selected as the base correction shot map, and the other preferred shot maps are set as the weight correction shot map.

Optionally, the calculating the pixel local standard deviation of the basic correction shooting image divides the basic correction shooting image into a local fusion area and a global fusion area based on the pixel local standard deviation, and the calculating includes:

performing block segmentation on the basic correction shooting image to generate a plurality of local area blocks with f, wherein f is required to be smaller than or equal to n;

the following is performed for each local area block:

acquiring adjacent local area blocks of the local area blocks to obtain one or more adjacent area blocks;

calculating pixel local standard deviation of the local area block and the adjacent area block;

if the pixel local standard deviation is greater than or equal to the pixel local threshold deviation, performing block fusion operation on the local area block with the pixel local standard deviation greater than or equal to the pixel local threshold deviation and the adjacent area block to obtain a fusion area block;

and (3) until all the local area blocks complete the calculation, judgment and fusion operation of the local standard deviation of the pixels, confirming all the fusion area blocks as local fusion areas, and confirming all the local area blocks and adjacent area blocks which do not execute the fusion operation as global fusion areas.

Optionally, the calculating the pixel local standard deviation of the local area block and the adjacent area block includes:

extracting edge pixels of a local area block and an adjacent area block with f x f respectively to obtain a local edge pixel set and an adjacent edge pixel set, wherein the local edge pixel set and the adjacent edge pixel set respectively comprise 4 sides, and the number of pixels of each side is f;

confirming edge pixels overlapped in the local edge pixel set and the adjacent edge pixel set to obtain an overlapped edge pixel set;

calculating pixel difference values between the overlapped edge pixels in the overlapped edge pixel set to obtain the overlapped edge difference values;

judging the size relation between the overlapping edge difference value and a preset overlapping edge threshold value, and if the overlapping edge difference value is larger than or equal to the overlapping edge threshold value, directly setting the local standard deviation of the pixel to infinity;

if the difference value of the coincident edges is smaller than the threshold value of the coincident edges, calculating the pixel difference value of the pixel set of the coincident edges and other 7 edges, and summing and dividing by 7 to obtain the local standard deviation of the pixels.

Optionally, the merging the pixel information of the weight correction shooting image into the local merging area to obtain a local merged area includes:

starting a lightweight fusion model which is trained in advance, wherein the lightweight fusion model consists of a convolutional neural network, and the number of network layers is not more than 9;

Extracting a local weight region of a region corresponding to the local fusion region from each weight correction shooting image, and taking the local weight region and the local fusion region as input data of a lightweight fusion model after training;

and (5) running the trained lightweight fusion model to obtain the local fused region.

In order to solve the above problems, the present invention further provides a photographing fusion device based on a camera array, the device comprising:

the object shooting module is used for receiving shooting instructions, and is connected with shooting equipment according to the shooting instructions, wherein the shooting equipment consists of a plurality of cameras which are distributed in a matrix form and are all located in the same plane, confirming focal length values of the shooting equipment, and shooting an object to be shot by the plurality of cameras according to the focal length values to obtain a plurality of shooting pictures, wherein the number of the shooting pictures is a multiple of the number of the cameras;

the gradient value calculation module is used for performing time classification on the plurality of shooting pictures according to different shooting time to obtain a plurality of groups of time shooting sets, wherein the number of the time shooting pictures in each group of time shooting sets is the same as that of cameras, and in each time shooting picture in each group of time shooting sets, a specified number of time shooting blocks with the size of n x n are extracted, and the pixel information entropy and the pixel gradient value of each time shooting block are calculated;

The fusion zone division module is used for selecting one group of time shooting sets from a plurality of groups of time shooting sets according to pixel information entropy and pixel gradient values to obtain a preferred shooting set, selecting a basic shooting image from the preferred shooting set, setting other preferred shooting images as weight shooting images, sequentially extracting basic pixel points from the basic shooting images, identifying basic pixel points and pixel values of the basic pixel points, selecting a pixel point to be weighted from the weight shooting images according to the basic pixel points, acquiring shooting distance parameters of the basic pixel points, judging whether related weight pixel points corresponding to the basic pixel points exist in the weight shooting images according to the shooting distance parameters and preset camera array parameters, setting the pixel values of the weight pixel points as pixel values of the basic pixel points if the related weight pixel points corresponding to the basic pixel points do not exist in the weight shooting images, acquiring the weight pixel points according to the basic pixel points in the weight shooting images, obtaining the weight pixel points to be weighted according to the related weight pixel points, and obtaining a local correction zone, and carrying out global correction on the basis of the local correction zone, and obtaining a local correction zone;

The image fusion module is used for fusing the pixel information of the weight correction shooting image in the local fusion area to obtain a local fused area, processing the global fusion area by taking the weight correction shooting image as a parameter for histogram equalization to obtain a global fused area, and combining the local fused area and the global fused area to obtain an imaging image of the photographing device.

In order to solve the above-mentioned problems, the present invention also provides an electronic apparatus including:

at least one processor; the method comprises the steps of,

a memory communicatively coupled to the at least one processor; wherein,

the memory stores instructions executable by the at least one processor to implement the camera array based photo fusion method described above.

In order to solve the above-mentioned problems, the present invention further provides a computer readable storage medium, in which at least one instruction is stored, the at least one instruction being executed by a processor in an electronic device to implement the camera array-based photographing fusion method described above.

Compared with the prior art, the method and the device have the advantages that the photographing instruction is received firstly, the photographing device is connected according to the photographing instruction, the photographing device is composed of the cameras, the cameras are distributed in a matrix form and are located in the same plane, the focal length value of the photographing device is confirmed, the objects to be photographed are photographed simultaneously according to the focal length value, and a plurality of photographing pictures are obtained, wherein the number of the photographing pictures is a multiple of the number of the cameras. The pressure of each camera for deep learning calculation is relieved, because the image processing process can be dispersed to a plurality of cameras instead of one camera, and a plurality of pictures obtained by shooting can be used for subsequent picture fusion operation, so that the imaging quality of the pictures is improved; further, according to different shooting time, performing time classification on the plurality of shooting images to obtain a plurality of groups of time shooting sets, wherein the number of the time shooting images of each group of time shooting sets is the same as the number of cameras, in each time shooting image of each group of time shooting sets, a specified number of time shooting blocks with the size of n x n are extracted, pixel information entropy and pixel gradient values of each time shooting block are calculated, and it is emphasized that image fusion is performed according to the pixel information entropy and the pixel gradient values, which is beneficial to effectively improving image quality in local areas and global areas, which means that deep learning processing is not needed on the whole image, and only optimization is needed for specific areas, so that the phenomenon of computing resource waste caused by deep learning in a large range is avoided; and finally, selecting one of the time shooting sets from the plurality of groups of time shooting sets according to pixel information entropy and pixel gradient values to obtain a preferred shooting set, selecting one basic correction shooting image from the preferred shooting set, setting other preferred shooting images as weight correction shooting images, calculating pixel local standard deviations of the basic correction shooting images, dividing the basic correction shooting images into local fusion areas and global fusion areas based on the pixel local standard deviations, fusing pixel information of the weight correction shooting images in the local fusion areas to obtain local fused areas, processing the global fusion areas by taking the weight correction shooting images as parameters of histogram equalization to obtain global fused areas, combining the local fused areas and the global fused areas to obtain imaging images of photographing equipment, and finally combining the processed local and global images to obtain high-quality imaging images. This merging process is based on efficient image processing and fusion techniques, rather than deep learning, and it is seen that excessive computational burden on the photographing apparatus is avoided. Therefore, the shooting fusion method, the shooting fusion device, the electronic equipment and the computer readable storage medium based on the camera array provided by the invention mainly aim to finish high-quality imaging on the premise of occupying less computing resources.

Drawings

Fig. 1 is a schematic flow chart of a photographing fusion method based on a camera array according to an embodiment of the present invention;

FIG. 2 is a functional block diagram of a camera-based fusion device according to an embodiment of the present invention;

fig. 3 is a schematic structural diagram of an electronic device for implementing the photographing fusion method based on a camera array according to an embodiment of the present invention.

The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.

Detailed Description

It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.

The embodiment of the application provides a photographing fusion method based on a camera array. The execution subject of the camera array-based photographing fusion method includes, but is not limited to, at least one of a server, a terminal, and the like, which can be configured to execute the method provided by the embodiment of the application. In other words, the camera array-based photographing fusion method may be performed by software or hardware installed in a terminal device or a server device. The service end includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, and the like.

Example 1:

referring to fig. 1, a flowchart of a camera-based fusion method according to an embodiment of the present invention is shown. In this embodiment, the photographing fusion method based on the camera array includes:

s1, receiving a photographing instruction, and connecting photographing equipment according to the photographing instruction, wherein the photographing equipment is composed of a plurality of cameras which are arranged in a matrix form and are all located in the same plane.

It should be explained that the photographing application scene of the embodiment of the invention is various, and an exemplary small piece is a plant scholars, and at present, a plurality of primary protection plants with excellent growth vigor are discovered in the field, so that the primary protection plants are planned to be photographed by photographing equipment. It can be understood that the first-class protected plants are of species of national protection, so that the pictures taken by the photographing device need to have high definition and no chromatic aberration, thereby facilitating subsequent researches.

Compared with the traditional photographing equipment, the embodiment of the invention mainly relies on the photographing equipment consisting of a plurality of cameras, and the cameras are distributed in a matrix form and all located in the same plane. The main purpose of the camera is that a plurality of cameras can simultaneously image, and the imaging error of single shooting due to focusing, light and other reasons can be prevented to the greatest extent.

S2, confirming a focal length value of photographing equipment, and simultaneously photographing an object to be photographed by using a plurality of cameras according to the focal length value to obtain a plurality of photographing graphs, wherein the number of the photographing graphs is a multiple of the number of the cameras.

It will be appreciated that the focus value is generally confirmed by the user. The display in the small Zhang Dianji photographing device is used for adjusting the focal length value through the prompt of the display, and the first-stage protection plants are photographed through the cameras at the same time after the focal length value is adjusted, so that a plurality of photographing pictures of the first-stage protection plants are obtained.

In addition, in order to ensure the imaging quality of each photographing, the number of photographing times is set to be at least 2, that is, if the number of cameras of a small-sized photographing device is 5, the 5 cameras simultaneously photograph at least 1, the obtained photographing images are also 10, and, supposedly, if the number of 5 cameras simultaneously photographs 3, 15 photographing images are obtained in total, that is, the number of photographing images is 3 times that of the cameras, and it is emphasized that, in order to ensure the maximum reduction of photographed objects, the number of photographing times set in the embodiment of the invention is at least 2.

S3, performing time classification on the plurality of shooting images according to different shooting times to obtain a plurality of groups of time shooting sets, wherein the number of the time shooting images of each group of time shooting sets is the same as that of cameras.

It can be understood that if the small-sized photographing device photographs the first-stage protected plants 3 times, each camera represents 15 photographing graphs generated in total, so that the small-sized photographing device can be divided into 3 time photographing sets, and each time photographing set is 5 time photographing graphs.

S4, extracting a specified number of time shooting blocks with the size of n x n from each time shooting image in each group of time shooting sets.

It should be explained that, since each camera is in the same plane and is very close to each other, the shooting environments are basically consistent, so that it is known from theory that the image obtained by each camera does not have too much different pixel differences, if there are too much different pixel differences, it is indicated that some cameras are interfered at the moment of imaging or the cameras are damaged, so that in order to ensure the imaging quality, the embodiment of the invention needs to correct the pixel errors of each group of time shooting sets first, wherein the pixel errors are represented by the pixel information entropy and the pixel gradient value, and the dicing operation needs to be performed on each time shooting image before calculating the pixel information entropy and the pixel gradient value. In detail, the extracting a specified number of time shooting blocks with n×n sizes in each time shooting chart in each time shooting set includes:

confirming a shooting block specification of the time shooting block based on the image specification, wherein the shooting block specification is n x n, and the value of n is smaller than or equal to

It should be understood that if the pixels of the whole time shooting image are directly used as the basis for calculating the pixel information entropy and the pixel gradient value, the calculation amount is too large, and the calculation load is brought to the shooting equipment.

For example, if each camera of the small-sized photographing apparatus generates 720×720 photographing images, M is 720, and n is not greater thanIt can be confirmed that the n value is 50, which means that a time photographing block having a size of 50×50 is selected from a plurality of designated positions of the photographing diagram of 720×720.

S5, calculating the pixel information entropy and the pixel gradient value of each time shooting block.

It should be understood that, it is necessary to fuse the shots obtained by multiple cameras in a short time and generate a high quality photo without chromatic aberration, and it is particularly important to select which shots in the early stage, if quality problems occur in the selection of the early shot, the subsequent picture fusion will be affected, so in the embodiment of the present invention, the time shot set screening operation needs to be performed according to the extracted time shot block, and the screening is based on the entropy and the pixel gradient value mainly derived from the pixel information.

In detail, the calculating the pixel information entropy and the pixel gradient value of each time photographing block includes:

calculating according to the pixel total value to obtain a pixel information entropy;

For example, if the sum of pixels of the time photographing block with the size of 50×50 is 20 ten thousand, it is further necessary to calculate the pixel information entropy, and in detail, the calculating the pixel information entropy according to the total pixel value includes:

wherein E is _i，j，s Entropy of pixel information representing the s-th time shooting block of the j-th time shooting diagram in the i-th time shooting set, p _s Representing each pixel value, p, in the s-th temporal shooting block _u Representing the pixel value of the u-th pixel in the s-th time photographing block.

For example, if the small sheet is photographed for first-stage protection plants to generate 3 groups of time photographing sets, the value of i is 1, 2 and 3 respectively, and the maximum value of j is 5 because the photographing device has 5 cameras.

Further, the performing convolution operation with the time photographing block using the 2 sets of gradient matrices to obtain pixel gradient values includes:

wherein M is _s，t Convolution pixels representing the t-th convolution operation performed by the s-th time capturing block of the j-th time capturing image in the I-th group of time capturing sets, |represents an absolute value symbol, I _3*3 Shooting a block for the s-th timeMatrix blocks of 3*3 dimensions;

It should be explained that, the convolution operation of the embodiment of the present invention adopts a corresponding multiplication method, that is, after generating 2 groups of gradient matrices 3*3, matrix blocks 3*3 are correspondingly cut out in the time shooting block according to the principle from left to right, and then pixel values of the matrix blocks 3*3 are correspondingly multiplied by 2 groups of gradient matrices 3*3 and added, so as to finally obtain convolution pixels. In addition, since the specification of the time shooting block is n×n, that is, a matrix block which cannot be completely split 3*3 may exist, the embodiment of the present invention adopts the zero padding operation to implement the split operation of 3*3.

S6, selecting one of the time shooting sets from the time shooting sets according to the pixel information entropy and the pixel gradient value to obtain a preferable shooting set.

In detail, selecting one of the plurality of sets of time shooting sets according to the pixel information entropy and the pixel gradient value to obtain a preferred shooting set comprises:

It should be explained that each group of time shooting set includes H time shooting pictures, where H represents the number of cameras, that is, after calculating the image pixel deviation value of each time shooting picture, according to a two-two subtraction method, the difference values of h+h-1+ … +2 groups of image pixel groups are calculated together, so as to determine the size relationship between each group of image pixel group difference values and the image pixel threshold value in sequence. For example, as described above, a total of 5 cameras may result in a total of 14 sets of 5+4+3+2 image pixel set differences.

Additionally, if 2 groups of time shooting pictures with the difference value of the image pixel groups larger than the image pixel threshold value exist in all the time shooting sets, the situation that the imaging deviation among a plurality of cameras is too large can be indicated, so that a user can be reminded of re-shooting, if the situation that 2 groups of time shooting pictures with the difference value of the image pixel groups larger than the image pixel threshold value still exist in the re-shot pictures, whether the cameras of shooting equipment have dust and other reasons can be checked, and the phenomenon that the imaging of the pictures after subsequent fusion is inconsistent with the actual objects is prevented.

It is emphasized that the method is simple and effective in calculation, and can rapidly screen out higher-quality shooting pictures on the premise of not occupying excessive calculation resources, so that sufficient preparation is made for subsequent picture fusion, and picture fusion operation is prevented from being influenced due to original picture quality problems.

S7, selecting a basic shooting diagram from the preferred shooting set, and setting other preferred shooting diagrams as weight shooting diagrams.

It should be explained that, in the embodiment of the present invention, the deep learning algorithm is avoided as the executor of the fusion operation, and the main reason is that the deep learning algorithm involves a large amount of operations, and if the ordinary photographing device supports the deep learning model as the executor of the fusion operation, it is necessary to add a CPU, GPU, etc. with high computing power into the photographing device, so that the situation is different from the actual situation. Therefore, the embodiment of the invention achieves the best image forming effect under the condition of minimum computing capability, on the basis of S1-S6, we have obtained a preferable shooting set with better image quality, then the next step is to execute image fusion on the preferable shooting set, so that a basic shooting image needs to be selected first, and the main function of the basic shooting image is to adjust the pixels of the basic shooting image according to the pixel weight values of other weight shooting images, thereby realizing the image fusion under the fastest speed and minimum computing quantity.

In detail, the selecting a basic shooting chart from the preferred shooting sets and setting other preferred shooting charts as weight shooting charts includes:

the preferred shot map with the smallest difference coefficient value is selected as a base shot map, and other preferred shot maps are set as weight shot maps.

It can be understood that, in fact, the pixel error of each preferred image in the preferred image capturing set is relatively small, so in order to amplify the pixel error between the preferred images, the embodiment of the present invention adopts a method of continuously performing the mean value calculation on the basis of the mean value, the median value, the maximum value and the minimum value, so as to select the most suitable preferred image capturing image as the base image capturing image.

And S8, sequentially extracting basic pixel points from the basic shooting diagram, identifying basic pixel points and pixel values of the basic pixel points, and selecting pixel points to be weighted from the weight shooting diagram according to the basic pixel points.

The base pixel points refer to the pixels in the base shooting image, and the base shooting image and the weight shooting image are shot by a camera with a standard of the image, so that the pixel value of the pixel point in the weight shooting image can be set according to the pixel value of the pixel point in the base shooting image. The pixel point positions of the pixels with the undetermined weights are the same as the pixel positions of the basic pixels in the basic shooting diagram, for example: and if the pixel point of the basic pixel point in the basic pixel map is (70, 80), the pixel point of the pixel point with the undetermined weight is also positioned at the (70, 80) position of the weight shooting map.

S9, acquiring shooting distance parameters of the basic pixel points, and judging whether related weight pixel points corresponding to the basic pixel points exist in the weight shooting diagram according to the shooting distance parameters and preset camera array parameters.

It may be appreciated that the shooting distance parameter refers to a distance between an actual position shot by the base pixel point and a camera, for example: when the object point shot by the basic pixel point is a point on a far petal, the shooting distance parameter is the distance between the point on the petal and the corresponding camera. The calculation method of the distance between the actual position photographed by the pixel point and the camera is the prior art, and is not described herein.

Further, the camera array parameter refers to a mutual distance parameter between the cameras. The related weight pixel points refer to pixel points of the weight shooting diagram, which shoot the same physical point as the basic pixel points.

It can be understood that, because the positions of the cameras are different, the shooting angles for the same physical point are different, so that the pixel points of the same physical point in the basic shooting image and the weight shooting image are different, and meanwhile, the physical point shot by a certain pixel point in the basic shooting image may not exist in the weight shooting image. The technology for judging whether the weight shooting image has the relevant weight pixel point according to the camera array parameter and the shooting distance parameter is the prior art (namely, judging whether the basic pixel point is present in the weight shooting image by a mode of connecting the physical point with the camera position and constructing a space coordinate system), and is not described herein.

And S10, if the related weight pixel point corresponding to the basic pixel point does not exist in the weight shooting diagram, setting the pixel value of the pixel point with the undetermined weight as the pixel value of the basic pixel point.

It can be understood that when the weight capturing image does not have the relevant weight pixel point corresponding to the basic pixel point, it indicates that the weight capturing image does not have the pixel point capable of assisting the basic pixel point, so that the pixel value of the relevant weight pixel point is directly set to be the same as the pixel value of the basic pixel point.

S11, if the related weight pixel points corresponding to the basic pixel points exist in the weight shooting image, acquiring pixel values of the related weight pixel points, setting the pixel values of the to-be-determined weight pixel points as the pixel values of the related weight pixel points, and obtaining a basic correction shooting image and a weight correction shooting image until the basic pixel points are extracted. Further, when the weight shooting graph has the relevant weight pixel point corresponding to the basic pixel point, it indicates that the weight shooting graph has the pixel point which can assist the basic pixel point to optimize the imaging effect, and therefore the pixel value of the undetermined weight pixel point should be set as the pixel value of the relevant weight pixel point.

S12, calculating the local standard deviation of the pixels of the basic correction shooting image, and dividing the basic correction shooting image into a local fusion area and a global fusion area based on the local standard deviation of the pixels.

It should be understood that, instead of merging the pixel information of all the weight correction shooting pictures into the basic correction shooting picture, it is confirmed that an area with imaging quality to be improved may occur in the basic correction shooting picture, namely, the global fusion area in the embodiment of the present invention, and more pixel information of the weight correction shooting picture is considered in the global fusion area.

In detail, the calculating the pixel local standard deviation of the basic correction shooting image, dividing the basic correction shooting image into a local fusion area and a global fusion area based on the pixel local standard deviation includes:

the following is performed for each local area block:

For example, the first-level protection plants shot by the small sheets are composed of 1 basic correction shooting image and 4 weight correction shooting images, and the 1 basic correction shooting image is 720 x 720 specification, so that the embodiment of the invention cuts the basic correction shooting image of 720 x 720 specification to generate a plurality of local area blocks with the size of 20 x 20, and plans to realize the division of the local fusion area and the global fusion area through the local standard deviation of pixels.

Further, the calculating the pixel local standard deviation of the local area block and the adjacent area block includes:

For example, if the base correction shooting map of the 720×720 specification is segmented to generate multiple local area blocks with the size of 20×20, and it is assumed that the local area blocks with the size of 6×20 share 4 adjacent area blocks, then local standard deviations of pixels of the local area blocks with the size of 6×20 and the 4 adjacent area blocks need to be calculated in sequence, and since the local area blocks have adjacent relations with other 4 adjacent area blocks, 1 edge must belong to a coincident edge, and pixels on the coincident edge are the coincident edge pixel set in the embodiment of the invention.

Continuously, the embodiment of the invention does not directly calculate the pixel difference value between the overlapped edge pixel set and other 7 edges, but calculates the pixel difference value between the overlapped edge pixels in the overlapped edge pixel set, for example, 20 pixels are shared in the overlapped edge pixel set, and the 20 pixels are subtracted from each other and added and divided by 10, namely the overlapped edge difference value in the embodiment of the invention, if the overlapped edge difference value is too large, the pixel fluctuation of the overlapped edge is indicated to be large, and the pixel fluctuation of the local area block corresponding to the overlapped edge and the adjacent area block is indicated to be huge, so that the pixel information of the shooting image needs to be corrected by considering other weights.

S13, merging pixel information of the weight correction shooting image into the local fusion area to obtain a local fused area, and processing the global fusion area by taking the weight correction shooting image as a parameter for histogram equalization to obtain the global fused area.

It should be explained that the merging the pixel information of the weight correction shooting image into the local merging area to obtain the local merged area includes:

It can be understood that the deep learning model has better picture fusion effect, but if the deep learning model is used repeatedly for many times, excessive calculation load is caused on the photographing equipment, so that the embodiment of the invention extracts the local fusion area which needs to be subjected to important fusion operation, and then constructs a lightweight fusion model, thereby realizing rapid fusion and ensuring the fusion quality.

In addition, histogram equalization is an operation of collecting histogram distribution of an image with smaller contrast in a smaller gray level range, so that better picture showing details are obtained.

S14, merging the local fused area and the global fused area to obtain an imaging diagram of the photographing device.

For example, the local fused region and the global fused region may be recombined and combined at the original image position, thereby obtaining an imaging diagram of the photographing apparatus.

Example 2:

fig. 2 is a functional block diagram of a camera array-based photographing fusion device according to an embodiment of the present invention.

The camera array-based photographing fusion apparatus 100 of the present invention may be installed in an electronic device. According to the functions implemented, the camera array-based photographing fusion device 100 may include an object photographing module 101, a gradient value calculating module 102, a fusion area dividing module 103, and a picture fusion module 104. The module of the invention, which may also be referred to as a unit, refers to a series of computer program segments, which are stored in the memory of the electronic device, capable of being executed by the processor of the electronic device and of performing a fixed function.

The object shooting module 101 is configured to receive a shooting instruction, and connect shooting equipment according to the shooting instruction, where the shooting equipment is composed of a plurality of cameras, the plurality of cameras are arranged in a matrix form and are all located in the same plane, confirm a focal length value of the shooting equipment, and shoot an object to be shot simultaneously by using the plurality of cameras according to the focal length value, so as to obtain a plurality of shooting pictures, where the number of the shooting pictures is a multiple of the number of the cameras;

the gradient value calculating module 102 is configured to perform time classification on multiple shot images according to different shooting times to obtain multiple groups of time shooting sets, where the number of the time shooting images in each group of time shooting sets is the same as the number of cameras, extract a specified number of time shooting blocks with n×n size in each time shooting image in each group of time shooting sets, and calculate pixel information entropy and pixel gradient value of each time shooting block;

The fusion area dividing module 103 is configured to select one of a plurality of time shooting sets from the plurality of time shooting sets according to pixel information entropy and pixel gradient values to obtain a preferred shooting set, select a base shooting image from the preferred shooting set, set other preferred shooting images as a weight shooting image, sequentially extract base pixel points in the base shooting image, identify base pixel points and pixel values of the base pixel points, select a pixel point with a weight to be determined according to the base pixel points in the weight shooting image, obtain shooting distance parameters of the base pixel points, judge whether a related weight pixel point corresponding to the base pixel point exists in the weight shooting image according to the shooting distance parameters and a preset camera array parameter, set the pixel value with the weight to be the pixel point with the weight to be determined as a pixel value of the base pixel point if the related weight pixel point corresponding to the base pixel point does not exist in the weight shooting image, obtain a correlation weight pixel point with the weight to be determined according to the base pixel point in the weight shooting image, and calculate a local difference of the base pixel point to be the local difference of the base pixel point to be corrected, and obtain a local difference of the base pixel to be corrected;

The picture fusion module 104 is configured to fuse pixel information of the weight correction shot image in the local fusion area to obtain a local fused area, process the global fusion area with the weight correction shot image as a histogram-averaged parameter to obtain a global fused area, and combine the local fused area and the global fused area to obtain an imaging image of the photographing device.

In detail, the modules in the camera array-based photographing fusion device 100 in the embodiment of the present invention use the same technical means as the camera array-based photographing fusion method described in fig. 1, and can produce the same technical effects, which are not described herein.

Example 3:

fig. 3 is a schematic structural diagram of an electronic device for implementing a camera fusion method based on a camera array according to an embodiment of the present invention.

The electronic device 1 may comprise a processor 10, a memory 11, a bus 12 and a communication interface 13, and may further comprise a computer program stored in the memory 11 and executable on the processor 10, such as a camera-array based photo fusion program.

The memory 11 includes at least one type of readable storage medium, including flash memory, a mobile hard disk, a multimedia card, a card memory (e.g., SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. The memory 11 may in some embodiments be an internal storage unit of the electronic device 1, such as a removable hard disk of the electronic device 1. The memory 11 may in other embodiments also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard) or the like, provided on the electronic device 1. Further, the memory 11 may also include both an internal storage unit and an external storage device of the electronic device 1. The memory 11 may be used not only for storing application software installed in the electronic device 1 and various types of data, such as codes of a camera-based photographing fusion program, etc., but also for temporarily storing data that has been output or is to be output.

The processor 10 may be comprised of integrated circuits in some embodiments, for example, a single packaged integrated circuit, or may be comprised of multiple integrated circuits packaged with the same or different functions, including one or more central processing units (CentralProcessingunit, CPU), microprocessors, digital processing chips, graphics processors, a combination of various control chips, and the like. The processor 10 is a control unit (control unit) of the electronic device, connects various parts of the entire electronic device using various interfaces and lines, executes various functions of the electronic device 1 and processes data by running or executing programs or modules (e.g., camera-array-based photo-fusion programs, etc.) stored in the memory 11, and calling data stored in the memory 11.

The bus may be an Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus may be classified as an address bus, a data bus, a control bus, etc. The bus is arranged to enable a connection communication between the memory 11 and at least one processor 10 etc.

Fig. 3 shows only an electronic device with components, it being understood by a person skilled in the art that the structure shown in fig. 3 does not constitute a limitation of the electronic device 1, and may comprise fewer or more components than shown, or may combine certain components, or may be arranged in different components.

For example, although not shown, the electronic device 1 may further include a power source (such as a battery) for supplying power to each component, and preferably, the power source may be logically connected to the at least one processor 10 through a power management device, so that functions of charge management, discharge management, power consumption management, and the like are implemented through the power management device. The power supply may also include one or more of any of a direct current or alternating current power supply, recharging device, power failure detection circuit, power converter or inverter, power status indicator, etc. The electronic device 1 may further include various sensors, bluetooth modules, wi-Fi modules, etc., which will not be described herein.

Further, the electronic device 1 may also comprise a network interface, optionally the network interface may comprise a wired interface and/or a wireless interface (e.g. WI-FI interface, bluetooth interface, etc.), typically used for establishing a communication connection between the electronic device 1 and other electronic devices.

The electronic device 1 may optionally further comprise a user interface, which may be a Display, an input unit, such as a Keyboard (Keyboard), or a standard wired interface, a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (organic light-emitting diode) touch, or the like. The display may also be referred to as a display screen or display unit, as appropriate, for displaying information processed in the electronic device 1 and for displaying a visual user interface.

It should be understood that the embodiments described are for illustrative purposes only and are not limited to this configuration in the scope of the patent application.

The camera array based photo fusion program stored in the memory 11 of the electronic device 1 is a combination of a plurality of instructions, which when executed in the processor 10, may implement:

selecting a basic shooting picture from the preferred shooting set, and setting other preferred shooting pictures as weight shooting pictures;

Specifically, the specific implementation method of the above instruction by the processor 10 may refer to descriptions of related steps in the corresponding embodiments of fig. 1 to 2, which are not repeated herein.

Further, the modules/units integrated in the electronic device 1 may be stored in a computer readable storage medium if implemented in the form of software functional units and sold or used as separate products. The computer readable storage medium may be volatile or nonvolatile. For example, the computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer memory, a Read-only memory (ROM).

The present invention also provides a computer readable storage medium storing a computer program which, when executed by a processor of an electronic device, can implement:

The modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical units, may be located in one place, or may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional module in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units can be realized in a form of hardware or a form of hardware and a form of software functional modules.

It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof.

Finally, it should be noted that the above-mentioned embodiments are merely for illustrating the technical solution of the present invention and not for limiting the same, and although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications and equivalents may be made to the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. The photographing fusion method based on the camera array is characterized by comprising the following steps of:

according to different shooting time, performing time classification on the plurality of shooting pictures to obtain a plurality of groups of time shooting sets, wherein the number of the time shooting pictures in each group of time shooting sets is the same as that of cameras; extracting a specified number of time shooting blocks with n x n size from each time shooting image in each time shooting set;

2. The camera array-based shot fusion method according to claim 1, wherein extracting a specified number of n x n time shooting blocks in each time shooting picture in each time shooting set comprises:

3. The camera array-based photo fusion method of claim 2, wherein the calculating the pixel information entropy and the pixel gradient value of each time shooting block comprises:

4. The camera array-based photo fusion method of claim 3, wherein performing a convolution operation with a time photographing block using 2 sets of gradient matrices to obtain pixel gradient values, comprises:

Wherein M is _s,t Convolution pixels representing the t-th convolution operation performed by the s-th time capturing block of the j-th time capturing image in the I-th group of time capturing sets, |represents an absolute value symbol, I _3*3 A 3*3-dimensional matrix block in the s-th time shooting block;

wherein T is _i,j,s The pixel gradient value representing the s-th time photographing block, U is the number of matrix pixels of the s-th time photographing block.

5. The camera array-based shot fusion method of claim 4, wherein selecting one of the plurality of sets of time shots based on pixel information entropy and pixel gradient values to obtain the preferred shot set comprises:

wherein P is _i,j Representing the image pixel deviation value of the jth time shooting picture in the ith group of time shooting sets, wherein C is the appointed number of time shooting blocks included in the jth time shooting picture;

6. The camera array-based shot fusion method according to claim 5, wherein selecting one basic correction shot from the preferred shot set and setting other preferred shot as the weight correction shot comprises:

7. The camera array-based shot fusion method of claim 6, wherein the calculating the pixel local standard deviation of the base correction shot map, dividing the base correction shot map into a local fusion area and a global fusion area based on the pixel local standard deviation, comprises:

the following is performed for each local area block:

8. The camera array-based photo fusion method of claim 7, wherein the calculating the pixel local standard deviation of the local area block and the neighboring area block comprises:

9. The camera array-based photographing fusion method as set forth in claim 8, wherein the merging the pixel information of the weight correction photographing map into the local fusion area to obtain the local fused area includes:

10. A camera array-based photographing fusion device, the device comprising: