CN108833785B - Fusion method and device of multi-view images, computer equipment and storage medium - Google Patents
Fusion method and device of multi-view images, computer equipment and storage medium Download PDFInfo
- Publication number
- CN108833785B CN108833785B CN201810717536.3A CN201810717536A CN108833785B CN 108833785 B CN108833785 B CN 108833785B CN 201810717536 A CN201810717536 A CN 201810717536A CN 108833785 B CN108833785 B CN 108833785B
- Authority
- CN
- China
- Prior art keywords
- image
- fused
- current
- target object
- definition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/95—Computational photography systems, e.g. light-field imaging systems
- H04N23/951—Computational photography systems, e.g. light-field imaging systems by using two or more images to influence resolution, frame rate or aspect ratio
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
Abstract
The invention discloses a method and a device for fusing multi-view images, computer equipment and a storage medium. The method comprises the following steps: acquiring an image to be fused of a target object under a standard shooting visual angle and a reference image of the target object under a non-standard shooting visual angle; calculating optical flow information between the image to be fused and the reference image; and fusing the image to be fused and the reference image according to the optical flow information to obtain a target image of the target object under a standard shooting view angle. By adopting the technical scheme, the embodiment of the invention can improve the fusion speed of the multi-view images and reduce the manpower and material resources consumed in the image fusion process.
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to a method and an apparatus for fusing multi-view images, a computer device, and a storage medium.
Background
In recent years, with the cost reduction of the camera sensor, multi-camera and even dense-array multi-view image acquisition equipment gradually become the mainstream trend of image acquisition.
Generally, the multi-view image acquisition device performs image fusion synthesis algorithm on the images acquired from the multi-view image acquisition device under each shooting view to generate an image of a target scene, so that compared with the image acquired by the single-camera image acquisition device, the multi-view image acquisition device has the advantages of high resolution, low noise, large information amount and the like. Currently, the following challenges need to be faced when performing multi-view graphics fusion: the conditions of parallax, shielding and the like exist among the pictures collected under different visual angles, and the parameters of the sensors and the lens of the multiple cameras are different, so that the pictures collected under different visual angles have differences of resolution, color, exposure degree and the like.
However, the inventor finds that the prior art has the following defects in the process of implementing the invention: in the prior art, when the above-mentioned war is carried out, the image is usually processed in a manual calibration mode, the processing speed is low, and long manpower and material resources are wasted.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method and an apparatus for fusing multi-view images, a computer device, and a storage medium, so as to solve the technical problem in the prior art that an image processing speed of a multi-view image capturing device is slow.
In a first aspect, an embodiment of the present invention provides a method for fusing multi-view images, including:
acquiring an image to be fused of a target object under a standard shooting visual angle and a reference image of the target object under a non-standard shooting visual angle;
calculating optical flow information between the image to be fused and the reference image;
and fusing the image to be fused and the reference image according to the optical flow information to obtain a target image of the target object under a standard shooting view angle.
In a second aspect, an embodiment of the present invention provides a device for fusing multi-view images, including:
the image acquisition module is used for acquiring an image to be fused of the target object under a standard shooting visual angle and a reference image of the target object under a non-standard shooting visual angle;
the optical flow information calculation module is used for calculating optical flow information between the image to be fused and the reference image;
and the image fusion module is used for fusing the image to be fused and the reference image according to the optical flow information to obtain a target image of the target object under a standard shooting visual angle.
In a third aspect, an embodiment of the present invention provides a computer device, including:
one or more processors;
a memory for storing one or more programs,
when the one or more programs are executed by the one or more processors, the one or more processors implement the method for fusing multi-view images according to the embodiment of the present invention.
In a fourth aspect, the embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, which when executed by a processor, implements the multi-view image fusion method according to the embodiment of the present invention.
In the technical scheme for fusing the multi-view images, an image to be fused of a target object acquired at a standard shooting angle and a reference image of the target object acquired at a non-standard shooting angle are acquired, optical flow information between the image to be fused and the reference image is calculated, the image to be fused and the reference image are fused according to the calculated optical flow information, and a target image of the target object at the standard shooting angle is obtained. By adopting the technical scheme, the embodiment of the invention can improve the fusion speed of the multi-view images and reduce the manpower and material resources consumed in the image fusion process.
Drawings
Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments made with reference to the following drawings:
fig. 1 is a schematic flowchart of a multi-view image fusion method according to an embodiment of the present invention;
fig. 2 is a schematic flowchart of a multi-view image fusion method according to a second embodiment of the present invention;
fig. 3 is a block diagram of a multi-view image fusion apparatus according to a third embodiment of the present invention;
fig. 4 is a schematic structural diagram of a computer device according to a fourth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some but not all of the relevant aspects of the present invention are shown in the drawings.
Example one
The embodiment of the invention provides a fusion method of multi-view images. The method can be executed by a multi-view image fusion device, wherein the device can be implemented by software and/or hardware, can be generally integrated in computer equipment with a multi-view image fusion function, and can be typically integrated in a multi-camera intelligent terminal or multi-view image acquisition equipment. Fig. 1 is a schematic flowchart of a method for fusing multi-view images according to an embodiment of the present invention, as shown in fig. 1, the method includes:
s101, acquiring an image to be fused of the target object under a standard shooting visual angle and a reference image of the target object under a non-standard shooting visual angle.
In this embodiment, the image to be fused of the target object and the reference image may be generated by local acquisition, or may be acquired from other devices. In consideration of timeliness of image fusion, preferably, if the home terminal has a multi-view image acquisition function, the target object may be photographed by an image acquisition device such as a camera, a camera or an image sensor configured at the home terminal, so as to obtain an image to be fused and a reference image of the target object; if the local terminal does not have a multi-view image acquisition function, if the local terminal is not provided with an image acquisition device such as a camera, a camera and an image sensor or the local terminal is only provided with an image acquisition device such as a camera, a camera or an image sensor, the image to be fused and the reference image of the target object can be acquired through other multi-view image acquisition devices, and at the moment, the image to be fused and the reference image which are fused at this time can be acquired from the multi-view image acquisition devices or the storage devices which store the image to be fused and the reference image of the target object correspondingly.
The standard shooting visual angle and the non-standard shooting visual angle can be preset by a developer or a user, and can also be adjusted according to the imaging effect of the target object under each shooting visual angle in the shooting process. The image to be fused and the reference image can be generated by shooting the target object by different cameras in the multi-view image acquisition device or different cameras in the multi-camera image acquisition device (such as a mobile phone with two cameras). The image to be fused and the reference image may only contain the target object, or may further contain other scene backgrounds. The resolution of the image to be fused and the reference image may be the same or different, and is not limited herein. In view of the imaging effect of the fused target image, it is preferable that the image to be fused and the reference image have the same resolution.
S102, calculating optical flow information between the image to be fused and the reference image.
In this embodiment, the optical flow is a parameter representing a position relationship of the target image in the time-varying image, and the optical flow information between the image to be fused and the reference image may represent a corresponding relationship between positions of the respective scenes in the image to be fused and the reference image. For example, if a certain position of the target object is imaged at a position of a pixel point a of the image to be fused, and the position is imaged at a position of a pixel point B of the reference image, the optical flow information between the image to be fused and the reference image may represent a relative positional relationship between the pixel point a and the pixel point B, for example, after the pixel point a moves several pixel points in the x and y directions, a position coordinate in the image to be fused is the same as a position coordinate of the pixel point B in the reference image.
In this step, the calculation method of the optical flow information between the image to be fused and the reference image may be set as needed. Specifically, the image to be fused and the reference image may be input into a neural network model having an optical flow estimation function, and optical flow information between the image to be fused and the reference image may be determined based on an output value of the neural network; the optical flow information between the image to be fused and the reference image may also be calculated based on a matching, frequency domain or gradient method, for example, the optical flow information may be determined by respectively locating and tracking main features of the target object in the image to be fused and the reference image, locating similar regions in the image to be fused and the reference image and calculating the optical flow information through displacement of the similar regions, calculating frequency domain information of the image to be fused and the reference image and determining the optical flow information based on the frequency domain information, or calculating a 2D velocity field of the target object changed from the image to be fused to the reference image by using the time-space differential of the brightness of the image to be fused and the reference image sequence, and further determining the optical flow information between the image to be fused and the reference image, which is not limited herein.
Here, it should be noted that the optical flow information between the image to be fused and the reference image may only include optical flow information between each pixel point of the image to be fused and each pixel point of the reference image, and may further include optical flow information between each area of the image to be fused and the corresponding area of the reference image, at this time, the optical flow information between the image to be fused and the reference image includes a plurality of sub-reference information, and each sub-optical flow information may describe optical flow information between each pixel point of the image to be fused and the reference image or optical flow information between each pixel area of the image to be fused and the corresponding pixel area of the reference image. The pixel region can be obtained by segmenting or downsampling the image to be fused and the reference image. The number of times of segmenting or downsampling the image to be fused and the reference image may be set as required, and when the number of times of segmenting or downsampling is multiple, the current segmentation or downsampling may be performed on the basis of the original image to be fused and the reference image, or on the basis of the pixel region obtained by the previous segmentation or downsampling, which is not limited herein.
Taking the pixel regions of the image to be fused and the reference image obtained by downsampling as an example, specifically, downsampling the image to be fused and the reference image for a set number of times may be performed respectively to obtain the feature map to be fused of the image to be fused at different definitions and the reference feature map of the reference image at different definitions; and calculating sub-optical flow information between the feature map to be fused and the reference feature map with the same definition. The feature graph to be fused comprises a feature graph corresponding to the image of the graph to be fused (namely the image of the graph to be fused), and the reference feature graph comprises a feature graph corresponding to the reference image (namely the image of the reference image); each sampling can be performed on the basis of the feature map to be fused and the reference feature map obtained by the last sampling, in the same sampling, a first sampling coefficient used for sampling the reference image is preferably the same as a second sampling coefficient used for sampling the image to be fused, and sampling coefficients used for different times of sampling can be the same or different. At this time, correspondingly, the sub-optical flow information between the feature map to be fused and the reference feature map can be obtained by calculating the corresponding relationship between the pixels of the feature map to be fused and the reference feature map (i.e., the feature map to be fused and the reference feature map with the same definition) obtained by sampling at the same time, that is, the optical flow information between the image to be fused and the reference image can be obtained.
For example, assuming that sampling coefficients used for each sampling are 1/2 (that is, a sampling step length is 2), and the sampling frequency is 4, when an image to be fused is sampled, the image to be fused itself may be determined as an original image to be fused obtained by sampling at the zeroth layer, and the original image to be fused feature map is sampled at the sampling step length 2, so as to obtain a first image to be fused feature map obtained by sampling at the first layer; sampling the first graph feature diagram to be fused by using the sampling step length 2 to obtain a second graph feature diagram to be fused obtained by sampling the second layer, sampling the second graph feature diagram to be fused by using the sampling step length 2 to obtain a third graph feature diagram to be fused obtained by sampling the third layer, and thus, sampling the graph image to be fused can be realized; with reference to the above process, reference feature maps of the reference image can be obtained, and are not described in detail herein.
S103, fusing the image to be fused and the reference image according to the optical flow information to obtain a target image of the target object under a standard shooting view angle.
Specifically, the corresponding relationship between the pixels of the image to be fused and the reference image may be determined according to the optical flow information between the image to be fused and the reference image (that is, a pixel point pair with the same shooting content in the image to be fused and the reference image is determined), and the color of the fused target pixel is determined according to the color information (RGB information) of the pixel with the corresponding relationship, so as to determine the color of the target pixel in the target image having the same position coordinate as the pixel in the image to be fused, thereby obtaining the target image of the target object under the standard shooting angle. The weight value adopted when the colors of the pixel point pairs with the objects are fused can be set according to needs, for example, the color of a first pixel point of an image to be fused and the color of a second pixel point in a reference image can be set to be 1; 1. 1: 5 or 0: 1, etc., without limitation.
For example, when the reference information includes a plurality of pieces of sub-reference information, the image to be fused and the reference image may be fused based on the sub-optical flow information, the feature map to be fused, and the reference image, so as to obtain a target image of the target object under a standard shooting view angle. If the definition can be in the sequence from low to high, firstly fusing the feature graph to be fused with the lowest definition and the reference feature graph to obtain a first sub-target graph; and performing up-sampling on the first sub-target map according to the reciprocal of a coefficient used in down-sampling or segmentation to obtain a first image to be fused under the second low definition, fusing the first image to be fused with the feature map to be fused and/or the reference feature map under the second low definition to obtain a second sub-target map, and repeating the steps to obtain a target image of the target object under the standard shooting view angle.
The fusion method of the multi-view images, provided by the embodiment of the invention, includes the steps of acquiring an image to be fused of a target object acquired at a standard shooting angle and a reference image of the target object acquired at a non-standard shooting angle, calculating optical flow information between the image to be fused and the reference image, and fusing the image to be fused and the reference image according to the calculated optical flow information to obtain a target image of the target object at the standard shooting angle. By adopting the technical scheme, the fusion speed of the multi-view images can be improved, and the manpower and material resources consumed in the image fusion process are reduced.
Example two
Fig. 2 is a schematic flow chart of a multi-view image fusion method according to a second embodiment of the present invention, and in this embodiment, optimization is performed on the basis of the above-mentioned embodiment, and in this embodiment, "fusion is performed on the image to be fused and the reference image based on the sub optical flow information, the feature map to be fused, and the reference image, so as to obtain a target image of a target object under a standard shooting view" is optimized as follows: performing view angle correction on the reference characteristic diagram according to the sub-optical flow information so as to correct the reference characteristic diagram into a corrected diagram under a standard shooting view angle; and sequentially fusing the feature graph to be fused and the correction graph under each definition according to the sequence of the definitions from small to large to obtain a target image of the target object under the standard shooting visual angle.
Further, before the acquiring the image to be fused of the target object under the standard shooting view angle and the reference image of the target object under the non-standard shooting view angle, the method further includes: acquiring an original image of a target object under a standard shooting visual angle, a first resolution of the original image and a second resolution of a reference image of the target object under a non-standard shooting visual angle; and converting the original image into an image to be fused with a second resolution according to the ratio between the second resolution and the first resolution.
Further, before the converting the original image into the image to be fused with the second resolution according to the ratio between the second resolution and the first resolution, the method further includes: determining that the first resolution is lower than the second resolution.
Further, the method for fusing multi-view images provided by this embodiment may further include: and shooting the target object by adopting a multi-view camera to obtain an original image of the target object under a standard shooting view angle and a reference image of the target object under a non-standard shooting view angle.
Correspondingly, as shown in fig. 2, the method for fusing multi-view images provided in this embodiment includes:
s201, shooting a target object by using a multi-view camera to obtain an original image of the target object under a standard shooting view angle and a reference image of the target object under a non-standard shooting view angle.
The multi-view camera can be any device with multiple shooting views, such as an intelligent terminal with multiple cameras or other multi-view image acquisition devices. The multi-view camera may be located outside the local terminal and establish a communication connection with the local terminal, or may be integrated inside the local terminal, which is not limited herein. Illustratively, when the multi-view camera exists independently from the home terminal, the home terminal may generate an image acquisition instruction when detecting that a current condition meets an image acquisition condition or when detecting that a user triggers an image acquisition request, and send the image acquisition instruction to the multi-view camera establishing connection with the home terminal, so as to control the multi-view camera to shoot a target object, thereby obtaining an original image and a reference image of the target object; when the multi-view camera is integrated in the home terminal, the multi-view camera can be directly controlled to acquire the original image and the reference image of the target object when the acquisition condition of the image is currently met or when the user triggers the image acquisition request is detected.
S202, acquiring an original image of the target object under a standard shooting visual angle, a first resolution of the original image and a second resolution of a reference image of the target object under a non-standard shooting visual angle.
Specifically, the original image may be acquired from the local or from a multi-view camera that captures the original image, the first resolution of the original image and the second resolution of the reference image may be determined based on image description information of the original image and the reference image, or may be determined by analyzing the number of pixels in the row direction and the column direction of the original image and the reference image or the size of the original image and the reference image, or may be determined based on parameters of a camera (a camera or an image sensor) that captures the original image and a camera (a camera or an image sensor) that captures the reference image, which is not limited in this embodiment.
S203, determining that the first resolution is lower than the second resolution.
S204, converting the original image into an image to be fused with a second resolution according to the ratio between the second resolution and the first resolution.
In this embodiment, the original image may be processed into an image to be fused having the same resolution as the reference image and the subsequent fusion operation is performed, regardless of the size between the resolutions of the original image and the reference image, or the image to be fused obtained after the resolution conversion of the original image and the reference image may be fused only when the resolution of the original image is less than or equal to the resolution of the image to be fused, which is not limited herein.
In view of the practicability of the target image generated by the fusion processing, it is preferable that the fusion processing is performed on the image to be fused converted from the original image and the reference image only when the first resolution of the original image is lower than the second resolution of the reference image and/or when the first resolution of the original image is equal to the second resolution of the reference image and the imaging of the target object in the original image is blocked, so as to further improve the imaging effect of the target object. At this time, when the first resolution of the original image is less than the second resolution of the reference image, S204-S209 may be performed to obtain the target image; when the first resolution of the original image is equal to the second resolution of the reference image and the imaging of the target image in the original image is blocked, S204 may not be executed, the original image is directly determined as the image to be fused, and S205-S209 are executed to obtain the target image.
For the case that the first resolution of the original image is equal to the second resolution of the reference image and the imaging of the target image in the original image is blocked, the object to be fused may be determined as needed, for example, all the pixels in the image to be fused (i.e., the original image) may be fused, or only the blocked area of the target object in the image to be fused may be fused, so as to further reduce the amount of calculation required in the fusion process. For the case that the first resolution of the original image is smaller than the second resolution of the reference image, the original image may be up-sampled according to a ratio between the second resolution and the first resolution to obtain an image to be fused with the second resolution. Here, the upsampling method may be selected as needed, and this embodiment does not limit this.
S205, acquiring an image to be fused of the target object under a standard shooting visual angle and a reference image of the target object under a non-standard shooting visual angle.
S206, downsampling the image to be fused and the reference image for a set number of times respectively to obtain the feature map to be fused of the image to be fused under different definitions and the reference feature map of the reference image under different definitions.
And S207, calculating sub optical flow information between the feature graph to be fused and the reference feature graph with the same definition.
And S208, carrying out view angle correction on the reference characteristic diagram according to the sub-optical flow information so as to correct the reference characteristic diagram into a corrected diagram under a standard shooting view angle.
In this embodiment, the shooting angle of view of the reference feature map may be corrected to a standard shooting angle of view according to the optical flow information to obtain a corrected map. At this time, for example, for a certain reference feature map, a feature map to be fused corresponding to the reference feature map (i.e. a feature map to be fused having the same definition as the reference feature map) may be determined first, and sub-optical flow information between the reference feature map and the feature map to be fused is obtained; and then determining the pixel points of each pixel point which need to move in the row direction and the column direction when the shooting visual angle of the reference characteristic diagram is converted into the standard shooting visual angle according to the sub-optical flow information, and then moving the corresponding pixel points in the reference characteristic diagram according to the pixel points, thereby obtaining a corrected diagram of the reference characteristic diagram under the standard shooting visual angle.
S209, sequentially fusing the feature graph to be fused and the correction graph under each definition according to the sequence of the definitions from small to large to obtain a target image of the target object under the standard shooting view angle.
In this embodiment, the feature images to be fused and the correction images at each definition may be sequentially fused to obtain a fusion image at each definition, then the fusion image with lower definition is up-sampled with the highest definition in each fusion image as a standard to obtain a plurality of images to be processed with the same definition as that of the fusion image with the highest resolution, and then each image to be processed is fused again to obtain a target image at a standard shooting view angle; or according to the sequence of the definitions from small to large, the feature graph to be fused, the correction graph and the intermediate graph obtained by sampling the fused graph obtained by fusing the previous definition are fused under each definition to obtain the current fused graph under the current definition, and so on until the current definition has no next definition higher than the definition of the current definition, at the moment, the fused target image can be obtained, and the definition is not limited.
In order to further improve the imaging effect of the target object in the target image obtained by fusion, preferably, the sequentially fusing the feature map to be fused and the correction map at each definition according to the order from small definition to large definition to obtain the target image of the target object at the standard shooting view angle may include: acquiring a current feature map under current definition, wherein the current feature map comprises a current feature map to be fused, a current correction map and a current intermediate image, and the current intermediate image is acquired by performing up-sampling on a last fused image obtained by fusing under previous definition; performing fusion processing on the current feature map to obtain a current fusion image; judging whether a next definition which is higher than the current definition and is adjacent to the current definition exists in the definition sequencing, if so, determining the next definition as the current definition, and returning to execute the operation of obtaining the current feature map under the current definition; and if not, determining the current fusion image as a target image of the target object under a standard shooting visual angle. Here, it should be noted that, if the current definition is the minimum definition in the definition sequence, that is, there is no previous definition in the definition sequence whose definition is lower than the current definition, the feature diagram to be fused and the reference feature diagram at the current definition may be fused to obtain the current fused image at the current definition.
It should be noted that, in this embodiment, the steps 202 to 209 may be executed by a neural network model at the integrated local end, that is, an original image and a reference image captured by a multi-view camera may be input to the neural network model, and the steps 202 to 209 are executed by the neural network model, where an output value of the neural network is a target object of the target object under a standard capturing view angle. Each feature graph to be fused and each reference feature graph can exist in a feature vector mode in the neural network, and the feature vector of a certain feature graph can describe the coordinate information and the color information of each pixel point of the feature graph, namely the feature graph can be restored by rendering the feature vector.
The method for fusing multi-view images provided by the third embodiment of the invention includes the steps of shooting an original image and a reference image by using a multi-view relative to a target object, obtaining a first resolution of the original image and a second resolution of the reference image, converting the original image into an image to be fused with the second resolution if the first resolution is lower than the second resolution, respectively performing downsampling on the image to be fused and the reference image for a set number of times to obtain a feature map to be fused and a reference feature map, modifying each reference feature map into a modified map under a standard shooting view according to sub-optical flow information between each feature map to be fused and the reference map, and sequentially fusing the feature map to be fused and the modified map under each definition according to a sequence from small definition to large definition to obtain the target image of the target object under the standard shooting view. By adopting the technical scheme, the fusion speed of the multi-view images can be improved, and the manpower and material resources consumed in the image fusion process are reduced; the influence of resolution, tone and/or exposure parameters on the image fusion effect can be reduced, camera parameters are not required to be calibrated manually in the fusion process, errors caused by inaccurate camera parameter calibration can be avoided, and the fusion effect of the multi-view image is further improved.
EXAMPLE III
The third embodiment of the present invention provides a multi-view image fusion device, which can be implemented by software and/or hardware, and can be generally integrated in a computer device with a multi-view image fusion function, typically, can be integrated in a multi-camera intelligent terminal or a multi-view image acquisition device, and can implement fusion of multi-view images by executing a multi-view image fusion method. Fig. 3 is a block diagram of a multi-view image fusion apparatus according to a third embodiment of the present invention, and as shown in fig. 3, the apparatus includes:
the image acquisition module 301 is configured to acquire an image to be fused of a target object under a standard shooting view and a reference image of the target object under a non-standard shooting view;
an optical flow information calculation module 302, configured to calculate optical flow information between the image to be fused and the reference image;
and the image fusion module 303 is configured to fuse the image to be fused and the reference image according to the optical flow information to obtain a target image of the target object under a standard shooting view angle.
In the multi-view image fusion device provided by the third embodiment of the present invention, the image acquisition module is used to acquire the image to be fused of the target object acquired at the standard shooting angle and the reference image of the target object acquired at the non-standard shooting angle, the optical flow information calculation module is used to calculate the optical flow information between the image to be fused and the reference image, and the image fusion module is used to fuse the image to be fused and the reference image according to the calculated optical flow information to obtain the target image of the target object at the standard shooting angle. By adopting the technical scheme, the fusion speed of the multi-view images can be improved, and the manpower and material resources consumed in the image fusion process are reduced.
In the above solution, the optical flow information calculating module 302 may include: the down-sampling unit is used for respectively performing down-sampling on the image to be fused and the reference image for set times to obtain a feature map to be fused of the image to be fused under different definitions and a reference feature map of the reference image under different definitions; the optical flow information computing unit is used for computing sub optical flow information between the feature graph to be fused and the reference feature graph with the same definition; accordingly, the image fusion module 303 may be configured to: and fusing the image to be fused and the reference image based on the sub-optical flow information, the feature graph to be fused and the reference image to obtain a target image of the target object under a standard shooting view angle.
In the foregoing solution, the image fusion module 303 may include: a view angle correction unit, configured to perform view angle correction on the reference feature map according to the sub-optical flow information, so as to correct the reference feature map into a corrected map at a standard shooting view angle; and the image fusion unit is used for sequentially fusing the feature graph to be fused and the correction graph under each definition according to the sequence of the definitions from small to large to obtain a target image of the target object under the standard shooting visual angle.
In the foregoing aspect, the image fusion unit may include: the characteristic diagram obtaining subunit is used for obtaining a current characteristic diagram under current definition, wherein the current characteristic diagram comprises a current characteristic diagram to be fused, a current correction diagram and a current intermediate image, and the current intermediate image is obtained by performing upsampling on a last fusion image obtained by fusion under previous definition; the feature map fusion subunit is used for performing fusion processing on the current feature map to obtain a current fusion image; the target image determining subunit is used for judging whether a next definition which is higher than the current definition and is adjacent to the current definition exists in the definition sequencing, if so, determining the next definition as the current definition, and returning to execute the operation of obtaining the current feature map under the current definition; and if not, determining the current fusion image as a target image of the target object under a standard shooting visual angle.
Further, the multi-view image fusion apparatus provided in this embodiment may further include: the resolution acquisition module is used for acquiring an original image of the target object under a standard shooting visual angle, a first resolution of the original image and a second resolution of a reference image of the target object under a non-standard shooting visual angle before acquiring the image to be fused of the target object under the standard shooting visual angle and the reference image of the target object under the non-standard shooting visual angle; and the image conversion module is used for converting the original image into an image to be fused with the second resolution according to the ratio between the second resolution and the first resolution.
Further, the multi-view image fusion apparatus provided in this embodiment may further include: a determining module, configured to determine that the first resolution is lower than the second resolution before the converting the original image into an image to be fused with the second resolution according to a ratio between the second resolution and the first resolution.
Further, the multi-view image fusion apparatus provided in this embodiment may further include: the image acquisition module is used for shooting the target object by adopting the multi-view camera to obtain an original image of the target object under a standard shooting view angle and a reference image of the target object under a non-standard shooting view angle.
The multi-view image fusion device provided by the third embodiment of the invention can execute the multi-view image fusion method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of executing the multi-view image fusion method. For the technical details that are not described in detail in this embodiment, reference may be made to the method for fusing multi-view images provided in any embodiment of the present invention.
Example four
Fig. 4 is a schematic structural diagram of an apparatus/terminal/server according to a fourth embodiment of the present invention, as shown in fig. 4, the apparatus/terminal/server includes a processor 40 and a memory 41, and may further include an input device 42 and an output device 43; the number of processors 40 in the device/terminal/server may be one or more, and one processor 40 is taken as an example in fig. 4; the processor 40, the memory 41, the input device 42 and the output device 43 in the device/terminal/server may be connected by a bus or other means, which is exemplified in fig. 4.
The memory 41, as a computer-readable storage medium, may be used to store software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the multi-view image fusion method in the embodiment of the present invention (for example, the image acquisition module 301, the optical flow information calculation module 302, and the image fusion module 303 in the multi-view image fusion apparatus). The processor 40 executes various functional applications of the device/terminal/server and data processing by running software programs, instructions and modules stored in the memory 41, that is, implements the above-described multi-view image fusion method.
The memory 41 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal, and the like. Further, the memory 41 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, memory 41 may further include memory located remotely from processor 40, which may be connected to devices/terminals/servers via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 42 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the apparatus/terminal/server. The output device 43 may include a display device such as a display screen.
An embodiment of the present invention further provides a storage medium containing computer-executable instructions, which when executed by a computer processor, perform a method for fusing multi-view images, the method including:
acquiring an image to be fused of a target object under a standard shooting visual angle and a reference image of the target object under a non-standard shooting visual angle;
calculating optical flow information between the image to be fused and the reference image;
and fusing the image to be fused and the reference image according to the optical flow information to obtain a target image of the target object under a standard shooting view angle.
Of course, the storage medium containing the computer-executable instructions provided by the embodiments of the present invention is not limited to the method operations described above, and may also perform related operations in the multi-view image fusion method provided by any embodiments of the present invention.
From the above description of the embodiments, it is obvious for those skilled in the art that the present invention can be implemented by software and necessary general hardware, and certainly, can also be implemented by hardware, but the former is a better embodiment in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which may be stored in a computer-readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a FLASH Memory (FLASH), a hard disk or an optical disk of a computer, and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute the methods according to the embodiments of the present invention.
It should be noted that, in the embodiment of the fusion apparatus for multi-view images, the included units and modules are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be implemented; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (7)
1. A method for fusing multi-view images, comprising:
acquiring an image to be fused of a target object under a standard shooting visual angle and a reference image of the target object under a non-standard shooting visual angle;
respectively carrying out downsampling on the image to be fused and the reference image for a set number of times to obtain a feature map to be fused of the image to be fused under different definitions and a reference feature map of the reference image under different definitions;
calculating sub-optical flow information between the feature graph to be fused and the reference feature graph with the same definition;
performing view angle correction on the reference characteristic diagram according to the sub-optical flow information so as to correct the reference characteristic diagram into a corrected diagram under a standard shooting view angle;
acquiring a current feature map under current definition, wherein the current feature map comprises a current feature map to be fused, a current correction map and a current intermediate image, and the current intermediate image is acquired by performing up-sampling on a last fused image obtained by fusing under previous definition;
performing fusion processing on the current feature map to obtain a current fusion image;
judging whether a next definition which is higher than the current definition and is adjacent to the current definition exists in the definition sequencing, if so, determining the next definition as the current definition, and returning to execute the operation of obtaining the current feature map under the current definition; and if not, determining the current fusion image as a target image of the target object under a standard shooting visual angle.
2. The method according to claim 1, wherein before the acquiring the image to be fused of the target object under the standard shooting view and the reference image of the target object under the non-standard shooting view, the method further comprises:
acquiring an original image of a target object under a standard shooting visual angle, a first resolution of the original image and a second resolution of a reference image of the target object under a non-standard shooting visual angle;
and converting the original image into an image to be fused with a second resolution according to the ratio between the second resolution and the first resolution.
3. The method according to claim 2, further comprising, before said converting said original image into an image to be fused having a second resolution according to a ratio between said second resolution and said first resolution:
determining that the first resolution is lower than the second resolution.
4. The method of claim 2, further comprising:
and shooting the target object by adopting a multi-view camera to obtain an original image of the target object under a standard shooting view angle and a reference image of the target object under a non-standard shooting view angle.
5. A device for fusing multi-view images, comprising:
the image acquisition module is used for acquiring an image to be fused of the target object under a standard shooting visual angle and a reference image of the target object under a non-standard shooting visual angle;
the optical flow information calculation module is used for calculating optical flow information between the image to be fused and the reference image;
the image fusion module is used for fusing the image to be fused and the reference image according to the optical flow information to obtain a target image of a target object under a standard shooting visual angle;
wherein the optical flow information calculation module includes: the down-sampling unit is used for respectively performing down-sampling on the image to be fused and the reference image for set times to obtain a feature map to be fused of the image to be fused under different definitions and a reference feature map of the reference image under different definitions; the optical flow information computing unit is used for computing sub optical flow information between the feature graph to be fused and the reference feature graph with the same definition; the image fusion module is configured to: fusing the image to be fused and the reference image based on the sub-optical flow information, the feature graph to be fused and the reference image to obtain a target image of the target object under a standard shooting view angle
The image fusion module includes: a view angle correction unit, configured to perform view angle correction on the reference feature map according to the sub-optical flow information, so as to correct the reference feature map into a corrected map at a standard shooting view angle; the image fusion unit is used for sequentially fusing the feature graph to be fused and the correction graph under each definition according to the sequence of the definitions from small to large to obtain a target image of the target object under a standard shooting visual angle;
the image fusion unit may include: the characteristic diagram obtaining subunit is used for obtaining a current characteristic diagram under current definition, wherein the current characteristic diagram comprises a current characteristic diagram to be fused, a current correction diagram and a current intermediate image, and the current intermediate image is obtained by performing upsampling on a last fusion image obtained by fusion under previous definition; the feature map fusion subunit is used for performing fusion processing on the current feature map to obtain a current fusion image; the target image determining subunit is used for judging whether a next definition which is higher than the current definition and is adjacent to the current definition exists in the definition sequencing, if so, determining the next definition as the current definition, and returning to execute the operation of obtaining the current feature map under the current definition; and if not, determining the current fusion image as a target image of the target object under a standard shooting visual angle.
6. A computer device, characterized in that the computer device comprises:
one or more processors;
a memory for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of multi-perspective image fusion of any of claims 1-4.
7. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out a method of fusing multi-view images according to any one of claims 1 to 4.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810717536.3A CN108833785B (en) | 2018-07-03 | 2018-07-03 | Fusion method and device of multi-view images, computer equipment and storage medium |
| PCT/CN2019/094553 WO2020007320A1 (en) | 2018-07-03 | 2019-07-03 | Method for fusing multi-visual angle images, apparatus, computer device, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810717536.3A CN108833785B (en) | 2018-07-03 | 2018-07-03 | Fusion method and device of multi-view images, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108833785A CN108833785A (en) | 2018-11-16 |
| CN108833785B true CN108833785B (en) | 2020-07-03 |
Family
ID=64134739
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810717536.3A Active CN108833785B (en) | 2018-07-03 | 2018-07-03 | Fusion method and device of multi-view images, computer equipment and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN108833785B (en) |
| WO (1) | WO2020007320A1 (en) |
Families Citing this family (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108833785B (en) * | 2018-07-03 | 2020-07-03 | 清华-伯克利深圳学院筹备办公室 | Fusion method and device of multi-view images, computer equipment and storage medium |
| CN112135088B (en) * | 2019-06-25 | 2024-04-16 | 北京京东尚科信息技术有限公司 | Method for displaying trial assembly effect, trial assembly terminal and storage medium |
| CN110929615B (en) * | 2019-11-14 | 2022-10-18 | RealMe重庆移动通信有限公司 | Image processing method, image processing apparatus, storage medium, and terminal device |
| CN110971825A (en) * | 2019-12-06 | 2020-04-07 | 深圳前海达闼云端智能科技有限公司 | Image correction method, electronic device and storage medium |
| CN111340866B (en) * | 2020-02-26 | 2024-03-01 | 腾讯科技（深圳）有限公司 | Depth image generation method, device and storage medium |
| CN111402118B (en) * | 2020-03-17 | 2023-03-24 | 腾讯科技（深圳）有限公司 | Image replacement method and device, computer equipment and storage medium |
| CN111724422B (en) * | 2020-06-29 | 2024-01-09 | 深圳市慧鲤科技有限公司 | Image processing method and device, electronic equipment and storage medium |
| CN117408903A (en) * | 2020-10-13 | 2024-01-16 | 深圳市锐尔觅移动通信有限公司 | Image processing method and device, storage medium and electronic equipment |
| CN113807349B (en) * | 2021-09-06 | 2023-06-20 | 海南大学 | Multi-view target identification method and system based on Internet of things |
| CN115641487B (en) * | 2022-08-26 | 2023-06-27 | 中子时代(青岛)创新科技有限公司 | Multi-stage judgment fusion method and system based on neutrons and X rays |
| CN116630220B (en) * | 2023-07-25 | 2023-11-21 | 江苏美克医学技术有限公司 | Fluorescent image depth-of-field fusion imaging method, device and storage medium |
| CN116912148B (en) * | 2023-09-12 | 2024-01-05 | 深圳思谋信息科技有限公司 | Image enhancement method, device, computer equipment and computer readable storage medium |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1267117A1 (en) * | 2001-06-12 | 2002-12-18 | Ichikoh Industries, Ltd. | Headlamp |
| CN103745449A (en) * | 2013-12-24 | 2014-04-23 | 南京理工大学 | Rapid and automatic mosaic technology of aerial video in search and tracking system |
| CN104869387A (en) * | 2015-04-19 | 2015-08-26 | 中国传媒大学 | Method for acquiring binocular image maximum parallax based on optical flow method |
| CN105046676A (en) * | 2015-08-27 | 2015-11-11 | 上海斐讯数据通信技术有限公司 | Image fusion method and equipment based on intelligent terminal |
| CN105844630A (en) * | 2016-03-21 | 2016-08-10 | 西安电子科技大学 | Binocular visual image super-resolution fusion de-noising method |
| CN106803899A (en) * | 2015-11-26 | 2017-06-06 | 华为技术有限公司 | The method and apparatus for merging image |
| CN106997579A (en) * | 2016-01-26 | 2017-08-01 | 华为技术有限公司 | The method and apparatus of image mosaic |
| CN107451952A (en) * | 2017-08-04 | 2017-12-08 | 追光人动画设计（北京）有限公司 | A kind of splicing and amalgamation method of panoramic video, equipment and system |
| CN108171744A (en) * | 2017-12-26 | 2018-06-15 | 努比亚技术有限公司 | Determining method, mobile terminal and the storage medium of disparity map in a kind of binocular virtualization |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP4377932B2 (en) * | 2007-07-26 | 2009-12-02 | 株式会社モルフォ | Panorama image generating apparatus and program |
| CN101901481B (en) * | 2010-08-11 | 2012-11-21 | 深圳市蓝韵实业有限公司 | Image mosaic method |
| CN102446366B (en) * | 2011-09-14 | 2013-06-19 | 天津大学 | Time-space jointed multi-view video interpolation and three-dimensional modeling method |
| CN103971352A (en) * | 2014-04-18 | 2014-08-06 | 华南理工大学 | Rapid image splicing method based on wide-angle lenses |
| CN108833785B (en) * | 2018-07-03 | 2020-07-03 | 清华-伯克利深圳学院筹备办公室 | Fusion method and device of multi-view images, computer equipment and storage medium |
-
2018
- 2018-07-03 CN CN201810717536.3A patent/CN108833785B/en active Active
-
2019
- 2019-07-03 WO PCT/CN2019/094553 patent/WO2020007320A1/en active Application Filing
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1267117A1 (en) * | 2001-06-12 | 2002-12-18 | Ichikoh Industries, Ltd. | Headlamp |
| CN103745449A (en) * | 2013-12-24 | 2014-04-23 | 南京理工大学 | Rapid and automatic mosaic technology of aerial video in search and tracking system |
| CN104869387A (en) * | 2015-04-19 | 2015-08-26 | 中国传媒大学 | Method for acquiring binocular image maximum parallax based on optical flow method |
| CN105046676A (en) * | 2015-08-27 | 2015-11-11 | 上海斐讯数据通信技术有限公司 | Image fusion method and equipment based on intelligent terminal |
| CN106803899A (en) * | 2015-11-26 | 2017-06-06 | 华为技术有限公司 | The method and apparatus for merging image |
| CN106997579A (en) * | 2016-01-26 | 2017-08-01 | 华为技术有限公司 | The method and apparatus of image mosaic |
| CN105844630A (en) * | 2016-03-21 | 2016-08-10 | 西安电子科技大学 | Binocular visual image super-resolution fusion de-noising method |
| CN107451952A (en) * | 2017-08-04 | 2017-12-08 | 追光人动画设计（北京）有限公司 | A kind of splicing and amalgamation method of panoramic video, equipment and system |
| CN108171744A (en) * | 2017-12-26 | 2018-06-15 | 努比亚技术有限公司 | Determining method, mobile terminal and the storage medium of disparity map in a kind of binocular virtualization |
Non-Patent Citations (4)
| Title |
|---|
| Sampling-based depth recovery in dynamic environments with motion assistance;Jun Tan, et al;《2013 Chinese Automation Congress》;20131108;第522-526页 * |
| 数字图像远程集中监控系统设计;方璐;《电子工程技术与标准化》;20010228;第19-20页 * |
| 林枝叶.面向全景视频拼接的图像融合算法及其GPU实现.《中国优秀硕士学位论文全文数据库 信息科技辑》.2017,(第2期),第9-15、22-25、59-67页. * |
| 面向全景视频拼接的图像融合算法及其GPU实现;林枝叶;《中国优秀硕士学位论文全文数据库 信息科技辑》;20170215(第2期);第9-15、22-25、59-67页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108833785A (en) | 2018-11-16 |
| WO2020007320A1 (en) | 2020-01-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108833785B (en) | Fusion method and device of multi-view images, computer equipment and storage medium | |
| CN108898567B (en) | Image noise reduction method, device and system | |
| KR102480245B1 (en) | Automated generation of panning shots | |
| US10762655B1 (en) | Disparity estimation using sparsely-distributed phase detection pixels | |
| US9979909B2 (en) | Automatic lens flare detection and correction for light-field images | |
| US10511776B2 (en) | Image fusion method and apparatus, and terminal device | |
| KR102464523B1 (en) | Method and apparatus for processing image property maps | |
| WO2019221013A4 (en) | Video stabilization method and apparatus and non-transitory computer-readable medium | |
| CN113129241B (en) | Image processing method and device, computer readable medium and electronic equipment | |
| JP5766077B2 (en) | Image processing apparatus and image processing method for noise reduction | |
| CN112991242A (en) | Image processing method, image processing apparatus, storage medium, and terminal device | |
| CN111383254A (en) | Depth information acquisition method and system and terminal equipment | |
| CN110689565B (en) | Depth map determination method and device and electronic equipment | |
| JP2017103756A (en) | Image data processing apparatus and method | |
| CN113470103B (en) | Method and device for determining camera acting distance in vehicle-road cooperation and road side equipment | |
| CN112150355B (en) | Image processing method and related equipment | |
| US11227166B2 (en) | Method and device for evaluating images, operating assistance method, and operating device | |
| CN111630569B (en) | Binocular matching method, visual imaging device and device with storage function | |
| CN112308809A (en) | Image synthesis method and device, computer equipment and storage medium | |
| JP6381212B2 (en) | Imaging apparatus and control method thereof | |
| JP6292947B2 (en) | Image processing apparatus, image processing method, and program | |
| JP2023127450A (en) | Image processing device, image processing method, and program | |
| CN114845055A (en) | Method and device for determining shooting parameters of image acquisition equipment and electronic equipment | |
| CN116934608A (en) | Image processing method, device, electronic equipment and storage medium | |
| CN116958164A (en) | Image processing method, device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20221201 Address after: 518000 2nd floor, building a, Tsinghua campus, Shenzhen University Town, Xili street, Nanshan District, Shenzhen City, Guangdong Province Patentee after: Shenzhen International Graduate School of Tsinghua University Address before: 518055 Nanshan Zhiyuan 1001, Xue Yuan Avenue, Nanshan District, Shenzhen, Guangdong. Patentee before: TSINGHUA-BERKELEY SHENZHEN INSTITUTE |