WO2020007320A1 - 多视角图像的融合方法、装置、计算机设备和存储介质 - Google Patents

多视角图像的融合方法、装置、计算机设备和存储介质 Download PDF

Info

Publication number
WO2020007320A1
WO2020007320A1 PCT/CN2019/094553 CN2019094553W WO2020007320A1 WO 2020007320 A1 WO2020007320 A1 WO 2020007320A1 CN 2019094553 W CN2019094553 W CN 2019094553W WO 2020007320 A1 WO2020007320 A1 WO 2020007320A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
fused
sharpness
target object
feature map
Prior art date
Application number
PCT/CN2019/094553
Other languages
English (en)
French (fr)
Inventor
方璐
戴琼海
郑海天
Original Assignee
清华-伯克利深圳学院筹备办公室
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 清华-伯克利深圳学院筹备办公室 filed Critical 清华-伯克利深圳学院筹备办公室
Publication of WO2020007320A1 publication Critical patent/WO2020007320A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/95—Computational photography systems, e.g. light-field imaging systems
    • H04N23/951—Computational photography systems, e.g. light-field imaging systems by using two or more images to influence resolution, frame rate or aspect ratio
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction

Definitions

  • the present application relates to the field of image processing technology, and for example, to a method, an apparatus, a computer device, and a storage medium for multi-view image fusion.
  • Multi-view image acquisition devices generally use image fusion and synthesis algorithms to fuse the images acquired under multiple shooting angles to generate images of the target scene. Therefore, compared with the pictures collected by the single-camera image acquisition device, it has High resolution, low noise, large amount of information, etc.
  • multi-perspective graphics fusion needs to face the following challenges: parallax, occlusion, etc. between pictures collected at different perspectives, and the sensors and lens parameters of multi-cameras are different, resulting in the resolution, color and Differences in exposure, etc.
  • embodiments of the present application provide a method, an apparatus, a computer device, and a storage medium for multi-view image fusion, so as to avoid the situation that the image processing speed of the multi-view image acquisition device in the related art is slow.
  • an embodiment of the present application provides a multi-view image fusion method, including: acquiring a target object to be fused in a standard shooting angle and a reference image of the target object in a non-standard shooting angle; and calculating the target Optical flow information between the fused image and the reference image; and fused the image to be fused and the reference image according to the optical flow information to obtain a target image of the target object at a standard shooting angle of view.
  • an embodiment of the present application provides a multi-view image fusion apparatus, including: an image acquisition module configured to acquire a target object to be fused in a standard shooting angle and a reference of the target object in a non-standard shooting angle An image; an optical flow information calculation module configured to calculate optical flow information between the image to be fused and the reference image; an image fusion module configured to calculate the image to be fused and the reference according to the optical flow information The images are fused to obtain the target image of the target object at the standard shooting angle of view.
  • an embodiment of the present application provides a computer device including: at least one processor; a memory configured to store at least one program, and when the at least one program is executed by the at least one processor, the at least one program
  • One processor implements the method for fusing multi-view images according to the embodiments of the present application.
  • an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored.
  • the program is executed by a processor, the multi-view image fusion method according to the embodiment of the present application is implemented.
  • FIG. 1 is a schematic flowchart of a multi-view image fusion method according to an embodiment of the present application
  • FIG. 2 is a schematic flowchart of a multi-view image fusion method according to another embodiment of the present application.
  • FIG. 3 is a structural block diagram of a multi-view image fusion apparatus according to an embodiment of the present application.
  • FIG. 4 is a schematic structural diagram of a computer device according to an embodiment of the present application.
  • FIG. 1 is a schematic flowchart of a multi-view image fusion method according to an embodiment of the present application. As shown in FIG. 1, the method includes steps S101 to S103.
  • step S101 an image to be fused of the target object in a standard shooting angle and a reference image of the target object in a non-standard shooting angle are obtained.
  • the image to be fused and the reference image of the target object may be collected and generated by the local end, or may be obtained from other devices.
  • the target object can be photographed by an image acquisition device such as a camera, camera, or image sensor configured at the local end to obtain the target.
  • the image to be fused and the reference image of the object if the local end does not have a multi-view image acquisition function, for example, the local end is not configured with an image acquisition device such as a camera, camera, and image sensor, or the local end is configured with only one camera, camera, or image sensor
  • the acquisition device can acquire the target object's to-be-fused image and reference image through other multi-view image acquisition equipment.
  • the multi-view image acquisition device or the storage of the to-be-fused image and reference image of the target object can be stored.
  • the device obtains the image to be fused and the reference image for the current fusion.
  • the standard shooting angle and the non-standard shooting angle can be set in advance by the developer or the user, and can also be adjusted according to the imaging effect of the target object in multiple shooting angles during the shooting process.
  • the image to be fused and the reference image may be generated by different cameras in a multi-view image acquisition device or different cameras in a multi-camera image acquisition device (such as a mobile phone with a dual camera, etc.).
  • the image to be fused and the reference image may include only the target object, and may also include other scene backgrounds.
  • the resolutions of the image to be fused and the reference image may be the same or different, which is not limited herein. Considering the imaging effect of the target image obtained by the fusion, in one embodiment, the image to be fused and the reference image may have the same resolution.
  • step S102 optical flow information between the image to be fused and the reference image is calculated.
  • the optical flow is a parameter that characterizes the positional relationship of the target image in the time-varying image.
  • the optical flow information between the image to be fused and the reference image can characterize the position of multiple scenes in the image to be fused and the reference image. Correspondence between. For example, assuming that a certain position of the target object is imaged at the pixel A position of the image to be fused and the position is imaged at the pixel B position of the reference image, the optical flow information between the image to be fused and the reference image can be Characterize the relative positional relationship between pixel A and pixel B. For example, the position coordinates of pixel A in the image to be fused after moving several pixels in the x and y directions are the same as the position coordinates of pixel B in the reference image. .
  • the calculation method of the optical flow information between the image to be fused and the reference image can be set as required.
  • the image to be fused and the reference image may be input into a neural network model with an optical flow estimation function, and the optical flow information between the image to be fused and the reference image is determined based on the output value of the neural network;
  • the optical flow information between the image to be fused and the reference image can be calculated based on matching, frequency domain or gradient methods.
  • the main features of the target object can be located and tracked in the image to be fused and the reference image to determine the optical flow information, Locate similar areas in the image to be fused and the reference image and calculate optical flow information through the displacement of the similar areas, calculate the frequency domain information of the image to be fused and the reference image and determine the optical flow information based on the frequency domain information, or use the to be fused
  • the space-time differential calculation of the brightness of the image and the reference image sequence changes the target object from the image to be fused to the 2D velocity field of the reference image, and then determines the optical flow information between the image to be fused and the reference image, which is not limited here.
  • the optical flow information between the image to be fused and the reference image may only include the optical flow information between each pixel point of the image to be fused and each pixel point of the reference image, and may also include each area of the image to be fused. Optical flow information between the corresponding areas of the reference image.
  • the optical flow information between the image to be fused and the reference image contains multiple sub-reference information, and each sub-optical flow information can describe the difference between the pixels of the image to be fused and the reference image.
  • the pixel region can be obtained by segmenting or downsampling the image to be fused and the reference image.
  • the number of times of segmentation or downsampling of the image to be fused and the reference image can be set as required.
  • the current segmentation or downsampling can be performed on the basis of the original image to be fused and the reference image. It can be performed on the basis of the pixel area obtained by the last segmentation or downsampling, which is not limited here.
  • the image to be fused and the reference image may be downsampled a set number of times to obtain the to-be-fused Feature to-be-fused image of the image at different definitions and reference feature map of the reference image at different definitions; calculate sub-optical flow information between the feature-to-be-fused image and reference feature map with the same definition.
  • the feature map to be fused includes the feature map corresponding to the image to be fused (that is, the map image to be fused itself), and the reference feature map includes the feature map corresponding to the reference image (that is, the reference image itself); each sample can be obtained at the previous sampling
  • the first sampling coefficient used for sampling the reference image may be the same as the second sampling coefficient used for sampling the image to be fused, and the sampling is different.
  • the sampling coefficients used can be the same or different.
  • the feature map to be fused can be obtained by calculating the corresponding relationship between the pixels of the feature map to be fused and the reference feature map (that is, the feature map to be fused and the reference feature map with the same definition).
  • the sub-optical flow information between the reference feature map and the reference feature map is the optical flow information between the image to be fused and the reference image.
  • the image to be fused can be determined as the first
  • the feature map of the original to-be-fused image obtained by sampling at the zero level is sampled at a sampling step size of 2 to obtain the feature map of the first to-be-fused image obtained by sampling at the first layer;
  • a feature map of a to-be-fused image is sampled to obtain a second feature map of the second to-be-fused image obtained by sampling at the second layer, and the feature map of the second to-be-fused image is sampled at a sampling step of 2 to obtain a third obtained by sampling at the third layer.
  • the feature map of the to-be-fused image can thus be used to sample the image of the to-be-fused image; referring to the above process, each reference feature map of the reference image can be obtained, which will not be described in detail here.
  • step S103 the image to be fused and the reference image are fused according to the optical flow information to obtain a target image of the target object at a standard shooting angle of view.
  • the correspondence between the pixels of the image to be fused and the reference image can be determined according to the optical flow information between the image to be fused and the reference image (that is, the pixels with the same shooting content in the image to be fused and the reference image are determined Point pair), according to the color information (RGB information) of the pixel point with the corresponding relationship, the color of the target pixel after fusion is determined, thereby determining the color of the target pixel in the target image with the same position coordinates as the pixel in the image to be fused, In this way, the target image of the target object under the standard shooting angle can be obtained.
  • the weight value used when fusing the color of the pixel point pair with the object can be set as needed, for example, the color of the first pixel point of the image to be fused and the color of the second pixel point in the reference image can be set according to 1; 1, 1: 5, or 0: 1, etc., are not limited here.
  • the to-be-fused image and the reference image may be fused based on the sub-optical flow information, the feature map to be fused, and the reference image to obtain The target image of the target object at a standard shooting angle of view.
  • the feature map to be fused with the lowest definition and the reference feature map are first fused to obtain the first sub-target map;
  • a sub-target image is up-sampled to obtain the first to-be-fused image at the second low-resolution, and the first to-be-fused image is fused with the to-be-fused feature map and / or the reference feature map at the low-resolution to obtain the first
  • Two sub-objective maps, and so on, can be used to obtain the target image of the target object under the standard shooting angle of view.
  • the multi-view image fusion method obtaineds a to-be-fused image of a target object collected at a standard shooting angle and a reference image of a target object collected at a non-standard shooting angle, and calculates the to-be-fused image and the reference According to the optical flow information between the images, the to-be-fused image and the reference image are fused according to the calculated optical flow information to obtain a target image of the target object at a standard shooting angle of view.
  • the fusion speed of the multi-view image can be improved, and the manpower and material resources consumed in the image fusion process can be reduced.
  • FIG. 2 is a schematic flowchart of a multi-view image fusion method according to another embodiment of the present application. This embodiment is refined on the basis of the foregoing embodiment.
  • the feature map to be fused and the reference image are fused to obtain the target image of the target object at a standard shooting angle.
  • Optical flow information performs perspective correction on the reference feature map to correct the reference feature map to a correction map at a standard shooting angle of view; in order of sharpness from small to large, the to-be-fused images in each sharpness are sequentially processed.
  • the feature map and the correction map are fused to obtain a target image of the target object under a standard shooting angle of view.
  • the method before the acquiring an image to be fused of the target object in a standard shooting angle and a reference image of the target object in a non-standard shooting angle, the method further includes: obtaining an original image of the target object in a standard shooting angle, A first resolution of the original image and a second resolution of a reference image of the target object at a non-standard shooting angle of view; and based on a ratio between the second resolution and the first resolution, The image is converted into an image to be fused with a second resolution.
  • the method before the converting the original image into a to-be-fused image having a second resolution according to a ratio between the second resolution and the first resolution, the method further includes: determining The first resolution is lower than the second resolution.
  • the multi-view image fusion method provided in this embodiment may further include: shooting a target object with a multi-view camera to obtain an original image of the target object at a standard shooting angle and a target object at a non-standard shooting angle Reference image below.
  • the multi-view image fusion method provided in this embodiment includes steps S201 to S209.
  • step S201 a multi-angle camera is used to shoot the target object, and an original image of the target object at a standard shooting angle and a reference image of the target object at a non-standard shooting angle are obtained.
  • the multi-view camera can be any device with multiple shooting angles, such as a smart terminal with multiple cameras or other multi-view image acquisition devices.
  • the multi-view camera can be located outside the local end and establish a communication connection with the local end, or it can be integrated inside the local end, which is not limited here.
  • the local end can generate an image acquisition instruction when it detects that the current conditions meet the image acquisition conditions or when it detects that the user triggers an image acquisition request, and the image acquisition instruction is generated.
  • step S202 an original image of the target object in a standard shooting angle, a first resolution of the original image, and a second resolution of a reference image of the target object in a non-standard shooting angle are obtained.
  • the original image may be obtained from the local end or from a multi-view camera that captured the original image
  • the first resolution of the original image and the second resolution of the reference image may be based on the image description of the original image and the reference image.
  • the information can also be determined by analyzing the number of pixels in the row and column directions of the original image and the reference image or the size of the original image and the reference image, or based on the camera (camera or image sensor) and the reference image that captured the original image
  • the parameters of the camera (camera or image sensor) are determined, which is not limited in this embodiment.
  • step S203 it is determined that the first resolution is lower than the second resolution.
  • step S204 the original image is converted into an image to be fused with a second resolution according to a ratio between the second resolution and the first resolution.
  • the original image may be processed into a to-be-fused image with the same resolution as the reference image and subsequent fusion operations may be performed, or only the original image may be processed.
  • fusion processing is performed on the image to be fused and the reference image obtained after the original image resolution conversion, which is not limited here.
  • the first resolution of the original image may be smaller than the second resolution of the reference image, and / or the first resolution of the original image is equal to the reference.
  • fusion processing is performed on the to-be-fused image obtained by converting the original image and the reference image to improve the imaging effect of the target object.
  • steps S204 to S209 may be performed to obtain the target image; when the first resolution of the original image is equal to the second resolution of the reference image
  • the original image may be directly determined as the image to be fused without performing step S204, and steps S205 to S209 may be performed to obtain the target image.
  • the object to be subjected to the fusion processing may be determined as needed, such as the image to be fused (that is, the original image) All the pixels in the image are subjected to fusion processing, or only the occlusion regions of the target object in the fused image may be subjected to fusion processing to reduce the amount of calculation required in the fusion process.
  • the original image may be up-sampled according to the ratio between the second resolution and the first resolution to obtain a candidate image having the second resolution. Fusion image.
  • the upsampling method can be selected according to needs, and this embodiment does not limit this.
  • step S205 an image to be fused of the target object in a standard shooting angle and a reference image of the target object in a non-standard shooting angle are obtained.
  • step S206 the image to be fused and the reference image are down-sampled for a set number of times to obtain a feature map to be fused at different definitions of the image to be fused and the reference image at different definitions.
  • Reference feature map below.
  • step S207 the sub-optical flow information between the feature map to be fused and the reference feature map with the same definition is calculated.
  • step S208 a perspective correction is performed on the reference feature map according to the sub-optical flow information, so as to modify the reference feature map into a correction map under a standard shooting angle of view.
  • the shooting angle of the reference feature map may be corrected to a standard shooting angle according to the optical flow information to obtain a correction map.
  • the feature map to be fused corresponding to the reference feature map (that is, the feature map to be fused with the same definition as the reference feature map) may be determined first, and obtained Sub-optical flow information between the reference feature map and the feature map to be fused; and then based on the sub-optical flow information, it is determined that when the shooting angle of the reference feature map is converted into a standard shooting angle, each pixel needs to be in the row direction and the column direction, respectively.
  • the number of pixels moved, and then the corresponding pixels in the reference feature map are moved according to the number of pixels, so as to obtain the correction map of the reference feature map at the standard shooting angle of view.
  • step S209 the feature map to be fused and the correction map at each sharpness are sequentially fused in the order of the sharpness from small to large to obtain a target image of the target object at a standard shooting angle of view.
  • the feature maps to be fused and the correction maps at each sharpness can be fused in order to obtain the fused map at each sharpness, and then the highest definition of multiple fusion maps is used as the standard to
  • the fusion map with a lower degree is up-sampled to obtain multiple to-be-processed images with the same sharpness as the highest-resolution fusion map, and then the multiple to-be-processed images are fused again to obtain the target image at a standard shooting angle; or
  • the fusion processing is performed on the feature map to be fused at each sharpness, the correction map, and the intermediate map sampled on the fusion map obtained from the previous sharpness fusion to obtain the current sharpness.
  • the current current fusion map is inferred in turn until the current resolution does not have a next resolution higher than its own resolution. At this time, the target image after fusion is obtained, which is not limited here.
  • Obtaining a target image of a target object at a standard shooting angle may include: acquiring a current feature map at a current definition, the current feature map including a current feature map to be fused, a current correction map, and a current intermediate image, the current intermediate image Obtained by upsampling the last fused image obtained by fusing under the previous definition; performing fusion processing on the current feature map to obtain the current fused image; judging whether there is higher than the current definition in the ranking of sharpness and The next sharpness adjacent to the current sharpness is determined based on a judgment result that there is a next sharpness higher than the current sharpness and adjacent to the current sharpness in the sharpness ranking, The sharpness is determined as the current sharpness, and returns to perform the operation of obtaining the current feature map at the current sharpness
  • the current sharpness is the smallest sharpness in the sharpness ranking, that is, there is no previous sharpness lower than the current sharpness in the sharpness ranking, then only the current sharpness can be used.
  • the feature map to be fused and the reference feature map are fused to obtain the current fused image at the current resolution.
  • the above steps S202 to S209 in this embodiment may be performed by a local neural network model integrated, that is, an original image and a reference image obtained by a multi-view camera may be input to the neural network model, and the neural network model executes the foregoing Steps S202 to S209.
  • the output value of the neural network is the target object of the target object in the standard shooting angle.
  • multiple feature maps to be fused and multiple reference feature maps can exist as feature vectors in a neural network.
  • the feature vector of a feature map can describe the coordinate information and color information of multiple pixels of the feature map. That is, by rendering the feature vector, the feature map can be restored.
  • the multi-view image fusion method uses multiple perspectives to shoot the target object to obtain an original image and a reference image, and obtains a first resolution of the original image and a second resolution of the reference image. If the resolution is lower than the second resolution, the original image is converted into the image to be fused with the second resolution, and the image to be fused and the reference image are down-sampled a set number of times to obtain the feature map to be fused and the reference feature map. According to the sub-optical flow information between the feature map to be fused and the reference map, the reference feature map is modified into a correction map at a standard shooting angle of view. Fusion with the correction map to get the target image of the target object in the standard shooting angle.
  • the multi-view image fusion speed be increased, and the manpower and material resources consumed in the image fusion process can be reduced; the influence of resolution, hue, and / or exposure parameters on the image fusion effect can also be reduced.
  • the camera parameters need not be manually calibrated during the fusion process, which can avoid errors caused by inaccurate camera parameter calibration and improve the fusion effect of multi-perspective pictures.
  • the embodiment of the present application provides a multi-view image fusion device.
  • the device may be implemented by software and / or hardware.
  • the device may be integrated in a computer device with a multi-view image fusion function.
  • the multi-view image fusion device may be integrated in a multi-camera intelligent terminal.
  • fusion of a multi-perspective image can be achieved by performing a multi-perspective image fusion method.
  • FIG. 3 is a structural block diagram of a multi-view image fusion device according to an embodiment of the present application. As shown in FIG. 3, an image acquisition module 301, an optical flow information calculation module 302, and an image fusion module 303 are shown.
  • the image acquisition module 301 is configured to acquire a to-be-fused image of the target object in a standard shooting angle of view and a reference image of the target object in a non-standard shooting angle of view.
  • the optical flow information calculation module 302 is configured to calculate optical flow information between the image to be fused and the reference image.
  • the image fusion module 303 is configured to fuse the to-be-fused image and the reference image according to the optical flow information to obtain a target image of the target object at a standard shooting angle of view.
  • the multi-view image fusion device obtaineds a to-be-fused image of a target object collected at a standard shooting angle and a reference image of a target object collected at a non-standard shooting angle through an image acquisition module, and uses optical flow information
  • the calculation module calculates optical flow information between the to-be-fused image and the reference image
  • the image fusion module fuses the to-be-fused image and the reference image according to the calculated optical flow information to obtain a target image of the target object at a standard shooting angle of view.
  • the fusion speed of the multi-view image can be improved, and the manpower and material resources consumed in the image fusion process can be reduced.
  • the optical flow information calculation module 302 may include: a down-sampling unit configured to perform down-sampling for the set number of times for the image to be fused and the reference image to obtain that the image to be fused is different in different The feature map to be fused in sharpness and the reference feature map of the reference image in different sharpness; the optical flow information calculation unit is configured to calculate the sub-light between the feature map to be fused and the reference feature map with the same sharpness
  • the image fusion module 303 may be configured to: based on the sub-optical flow information, the feature map to be fused, and the reference image, fuse the to-be-fused image and the reference image to obtain The target image of the target object at a standard shooting angle of view.
  • the image fusion module 303 may include a perspective correction unit configured to perform perspective correction on the reference feature map according to the sub-optical flow information, so as to modify the reference feature map to a standard shooting angle
  • the correction image below; the image fusion unit is set to sequentially fuse the feature image to be fused and the correction image at each resolution in order of sharpness from small to large to obtain a target image of the target object at a standard shooting angle of view.
  • the image fusion unit may include: a feature map acquisition subunit configured to obtain a current feature map at a current definition, where the current feature map includes a current feature map to be fused, a current correction map, and a current intermediate image
  • the current intermediate image is obtained by upsampling a previous fused image obtained by fusing under a previous definition; a feature map fusion subunit is configured to perform a fusion process on the current feature map to obtain a current fused image; a target
  • the image determination subunit is configured to determine whether there is a next sharpness higher than the current sharpness and adjacent to the current sharpness in the sharpness ranking, and based on the sharpness ranking, there is a higher sharpness than the current sharpness and A judgment result of a next sharpness adjacent to the current sharpness, determining the next sharpness as the current sharpness, and returning to perform an operation of acquiring a current feature map at the current sharpness; There is no judgment result that is higher than the current sharpness and is next to the current sharpness.
  • the multi-view image fusion apparatus may further include: a resolution acquisition module configured to acquire the to-be-fused image of the target object in a standard shooting angle and the target object in a non-standard shooting Before a reference image in a viewing angle, obtaining an original image of the target object in a standard shooting angle, a first resolution of the original image, and a second resolution of a reference image of the target in a non-standard shooting angle; an image conversion module, It is set to convert the original image into an image to be fused with a second resolution according to a ratio between the second resolution and the first resolution.
  • a resolution acquisition module configured to acquire the to-be-fused image of the target object in a standard shooting angle and the target object in a non-standard shooting Before a reference image in a viewing angle, obtaining an original image of the target object in a standard shooting angle, a first resolution of the original image, and a second resolution of a reference image of the target in a non-standard shooting angle
  • an image conversion module It is set to convert the
  • the multi-view image fusion apparatus may further include: a determining module configured to set the selected image according to a ratio between the second resolution and the first resolution. Before the original image is converted into an image to be fused with a second resolution, it is determined that the first resolution is lower than the second resolution.
  • the multi-view image fusion device may further include: an image acquisition module configured to use a multi-view camera to shoot the target object to obtain the original image and target of the target object in a standard shooting angle A reference image of the subject at a non-standard shooting angle.
  • the multi-view image fusion apparatus provided by the embodiment of the present application can execute the multi-view image fusion method provided by any embodiment of the present application, and has corresponding function modules for performing the multi-view image fusion method.
  • the multi-view image fusion apparatus can execute the multi-view image fusion method provided by any embodiment of the present application, and has corresponding function modules for performing the multi-view image fusion method.
  • FIG. 4 is a schematic structural diagram of a device / terminal / server provided by an embodiment of the present application.
  • the device / terminal / server includes a processor 40 and a memory 41, and may further include an input device 42 and an output device 43.
  • the number of processors 40 in the device / terminal / server may be one or more, and one processor 40 is taken as an example in FIG. 4; the processor 40, the memory 41, the input device 42 and the output device in the device / terminal / server 43 may be connected through a bus or other methods. In FIG. 4, connection through a bus is taken as an example.
  • the memory 41 is a computer-readable storage medium, and may be configured to store software programs, computer-executable programs, and modules, such as program instructions / modules corresponding to the multi-view image fusion method in the embodiments of the present application (for example, multi-view images (The image acquisition module 301, the optical flow information calculation module 302, and the image fusion module 303) in the fusion device.
  • the processor 40 executes multiple functional applications and data processing of the device / terminal / server by running software programs, instructions, and modules stored in the memory 41, that is, implementing the above-mentioned multi-view image fusion method.
  • the memory 41 may mainly include a storage program area and a storage data area, where the storage program area may store an operating system and application programs required for at least one function; the storage data area may store data created according to the use of the terminal, and the like.
  • the memory 41 may include a high-speed random access memory, and may further include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage device.
  • the memory 41 may include memory remotely set with respect to the processor 40, and these remote memories may be connected to the device / terminal / server through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
  • the input device 42 may be configured to receive inputted numeric or character information and generate key signal inputs related to user settings and function control of the device / terminal / server.
  • the output device 43 may include a display device such as a display screen.
  • An embodiment of the present application further provides a storage medium containing computer-executable instructions.
  • the method is configured to perform a multi-view image fusion method.
  • the method includes: The image to be fused in a standard shooting angle and a reference image of the target object in a non-standard shooting angle; calculating optical flow information between the to-be-fused image and the reference image; and comparing the to-be-fused image according to the optical flow information
  • the image and the reference image are fused to obtain a target image of the target object under a standard shooting angle of view.
  • a storage medium containing computer-executable instructions provided in the embodiments of the present application is not limited to the method operations described above, and may also perform fusion of multi-view images provided by any embodiment of the present application. Related operations in the method.
  • the present application can be implemented by software and necessary general-purpose hardware, and of course, can also be implemented by hardware.
  • the technical solution of this application that is essential or contributes to related technologies can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as a computer floppy disk, Read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including several instructions to make a computer device (can be a personal computer, A server, or a network device, etc.) executes the method described in the embodiment of the present application.
  • a computer-readable storage medium such as a computer floppy disk, Read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc.
  • the multiple units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized
  • the specific names of the multiple functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the present application.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computing Systems (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Image Processing (AREA)
  • Studio Devices (AREA)

Abstract

本申请公开了一种多视角图像的融合方法、装置、计算机设备和存储介质。该方法包括：获取目标对象在标准拍摄视角下的待融合图像以及目标对象在非标准拍摄视角下的参考图像；计算所述待融合图像和所述参考图像之间的光流信息；根据所述光流信息对所述待融合图像和所述参考图像进行融合，得到目标对象在标准拍摄视角下的目标图像。

Description

多视角图像的融合方法、装置、计算机设备和存储介质
本申请要求在2018年07月03日提交中国专利局、申请号为201810717536.3的中国专利申请的优先权，该申请的全部内容通过引用结合在本申请中。
技术领域
本申请涉及图像处理技术领域，例如涉及一种多视角图像的融合方法、装置、计算机设备和存储介质。
背景技术
近年来，随着摄像传感器成本的降低，多相机、多摄像头乃至密集阵列的多视角图像采集设备逐渐成为了图像采集的主流趋势。
多视角图像采集设备一般通过图像融合合成算法，对其在多种拍摄视角下采集得到的图像进行融合生成目标景物的图像，因此，其相比于单相机图像采集设备采集的图片而言，具有分辨率高、噪声小、信息量大等优点。目前，在进行多视角图形融合时需要面临以下挑战：不同视角下采集的图片间存在视差、遮挡等情况，多相机的传感器与镜头参数不同，导致不同视角下采集的图片存在分辨率、颜色和曝光程度等差异，等等。
但是，申请人在实现本申请的过程中发现：相关技术在面对上述挑战时，通常需要通过人工标定的方式对图像进行处理，处理速度较慢，且需要浪费较长的人力与物力。
发明内容
以下是对本文详细描述的主题的概述。本概述并非是为了限制权利要求的保护范围。
有鉴于此，本申请实施例提供一种多视角图像的融合方法、装置、计算机设备和存储介质，以避免相关技术中多视点图像采集设备图像处理速度较慢的情况。
第一方面，本申请实施例提供了一种多视角图像的融合方法，包括：获取目标对象在标准拍摄视角下的待融合图像以及目标对象在非标准拍摄视角下的 参考图像；计算所述待融合图像和所述参考图像之间的光流信息；根据所述光流信息对所述待融合图像和所述参考图像进行融合，得到目标对象在标准拍摄视角下的目标图像。
第二方面，本申请实施例提供了一种多视角图像的融合装置，包括：图像获取模块，设置为获取目标对象在标准拍摄视角下的待融合图像以及目标对象在非标准拍摄视角下的参考图像；光流信息计算模块，设置为计算所述待融合图像和所述参考图像之间的光流信息；图像融合模块，设置为根据所述光流信息对所述待融合图像和所述参考图像进行融合，得到目标对象在标准拍摄视角下的目标图像。
第三方面，本申请实施例提供了一种计算机设备，包括：至少一个处理器；存储器，设置为存储至少一个程序，当所述至少一个程序被所述至少一个处理器执行，使得所述至少一个处理器实现如本申请实施例所述的多视角图像的融合方法。
第四方面，本申请实施例还提供了一种计算机可读存储介质，，其上存储有计算机程序，该程序被处理器执行时实现如本申请实施例所述的多视角图像的融合方法。
在阅读并理解了附图和详细描述后，可以明白其他方面。
附图说明
通过阅读参照以下附图所作的对非限制性实施例所作的详细描述：
图1为本申请实施例提供的一种多视角图像的融合方法的流程示意图；
图2为本申请另一实施例提供的一种多视角图像的融合方法的流程示意图；
图3为本申请实施例提供的一种多视角图像的融合装置的结构框图；
图4为本申请实施例提供的一种计算机设备的结构示意图。
具体实施方式
下面结合附图和实施例对本申请作详细说明。可以理解的是，此处所描述的具体实施例仅仅用于解释本申请，而非对本申请的限定。另外还需要说明的是，为了便于描述，附图中仅示出了与本申请相关的部分而非全部内容。
本申请实施例提供一种多视角图像的融合方法。该方法可以由多视角图像的融合装置执行，其中，该装置可以由软件和/或硬件实现，一般可集成在具有多视角图像融合功能的计算机设备中，典型的，可以集成在多摄像头智能终端或多视角图像采集设备中。图1是本申请实施例提供的一种多视角图像的融合方法的流程示意图，如图1所示，该方法包括步骤S101至步骤S103。
在步骤S101中，获取目标对象在标准拍摄视角下的待融合图像以及目标对象在非标准拍摄视角下的参考图像。
本实施例中，目标对象的待融合图像和参考图像可以通过本端采集生成，也可以从其他设备中获取。考虑到图像融合的及时性，在一实施例中，如果本端具有多视角图像采集功能，则可以通过本端配置的相机、摄像头或图像传感器等图像采集装置对目标对象进行拍摄，以得到目标对象的待融合图像和参考图像；如果本端不具有多视角图像采集功能，如本端未配置相机、摄像头和图像传感器等图像采集装置或者本端只配置有一个相机、摄像头或图像传感器等图像采集装置，则可以通过其他多视角图像采集设备采集目标对象的待融合图像和参考图像，此时，相应的，可以从多视角图像采集设备或者存储有目标对象的待融合图像及参考图像的存储设备中获取本次融合的待融合图像和参考图像。
其中，标准拍摄视角和非标准拍摄视角可以由开发商或用户预先进行设置，也可以在拍摄过程中根据多拍摄视角下目标对象的成像效果进行调整。待融合图像和参考图像可以由多视角图像采集设备中的不同相机或多摄像头图像采集设备(如具有双摄像头的手机等)中的不同摄像头对目标对象进行拍摄生成。待融合图像和参考图像中可以仅包含目标对象，也可以包含其他的景物背景。待融合图像和参考图像的分辨率可以相同或不同，此处不作限制。考虑到融合得到的目标图像的成像效果，在一实施例中，待融合图像和参考图像可以具有相同的分辨率。
在步骤S102中，计算所述待融合图像和所述参考图像之间的光流信息。
本实施例中，光流为表征目标图像在时变图像中的位置关系的参数，待融合图像和参考图像之间的光流信息可以表征多个景物在待融合图像和参考图像中的位置之间的对应关系。举例而言，假设目标对象的某一个位置在待融合图像的像素点A位置处成像，该位置在参考图像的像素点B位置处成像，那么待 融合图像和参考图像之间的光流信息可以表征像素点A与像素点B的相对位置关系，如像素点A分别在x和y方向上移动几个像素点后在待融合图像中的位置坐标与像素点B在参考图像中的位置坐标相同。
本步骤中，待融合图像和参考图像之间的光流信息的计算方法可以根据需要设置。在一实施例中，可以将待融合图像和参考图像输入到具有光流估计功能的神经网络模型中，并基于该神经网络的输出值确定待融合图像和参考图像之间的光流信息；也可以基于匹配、频域或梯度的方法计算待融合图像和参考图像之间的光流信息，如可以在待融合图像和参考图像中分别对目标对象的主要特征进行定位和跟踪确定光流信息、对待融合图像和参考图像中的类似区域进行定位并通过相似区域的位移计算光流信息、计算待融合图像和参考图像的频域信息并基于该频域信息确定光流信息，或者，利用待融合图像和参考图像序列亮度的时空微分计算目标对象由待融合图像变化至参考图像的2D速度场，进而确定待融合图像和参考图像之间的光流信息，此处不作限制。
在此，需要说明的是，待融合图像和参考图像之间的光流信息可以仅仅包含待融合图像各像素点和参考图像各像素点之间的光流信息，还可以包含待融合图像各区域与参考图像相应区域之间的光流信息，此时，待融合图像和参考图像之间的光流信息包含多个子参考信息，每个子光流信息可以描述待融合图像和参考图像各像素点之间的光流信息或者待融合图像的各像素区域和参考图像对应像素区域之间的光流信息。其中，像素区域可以通过对待融合图像及参考图像进行分割或下采样获得。对待融合图像和参考图像进行分割或下采样的次数可以根据需要设置，当分割或下采样次数为多次时，当前次分割或下采样可以在原始待融合图像和参考图像的基础上进行，也可以在上次分割或下采样得到的像素区域的基础上进行，此处不作限制。
以通过下采样得到待融合图像和参考图像的像素区域为例，在一实施例中，可以首先分别对所述待融合图像和所述参考图像进行设定次数的下采样，得到所述待融合图像在不同清晰度下的待融合特征图和所述参考图像在不同清晰度下的参考特征图；计算具有相同清晰度的待融合特征图和参考特征图之间的子光流信息。其中，待融合特征图包括待融合图图像对应的特征图(即待融合图图像自身)，参考特征图包括参考图像对应的特征图(即参考图像自身)；每次采样可以在上次采样得到的待融合特征图和参考特征图的基础上进行，同次采 样中，对参考图像进行采样所使用的第一采样系数可以与对待融合图像进行采样所使用的第二采样系数相同，不同次采样所使用的采样系数可以相同或不同。此时，相应的，可以通过计算同次采样得到的待融合特征图和参考特征图(即具有相同清晰度的待融合特征图和参考特征图)各像素之间的对应关系得到待融合特征图和参考特征图之间的子光流信息，即得到待融合图像和参考图像之间的光流信息。
示例性的，假设每次采样所使用的采样系数均为1/2(即采样步长为2)，采样次数为4，则在对待融合图像进行采样时，可以将待融合图像自身确定为第零层采样得到的原始待融合特征图，以采样步长2对该原始待融合图特征图进行采样，得到第一层采样得到的第一待融合图特征图；以采样步长2对该第一待融合图特征图进行采样，得到第二层采样得到的第二待融合图特征图，以采样步长2对该第二待融合图特征图进行采样，得到第三层采样得到的第三待融合图特征图，由此，即可以实现对待融合图图像的采样；参照上述过程，可以得到参考图像的各参考特征图，此处不再详述。
在步骤S103中，根据所述光流信息对所述待融合图像和所述参考图像进行融合，得到目标对象在标准拍摄视角下的目标图像。
在一实施例中，可以根据待融合图像与参考图像之间的光流信息确定待融合图像和参考图像各像素点之间的对应关系(即决定待融合图像和参考图像中拍摄内容相同的像素点对)，根据具有对应关系的像素点的颜色信息(RGB信息)确定融合后的目标像素的颜色，从而确定目标图像中与待融合图像中该像素具有相同位置坐标的目标像素的颜色，由此，即可以得到目标对象在标准拍摄视角下的目标图像。其中，在对具有对象的像素点对的颜色进行融合时所采用的权重值可以根据需要设置，如可以将待融合图像的第一像素点的颜色与参考图像中的第二像素点的颜色按照1；1、1：5或0：1等进行融合，此处不作限制。
示例性的，当参考信息包含多个子参考信息时，可以基于所述子光流信息、所述待融合特征图和所述参考图像，对所述待融合图像和所述参考图像进行融合，得到目标对象在标准拍摄视角下的目标图像。如可以按照清晰度由低到高的顺序，首先对清晰度最低的待融合特征图和参考特征图进行融合，得到第一子目标图；按照下采样或分割时所使用系数的倒数对该第一子目标图进行上采样得到次低清晰度下的第一待融合图，将该第一待融合图与该次低清晰度下的 待融合特征图和/或参考特征图进行融合，得到第二子目标图，依次类推，即可得到目标对象在标准拍摄视角下的目标图像。
本申请实施例提供的多视角图像的融合方法，获取在标准拍摄角度下采集的目标对象的待融合图像以及在非标准拍摄角度下采集的目标对象的参考图像，计算该待融合图像和该参考图像之间的光流信息，根据计算得到的光流信息对待融合图像和参考图像进行融合，得到目标对象在标准拍摄视角下的目标图像。本实施例通过采用上述技术方案，能够提高多视角图像的融合速度，减少图像融合过程中所耗费的人力与物力。
图2为本申请实另一施例提供的一种多视角图像的融合方法的流程示意图，本实施例在上述实施例的基础上进行细化，在本实施例中，将“基于所述子光流信息、所述待融合特征图和所述参考图像，对所述待融合图像和所述参考图像进行融合，得到目标对象在标准拍摄视角下的目标图像”细化为：根据所述子光流信息对所述参考特征图进行视角修正，以将所述参考特征图修正为在标准拍摄视角下的修正图；按照清晰度从小到大的顺序，依次对每个清晰度下的待融合特征图和修正图进行融合，得到目标对象在标准拍摄视角下的目标图像。
在一实施例中，在所述获取目标对象在标准拍摄视角下的待融合图像以及目标对象在非标准拍摄视角下的参考图像之前，还包括：获取目标对象在标准拍摄视角下的原始图像、所述原始图像的第一分辨率以及目标对象在非标准拍摄视角下的参考图像的第二分辨率；根据所述第二分辨率和所述第一分辨率之间的比值，将所述原始图像转换为具有第二分辨率的待融合图像。
在一实施例中，在所述根据所述第二分辨率和所述第一分辨率之间的比值，将所述原始图像转换为具有第二分辨率的待融合图像之前，还包括：确定所述第一分辨率低于所述第二分辨率。
在一实施例中，本实施例提供的多视角图像的融合方法还可以包括：采用多视角相机对目标对象进行拍摄，得到目标对象在标准拍摄视角下的原始图像和目标对象在非标准拍摄视角下的参考图像。
如图2所示，本实施例提供的多视角图像的融合方法包括步骤S201至步骤S209。
在步骤S201中，采用多视角相机对目标对象进行拍摄，得到目标对象在标 准拍摄视角下的原始图像和目标对象在非标准拍摄视角下的参考图像。
其中，多视角相机可以为任意具有多个拍摄视角的设备，如具有多个摄像头的智能终端或其他多视角图像采集设备等。该多视角相机可以位于本端外部且与本端建立通信连接，也可以集成于本端内部，此处不作限制。示例性的，当多视角相机独立于本端存在时，本端可以在检测到当前条件符合图像的采集条件时或在检测到用户触发图像采集请求时生成图像采集指令，并将该图像采集指令发送给与本端建立连接的多视角相机，从而控制多视角相机对目标对象进行拍摄，得到目标对象的原始图像和参考图像；当多视角相机集成于本端内部时，可以在检测到当前符合图像的采集条件时或检测到用户触发图像采集请求时直接控制多视角相机采集目标对象的原始图像和参考图像。
在步骤S202中，获取目标对象在标准拍摄视角下的原始图像、所述原始图像的第一分辨率以及目标对象在非标准拍摄视角下的参考图像的第二分辨率。
在一实施例中，原始图像可以从本端或从拍摄该原始图像的多视角相机中获取，原始图像的第一分辨率和参考图像的第二分辨率可以基于原始图像和参考图像的图像描述信息确定，也可以通过分析原始图像和参考图像行方向和列方向上的像素个数或者原始图像和参考图像的大小确定，或者，基于拍摄原始图像的相机(摄像头或图像传感器)和拍摄参考图像的相机(摄像头或图像传感器)的参数确定，本实施例并不对此进行限制。
在步骤S203中，确定所述第一分辨率低于所述第二分辨率。
在步骤S204中，根据所述第二分辨率和所述第一分辨率之间的比值，将所述原始图像转换为具有第二分辨率的待融合图像。
本实施例中，可以不考虑原始图像和参考图像的分辨率之间的大小，将原始图像处理为与参考图像具有相同分辨率的待融合图像并执行后续的融合操作，也可以仅在原始图像的分辨率小于或等于待融合图像的分辨率时对原始图像分辨率转换后得到的待融合图像与参考图像进行融合处理，此处不作限制。
考虑到融合处理所生成目标图像的实用性，在一实施例中，可以仅在原始图像的第一分辨率小于参考图像的第二分辨率，和/或，原始图像的第一分辨率等于参考图像的第二分辨率且目标对象在原始图像中的成像存在遮挡时对原始图像转换得到的待融合图像与参考图像进行融合处理，以提高目标对象的成像效果。此时，当原始图像的第一分辨率小于参考图像的第二分辨率时，可以执 行步骤S204-步骤S209，以得到目标图像；当原始图像的第一分辨率等于参考图像的第二分辨率且目标图像在原始图像中的成像存在遮挡时，可以不执行步骤S204，直接将原始图像确定为待融合图像，并执行步骤S205-步骤S209，得到目标图像。
针对原始图像的第一分辨率等于参考图像的第二分辨率且目标图像在原始图像中的成像存在遮挡的情况，进行融合处理的对象可以根据需要确定，如可以对待融合图像(即原始图像)中的全部像素点均进行融合处理，也可以仅对待融合图像中目标对象的遮挡区域进行融合处理，以减少融合过程中所需的计算量。针对原始图像的第一分辨率小于参考图像的第二分辨率的情况，可以根据第二分辨率和第一分辨率之间的比值对原始图像进行上采样，以得到具有第二分辨率的待融合图像。在此，上采样方法可以根据需要选取，本实施例并不对此进行限制。
在步骤S205中，获取目标对象在标准拍摄视角下的待融合图像以及目标对象在非标准拍摄视角下的参考图像。
在步骤S206中，分别对所述待融合图像和所述参考图像进行设定次数的下采样，得到所述待融合图像在不同清晰度下的待融合特征图和所述参考图像在不同清晰度下的参考特征图。
在步骤S207中，计算具有相同清晰度的待融合特征图和参考特征图之间的子光流信息。
在步骤S208中，根据所述子光流信息对所述参考特征图进行视角修正，以将所述参考特征图修正为在标准拍摄视角下的修正图。
本实施例中，可以根据光流信息将参考特征图的拍摄视角修正为标准拍摄视角，以得到修正图。此时，示例性的，就某一参考特征图而言，可以首先确定与该参考特征图对应的待融合特征图(即与该参考特征图具有相同清晰度的待融合特征图)，并获取该参考特征图与该待融合特征图之间的子光流信息；然后根据该子光流信息确定参考特征图的拍摄视角转换为标准拍摄视角时各像素点在行方向和列方向上分别需要移动的像素点数，然后根据各像素点数对参考特征图中对应的像素点进行移动，从而得到参考特征图在标准拍摄视角下的修正图。
在步骤S209中，按照清晰度从小到大的顺序，依次对每个清晰度下的待融 合特征图和修正图进行融合，得到目标对象在标准拍摄视角下的目标图像。
本实施例中，可以依次对每个清晰度下的待融合特征图和修正图进行融合，得到每个清晰度下的融合图，然后以多个融合图中的最高清晰度为标准，对清晰度较低的融合图进行上采样，得到与分辨率最高的融合图清晰度相同的多个待处理图像，然后再次对多个待处理图像进行融合以得到标准拍摄视角下的目标图像；也可以按照清晰度从小到大的顺序，对每个清晰度下的待融合特征图、修正图以及对在上一清晰度融合得到的融合图上采样得到的中间图进行融合处理，以得到当前清晰度下的当前融合图，依次类推，直至当前清晰度不存在比自身清晰度高的下一清晰度为止，此时，即可得到融合后的目标图像，此处不作限制。
为了提高目标对象在融合得到的目标图像中的成像效果，在一实施例中，所述按照清晰度从小到大的顺序，依次对每个清晰度下的待融合特征图和修正图进行融合，得到目标对象在标准拍摄视角下的目标图像，可以包括：获取当前清晰度下的当前特征图，所述当前特征图包括当前待融合特征图、当前修正图和当前中间图像，所述当前中间图像通过对在上一清晰度下融合得到的上一融合图像进行上采样获得；对所述当前特征图进行融合处理，得到当前融合图像；判断清晰度排序中是否存在高于所述当前清晰度且与所述当前清晰度相邻的下一清晰度，基于清晰度排序中存在高于所述当前清晰度且与所述当前清晰度相邻的下一清晰度的判断结果，将所述下一清晰度确定为当前清晰度，并返回执行获取当前清晰度下的当前特征图的操作；基于清晰度排序中不存在高于所述当前清晰度且与所述当前清晰度相邻的下一清晰度的判断结果，将所述当前融合图像确定为目标对象在标准拍摄视角下的目标图像。在此，需要指出的是，如果当前清晰度为清晰度排序中的最小清晰度，即清晰度排序中不存在清晰度低于当前清晰度的上一清晰度，则可以仅对当前清晰度下待融合特征图和参考特征图进行融合得到当前清晰度下的当前融合图像。
需要说明的是，本实施例上述步骤S202至步骤S209可以由集成本端的神经网络模型进行执行，即可以向神经网络模型输入多视角相机拍摄得到的原始图像和参考图像，由神经网络模型执行上述步骤S202至步骤S209，此时，神经网络的输出值即为目标对象在标准拍摄视角下的目标对象。其中，多个待融合特征图和多个参考特征图在神经网络中可以以特征向量的方式存在，某一特征 图的特征向量可以描述该特征图的多个像素点的坐标信息和颜色信息，即通过渲染该特征向量，即可将该特征图还原。
本申请实施例提供的多视角图像的融合方法，采用多视角相对目标对象进行拍摄得到原始图像和参考图像，获取该原始图像的第一分辨率和该参考图像的第二分辨率，如果第一分辨率低于第二分辨率，则将原始图像转换为具有第二分辨率的待融合图像，分别对待融合图像和参考图像进行设定次数的下采样，得到待融合特征图和参考特征图，根据待融合特征图和参考图之间的子光流信息将参考特征图修正为标准拍摄视角下的修正图，按照清晰度从小到大的顺序，依次对每个清晰度下的待融合特征图和修正图进行融合，得到目标对象在标准拍摄视角下的目标图像。本实施例通过采用上述技术方案，不但可以提高多视角图像的融合速度，减少图像融合过程中所耗费的人力与物力；还可以减少辨率、色调和/或曝光参数对图片融合效果的影响，且融合过程中无需人工对相机参数进行标定，能够避免相机参数标定不准所导致的误差，提高多视角图片的融合效果。
本申请实施例提供一种多视角图像的融合装置，该装置可以由软件和/或硬件实现，一般可集成在具有多视角图像融合功能的计算机设备中，典型的，可以集成在多摄像头智能终端或多视角图像采集设备中，可通过执行多视角图像的融合方法实现对多视角图像的融合。图3为本申请实施例提供的多视角图像融合装置的结构框图，如图3所示图像获取模块301，光流信息计算模块302和图像融合模块303。
图像获取模块301，设置为获取目标对象在标准拍摄视角下的待融合图像以及目标对象在非标准拍摄视角下的参考图像。
光流信息计算模块302，设置为计算所述待融合图像和所述参考图像之间的光流信息。
图像融合模块303，设置为根据所述光流信息对所述待融合图像和所述参考图像进行融合，得到目标对象在标准拍摄视角下的目标图像。
本申请实施例提供的多视角图像的融合装置，通过图像获取模块获取在标准拍摄角度下采集的目标对象的待融合图像以及在非标准拍摄角度下采集的目标对象的参考图像，通过光流信息计算模块计算该待融合图像和该参考图像之 间的光流信息，通过图像融合模块根据计算得到的光流信息对待融合图像和参考图像进行融合，得到目标对象在标准拍摄视角下的目标图像。本实施例通过采用上述技术方案，能够提高多视角图像的融合速度，减少图像融合过程中所耗费的人力与物力。
在上述方案中，所述光流信息计算模块302可以包括：下采样单元，设置为分别对所述待融合图像和所述参考图像进行设定次数的下采样，得到所述待融合图像在不同清晰度下的待融合特征图和所述参考图像在不同清晰度下的参考特征图；光流信息计算单元，设置为计算具有相同清晰度的待融合特征图和参考特征图之间的子光流信息；相应的，所述图像融合模块303可以设置为：基于所述子光流信息、所述待融合特征图和所述参考图像对所述待融合图像和所述参考图像进行融合，得到目标对象在标准拍摄视角下的目标图像。
在上述方案中，所述图像融合模块303可以包括：视角修正单元，设置为根据所述子光流信息对所述参考特征图进行视角修正，以将所述参考特征图修正为在标准拍摄视角下的修正图；图像融合单元，设置为按照清晰度从小到大的顺序，依次对每个清晰度下的待融合特征图和修正图进行融合，得到目标对象在标准拍摄视角下的目标图像。
在上述方案中，所述图像融合单元可以包括：特征图获取子单元，设置为获取当前清晰度下的当前特征图，所述当前特征图包括当前待融合特征图、当前修正图和当前中间图像，所述当前中间图像通过对在上一清晰度下融合得到的上一融合图像进行上采样获得；特征图融合子单元，设置为对所述当前特征图进行融合处理，得到当前融合图像；目标图像确定子单元，设置为判断清晰度排序中是否存在高于所述当前清晰度且与所述当前清晰度相邻的下一清晰度，基于清晰度排序中存在高于所述当前清晰度且与所述当前清晰度相邻的下一清晰度的判断结果，将所述下一清晰度确定为当前清晰度，并返回执行获取当前清晰度下的当前特征图的操作；基于清晰度排序中不存在高于所述当前清晰度且与所述当前清晰度相邻的下一清晰度的判断结果，将所述当前融合图像确定为目标对象在标准拍摄视角下的目标图像。
在一实施例中，本实施例提供的多视角图像的融合装置还可以包括：分辨率获取模块，设置为在所述获取目标对象在标准拍摄视角下的待融合图像以及目标对象在非标准拍摄视角下的参考图像之前，获取目标对象在标准拍摄视角 下的原始图像、所述原始图像的第一分辨率以及目标对象在非标准拍摄视角下的参考图像的第二分辨率；图像转换模块，设置为根据所述第二分辨率和所述第一分辨率之间的比值，将所述原始图像转换为具有第二分辨率的待融合图像。
在一实施例中，本实施例提供的多视角图像的融合装置还可以包括：确定模块，设置为在所述根据所述第二分辨率和所述第一分辨率之间的比值，将所述原始图像转换为具有第二分辨率的待融合图像之前，确定所述第一分辨率低于所述第二分辨率。
在一实施例中，本实施例提供的多视角图像的融合装置还可以包括：图像采集模块，设置为采用多视角相机对目标对象进行拍摄，得到目标对象在标准拍摄视角下的原始图像和目标对象在非标准拍摄视角下的参考图像。
本申请实施例提供的多视角图像的融合装置可执行本申请任意实施例提供的多视角图像的融合方法，具备执行多视角图像的融合方法相应的功能模块。未在本实施例中详尽描述的技术细节，可参见本申请任意实施例所提供的多视角图像的融合方法。
图4为本申请实施例提供的一种设备/终端/服务器的结构示意图，如图4所示，该设备/终端/服务器包括处理器40和存储器41，还可以包括输入装置42和输出装置43；设备/终端/服务器中处理器40的数量可以是一个或多个，图4中以一个处理器40为例；设备/终端/服务器中的处理器40、存储器41、输入装置42和输出装置43可以通过总线或其他方式连接，图4中以通过总线连接为例。
存储器41作为一种计算机可读存储介质，可设置为存储软件程序、计算机可执行程序以及模块，如本申请实施例中的多视角图像的融合方法对应的程序指令/模块(例如，多视角图像的融合装置中的图像获取模块301、光流信息计算模块302和图像融合模块303)。处理器40通过运行存储在存储器41中的软件程序、指令以及模块，从而执行设备/终端/服务器的多种功能应用以及数据处理，即实现上述的多视角图像的融合方法。
存储器41可主要包括存储程序区和存储数据区，其中，存储程序区可存储操作系统、至少一个功能所需的应用程序；存储数据区可存储根据终端的使用所创建的数据等。此外，存储器41可以包括高速随机存取存储器，还可以包括 非易失性存储器，例如至少一个磁盘存储器件、闪存器件、或其他非易失性固态存储器件。在一些实例中，存储器41可包括相对于处理器40远程设置的存储器，这些远程存储器可以通过网络连接至设备/终端/服务器。上述网络的实例包括但不限于互联网、企业内部网、局域网、移动通信网及其组合。
输入装置42可设置为接收输入的数字或字符信息，以及产生与设备/终端/服务器的用户设置以及功能控制有关的键信号输入。输出装置43可包括显示屏等显示设备。
本申请实施例还提供一种包含计算机可执行指令的存储介质，所述计算机可执行指令在由计算机处理器执行时设置为执行一种多视角图像的融合方法，该方法包括：获取目标对象在标准拍摄视角下的待融合图像以及目标对象在非标准拍摄视角下的参考图像；计算所述待融合图像和所述参考图像之间的光流信息；根据所述光流信息对所述待融合图像和所述参考图像进行融合，得到目标对象在标准拍摄视角下的目标图像。
当然,本申请实施例所提供的一种包含计算机可执行指令的存储介质,其计算机可执行指令不限于如上所述的方法操作,还可以执行本申请任意实施例所提供的多视角图像的融合方法中的相关操作。
通过以上关于实施方式的描述，所属领域的技术人员可以清楚地了解到，本申请可借助软件及必需的通用硬件来实现，当然也可以通过硬件实现。基于这样的理解，本申请的技术方案本质上或者说对相关技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品可以存储在计算机可读存储介质中，如计算机的软盘、只读存储器(Read-Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、闪存(FLASH)、硬盘或光盘等，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请实施例所述的方法。
值得注意的是，上述多视角图像的融合装置的实施例中，所包括的多个单元和模块只是按照功能逻辑进行划分的，但并不局限于上述的划分，只要能够实现相应的功能即可；另外，多个功能单元的具体名称也只是为了便于相互区分，并不用于限制本申请的保护范围。

Claims (10)

  1. 一种多视角图像的融合方法，包括：
    获取目标对象在标准拍摄视角下的待融合图像以及目标对象在非标准拍摄视角下的参考图像；
    计算所述待融合图像和所述参考图像之间的光流信息；
    根据所述光流信息对所述待融合图像和所述参考图像进行融合，得到目标对象在标准拍摄视角下的目标图像。
  2. 根据权利要求1所述的方法，其中，所述计算所述待融合图像和所述参考图像之间的光流信息，包括：
    分别对所述待融合图像和所述参考图像进行设定次数的下采样，得到所述待融合图像在不同清晰度下的待融合特征图和所述参考图像在不同清晰度下的参考特征图；
    计算具有相同清晰度的待融合特征图和参考特征图之间的子光流信息；
    所述根据所述光流信息对所述待融合图像和所述参考图像进行融合，得到目标对象在标准拍摄视角下的目标图像，包括：
    基于所述子光流信息、所述待融合特征图和所述参考图像，对所述待融合图像和所述参考图像进行融合，得到目标对象在标准拍摄视角下的目标图像。
  3. 根据权利要求2所述的方法，其中，所述基于所述子光流信息、所述待融合特征图和所述参考图像，对所述待融合图像和所述参考图像进行融合，得到目标对象在标准拍摄视角下的目标图像，包括：
    根据所述子光流信息对所述参考特征图进行视角修正，以将所述参考特征图修正为在标准拍摄视角下的修正图；
    按照清晰度从小到大的顺序，依次对每个清晰度下的待融合特征图和修正图进行融合，得到目标对象在标准拍摄视角下的目标图像。
  4. 根据权利要求3所述的方法，其中，所述按照清晰度从小到大的顺序，依次对每个清晰度下的待融合特征图和修正图进行融合，得到目标对象在标准拍摄视角下的目标图像，包括：
    获取当前清晰度下的当前特征图，所述当前特征图包括当前待融合特征图、当前修正图和当前中间图像，所述当前中间图像通过对在上一清晰度下融合得到的上一融合图像进行上采样获得；
    对所述当前特征图进行融合处理，得到当前融合图像；
    判断清晰度排序中是否存在高于所述当前清晰度且与所述当前清晰度相邻的下一清晰度，基于清晰度排序中存在高于所述当前清晰度且与所述当前清晰度相邻的下一清晰度的判断结果，将所述下一清晰度确定为当前清晰度，并返回执行获取当前清晰度下的当前特征图的操作；基于清晰度排序中不存在高于所述当前清晰度且与所述当前清晰度相邻的下一清晰度的判断结果，将所述当前融合图像确定为目标对象在标准拍摄视角下的目标图像。
  5. 根据权利要求1所述的方法，在所述获取目标对象在标准拍摄视角下的待融合图像以及目标对象在非标准拍摄视角下的参考图像之前，还包括：
    获取目标对象在标准拍摄视角下的原始图像、所述原始图像的第一分辨率以及目标对象在非标准拍摄视角下的参考图像的第二分辨率；
    根据所述第二分辨率和所述第一分辨率之间的比值，将所述原始图像转换为具有第二分辨率的待融合图像。
  6. 根据权利要求5所述的方法，在所述根据所述第二分辨率和所述第一分辨率之间的比值，将所述原始图像转换为具有第二分辨率的待融合图像之前，还包括：
    确定所述第一分辨率低于所述第二分辨率。
  7. 根据权利要求5所述的方法，还包括：
    采用多视角相机对目标对象进行拍摄，得到目标对象在标准拍摄视角下的原始图像和目标对象在非标准拍摄视角下的参考图像。
  8. 一种多视角图像的融合装置，包括：
    图像获取模块，设置为获取目标对象在标准拍摄视角下的待融合图像以及目标对象在非标准拍摄视角下的参考图像；
    光流信息计算模块，设置为计算所述待融合图像和所述参考图像之间的光流信息；
    图像融合模块，设置为根据所述光流信息对所述待融合图像和所述参考图像进行融合，得到目标对象在标准拍摄视角下的目标图像。
  9. 一种计算机设备，所述计算机设备包括：
    至少一个处理器；
    存储器，设置为存储至少一个或多个程序，
    当所述至少一个程序被所述一个或多个处理器执行，使得所述至少一个处 理器实现如权利要求1-7中任一所述的多视角图像的融合方法。
  10. 一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现如权利要求1-7中任一所述的多视角图像的融合方法。
PCT/CN2019/094553 2018-07-03 2019-07-03 多视角图像的融合方法、装置、计算机设备和存储介质 WO2020007320A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810717536.3A CN108833785B (zh) 2018-07-03 2018-07-03 多视角图像的融合方法、装置、计算机设备和存储介质
CN201810717536.3 2018-07-03

Publications (1)

Publication Number Publication Date
WO2020007320A1 true WO2020007320A1 (zh) 2020-01-09

Family

ID=64134739

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/094553 WO2020007320A1 (zh) 2018-07-03 2019-07-03 多视角图像的融合方法、装置、计算机设备和存储介质

Country Status (2)

Country Link
CN (1) CN108833785B (zh)
WO (1) WO2020007320A1 (zh)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111340866A (zh) * 2020-02-26 2020-06-26 腾讯科技（深圳）有限公司 深度图像生成方法、装置及存储介质
CN111402118A (zh) * 2020-03-17 2020-07-10 腾讯科技（深圳）有限公司 图像替换方法、装置、计算机设备和存储介质
CN111724422A (zh) * 2020-06-29 2020-09-29 深圳市慧鲤科技有限公司 图像处理方法及装置、电子设备及存储介质
CN112184610A (zh) * 2020-10-13 2021-01-05 深圳市锐尔觅移动通信有限公司 图像处理方法及装置、存储介质和电子设备
CN115641487A (zh) * 2022-08-26 2023-01-24 青岛元动芯能源科技有限公司 一种基于中子和x射线的多级判定融合方法和系统
CN116630220A (zh) * 2023-07-25 2023-08-22 江苏美克医学技术有限公司 一种荧光图像景深融合成像方法、装置及存储介质

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108833785B (zh) * 2018-07-03 2020-07-03 清华-伯克利深圳学院筹备办公室 多视角图像的融合方法、装置、计算机设备和存储介质
CN112135088B (zh) * 2019-06-25 2024-04-16 北京京东尚科信息技术有限公司 一种显示试装效果的方法、试装终端及存储介质
CN110929615B (zh) * 2019-11-14 2022-10-18 RealMe重庆移动通信有限公司 图像处理方法、图像处理装置、存储介质与终端设备
CN115334245A (zh) * 2019-12-06 2022-11-11 达闼机器人股份有限公司 一种图像矫正方法、装置、电子设备及存储介质
CN112581365B (zh) * 2020-11-10 2024-09-06 北京拙河光场科技有限公司 一种跨尺度自适应信息映射成像方法及装置、介质
CN113807349B (zh) * 2021-09-06 2023-06-20 海南大学 基于物联网的多视角目标识别方法及系统
CN116912148B (zh) * 2023-09-12 2024-01-05 深圳思谋信息科技有限公司 图像增强方法、装置、计算机设备及计算机可读存储介质
CN117291954A (zh) * 2023-09-21 2023-12-26 小红书科技有限公司 生成光流数据集的方法、相关方法及相关产品

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090028462A1 (en) * 2007-07-26 2009-01-29 Kensuke Habuka Apparatus and program for producing a panoramic image
CN101901481A (zh) * 2010-08-11 2010-12-01 深圳市蓝韵实业有限公司 一种图像拼接方法
CN102446366A (zh) * 2011-09-14 2012-05-09 天津大学 时空联合多视角视频插值及三维建模方法
CN103745449A (zh) * 2013-12-24 2014-04-23 南京理工大学 一种搜跟系统中航拍视频的快速自动拼接技术
CN103971352A (zh) * 2014-04-18 2014-08-06 华南理工大学 一种基于广角镜头的快速图像拼接方法
CN108833785A (zh) * 2018-07-03 2018-11-16 清华-伯克利深圳学院筹备办公室 多视角图像的融合方法、装置、计算机设备和存储介质

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2002367416A (ja) * 2001-06-12 2002-12-20 Ichikoh Ind Ltd ヘッドランプ
CN104869387B (zh) * 2015-04-19 2017-01-11 中国传媒大学 基于光流法的双目图像最大视差获取方法
CN105046676A (zh) * 2015-08-27 2015-11-11 上海斐讯数据通信技术有限公司 一种基于智能终端的图像融合方法及设备
CN106803899B (zh) * 2015-11-26 2019-10-01 华为技术有限公司 合并图像的方法和装置
CN106997579B (zh) * 2016-01-26 2020-01-03 华为技术有限公司 图像拼接的方法和装置
CN105844630B (zh) * 2016-03-21 2018-11-16 西安电子科技大学 一种双目视觉的图像超分辨率融合去噪方法
CN107451952B (zh) * 2017-08-04 2020-11-03 追光人动画设计（北京）有限公司 一种全景视频的拼接融合方法、设备以及系统
CN108171744A (zh) * 2017-12-26 2018-06-15 努比亚技术有限公司 一种双目虚化中视差图的确定方法、移动终端及存储介质

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090028462A1 (en) * 2007-07-26 2009-01-29 Kensuke Habuka Apparatus and program for producing a panoramic image
CN101901481A (zh) * 2010-08-11 2010-12-01 深圳市蓝韵实业有限公司 一种图像拼接方法
CN102446366A (zh) * 2011-09-14 2012-05-09 天津大学 时空联合多视角视频插值及三维建模方法
CN103745449A (zh) * 2013-12-24 2014-04-23 南京理工大学 一种搜跟系统中航拍视频的快速自动拼接技术
CN103971352A (zh) * 2014-04-18 2014-08-06 华南理工大学 一种基于广角镜头的快速图像拼接方法
CN108833785A (zh) * 2018-07-03 2018-11-16 清华-伯克利深圳学院筹备办公室 多视角图像的融合方法、装置、计算机设备和存储介质

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111340866A (zh) * 2020-02-26 2020-06-26 腾讯科技（深圳）有限公司 深度图像生成方法、装置及存储介质
CN111340866B (zh) * 2020-02-26 2024-03-01 腾讯科技（深圳）有限公司 深度图像生成方法、装置及存储介质
CN111402118A (zh) * 2020-03-17 2020-07-10 腾讯科技（深圳）有限公司 图像替换方法、装置、计算机设备和存储介质
CN111724422A (zh) * 2020-06-29 2020-09-29 深圳市慧鲤科技有限公司 图像处理方法及装置、电子设备及存储介质
CN111724422B (zh) * 2020-06-29 2024-01-09 深圳市慧鲤科技有限公司 图像处理方法及装置、电子设备及存储介质
CN112184610A (zh) * 2020-10-13 2021-01-05 深圳市锐尔觅移动通信有限公司 图像处理方法及装置、存储介质和电子设备
CN112184610B (zh) * 2020-10-13 2023-11-28 深圳市锐尔觅移动通信有限公司 图像处理方法及装置、存储介质和电子设备
CN115641487A (zh) * 2022-08-26 2023-01-24 青岛元动芯能源科技有限公司 一种基于中子和x射线的多级判定融合方法和系统
CN115641487B (zh) * 2022-08-26 2023-06-27 中子时代(青岛)创新科技有限公司 一种基于中子和x射线的多级判定融合方法和系统
CN116630220A (zh) * 2023-07-25 2023-08-22 江苏美克医学技术有限公司 一种荧光图像景深融合成像方法、装置及存储介质
CN116630220B (zh) * 2023-07-25 2023-11-21 江苏美克医学技术有限公司 一种荧光图像景深融合成像方法、装置及存储介质

Also Published As

Publication number Publication date
CN108833785A (zh) 2018-11-16
CN108833785B (zh) 2020-07-03

Similar Documents

Publication Publication Date Title
WO2020007320A1 (zh) 多视角图像的融合方法、装置、计算机设备和存储介质
US10540806B2 (en) Systems and methods for depth-assisted perspective distortion correction
KR102351542B1 (ko) 시차 보상 기능을 갖는 애플리케이션 프로세서, 및 이를 구비하는 디지털 촬영 장치
WO2021073331A1 (zh) 基于终端设备的变焦虚化图像获取方法和装置
US10389948B2 (en) Depth-based zoom function using multiple cameras
US10762655B1 (en) Disparity estimation using sparsely-distributed phase detection pixels
CN100364319C (zh) 图像处理方法和图像处理装置
JP5954668B2 (ja) 画像処理装置、撮像装置および画像処理方法
CN113129241B (zh) 图像处理方法及装置、计算机可读介质、电子设备
WO2014023231A1 (zh) 宽视场超高分辨率光学成像系统及方法
US9131155B1 (en) Digital video stabilization for multi-view systems
JP6656035B2 (ja) 画像処理装置、撮像装置および画像処理装置の制御方法
EP3629570A2 (en) Image capturing apparatus and image recording method
US20220358619A1 (en) Automatic dolly zoom image processing device
KR101324250B1 (ko) 영상 처리를 통한 광축 오차 보상 방법 및 시스템, 광축 오차 보상 기능이 구현된 줌 카메라
JP2018503195A (ja) 物体検出方法及び物体検出装置
US10397540B2 (en) Method for obtaining and merging multi-resolution data
JP4394487B2 (ja) ステレオ画像処理装置
JP6483661B2 (ja) 撮像制御装置、撮像制御方法およびプログラム
JP6648916B2 (ja) 撮像装置
JP6381212B2 (ja) 撮像装置及びその制御方法
JP2018005337A (ja) 画像処理装置、撮像装置、画像処理方法、及びプログラム
JP2016024490A (ja) 画像処理装置、画像処理方法およびプログラム
KR101323195B1 (ko) 2차원 영상에서 3차원 영상 생성 방법
JP2023127450A (ja) 画像処理装置、画像処理方法およびプログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19829771

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19829771

Country of ref document: EP

Kind code of ref document: A1