WO2023005170A1 - 全景视频的生成方法和装置 - Google Patents

全景视频的生成方法和装置 Download PDF

Info

Publication number
WO2023005170A1
WO2023005170A1 PCT/CN2022/072978 CN2022072978W WO2023005170A1 WO 2023005170 A1 WO2023005170 A1 WO 2023005170A1 CN 2022072978 W CN2022072978 W CN 2022072978W WO 2023005170 A1 WO2023005170 A1 WO 2023005170A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
video frame
panoramic
video
frame images
Prior art date
Application number
PCT/CN2022/072978
Other languages
English (en)
French (fr)
Inventor
饶童
朱毅
黄乘风
Original Assignee
贝壳技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 贝壳技术有限公司 filed Critical 贝壳技术有限公司
Publication of WO2023005170A1 publication Critical patent/WO2023005170A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G06T7/337—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
    • G06V10/751—Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/156—Mixing image signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/698—Control of cameras or camera modules for achieving an enlarged field of view, e.g. panoramic image capture
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/2622—Signal amplitude transition in the zone between image portions, e.g. soft edges
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/265—Mixing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/10—Image acquisition
    • G06V10/16—Image acquisition using multiple overlapping images; Image stitching

Definitions

  • the present disclosure relates to the technical field of image processing, in particular to a method and device for generating a panoramic video.
  • Panorama is divided into virtual reality and three-dimensional reality.
  • Virtual reality is a scene that simulates reality produced by software.
  • the 3D real scene is to use the camera to take the real scene photos, after special stitching and processing, finally get the 3D real scene pictures.
  • the current panoramic video needs to be captured by a panoramic camera that supports video recording, which requires relatively high shooting equipment.
  • the panoramic video has a lot of content, and uploading/downloading of corresponding data resources requires a relatively good network environment and network bandwidth.
  • Embodiments of the present disclosure provide a method and device for generating a panoramic video, which does not require a panoramic camera supporting video recording, and has lower requirements on the network environment than the method of presenting animations for the panoramic video.
  • a method for generating a panoramic video including:
  • a panoramic video is generated based on the plurality of panoramic frame images.
  • the image acquisition positions and image acquisition angles among the plurality of video frame images are the same, and the feature matching is performed on the panoramic image and the plurality of video frame images, and based on Feature matching result, performing image fusion with each video frame image in the plurality of video frame images respectively with the panoramic image, including:
  • image fusion is performed on each video frame image in the plurality of video frame images and the panoramic image respectively.
  • the performing image fusion of each video frame image in the multiple video frame images with the panoramic image based on the perspective transformation mapping matrix includes:
  • mapping image from the selected video frame image to the panoramic image based on the perspective transformation mapping matrix
  • Image fusion is performed on the plurality of color-adjusted video frame images and the panoramic image respectively.
  • the color adjustment of the plurality of video frame images based on the color difference of the same-named pixel between the mapping image and the panoramic image includes:
  • Color adjustment is performed on the plurality of video frame images based on the color mapping function or the color lookup table.
  • the performing image fusion of each video frame image in the multiple video frame images with the panoramic image based on the perspective transformation mapping matrix includes:
  • feathering is performed after replacing the pixels in the corresponding image fusion area in the panoramic image with the pixels of the video frame image.
  • the feature matching is performed on the panoramic image and the plurality of video frame images, and based on the feature matching result, each video frame image in the plurality of video frame images is respectively matched with
  • the panoramic image is subjected to image fusion, including:
  • image fusion is performed on each video frame image and the panoramic image respectively.
  • performing image fusion on the plurality of video frame images and the panoramic image based on the perspective transformation mapping matrix between each video frame image and the panoramic image include:
  • Image fusion is performed on the plurality of color-adjusted video frame images and the panoramic image respectively.
  • performing image fusion on the plurality of video frame images and the panoramic image based on the perspective transformation mapping matrix between each video frame image and the panoramic image include:
  • Feathering is performed after replacing the pixels of the panoramic image in the image fusion area with the pixels of the plurality of video frame images respectively.
  • a second aspect of the embodiments of the present disclosure provides a device for generating a panoramic video, including:
  • a panoramic image acquisition module configured to acquire a panoramic image including a target scene
  • a video frame image acquisition module configured to acquire the video collected for the target scene, and select a plurality of video frame images from the video
  • the panoramic frame image acquisition module is used to perform feature matching on the panoramic image and the multiple video frame images, and based on the feature matching results, each video frame image in the multiple video frame images is respectively compared with the Perform image fusion on the panoramic image to obtain multiple panoramic frame images;
  • An animation generation module configured to generate a panoramic video based on the plurality of panoramic frame images.
  • the panoramic frame image acquisition module includes:
  • a video frame image acquisition unit configured to select a video frame image from the plurality of video frame images
  • a first perspective transformation mapping matrix generating unit configured to perform feature point matching on the selected video frame image and the panoramic image, and generate a perspective transformation between the selected video frame image and the panoramic image based on the feature point matching result mapping matrix;
  • the first image fusion unit is configured to perform image fusion on each video frame image in the plurality of video frame images and the panoramic image based on the perspective transformation mapping matrix.
  • the first image fusion unit includes:
  • a first mapping image generating subunit configured to generate a mapping image mapped from the selected video frame image to the panoramic image based on the perspective transformation mapping matrix
  • the first color adjustment subunit is configured to perform color adjustment on the plurality of video frame images based on the color difference of the same-named pixel between the mapping image and the panoramic image;
  • the first image fusion sub-unit is configured to perform image fusion on the plurality of color-adjusted video frame images and the panoramic image respectively.
  • the first color adjustment subunit is configured to generate a color mapping function or a color lookup table based on the color difference of the same-named pixel between the mapped image and the panoramic image, and based on The color mapping function or the color lookup table respectively performs color adjustment on the plurality of video frame images.
  • the first image fusion unit includes:
  • An image fusion area determining subunit configured to determine an image fusion area corresponding to each video frame image in the panoramic image based on the perspective transformation mapping matrix
  • the image fusion subunit is used for performing feathering after replacing the pixels in the corresponding image fusion area in the panoramic image with the pixels of the video frame image for each video frame image.
  • the panoramic frame image acquisition module includes:
  • the second perspective transformation mapping matrix generation unit is configured to perform feature point matching on each video frame image in the plurality of video frame images with the panoramic image, and generate each video frame image based on the feature point matching result and the perspective transformation mapping matrix between the panoramic image;
  • the second image fusion unit is configured to perform image fusion on each video frame image and the panoramic image based on the perspective transformation mapping matrix between the each video frame image and the panoramic image.
  • the second image fusion unit includes:
  • the second mapping image generation subunit is used to generate multiple mapping images based on the perspective transformation mapping matrix between each video frame image and the panoramic image, and the multiple mapping images include the a frame image is mapped to a mapping image of the panoramic image;
  • the second color adjustment subunit is used to adjust the color of the plurality of video frame images based on the color difference of the same-named pixel between the plurality of mapping images and the panoramic image;
  • the second image fusion sub-unit is configured to perform image fusion on the plurality of color-adjusted video frame images and the panoramic image respectively.
  • an electronic device including:
  • the processor is configured to execute the computer program stored in the memory, and when the computer program is executed, realize the panoramic video generation method described in the first aspect above.
  • a fourth aspect of the embodiments of the present disclosure provides a computer-readable storage medium, on which a computer program is stored.
  • the computer program is executed by a processor, the panoramic video generation method described in the above-mentioned first aspect is implemented.
  • the method and device for generating a panoramic video, and the method and device for image fusion of the present disclosure can generate an animation video in which a target scene undergoes screen changes by fusing a panoramic image including a target scene with a video frame image for the target scene.
  • the video frame images can only be collected by a monocular camera, and there is no need to use a panoramic camera that supports video recording.
  • the embodiment of the present disclosure only needs to collect the video of the moving part in the panorama, which can greatly reduce the dependence on the quality of the network environment. Therefore, compared with the way of presenting the animation of the panorama video, it has lower requirements on the network environment.
  • FIG. 1 is a flowchart of a method for generating a panoramic video according to an embodiment of the present disclosure.
  • FIG. 2 is a panoramic image of a target in one example of the present disclosure.
  • Fig. 3 is a schematic diagram of a picture change area of a panoramic video in an example of the present disclosure.
  • Fig. 4 is a schematic diagram of pasting a video frame image directly into a panoramic image without processing in an example of the present disclosure.
  • FIG. 5 is a panorama frame image after color adjustment of FIG. 4 .
  • Fig. 6 is a structural block diagram of an apparatus for generating a panoramic video according to an embodiment of the disclosure.
  • Fig. 7 is a schematic structural diagram of an application embodiment of the electronic device of the present disclosure.
  • plural may refer to two or more than two, and “at least one” may refer to one, two or more than two.
  • the term "and/or" in the present disclosure is only an association relationship describing associated objects, indicating that there may be three relationships, for example, A and/or B may indicate: A exists alone, and A and B exist simultaneously , there are three cases of B alone.
  • the character "/" in the present disclosure generally indicates that the contextual objects are an "or" relationship.
  • Embodiments of the present disclosure may be applied to electronic devices such as terminal devices, computer systems, servers, etc., which may operate with numerous other general purpose or special purpose computing system environments or configurations.
  • Examples of well known terminal devices, computing systems, environments and/or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick client Computers, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, networked personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the foregoing, etc.
  • Electronic devices such as terminal devices, computer systems, servers, etc. may be described in the general context of computer system-executable instructions, such as program modules, being executed by the computer system.
  • program modules may include routines, programs, objects, components, logic, data structures, etc., that perform particular tasks or implement particular abstract data types.
  • the computer system/server can be practiced in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network.
  • program modules may be located in both local and remote computing system storage media including storage devices.
  • FIG. 1 is a flowchart of a method for generating a panoramic video according to an embodiment of the present disclosure. As shown in Figure 1, the generation method of the panoramic video of the embodiment of the present disclosure includes:
  • S1 Obtain a panoramic image including a target scene.
  • the panorama image may be a panorama of a set of rooms
  • the target scene may be a video shooting scene of one of the rooms in the set of rooms.
  • the panoramic image may be a panoramic image obtained after multiple images are taken by a monocular camera and processed by an image processing technology on the multiple images.
  • image processing technology includes image mosaic and image fusion.
  • the panoramic image may also be obtained by shooting with a panoramic camera.
  • S2 Obtain the video collected for the target scene, and select multiple video frame images from the video.
  • a monocular camera can be aimed at a target scene to shoot a video, and then a plurality of video frame images can be selected from the video.
  • the multiple video frame images may include multiple key frame images of the video, or the multiple video frame images include all video frame images of the video.
  • the method of selecting the key frame of the selected video may adopt a key frame extraction method based on a shot, a key frame extraction method based on motion analysis, or a method based on video clustering.
  • the key frame extraction method based on the shot first divides the source video file according to the shot changes according to some technical means, and then selects the first and last two frames in each shot of the video as key frames.
  • the optical flow of object movement is analyzed in the video shot, and the video frame with the least number of optical flow movements in the video shot is selected as the extracted key frame each time.
  • video clustering methods first, a cluster center is initialized.
  • the present disclosure does not limit the key frame extraction method.
  • S3 Perform feature matching on the panoramic image and multiple video frame images, and based on the feature matching results, perform image fusion on each of the multiple video frame images with the panoramic image to obtain multiple panoramic frame images.
  • feature matching includes feature collection and feature point matching.
  • the feature matching results may include feature point matching results between the panoramic image and multiple video frame images.
  • the screen changes only occur in the region fused with each video frame image.
  • S4 Generate a panoramic video based on multiple panoramic frame images.
  • the video frame image can only be collected by a monocular camera, and there is no need to use a panoramic camera that supports video recording.
  • FIG. 2 is a target panoramic image in an example of the present disclosure
  • FIG. 3 is a schematic diagram of a frame change area of a frame of panoramic video in an example of the present disclosure.
  • the panoramic video only the pixels in the picture change area change, and the other image areas do not change.
  • the embodiment of the present disclosure only needs to collect the video of the moving part in the panorama, which can greatly reduce the dependence on the quality of the network environment. Therefore, compared with the way of presenting the animation of the panorama video, it has lower requirements on the network environment.
  • the image acquisition positions and image acquisition angles among the multiple video frame images are the same.
  • multiple video frame images can be obtained by shooting video with a pan-tilt camera.
  • the shooting parameters include not only image acquisition position and image acquisition angle, but also parameters such as image acquisition focal length and image resolution, that is, except that the image acquisition positions and image acquisition angles between multiple video frame images are the same, multiple The image capture focal length and image resolution are also the same between video frame images.
  • step S3 includes:
  • S3-1 Select a video frame image from multiple video frame images.
  • the key frame image at the front of the video playback time point can be used as the video frame image of the selected area; when a plurality of video frame images include all video frame images of the video
  • the first frame image of the video can be used as the video frame image of the selection area.
  • S3-2 Perform feature point matching on the selected video frame image and the panoramic image, and generate a perspective transformation mapping matrix between the selected video frame image and the panoramic image based on the feature point matching results.
  • feature extraction is performed on selected video frame images to obtain first image features
  • feature extraction is performed on panoramic images to obtain second image features.
  • the same sign extraction weight is used.
  • a perspective transformation mapping matrix for mapping from the selected video frame image to the panoramic image is determined.
  • the perspective transformation mapping matrix is a 3x3 matrix, which implicitly expresses the relative rotation, translation and plane parameters between the plane of the video frame image and the plane of the panoramic image in space.
  • S3-3 Based on the perspective transformation mapping matrix, perform image fusion on each of the multiple video frame images and the panoramic image. Wherein, since the image acquisition positions and image acquisition angles among the plurality of video frame images are the same, the perspective transformation mapping matrix between each video frame image and the panoramic image in the plurality of video frame images is also completely the same. Based on the same perspective transformation mapping matrix, image fusion is performed on each of the multiple video frame images and the panoramic image to obtain multiple panoramic frame images.
  • Fig. 4 is a schematic diagram of pasting a video frame image directly into a panoramic image without processing in an example of the present disclosure. As shown in Figure 4, there is a clear visual difference between the image fusion area and the panoramic image.
  • Fig. 5 is the panorama frame image after the color adjustment of Fig. 4. As shown in Fig. 5, after the color adjustment, the visual difference between the image fusion area and other areas is effectively reduced.
  • each video frame image in the plurality of video frame images is respectively image-fused with the panoramic image
  • the The position of the image fusion area in the panoramic image remains unchanged, so that the perspective transformation mapping matrix can only be established once, and then based on the perspective transformation mapping matrix, each video frame image in multiple video frame images can be imaged separately with the panoramic image fusion.
  • step S3-3 includes:
  • S3-3-A-1 Generate a mapping image from the selected video frame image to the panoramic image based on the perspective transformation mapping matrix. That is, the mapping image can be generated by performing mapping transformation on the selected video frame image through the perspective transformation mapping matrix.
  • FIG. 2 is a panoramic image in one example of the present disclosure
  • FIG. 3 is a panoramic image including a map image in one example of the present disclosure.
  • S3-3-A-2 Based on the color difference of the pixel with the same name between the mapped image and the panoramic image, color adjustment is performed on multiple video frame images respectively.
  • the image point with the same name is also called the corresponding image point, which refers to the image point of any target point on different photos.
  • the color differences include grayscale differences, RGB differences, and the like.
  • the color difference can be obtained by comparing the pixel values of pixels with the same name. Taking the grayscale image difference as an example, assuming that any pixel with the same name on the corresponding area I2 of the first mapped image I1 and the target panoramic image is p1(x1, y1) and p2(x2, y2), then the corresponding grayscale image
  • is the color difference.
  • the color mapping relationship between the same-named pixels is determined based on the color difference of the same-named pixels. Based on the color mapping relationship, color adjustment is performed on multiple video frame images respectively.
  • S3-3-A-3 Perform image fusion on the multiple color-adjusted video frame images and the panoramic image respectively.
  • the visual difference after fusion of the video frame image and the panoramic image can be reduced through color adjustment.
  • step S3-3-A-3 includes: generating a color mapping function or a color lookup table based on the color difference of the same-named pixel between the mapped image and the panoramic image; A lookup table is used to perform color adjustment on multiple video frame images respectively.
  • the color difference of the same-named pixel between the mapped image corresponding to the selected video frame image and the panoramic image is minimized.
  • the mapping function used is the second-order Taylor series expansion of the exponential function, because it is believed that the general color difference will not be too large, and the high-order information will quickly drop to zero.
  • Color adjustment is performed on the pixels of the selected video frame image based on the color difference mapping function. For each of the remaining video frame images in multiple video frame images except the previously selected video frame image, obtain all pixel information of each video frame image, and then calculate the pixel points of each video frame image based on the color difference mapping function Make color adjustments.
  • the mapping image corresponding to the selected video frame image and the panoramic image can have the same name
  • the color difference of the image point is minimal.
  • taking a grayscale image as an example a group of pixel points are found on the mapped image, and their intensity values cover 0-255. Find the intensity value corresponding to the image point with the same name on the panoramic image, and build a lookup table. where missing data are constructed by interpolation.
  • the disclosure can quickly and accurately establish the mapping relationship between the mapping image and the pixel with the same name in the target panoramic image by means of a look-up table.
  • Color adjustment is performed on the pixels of the selected video frame image based on the color lookup table. For each of the remaining video frame images in multiple video frame images except the previously selected video frame image, obtain all pixel information of each video frame image, and then perform pixel point information based on the color lookup table for each video frame image Make color adjustments.
  • step S3-3 includes:
  • S3-3-B-1 Based on the perspective transformation mapping matrix, determine the image fusion area corresponding to each video frame image in the panoramic image. Wherein, since the image acquisition positions and image acquisition angles among the plurality of video frame images are the same, when each video frame image in the plurality of video frame images is respectively image-fused with the panoramic image, the The position of the image fusion area remains unchanged.
  • step S3 includes: performing feature point matching on each video frame image in a plurality of video frame images and the panoramic image, and generating each video frame image and the panoramic image based on the feature point matching results
  • the perspective transformation mapping matrix between each video frame image and the panoramic image are based on the perspective transformation mapping matrix between each video frame image and the panoramic image for image fusion.
  • the fusion position of each video frame image in the corresponding panoramic image is adjusted to ensure the fusion effect of each video frame image.
  • image fusion is performed on multiple video frame images and the panoramic image respectively, including: based on the perspective between each video frame image and the panoramic image Transform the mapping matrix to generate multiple mapping images, the multiple mapping images include mapping images from each video frame image to the panoramic image; based on the color difference of the same-named pixels between the multiple mapping images and the panoramic image, respectively for multiple Color adjustment is performed on each video frame image; image fusion is performed on multiple color-adjusted video frame images with the panoramic image respectively.
  • the perspective transformation mapping matrix is established for each video frame image, it is necessary to perform projective transformation on the perspective transformation mapping matrix corresponding to each video frame image, so that the mapping corresponding to each video frame image can be obtained image, and then perform color adjustment based on the mapping image corresponding to each video frame image, which can ensure the fusion effect of each video frame image.
  • the apparatus for generating a panoramic video provided by any embodiment of the present disclosure may be used to implement the method for generating a panoramic video in the foregoing embodiments of the present disclosure.
  • the apparatus for generating panoramic video provided by any embodiment of the present disclosure may be set on the terminal device, or on the server, or partly on the terminal device and partly on the server.
  • the apparatus for generating a panoramic video provided in an exemplary embodiment may include: a processor; a memory for storing instructions executable by the processor; the processor is used for reading the executable instructions from the memory, And execute the instructions to implement the panoramic video generation method provided by the exemplary embodiments of the present disclosure.
  • Fig. 6 is a structural block diagram of an apparatus for generating a panoramic video according to an embodiment of the disclosure.
  • the apparatus for generating a panoramic video in the embodiment of the present disclosure includes: a panoramic image acquisition module 610 , a video frame image acquisition module 620 , a panoramic frame image acquisition module 630 and an animation generation module 640 .
  • the panoramic image acquisition module 610 is used to acquire a panoramic image including a target scene.
  • the video frame image acquisition module 620 is used to acquire the video collected for the target scene, and select a plurality of video frame images from the video.
  • the panoramic frame image acquisition module 630 is used to perform feature matching on the panoramic image and multiple video frame images, and based on the feature matching results, image fusion is performed on each video frame image in the multiple video frame images with the panoramic image to obtain multiple panoramic frame images.
  • the animation generation module 640 is used to generate a panoramic video based on multiple panoramic frame images.
  • the panoramic frame image acquisition module 630 includes:
  • a video frame image acquisition unit used to select a video frame image from a plurality of video frame images
  • the first perspective transformation mapping matrix generating unit is used to perform feature point matching on the selected video frame image and the panoramic image, and generate a perspective transformation mapping matrix between the selected video frame image and the panoramic image based on the feature point matching result;
  • the first image fusion unit is configured to perform image fusion on each of the plurality of video frame images and the panoramic image based on the perspective transformation mapping matrix.
  • the first image fusion unit includes:
  • the first mapping image generation subunit is used to generate a mapping image from the selected video frame image to the panoramic image based on the perspective transformation mapping matrix;
  • the first color adjustment subunit is used to adjust the color of multiple video frame images based on the color difference of the same-named pixel between the mapped image and the panoramic image;
  • the first image fusion sub-unit is configured to perform image fusion on the plurality of color-adjusted video frame images and the panoramic image respectively.
  • the first color adjustment subunit is used to generate a color mapping function or a color lookup table based on the color difference of the pixel with the same name between the mapped image and the panoramic image, and The table is used to adjust the color of multiple video frame images respectively.
  • the first image fusion unit includes:
  • the image fusion area determination subunit is used to determine the image fusion area corresponding to each video frame image in the panoramic image based on the perspective transformation mapping matrix;
  • the image fusion subunit is used for performing feathering after replacing the pixels in the corresponding image fusion area in the panoramic image with the pixels of the video frame image for each video frame image.
  • the panoramic frame image acquisition module 630 includes:
  • the second perspective transformation mapping matrix generation unit is used to perform feature point matching on each video frame image in the plurality of video frame images and the panoramic image, and generate a relationship between each video frame image and the panoramic image based on the feature point matching result perspective transformation mapping matrix;
  • the second image fusion unit is configured to perform image fusion on each video frame image and the panoramic image based on the perspective transformation mapping matrix between each video frame image and the panoramic image.
  • the second image fusion unit includes:
  • the second mapping image generation subunit is used to generate multiple mapping images based on the perspective transformation mapping matrix between each video frame image and the panoramic image, and the multiple mapping images include mapping from each video frame image to the panoramic image image;
  • the second color adjustment subunit is used to adjust the color of multiple video frame images based on the color difference of the same-named pixel between the multiple mapped images and the panoramic image;
  • the second image fusion sub-unit is configured to perform image fusion on the plurality of color-adjusted video frame images and the panoramic image respectively.
  • the apparatus for generating panoramic video in this embodiment of the present disclosure is a device corresponding to the method for generating panoramic video. In order to reduce redundancy, the specific implementation of the apparatus for generating panoramic video in this embodiment of the present disclosure will not be repeated.
  • an electronic device including:
  • the processor is configured to execute the computer program stored in the memory, and when the computer program is executed, realize the panoramic video generation method described in any one of the above-mentioned embodiments of the present disclosure.
  • Fig. 7 is a schematic structural diagram of an application embodiment of the electronic device of the present disclosure.
  • the electronic device may be either or both of the first device and the second device, or a stand-alone device independent of them, and the stand-alone device may communicate with the first device and the second device to receive collected data from them. input signal.
  • an electronic device includes one or more processors and memory.
  • the processor may be a central processing unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.
  • CPU central processing unit
  • the processor may control other components in the electronic device to perform desired functions.
  • the memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory.
  • the volatile memory may include, for example, random access memory (RAM) and/or cache memory (cache).
  • the non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, and the like.
  • One or more computer program instructions may be stored on the computer-readable storage medium, and the processor may execute the program instructions to implement the panoramic video generation method and/or the various embodiments of the present disclosure described above. other desired features.
  • the electronic device may further include: an input device and an output device, and these components are interconnected through a bus system and/or other forms of connection mechanisms (not shown).
  • the input device may also include, for example, a keyboard, a mouse, and the like.
  • the output device can output various information to the outside, including determined distance information, direction information, and the like.
  • the output devices may include, for example, displays, speakers, printers, and communication networks and their connected remote output devices, among others.
  • the electronic device may also include any other suitable components according to specific applications.
  • embodiments of the present disclosure may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the functions described in the foregoing sections of this specification. Steps in the method for generating panoramic video in various embodiments of the present disclosure.
  • the computer program product can be written in any combination of one or more programming languages to execute the program codes for performing the operations of the embodiments of the present disclosure, and the programming languages include object-oriented programming languages, such as Java, C++, etc. , also includes conventional procedural programming languages, such as the "C" language or similar programming languages.
  • the program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server to execute.
  • embodiments of the present disclosure may also be a computer-readable storage medium, on which computer program instructions are stored, and the computer program instructions, when executed by a processor, cause the processor to execute the method according to the present invention described in the above part of this specification.
  • the steps in the method for generating panoramic video in various embodiments are disclosed.
  • the computer readable storage medium may employ any combination of one or more readable media.
  • the readable medium may be a readable signal medium or a readable storage medium.
  • the readable storage medium may include, but not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: electrical connection with one or more conductors, portable disk, hard disk, random access memory (RAM), read only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
  • the methods and apparatus of the present disclosure may be implemented in many ways.
  • the methods and apparatuses of the present disclosure may be implemented by software, hardware, firmware or any combination of software, hardware, and firmware.
  • the above sequence of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the sequence specifically described above unless specifically stated otherwise.
  • the present disclosure can also be implemented as programs recorded in recording media, the programs including machine-readable instructions for realizing the method according to the present disclosure.
  • the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.
  • each component or each step can be decomposed and/or reassembled. These decompositions and/or recombinations should be considered equivalents of the present disclosure.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Image Processing (AREA)
  • Studio Devices (AREA)

Abstract

本公开实施例公开了一种全景视频的生成方法和装置，其中，该生成方法包括：获取包括目标场景的全景图像；获取针对所述目标场景采集的视频，并从所述视频中选取出多个视频帧图像；对所述全景图像与所述多个视频帧图像进行特征匹配，并基于特征匹配结果，将所述多个视频帧图像中的每个视频帧图像分别与所述全景图像进行图像融合，得到多个全景帧图像；基于所述多个全景帧图像生成全景视频。本公开无需采用支持视频录制的全景相机，仅需单目相机配合全景图像即可生成全景视频，且相对于全景视频呈现动画的方式对网络环境要求低。

Description

全景视频的生成方法和装置
相关申请的交叉引用
本公开要求2021年07月30日提交的中国专利申请202110873542.X的权益，该申请的内容通过引用被合并于本文。
技术领域
本公开涉及图像处理技术领域，尤其是一种全景视频的生成方法和装置。
背景技术
全景分为虚拟现实和三维实景。虚拟现实是利用软件制作出来的模拟现实的场景。三维实景是利用相机拍摄实景照片，经过特殊的拼合、处理，最终得到三维实景图片。
目前的全景视频，需要通过支持视频录制的全景相机来采集，对于拍摄设备要求较高。此外，全景视频的内容较多，对应的数据资源的上传/下载均需要较好的网络环境以及网络带宽。
发明内容
本公开实施例提供一种全景视频的生成方法和装置，无需采用支持视频录制的全景相机，且相对于全景视频呈现动画的方式对网络环境要求低。
本公开实施例的第一个方面，提供一种全景视频的生成方法，包括：
获取包括目标场景的全景图像；
获取针对所述目标场景采集的视频，并从所述视频中选取出多个视频帧图像；
对所述全景图像与所述多个视频帧图像进行特征匹配，并基于特征匹配结果，将所述多个视频帧图像中的每个视频帧图像分别与所述全景图像进行图像融合，得到多个全景帧图像；
基于所述多个全景帧图像生成全景视频。
在一种可选的方式中，所述多个视频帧图像之间的图像采集位置和图像采集角度均相同，所述对所述全景图像与所述多个视频帧图像进行特征匹配，并基于特征匹配结果，将所述多个视频帧图像中的每个视频帧图像分别与所述全景图像进行图像融合，包括：
从所述多个视频帧图像中选取一视频帧图像；
对选取的视频帧图像与所述全景图像进行特征点匹配，基于特征点匹配结果生成所述选取的视频帧图像与所述全景图像之间的透视变换映射矩阵；
基于所述透视变换映射矩阵，分别将所述多个视频帧图像中的每个视频帧图像与所述全景图像进行图像融合。
在一种可选的方式中，所述基于所述透视变换映射矩阵，分别将所述多个视频帧图像中的每个视频帧图像与所述全景图像进行图像融合，包括：
基于所述透视变换映射矩阵，生成从所述选取的视频帧图像映射到所述全景图像的映射图像；
基于所述映射图像与所述全景图像之间的同名像点的色彩差异，分别对所述多个视频帧图像进行色彩调整；
将色彩调整后的多个视频帧图像分别与所述全景图像进行图像融合。
在一种可选的方式中，所述基于所述映射图像与所述全景图像之间的同名像点的色彩差异，分别对所述多个视频帧图像进行色彩调整，包括：
基于所述映射图像与所述全景图像之间的同名像点的色彩差异，生成色彩映射函数或色彩查找表；
基于所述色彩映射函数或所述色彩查找表，分别对所述多个视频帧图像进行色彩调整。
在一种可选的方式中，所述基于所述透视变换映射矩阵，分别将所述多个视频帧图像中的每个视频帧图像与所述全景图像进行图像融合，包括：
基于所述透视变换映射矩阵，确定所述全景图像中与所述每个视频帧图像对应的图像融合区域；
针对每个视频帧图像，将所述全景图像中对应的图像融合区域内的像素点替换为视频帧图像的像素点后，进行羽化。
在一种可选的方式中，所述对所述全景图像与所述多个视频帧图像进行特征匹配，并基于特征匹配结果，将所述多个视频帧图像中的各视频帧图像分别与所述全景图像进行图像融合，包括：
将所述多个视频帧图中的每个视频帧图像与所述全景图像进行特征点匹配，基于特征点匹配结果生成所述每个视频帧图像与所述全景图像之间的透视变换映射矩阵；
基于所述每个视频帧图像与所述全景图像之间的透视变换映射矩阵，将所述每个视频帧图像分别与所述全景图像进行图像融合。
在一种可选的方式中，所述基于所述每个视频帧图像与所述全景图像之间的透视变换映射矩阵，将所述多个视频帧图像分别与所述全景图像进行图像融合，包括：
基于所述每个视频帧图像与所述全景图像之间的透视变换映射矩阵，生成多个映射图像，所述多个映射图像包括从所述每个视频帧图像映射到所述全景图像的映射图像；
基于所述多个映射图像与所述全景图像之间的同名像点的色彩差异，分别对所述多个视频帧图像进行色彩调整；
将色彩调整后的多个视频帧图像分别与所述全景图像进行图像融合。
在一种可选的方式中，所述基于所述每个视频帧图像与所述全景图像之间的透视变换映射矩阵，将所述多个视频帧图像分别与所述全景图像进行图像融合，包括：
基于所述每个视频帧图像与所述全景图像之间的透视变换映射矩阵，确定所述每个视频帧图像与所述全景图像的图像融合区域；
将所述全景图像在所述图像融合区域内的像素点分别替换为所述多个视频帧图像的像素点后，进行羽化。
本公开实施例的第二个方面，提供一种全景视频的生成装置，包括：
全景图像获取模块，用于获取包括目标场景的全景图像；
视频帧图像获取模块，用于获取针对所述目标场景采集的视频，并从所述视频中选取出多个视频帧图像；
全景帧图像获取模块，用于对所述全景图像与所述多个视频帧图像进行特征匹配，并基于特征匹配结果，将所述多个视频帧图像中的每个视频帧图像分别与所述全景图像进行图像融合，得到多个全景帧图像；
动画生成模块，用于基于所述多个全景帧图像生成全景视频。
在一种可选的方式中，所述多个视频帧图像之间的图像采集位置和图像采集角度均相同；所述全景帧图像获取模块包括：
视频帧图像获取单元，用于从所述多个视频帧图像中选取一视频帧图像；
第一透视变换映射矩阵生成单元，用于对选取的视频帧图像与所述全景图像进行特征点匹配，基于特征点匹配结果生成所述选取的视频帧图像与所述全景图像之间的透视变换映射矩阵；
第一图像融合单元，用于基于所述透视变换映射矩阵，分别将所述多个视频帧图像中的每个视频帧图像与所述全景图像进行图像融合。
在一种可选的方式中，所述第一图像融合单元包括：
第一映射图像生成子单元，用于基于所述透视变换映射矩阵，生成从所述选取的视频帧图像映射到所述全景图像的映射图像；
第一色彩调整子单元，用于基于所述映射图像与所述全景图像之间的同名像点的色彩差异，分别对所述多个视频帧图像进行色彩调整；
第一图像融合子单元，用于将色彩调整后的多个视频帧图像分别与所述全景图像进行图像融合。
在一种可选的方式中，所述第一色彩调整子单元用于基于所述映射图像与所述全景图像之间的同名像点的色彩差异，生成色彩映射函数或色彩查找表，并基于所述色彩映射函数或所述色彩查找表，分别对所述多个视频帧图像进行色彩调整。
在一种可选的方式中，所述第一图像融合单元包括：
图像融合区域确定子单元，用于基于所述透视变换映射矩阵，确定所述全景图像中与所述每个视频帧图像对应的图像融合区域；
图像融合子单元，用于对每个视频帧图像，将所述全景图像中对应的图像融合区域内的像素点替换为视频帧图像的像素点后，进行羽化。
在一种可选的方式中，所述全景帧图像获取模块包括：
第二透视变换映射矩阵生成单元，用于将所述多个视频帧图中的每个视频帧图像分别与所述全景图像进行特征点匹配，基于特征点匹配结果生成所述每个视频帧图像与所述全景图像之间的透视变换映射矩阵；
第二图像融合单元，用于基于所述每个视频帧图像与所述全景图像之间的透视变换映射矩阵，将所述每个视频帧图像分别与所述全景图像进行图像融合。
在一种可选的方式中，所述第二图像融合单元包括：
第二映射图像生成子单元，用于基于所述每个视频帧图像与所述全景图像之间的透视变换映射矩阵，生成多个映射图像，所述多个映射图像包括从所述每个视频帧图像映射到所述全景图像的映射图像；
第二色彩调整子单元，用于基于所述多个映射图像与所述全景图像之间的同名像点的色彩差异，分别对所述多个视频帧图像进行色彩调整；
第二图像融合子单元，用于将色彩调整后的多个视频帧图像分别与所述全景图像进行图像融合。
本公开实施例的第三个方面，提供一种电子设备，包括：
存储器，用于存储计算机程序；
处理器，用于执行所述存储器中存储的计算机程序，且所述计算机程序被执行时，实现上述第一方面所述的全景视频的生成方法。
本公开实施例的第四个方面，提供一种计算机可读存储介质，其上存储有计算机程序，该计算机程序被处理器执行时，实现上述第一方面所述的全景视频的生成方法。
本公开的全景视频的生成方法和装置、以及图像融合方法和装置，通过将包括目标场景的全景图像与针对该目标场景的视频帧图像进行融合，可以生成以目标场景进行画面变化的动画视频。本公开的实施例中，视频帧图像仅通过单目相机采集即可，无需采用支持视频录制的全景相机。此外，本公开实施例仅需采集全景中运动部分的视频，可以大大减少对网络环境质量的依赖，因此相对于全景视频呈现动画的方式对网络环境要求低。
下面通过附图和实施例，对本公开的技术方案做进一步的详细描述。
附图说明
构成说明书的一部分的附图描述了本公开的实施例，并且连同描述一起用于解释本公开的原理。
参照附图，根据下面的详细描述，可以更加清楚地理解本公开，其中：
图1为本公开实施例的全景视频的生成方法的流程图。
图2是本公开一个示例中的目标全景图像。
图3是本公开一个示例中全景视频的画面变化区域的示意图。
图4是本公开一个示例中将一帧视频帧图像未做处理直接粘贴在全景图像中的示意图。
图5是对图4进行色彩调整后的全景帧图像。
图6为本公开实施例的全景视频的生成装置的结构框图。
图7为本公开电子设备一个应用实施例的结构示意图。
具体实施方式
现在将参照附图来详细描述本公开的各种示例性实施例。应注意到：除非另外具 体说明，否则在这些实施例中阐述的部件和步骤的相对布置、数字表达式和数值不限制本公开的范围。
本领域技术人员可以理解，本公开实施例中的“第一”、“第二”等术语仅用于区别不同步骤、设备或模块等，既不代表任何特定技术含义，也不表示它们之间的必然逻辑顺序。
还应理解，在本公开实施例中，“多个”可以指两个或两个以上，“至少一个”可以指一个、两个或两个以上。
还应理解，对于本公开实施例中提及的任一部件、数据或结构，在没有明确限定或者在前后文给出相反启示的情况下，一般可以理解为一个或多个。
另外，本公开中术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本公开中字符“/”，一般表示前后关联对象是一种“或”的关系。
还应理解，本公开对各个实施例的描述着重强调各个实施例之间的不同之处，其相同或相似之处可以相互参考，为了简洁，不再一一赘述。
以下对至少一个示例性实施例的描述实际上仅仅是说明性的，决不作为对本公开及其应用或使用的任何限制。
对于相关领域普通技术人员已知的技术、方法和设备可能不作详细讨论，但在适当情况下，所述技术、方法和设备应当被视为说明书的一部分。
应注意到：相似的标号和字母在下面的附图中表示类似项，因此，一旦某一项在一个附图中被定义，则在随后的附图中不需要对其进行进一步讨论。
本公开实施例可以应用于终端设备、计算机系统、服务器等电子设备，其可与众多其它通用或专用计算系统环境或配置一起操作。适于与终端设备、计算机系统、服务器等电子设备一起使用的众所周知的终端设备、计算系统、环境和/或配置的例子包括但不限于：个人计算机系统、服务器计算机系统、瘦客户机、厚客户机、手持或膝上设备、基于微处理器的系统、机顶盒、可编程消费电子产品、网络个人电脑、小型计算机系统﹑大型计算机系统和包括上述任何系统的分布式云计算技术环境，等等。
终端设备、计算机系统、服务器等电子设备可以在由计算机系统执行的计算机系统可执行指令(诸如程序模块)的一般语境下描述。通常，程序模块可以包括例程、程序、目标程序、组件、逻辑、数据结构等等，它们执行特定的任务或者实现特定的抽象数据类型。计算机系统/服务器可以在分布式云计算环境中实施，分布式云计算环境中，任务是由通过通信网络链接的远程处理设备执行的。在分布式云计算环境中，程序模块可以位于包括存储设备的本地或远程计算系统存储介质上。
图1为本公开实施例的全景视频的生成方法的流程图。如图1所示，本公开实施例的全景视频的生成方法，包括：
S1：获取包括目标场景的全景图像。
其中，全景图像可以是一套房间的全景图，目标场景可以是该套房间内的其中一个房间的视频拍摄场景。在本公开实施例中，全景图像可以是通过单目相机拍摄多张图 像后，通过对多张图像进行图像处理技术处理后得到全景图像。其中，图像处理技术包括图像拼接和图像融合等。此外，全景图像也可以是通过全景相机拍摄得到的。
S2：获取针对目标场景采集的视频，并从视频中选取出多个视频帧图像。
其中，本公开的实施例可以通过单目相机对准目标场景拍摄视频，然后从视频中选取出多个视频帧图像。其中，多个视频帧图像可以包括该视频的多个关键帧图像，或者多个视频帧图像包括该视频的所有视频帧图像。
其中，选取视频选取关键帧的方式可以采用基于镜头的关键帧提取方法，可以采用基于运动分析的关键帧提取方法，或采用基于视频聚类的方法。基于镜头的关键帧提取方法，先按照某种技术手段把源视频文件按照镜头变化分割，然后在视频每个镜头中选择首、尾两帧作为关键帧。基于运动分析的方法，在视频镜头中分析物体运动的光流量，每次选择视频镜头中光流移动次数最少的视频帧作为提取到的关键帧。基于视频聚类的方法，首先，初始化一个聚类中心。其次，通过计算聚类中心与当前帧之间的范围，确定被分为类的参考帧或者作为类的新聚类中心。最后，我们选择离聚类中心最近的视频帧处理成关键帧。需要说明的是，本公开并不限定关键帧的提取方法。
S3：对全景图像与多个视频帧图像进行特征匹配，并基于特征匹配结果，将多个视频帧图像中的每个视频帧图像分别与全景图像进行图像融合，得到多个全景帧图像。
其中，特征匹配包括特征采集和特征点匹配。特征匹配结果可以包括全景图像分别与多个视频帧图像的特征点匹配结果。在本公开中，多个全景帧图像中，仅在与每个视频帧图像融合的区域发生画面变化。
S4：基于多个全景帧图像生成全景视频。
在本实施例中，本公开实施例中，视频帧图像仅通过单目相机采集即可，无需采用支持视频录制的全景相机。图2是本公开一个示例中的目标全景图像，图3是本公开一个示例中一帧全景视频的画面变化区域的示意图。如图2和图3所示，全景视频中仅在画面变化区域内的像素发生变化，其他图像区域内不发生变化。本公开实施例仅需采集全景中运动部分的视频，可以大大减少对网络环境质量的依赖，因此相对于全景视频呈现动画的方式对网络环境要求低。
在本公开的一个实施例中，多个视频帧图像之间的图像采集位置和图像采集角度均相同。其中，多个视频帧图像可以通过云台相机拍摄视频得到。需要说明的是，云台相机在拍摄视频时，拍摄参数保持不变。其中，拍摄参数除了包括图像采集位置和图像采集角以外、还包括图像采集焦距和图像分辨率等参数，即除了多个视频帧图像之间的图像采集位置和图像采集角度均相同以外，多个视频帧图像之间的图像采集焦距和图像分辨率也均相同。相应地，步骤S3包括：
S3-1：从多个视频帧图像中选取一视频帧图像。其中，当多个视频帧图像仅包括多个关键帧图像时，可以将视频播放时间点最靠前的关键帧图像作为选区的视频帧图像；当多个视频帧图像包括视频的所有视频帧图像时，可以将视频的第一帧图像作为选区的视频帧图像。
S3-2：对选取的视频帧图像与全景图像进行特征点匹配，基于特征点匹配结果生成 选取的视频帧图像与全景图像之间的透视变换映射矩阵。
具体地，对选取的视频帧图像进行特征提取得到第一图像特征，并对全景图像进行特征提取得到第二图像特征。其中，对取的视频帧图像和全景图像进行特征提取时，使用相同地体征提取权重。
通过对第一图像特征和第二图像特征进行特征点匹配，可以得到选取的视频帧图像与和全景图像之间的匹配点对。
基于选取的视频帧图像与和全景图像之间的匹配点对，确定从选取的视频帧图像映射到全景图像的透视变换映射矩阵。其中，透视变换映射矩阵是3x3矩阵，该矩阵隐式表达了空间中视频帧图像的平面与全景图像的平面之间的相对旋转、平移以及平面参数。
S3-3：基于透视变换映射矩阵，分别将多个视频帧图像中的每个视频帧图像与全景图像进行图像融合。其中，由于多个视频帧图像之间的图像采集位置和图像采集角度均相同，因此多个视频帧图像中的每个视频帧图像与全景图像之间的透视变换映射矩阵也完全相同。基于同样的透视变换映射矩阵，分别将多个视频帧图像中的每个视频帧图像与全景图像进行图像融合，可以得到多个全景帧图像。
图4是本公开一个示例中将一帧视频帧图像未做处理直接粘贴在全景图像中的示意图。如图4所示，图像融合区域与全景图像存在明显视觉差异。图5是对图4进行色彩调整后的全景帧图像，如图5所示，经过色彩调整后，有效降低了图像融合区域与其他区域的视觉差异。
在本实施例中，由于多个视频帧图像之间的图像采集位置和图像采集角度均相同，因此在将多个视频帧图像中的每个视频帧图像分别与全景图像进行图像融合时，在全景图像中的图像融合区域位置保持不变，进而可以仅需建立一次透视变换映射矩阵，然后基于透视变换映射矩阵既可实现多个视频帧图像中的每个视频帧图像分别与全景图像进行图像融合。
在本公开的一个实施例中，步骤S3-3包括：
S3-3-A-1：基于透视变换映射矩阵，生成从选取的视频帧图像映射到全景图像的映射图像。即通过透视变换映射矩阵对选取的视频帧图像进行映射变换，既可生成该映射图像。
图2是本公开一个示例中的全景图像，图3是本公开一个示例中包括映射图像的全景图像。
S3-3-A-2：基于映射图像与全景图像之间的同名像点的色彩差异，分别对多个视频帧图像进行色彩调整。同名像点也称相应像点，指任一目标点在不同相片上的构像点。
其中，需要首先获取映射图像与全景图像之间的同名像点的色彩差异。在本实施例中，色彩差异包括灰度图差异和RGB差异等。色彩差异可以通过同名像点的像素值比对的方式得到。以灰度图差异为例，假设第一映射图像I1和目标全景图像的对应区域I2上任意同名像点分别为p1(x1，y1)和p2(x2，y2)，则对应的灰度图上的强度差||I1(x1，y1)-I2(x2，y2)||，即为色彩差异。在得到映射图像与全景图像之间的同名像点的色彩差 异后，基于同名像点的色彩差异，确定同名像点之间的色彩映射关系。基于色彩映射关系，分别对多个视频帧图像进行色彩调整。
S3-3-A-3：将色彩调整后的多个视频帧图像分别与全景图像进行图像融合。
在本实施例中，通过色彩调整可以降低视频帧图像与全景图像进行融合后的视觉差异。
在本公开的一个实施例中，步骤S3-3-A-3包括：基于映射图像与全景图像之间的同名像点的色彩差异，生成色彩映射函数或色彩查找表；基于色彩映射函数或色彩查找表，分别对多个视频帧图像进行色彩调整。
在公开的一个示例中，通过求解一个色彩映射函数，使得选取的视频帧图像对应的映射图像与全景图像之间的同名像点的色彩差异最小。其中，使用的映射函数为指数函数的二阶泰勒级数展开，因为认为一般色差不会太大，同时高阶信息会迅速降零。基于色差映射函数对选取的视频帧图像的像素点进行色彩调整。对于多个视频帧图像中除了之前选取的视频帧图像之外剩余的每个视频帧图像，获取每个视频帧图像的所有像素点信息，然后基于色差映射函数对每个视频帧图像的像素点进行色彩调整。
在本公开的另一个示例中，通过建立选取的视频帧图像的像素点与全景图像的像素点之间的色彩查找表，可以使得选取的视频帧图像对应的映射图像与全景图像之间的同名像点的色彩差异最小。在本公开的一个示例中，以灰度图为例，在映射图像上找到一组像素点，使其强度值覆盖0～255。在全景图像上找到同名像点对应的强度值，构建查找表。其中，缺失的数据通过插值构造。本公开通过查指表的方式可以快速准确地建立映射图像与目标全景图像的同名像点的映射关系。基于色彩查找表对选取的视频帧图像的像素点进行色彩调整。对于多个视频帧图像中除了之前选取的视频帧图像之外剩余的每个视频帧图像，获取每个视频帧图像的所有像素点信息，然后基于色彩查找表对每个视频帧图像的像素点进行色彩调整。
在本公开的另一个实施例中，步骤S3-3包括：
S3-3-B-1：基于透视变换映射矩阵，确定全景图像中与每个视频帧图像对应的图像融合区域。其中，由于多个视频帧图像之间的图像采集位置和图像采集角度均相同，因此在将多个视频帧图像中的每个视频帧图像分别与全景图像进行图像融合时，在全景图像中的图像融合区域位置保持不变。
S3-3-B-2：对每个视频帧图像，将全景图像中对应的图像融合区域内的像素点替换为视频帧图像的像素点后，进行羽化。其中，羽化原理是令第一映射图像边缘衔接部分虚化，起到渐变的作用从而达到自然衔接的效果。
在本发明的另一个实施例中，步骤S3包括：将多个视频帧图中的每个视频帧图像分别与全景图像进行特征点匹配，基于特征点匹配结果生成每个视频帧图像与全景图像之间的透视变换映射矩阵；基于每个视频帧图像与全景图像之间的透视变换映射矩阵，将每个视频帧图像分别与全景图像进行图像融合。
在本实施例中，当采集视频时，如果图像采集参数发生变化，例如图像采集装置发生了抖动导致多个视频帧图像之间的图像采集位置和图像采集角度发生了改变，需要 对每个视频帧图像分别建立透视变换映射矩阵，进而基于每个视频帧图像分别建立的透视变换映射矩阵，调整每个视频帧图像在对应的全景图像中的融合位置，保证每个视频帧图像的融合效果。
相应地，在基于每个视频帧图像与全景图像之间的透视变换映射矩阵，将多个视频帧图像分别与全景图像进行图像融合，包括：基于每个视频帧图像与全景图像之间的透视变换映射矩阵，生成多个映射图像，多个映射图像包括从每个视频帧图像映射到全景图像的映射图像；基于多个映射图像与全景图像之间的同名像点的色彩差异，分别对多个视频帧图像进行色彩调整；将色彩调整后的多个视频帧图像分别与全景图像进行图像融合。
在本实施例中，当对每个视频帧图像分别建立了透视变换映射矩阵后，需要针对每个视频帧图像对应的透视变换映射矩阵进行投射变换，从而可以得到每个视频帧图像对应的映射图像，然后基于每个视频帧图像对应的映射图像进行色彩调整，可以保证每个视频帧图像的融合效果。
本公开任一实施例提供的全景视频的生成装置可用于实现本公开上述实施例中全景视频的生成方法。本公开任一实施例提供的全景视频的生成装置可以设置在终端设备上，也可以设置在服务端上，或者部分设置在终端设备上，部分设置在服务端上。示例性实施例提供的全景视频的生成装置可以包括：处理器；用于存储所述处理器可执行指令的存储器；所述处理器，用于从所述存储器中读取所述可执行指令，并执行所述指令以实现本公开示例性实施例提供的全景视频的生成方法。
图6为本公开实施例的全景视频的生成装置的结构框图。如图6所示，本公开实施例的全景视频的生成装置，包括：全景图像获取模块610、视频帧图像获取模块620、全景帧图像获取模块630和动画生成模块640。
全景图像获取模块610用于获取包括目标场景的全景图像。视频帧图像获取模块620用于获取针对目标场景采集的视频，并从视频中选取出多个视频帧图像。全景帧图像获取模块630用于对全景图像与多个视频帧图像进行特征匹配，并基于特征匹配结果，将多个视频帧图像中的每个视频帧图像分别与全景图像进行图像融合，得到多个全景帧图像。动画生成模块640用于基于多个全景帧图像生成全景视频。
在本公开的一个实施例中，多个视频帧图像之间的图像采集位置和图像采集角度均相同。全景帧图像获取模块630包括：
视频帧图像获取单元，用于从多个视频帧图像中选取一视频帧图像；
第一透视变换映射矩阵生成单元，用于对选取的视频帧图像与全景图像进行特征点匹配，基于特征点匹配结果生成选取的视频帧图像与全景图像之间的透视变换映射矩阵；
第一图像融合单元，用于基于透视变换映射矩阵，分别将多个视频帧图像中的每个视频帧图像与全景图像进行图像融合。
在本公开的一个实施例中，第一图像融合单元包括：
第一映射图像生成子单元，用于基于透视变换映射矩阵，生成从选取的视频帧图 像映射到全景图像的映射图像；
第一色彩调整子单元，用于基于映射图像与全景图像之间的同名像点的色彩差异，分别对多个视频帧图像进行色彩调整；
第一图像融合子单元，用于将色彩调整后的多个视频帧图像分别与全景图像进行图像融合。
在本公开的一个实施例中，第一色彩调整子单元用于基于映射图像与全景图像之间的同名像点的色彩差异，生成色彩映射函数或色彩查找表，并基于色彩映射函数或色彩查找表，分别对多个视频帧图像进行色彩调整。
在本公开的一个实施例中，第一图像融合单元包括：
图像融合区域确定子单元，用于基于透视变换映射矩阵，确定全景图像中与每个视频帧图像对应的图像融合区域；
图像融合子单元，用于对每个视频帧图像，将全景图像中对应的图像融合区域内的像素点替换为视频帧图像的像素点后，进行羽化。
在本公开的一个实施例中，全景帧图像获取模块630包括：
第二透视变换映射矩阵生成单元，用于将多个视频帧图中的每个视频帧图像分别与全景图像进行特征点匹配，基于特征点匹配结果生成每个视频帧图像与全景图像之间的透视变换映射矩阵；
第二图像融合单元，用于基于每个视频帧图像与全景图像之间的透视变换映射矩阵，将每个视频帧图像分别与全景图像进行图像融合。
在本公开的一个实施例中，第二图像融合单元包括：
第二映射图像生成子单元，用于基于每个视频帧图像与全景图像之间的透视变换映射矩阵，生成多个映射图像，多个映射图像包括从每个视频帧图像映射到全景图像的映射图像；
第二色彩调整子单元，用于基于多个映射图像与全景图像之间的同名像点的色彩差异，分别对多个视频帧图像进行色彩调整；
第二图像融合子单元，用于将色彩调整后的多个视频帧图像分别与全景图像进行图像融合。
需要说明的是，本公开实施例的全景视频的生成装置是与全景视频的生成方法对应的装置，为了减少冗余，对本公开实施例的全景视频的生成装置的具体实施方式不作赘述。
另外，本公开实施例还提供了一种电子设备，包括：
存储器，用于存储计算机程序；
处理器，用于执行所述存储器中存储的计算机程序，且所述计算机程序被执行时，实现本公开上述任一实施例所述的全景视频的生成方法。
图7为本公开电子设备一个应用实施例的结构示意图。下面，参考图7来描述根据本公开实施例的电子设备。该电子设备可以是第一设备和第二设备中的任一个或两者、或与它们独立的单机设备，该单机设备可以与第一设备和第二设备进行通信，以从它们 接收所采集到的输入信号。
如图7所示，电子设备包括一个或多个处理器和存储器。
处理器可以是中央处理单元(CPU)或者具有数据处理能力和/或指令执行能力的其他形式的处理单元，并且可以控制电子设备中的其他组件以执行期望的功能。
存储器可以包括一个或多个计算机程序产品，所述计算机程序产品可以包括各种形式的计算机可读存储介质，例如易失性存储器和/或非易失性存储器。所述易失性存储器例如可以包括随机存取存储器(RAM)和/或高速缓冲存储器(cache)等。所述非易失性存储器例如可以包括只读存储器(ROM)、硬盘、闪存等。在所述计算机可读存储介质上可以存储一个或多个计算机程序指令，处理器可以运行所述程序指令，以实现上文所述的本公开的各个实施例的全景视频的生成方法以及/或者其他期望的功能。
在一个示例中，电子设备还可以包括：输入装置和输出装置，这些组件通过总线系统和/或其他形式的连接机构(未示出)互连。
此外，该输入设备还可以包括例如键盘、鼠标等等。
该输出装置可以向外部输出各种信息，包括确定出的距离信息、方向信息等。该输出设备可以包括例如显示器、扬声器、打印机、以及通信网络及其所连接的远程输出设备等等。
当然，为了简化，图7中仅示出了该电子设备中与本公开有关的组件中的一些，省略了诸如总线、输入/输出接口等等的组件。除此之外，根据具体应用情况，电子设备还可以包括任何其他适当的组件。
除了上述方法和设备以外，本公开的实施例还可以是计算机程序产品，其包括计算机程序指令，所述计算机程序指令在被处理器运行时使得所述处理器执行本说明书上述部分中描述的根据本公开各种实施例的全景视频的生成方法中的步骤。
所述计算机程序产品可以以一种或多种程序设计语言的任意组合来编写用于执行本公开实施例操作的程序代码，所述程序设计语言包括面向对象的程序设计语言，诸如Java、C++等，还包括常规的过程式程序设计语言，诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算设备上执行、部分地在用户设备上执行、作为一个独立的软件包执行、部分在用户计算设备上部分在远程计算设备上执行、或者完全在远程计算设备或服务器上执行。
此外，本公开的实施例还可以是计算机可读存储介质，其上存储有计算机程序指令，所述计算机程序指令在被处理器运行时使得所述处理器执行本说明书上述部分中描述的根据本公开各种实施例的全景视频的生成方法中的步骤。
所述计算机可读存储介质可以采用一个或多个可读介质的任意组合。可读介质可以是可读信号介质或者可读存储介质。可读存储介质例如可以包括但不限于电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。可读存储介质的更具体的例子(非穷举的列表)包括：具有一个或多个导线的电连接、便携式盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或 者上述的任意合适的组合。
本领域普通技术人员可以理解：实现上述方法实施例的全部或部分步骤可以通过程序指令相关的硬件来完成，前述的程序可以存储于一计算机可读取存储介质中，该程序在执行时，执行包括上述方法实施例的步骤；而前述的存储介质包括：ROM、RAM、磁碟或者光盘等各种可以存储程序代码的介质。
以上结合具体实施例描述了本公开的基本原理，但是，需要指出的是，在本公开中提及的优点、优势、效果等仅是示例而非限制，不能认为这些优点、优势、效果等是本公开的各个实施例必须具备的。另外，上述公开的具体细节仅是为了示例的作用和便于理解的作用，而非限制，上述细节并不限制本公开为必须采用上述具体的细节来实现。
本说明书中各个实施例均采用递进的方式描述，每个实施例重点说明的都是与其它实施例的不同之处，各个实施例之间相同或相似的部分相互参见即可。对于系统实施例而言，由于其与方法实施例基本对应，所以描述的比较简单，相关之处参见方法实施例的部分说明即可。
本公开中涉及的器件、装置、设备、系统的方框图仅作为例示性的例子并且不意图要求或暗示必须按照方框图示出的方式进行连接、布置、配置。如本领域技术人员将认识到的，可以按任意方式连接、布置、配置这些器件、装置、设备、系统。诸如“包括”、“包含”、“具有”等等的词语是开放性词汇，指“包括但不限于”，且可与其互换使用。这里所使用的词汇“或”和“和”指词汇“和/或”，且可与其互换使用，除非上下文明确指示不是如此。这里所使用的词汇“诸如”指词组“诸如但不限于”，且可与其互换使用。
可能以许多方式来实现本公开的方法和装置。例如，可通过软件、硬件、固件或者软件、硬件、固件的任何组合来实现本公开的方法和装置。用于所述方法的步骤的上述顺序仅是为了进行说明，本公开的方法的步骤不限于以上具体描述的顺序，除非以其它方式特别说明。此外，在一些实施例中，还可将本公开实施为记录在记录介质中的程序，这些程序包括用于实现根据本公开的方法的机器可读指令。因而，本公开还覆盖存储用于执行根据本公开的方法的程序的记录介质。
还需要指出的是，在本公开的装置、设备和方法中，各部件或各步骤是可以分解和/或重新组合的。这些分解和/或重新组合应视为本公开的等效方案。
提供所公开的方面的以上描述以使本领域的任何技术人员能够做出或者使用本公开。对这些方面的各种修改对于本领域技术人员而言是非常显而易见的，并且在此定义的一般原理可以应用于其他方面而不脱离本公开的范围。因此，本公开不意图被限制到在此示出的方面，而是按照与在此公开的原理和新颖的特征一致的最宽范围。
为了例示和描述的目的已经给出了以上描述。此外，此描述不意图将本公开的实施例限制到在此公开的形式。尽管以上已经讨论了多个示例方面和实施例，但是本领域技术人员将认识到其某些变型、修改、改变、添加和子组合。

Claims (16)

  1. 一种全景视频的生成方法，其特征在于，包括：
    获取包括目标场景的全景图像；
    获取针对所述目标场景采集的视频，并从所述视频中选取出多个视频帧图像；
    对所述全景图像与所述多个视频帧图像进行特征匹配，并基于特征匹配结果，将所述多个视频帧图像中的每个视频帧图像分别与所述全景图像进行图像融合，得到多个全景帧图像；
    基于所述多个全景帧图像生成全景视频。
  2. 根据权利要求1所述的全景视频的生成方法，其特征在于，所述多个视频帧图像之间的图像采集位置和图像采集角度均相同；所述对所述全景图像与所述多个视频帧图像进行特征匹配，并基于特征匹配结果，将所述多个视频帧图像中的每个视频帧图像分别与所述全景图像进行图像融合，包括：
    从所述多个视频帧图像中选取一视频帧图像；
    对选取的视频帧图像与所述全景图像进行特征点匹配，基于特征点匹配结果生成所述选取的视频帧图像与所述全景图像之间的透视变换映射矩阵；
    基于所述透视变换映射矩阵，分别将所述多个视频帧图像中的每个视频帧图像与所述全景图像进行图像融合。
  3. 根据权利要求2所述的全景视频的生成方法，其特征在于，所述基于所述透视变换映射矩阵，分别将所述多个视频帧图像中的每个视频帧图像与所述全景图像进行图像融合，包括：
    基于所述透视变换映射矩阵，生成从所述选取的视频帧图像映射到所述全景图像的映射图像；
    基于所述映射图像与所述全景图像之间的同名像点的色彩差异，分别对所述多个视频帧图像进行色彩调整；
    将色彩调整后的多个视频帧图像分别与所述全景图像进行图像融合。
  4. 根据权利要求3所述的全景视频的生成方法，其特征在于，所述基于所述映射图像与所述全景图像之间的同名像点的色彩差异，分别对所述多个视频帧图像进行色彩调整，包括：
    基于所述映射图像与所述全景图像之间的同名像点的色彩差异，生成色彩映射函数或色彩查找表；
    基于所述色彩映射函数或所述色彩查找表，分别对所述多个视频帧图像进行色彩调整。
  5. 根据权利要求2所述的全景视频的生成方法，其特征在于，所述基于所述透视变换映射矩阵，分别将所述多个视频帧图像中的每个视频帧图像与所述全景图像进行图像融合，包括：
    基于所述透视变换映射矩阵，确定所述全景图像中与所述每个视频帧图像对应的图像融合区域；
    对每个视频帧图像，将所述全景图像中对应的图像融合区域内的像素点替换为视频帧图像的像素点后，进行羽化。
  6. 根据权利要求1所述的全景视频的生成方法，其特征在于，所述对所述全景图像与所述多个视频帧图像进行特征匹配，并基于特征匹配结果，将所述多个视频帧图像中的各视频帧图像分别与所述全景图像进行图像融合，包括：
    将所述多个视频帧图中的每个视频帧图像与所述全景图像进行特征点匹配，基于特征点匹配结果生成所述每个视频帧图像与所述全景图像之间的透视变换映射矩阵；
    基于所述每个视频帧图像与所述全景图像之间的透视变换映射矩阵，将所述每个视频帧图像分别与所述全景图像进行图像融合。
  7. 根据权利要求6所述的全景视频的生成方法，其特征在于，所述基于所述每个视频帧图像与所述全景图像之间的透视变换映射矩阵，将所述多个视频帧图像分别与所述全景图像进行图像融合，包括：
    基于所述每个视频帧图像与所述全景图像之间的透视变换映射矩阵，生成多个映射图像，所述多个映射图像包括从所述每个视频帧图像映射到所述全景图像的映射图像；
    基于所述多个映射图像与所述全景图像之间的同名像点的色彩差异，分别对所述多个视频帧图像进行色彩调整；
    将色彩调整后的多个视频帧图像分别与所述全景图像进行图像融合。
  8. 一种全景视频的生成装置，其特征在于，包括：
    全景图像获取模块，用于获取包括目标场景的全景图像；
    视频帧图像获取模块，用于获取针对所述目标场景采集的视频，并从所述视频中选取出多个视频帧图像；
    全景帧图像获取模块，用于对所述全景图像与所述多个视频帧图像进行特征匹配，并基于特征匹配结果，将所述多个视频帧图像中的每个视频帧图像分别与所述全景图像进行图像融合，得到多个全景帧图像；
    全景视频生成模块，用于基于所述多个全景帧图像生成全景视频。
  9. 根据权利要求8所述的装置，其特征在于，所述多个视频帧图像之间的图像采集位置和图像采集角度均相同；所述全景帧图像获取模块包括：
    视频帧图像获取单元，用于从所述多个视频帧图像中选取一视频帧图像；
    第一透视变换映射矩阵生成单元，用于对选取的视频帧图像与所述全景图像进行特征点匹配，基于特征点匹配结果生成所述选取的视频帧图像与所述全景图像之间的透视变换映射矩阵；
    第一图像融合单元，用于基于所述透视变换映射矩阵，分别将所述多个视频帧图像中的每个视频帧图像与所述全景图像进行图像融合。
  10. 根据权利要求9所述的装置，其特征在于，所述第一图像融合单元包括：
    第一映射图像生成子单元，用于基于所述透视变换映射矩阵，生成从所述选取的视频帧图像映射到所述全景图像的映射图像；
    第一色彩调整子单元，用于基于所述映射图像与所述全景图像之间的同名像点的色彩差异，分别对所述多个视频帧图像进行色彩调整；
    第一图像融合子单元，用于将色彩调整后的多个视频帧图像分别与所述全景图像进行图像融合。
  11. 根据权利要求10所述的装置，其特征在于，所述第一色彩调整子单元用于：基于所述映射图像与所述全景图像之间的同名像点的色彩差异，生成色彩映射函数或色彩查找表，并基于所述色彩映射函数或所述色彩查找表，分别对所述多个视频帧图像进行色彩调整。
  12. 根据权利要求9所述的装置，其特征在于，所述第一图像融合单元包括：
    图像融合区域确定子单元，用于基于所述透视变换映射矩阵，确定所述全景图像中与所述每个视频帧图像对应的图像融合区域；
    图像融合子单元，用于对每个视频帧图像，将所述全景图像中对应的图像融合区域内的像素点替换为视频帧图像的像素点后，进行羽化。
  13. 根据权利要求8所述的装置，其特征在于，所述全景帧图像获取模块包括：
    第二透视变换映射矩阵生成单元，用于将所述多个视频帧图中的每个视频帧图像分别与所述全景图像进行特征点匹配，基于特征点匹配结果生成所述每个视频帧图像与所述全景图像之间的透视变换映射矩阵；
    第二图像融合单元，用于基于所述每个视频帧图像与所述全景图像之间的透视变换映射矩阵，将所述每个视频帧图像分别与所述全景图像进行图像融合。
  14. 根据权利要求13所述的装置，其特征在于，所述第二图像融合单元包括：
    第二映射图像生成子单元，用于基于所述每个视频帧图像与所述全景图像之间的透视变换映射矩阵，生成多个映射图像，所述多个映射图像包括从所述每个视频帧图像映射到所述全景图像的映射图像；
    第二色彩调整子单元，用于基于所述多个映射图像与所述全景图像之间的同名像点 的色彩差异，分别对所述多个视频帧图像进行色彩调整；
    第二图像融合子单元，用于将色彩调整后的多个视频帧图像分别与所述全景图像进行图像融合。
  15. 一种电子设备，其特征在于，包括：
    存储器，用于存储计算机程序；
    处理器，用于执行所述存储器中存储的计算机程序，且所述计算机程序被执行时，实现上述权利要求1-7任一所述的全景视频的生成方法。
  16. 一种计算机可读存储介质，其上存储有计算机程序，其特征在于，该计算机程序被处理器执行时，实现上述权利要求1-7任一所述的全景视频的生成方法。
PCT/CN2022/072978 2021-07-30 2022-01-20 全景视频的生成方法和装置 WO2023005170A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110873542.XA CN113572978A (zh) 2021-07-30 2021-07-30 全景视频的生成方法和装置
CN202110873542.X 2021-07-30

Publications (1)

Publication Number Publication Date
WO2023005170A1 true WO2023005170A1 (zh) 2023-02-02

Family

ID=78169505

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/072978 WO2023005170A1 (zh) 2021-07-30 2022-01-20 全景视频的生成方法和装置

Country Status (3)

Country Link
US (1) US11812154B2 (zh)
CN (1) CN113572978A (zh)
WO (1) WO2023005170A1 (zh)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113572978A (zh) * 2021-07-30 2021-10-29 北京房江湖科技有限公司 全景视频的生成方法和装置
CN114007017A (zh) * 2021-11-18 2022-02-01 浙江博采传媒有限公司 一种视频生成方法、装置及存储介质
CN114862989B (zh) * 2022-05-20 2023-03-28 远峰科技股份有限公司 一种全景环视图像的盲区填充方法与装置

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105844256A (zh) * 2016-04-07 2016-08-10 广州盈可视电子科技有限公司 一种全景视频帧图像处理方法和装置
WO2020218024A1 (ja) * 2019-04-24 2020-10-29 日本電信電話株式会社 パノラマ映像合成装置、パノラマ映像合成方法、及びパノラマ映像合成プログラム
CN111866523A (zh) * 2020-07-24 2020-10-30 北京爱笔科技有限公司 全景视频合成方法、装置、电子设备和计算机存储介质
CN112085659A (zh) * 2020-09-11 2020-12-15 中德（珠海）人工智能研究院有限公司 一种基于球幕相机的全景拼接融合方法、系统及存储介质
CN113572978A (zh) * 2021-07-30 2021-10-29 北京房江湖科技有限公司 全景视频的生成方法和装置

Family Cites Families (34)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5850352A (en) * 1995-03-31 1998-12-15 The Regents Of The University Of California Immersive video, including video hypermosaicing to generate from multiple video views of a scene a three-dimensional video mosaic from which diverse virtual video scene images are synthesized, including panoramic, scene interactive and stereoscopic images
US5729471A (en) * 1995-03-31 1998-03-17 The Regents Of The University Of California Machine dynamic selection of one video camera/image of a scene from multiple video cameras/images of the scene in accordance with a particular perspective on the scene, an object in the scene, or an event in the scene
US7594177B2 (en) * 2004-12-08 2009-09-22 Microsoft Corporation System and method for video browsing using a cluster index
US8300083B2 (en) * 2007-07-20 2012-10-30 Hewlett-Packard Development Company, L.P. Position relationships associated with image capturing devices
US9007432B2 (en) * 2010-12-16 2015-04-14 The Massachusetts Institute Of Technology Imaging systems and methods for immersive surveillance
US10848731B2 (en) * 2012-02-24 2020-11-24 Matterport, Inc. Capturing and aligning panoramic image and depth data
KR101946019B1 (ko) * 2014-08-18 2019-04-22 삼성전자주식회사 파노라믹 동영상을 생성하는 동영상 처리 장치 및 그 방법
US11205305B2 (en) * 2014-09-22 2021-12-21 Samsung Electronics Company, Ltd. Presentation of three-dimensional video
US10257494B2 (en) * 2014-09-22 2019-04-09 Samsung Electronics Co., Ltd. Reconstruction of three-dimensional video
KR20160115466A (ko) * 2015-03-27 2016-10-06 한국전자통신연구원 파노라믹 비디오를 스티칭하는 장치 및 이를 위한 스티칭 방법
JP6775776B2 (ja) * 2017-03-09 2020-10-28 株式会社岩根研究所 自由視点移動表示装置
CN107509110A (zh) 2017-08-03 2017-12-22 乐蜜有限公司 一种视频文件的循环播放方法及装置
US10587821B2 (en) * 2018-05-17 2020-03-10 Lockheed Martin Corporation High speed image registration system and methods of use
CN109361880A (zh) * 2018-11-30 2019-02-19 三星电子（中国）研发中心 一种展示静态图片对应的动态图片或视频的方法及系统
CN109697696B (zh) * 2018-12-24 2019-10-18 北京天睿空间科技股份有限公司 用于全景视频的补盲方法
CN111385490B (zh) * 2018-12-28 2021-07-13 清华大学 视频拼接方法及装置
CN109951654B (zh) * 2019-03-06 2022-02-15 腾讯科技（深圳）有限公司 一种视频合成的方法、模型训练的方法以及相关装置
CN110324718B (zh) 2019-08-05 2021-09-07 北京字节跳动网络技术有限公司 音视频生成方法、装置、电子设备及可读介质
CN111178191B (zh) * 2019-11-11 2022-01-11 贝壳找房(北京)科技有限公司 信息播放方法、装置、计算机可读存储介质及电子设备
US11270448B2 (en) * 2019-11-26 2022-03-08 Microsoft Technology Licensing, Llc Using machine learning to selectively overlay image content
US10999530B1 (en) * 2019-11-26 2021-05-04 L'oreal Techniques for generating time-series images of changes in personal appearance
US11354883B2 (en) * 2019-12-30 2022-06-07 Sensetime International Pte. Ltd. Image processing method and apparatus, and electronic device
US11146727B2 (en) * 2020-03-16 2021-10-12 Ke.Com (Beijing) Technology Co., Ltd. Method and device for generating a panoramic image
US11620730B2 (en) * 2020-03-23 2023-04-04 Realsee (Beijing) Technology Co., Ltd. Method for merging multiple images and post-processing of panorama
CN111601033A (zh) * 2020-04-27 2020-08-28 北京小米松果电子有限公司 视频处理方法、装置及存储介质
CN115835010A (zh) * 2020-08-27 2023-03-21 荣耀终端有限公司 一种拍摄方法及终端
CN112562721B (zh) 2020-11-30 2024-04-16 清华珠三角研究院 一种视频翻译方法、系统、装置及存储介质
US11461975B2 (en) * 2020-12-03 2022-10-04 Realsee (Beijing) Technology Co., Ltd. Method and apparatus for generating guidance among viewpoints in a scene
US11683462B2 (en) * 2021-06-04 2023-06-20 Dierks Technology, Inc. Matching segments of video for virtual display of a space
CN113129211B (zh) * 2021-06-16 2021-08-17 贝壳技术有限公司 光心对齐检测方法和装置、存储介质、电子设备
US20230022444A1 (en) * 2021-07-26 2023-01-26 Samsung Electronics Co., Ltd. Electronic device generating image and method for operating the same
CN113592935A (zh) * 2021-07-27 2021-11-02 贝壳技术有限公司 深度图像生成方法和装置
CN113450258B (zh) * 2021-08-31 2021-11-05 贝壳技术有限公司 视角转换方法和装置、存储介质、电子设备
CN113989166B (zh) * 2021-12-27 2022-03-15 贝壳技术有限公司 全景图的垂直矫正方法和装置、电子设备和存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105844256A (zh) * 2016-04-07 2016-08-10 广州盈可视电子科技有限公司 一种全景视频帧图像处理方法和装置
WO2020218024A1 (ja) * 2019-04-24 2020-10-29 日本電信電話株式会社 パノラマ映像合成装置、パノラマ映像合成方法、及びパノラマ映像合成プログラム
CN111866523A (zh) * 2020-07-24 2020-10-30 北京爱笔科技有限公司 全景视频合成方法、装置、电子设备和计算机存储介质
CN112085659A (zh) * 2020-09-11 2020-12-15 中德（珠海）人工智能研究院有限公司 一种基于球幕相机的全景拼接融合方法、系统及存储介质
CN113572978A (zh) * 2021-07-30 2021-10-29 北京房江湖科技有限公司 全景视频的生成方法和装置

Also Published As

Publication number Publication date
US20230033267A1 (en) 2023-02-02
CN113572978A (zh) 2021-10-29
US11812154B2 (en) 2023-11-07

Similar Documents

Publication Publication Date Title
WO2023005170A1 (zh) 全景视频的生成方法和装置
CN110264509B (zh) 确定图像捕捉设备的位姿的方法、装置及其存储介质
CN107113381B (zh) 时空局部变形及接缝查找的容差视频拼接方法、装置及计算机可读介质
US10748265B2 (en) Arbitrary view generation
WO2016155377A1 (zh) 图片展示方法和装置
US10321109B1 (en) Large volume video data transfer over limited capacity bus
WO2018228436A1 (zh) 双视角图像校准及图像处理方法、装置、存储介质和电子设备
WO2015200782A1 (en) 3-d model generation
US10586378B2 (en) Stabilizing image sequences based on camera rotation and focal length parameters
JP7247327B2 (ja) ダイナミックデプス画像を撮影して編集するための手法
CN111429354B (zh) 图像拼接及全景图拼接方法和装置、存储介质、电子设备
EP3660663A1 (en) Delivering virtualized content
CN111402404B (zh) 全景图补全方法、装置、计算机可读存储介质及电子设备
CN111402136B (zh) 全景图生成方法、装置、计算机可读存储介质及电子设备
WO2022262273A1 (zh) 光心对齐检测方法和装置、存储介质、电子设备
CN113132708B (zh) 利用鱼眼相机获取三维场景图像的方法和装置、设备和介质
US9723206B1 (en) Enabling a true surround view of a 360 panorama via a dynamic cylindrical projection of the panorama
KR20210057925A (ko) 스트리밍 서버 및 이를 이용한 다시점 동영상에서의 객체 처리 방법
CN111260544B (zh) 数据处理方法及装置、电子设备和计算机存储介质
WO2018100230A1 (en) Method and apparatuses for determining positions of multi-directional image capture apparatuses
CN108920598B (zh) 全景图浏览方法、装置、终端设备、服务器及存储介质
CN112465716A (zh) 图像转换方法、装置、计算机可读存储介质及电子设备
CN112561795A (zh) 基于Spark和OpenCV的实时生成全景图实现系统及方法
Hayashida et al. Real-time human detection using spherical camera for web browser-based telecommunications
KR102561903B1 (ko) 클라우드 서버를 이용한 ai 기반의 xr 콘텐츠 서비스 방법

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22847791

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE