WO2020135394A1 - 视频拼接方法及装置 - Google Patents

视频拼接方法及装置 Download PDF

Info

Publication number
WO2020135394A1
WO2020135394A1 PCT/CN2019/127805 CN2019127805W WO2020135394A1 WO 2020135394 A1 WO2020135394 A1 WO 2020135394A1 CN 2019127805 W CN2019127805 W CN 2019127805W WO 2020135394 A1 WO2020135394 A1 WO 2020135394A1
Authority
WO
WIPO (PCT)
Prior art keywords
target frame
video
frame
feature point
point pair
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2019/127805
Other languages
English (en)
French (fr)
Inventor
吴华强
何青林
胡晨
王旭光
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tsinghua University
Original Assignee
Tsinghua University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tsinghua University filed Critical Tsinghua University
Priority to US17/049,347 priority Critical patent/US11538177B2/en
Publication of WO2020135394A1 publication Critical patent/WO2020135394A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/265—Mixing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G06T7/337—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00—Indexing scheme for image data processing or generation, in general
    • G06T2200/32—Indexing scheme for image data processing or generation, in general involving image mosaicing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Definitions

  • the embodiments of the present disclosure relate to a video stitching method and device.
  • Video stitching is an important technology in the field of computer vision.
  • the process of video splicing is to splice and synthesize the video images collected by multiple cameras with overlapping areas on the same view plane to form a panoramic video with higher resolution and wider viewing angle.
  • the cost of obtaining panoramic video through video stitching technology is lower and the practicality is higher.
  • video stitching technology has been applied to various fields such as security monitoring, aerial reconnaissance, display and exhibition, and visual entertainment.
  • At least one embodiment of the present disclosure provides a video splicing method for splicing a first video and a second video, including: performing respectively on the first target frame of the first video and the second target frame of the second video Feature extraction, perform feature matching on the features extracted from the first target frame and the second target frame respectively, and filter the matched features to obtain the features of the first target frame and the second target frame A first feature point pair set; forward tracking the forward feature point pair set of the first target frame and the second target frame to the first target frame and the second target frame to obtain the first A second feature point pair set of a target frame and the second target frame; backtracking the backward feature point pair set of the first target frame and the second target frame to the first target frame and The second target frame, obtaining a third feature point pair set of the first target frame and the second target frame; and according to the first feature of the first target frame and the second target frame
  • the union of the point pair set, the second feature point pair set and the third feature point pair set calculates the geometric transformation relationship between the first target frame
  • the video stitching method provided by some embodiments of the present disclosure further includes: merging the first feature point pair set and the second feature point pair set of the first target frame and the second target frame A set of forward feature point pairs that is one frame after the first target frame of the first video and one frame after the second target frame of the second video is set.
  • the video stitching method provided by some embodiments of the present disclosure further includes: if the first target frame and the second target frame are initial frames to be stitched, forward tracking is not performed, and the first target is used according to the first target The union of the first feature point pair set and the third feature point pair set of the frame and the second target frame to calculate the geometric transformation relationship between the first target frame and the second target frame , Using the first feature point pair set of the first target frame and the second target frame as the frame after the first target frame of the first video and after the second target frame of the second video A set of forward feature point pairs for one frame.
  • the video stitching method provided by some embodiments of the present disclosure further includes: if the number of video frames to be stitched after the first target frame and the second target frame is less than the number of backward tracking target frames, then the The final frame to be spliced of the first video and the second video is used as a starting frame for backward tracking.
  • the video stitching method provided by some embodiments of the present disclosure further includes: if the first target frame and the second target frame are of a frame of the first video and the second video to be spliced
  • the first feature point pair set of the first target frame and the second target frame is used as the previous frame and the second video of the first target frame of the first video
  • the set of backward feature points of the previous frame of the second target frame of the second target frame if the first target frame and the second target frame are a frame of the first video and the second video to be stitched
  • the middle frame of the backward tracking of, the union of the first feature point pair set and the third feature point pair set of the first target frame and the second target frame is used as the first video
  • the video splicing method further includes: if the first target frame and the second target frame are the final frames to be spliced, backward tracking is not performed, according to the first target The union of the first feature point pair set and the second feature point pair set of the frame and the second target frame to calculate the geometric transformation relationship between the first target frame and the second target frame .
  • the number of target frames for backward tracking is 1-20 frames.
  • the video stitching method provided by some embodiments of the present disclosure further includes: if the number of feature point pairs of the second feature point pair set of the first target frame and the second target frame is greater than a first threshold, Then, the second feature point pair set of the first target frame and the second target frame is randomly filtered.
  • the video stitching method provided by some embodiments of the present disclosure further includes: if the number of feature point pairs of the third feature point pair set of the first target frame and the second target frame is greater than a second threshold, Then, the third feature point pair set of the first target frame and the second target frame is randomly filtered.
  • the video stitching method provided by some embodiments of the present disclosure further includes: according to the geometric transformation relationship between the first target frame and the second target frame, the first target frame and the second target The frames are stitched, and the panoramic images of the first target frame and the second target frame are output.
  • the video stitching method provided by some embodiments of the present disclosure further includes: taking each frame of the first video and the second video to be stitched as the first target frame and the second target frame, respectively , Output the panoramic image of each frame to be spliced to form a panoramic video.
  • the first video and the second video are online videos or offline videos.
  • At least one embodiment of the present disclosure further provides a video splicing apparatus, including: a memory; and a processor; wherein the memory stores executable instructions, and the executable instructions can be executed by the processor to implement any of the present disclosure The video stitching method described in the embodiment.
  • Figure 1 is a schematic diagram of a video stitching principle
  • FIG. 3 is a flowchart of a video stitching method provided by some embodiments of the present disclosure.
  • FIG. 4 is a flowchart corresponding to forward tracking in the video stitching method shown in FIG. 3 provided by some embodiments of the present disclosure
  • FIG. 5 is a flowchart of backward tracking in the video stitching method shown in FIG. 3 provided by some embodiments of the present disclosure.
  • FIG. 6 is a schematic diagram of a video splicing device provided by some embodiments of the present disclosure.
  • Figure 1 shows a video stitching principle.
  • the first video V_A includes N frames to be stitched (for example, IM_A(1), ..., IM_A(t), ..., IM_A(N)), and correspondingly, the second video V_B also includes to-be-spliced N frames (for example, IM_B(1), ..., IM_B(t), ..., IM_B(N)).
  • the principle of video splicing is to splice each frame of the first video V_A to be spliced with each frame of the second video V_B to one-to-one correspondence, for example, the initial frame IM_A(1 of the first video V_A to be spliced ) Splicing with the initial frame IM_B(1) of the second video V_B to be spliced to obtain the first splicing frame IM_PV(1), ..., the t-th frame IM_A(t) of the first video V_A to be spliced and the second video V_B
  • the image boundaries of each frame of the first video V_A and the second video V_B to be spliced must partially overlap.
  • overlapping areas of the image boundaries of the different frames of the first video V_A and the second video V_B to be spliced may be the same or different, which is not limited in this disclosure.
  • the object of video stitching may not be limited to the above two channels of video, for example, it may include three or more channels of video, as long as these videos can perform video stitching.
  • the objects of video splicing include three channels of video.
  • the first video and the second video can be spliced to form an intermediate video, and then the intermediate video and the third video can be spliced to form the desired panorama. video.
  • the present disclosure includes but is not limited to this.
  • the first channel video and the third channel video can be separately stitched with the second channel video to form a panoramic video based on the perspective of the second channel video.
  • first video V_A and the second video V_B may be offline videos, that is, before the video stitching, each frame of the first video V_A and the second video V_B to be stitched is obtained; the first video V_A and The second video V_B may also be an online video.
  • the video splicing device receives and processes new frames to be spliced while processing and splicing the already acquired frames to be spliced, that is, real-time or quasi-real-time splicing of the online video.
  • FIG. 2 shows a flowchart of a video stitching method.
  • the video splicing method includes processing each frame of the first video V_A and the second video V_B to be spliced according to steps S10 to S40.
  • Step S10 Perform feature extraction on the first target frame IM_A(t) of the first video and the second target frame IM_B(t) of the second video, respectively, to obtain respective feature point sets ⁇ A(t)_F(m) ⁇ And ⁇ B(t)_F(n) ⁇ .
  • scale-invariant feature transform SIFT
  • accelerated robust feature Speeded UpRobust Features, SURF
  • ORB Oriented FAST and Rotated BRIEF
  • Step S20 The feature point sets ⁇ A(t)_F[m] ⁇ and ⁇ B(t)_F[n] ⁇ extracted from the first target frame IM_A(t) and the second target frame IM_B(t), respectively Perform feature matching to obtain the matching feature point pair set ⁇ (A(t)_MF[s], B(t)_MF[s]) ⁇ .
  • the K-Nearest (KNN) algorithm can be used to perform feature matching on the separately extracted feature point sets ⁇ A(t)_F[m] ⁇ and ⁇ B(t)_F[n] ⁇ , for example
  • KNN K-Nearest
  • For a feature point in the first target frame IM_A(t), find the most similar feature point in the second target frame IM_B(t), thereby forming a matching feature point pair (A(t)_MF[s], B(t)_MF[s]), further, multiple matching feature point pairs form a matching feature point pair set ⁇ (A(t)_MF[s], B(t)_MF[s]) ⁇ , where s is Formal parameters, s 1, 2,...
  • Step S30 Screen the matching feature point pair set ⁇ (A(t)_MF[s],B(t)_MF[s]) ⁇ to obtain the first feature point pair set ⁇ (A(t)_FMF[s] ,B(t)_FMF[s]) ⁇ .
  • step S20 Due to the fact that there may be a false match when performing feature matching in step S20, it is usually necessary to filter the matching feature point pair set ⁇ (A(t)_MF[s], B(t)_MF[s]) ⁇ to The matching feature point pairs obtained by erroneous matching are removed, and the matching feature point pairs obtained by the best matching are left to form the first feature point pair set ⁇ (A(t)_FMF[s],B(t)_FMF[ s]) ⁇ .
  • a random sampling consensus (RANdom SAmple Consensus, RANSAC) algorithm can be used to filter the matched feature point pair set ⁇ (A(t)_MF[s], B(t)_MF[s]) ⁇ obtained in step S20.
  • RANSAC Random SAmple Consensus
  • Step S40 Calculate the first target frame IM_A(t) and the second target frame IM_B(t) according to the first feature point pair set ⁇ (A(t)_FMF[s], B(t)_FMF[s]) ⁇ The geometric transformation relationship between them determines the homography matrix between the first target frame IM_A(t) and the second target frame IM_B(t).
  • the least square method may be employed to calculate the homography matrix between the first target frame IM_A(t) and the second target frame IM_B(t).
  • the homography matrix corresponding to the best match obtained by using the RANSAC algorithm in step S30 may be used as the homography between the first target frame IM_A(t) and the second target frame IM_B(t) matrix.
  • the algorithm used to calculate the geometric transformation relationship between the first target frame IM_A(t) and the second target frame IM_B(t) is not limited.
  • the video stitching method shown in FIG. 2 can be based on the camera calibration results of the first target frame IM_A(t) and the second target frame IM_B(t), that is, the first target frame IM_A(t) and the first target frame of the first video V_A
  • steps S10 to S40 only the feature points of the target frames to be stitched (ie, the first target frame and the second target frame) are used for feature matching, and then Complete the camera calibration.
  • This method of camera calibration based on a single target frame does not consider the temporal continuity of the front and back frames of the video, which may cause the camera calibration of different target frames to change irregularly with time, resulting in the playback of the panoramic video. At the time, the transformation of some adjacent splicing frames may appear obvious jitter.
  • At least one embodiment of the present disclosure provides a video splicing method for splicing a first video and a second video.
  • the video splicing method includes: performing respectively on a first target frame of the first video and a second target frame of the second video Feature extraction, perform feature matching on the features extracted from the first target frame and the second target frame respectively, and filter the matched features to obtain the first feature point pair set of the first target frame and the second target frame;
  • the forward feature point pair set of the first target frame and the second target frame forward traces to the first target frame and the second target frame to obtain the second feature point pair set of the first target frame and the second target frame;
  • the backward feature point pair set of a target frame and the second target frame is traced back to the first target frame and the second target frame to obtain the third feature point pair set of the first target frame and the second target frame;
  • the union of the first feature point pair set, the second feature point pair set, and the third feature point pair set of a target frame and the second target frame calculates the
  • At least one embodiment of the present disclosure also provides a video splicing device corresponding to the above video splicing method.
  • the video stitching method and device provided by the embodiments of the present disclosure combine feature matching and bidirectional feature tracking to perform camera calibration, improve the robustness of feature point pairs used for camera calibration, and improve the continuity and stability of camera calibration.
  • FIG. 3 shows a flowchart of a video stitching method. As shown in FIG. 3, the video stitching method includes steps S100 to S400.
  • S100 Feature extraction is performed on the first target frame IM_A(t) of the first video and the second target frame IM_B(t) of the second video, respectively.
  • the first target frame IM_A(t) and the second target frame IM_B( t) The features extracted in each are feature matched, and the matched features are filtered to obtain the first feature point pair set of the first target frame and the second target frame ⁇ (A(t)_FMF[s],B(t )_FMF[s]) ⁇ .
  • step S100 may include steps S10 to S30 of the existing video splicing method shown in FIG. 2, that is, for the detailed description and technical effects of step S100, reference may be made to the above description for step S10 to step S30 , This disclosure will not repeat them here.
  • Step S200 Forward the pair of forward feature points of the first target frame IM_A(t) and the second target frame IM_B(t) ⁇ (A(t)_FTF[s], B(t)_FTF[s]) ⁇ Tracing to the first target frame IM_A(t) and the second target frame IM_B(t), the second feature point pair set of the first target frame IM_A(t) and the second target frame IM_B(t) ⁇ (A( t)_FTMF[s],B(t)_FTMF[s]) ⁇ .
  • the Kanade-Lucas-Tomasi tracking algorithm can be used for forward tracking, which is not limited in this disclosure.
  • the forward feature point pair set of the target frames of the first video and the second video to be spliced that is, the first target frame and the second target frame
  • ⁇ (A(t)_FTF[s],B(t )_FTF[s]) ⁇ is obtained based on the frame before the target frame, therefore, the video stitching method considers the continuity in time between the frame before the target frame of the video and the target frame.
  • Step S300 After the backward feature point pair set of the first target frame IM_A(t) and the second target frame IM_B(t) ⁇ (A(t)_BTF[s], B(t)_BTF[s]) ⁇ Tracing to the first target frame IM_A(t) and the second target frame IM_B(t), the third feature point pair set of the first target frame IM_A(t) and the second target frame IM_B(t) is obtained ⁇ (A( t)_BTMF[s],B(t)_BTMF[s]) ⁇ .
  • Kanade-Lucas-Tomasi tracking algorithm or the like may be used for backward tracking, which is not limited in this disclosure.
  • the backward feature point pair set ⁇ (A(t)_FTF[s],B(t)_FTF[s]) ⁇ of the target frame to be spliced of the first video and the second video is based on the target frame The subsequent frames are obtained. Therefore, this video splicing method takes into account the temporal continuity of the target frame of the video and the frames after the target frame.
  • Step S400 According to the first feature point pair set ⁇ (A(t)_FMF[s], B(t)_FMF[s]) ⁇ of the first target frame IM_A(t) and the second target frame IM_B(t), The second feature point pair set ⁇ (A(t)_FTMF[s],B(t)_FTMF[s]) ⁇ and the third feature point pair set ⁇ (A(t)_BTMF[s],B(t)_BTMF [s]) ⁇ The union of the first target frame IM_A(t) and the second target frame IM_B(t) is calculated.
  • the camera calibration can be performed according to the union of the three.
  • the geometric transformation relationship between the first target frame IM_A(t) and the second target frame IM_B(t) may be calculated using the least square method or the RANSAC algorithm, etc. to determine the corresponding homography matrix. It should be noted that in this embodiment, the algorithm used to calculate the geometric transformation relationship between the first target frame IM_A(t) and the second target frame IM_B(t) is not limited.
  • the video stitching method shown in FIG. 3 combines feature matching (step S100) and bidirectional feature tracking (forward tracking in step S200 and backward tracking in step S300) for camera calibration, which improves camera calibration.
  • the robustness of the feature point pairs improves the continuity and stability of camera calibration.
  • FIG. 4 shows a flowchart corresponding to forward tracking in the video stitching method shown in FIG. 3.
  • the process of forward tracking corresponds to the above step S200.
  • the initial frames IM_A(1) and IM_B(1) of the first video and the second video to be spliced are the starting frames of forward tracking.
  • Each frame is tracked forward.
  • the key of forward tracking is to determine the forward feature point pair set of each frame to be spliced of the first video and the second video.
  • the video stitching method provided by an embodiment of the present disclosure may further include: pairing the first feature points of the first target frame IM_A(t) and the second target frame IM_B(t) ⁇ (A(t )_FMF[s],B(t)_FMF[s]) ⁇ and the second feature point pair set ⁇ (A(t)_FTMF[s],B(t)_FTMF[s]) ⁇ as the first The frame IM_A(t+1) after the first target frame IM_A(t) of the video (not shown in FIG. 4) and the frame IM_B(t) after the second target frame IM_B(t) of the second video +1) (not shown in FIG. 4) forward feature point pair set ⁇ (A(t+1)_FTF[s], B(t+1)_FTF[s]) ⁇ .
  • first target frame IM_A(t) and the second target frame IM_B(t) are the initial frames to be stitched (ie, IM_A(1) and IM_B(1))
  • first target frame IM_A(t) and the second target frame IM_B(t) are the initial frames to be stitched (ie, IM_A(1) and IM_B(1))
  • the first target frame IM_A(t) and the second target frame IM_B(t) does not have a corresponding forward feature point pair set ⁇ (A(1)_FTF[s],B(1)_FTF[s]) ⁇ , forward tracking cannot be performed, so the first target The frame IM_A(1) and the second target frame IM_B(1) have no corresponding second feature point pair set ⁇ (A(1)_FTMF[s],B(1)_FTMF[s]) ⁇ , so for the first target The frame IM_A(1) and the second target frame IM_B(1) need to be processed separately.
  • the video stitching method provided by the embodiments of the present disclosure may further include: if the first target frame and the second target frame are the initial frames IM_A(1) and IM_B(1) to be spliced, then no Forward tracking, based on the first feature point pair set of the first target frame IM_A(1) and the second target frame IM_B(1) ⁇ (A(1)_FMF[s],B(1)_FMF[s]) ⁇ The union of the third feature point pair set ⁇ (A(1)_BTMF[s],B(1)_BTMF[s]) ⁇ calculates the first target frame IM_A(1) and the second target frame IM_B(1) The geometric transformation relationship between; and as shown in Figure 4, the first target frame IM_A (1) and the second target frame IM_B (1) the first feature point pair set ⁇ (A(1)_FMF[s], B(1)_FMF[s]) ⁇ as the first target frame IM_A(1) after the first video IM_A(2) (not shown in FIG. 4) and the second target frame
  • FIG. 5 shows a flowchart corresponding to the backward tracking in the video stitching method shown in FIG. 3.
  • the process of backward tracking corresponds to the above step S300.
  • the target frames IM_A(t) and IM_B(t) of the first video and the second video to be stitched are backward tracked, the first video and the second video are to be stitched
  • the target frames IM_A(t) and IM_B(t) are the kth frame after IM_A(t+k) and IM_B(t+k) as the starting frame for backward tracking.
  • the starting frame for backward tracking is IM_A( Each frame before t+k) and IM_B(t+k) performs backward tracking until the target frames IM_A(t) and IM_B(t) to be stitched are tracked.
  • k is the number of backward tracking target frames, which can be an integer greater than 0.
  • the key to backward tracking is to determine the starting frame of backward tracking and the set of backward feature points for each frame of the first video and the second video to be spliced.
  • the video stitching method provided by the embodiments of the present disclosure may further include: if the number of video frames to be stitched after the first target frame IM_A(t) and the second target frame IM_B(t) is less than the backward direction When tracking the target frame number k, the final frames IM_A(N) and IM_B(N) of the first video and the second video to be spliced are used as the starting frame for backward tracking.
  • the case where the number of video frames to be spliced after the target frame to be spliced is less than the number k of the backward tracking target frame generally occurs when k is greater than 1.
  • k is greater than 1.
  • the number of video frames is k-1,...,1, which is less than the number of target frames for backward tracking k, that is, k backward tracking cannot be performed, so the final frames IM_A(N) and IM_B(N) to be stitched are used as these waiting
  • the video stitching method provided by the embodiments of the present disclosure may further include: if the first target frame and the second target frame are some of the first video and the second video to be spliced The starting frame of the backward tracking of a frame, the first feature point pair of the first target frame and the second target frame are used as the previous frame of the first target frame of the first video and the second target frame of the second video The set of backward feature points of the previous frame.
  • the first feature point pair of the first target frame IM_A(t+k) and the second target frame IM_B(t+k) is set ⁇ (A(t +k)_FMF[s], B(t+k)_FMF[s]) ⁇ as the first frame IM_A(t+k-1) of the first target frame IM_A(t+k) of the first video ( Figure 5 And the backward feature point pair set of the previous frame IM_B(t+k-1) (not shown in FIG. 5) of the second target frame IM_B(t+k) of the second video (not shown in FIG. 5) ((A (t+k-1)_BTF[s],B(t+k-1)_BTF[s])
  • the video stitching method provided by the embodiments of the present disclosure may further include: if the first target frame and the second target frame are some of the first video and the second video to be spliced
  • the intermediate frame of the backward tracking of one frame, the union of the first feature point pair set and the third feature point pair set of the first target frame and the second target frame is taken as the previous one of the first target frame of the first video
  • the set of backward feature points of the frame and the previous frame of the second target frame of the second video may further include: if the first target frame and the second target frame are some of the first video and the second video to be spliced
  • the intermediate frame of the backward tracking of one frame, the union of the first feature point pair set and the third feature point pair set of the first target frame and the second target frame is taken as the previous one of the first target frame of the first video
  • the set of backward feature points of the frame and the previous frame of the second target frame of the second video may further include: if the first target frame and the second target frame are
  • IM_A(t) and IM_B(t) are taken as the first target frame and the second target frame, which are some of the first video and the second video to be spliced
  • An intermediate frame for backward tracking of a frame IM_A(t-1) and IM_B(t-1) not shown in FIG.
  • k may also be an integer with a value greater than 2.
  • the video stitching method provided by the embodiment of the present disclosure may not include the intermediate frame for backward tracking
  • the video stitching method provided by the embodiment of the present disclosure may also include processing of the intermediate frame for backward tracking, which does not affect the realization of video stitching.
  • the video stitching method provided by the embodiments of the present disclosure may further include: if the first target frame and the second target frame are the final frames IM_A(N) and IM_B(N) to be stitched, then no Backward tracking, according to the first feature point pair set of the first target frame IM_A(N) and the second target frame IM_B(N) ⁇ (A(N)_FMF[s],B(N)_FMF[s]) ⁇ The union of the second feature point pair set ⁇ (A(N)_FTMF[s], B(N)_FTMF[s]) ⁇ calculates the first target frame IM_A(N) and the second target frame IM_B( N) The geometric transformation relationship between.
  • the value of the backward tracking target frame number k is not limited.
  • the backward tracking target frame number k may be, for example, 1-20, such as 1-10, such as 1-5, such as 1, so that the video stitching method can be used to stitch offline videos.
  • the backward tracking target frame number k may be N-1.
  • the backward tracking process and the forward tracking process may be regarded as Do the same (the difference is that forward tracking is in order and backward tracking is in reverse order).
  • the backward tracking target frame number k may take an integer greater than 20, which is not limited in this disclosure. It should be noted that for online video, the larger the value of the backward tracking target frame number k, the worse the quasi-real-time performance of video stitching, but the better the temporal continuity between the target frame and the frames after the target frame.
  • the video stitching method provided by the embodiments of the present disclosure may further include: if the second feature point pair set of the first target frame IM_A(t) and the second target frame IM_B(t) ⁇ (A( t)_FTMF[s],B(t)_FTMF[s]) ⁇ The number of feature point pairs is greater than the first threshold, then the second target frame IM_A(t) and the second target frame IM_B(t) The feature points are randomly filtered on the set ⁇ (A(t)_FTMF[s], B(t)_FTMF[s]) ⁇ .
  • the random filtering of the second feature point pair set of the target frame is to randomly discard the feature point pair of the second feature point pair set, so that the The number of remaining feature point pairs in the second feature point pair set does not exceed the first threshold, so that the corresponding weight of the forward feature point pair set of the second feature point pair set in the next frame of the target frame is relatively stable, and then Maintain the stability of the forward tracking of the next frame of the target frame.
  • the video stitching method provided by the embodiments of the present disclosure may further include: if the third feature point pair set of the first target frame IM_A(t) and the second target frame IM_B(t) ⁇ (A( t)_BTMF[s],B(t)_BTMF[s]) ⁇ The number of feature point pairs is greater than the second threshold, then the third target frame IM_A(t) and the third target frame IM_B(t) of the third The feature points are randomly filtered on the set ⁇ (A(t)_BTMF[s], B(t)_BTMF[s]) ⁇ .
  • the random filtering of the third feature point pair set of the target frame is to randomly discard the feature point pair of the third feature point pair set, so that the The number of remaining feature point pairs in the third feature point pair set does not exceed the second threshold, so that the corresponding weight of the backward feature point pair set of the third feature point pair set in the previous frame of the target frame can be kept relatively stable, and thus can Maintain the stability of backward tracking of the previous frame of the target frame.
  • the number of feature point pairs in the second feature point pair set and the third feature point pair set in the target frame is limited by random filtering, and the second feature point pair set and the third feature point pair set can also be maintained
  • the weights corresponding to the union of the first feature point pair set, the second feature point pair set, and the third feature point pair set of the target frame are relatively stable, so that the first target frame and the second target frame are calculated according to the union
  • the homography matrix obtained by the geometric transformation relationship between them is equivalent to being time-weighted. Therefore, the homography matrix corresponding to the adjacent frames to be spliced has a smoother temporal transformation, achieving stable camera calibration.
  • the video stitching method provided by the embodiments of the present disclosure may further include: stitching the first target frame and the second target frame according to the geometric transformation relationship between the first target frame and the second target frame And output the panoramic images of the first target frame and the second target frame.
  • a stitching seam may appear on the obtained panoramic image due to the difference in exposure and other factors in the overlapping part of the boundary of the two frame images, therefore, The pixel values of the overlapping parts of the two frames of images can also be weighted and fused to eliminate the stitching seam.
  • the video stitching method provided by an embodiment of the present disclosure may further include: taking each frame of the first video and the second video to be stitched as the first target frame and the second target frame, respectively, and outputting the The panoramic images of each frame stitched together form a panoramic video.
  • the process of forming the panoramic video reference may be made to FIG. 1 and the description about FIG. 1 in the present disclosure, which will not be repeated here.
  • the first video and the second video may be online videos or offline videos. It should be noted that, when both the first video and the second video are online videos, the number k of the backward tracking target frame in the video stitching method provided by the embodiment of the present disclosure may be set to a small value, such as 1- 20, such as 1-10, such as 1-5, such as 1, to achieve quasi-real-time stitching.
  • the flow of the above video stitching method may include more or fewer steps, and these steps may be executed sequentially or in parallel.
  • the execution order of each step is subject to the realization of video splicing, and is not limited by the sequence number mark of each step.
  • At least one embodiment of the present disclosure also provides a video splicing device.
  • the video splicing device 1 includes a memory 10 and a processor 20.
  • the memory 10 stores executable instructions that can be executed by the processor 20 to implement the video stitching method provided by any embodiment of the present disclosure.
  • the memory 10 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory.
  • the volatile memory may include, for example, random access memory (RAM) and/or cache memory.
  • the non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, and the like.
  • the above-mentioned executable instructions may be stored on the computer-readable storage medium.
  • the processor 20 may be implemented in at least one hardware form of a digital signal processor (DSP), a field programmable gate array (FPGA), and a programmable logic array (PLA), which may include one or more central processing units ( CPU) or other forms of processing units with data processing capabilities and/or instruction execution capabilities, and can control the necessary structures, units or modules (not shown in FIG. 6) in the video splicing apparatus 1 to perform desired functions.
  • DSP digital signal processor
  • FPGA field programmable gate array
  • PDA programmable logic array
  • CPU central processing units
  • the processor 20 may execute the above executable instructions to perform video stitching according to the video stitching method provided by any embodiment of the present disclosure.
  • the embodiments of the present disclosure do not provide the entire structure of the video splicing device.
  • the video splicing device may further include other necessary structures, units, or modules.
  • the video splicing device 1 may further include a video receiving unit, a panoramic video output unit, and the like, which are not limited in the embodiments of the present disclosure.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Studio Devices (AREA)
  • Image Analysis (AREA)

Abstract

一种视频拼接方法及装置。该视频拼接方法用于拼接第一视频和第二视频，包括：对第一视频的第一目标帧和第二视频的第二目标帧进行特征提取、特征匹配和筛选，获得第一目标帧和第二目标帧的第一特征点对集；对第一目标帧和第二目标帧进行前向追踪，获得第一目标帧和第二目标帧的第二特征点对集；对第一目标帧和第二目标帧进行后向追踪，获得第一目标帧和第二目标帧的第三特征点对集；以及根据第一目标帧和第二目标帧的第一特征点对集、第二特征点对集以及第三特征点对集的并集，计算第一目标帧和第二目标帧之间的几何变换关系。

Description

视频拼接方法及装置
本申请要求于2018年12月28日递交、题为“视频拼接方法及装置”的中国专利申请第201811625266.X号的优先权，在此全文引用上述中国专利申请公开的内容以作为本申请的一部分。
技术领域
本公开的实施例涉及一种视频拼接方法及装置。
背景技术
视频拼接是计算机视觉领域中的一项重要的技术。视频拼接的过程是将多个摄像机采集到的具有重叠区域的视频图像拼接合成到同一视面上，形成一个分辨率更高、视角更广的全景视频。与具有广角镜头的全景摄像机相比，通过视频拼接技术获得全景视频的成本更低，实用性更高。在实际生活中，视频拼接技术已经运用到了安防监控、空中侦察、展示展览、视觉娱乐等各个领域。
发明内容
本公开至少一实施例提供一种视频拼接方法，用于拼接第一视频和第二视频，包括：对所述第一视频的第一目标帧和所述第二视频的第二目标帧分别进行特征提取，对从所述第一目标帧和所述第二目标帧中分别提取的特征进行特征匹配，并对匹配的特征进行筛选，获得所述第一目标帧和所述第二目标帧的第一特征点对集；将所述第一目标帧和所述第二目标帧的前向特征点对集前向追踪到所述第一目标帧和所述第二目标帧，获得所述第一目标帧和所述第二目标帧的第二特征点对集；将所述第一目标帧和所述第二目标帧的后向特征点对集后向追踪到所述第一目标帧和所述第二目标帧，获得所述第一目标帧和所述第二目标帧的第三特征点对集；以及根据所述第一目标帧和所述第二目标帧的所述第一特征点对集、所述第二特征点对集以及所述第三特征点对集的并集，计算所述第一目标帧和所述第二目标帧之间的几何变换关系。
例如，本公开一些实施例提供的视频拼接方法，还包括：将所述第一目标帧和所述第二目标帧的所述第一特征点对集与所述第二特征点对集的并集作为所述第一视频的第一目标帧的后一帧和所述第二视频的第二目标帧的后一帧的前向特征点对集。
例如，本公开一些实施例提供的视频拼接方法，还包括：若所述第一目标帧和所述第二目标帧为待拼接的初始帧，则不进行前向追踪，根据所述第一目标帧和所述第二目标帧的所述第一特征点对集与所述第三特征点对集的并集，计算所述第一目标帧和所述第二目标帧之间的几何变换关系，将所述第一目标帧和所述第二目标帧的第一特征点对集作为所述第一视频的第一目标帧的后一帧和所述第二视频的第二目标帧的后一帧的前向特征点对集。
例如，本公开一些实施例提供的视频拼接方法，还包括：若所述第一目标帧和所述第二目标帧之后的待拼接的视频帧数小于后向追踪目标帧数，则将所述第一视频和所述第二视频的待拼接的最终帧作为后向追踪的起点帧。
例如，本公开一些实施例提供的视频拼接方法，还包括：若所述第一目标帧和所述第二目标帧为所述第一视频和所述第二视频的待拼接的某一帧的后向追踪的起点帧，则将所述第一目标帧和所述第二目标帧的第一特征点对集作为所述第一视频的第一目标帧的前一帧和所述第二视频的第二目标帧的前一帧的后向特征点对集；若所述第一目标帧和所述第二目标帧为所述第一视频和所述第二视频的待拼接的某一帧的后向追踪的中间帧，则将所述第一目标帧和所述第二目标帧的所述第一特征点对集与所述第三特征点对集的并集作为所述第一视频的第一目标帧的前一帧和所述第二视频的第二目标帧的前一帧的后向特征点对集。
例如，本公开一些实施例提供的视频拼接方法，还包括：若所述第一目标帧和所述第二目标帧为待拼接的最终帧，则不进行后向追踪，根据所述第一目标帧和所述第二目标帧的所述第一特征点对集与所述第二特征点对集的并集，计算所述第一目标帧和所述第二目标帧之间的几何变换关系。
例如，在本公开一些实施例提供的视频拼接方法中，所述后向追踪目标帧数为1-20帧。
例如，本公开一些实施例提供的视频拼接方法，还包括：若所述第一 目标帧和所述第二目标帧的所述第二特征点对集的特征点对的数量大于第一阈值，则对所述第一目标帧和所述第二目标帧的所述第二特征点对集进行随机滤波。
例如，本公开一些实施例提供的视频拼接方法，还包括：若所述第一目标帧和所述第二目标帧的所述第三特征点对集的特征点对的数量大于第二阈值，则对所述第一目标帧和所述第二目标帧的所述第三特征点对集进行随机滤波。
例如，本公开一些实施例提供的视频拼接方法，还包括：根据所述第一目标帧和所述第二目标帧之间的几何变换关系，对所述第一目标帧和所述第二目标帧进行拼接，并输出所述第一目标帧和所述第二目标帧的全景图像。
例如，本公开一些实施例提供的视频拼接方法，还包括：将所述第一视频和所述第二视频的待拼接的每一帧分别作为所述第一目标帧和所述第二目标帧，输出所述待拼接的每一帧的全景图像，形成全景视频。
例如，在本公开一些实施例提供的视频拼接方法中，所述第一视频和所述第二视频为在线视频或者离线视频。
本公开至少一实施例还提供一种视频拼接装置，包括：存储器；和处理器；其中，所述存储器存储有可执行指令，所述可执行指令可由所述处理器执行以实现本公开任一实施例所述的视频拼接方法。
附图说明
为了更清楚地说明本公开实施例的技术方案，下面将对实施例的附图作简单地介绍，显而易见地，下面描述中的附图仅仅涉及本公开的一些实施例，而非对本公开的限制。
图1为一种视频拼接原理的示意图；
图2为一种视频拼接方法的流程图；
图3为本公开一些实施例提供的一种视频拼接方法的流程图；
图4为本公开一些实施例提供的一种对应于图3所示的视频拼接方法中的前向追踪的流程图；
图5为本公开一些实施例提供的一种对应于图3所示的视频拼接方法中的后向追踪的流程图；以及
图6为本公开一些实施例提供的一种视频拼接装置的示意图。
具体实施方式
为使本公开实施例的目的、技术方案和优点更加清楚，下面将结合本公开实施例的附图，对本公开实施例的技术方案进行清楚、完整地描述。显然，所描述的实施例是本公开的一部分实施例，而不是全部的实施例。基于所描述的本公开的实施例，本领域普通技术人员在无需创造性劳动的前提下所获得的所有其他实施例，都属于本公开保护的范围。
除非另外定义，本公开使用的技术术语或者科学术语应当为本公开所属领域内具有一般技能的人士所理解的通常意义。本公开中使用的“第一”、“第二”以及类似的词语并不表示任何顺序、数量或者重要性，而只是用来区分不同的组成部分。同样，“一个”、“一”或者“该”等类似词语也不表示数量限制，而是表示存在至少一个。“包括”或者“包含”等类似的词语意指出现该词前面的元件或者物件涵盖出现在该词后面列举的元件或者物件及其等同，而不排除其他元件或者物件。“连接”或者“相连”等类似的词语并非限定于物理的或者机械的连接，而是可以包括电性的连接，不管是直接的还是间接的。“上”、“下”、“左”、“右”等仅用于表示相对位置关系，当被描述对象的绝对位置改变后，则该相对位置关系也可能相应地改变。
图1示出了一种视频拼接原理。如图1所示，第一视频V_A包括待拼接的N帧(例如，IM_A(1)，…，IM_A(t)，…，IM_A(N))，对应地，第二视频V_B也包括待拼接的N帧(例如，IM_B(1)，…，IM_B(t)，…，IM_B(N))。视频拼接的原理是将第一视频V_A的待拼接的每一帧与第二视频V_B的待拼接的每一帧一一对应拼接，例如，将第一视频V_A的待拼接的初始帧IM_A(1)与第二视频V_B的待拼接的初始帧IM_B(1)拼接得到第一拼接帧IM_PV(1)，…，将第一视频V_A的待拼接的第t帧IM_A(t)与第二视频V_B的待拼接的第t帧IM_B(t)拼接得到第t拼接帧IM_PV(t)，…，将第一视频V_A的待拼接的最终帧IM_A(N)与第二视频V_B的待拼接的最终帧IM_B(N)拼接得到最终拼接帧IM_PV(t)，该多个拼接帧(IM_PV(1)，…，IM_PV(t)，…，IM_PV(N))依序形成一个全景视频PV。
需要说明的是，为了能让视频拼接装置自动进行视频拼接，要求第一 视频V_A和第二视频V_B待拼接的每一帧的图像边界具有部分重叠，例如，如图1所示，IM_A(t)和IM_B(t)的图像边界具有重叠区域OA，其中t＝1，…，N。还需要说明的是，第一视频V_A和第二视频V_B待拼接的不同帧的图像边界的重叠区域可以相同，也可以不同，本公开对此不作限制。
需要说明的是，视频拼接的对象可以不限于上述两路视频，例如可以包括三路或者更多路视频，只要这些视频能够进行视频拼接即可。例如，在一些示例中，视频拼接的对象包括三路视频，可以先将第一路视频与第二路视频进行拼接形成中间视频，再将该中间视频与第三路视频进行拼接形成需要的全景视频。需要说明的是，本公开包括但不限于此，例如，还可以将第一路视频和第三路视频分别与第二路视频拼接，最后形成基于第二路视频的视角的全景视频。
需要说明的是，第一视频V_A和第二视频V_B可以是离线视频，即在进行视频拼接前，就获得了第一视频V_A和第二视频V_B待拼接的每一帧；第一视频V_A和第二视频V_B也可以是在线视频，视频拼接装置一边接收新的待拼接的帧，一边对已经获得的待拼接的帧进行处理和拼接，即对在线视频进行实时或准实时拼接。
图2示出了一种视频拼接方法的流程图。该视频拼接方法包括按步骤S10至步骤S40对第一视频V_A和第二视频V_B待拼接的每一帧进行处理。下面以第一视频V_A的待拼接的第t帧IM_A(t)作为第一目标帧、第二视频V_B的待拼接的第t帧IM_B(t)作为第二目标帧对步骤S10至步骤S40进行说明，其中，t＝1，…，N。
步骤S10：对第一视频的第一目标帧IM_A(t)和第二视频的第二目标帧IM_B(t)分别进行特征提取，获得各自的特征点集{A(t)_F(m)}和{B(t)_F(n)}。
例如，可以根据视频图像的特点选择性地采用例如尺度不变特征变换(Scale-Invariant Feature Transform，SIFT)算法或加速稳健特征(Speeded Up Robust Features，SURF)算法或ORB(Oriented FAST and Rotated BRIEF)算法等进行特征提取，本公开对此不作限制。
需要说明的是，m、n均为形式参数，其中m＝1，2，…，n＝1，2，…。
步骤S20：对从第一目标帧IM_A(t)和第二目标帧IM_B(t)中分别提取 的特征点集{A(t)_F[m]}和{B(t)_F[n]}进行特征匹配，获得匹配特征点对集{(A(t)_MF[s],B(t)_MF[s])}。
例如，可以采用K最近邻(K-Nearest Neighbor，KNN)算法等对分别提取的特征点集{A(t)_F[m]}和{B(t)_F[n]}进行特征匹配，例如针对第一目标帧IM_A(t)中的某一特征点，在第二目标帧IM_B(t)中找到与其最相似的特征点，从而形成匹配特征点对(A(t)_MF[s],B(t)_MF[s])，进一步地，多个匹配特征点对形成匹配特征点对集{(A(t)_MF[s],B(t)_MF[s])}，其中s为形式参数，s＝1，2，…。例如，KNN算法中可以选择K＝2，本公开包括但不限于此。需要说明的是，在本公开中，对进行特征匹配采用的算法不作限制。
步骤S30：对匹配特征点对集{(A(t)_MF[s],B(t)_MF[s])}进行筛选，获得第一特征点对集{(A(t)_FMF[s],B(t)_FMF[s])}。
由于在步骤S20中进行特征匹配时可能存在错误匹配的情形，因此通常需要对匹配特征点对集{(A(t)_MF[s],B(t)_MF[s])}进行筛选，以剔除其中的错误匹配获得的匹配特征点对，留下其中的最佳匹配获得的匹配特征点对以形成第一特征点对集{(A(t)_FMF[s],B(t)_FMF[s])}。例如，可以采用随机抽样一致(RANdom SAmple Consensus，RANSAC)算法等对步骤S20获得的匹配特征点对集{(A(t)_MF[s],B(t)_MF[s])}进行筛选。需要说明的是，在本公开中，对进行筛选采用的算法不作限制。
步骤S40：根据第一特征点对集{(A(t)_FMF[s],B(t)_FMF[s])}计算第一目标帧IM_A(t)和第二目标帧IM_B(t)之间的几何变换关系，确定第一目标帧IM_A(t)和第二目标帧IM_B(t)之间的单应矩阵。
例如，在一些示例中，可以采用最小二乘法计算第一目标帧IM_A(t)和第二目标帧IM_B(t)之间的单应矩阵。例如，在一些示例中，可以将步骤S30中采用RANSAC算法进行筛选得到的最佳匹配对应的单应矩阵作为第一目标帧IM_A(t)和第二目标帧IM_B(t)之间的单应矩阵。需要说明的是，在本公开中，对计算第一目标帧IM_A(t)和第二目标帧IM_B(t)之间的几何变换关系采用的算法不作限制。
需要说明的是，上述计算第一目标帧IM_A(t)和第二目标帧IM_B(t)之间的几何变换关系，确定第一目标帧IM_A(t)和第二目标帧IM_B(t)之间的单应矩阵，即为对第一目标帧IM_A(t)和第二目标帧IM_B(t)进行相机标 定。
图2所示的视频拼接方法可以根据对第一目标帧IM_A(t)和第二目标帧IM_B(t)进行相机标定的结果，即第一视频V_A的第一目标帧IM_A(t)和第二视频V_B的第二目标帧IM_B(t)之间的几何变换关系(单应矩阵)，将第一目标帧IM_A(t)和第二目标帧IM_B(t)拼接得到对应的拼接帧IM_PV(t)；进一步地，令t＝1，…，N，可以得到多个拼接帧IM_PV(1)，…，IM_PV(N)，从而得到全景视频PV。
需要说明的是，图2所示的视频拼接方法在步骤S10至步骤S40中，仅使用了待拼接的目标帧(即第一目标帧和第二目标帧)的特征点来进行特征匹配，进而完成相机标定。这种基于单一的目标帧进行相机标定的方法，未考虑视频的前后帧在时间上的连续性，可能会导致不同目标帧的相机标定随时间变化毫无规律，最终导致得到的全景视频在播放时，某些相邻拼接帧的变换可能出现明显的抖动现象。
本公开至少一实施例提供一种视频拼接方法，用于拼接第一视频和第二视频，该视频拼接方法包括：对第一视频的第一目标帧和第二视频的第二目标帧分别进行特征提取，对从第一目标帧和第二目标帧中分别提取的特征进行特征匹配，并对匹配的特征进行筛选，获得第一目标帧和第二目标帧的第一特征点对集；将第一目标帧和第二目标帧的前向特征点对集前向追踪到第一目标帧和第二目标帧，获得第一目标帧和第二目标帧的第二特征点对集；将第一目标帧和第二目标帧的后向特征点对集后向追踪到第一目标帧和第二目标帧，获得第一目标帧和第二目标帧的第三特征点对集；以及根据第一目标帧和第二目标帧的第一特征点对集、第二特征点对集以及第三特征点对集的并集，计算第一目标帧和第二目标帧之间的几何变换关系。
本公开至少一实施例还提供一种对应于上述视频拼接方法的视频拼接装置。
本公开的实施例提供的视频拼接方法及装置结合特征匹配和双向特征追踪进行相机标定，提高了用于相机标定的特征点对的鲁棒性，改进了相机标定的连续性和稳定性。
下面结合附图对本公开的实施例进行详细说明。
图3示出了一种视频拼接方法的流程图。如图3所示，该视频拼接方 法包括步骤S100至步骤S400。
S100：对第一视频的第一目标帧IM_A(t)和第二视频的第二目标帧IM_B(t)分别进行特征提取，对从第一目标帧IM_A(t)和第二目标帧IM_B(t)中分别提取的特征进行特征匹配，并对匹配的特征进行筛选，获得第一目标帧和第二目标帧的第一特征点对集{(A(t)_FMF[s],B(t)_FMF[s])}。
例如，在一些示例中，步骤S100可以包括图2所示的现有的视频拼接方法的步骤S10至步骤S30，即关于步骤S100的详细描述和技术效果可以参考上述对于步骤S10至步骤S30的描述，本公开对此不再赘述。
步骤S200：将第一目标帧IM_A(t)和第二目标帧IM_B(t)的前向特征点对集{(A(t)_FTF[s],B(t)_FTF[s])}前向追踪到第一目标帧IM_A(t)和第二目标帧IM_B(t)，获得第一目标帧IM_A(t)和第二目标帧IM_B(t)的第二特征点对集{(A(t)_FTMF[s],B(t)_FTMF[s])}。
例如，在一些示例中，可以采用Kanade-Lucas-Tomasi追踪算法等进行前向追踪，本公开对此不作限制。需要说明的是，第一视频和第二视频待拼接的目标帧(即第一目标帧和第二目标帧)的前向特征点对集{(A(t)_FTF[s],B(t)_FTF[s])}是基于该目标帧之前的帧获得的，因此，该视频拼接方法考虑了视频的目标帧之前的帧与目标帧在时间上的连续性。
步骤S300：将第一目标帧IM_A(t)和第二目标帧IM_B(t)的后向特征点对集{(A(t)_BTF[s],B(t)_BTF[s])}后向追踪到第一目标帧IM_A(t)和第二目标帧IM_B(t)，获得第一目标帧IM_A(t)和第二目标帧IM_B(t)的第三特征点对集{(A(t)_BTMF[s],B(t)_BTMF[s])}。
例如，在一些示例中，可以采用Kanade-Lucas-Tomasi追踪算法等进行后向追踪，本公开对此不作限制。需要说明的是，第一视频和第二视频待拼接的目标帧的后向特征点对集{(A(t)_FTF[s],B(t)_FTF[s])}是基于该目标帧之后的帧获得的，因此，该视频拼接方法考虑了视频的目标帧与目标帧之后的帧在时间上的连续性。
步骤S400：根据第一目标帧IM_A(t)和第二目标帧IM_B(t)的第一特征点对集{(A(t)_FMF[s],B(t)_FMF[s])}、第二特征点对集{(A(t)_FTMF[s],B(t)_FTMF[s])}以及第三特征点对集{(A(t)_BTMF[s],B(t)_BTMF[s])}的并集，计算第一目标帧IM_A(t)和第二目标帧IM_B(t)之间的几何变换关系。
由于第一特征点对集、第二特征点对集以及第三特征点对集中的特征 点对可能存在重复情形，因此，可以根据三者的并集进行相机标定。例如，在一些示例中，可以采用最小二乘法或RANSAC算法等计算第一目标帧IM_A(t)和第二目标帧IM_B(t)之间的几何变换关系，确定相应的单应矩阵。需要说明的是，在本实施例中，对计算第一目标帧IM_A(t)和第二目标帧IM_B(t)之间的几何变换关系采用的算法不作限制。
需要说明的是，图3所示的视频拼接方法结合了特征匹配(步骤S100)和双向特征追踪(步骤S200的前向追踪和步骤S300的后向追踪)进行相机标定，提高了用于相机标定的特征点对的鲁棒性，改进了相机标定的连续性和稳定性。
图4示出了一种对应于图3所示的视频拼接方法中的前向追踪的流程图。前向追踪的过程对应于上述步骤S200，以第一视频和第二视频待拼接的初始帧IM_A(1)和IM_B(1)为前向追踪的起点帧，依据顺序对其后的待拼接的每一帧进行前向追踪。前向追踪的关键是确定第一视频和第二视频待拼接的每一帧的前向特征点对集。
如图4所示，本公开的实施例提供的视频拼接方法还可以包括：将第一目标帧IM_A(t)和第二目标帧IM_B(t)的第一特征点对集{(A(t)_FMF[s],B(t)_FMF[s])}与第二特征点对集{(A(t)_FTMF[s],B(t)_FTMF[s])}的并集作为第一视频的第一目标帧IM_A(t)的后一帧IM_A(t+1)(图4中未示出)和所述第二视频的第二目标帧IM_B(t)的后一帧IM_B(t+1)(图4中未示出)的前向特征点对集{(A(t+1)_FTF[s],B(t+1)_FTF[s])}。
特别地，若第一目标帧IM_A(t)和第二目标帧IM_B(t)为待拼接的初始帧(即IM_A(1)和IM_B(1))，由于第一目标帧IM_A(t)和第二目标帧IM_B(t)没有对应的前向特征点对集{(A(1)_FTF[s],B(1)_FTF[s])}，则无法进行前向追踪，从而第一目标帧IM_A(1)和第二目标帧IM_B(1)没有对应的第二特征点对集{(A(1)_FTMF[s],B(1)_FTMF[s])}，因而对于第一目标帧IM_A(1)和第二目标帧IM_B(1)需要另作处理。
例如，在一些示例中，本公开的实施例提供的视频拼接方法还可以包括：若第一目标帧和第二目标帧为待拼接的初始帧IM_A(1)和IM_B(1)，则不进行前向追踪，根据第一目标帧IM_A(1)和第二目标帧IM_B(1)的第一特征点对集{(A(1)_FMF[s],B(1)_FMF[s])}与第三特征点对集{(A(1)_BTMF[s],B(1)_BTMF[s])}的并集，计算第一目标帧IM_A(1)和第二 目标帧IM_B(1)之间的几何变换关系；以及如图4所示，将第一目标帧IM_A(1)和第二目标帧IM_B(1)的第一特征点对集{(A(1)_FMF[s],B(1)_FMF[s])}作为第一视频的第一目标帧IM_A(1)的后一帧IM_A(2)(图4中未示出)和第二视频的第二目标帧IM_B(1)的后一帧IM_B(2)(图4中未示出)的前向特征点对集{(A(2)_FTF[s],B(2)_FTF[s])}。
图5示出了一种对应于图3所示的视频拼接方法中的后向追踪的流程图。后向追踪的过程对应于上述步骤S300，当对第一视频和第二视频待拼接的目标帧IM_A(t)和IM_B(t)进行后向追踪时，以第一视频和第二视频待拼接的目标帧IM_A(t)和IM_B(t)之后的第k帧即IM_A(t+k)和IM_B(t+k)为后向追踪的起点帧，依据倒序对后向追踪的起点帧IM_A(t+k)和IM_B(t+k)之前的每一帧进行后向追踪，直到追踪到待拼接的目标帧IM_A(t)和IM_B(t)为止。其中，k为后向追踪目标帧数，可以取值为大于0的整数。后向追踪的关键是确定后向追踪的起点帧，以及第一视频和第二视频待拼接的每一帧的后向特征点对集。
例如，在一些示例中，本公开的实施例提供的视频拼接方法还可以包括：若第一目标帧IM_A(t)和第二目标帧IM_B(t)之后的待拼接的视频帧数小于后向追踪目标帧数k，则将第一视频和第二视频的待拼接的最终帧IM_A(N)和IM_B(N)作为后向追踪的起点帧。
需要说明的是，待拼接的目标帧之后的待拼接的视频帧数小于后向追踪目标帧数k的情况一般出现在k大于1时。此时，对于待拼接的目标帧IM_A(N-k+1)和IM_B(N-k+1)，…，IM_A(N-1)和IM_B(N-1)，由于其后的待拼接的视频帧数分别为k-1，…，1，小于后向追踪目标帧数k，即无法进行k次后向追踪，因此将待拼接的最终帧IM_A(N)和IM_B(N)作为这些待拼接的目标帧的后向追踪的起点帧，并相应减少这些待拼接的目标帧的后向追踪目标帧数。
需要说明的是，从后向追踪的起点帧后向追踪到待拼接的目标帧的过程与从前向追踪的起点帧(例如，待拼接的初始帧)前向追踪到待拼接的目标帧的过程相似，因此关于后向追踪过程的细节和原理可以参考前向追踪过程的细节和原理的描述。
例如，在一些示例中，如图5所示，本公开的实施例提供的视频拼接方法还可以包括：若第一目标帧和第二目标帧为第一视频和第二视频的待 拼接的某一帧的后向追踪的起点帧，则将第一目标帧和第二目标帧的第一特征点对集作为第一视频的第一目标帧的前一帧和第二视频的第二目标帧的前一帧的后向特征点对集。具体地，例如，以IM_A(t+k)和IM_B(t+k)作为第一目标帧和第二目标帧，其为第一视频和第二视频的待拼接的某一帧(IM_A(t)和IM_B(t))的后向追踪的起点帧，则将第一目标帧IM_A(t+k)和第二目标帧IM_B(t+k)的第一特征点对集{(A(t+k)_FMF[s],B(t+k)_FMF[s])}作为第一视频的第一目标帧IM_A(t+k)的前一帧IM_A(t+k-1)(图5中未示出)和第二视频的第二目标帧IM_B(t+k)的前一帧IM_B(t+k-1)(图5中未示出)的后向特征点对集{(A(t+k-1)_BTF[s],B(t+k-1)_BTF[s])}。
例如，在一些示例中，如图5所示，本公开的实施例提供的视频拼接方法还可以包括：若第一目标帧和第二目标帧为第一视频和第二视频的待拼接的某一帧的后向追踪的中间帧，则将第一目标帧和第二目标帧的第一特征点对集与第三特征点对集的并集作为第一视频的第一目标帧的前一帧和第二视频的第二目标帧的前一帧的后向特征点对集。具体地，以后向追踪目标帧数k＝2为例，取IM_A(t)和IM_B(t)作为第一目标帧和第二目标帧，其为第一视频和第二视频的待拼接的某一帧(图5中未示出的IM_A(t-1)和IM_B(t-1))的后向追踪的中间帧，此时IM_A(t+1)和IM_B(t+1)为待拼接的IM_A(t-1)和IM_B(t-1))的后向追踪的起点帧，则将第一目标帧IM_A(t)和第二目标帧IM_B(t)的第一特征点对集{(A(t)_FMF[s],B(t)_FMF[s])}与第三特征点对集{(A(t)_BTMF[s],B(t)_BTMF[s])}的并集作为第一视频的第一目标帧IM_A(t)的前一帧IM_A(t-1)和第二视频的第二目标帧IM_B(t)的前一帧IM_B(t-1)的后向特征点对集{(A(t-1)_BTF[s],B(t-1)_BTF[s])}。需要说明的是，此处k＝2为示例性的，本公开包括但不限于此，例如，k还可以为取值大于2的整数。还需要说明的是，当k＝1时，后向追踪过程中不存在后向追踪的中间帧，此时，本公开实施例提供的视频拼接方法中可以不包括对后向追踪的中间帧的处理；另外，本公开实施例提供的视频拼接方法中也可以包括对后向追踪的中间帧的处理，并不影响视频拼接的实现。
需要说明的是，前向追踪过程中，待拼接的初始帧IM_A(1)和IM_B(1)无法进行前向追踪，从而没有对应的第二特征点对集，因而需要另作处理；相应地，后向追踪过程中，待拼接的最终帧IM_A(N)和IM_B(N)无法进行 后向追踪，从而没有对应的第三特征点对集，因而也需要另作处理。
例如，在一些示例中，本公开的实施例提供的视频拼接方法还可以包括：若第一目标帧和第二目标帧为待拼接的最终帧IM_A(N)和IM_B(N)，则不进行后向追踪，根据第一目标帧IM_A(N)和第二目标帧IM_B(N)的第一特征点对集{(A(N)_FMF[s],B(N)_FMF[s])}与第二特征点对集{(A(N)_FTMF[s],B(N)_FTMF[s])}的并集，计算第一目标帧IM_A(N)和所述第二目标帧IM_B(N)之间的几何变换关系。
需要说明的是，在本公开的实施例提供的视频拼接方法中，对后向追踪目标帧数k的取值不作限制。例如，在一些示例中，后向追踪目标帧数k可以取值为例如1-20，例如1-10，例如1-5，例如1，从而该视频拼接方法既可以用于对离线视频进行拼接，又可以用于对在线视频进行准实时的拼接。例如，在一些示例中，对于待拼接的视频总帧数为N的离线视频，后向追踪目标帧数k可以取值为N-1，此时，后向追踪过程与前向追踪过程可以视作相同(差别是前向追踪是顺序，后向追踪是倒序)。例如，在一些实例中，对于在线视频，后向追踪目标帧数k可以取值为大于20的整数，本公开对此不作限制。需要说明的是，对于在线视频，后向追踪目标帧数k的取值越大，视频拼接的准实时性越差，但是，目标帧与目标帧之后的帧在时间上的连续性越好。
例如，在一些示例中，本公开的实施例提供的视频拼接方法还可以包括：若第一目标帧IM_A(t)和第二目标帧IM_B(t)的第二特征点对集{(A(t)_FTMF[s],B(t)_FTMF[s])}的特征点对的数量大于第一阈值，则对第一目标帧IM_A(t)和第二目标帧IM_B(t)的第二特征点对集{(A(t)_FTMF[s],B(t)_FTMF[s])}进行随机滤波。需要说明的是，对目标帧(第一目标帧和第二目标帧)的第二特征点对集进行随机滤波，是对该第二特征点对集的特征点对进行随机丢弃，以使该第二特征点对集中剩余的特征点对数量不超过第一阈值，从而可以保持第二特征点对集在目标帧的后一帧的前向特征点对集中对应的权重的相对稳定，进而可以保持对目标帧的后一帧进行前向追踪的稳定性。
例如，在一些示例中，本公开的实施例提供的视频拼接方法还可以包括：若第一目标帧IM_A(t)和第二目标帧IM_B(t)的第三特征点对集{(A(t)_BTMF[s],B(t)_BTMF[s])}的特征点对的数量大于第二阈值，则对第 一目标帧IM_A(t)和第二目标帧IM_B(t)的第三特征点对集{(A(t)_BTMF[s],B(t)_BTMF[s])}进行随机滤波。需要说明的是，对目标帧(第一目标帧和第二目标帧)的第三特征点对集进行随机滤波，是对该第三特征点对集的特征点对进行随机丢弃，以使该第三特征点对集中剩余的特征点对数量不超过第二阈值，从而可以保持第三特征点对集在目标帧的前一帧的后向特征点对集中对应的权重的相对稳定，进而可以保持对目标帧的前一帧进行后向追踪的稳定性。
需要说明的是，通过随机滤波对目标帧的第二特征点对集和第三特征点对集中的特征点对的数量进行限制，还可以保持第二特征点对集和第三特征点对集在目标帧的第一特征点对集、第二特征点对集以及第三特征点对集的并集中对应的权重的相对稳定，从而根据该并集计算第一目标帧和第二目标帧之间的几何变换关系得到的单应矩阵相当于进行了时间上的加权，因此相邻的待拼接的帧对应的单应矩阵在时间上的变换更加平缓，实现了稳定的相机标定。
例如，在一些示例中，本公开的实施例提供的视频拼接方法还可以包括：根据第一目标帧和第二目标帧之间的几何变换关系，对第一目标帧和第二目标帧进行拼接，并输出第一目标帧和第二目标帧的全景图像。需要说明的是，在对第一目标帧和第二目标帧进行拼接时，在该两帧图像的边界的重叠部分可能由于曝光等因素的不同导致得到的全景图像上出现拼接缝，因此，还可以将该两帧图像的重叠部分的像素值进行加权融合以消除拼接缝。
例如，在一些示例中，本公开的实施例提供的视频拼接方法还可以包括：将第一视频和第二视频的待拼接的每一帧分别作为第一目标帧和第二目标帧，输出待拼接的每一帧的全景图像，形成全景视频。该形成全景视频的过程可以参考图1以及本公开中关于图1的描述，在此不再赘述。
例如，在本公开的实施例提供的视频拼接方法中，第一视频和第二视频可以为在线视频或者离线视频。需要说明的是，当第一视频和第二视频都为在线视频时，可以将本公开实施例提供的视频拼接方法中的后向追踪目标帧数k设置为一个较小的数值，例如1-20，例如1-10，例如1-5，例如1，以实现准实时的拼接。
需要说明的是，在本公开的实施例中，上述视频拼接方法的流程可以 包括更多或更少的步骤，这些步骤可以顺序执行或并行执行。在上文描述的视频拼接方法的流程中，各个步骤的执行顺序以实现视频拼接为准，并不受各个步骤的序号标记的限制。
本公开至少一实施例还提供一种视频拼接装置。如图6所示，该视频拼接装置1包括存储器10和处理器20。存储器10存储有可执行指令，该可执行指令可由处理器20执行以实现本公开任一实施例提供的视频拼接方法。
存储器10可以包括一个或多个计算机程序产品，所述计算机程序产品可以包括各种形式的计算机可读存储介质，例如易失性存储器和/或非易失性存储器。所述易失性存储器例如可以包括随机存取存储器(RAM)和/或高速缓冲存储器(cache)等。所述非易失性存储器例如可以包括只读存储器(ROM)、硬盘、闪存等。在所述计算机可读存储介质上可以存储有上述可执行指令。
处理器20可以采用数字信号处理器(DSP)、现场可编程门阵列(FPGA)、可编程逻辑阵列(PLA)中的至少一种硬件形式来实现，其可以包括一个或多个中央处理单元(CPU)或者具有数据处理能力和/或指令执行能力的其它形式的处理单元，并且可以控制视频拼接装置1中的必要的结构、单元或者模块(图6中未示出)以执行期望的功能。例如，处理器20可以执行上述可执行指令以根据本公开任一实施例提供的视频拼接方法进行视频拼接。
需要说明的是，本公开的实施例并没有给出该视频拼接装置的全部结构。为使该视频拼接装置实现本公开的实施例提供的视频拼接方法，本领域技术人员可以理解，该视频拼接装置还可以包括其他必要的结构、单元或者模块。例如，该视频拼接装置1还可以包括视频接收单元、全景视频输出单元等，本公开的实施例对此不作限制。
以上所述，仅为本公开的具体实施方式，但本公开的保护范围并不局限于此，本公开的保护范围应以权利要求的保护范围为准。

Claims (13)

  1. 一种视频拼接方法，用于拼接第一视频和第二视频，包括：
    对所述第一视频的第一目标帧和所述第二视频的第二目标帧分别进行特征提取，对从所述第一目标帧和所述第二目标帧中分别提取的特征进行特征匹配，并对匹配的特征进行筛选，获得所述第一目标帧和所述第二目标帧的第一特征点对集；
    将所述第一目标帧和所述第二目标帧的前向特征点对集前向追踪到所述第一目标帧和所述第二目标帧，获得所述第一目标帧和所述第二目标帧的第二特征点对集；
    将所述第一目标帧和所述第二目标帧的后向特征点对集后向追踪到所述第一目标帧和所述第二目标帧，获得所述第一目标帧和所述第二目标帧的第三特征点对集；以及
    根据所述第一目标帧和所述第二目标帧的所述第一特征点对集、所述第二特征点对集以及所述第三特征点对集的并集，计算所述第一目标帧和所述第二目标帧之间的几何变换关系。
  2. 根据权利要求1所述的视频拼接方法，还包括：
    将所述第一目标帧和所述第二目标帧的所述第一特征点对集与所述第二特征点对集的并集作为所述第一视频的第一目标帧的后一帧和所述第二视频的第二目标帧的后一帧的前向特征点对集。
  3. 根据权利要求2所述的视频拼接方法，还包括：若所述第一目标帧和所述第二目标帧为待拼接的初始帧，则不进行前向追踪，根据所述第一目标帧和所述第二目标帧的所述第一特征点对集与所述第三特征点对集的并集，计算所述第一目标帧和所述第二目标帧之间的几何变换关系，将所述第一目标帧和所述第二目标帧的第一特征点对集作为所述第一视频的第一目标帧的后一帧和所述第二视频的第二目标帧的后一帧的前向特征点对集。
  4. 根据权利要求1-3任一项所述的视频拼接方法，还包括：若所述第一目标帧和所述第二目标帧之后的待拼接的视频帧数小于后向追踪目标帧数，则将所述第一视频和所述第二视频的待拼接的最终帧作为后向追踪的起点帧。
  5. 根据权利要求4所述的视频拼接方法，还包括：若所述第一目标帧和所述第二目标帧为所述第一视频和所述第二视频的待拼接的某一帧的后向追踪的起点帧，则将所述第一目标帧和所述第二目标帧的第一特征点对集作为所述第一视频的第一目标帧的前一帧和所述第二视频的第二目标帧的前一帧的后向特征点对集；
    若所述第一目标帧和所述第二目标帧为所述第一视频和所述第二视频的待拼接的某一帧的后向追踪的中间帧，则将所述第一目标帧和所述第二目标帧的所述第一特征点对集与所述第三特征点对集的并集作为所述第一视频的第一目标帧的前一帧和所述第二视频的第二目标帧的前一帧的后向特征点对集。
  6. 根据权利要求5所述的视频拼接方法，还包括：若所述第一目标帧和所述第二目标帧为待拼接的最终帧，则不进行后向追踪，根据所述第一目标帧和所述第二目标帧的所述第一特征点对集与所述第二特征点对集的并集，计算所述第一目标帧和所述第二目标帧之间的几何变换关系。
  7. 根据权利要求6所述的视频拼接方法，其中，所述后向追踪目标帧数为1-20帧。
  8. 根据权利要求4-7任一项所述的视频拼接方法，还包括：若所述第一目标帧和所述第二目标帧的所述第二特征点对集的特征点对的数量大于第一阈值，则对所述第一目标帧和所述第二目标帧的所述第二特征点对集进行随机滤波。
  9. 根据权利要求8所述的视频拼接方法，还包括：若所述第一目标帧和所述第二目标帧的所述第三特征点对集的特征点对的数量大于第二阈值，则对所述第一目标帧和所述第二目标帧的所述第三特征点对集进行随机滤波。
  10. 根据权利要求1-9任一项所述的视频拼接方法，还包括：根据所述第一目标帧和所述第二目标帧之间的几何变换关系，对所述第一目标帧和所述第二目标帧进行拼接，并输出所述第一目标帧和所述第二目标帧的全景图像。
  11. 根据权利要求10所述的视频拼接方法，还包括：将所述第一视频和所述第二视频的待拼接的每一帧分别作为所述第一目标帧和所述第二目标帧，输出所述待拼接的每一帧的全景图像，形成全景视频。
  12. 根据权利要求11所述的视频拼接方法，其中，所述第一视频和所述第二视频为在线视频或者离线视频。
  13. 一种视频拼接装置，包括：
    存储器；和
    处理器；
    其中，所述存储器存储有可执行指令，所述可执行指令可由所述处理器执行以实现根据权利要求1-12任一项所述的视频拼接方法。
PCT/CN2019/127805 2018-12-28 2019-12-24 视频拼接方法及装置 Ceased WO2020135394A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US17/049,347 US11538177B2 (en) 2018-12-28 2019-12-24 Video stitching method and device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201811625266.XA CN111385490B (zh) 2018-12-28 2018-12-28 视频拼接方法及装置
CN201811625266.X 2018-12-28

Publications (1)

Publication Number Publication Date
WO2020135394A1 true WO2020135394A1 (zh) 2020-07-02

Family

ID=71126350

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/127805 Ceased WO2020135394A1 (zh) 2018-12-28 2019-12-24 视频拼接方法及装置

Country Status (3)

Country Link
US (1) US11538177B2 (zh)
CN (1) CN111385490B (zh)
WO (1) WO2020135394A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114363568A (zh) * 2021-12-31 2022-04-15 桂林长海发展有限责任公司 基于多视频拼接的移动目标监测方法、系统和存储介质
CN116567350A (zh) * 2023-05-19 2023-08-08 上海国威互娱文化科技有限公司 全景视频数据处理方法及系统

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113014799B (zh) * 2021-01-28 2023-01-31 维沃移动通信有限公司 图像显示方法、装置和电子设备
US20220414825A1 (en) * 2021-06-29 2022-12-29 V5 Technologies Co., Ltd. Image stitching method
CN113572978A (zh) * 2021-07-30 2021-10-29 北京房江湖科技有限公司 全景视频的生成方法和装置
CN114125324B (zh) * 2021-11-08 2024-02-06 北京百度网讯科技有限公司 一种视频拼接方法、装置、电子设备及存储介质
CN117541764B (zh) * 2024-01-09 2024-04-05 北京大学 一种图像拼接方法、电子设备及存储介质

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101646022A (zh) * 2009-09-04 2010-02-10 深圳华为通信技术有限公司 一种图像拼接的方法、系统
US20160286138A1 (en) * 2015-03-27 2016-09-29 Electronics And Telecommunications Research Institute Apparatus and method for stitching panoramaic video
CN106991644A (zh) * 2016-01-20 2017-07-28 上海慧体网络科技有限公司 一种基于运动场多路摄像头进行视频拼接的方法
CN108307200A (zh) * 2018-01-31 2018-07-20 深圳积木易搭科技技术有限公司 一种在线视频拼接方法系统
TWI639136B (zh) * 2017-11-29 2018-10-21 國立高雄科技大學 即時視訊畫面拼接方法
CN108737743A (zh) * 2017-04-14 2018-11-02 中国科学院苏州纳米技术与纳米仿生研究所 基于图像拼接的视频拼接装置及视频拼接方法
CN108921776A (zh) * 2018-05-31 2018-11-30 深圳市易飞方达科技有限公司 一种基于无人机的图像拼接方法及装置

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103226822B (zh) * 2013-05-15 2015-07-29 清华大学 医疗影像拼接方法
CN108229280B (zh) * 2017-04-20 2020-11-13 北京市商汤科技开发有限公司 时域动作检测方法和系统、电子设备、计算机存储介质
CN109218727B (zh) * 2017-06-30 2021-06-25 书法报视频媒体(湖北)有限公司 视频处理的方法和装置
CN108470354B (zh) * 2018-03-23 2021-04-27 云南大学 视频目标跟踪方法、装置和实现装置

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101646022A (zh) * 2009-09-04 2010-02-10 深圳华为通信技术有限公司 一种图像拼接的方法、系统
US20160286138A1 (en) * 2015-03-27 2016-09-29 Electronics And Telecommunications Research Institute Apparatus and method for stitching panoramaic video
CN106991644A (zh) * 2016-01-20 2017-07-28 上海慧体网络科技有限公司 一种基于运动场多路摄像头进行视频拼接的方法
CN108737743A (zh) * 2017-04-14 2018-11-02 中国科学院苏州纳米技术与纳米仿生研究所 基于图像拼接的视频拼接装置及视频拼接方法
TWI639136B (zh) * 2017-11-29 2018-10-21 國立高雄科技大學 即時視訊畫面拼接方法
CN108307200A (zh) * 2018-01-31 2018-07-20 深圳积木易搭科技技术有限公司 一种在线视频拼接方法系统
CN108921776A (zh) * 2018-05-31 2018-11-30 深圳市易飞方达科技有限公司 一种基于无人机的图像拼接方法及装置

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114363568A (zh) * 2021-12-31 2022-04-15 桂林长海发展有限责任公司 基于多视频拼接的移动目标监测方法、系统和存储介质
CN116567350A (zh) * 2023-05-19 2023-08-08 上海国威互娱文化科技有限公司 全景视频数据处理方法及系统
CN116567350B (zh) * 2023-05-19 2024-04-19 上海国威互娱文化科技有限公司 全景视频数据处理方法及系统

Also Published As

Publication number Publication date
US11538177B2 (en) 2022-12-27
US20210264622A1 (en) 2021-08-26
CN111385490A (zh) 2020-07-07
CN111385490B (zh) 2021-07-13

Similar Documents

Publication Publication Date Title
CN111385490B (zh) 视频拼接方法及装置
US10297034B2 (en) Systems and methods for fusing images
US10395341B2 (en) Panoramic image generation method and apparatus for user terminal
CN109040575B (zh) 全景视频的处理方法、装置、设备、计算机可读存储介质
WO2021043295A1 (zh) 一种全景视频的目标追踪方法、装置及便携式终端
US20160286138A1 (en) Apparatus and method for stitching panoramaic video
CN110738599B (zh) 图像拼接方法、装置、电子设备及存储介质
US9386229B2 (en) Image processing system and method for object-tracing
Bose et al. Visual odometry for pixel processor arrays
JP2020053774A (ja) 撮像装置および画像記録方法
CN108513057A (zh) 图像处理方法及装置
CN112367464A (zh) 图像输出方法、装置及电子设备
CN109889736B (zh) 基于双摄像头、多摄像头的图像获取方法、装置及设备
JP2025510536A (ja) 画像処理方法及び装置、並びにデバイス
CN107454328A (zh) 图像处理方法、装置、计算机可读存储介质和计算机设备
CN113395434B (zh) 一种预览图像虚化方法、存储介质及终端设备
WO2019052197A1 (zh) 飞行器参数设定方法和装置
JP2016111561A (ja) 情報処理装置、システム、情報処理方法及びプログラム
CN119094809A (zh) 亿级像素融合视频的图像拼接方法、装置及系统
Wang et al. Pixel-wise video stabilization
CN104038668A (zh) 一种全景视频显示方法及系统
CN113938605A (zh) 拍照方法、装置、设备及介质
CN106791298A (zh) 一种具有双摄像头的终端及拍照方法
CN108540715B (zh) 一种图片处理方法、电子设备和计算机可读存储介质
CN114170417A (zh) 目标对象识别方法、装置、计算机设备和存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19904990

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19904990

Country of ref document: EP

Kind code of ref document: A1