US20160142643A1 - Parallax tolerant video stitching with spatial-temporal localized warping and seam finding - Google Patents
Parallax tolerant video stitching with spatial-temporal localized warping and seam finding Download PDFInfo
- Publication number
- US20160142643A1 US20160142643A1 US14/540,960 US201414540960A US2016142643A1 US 20160142643 A1 US20160142643 A1 US 20160142643A1 US 201414540960 A US201414540960 A US 201414540960A US 2016142643 A1 US2016142643 A1 US 2016142643A1
- Authority
- US
- United States
- Prior art keywords
- spatial
- temporal
- target
- warping
- maps
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 claims abstract description 71
- 230000008569 process Effects 0.000 claims abstract description 37
- 230000002123 temporal effect Effects 0.000 claims description 63
- 230000006870 function Effects 0.000 claims description 20
- 230000000007 visual effect Effects 0.000 claims description 15
- 238000004590 computer program Methods 0.000 claims description 5
- 238000002372 labelling Methods 0.000 claims description 4
- 230000009466 transformation Effects 0.000 description 9
- 238000012545 processing Methods 0.000 description 8
- 230000033001 locomotion Effects 0.000 description 5
- 238000005259 measurement Methods 0.000 description 5
- 101100219315 Arabidopsis thaliana CYP83A1 gene Proteins 0.000 description 3
- 101100269674 Mus musculus Alyref2 gene Proteins 0.000 description 3
- 101100140580 Saccharomyces cerevisiae (strain ATCC 204508 / S288c) REF2 gene Proteins 0.000 description 3
- 238000010276 construction Methods 0.000 description 3
- 239000011159 matrix material Substances 0.000 description 3
- 238000005457 optimization Methods 0.000 description 3
- 230000002093 peripheral effect Effects 0.000 description 3
- 230000003068 static effect Effects 0.000 description 3
- 206010044565 Tremor Diseases 0.000 description 2
- 230000004075 alteration Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 230000005055 memory storage Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000001131 transforming effect Effects 0.000 description 2
- 101100152598 Arabidopsis thaliana CYP73A5 gene Proteins 0.000 description 1
- 101000806846 Homo sapiens DNA-(apurinic or apyrimidinic site) endonuclease Proteins 0.000 description 1
- 101000835083 Homo sapiens Tissue factor pathway inhibitor 2 Proteins 0.000 description 1
- 102100026134 Tissue factor pathway inhibitor 2 Human genes 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000007123 defense Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000002156 mixing Methods 0.000 description 1
- 230000011514 reflex Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000000087 stabilizing effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/265—Mixing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/60—Editing figures and text; Combining figures or text
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
-
- G06T5/006—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/80—Geometric correction
Definitions
- the present disclosure relates generally to video processing, and more particularly, to a system and method for parallax tolerant video stitching with spatial-temporal localized warping and seam finding.
- a method of parallax tolerant video stitching includes determining a plurality of video sequences to be stitched together; performing a spatial-temporal localized warping computation process on the video sequences to determine a plurality of target warping maps; warping a plurality of frames among the video sequences into a plurality of target virtual frames using the target warping maps; performing a spatial-temporal content-based seam finding process on the target virtual frames to determine a plurality of target seam maps; and stitching the video sequences together using the target seam maps.
- an apparatus for parallax tolerant video stitching includes at least one memory and at least one processor coupled to the at least one memory.
- the at least one processor is configured to determine a plurality of video sequences to be stitched together, perform a spatial-temporal localized warping computation process on the video sequences to determine a plurality of target warping maps, warp a plurality of frames among the video sequences into a plurality of target virtual frames using the target warping maps, perform a spatial-temporal content-based seam finding process on the target virtual frames to determine a plurality of target seam maps, and stitch the video sequences together using the target seam maps.
- a non-transitory computer readable medium embodying a computer program.
- the computer program includes computer readable program code for determining a plurality of video sequences to be stitched together; performing a spatial-temporal localized warping computation process on the video sequences to determine a plurality of target warping maps; warping a plurality of frames among the video sequences into a plurality of target virtual frames using the target warping maps; performing a spatial-temporal content-based seam finding process on the target virtual frames to determine a plurality of target seam maps; and stitching the video sequences together using the target seam maps.
- FIG. 1A illustrates an example of parallax artifacts caused by using a single global homography
- FIG. 1B illustrates parallax artifacts that are corrected by using a homography mesh
- FIGS. 2A and 2B illustrate an example of parallax artifacts caused by directly applying two-dimensional (2D) stitching technique in a video
- FIG. 3 illustrates an overall workflow of video stitching according to this disclosure
- FIG. 4 illustrates a detailed view of a spatial-temporal localized warping framework that implements the functions of a spatial-temporal localized warping computation block described in FIG. 3 , according to this disclosure;
- FIG. 5 illustrates a detailed view of a spatial-temporal content-based seam finding framework that implements the functions of a spatial-temporal content-based seam finding block described in FIG. 3 , according to this disclosure;
- FIG. 6 illustrates an example of a graph that can be constructed using a spatial-temporal graph construction process according to this disclosure
- FIG. 7 illustrates an example method for video stitching according to this disclosure.
- FIG. 8 illustrates an example of a computing device for performing a video stitching workflow according to this disclosure.
- FIGS. 1A through 8 discussed below, and the various embodiments used to describe the principles of the present invention in this patent document are by way of illustration only and should not be construed in any way to limit the scope of the invention. Those skilled in the art will understand that the principles of the invention may be implemented in any type of suitably arranged device or system.
- each image can be stitched to the reference image with localized homographies to significantly reduce the parallax artifacts, as illustrated in FIG. 1B .
- embodiments of this disclosure provide a video stitching system and method that include a spatial-temporal localized warping framework and a spatial-temporal content-based seam finding framework.
- the spatial-temporal localized warping framework addresses the artifacts caused by moving objects in video stitching.
- the framework includes a spatial-temporal cost function to determine the optimal localized warping maps to align videos to a reference video by preserving the spatial-temporal local alignment, preserving the spatial-temporal global alignment, and maintaining the spatial-temporal smoothness.
- the spatial-temporal content-based seam finding framework addresses issues caused by both the inconsistent stitching seams and the undesired seams cutting through salient foreground objects.
- the framework includes a spatial-temporal content-based graph-cut seam finding mechanism.
- a spatial-temporal graph is constructed; the graph contains both spatial and temporal edges, and takes into account the objectness of the pixels.
- the optimal flow seam that is found based on the graph can stitch the videos together more consistently as well as avoid cutting through salient foreground objects.
- FIG. 3 illustrates an overall workflow of video stitching according to this disclosure.
- the workflow 300 shown in FIG. 3 is for illustration only. Other embodiments of the workflow 300 may be used without departing from the scope of this disclosure.
- a reference video sequence is defined, which can be any one of the n video sequences 301 a - 301 n .
- a primary objective of video stitching is to generate a larger video sequence by stitching the corresponding frames of the n video sequences 301 a - 301 n to the reference video sequence.
- Let ⁇ t denote the frame in the reference video sequence at time t
- I i,t denote the frame in the i-th video sequence at time t.
- the video stitching workflow 300 includes two function blocks that enable parallax-tolerant video stitching: the spatial-temporal localized warping computation block 310 and the spatial-temporal content-based seam finding block 320 .
- the spatial-temporal localized warping computation block 310 uses the video sequences 301 a - 301 n to determine a set of target warping maps M i,t 302 .
- Each frame I i,t is warped into a target virtual frame ⁇ i,t 303 using a corresponding target warping map M i,t 302 .
- the spatial-temporal content-based seam finding block 320 uses the target virtual frames 303 to determine a set of target seam maps 304 .
- the function blocks 310 , 320 will now be described in greater detail.
- FIG. 4 illustrates a detailed view of a spatial-temporal localized warping framework that implements the functions of the spatial-temporal localized warping computation block 310 according to this disclosure.
- the spatial-temporal localized warping framework 400 may be used in connection with the video stitching workflow 300 in FIG. 3 .
- the spatial-temporal localized warping framework 400 shown in FIG. 4 is for illustration only. Other embodiments of the framework 400 may be used without departing from the scope of this disclosure.
- Each target warping map M i,t includes information for transforming (or warping) the original frame I i,t to a target virtual frame ⁇ i,t where ⁇ i,t aligns with the reference frame ⁇ t .
- Various keypoint extraction techniques can be used to extract the visual keypoints, such as the 2D or 3D Harris corner detectors.
- Various descriptors can be used for d i,t,k , such as the SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), or FAST (Features from Accelerated Segment Test) descriptors.
- Each spatial global homography ⁇ tilde over (S) ⁇ i is a 3 by 3 transformation matrix that transforms each frame I i,t to align with the reference frame ⁇ t .
- each temporal global homography T i,t is a 3 by 3 transformation matrix that transforms each frame I i,t to align with a temporal reference frame Î i,t for the i-th video sequence.
- each frame I i,t By transforming each frame I i,t using the temporal global homography T it to align with the temporal reference frame Î i,t , the benefit of stabilizing the original video frames I i,t can be automatically realized by using a static global camera path defined by A i . This is beneficial to the final stitching result when a small amount of camera shakiness exists during the video capture. Such shakiness can occur when the camera system is not completely physically stabilized, for example, when the camera system is used outdoor in strong wind.
- each input video frame I i,t is transformed to a pre-warped video frame ⁇ i,t according to the equation:
- ⁇ i,t ⁇ tilde over (S) ⁇ i T i,t I i,t ,
- a x n ⁇ y n uniform grid is defined that divides each image into x n ⁇ y n uniform cells.
- a set of target vertices ⁇ circumflex over (V) ⁇ i,t,k , k 1, . . .
- the coefficients can be determined using any of a number of different methods, such as the inverse bilinear interpolation method described in REF2. Therefore, minimizing E ds encourages the final target vertices to transform each original frame I i,t to align with the reference image ⁇ t by matching their corresponding keypoints.
- the coefficients can be determined using the same method as in the preceding paragraph. Therefore, minimizing E dt encourages the final target vertices to transform each original frame I i,t to align with the reference image ⁇ t while maintaining the temporal correspondence alignment.
- the parameter E gs measures spatial global alignment.
- the parameter E gt measures temporal global alignment.
- r ⁇ t denote a temporal neighborhood of time frame t.
- the weight value ⁇ i,t,l is determined by the scale of pixel movement in the spatial neighborhood of the pre-warped vertex V i,t,l . That is, if the scene remains static in the spatial neighborhood of the pre-warped vertex V i,t,l , the corresponding vertex ⁇ circumflex over (V) ⁇ i,t,l is encouraged to remain the same through time, i.e., ⁇ i,t,l should take a large value close to 1.
- the scale of movement determined using the pre-warped temporal matching pairs in the spatial neighborhood of the pre-warped vertex V i,t,l is used to determine the weight value ⁇ i,t,l .
- other motion measurements such as the ones based on optical flow can also be used to determine ⁇ i,t,l .
- the parameter E ss measures spatial smoothness.
- ⁇ denote a set of triplets, where each triplet in ⁇ contains three vertices V i,t,l (1), V i,t,l (2), V i,t,l (3), which define a triangle.
- the vertex V i,t,l (1) can be represented by the other vertices V i,t,l (2), V i,t,l (3) according to the following:
- E ss encourages the mesh cells to undergo a similarity transformation spatially, which helps to reduce the local distortion during optimization.
- the value ⁇ s is a weight assigned to each triangle, which is determined by the spatial edge saliency in the triangle and helps to distribute more distortion to less salient regions.
- the parameter E st measures temporal smoothness.
- ⁇ denote a set of triplets, where each triplet in ⁇ contains three vertices V i,t,l (1), V i,t,l (2), V i,t,l (3), which define a triangle.
- the vertex V i,t,l (1) can be represented by the other vertices V i,t,l (2), V i,t,l (3) as:
- E st encourages the mesh cells to undergo a similarity transformation temporally, which helps to reduce the local distortion during optimization.
- the value ⁇ t is a weight assigned to each triangle, which is determined by the temporal edge saliency in the triangle and helps to distribute more distortion to less salient regions.
- the weights ⁇ 0, ⁇ 0, ⁇ 0, ⁇ 0, ⁇ 0, ⁇ 0 are assigned to each term in the cost function in Equation (1) to balance the importance of different terms in optimization.
- the cost function in Equation (1) reduces to the content-preserving warping method for static image stitching developed in REF2.
- each mesh cell C j its four vertices ⁇ circumflex over (V) ⁇ i,t,j (1), ⁇ circumflex over (V) ⁇ i,t,j (2), ⁇ circumflex over (V) ⁇ i,t,j (3), ⁇ circumflex over (V) ⁇ i,t,j (4) and V i,t,j (1), V i,t,j (2), V i,t,j (3), V i,t,j (4) define a perspective transformation H i,t,j to transform the pixels of image I i,t in the mesh cell C j to align with the corresponding mesh cell ⁇ tilde over (C) ⁇ j in the reference image ⁇ i .
- FIG. 5 illustrates a detailed view of a spatial-temporal content-based seam finding framework that implements the functions of the spatial-temporal content-based seam finding block 320 according to this disclosure.
- the spatial-temporal content-based seam finding framework 500 may be used in connection with the video stitching workflow 300 in FIG. 3 .
- the spatial-temporal content-based seam finding framework 500 shown in FIG. 5 is for illustration only. Other embodiments of the framework 500 may be used without departing from the scope of this disclosure.
- the first step of the spatial-temporal content-based seam finding framework 500 is the spatial-temporal objectness computation process 501 .
- an objectness value o i,j,t,k ⁇ [0,1] is assigned to each overlapping pixel p i,j,t,k between ⁇ i,t and ⁇ j,t .
- the objectness value o i,j,t,k measures the level of object saliency of the pixel p i,j,t,k . The more salient the pixel p i,j,t,k is, the larger the value o i,j,t,k has, and it is less desirable that the target seam cuts through the pixel p i,j,t,k . There are a number of different methods of determining the objectness value o i,j,t,k . For example, if the pixel is on a human face, the target seam is not encouraged to cut through the human face to avoid artifacts.
- the value f i,j,t,k is the distance from the pixel p i,j,t,k to an automatically detected human face, and e i,j,t,k is the distance from the pixel p i,j,t,k to a close-by strong moving edge.
- the values a, b are the weights to balance these two terms.
- FIG. 6 illustrates an example of such a graph constructed according to this disclosure.
- the graph 600 includes a plurality of graph nodes 601 .
- Each graph node 601 is an overlapping pixel p i,j,t,k .
- the spatial edge is the edge between two graph nodes that corresponds to pixels at the same time index but different spatial locations.
- the temporal edge is the edge between two graph nodes that corresponds to pixels at the same spatial location but different time indices.
- the spatial edge 602 between pixel p i,j,t,k and p i,j,t,l is defined as E s i,j,t (k,l) according to the following:
- D( ⁇ i,t (k), ⁇ j,t (k)) is the distance measurement between pixel value ⁇ i,t (k) and pixel value ⁇ j,t (k), and ⁇ i,t (k) is the pixel value of the k-th pixel in frame ⁇ i,t .
- Various distance measurements can be used to determine D( ⁇ i,t (k), ⁇ j,t (k)). For example, in one embodiment:
- E s i,j,t ( k,l ) o i,j,t,k ⁇ i,t ( k ) ⁇ ⁇ j,t ( k ) ⁇ + o i,j,t,l ⁇ i,t ( l ) ⁇ ⁇ j,t ( l ) ⁇ .
- the temporal edge 603 between pixel p i,j,t,k and p i,j,t+1,k is defined as E t i,j,k (t,t+1) according to the following:
- E t i,j,k ( t,t+ 1) ( o i,j,t,k +o i,j,t+1,k )( D ( ⁇ i,t ( k ), ⁇ i,t+1 ( k ))+ D ( ⁇ j,t ( k ), ⁇ j,t+1 ( k ))/2,
- D( ⁇ i,t (k), ⁇ i,t+1 (k)) is the distance measurement between pixel value ⁇ i,t (k) and pixel value ⁇ i,t+1 (k).
- Various distance measurements can be used to determine D( ⁇ i,t (k), ⁇ i,t+1 (k)). For example in one embodiment:
- E t i,j,k ( t,t+ 1) ( o i,j,t,k +o i,j,t+1,k )( ⁇ I i,t ( k ) ⁇ I i,t+1 ( k ) ⁇ + ⁇ I j,t ( k ) ⁇ I j,t+1 ( k ) ⁇ )/2.
- the max-flow seam computation process 503 is performed to find the optimal labeling ⁇ i,j,t,k of every overlapping pixel p i,j,t,k .
- the labeling ⁇ i,j,t,k is either source or sink, and is determined by finding a minimal-edge-cost path to cut the graph.
- ⁇ i,j,t,k is source, the corresponding pixel in the final stitched image will take the pixel value from ⁇ i,t , and if ⁇ i,j,t,k is sink, the corresponding pixel in the final stitched image will take the pixel value from ⁇ j,t .
- the above process is conducted iteratively by adding frames to the stitched result one by one. That is, frame ⁇ 1,t and ⁇ 2,t are first stitched together, and then frame ⁇ 3,t is added in to stitch with the stitched result of frame ⁇ 1,t and ⁇ 2,t , and so on.
- various color correction, gain compensation, and blending techniques can be used to visually enhance the stitched result.
- FIG. 7 illustrates an example method for video stitching according to this embodiment.
- the method 700 is described as being used with a computing device capable of video processing, such as the computing device 800 of FIG. 8 (described below).
- the method 700 could be used by any suitable device and in any suitable system.
- a plurality of video sequences are determined to be stitched together. In some embodiments, this may include a computing device determining the video sequences 301 a - 301 n in FIG. 3 .
- a spatial-temporal localized warping computation process is performed on the video sequences to determine a plurality of target warping maps. In some embodiments, this may include the spatial-temporal localized warping framework 400 performing the functions of the spatial-temporal localized warping computation block 310 in FIG. 3 .
- a plurality of frames among the video sequences are warped into a plurality of target virtual frames using the target warping maps determined in step 703 .
- a spatial-temporal content-based seam finding process is performed on the target virtual frames to determine a plurality of target seam maps. In some embodiments, this may include the spatial-temporal content-based seam finding framework 500 performing the functions of the spatial-temporal content-based seam finding block 320 in FIG. 3 .
- the video sequences are stitched together using the target seam maps.
- FIG. 7 illustrates one example of a method 700 for video stitching
- various changes may be made to FIG. 7 .
- steps in FIG. 7 could overlap, occur in parallel, occur in a different order, or occur any number of times.
- FIG. 8 illustrates an example of a computing device 800 for performing the video stitching workflow 300 of FIG. 3 or the video stitching method 700 of FIG. 7 .
- the computing device 800 includes a computing block 803 with a processing block 805 and a system memory 807 .
- the processing block 805 may be any type of programmable electronic device for executing software instructions, but will conventionally be one or more microprocessors.
- the system memory 807 may include both a read-only memory (ROM) 809 and a random access memory (RAM) 811 .
- ROM read-only memory
- RAM random access memory
- both the read-only memory 809 and the random access memory 811 may store software instructions for execution by the processing block 805 .
- the processing block 805 and the system memory 807 are connected, either directly or indirectly, through a bus 813 or alternate communication structure, to one or more peripheral devices.
- the processing block 805 or the system memory 807 may be directly or indirectly connected to one or more additional memory storage devices 815 .
- the memory storage devices 815 may include, for example, a “hard” magnetic disk drive, a solid state disk drive, an optical disk drive, and a removable disk drive.
- the processing block 805 and the system memory 807 also may be directly or indirectly connected to one or more input devices 817 and one or more output devices 819 .
- the input devices 817 may include, for example, a keyboard, a pointing device (such as a mouse, touchpad, stylus, trackball, or joystick), a touch screen, a scanner, a camera, and a microphone.
- the output devices 819 may include, for example, a display device, a printer and speakers. Such a display device may be configured to display video images.
- one or more of the peripheral devices 815 - 819 may be internally housed with the computing block 803 .
- one or more of the peripheral devices 815 - 819 may be external to the housing for the computing block 803 and connected to the bus 813 through, for example, a Universal Serial Bus (USB) connection or a digital visual interface (DVI) connection.
- USB Universal Serial Bus
- DVI digital visual interface
- the computing block 803 may also be directly or indirectly connected to one or more network interfaces cards (NIC) 821 , for communicating with other devices making up a network.
- the network interface cards 821 translate data and control signals from the computing block 803 into network messages according to one or more communication protocols, such as the transmission control protocol (TCP) and the Internet protocol (IP).
- TCP transmission control protocol
- IP Internet protocol
- the network interface cards 821 may employ any suitable connection agent (or combination of agents) for connecting to a network, including, for example, a wireless transceiver, a modem, or an Ethernet connection.
- computing device 800 is illustrated as an example only, and it not intended to be limiting. Various embodiments of this disclosure may be implemented using one or more computing devices that include the components of the computing device 800 illustrated in FIG. 8 , or which include an alternate combination of components, including components that are not shown in FIG. 8 . For example, various embodiments of the invention may be implemented using a multi-processor computer, a plurality of single and/or multiprocessor computers arranged into a network, or some combination of both.
- the embodiments described herein provide a solution for parallax tolerant video stitching
- the computed localized warping maps are able to align frames from multiple videos by optimally preserving the spatial and temporal data alignment and the spatial temporal smoothness.
- the resulting warped frames are spatially well aligned with localized warping, and are temporally consistent.
- the resulting seams can be used to stitch frames from multiple videos together with good temporal consistency while avoiding cutting through salient foreground objects to avoid artifacts.
- a computer program that is formed from computer readable program code and that is embodied in a computer readable medium.
- computer readable program code includes any type of computer code, including source code, object code, and executable code.
- computer readable medium includes any type of medium capable of being accessed by a computer, such as read only memory (ROM), random access memory (RAM), a hard disk drive, a compact disc (CD), a digital video disc (DVD), or any other type of memory.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
- Geometry (AREA)
Abstract
Description
- The present disclosure relates generally to video processing, and more particularly, to a system and method for parallax tolerant video stitching with spatial-temporal localized warping and seam finding.
- The increasingly powerful computation, large storage, and expanding transmission bandwidths have enabled a wide variety of applications in the market, which provides modern users all kinds of visual experiences. For instance, with the advent of high-definition displaying devices such as very large screens and Ultra-HD TVs, there has been a strong interest in generating high-quality videos with an ultra-large Field-of-View (FoV) that can give users immersive media experiences. A variety of devices and methods have been developed to construct large FoV images. Very expensive, high-end camera systems are used by professional agencies for this purpose, such as the AWARE-2 camera used in the defense industry, which is a monocentric, multi-scale camera that includes a spherically symmetric objective lens surrounded by an array of secondary microcameras. For groups with a smaller budget (e.g., independent photographers or even amateur consumers), a camera system that can obtain reasonable quality but with much less expense is desired.
- According to one embodiment, there is provided a method of parallax tolerant video stitching. The method includes determining a plurality of video sequences to be stitched together; performing a spatial-temporal localized warping computation process on the video sequences to determine a plurality of target warping maps; warping a plurality of frames among the video sequences into a plurality of target virtual frames using the target warping maps; performing a spatial-temporal content-based seam finding process on the target virtual frames to determine a plurality of target seam maps; and stitching the video sequences together using the target seam maps.
- According to another embodiment, there is provided an apparatus for parallax tolerant video stitching. The apparatus includes at least one memory and at least one processor coupled to the at least one memory. The at least one processor is configured to determine a plurality of video sequences to be stitched together, perform a spatial-temporal localized warping computation process on the video sequences to determine a plurality of target warping maps, warp a plurality of frames among the video sequences into a plurality of target virtual frames using the target warping maps, perform a spatial-temporal content-based seam finding process on the target virtual frames to determine a plurality of target seam maps, and stitch the video sequences together using the target seam maps.
- According to yet another embodiment, there is provided a non-transitory computer readable medium embodying a computer program. The computer program includes computer readable program code for determining a plurality of video sequences to be stitched together; performing a spatial-temporal localized warping computation process on the video sequences to determine a plurality of target warping maps; warping a plurality of frames among the video sequences into a plurality of target virtual frames using the target warping maps; performing a spatial-temporal content-based seam finding process on the target virtual frames to determine a plurality of target seam maps; and stitching the video sequences together using the target seam maps.
- For a more complete understanding of the present disclosure, and the advantages thereof, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, wherein like numbers designate like objects, and in which:
-
FIG. 1A illustrates an example of parallax artifacts caused by using a single global homography; -
FIG. 1B illustrates parallax artifacts that are corrected by using a homography mesh; -
FIGS. 2A and 2B illustrate an example of parallax artifacts caused by directly applying two-dimensional (2D) stitching technique in a video; -
FIG. 3 illustrates an overall workflow of video stitching according to this disclosure; -
FIG. 4 illustrates a detailed view of a spatial-temporal localized warping framework that implements the functions of a spatial-temporal localized warping computation block described inFIG. 3 , according to this disclosure; -
FIG. 5 illustrates a detailed view of a spatial-temporal content-based seam finding framework that implements the functions of a spatial-temporal content-based seam finding block described inFIG. 3 , according to this disclosure; -
FIG. 6 illustrates an example of a graph that can be constructed using a spatial-temporal graph construction process according to this disclosure; -
FIG. 7 illustrates an example method for video stitching according to this disclosure; and -
FIG. 8 illustrates an example of a computing device for performing a video stitching workflow according to this disclosure. -
FIGS. 1A through 8 , discussed below, and the various embodiments used to describe the principles of the present invention in this patent document are by way of illustration only and should not be construed in any way to limit the scope of the invention. Those skilled in the art will understand that the principles of the invention may be implemented in any type of suitably arranged device or system. - The following documents are hereby incorporated into the present disclosure as if fully set forth herein: (i) Brady et al., “Multiscale gigapixel photography,” Nature 486:386-389, 2012 (hereinafter “REF1”); (ii) F. Zhang and F. Liu, “Parallax-tolerant image stitching,” IEEE CVPR, 2014 (hereinafter “REF2”); and (iii) Szeliski, “Image alignment and stitching: A tutorial,” Foundations and Trends in Computer Graphics and Computer Vision, 2006 (hereinafter “REF3”).
- Using affordable cameras, such as non DSLR (digital single-lens reflex) or mobile cameras, many methods have been developed for generating large Field-of-View (FoV) 2D photo panoramas. These methods does not require recovering the geometric and photometric scene models, but they require that the captured scene be planar or distant, or that the camera viewpoints be closely located, in which cases each image can be stitched to a reference image by using a single global homography. When such requirements are not met perfectly, i.e., when a single global homography is not enough to stitch an image to the reference image (which is generally the case in real applications), the resulting stitched panorama usually presents different levels of parallax artifacts such as ghosting and distortion, as illustrated in
FIG. 1A . - To alleviate the problem of parallax artifacts, one or more localized content-preserving warping algorithms have been developed. By using a homography mesh instead of a single global homography, each image can be stitched to the reference image with localized homographies to significantly reduce the parallax artifacts, as illustrated in
FIG. 1B . However, it is quite difficult to generalize the previous 2D panorama methods to construct video panoramas if the video contains non-negligible medium- to large-size moving objects. Directly applying the previous 2D panorama methods to stitch individual video frames will result in severe artifacts, not only around the object region due to the object movement, but also for the overall stitched video due to the inconsistency in 2D warps and/or stitching seams, as illustrated inFIG. 2B (compared to an earlier frame inFIG. 2A , which shows little or no artifacts). - To resolve these issues, embodiments of this disclosure provide a video stitching system and method that include a spatial-temporal localized warping framework and a spatial-temporal content-based seam finding framework. The spatial-temporal localized warping framework addresses the artifacts caused by moving objects in video stitching. The framework includes a spatial-temporal cost function to determine the optimal localized warping maps to align videos to a reference video by preserving the spatial-temporal local alignment, preserving the spatial-temporal global alignment, and maintaining the spatial-temporal smoothness.
- The spatial-temporal content-based seam finding framework addresses issues caused by both the inconsistent stitching seams and the undesired seams cutting through salient foreground objects. The framework includes a spatial-temporal content-based graph-cut seam finding mechanism. A spatial-temporal graph is constructed; the graph contains both spatial and temporal edges, and takes into account the objectness of the pixels. The optimal flow seam that is found based on the graph can stitch the videos together more consistently as well as avoid cutting through salient foreground objects.
-
FIG. 3 illustrates an overall workflow of video stitching according to this disclosure. Theworkflow 300 shown inFIG. 3 is for illustration only. Other embodiments of theworkflow 300 may be used without departing from the scope of this disclosure. - To better explain the
video stitching workflow 300, it is assumed that there are n video sequences 301 a-301 n that are to be stitched together. A reference video sequence is defined, which can be any one of the n video sequences 301 a-301 n. A primary objective of video stitching is to generate a larger video sequence by stitching the corresponding frames of the n video sequences 301 a-301 n to the reference video sequence. Let Ĩt denote the frame in the reference video sequence at time t, and let Ii,t denote the frame in the i-th video sequence at time t. Using video stitching, a virtual frame I′t is generated by stitching Ii,t, i=1, . . . , n to Ĩt at different times t=1, . . . , m. - The
video stitching workflow 300 includes two function blocks that enable parallax-tolerant video stitching: the spatial-temporal localizedwarping computation block 310 and the spatial-temporal content-basedseam finding block 320. The spatial-temporal localizedwarping computation block 310 uses the video sequences 301 a-301 n to determine a set of targetwarping maps M i,t 302. Each frame Ii,t is warped into a target virtual frame Ĩi,t 303 using a corresponding targetwarping map M i,t 302. The spatial-temporal content-basedseam finding block 320 uses the targetvirtual frames 303 to determine a set oftarget seam maps 304. The function blocks 310, 320 will now be described in greater detail. -
FIG. 4 illustrates a detailed view of a spatial-temporal localized warping framework that implements the functions of the spatial-temporal localizedwarping computation block 310 according to this disclosure. The spatial-temporallocalized warping framework 400 may be used in connection with thevideo stitching workflow 300 inFIG. 3 . The spatial-temporallocalized warping framework 400 shown inFIG. 4 is for illustration only. Other embodiments of theframework 400 may be used without departing from the scope of this disclosure. - As shown in
FIG. 4 , the spatial-temporallocalized warping framework 400 takes a set of video sequences Ii,t, i=1, . . . , n, t=1, . . . , m (represented inFIG. 4 by the video sequences 301 a-301 n), and determines a set of target warping maps Mi,t, i=1, . . . , n, t=1, . . . , m (represented inFIG. 4 by the target warping maps 302). Each target warping map Mi,t includes information for transforming (or warping) the original frame Ii,t to a target virtual frame Ĩi,t where Ĩi,t aligns with the reference frame Ĩt. - The first step of the spatial-temporal
localized warping framework 400 is to take the set of video sequences Ii,t, i=1, . . . , n, t=1, . . . , m (video sequences 301 a-301 n), and extract a set of visual keypoints (Pi,t,k,di,t,k), k=1, . . . , Ki (keypoints 401 a-401 n) from each video sequence, where di,t,k is a visual descriptor. Each visual keypoint (Pi,t,k,di,t,k)i records the spatial-temporal location of the keypoint in the corresponding video sequence. - Together, the parameter Pi,t,k=(xi,t,k,yi,t,k) and the visual descriptor di,t,k describe the local visual characteristics around the visual keypoint in the corresponding video sequence. Various keypoint extraction techniques can be used to extract the visual keypoints, such as the 2D or 3D Harris corner detectors. Various descriptors can be used for di,t,k, such as the SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), or FAST (Features from Accelerated Segment Test) descriptors.
- Using the set of visual keypoints (Pi,t,k,di,t,k), k=1, . . . , Ki (keypoints 401 a-401 n), the spatial-temporal
localized warping framework 400 determines a set of spatial global homographies {tilde over (S)}i, i=1, . . . , n (spatial global homographies 402), and a set of temporal global homographies Ti,t, i=1, . . . , n, t=1, . . . , m (temporal global homographies 403). Each spatial global homography {tilde over (S)}i is a 3 by 3 transformation matrix that transforms each frame Ii,t to align with the reference frame Ĩt. Similarly, each temporal global homography Ti,t is a 3 by 3 transformation matrix that transforms each frame Ii,t to align with a temporal reference frame Îi,t for the i-th video sequence. - In a preferred embodiment, the temporal reference frame Îi,t can be determined using two steps. First, an averaged temporal global homography Ai can be calculated as Ai=avgΣtAi(t,t+1), where Ai(t,t+1) is a 3 by 3 transformation matrix to transform frame Ii,t+1 to align with frame Ii,t. Then, in the second step, the temporal reference frame Îi,t can be calculated as Îi,t=Ai (t−1)Ii,1. By transforming each frame Ii,t using the temporal global homography Tit to align with the temporal reference frame Îi,t, the benefit of stabilizing the original video frames Ii,t can be automatically realized by using a static global camera path defined by Ai. This is beneficial to the final stitching result when a small amount of camera shakiness exists during the video capture. Such shakiness can occur when the camera system is not completely physically stabilized, for example, when the camera system is used outdoor in strong wind.
- In a preferred embodiment, temporal matching pairs (Pi,t,l,Pi,t+1,l), l=1, . . . , Li,t can be determined based on the similarity between keypoints (Pi,t,k,di,t,k) and (Pi,t+1,k,di,t+1,k), and Ai(t,t+1) can be determined based on the temporal matching pairs (Pi,t,l,Pi,t+1,l), l=1, . . . , Li,t using Ransac and outlier rejection. Using the averaged temporal global homography Ai, the first item in the temporal matching pairs Pi,t,l, l=1, . . . , Li,t can be transformed to new locations P′i,t,l, l=1, . . . , Li,t, and the temporal global homography Ti,t is determined based on the matching pairs (P′i,t,l,Pi,t+1,l), l=1, . . . , Li,t using Ransac and outlier rejection. At the same time, spatial matching pairs (Pi,t,l,{tilde over (P)}t,l), l=1, . . . , Li can be found based on the similarity between keypoints (Pi,t,k,di,t,k) and ({tilde over (P)}t,k,{tilde over (d)}t,k), and the spatial global homography {tilde over (S)}i can be determined using all the spatial matching pairs at different times (Pi,t,l,Pj,t,l), l=1, . . . , Li,j, t=1, . . . , m using Ransac and outlier rejection, where ({tilde over (P)}t,k,{tilde over (d)}t,k), k=1, . . . , {tilde over (K)} are the keypoints extracted over the reference video sequence, which can be any one of the n input video sequences 301 a-301 n.
- A
pre-warping process 404 uses the spatial global homographies {tilde over (S)}i, i=1, . . . , n (spatial global homographies 402) and the temporal global homographies Tit, i=1, . . . , n, t=1, . . . , m (temporal global homographies 403). In thepre-warping process 404, each input video frame Ii,t is transformed to a pre-warped video frame Īi,t according to the equation: -
Ī i,t ={tilde over (S)} i T i,t I i,t, - the temporal matching pairs (Pi,t,l,Pi,t+1,l), l=1, . . . , Li,t are transformed to a set of pre-warped temporal matching pairs (
P i,t,l,P i,t+1,l), l=1, . . . , Li,t (pre-warped temporal matching pairs 405) according to the equation: -
P i,t,l ={tilde over (S)} i T i,t P i,t,l ,P i,t+1,l ={tilde over (S)} i T i,t+1 P i,t+1,l, - and the spatial matching pairs (Pi,t,l,{tilde over (P)}t,l), l=1, . . . , Li, t=1, . . . , m are transformed to a set of pre-warped spatial matching pairs (
P i,t,l,{tilde over (P)}t,l), l=1, . . . , Li (pre-warped spatial matching pairs 406) according to the equation: -
P i,t,l ={tilde over (S)} i T i,t P i,t,l. - A xn×yn uniform grid is defined that divides each image into xn×yn uniform cells. Let Vi,t,k, k=1, . . . , (xn+1)(yn+1) and
V i,t,k, k=1, . . . , (xn+1)(yn+1) denote the vertices of the grid mesh in image Ii,t and the pre-warped image Īi,t, respectively. In the spatial-temporal localized warping computation process, a set of target vertices {circumflex over (V)}i,t,k, k=1, . . . , (xn+1)(yn+1) (target vertices 407) are determined based on the input vertices Vi,t,k, k=1, . . . , (xn+1)(yn+1) andV i,t,k, k=1, . . . , (xn+1)(yn+1), and the input pre-warped spatial matching pairs (P i,t,l,{tilde over (P)}t,l), l=1, . . . , Li and pre-warped temporal matching pairs (P i,t,l,P i,t+1,l), l=1, . . . , Li,t. For each mesh cell Cj, its four vertices {circumflex over (V)}i,t,j(1), {circumflex over (V)}i,t,j(2), {circumflex over (V)}i,t,j(3), {circumflex over (V)}i,t,j(4) and Vi,t,j(1), Vi,t,j(2), Vi,t,j(3), Vi,t,j(4) defines a perspective transformation Hi,t,j to transform the pixels of image Ii,t in the mesh cell Cj to align with the corresponding mesh cell {tilde over (C)}j in the reference image Ĩt. In a preferred embodiment, {circumflex over (V)}i,t,k, k=1, . . . , (xn+1)(yn+1), i=1, . . . , n, t=1, . . . , m is determined by minimizing the following cost function: -
- The parameter Eds measures spatial local alignment, where (
P i,t,l,{tilde over (P)}t,l) is a pre-warpedspatial matching pair 406, andP i,t,l is represented by a linear combination of the four verticesV i,t,l(k), k=1, . . . , 4 that containsP i,t,l with coefficients λi,t,l(k), k=1, . . . , 4. The coefficients can be determined using any of a number of different methods, such as the inverse bilinear interpolation method described in REF2. Therefore, minimizing Eds encourages the final target vertices to transform each original frame Ii,t to align with the reference image Ĩt by matching their corresponding keypoints. - The parameter Edt measures temporal local alignment, where (
P i,t,l,P i,t+1,l) is a pre-warpedtemporal matching pair 405, andP i,t,l is represented by a linear combination of the four verticesV i,t,l(k), k=1, . . . , 4 that containsP i,t,l with coefficients λi,t,l(k), k=1, . . . , 4. The coefficients can be determined using the same method as in the preceding paragraph. Therefore, minimizing Edt encourages the final target vertices to transform each original frame Ii,t to align with the reference image Ĩt while maintaining the temporal correspondence alignment. - The parameter Egs measures spatial global alignment. When there is no pre-warped spatial matching pairs in the spatial neighborhood of the pre-warped vertex
V i,t,l, the corresponding vertex {circumflex over (V)}i,t,l is encouraged to be the same as the pre-warped vertexV i,t,l, and therefore, τi,t,l=1. Otherwise, τi,t,l=0. - The parameter Egt measures temporal global alignment. Let rεΩt denote a temporal neighborhood of time frame t. When there is no pre-warped temporal matching pairs in the spatial neighborhood of the pre-warped vertex
V i,t,l, the corresponding vertex {circumflex over (V)}i,t,l is encouraged to remain the same through time (i.e., remain unchanged within the temporal neighborhood Ωt), and therefore, σi,t,l=1. When there exist pre-warped temporal matching pairs in the spatial neighborhood of the pre-warped vertexV i,t,l, the weight value σi,t,l is determined by the scale of pixel movement in the spatial neighborhood of the pre-warped vertexV i,t,l. That is, if the scene remains static in the spatial neighborhood of the pre-warped vertexV i,t,l, the corresponding vertex {circumflex over (V)}i,t,l is encouraged to remain the same through time, i.e., σi,t,l should take a large value close to 1. When there exists substantial scene movement in the spatial neighborhood of the pre-warped vertexV i,t,l, σi,t,l should take a small value close to 0. In a preferred embodiment, the scale of movement determined using the pre-warped temporal matching pairs in the spatial neighborhood of the pre-warped vertexV i,t,l is used to determine the weight value σi,t,l. In other embodiments, other motion measurements such as the ones based on optical flow can also be used to determine σi,t,l. - The parameter Ess measures spatial smoothness. Let Δ denote a set of triplets, where each triplet in Δ contains three vertices
V i,t,l(1),V i,t,l(2),V i,t,l(3), which define a triangle. The vertexV i,t,l(1) can be represented by the other verticesV i,t,l(2),V i,t,l(3) according to the following: -
- If the triangle undergoes a similarity transformation, its coordinates in the local coordinate system will remain the same. Therefore, minimizing Ess encourages the mesh cells to undergo a similarity transformation spatially, which helps to reduce the local distortion during optimization. The value
ω s is a weight assigned to each triangle, which is determined by the spatial edge saliency in the triangle and helps to distribute more distortion to less salient regions. - The parameter Est measures temporal smoothness. Again, let Δ denote a set of triplets, where each triplet in Δ contains three vertices
V i,t,l(1),V i,t,l(2),V i,t,l(3), which define a triangle. The vertexV i,t,l(1) can be represented by the other verticesV i,t,l(2),V i,t,l(3) as: -
- If the triangle undergoes a similarity transformation, its coordinates in the local coordinate system will remain the same. Therefore, minimizing Est encourages the mesh cells to undergo a similarity transformation temporally, which helps to reduce the local distortion during optimization. The value
ω t is a weight assigned to each triangle, which is determined by the temporal edge saliency in the triangle and helps to distribute more distortion to less salient regions. - The weights φ≧0, α≧0, β≧0, φ≧0, θ≧0 are assigned to each term in the cost function in Equation (1) to balance the importance of different terms in optimization. When φ=0, φ=0, θ=0, the cost function in Equation (1) reduces to the content-preserving warping method for static image stitching developed in REF2.
- After obtaining the target vertices {circumflex over (V)}i,t,k, k=1, . . . , (xn+1)(yn+1), i=1, . . . , n, t=1, . . . , m, the set of target warping maps Mi,t, i=1, . . . , n, t=1, . . . , m can be determined based on the original vertices, Vi,t,k, k=1, . . . , (xn+1)(yn+1), i=1, . . . , n, t=1, . . . , m and the target vertices {circumflex over (V)}i,t,k, k=1, . . . , (xn+1)(yn+1), i=1, . . . , n, t=1, . . . , m. There are several ways to determine the target warping maps. In a preferred embodiment, for each mesh cell Cj, its four vertices {circumflex over (V)}i,t,j(1), {circumflex over (V)}i,t,j(2), {circumflex over (V)}i,t,j(3), {circumflex over (V)}i,t,j(4) and Vi,t,j(1), Vi,t,j(2), Vi,t,j(3), Vi,t,j(4) define a perspective transformation Hi,t,j to transform the pixels of image Ii,t in the mesh cell Cj to align with the corresponding mesh cell {tilde over (C)}j in the reference image Ĩi. The target warping map Mi,t is simply formed as the set of Hi,t,j, j=1, . . . , xnyn, and the whole image Ii,t can be warped by Mi,t cell by cell into the target virtual frame Ĩi,t (target virtual frame 303).
-
FIG. 5 illustrates a detailed view of a spatial-temporal content-based seam finding framework that implements the functions of the spatial-temporal content-basedseam finding block 320 according to this disclosure. The spatial-temporal content-basedseam finding framework 500 may be used in connection with thevideo stitching workflow 300 inFIG. 3 . The spatial-temporal content-basedseam finding framework 500 shown inFIG. 5 is for illustration only. Other embodiments of theframework 500 may be used without departing from the scope of this disclosure. - As shown in
FIG. 5 , the spatial-temporal content-basedseam finding framework 500 takes the set of target virtual frames Ĩi,t, i=1, . . . , n, t=1, . . . , m (represented inFIG. 5 by the target virtual frames 303), and determines a set of target seam maps Zt, t=1, . . . , m (represented inFIG. 5 by the target seam maps 304). Each seam map Zt includes information associated with composing the final stitched virtual frame I′t from the warped virtual target frames Ĩi,t, i=1, . . . , n. - The first step of the spatial-temporal content-based
seam finding framework 500 is the spatial-temporalobjectness computation process 501. Given a pair of target virtual frame sequences Ĩi,t, t=1, . . . , m and Ĩj,t, t=1, . . . , m, in the spatial-temporalobjectness computation process 501, an objectness value oi,j,t,kε[0,1] is assigned to each overlapping pixel pi,j,t,k between Ĩi,t and Ĩj,t. The objectness value oi,j,t,k measures the level of object saliency of the pixel pi,j,t,k. The more salient the pixel pi,j,t,k is, the larger the value oi,j,t,k has, and it is less desirable that the target seam cuts through the pixel pi,j,t,k. There are a number of different methods of determining the objectness value oi,j,t,k. For example, if the pixel is on a human face, the target seam is not encouraged to cut through the human face to avoid artifacts. As another example, if the pixel is on a fast moving object and is close to strong structural edges, the target seam is not encouraged to cut through the pixel to avoid artifacts. In a preferred embodiment, thecomputation process 501 takes into account the above factors in computing the objectness value, where oi,j,t,k=a*fi,j,t,k+b*ei,j,t,k. The value fi,j,t,k is the distance from the pixel pi,j,t,k to an automatically detected human face, and ei,j,t,k is the distance from the pixel pi,j,t,k to a close-by strong moving edge. The values a, b are the weights to balance these two terms. - After that, a spatial-temporal graph can be constructed using the spatial-temporal
graph construction process 502.FIG. 6 illustrates an example of such a graph constructed according to this disclosure. As shown inFIG. 6 , thegraph 600 includes a plurality ofgraph nodes 601. Eachgraph node 601 is an overlapping pixel pi,j,t,k. There are two types of edges between each pair of graph nodes: a spatial edge (represented by the spatial edge 602) and a temporal edge (represented by the temporal edge 603). The spatial edge is the edge between two graph nodes that corresponds to pixels at the same time index but different spatial locations. The temporal edge is the edge between two graph nodes that corresponds to pixels at the same spatial location but different time indices. Specifically, thespatial edge 602 between pixel pi,j,t,k and pi,j,t,l is defined as Es i,j,t(k,l) according to the following: -
E s i,j,t(k,l)=o i,j,t,k D(Ĩ i,t(k),Ĩ j,t(k))+o i,j,t,l D(Ĩ i,t(l),Ĩ j,t(l)), - where D(Ĩi,t(k),Ĩj,t(k)) is the distance measurement between pixel value Ĩi,t(k) and pixel value Ĩj,t(k), and Ĩi,t(k) is the pixel value of the k-th pixel in frame Ĩi,t. Various distance measurements can be used to determine D(Ĩi,t(k),Ĩj,t(k)). For example, in one embodiment:
-
E s i,j,t(k,l)=o i,j,t,k ∥Ĩ i,t(k)−Ĩ j,t(k)∥+o i,j,t,l ∥Ĩ i,t(l)−Ĩ j,t(l)∥. - The
temporal edge 603 between pixel pi,j,t,k and pi,j,t+1,k is defined as Et i,j,k(t,t+1) according to the following: -
E t i,j,k(t,t+1)=(o i,j,t,k +o i,j,t+1,k)(D(Ĩ i,t(k),Ĩ i,t+1(k))+D(Ĩ j,t(k),Ĩ j,t+1(k)))/2, - where D(Ĩi,t(k),Ĩi,t+1(k)) is the distance measurement between pixel value Ĩi,t(k) and pixel value Ĩi,t+1(k). Various distance measurements can be used to determine D(Ĩi,t(k),Ĩi,t+1(k)). For example in one embodiment:
-
E t i,j,k(t,t+1)=(o i,j,t,k +o i,j,t+1,k)(∥I i,t(k)−I i,t+1(k)∥+∥I j,t(k)−I j,t+1(k)∥)/2. - Without loss of generality, assume that image Ĩi,t is source and Ĩj,t is sink, the overlapping pixel that is on the boundary of the overlapped regions between Ĩi,t and Ĩj,t is given an edge to its closest image (either source or sink), with infinity edge weight.
- Then, returning to
FIG. 5 , after the graph is constructed using the spatial-temporalgraph construction process 502, the max-flow seam computation process 503 is performed to find the optimal labeling ηi,j,t,k of every overlapping pixel pi,j,t,k. The labeling ηi,j,t,k is either source or sink, and is determined by finding a minimal-edge-cost path to cut the graph. If ηi,j,t,k is source, the corresponding pixel in the final stitched image will take the pixel value from Ĩi,t, and if ηi,j,t,k is sink, the corresponding pixel in the final stitched image will take the pixel value from Ĩj,t. - To determine the final target seam map Zt, the above process is conducted iteratively by adding frames to the stitched result one by one. That is, frame Ĩ1,t and Ĩ2,t are first stitched together, and then frame Ĩ3,t is added in to stitch with the stitched result of frame Ĩ1,t and Ĩ2,t, and so on.
- Once the set of target seam maps Zt, t=1, . . . , m (target seam maps 304) are obtained, various color correction, gain compensation, and blending techniques can be used to visually enhance the stitched result.
-
FIG. 7 illustrates an example method for video stitching according to this embodiment. For ease of explanation, themethod 700 is described as being used with a computing device capable of video processing, such as thecomputing device 800 ofFIG. 8 (described below). However, themethod 700 could be used by any suitable device and in any suitable system. - At
step 701, a plurality of video sequences are determined to be stitched together. In some embodiments, this may include a computing device determining the video sequences 301 a-301 n inFIG. 3 . Atstep 703, a spatial-temporal localized warping computation process is performed on the video sequences to determine a plurality of target warping maps. In some embodiments, this may include the spatial-temporallocalized warping framework 400 performing the functions of the spatial-temporal localizedwarping computation block 310 inFIG. 3 . - At
step 705, a plurality of frames among the video sequences are warped into a plurality of target virtual frames using the target warping maps determined instep 703. Atstep 707, a spatial-temporal content-based seam finding process is performed on the target virtual frames to determine a plurality of target seam maps. In some embodiments, this may include the spatial-temporal content-basedseam finding framework 500 performing the functions of the spatial-temporal content-basedseam finding block 320 inFIG. 3 . Then, atstep 709, the video sequences are stitched together using the target seam maps. - Although
FIG. 7 illustrates one example of amethod 700 for video stitching, various changes may be made toFIG. 7 . For example, while shown as a series of steps, various steps inFIG. 7 could overlap, occur in parallel, occur in a different order, or occur any number of times. -
FIG. 8 illustrates an example of acomputing device 800 for performing thevideo stitching workflow 300 ofFIG. 3 or thevideo stitching method 700 ofFIG. 7 . As shown inFIG. 8 , thecomputing device 800 includes acomputing block 803 with aprocessing block 805 and asystem memory 807. Theprocessing block 805 may be any type of programmable electronic device for executing software instructions, but will conventionally be one or more microprocessors. Thesystem memory 807 may include both a read-only memory (ROM) 809 and a random access memory (RAM) 811. As will be appreciated by those of skill in the art, both the read-only memory 809 and therandom access memory 811 may store software instructions for execution by theprocessing block 805. - The
processing block 805 and thesystem memory 807 are connected, either directly or indirectly, through abus 813 or alternate communication structure, to one or more peripheral devices. For example, theprocessing block 805 or thesystem memory 807 may be directly or indirectly connected to one or more additionalmemory storage devices 815. Thememory storage devices 815 may include, for example, a “hard” magnetic disk drive, a solid state disk drive, an optical disk drive, and a removable disk drive. Theprocessing block 805 and thesystem memory 807 also may be directly or indirectly connected to one ormore input devices 817 and one ormore output devices 819. Theinput devices 817 may include, for example, a keyboard, a pointing device (such as a mouse, touchpad, stylus, trackball, or joystick), a touch screen, a scanner, a camera, and a microphone. Theoutput devices 819 may include, for example, a display device, a printer and speakers. Such a display device may be configured to display video images. With various examples of thecomputing device 800, one or more of the peripheral devices 815-819 may be internally housed with thecomputing block 803. Alternately, one or more of the peripheral devices 815-819 may be external to the housing for thecomputing block 803 and connected to thebus 813 through, for example, a Universal Serial Bus (USB) connection or a digital visual interface (DVI) connection. - With some implementations, the
computing block 803 may also be directly or indirectly connected to one or more network interfaces cards (NIC) 821, for communicating with other devices making up a network. Thenetwork interface cards 821 translate data and control signals from thecomputing block 803 into network messages according to one or more communication protocols, such as the transmission control protocol (TCP) and the Internet protocol (IP). Also, thenetwork interface cards 821 may employ any suitable connection agent (or combination of agents) for connecting to a network, including, for example, a wireless transceiver, a modem, or an Ethernet connection. - It should be appreciated that the
computing device 800 is illustrated as an example only, and it not intended to be limiting. Various embodiments of this disclosure may be implemented using one or more computing devices that include the components of thecomputing device 800 illustrated inFIG. 8 , or which include an alternate combination of components, including components that are not shown inFIG. 8 . For example, various embodiments of the invention may be implemented using a multi-processor computer, a plurality of single and/or multiprocessor computers arranged into a network, or some combination of both. - The embodiments described herein provide a solution for parallax tolerant video stitching By jointly minimizing the spatial-temporal cost function in the spatial-temporal localized warping framework, the computed localized warping maps are able to align frames from multiple videos by optimally preserving the spatial and temporal data alignment and the spatial temporal smoothness. As a result, the resulting warped frames are spatially well aligned with localized warping, and are temporally consistent.
- By finding the optimal spatial-temporal seams that take into account the objectness of the pixels in the spatial-temporal content-based seam finding framework, the resulting seams can be used to stitch frames from multiple videos together with good temporal consistency while avoiding cutting through salient foreground objects to avoid artifacts.
- In some embodiments, some or all of the functions or processes of the one or more of the devices are implemented or supported by a computer program that is formed from computer readable program code and that is embodied in a computer readable medium. The phrase “computer readable program code” includes any type of computer code, including source code, object code, and executable code. The phrase “computer readable medium” includes any type of medium capable of being accessed by a computer, such as read only memory (ROM), random access memory (RAM), a hard disk drive, a compact disc (CD), a digital video disc (DVD), or any other type of memory.
- It may be advantageous to set forth definitions of certain words and phrases used throughout this patent document. The terms “include” and “comprise,” as well as derivatives thereof, mean inclusion without limitation. The term “or” is inclusive, meaning and/or. The phrases “associated with” and “associated therewith,” as well as derivatives thereof, mean to include, be included within, interconnect with, contain, be contained within, connect to or with, couple to or with, be communicable with, cooperate with, interleave, juxtapose, be proximate to, be bound to or with, have, have a property of, or the like.
- While this disclosure has described certain embodiments and generally associated methods, alterations and permutations of these embodiments and methods will be apparent to those skilled in the art. Accordingly, the above description of example embodiments does not define or constrain this disclosure. Other changes, substitutions, and alterations are also possible without departing from the spirit and scope of this disclosure, as defined by the following claims.
Claims (20)
E=E ds +φE dt +αE gs +βE gt +φE ss +θE st,
E=E ds +φE dt +αE gs +βE gt +φE ss +θE st,
E=E ds +φE dt +αE gs +βE gt+φss +θE st,
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/540,960 US9363449B1 (en) | 2014-11-13 | 2014-11-13 | Parallax tolerant video stitching with spatial-temporal localized warping and seam finding |
CN201580062015.3A CN107113381B (en) | 2014-11-13 | 2015-11-11 | Tolerance video splicing method, device and computer readable medium for spatio-temporal local deformation and seam search |
EP15859688.2A EP3218870B1 (en) | 2014-11-13 | 2015-11-11 | Parallax tolerant video stitching with spatial-temporal localized warping and seam finding |
PCT/CN2015/094320 WO2016074620A1 (en) | 2014-11-13 | 2015-11-11 | Parallax tolerant video stitching with spatial-temporal localized warping and seam finding |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/540,960 US9363449B1 (en) | 2014-11-13 | 2014-11-13 | Parallax tolerant video stitching with spatial-temporal localized warping and seam finding |
Publications (2)
Publication Number | Publication Date |
---|---|
US20160142643A1 true US20160142643A1 (en) | 2016-05-19 |
US9363449B1 US9363449B1 (en) | 2016-06-07 |
Family
ID=55953746
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/540,960 Active US9363449B1 (en) | 2014-11-13 | 2014-11-13 | Parallax tolerant video stitching with spatial-temporal localized warping and seam finding |
Country Status (4)
Country | Link |
---|---|
US (1) | US9363449B1 (en) |
EP (1) | EP3218870B1 (en) |
CN (1) | CN107113381B (en) |
WO (1) | WO2016074620A1 (en) |
Cited By (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20170161927A1 (en) * | 2015-12-02 | 2017-06-08 | Caterpillar Inc. | Systems and Methods for Stitching Metallographic and Stereoscopic Images |
US20180176277A1 (en) * | 2016-12-16 | 2018-06-21 | Industrial Technology Research Institute | Video streaming stitching and transmitting method, video streaming gateway and video streaming viewer |
US20190068876A1 (en) * | 2017-08-29 | 2019-02-28 | Nokia Technologies Oy | Method Of Image Alignment For Stitching Using A Hybrid Strategy |
CN109544460A (en) * | 2017-09-22 | 2019-03-29 | 宝沃汽车(中国)有限公司 | Image correction method, device and vehicle |
CN110099220A (en) * | 2019-06-17 | 2019-08-06 | 广东中星微电子有限公司 | A kind of panorama mosaic method and device |
WO2020012072A1 (en) | 2018-07-11 | 2020-01-16 | Nokia Technologies Oy | Method and apparatus for virtual reality content stitching control with network based media processing |
CN111507904A (en) * | 2020-04-22 | 2020-08-07 | 华中科技大学 | Image splicing method and device for microcosmic printed patterns |
US10742960B1 (en) * | 2016-06-28 | 2020-08-11 | Altia Systems, Inc. | Panoramic 3D camera |
CN111932594A (en) * | 2020-09-18 | 2020-11-13 | 西安拙河安见信息科技有限公司 | Billion pixel video alignment method and device based on optical flow and medium |
WO2021003263A1 (en) * | 2019-07-01 | 2021-01-07 | Geomagical Labs, Inc. | Method and system for image generation |
US10931851B2 (en) * | 2017-08-21 | 2021-02-23 | Gopro, Inc. | Image stitching with electronic rolling shutter correction |
US11288769B2 (en) | 2019-04-15 | 2022-03-29 | Tata Consultancy Services Limited | System and method for stitching images using non-linear optimization and multi-constraint cost function minimization |
Families Citing this family (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10186067B2 (en) * | 2016-10-25 | 2019-01-22 | Aspeed Technology Inc. | Method and apparatus for generating panoramic image with rotation, translation and warping process |
CN108307200B (en) * | 2018-01-31 | 2020-06-09 | 深圳积木易搭科技技术有限公司 | Online video splicing method and system |
KR102198217B1 (en) | 2018-04-11 | 2021-01-04 | 한국전자통신연구원 | Look-up table based stitching image generating apparatus and method |
KR102383669B1 (en) * | 2018-10-23 | 2022-04-06 | 한국전자통신연구원 | Parallax minimized stitching using HLBP descriptor |
CN112073749A (en) * | 2020-08-07 | 2020-12-11 | 中国科学院计算技术研究所 | Sign language video synthesis method, sign language translation system, medium and electronic equipment |
US11210840B1 (en) * | 2020-08-27 | 2021-12-28 | Aspeed Technology Inc. | Transform method for rendering post-rotation panoramic images |
CN112465704B (en) * | 2020-12-07 | 2024-02-06 | 清华大学深圳国际研究生院 | Global-local self-adaptive optimized panoramic light field splicing method |
CN114125178A (en) * | 2021-11-16 | 2022-03-01 | 阿里巴巴达摩院(杭州)科技有限公司 | Video splicing method, device and readable medium |
Family Cites Families (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7119837B2 (en) * | 2002-06-28 | 2006-10-10 | Microsoft Corporation | Video processing system and method for automatic enhancement of digital video |
EP1854282B1 (en) * | 2004-11-05 | 2019-02-27 | Briefcam Ltd. | Method and system for spatio-temporal video warping |
KR101216161B1 (en) * | 2005-03-31 | 2012-12-27 | 유클리드 디스커버리스, 엘엘씨 | Apparatus and method for processing video data |
US7460730B2 (en) * | 2005-08-04 | 2008-12-02 | Microsoft Corporation | Video registration and image sequence stitching |
US20070030396A1 (en) * | 2005-08-05 | 2007-02-08 | Hui Zhou | Method and apparatus for generating a panorama from a sequence of video frames |
TWI303782B (en) * | 2006-03-10 | 2008-12-01 | Sony Taiwan Ltd | An optimized video stitching mehtod for asic implementation |
JP5355422B2 (en) * | 2007-02-01 | 2013-11-27 | イッサム・リサーチ・デヴェロップメント・カンパニー・オヴ・ザ・ヘブルー・ユニヴァーシティ・オヴ・イェルサレム | Method and system for video indexing and video synopsis |
US8270767B2 (en) * | 2008-04-16 | 2012-09-18 | Johnson Controls Technology Company | Systems and methods for providing immersive displays of video camera information from a plurality of cameras |
CN101635833B (en) * | 2008-07-22 | 2012-02-15 | 深圳市朗驰欣创科技有限公司 | Method, device and system for video monitoring |
US8947502B2 (en) * | 2011-04-06 | 2015-02-03 | Qualcomm Technologies, Inc. | In camera implementation of selecting and stitching frames for panoramic imagery |
US8863182B1 (en) * | 2012-02-17 | 2014-10-14 | Google Inc. | In-stream video stitching |
TWI435162B (en) * | 2012-10-22 | 2014-04-21 | Nat Univ Chung Cheng | Low complexity of the panoramic image and video bonding method |
-
2014
- 2014-11-13 US US14/540,960 patent/US9363449B1/en active Active
-
2015
- 2015-11-11 CN CN201580062015.3A patent/CN107113381B/en active Active
- 2015-11-11 WO PCT/CN2015/094320 patent/WO2016074620A1/en active Application Filing
- 2015-11-11 EP EP15859688.2A patent/EP3218870B1/en active Active
Cited By (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9721371B2 (en) * | 2015-12-02 | 2017-08-01 | Caterpillar Inc. | Systems and methods for stitching metallographic and stereoscopic images |
US20170161927A1 (en) * | 2015-12-02 | 2017-06-08 | Caterpillar Inc. | Systems and Methods for Stitching Metallographic and Stereoscopic Images |
US10742960B1 (en) * | 2016-06-28 | 2020-08-11 | Altia Systems, Inc. | Panoramic 3D camera |
US11089287B1 (en) * | 2016-06-28 | 2021-08-10 | Altia Systems Inc | Panoramic 3D camera |
US20180176277A1 (en) * | 2016-12-16 | 2018-06-21 | Industrial Technology Research Institute | Video streaming stitching and transmitting method, video streaming gateway and video streaming viewer |
CN108206940A (en) * | 2016-12-16 | 2018-06-26 | 财团法人工业技术研究院 | Video streaming connection and transmission method, gateway device and viewing device |
US10701122B2 (en) * | 2016-12-16 | 2020-06-30 | Industrial Technology Research Institute | Video streaming stitching and transmitting method, video streaming gateway and video streaming viewer |
US11962736B2 (en) | 2017-08-21 | 2024-04-16 | Gopro, Inc. | Image stitching with electronic rolling shutter correction |
US10931851B2 (en) * | 2017-08-21 | 2021-02-23 | Gopro, Inc. | Image stitching with electronic rolling shutter correction |
US20190068876A1 (en) * | 2017-08-29 | 2019-02-28 | Nokia Technologies Oy | Method Of Image Alignment For Stitching Using A Hybrid Strategy |
CN109544460A (en) * | 2017-09-22 | 2019-03-29 | 宝沃汽车(中国)有限公司 | Image correction method, device and vehicle |
WO2020012072A1 (en) | 2018-07-11 | 2020-01-16 | Nokia Technologies Oy | Method and apparatus for virtual reality content stitching control with network based media processing |
EP3821597A4 (en) * | 2018-07-11 | 2022-04-06 | Nokia Technologies Oy | Method and apparatus for virtual reality content stitching control with network based media processing |
US11288769B2 (en) | 2019-04-15 | 2022-03-29 | Tata Consultancy Services Limited | System and method for stitching images using non-linear optimization and multi-constraint cost function minimization |
CN110099220A (en) * | 2019-06-17 | 2019-08-06 | 广东中星微电子有限公司 | A kind of panorama mosaic method and device |
WO2021003263A1 (en) * | 2019-07-01 | 2021-01-07 | Geomagical Labs, Inc. | Method and system for image generation |
US11830163B2 (en) | 2019-07-01 | 2023-11-28 | Geomagical Labs, Inc. | Method and system for image generation |
CN111507904A (en) * | 2020-04-22 | 2020-08-07 | 华中科技大学 | Image splicing method and device for microcosmic printed patterns |
CN111932594A (en) * | 2020-09-18 | 2020-11-13 | 西安拙河安见信息科技有限公司 | Billion pixel video alignment method and device based on optical flow and medium |
Also Published As
Publication number | Publication date |
---|---|
WO2016074620A1 (en) | 2016-05-19 |
CN107113381A (en) | 2017-08-29 |
EP3218870A1 (en) | 2017-09-20 |
CN107113381B (en) | 2020-06-26 |
EP3218870A4 (en) | 2017-09-20 |
EP3218870B1 (en) | 2020-01-08 |
US9363449B1 (en) | 2016-06-07 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9363449B1 (en) | Parallax tolerant video stitching with spatial-temporal localized warping and seam finding | |
Perazzi et al. | Panoramic video from unstructured camera arrays | |
US9591237B2 (en) | Automated generation of panning shots | |
US11663733B2 (en) | Depth determination for images captured with a moving camera and representing moving features | |
US7006709B2 (en) | System and method deghosting mosaics using multiperspective plane sweep | |
US20170358092A1 (en) | Multi-view scene segmentation and propagation | |
US9014470B2 (en) | Non-rigid dense correspondence | |
US20140118482A1 (en) | Method and apparatus for 2d to 3d conversion using panorama image | |
US8619098B2 (en) | Methods and apparatuses for generating co-salient thumbnails for digital images | |
CN111866523B (en) | Panoramic video synthesis method and device, electronic equipment and computer storage medium | |
KR102587298B1 (en) | Real-time omnidirectional stereo matching method using multi-view fisheye lenses and system therefore | |
US20220321859A1 (en) | Real-time omnidirectional stereo matching method using multi-view fisheye lenses and system thereof | |
Ji et al. | Virtual Home Staging: Inverse Rendering and Editing an Indoor Panorama under Natural Illumination | |
US11893704B2 (en) | Image processing method and device therefor | |
Fujita et al. | Cost volume refinement filter for post filtering of visual corresponding | |
Dudek et al. | Robust global and local color matching in stereoscopic omnidirectional content | |
Lumentut et al. | 6-DOF motion blur synthesis and performance evaluation of light field deblurring | |
JP2014010806A (en) | Light source information extraction device, light source information extraction method, and light source information extraction program | |
US20240135632A1 (en) | Method and appratus with neural rendering based on view augmentation | |
Guo et al. | Efficient high dynamic range video using multi-exposure CNN flow | |
Heo et al. | Depth maps and high-dynamic range image generation from alternating exposure multiview images | |
Ma et al. | Image stitching using sigmoid function to create perception mask | |
Guo et al. | Efficient view manipulation for cuboid-structured images | |
Noda et al. | Generation of Omnidirectional Image Without Photographer | |
Zhang et al. | A gigapixel image mosaicking approach based on SURF and color transfer |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: FUTUREWEI TECHNOLOGIES, INC., TEXAS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:JIANG, WEI;GU, JINWEI;SIGNING DATES FROM 20141113 TO 20150315;REEL/FRAME:035195/0855 |
|
AS | Assignment |
Owner name: FUTUREWEI TECHNOLOGIES, INC., TEXAS Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE DOC DATE OF THE FIRST INVENTOR JIANG, WEI. PREVIOUSLY RECORDED AT REEL: 035195 FRAME: 0855. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT;ASSIGNORS:JIANG, WEI;GU, JINWEI;SIGNING DATES FROM 20150312 TO 20150315;REEL/FRAME:035752/0497 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 8 |