CN102957869B - Video stabilization - Google Patents
Video stabilization Download PDFInfo
- Publication number
- CN102957869B CN102957869B CN201210363053.0A CN201210363053A CN102957869B CN 102957869 B CN102957869 B CN 102957869B CN 201210363053 A CN201210363053 A CN 201210363053A CN 102957869 B CN102957869 B CN 102957869B
- Authority
- CN
- China
- Prior art keywords
- frame
- video camera
- displacement
- video signal
- pixel displacement
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/14—Systems for two-way working
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
- H04N23/681—Motion detection
- H04N23/6812—Motion detection based on additional sensors, e.g. acceleration sensors
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
- H04N23/682—Vibration or motion blur correction
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
- H04N23/682—Vibration or motion blur correction
- H04N23/683—Vibration or motion blur correction performed by a processor, e.g. controlling the readout of an image memory
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/14—Systems for two-way working
- H04N7/141—Systems for two-way working between two video terminals, e.g. videophone
- H04N7/142—Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
- H04N2007/145—Handheld terminals
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Studio Devices (AREA)
Abstract
A kind of for transmitting the method for video signal, equipment and computer program from subscriber equipment.On a user device, video camera is used to capture multiple video signal frame.The functional status of equipment will be determined, and before being transmitted, stablize described video signal selectively based on described functional status.
Description
Technical field
The present invention relates to stable (stabilisation) of video signal.Especially, the present invention relates to use video camera to catch
Obtain video signal frame and the motion to video camera compensates, thus stablize video signal.
Background technology
Video camera can be used for the sequence of the image that capture uses as video signal frame.Video camera can be fixed to surely
Fixed object, such as, can be arranged on video camera on the support of spider etc, thus keeps capturing frame of video when
Video camera is static.But, video camera often realizes in a mobile device, and may not be arranged on fixing object, example
As, video camera may be kept or be positioned in the moving object of vehicle etc.When video camera capture video signal frame, take the photograph
The mobile of camera may cause video signal itself undesirable movement occur.
It is a kind of method being not intended to movement compensated in video signal that image stabilization processes.Some systems are held
Row estimation produces the motion vector processed for image stabilization.At " the Online that Julan Yang et al. delivers
Video Stabilization Based on Particle Filters " literary composition describes a kind of such system.Image
Stable algorithm can include three major parts: estimation, motion smoothing and motion compensation.Estimation parts can be estimated
Meter local motion vector within video signal, and calculate global motion vector based on these partial estimation.Then, motion
Smoothing unit can process the filtering carried out for estimated global motion vector, in order to value that smoothing computation obtains and
Prevent that the biggest undesirable difference occurs between the motion vector previously calculated.Afterwards, motion compensation parts can with
The side that the global motion vector that filtered is contrary moves up image, thus stablizes video signal.Described motion compensation parts can
With to rotating, distort or the complex transformations of scaling etc taking in.
Process to perform image stabilization based on motion vector as mentioned above, then be likely to require substantial amounts of process and provide
Source.Wanting real-time stabilization video signal, namely using (such as in video call while video camera capture video signal
Transmit or export from equipment) video signal of stable release when, this likely can become a problem.Additionally,
Performing to have equally the when that equipment that image stabilization processes being the small type mobile devices of mobile phone this kind of process resource-constrained can
Problem can be there will be.
In recent years, the manufacture of motion sensor has become simpler and cheap, and the size of motion sensor is also
Significantly diminish.Now, it is feasible for realizing motion sensor in a mobile device.Motion sensor produces representative sensor motion
Sampling.At " the Using Sensors for Efficient Video Coding in Hand-that Andy L. Lin delivers
Held devices " and " the Accelerometer Based Digital Video that delivers of Martin Drahansk et al.
Stabilization for General Security Surveillance Systems " these two existing documents are addressed
Use and be derived from the data of motion sensor to stablize the probability of video signal.
Known handling implement " VirtualDub " provides needs the off-line stabilized treatment of pruning video signal (namely
Non real-time).
A kind of mechanism stable for online (the most real-time) sending side numeral is proposed in Britain's application 1109071.9.
This mechanism is all effective in a lot of situations, but it needs to prune the video transmitted.And this is likely to affect and is transmitted
The quality of video.
Summary of the invention
According to an aspect of the invention, it is provided a kind of method transmitting video signal from subscriber equipment, the method bag
Include: use video camera to capture multiple video signal frames on a user device；Determine the functional status of equipment；And be transmitted
Before, to stablize selectively video signal based on functional status.
Preferably, described functional status is the movement degree of video camera, and the method includes: the motion of surveillance camera and
By it compared with a threshold value.
Described supervision may include that using the motion sensor being associated with video camera to produce represents camera motion
Multiple samplings；These samplings are used to determine video camera displacement between the successive frame that described video camera captures；And determine
The pixel displacement of the motion in video signal between the successive frame that video camera displacement determined by representative is caused, and the method
Also include: pixel displacement is compared with threshold value；If pixel displacement exceeds described threshold value, the most stable described
Video signal；In the case of not implementing stabilized treatment, otherwise transmit video signal.
Preferably, the motion of video camera is rotary motion, and motion sensor is rotational motion sensor, and video camera
Displacement is the angular displacement of video camera.
Subscriber equipment includes forward direction video camera and backward video camera, when functional status is to have selected forward direction video camera, this
Time can transmit video signal in the case of not performing stabilized treatment.Preferably, have selected equipment rear to video camera
Time, at this moment video signal can be carried out stable.
Preferably, the sample rate using the sampling of motion sensor generation is higher than the frame rate of video signal.
Video camera and motion sensor may be mounted at inside mobile device.
Stablize the step of video signal to may include that pixel displacement is filtered；And according to the pixel position filtered
In-migration moves the image of at least one frame in the first and second frames.
The step that pixel displacement is filtered may include that based on described be that the location of pixels that determines of second frame comes really
Determine accumulation pixel displacement；And it is based upon accumulation pixel displacement and the filtering accumulation for first frame that second frame determines
The weighted sum of pixel displacement determines the pixel displacement of filtering accumulation for second frame.
Preferably, at least one in the following is added time migration: (i) the multiple frames captured；And (2)
Produced multiple samplings, so that the timing of the timing of the multiple frames captured and produced multiple samplings matches.
If pixel displacement exceeds threshold value, then intervalometer can be used to determine whether to have passed through the predetermined period,
And have passed through this period when, only just stablize described video signal.If pixel displacement is without departing from described threshold value,
Then can reset this intervalometer.
According to the second aspect of the invention, it is provided that a kind of equipment for stable video signal, this equipment includes: quilt
It is configured to capture the video camera of multiple frames of video signal；For determining the device of the functional status of equipment；And for passing
Stablize the device of described video signal selectively based on described functional status before defeated.
Preferably, described functional status is the movement degree of video camera, and this equipment includes: move also for surveillance camera
And by its device compared with a threshold value.
This equipment can also include: the motion sensor being associated with video camera, and it is configured to generation and represents video camera
Multiple samplings of motion；Pixel displacement determines parts, and it is configured to use described sampling to determine that video camera is at this video camera
Pixel displacement between the successive frame of capture, described pixel displacement represents the video between the successive frame that camera motion is caused
Signal movement；Comparing unit, it is configured to compare pixel displacement and a predetermined threshold；And motion compensation parts,
It is configured to: before being transmitted, if pixel displacement is beyond described threshold value, then stablizes described video signal, otherwise, in transmission
The most do not realize stabilized treatment.
Preferably, the motion of video camera is rotary motion, and motion sensor is rotational motion sensor, and video camera
Displacement is the angular displacement of video camera.
Described motion sensor can be gyrocompass motion sensors.Described equipment can be mobile device.
According to the third aspect of the present invention, it is provided that a kind of computer program for stable video signal, institute
State computer program to be included in the computer-readable medium of non-transitory, and be configured to when on device handler
The operation of any claim in execution claim 1-13 when of operation.
Inventors have realised that for the resolution maximizing output video, it is desirable to only realize needs when
Video stabilization processes, in order to avoid unnecessarily pruning the video signal transmitted.
Accompanying drawing explanation
In order to be more fully understood that the present invention and illustrate how to implement the present invention, now will be by way of example with reference to following
Accompanying drawing, wherein:
Fig. 1 a and 1b shows the equipment according to preferred embodiment；
Fig. 2 is the flow chart of the process for stable video signal according to preferred embodiment；
Fig. 3 is the flow chart that the supervision according to preferred embodiment processes；
Fig. 4 is that the example graphic of the camera shutter position elapsed in time represents；
Fig. 5 is that the example graphic of the angular velocity of the video camera elapsed in time represents；
Fig. 6 is the flow chart of the stabilized treatment according to preferred embodiment；And
Fig. 7 is the expression of the image before and after pruning modes.
Detailed description of the invention
Only by way of example, the preferred embodiments of the present invention will be described now.
Fig. 1 a and 1b shows the equipment 100 according to a preferred embodiment.For example, equipment 100 can be mobile
Equipment, such as mobile phone or other handheld devices.Equipment 100 includes with screen 101 towards the forward direction video camera of equidirectional
104, and with the backward video camera 102 in opposite direction of screen 101.Equipment 100 also includes motion sensor 110, CPU 106
And memorizer 108.Each in forward direction video camera 104 and backward video camera 102 is configured to be easily selected by a user
Time capture image.The image captured may be used for forming video signal, and thus each image is used as video signal
Frame, and capture image with the frame rate of video signal.For example, described frame rate can be 25 frames per second, but this
A little video cameras can also be with different frame rate work.It is about per second 15 for realizing the minimum frame speed of moving image perception
Frame, but this can depend on watching the content in people of video image and video signal (namely at the theme of video signal
In have how many motion).Motion sensor 110 is configured to produce the sampling of the motion of the equipment of representative 100.Due to motion sensor
110 and video camera 104,102 all be in equipment 100, therefore they each other be association, so, motion sensor
104 samplings produced may be used for representing any one motion in video camera 104,102.As known in the art,
CPU 106 is configured to perform calculating on the appliance 100 and processes.And as known in the art, memorizer 108 is used for
Store data in device 100.As known in the art, parts 102,104,106 and 108 each other can by via
The bus (the most not display) of equipment 100 sends data and communicates.
Regulator described here may be at three kinds of states.This regulator can be as operation on CPU 106
Code realizes.
The state of regulator is functional status based on mobile device and selected.The first state is that regulator is closed
And do not realize the process of stable video signal as described below.When forward direction video camera 104 captures video signal, at regulator
In the first state.Owing to pruning modes can make visual angle narrow, and the process of regulator Steady Background Light likely can be believed at video
Increasing more motion in number, therefore, stabilized treatment is disabled on forward direction video camera 104.
Second state of regulator is supervision equipment high frequency motion (vibrations (shakiness)) but does not enters video signal
Row is pruned.When backward video camera 102 captures video signal, regulator will be in the second state.
The third state of regulator is application stabilized treatment, thus prunes the video signal of backward video camera 102 capture.
Fig. 2 shows that the use equipment 100 according to preferred embodiment stablizes the process of video signal.
In step S201, this process determines whether video signal is captured by forward direction video camera 104.It is to say, really
Determine whether regulator is in the first state.If video signal is captured by forward direction video camera 104, then this process not followed by
Continue and carry out.Capture (it is to say, video signal is by backward video camera 102 if video signal is not forward direction video camera 104
Capture), then this process advances to step S202.
In step S202, the vibrations of backward video camera 102 will be monitored.
Show supervision step S202 in figure 3 in greater detail.In step S302, backward video camera 102 capture will by with
Make the image of video signal frame.Such as, video camera 102 can have a record during the time distributing to video signal frame
The photosensor array of the energy level of incident light on a sensor.The shutter of video camera 102 is used for separating in time these
Frame, thus during each frame, described shutter all can be opened in a period and be closed in another period.Caught
The video signal frame obtained is provided to preprocessor (such as being realized in processing component) by CPU 106.Using this area
Before frame is encoded by the video coding technique known, described preprocessor is operable to the figure in stable video signal frame
Picture.
In step S304, when video camera 102 captures video signal frame, motion sensor 110 will produce the equipment of representative
The sampling of the motion of 100.Such as, motion sensor 110 can be the rotational motion sensor of gyroscope etc.Gyroscope 110
The angular velocity of equipment 100 is measured, and output represents the sampling of described angular velocity at a specific intervals.Described interval was both
Can be aturegularaintervals, it is also possible to be not aturegularaintervals.Preferably, the average sample rate from the sampling of gyroscope 110 output is wanted
Higher than the frame rate of video signal, but this is not required in that.Such as, the sample rate from gyroscope 110 output can be per second
60 samplings, the maximum generally vibration frequency of this sample rate reflection equipment 100, and currently unrelated with frame rate.Gyro
The sampling that instrument 110 produces will be provided to preprocessor.
In step S306, the video camera 102 angular displacement between two video signal frames (frame 1 and frame 2) will be determined.Should
Determine and can be performed by the processing component of CPU 106.Inventor is it has been determined that in order to use the data from gyroscope 110
Effectively determine the angular displacement between two frames, the time interval between the exposure time midpoint of the frame of video camera 102 capture
Upper angular velocity is integrated being highly useful.Inventor also determines that, due to may not by the sample rate of gyroscope 110 with
The frame rate of video camera 102 synchronizes, and therefore, this process likely can be the most problematic, especially following state is occurring
Time:
Video camera 102 is adjusted to regulate time of exposure (a lot of video cameras are not always the case) according to available light；
The timestamp of video signal frame that video camera 102 provides be associated with shutter closing time (i.e. the end time of frame,
This is opposition with the midpoint of the time of exposure of frame)；And
At the midpoint of the time of exposure of frame, gyro data is unavailable.
As it has been described above, preprocessor receives the frame of video from video camera 105, and also can receive from gyroscope 110
Sampling.Sampling with at least equal with the frame rate of the video signal of video camera 105 capture speed from gyroscope 110
It is provided to preprocessor (such as with aturegularaintervals).By using higher sample rate in gyroscope 110, can be given more
Accurate angle estimation, but higher in the cost of CPU use.
The timestamp t of first frame (frame 1) that video camera 102 is supplied in video signal1Indicate the end time of this frame,
Namely the shutter Guan Bi of video camera 102 is with the time of end frame 1.Equally, video camera 102 be supplied in video signal second
The timestamp t of individual frame (frame 2)2Indicating the end time of described frame, namely the shutter of video camera 102 closes with end frame 2
Time.In order to determine the equipment 100 angular displacement (θ) between first frame and second frame, come with the timestamp using frame
Represent that the time of frame is different, be the midpoint of the time of exposure using frame 1 and frame 2 the most accurately.The exposure of first second frame
The light time is to use e1And e2Represent.Angular displacement is by time t1-0.5e1With time t2-0.5e2Between to equipment 100
Angular velocity (sampled representation with exporting from gyroscope 110) is integrated determining.Thus, the angular displacement between frame 1 and frame 2
It is given by:
。
Fig. 4 is that the example graphic of the shutter position of the video camera 105 elapsed in time represents.The shutter of video camera 102 exists
At time t at the end of frame 11Guan Bi.The shutter of this video camera 102 reopens again, in order to capture frame 2, then can tie at frame 2
At time t during bundle2Guan Bi.In the diagram, the time of exposure of frame 1 is shown as e1, and the time of exposure of frame 2 is shown as
e2.Use T in the diagram12Represent the time that angular velocity is integrated.Be it will be seen that by viewing Fig. 4, time T12On long-pending
Divide corresponding to the exposure time midpoint at first frame (at time t1-0.5e1) with the exposure time midpoint of second frame (time
Between t2-0.5e2The integration carried out between).Fig. 4 shows that the opening time of shutter is equal with the closing time of shutter, but this is only
It it is an example.(realizing short exposure time) in certain embodiments, the time that shutter is opened is smaller than the time of shutter Guan Bi.
By contrast, (realize long time of exposure) in other embodiments, the time of time shutter the to be longer than Guan Bi that shutter is opened.
Due to the Timing Synchronization of the video signal frame that the sampling of gyroscope 110 does not captures with video camera 12, it is therefore possible to
Occur that gyroscope 110 does not produce the situation of sampling at the midpoint of frame (frame 1 and frame 2).In this case, equipment 100 is at frame
Midpoint angular velocity can by insert gyroscope 110 generate sampling representated by angular velocity and be determined.This angular velocity
By putting execution at any time and insert and evaluated, and the midpoint of the time of exposure of frame will be defined on according to above-mentioned
The integration interval that equation uses the when of calculating angular displacement.
Fig. 5 is that the example graphic of the angular velocity of the video camera 102 elapsed in time represents.In Figure 5 gyroscope 110 is produced
It is sampling 502,504,506,508 and 510 that the sampling of the angular velocity given birth to and represent equipment 100 is shown as.It can be seen that
In example shown in Fig. 5, the sampling timing of gyroscope 110 is not rule.Such as, the time between sampling 504 and 506 is wanted
It is shorter than the time between sampling 506 and 508.In Figure 5, the dotted line that connection is sampled shows can be according to the time and by inserting
Enter the angular velocity representated by sampling that gyroscope 110 produces and the value of angular velocity being determined.The angular velocity inserted (uses dotted line
Display) can be at time (t1-0.5e1) and (t2-0.5e2Integration between), in order to determine that video camera 102 is at the first frame and second
Angular displacement between frame.Fig. 5 shows from the simple linear interpolation between the sampling of gyroscope 110.In other embodiments,
More advanced interpolation also can use.
A kind of situation likely occurs, will stable frame be wherein after the last sampling from gyroscope 110
Preprocessor receives.Such as, when capturing frame 2 on video camera 102, described frame 2 can be at gyroscope
Midpoint (t at the time of exposure of frame 22-0.5e2Receive on preprocessor before producing any sampling after).Such as, frame 2
Can be to receive on preprocessor before the sampling 510 shown in Fig. 5.In this condition, can draw in video streaming
Enter to postpone, in order on preprocessor, receive sampling 510 processing before frame 2, thus allow preprocessor process frame 2 it
Before determine time (t2-0.5e2) angular velocity.Alternatively, angular velocity can be from previous receipt from the sampling of gyroscope 110
Extrapolation obtains, in order to determine that equipment 100 is at time (t2-0.5e2) angular velocity.
If video camera 102 the most mobile (such as fixed placement equipment 100), then gyroscope 110 can be disabled, in order to
Save battery life.Nonmotile state can be determined by the feedback from video encoder, wherein said Video coding
Device can be realized by preprocessor and after image stability method described here to encoding video signal performing.
As a part for coded treatment, described video encoder can perform estimation, and thus may determine that video camera is
No movement.Additionally, video camera 102 moves when, kinestate at this moment can also be determined and uses it for enabling gyro
Instrument 110.When equipment 100 do not have motion state in work time, can slowly interval in poll motion sensor 110,
To determine whether equipment 100 starts again at movement.According to the hardware realized in the operating system of equipment 100 and application programming
Interface (API), may have the mode that computing cost is less mobile to determine when equipment 100 starts.
Operation timing for video camera 102 and the hardware of gyroscope 110 may not be mated.Reason among these is likely
It is described video camera 102 and gyroscope 119 is to realize in independent hardware chip.Therefore, to adopting that gyroscope 110 generates
The timestamp of any one (or all both) in sample and video signal frame adds a skew and is probably and is highly profitable
's.In this way, the timing from the sampling of gyroscope 110 correctly can be mated with the timing of video signal frame.For
Specific hardware chip combination, described skew can be a constant.Thus, delay can be by calculated off line and at equipment 100
Upper use, without causing treatment loss for method described herein.
Back with reference to Fig. 3, in step S308, the pixel displacement of the motion representing video camera 102 will be determined.Generally,
The rotation of video camera 102 can result in and approximation unrelated with the distance of the object in image in the two field picture of video signal
Constant pixel displacement.This is contrasted with linear camera motion, wherein for described linear camera moves, and pixel
Displacement is the function of the distance to object.Depend on taking the photograph by the function (or algorithm) of the rotation map of equipment 100 to pixel displacement
The resolution of the image that the parameter (focal length of such as video camera 102 and camera lens width) of camera 102 and video camera 102 capture
Rate.Encoder feedback is determined for the precision of the sampling that gyroscope 110 produces, and for adaptive mapping algorithm.Additionally,
Also have some about motion and situations of object displacement, the most described herein based on the sampling from gyroscope 110 stablize
(such as the rotation around the video camera 102 of user's face, user's face is likely in frame for model inaccuracy
Between be stable, but gyroscope 110 will detect that rotation, thus, described stabilized treatment will attempt Steady Background Light), this
A little can be detected by encoder and feed back to stable algorithm.Stable algorithm can be carried out adaptation in this way.
The pixel displacement determined in step S308 represents the video signal two field picture that the motion of video camera 102 results in
In motion amplitude (with the sports of image subject in contrast).In this way, the pixel position determined in step S308
Move the undesirable high frequency motion (vibrations) in the image representing video signal frame.
Back with reference to Fig. 2, after monitoring step S202, in step S204, the motion of video camera 102 caused
Motion amplitude in the image of video signal frame will be compared with predetermined threshold, in order to determine that whether equipment 100 is in vibrations.
If determining equipment 100 not vibrations in step S204, then this process reversely advances to monitor step S202,
Prune thus without to video signal, so that video signal keeps maximum output video resolution.Once make this to determine,
Then the intervalometer (the most not display) realized in device 100 is resetted.
In step S204, if it is determined that equipment 100 is in vibrations, then this process advances to step S206.In step S206
Middle can determine whether starts whether to have passed through a period when last reset intervalometer.
If not yet through the described period, then this process reversely advances to monitor step S202.If having already been through institute
State the period, then this process advances to stabilizing step S208.
Hereinafter will be more fully described stabilizing step S208 with reference to Fig. 6.
In step S602, the pixel displacement determined in step S308 will be filtered.Perform this process to smooth that
Elapse in time in image stabilization processes a bit and be applied to the change of video signal, thus provide through stabilized treatment more
Smooth video signal.Wave filter for being filtered pixel displacement can design in different ways, and such as, it sets
Meter mode can depend on the resolution of the image that video camera 102 captures, and can be applied to the acceptable delay of video signal, with
And the license pruning rate of the raw video signal image received from video camera 102 can be applied on preprocessor.As an example
Son, in processing in image stabilization for the high frequency change of the pixel displacement of application, the frame of video that resolution is higher can be from it
Bigger algorithm is benefited.On the other hand, pruning rate can arrange a hard restriction for maximum algorithm.
Can use exponential filter pixel displacement being filtered according to below equation:
X_filt (n)=(1-w) * x_filt (n-1)+w*x (n),
Wherein n represents the frame number of video signal, and x represents the accumulation displacement according to the pixel displacement determined in step S308
(or " position "), and x_filt represents the accumulation displacement that filtered, the wherein said accumulation displacement filtered subsequently will by with
In determining how calibration input picture, in order to stablized as described in greater detail below.In this way, institute
State wave filter and act as an exponential filter.When the movement stops, x_filt-x will converge to zero, it means that image does not has
There is movement.Described wave filter is by by the corresponding filtered pixel displacement the most of previous frame and be present frame in step S308
The pixel displacement determined is as the basis of filtered pixel displacement, in order to those have elapsed in time to eliminate (smooth out)
Determine the change of pixel displacement.The weighting of the displacement of filtered pixel being applied to previous frame is (1-w), and is applied to as currently
The weighting of the pixel displacement that frame determines is then w.Thus, adjust weighting parameters w and will adjust wave filter to pixel displacement the most (x)
In response for changing.Compared with finite impulse response (FIR) wave filter, output x_filt is being pruned and makes at it
When scope [x-crop, x+crop], be more suitable for is recurrence (infinite-duration impulse response (IIR)) wave filter, because after Xiu Jianing
Value will be fed back to filter loop, and follow-up x_filt output can be made to be not susceptible to prune.
Weighting parameters w is adapted to resolution and the i.e. frame rate of video signal, in order to the perseverance that acquisition hertz is measured
Earnest reason cut-off frequency.If wave filter is ideal filter, then physics cut-off frequency will define and be introduced in x_filt
In the highest frequency component of change of x.The change of the x with frequency more higher than cut-off frequency will be declined by ideal filter
Subtract, and will not be in appearance in x_filt.But, this wave filter is not ideal filter, and just because of this, cut-off frequency
Defining such highest frequency, the decay applied by wave filter for it is less than 3dB.Therefore, nonideal wave filter is come
Say, exist below some decay at cut-off frequency, and more than cut-off frequency, also there is no complete attenuation.Wave filter output will
It is trimmed to about, so that the difference between x_filt and x is not more than frame and prunes size.W is adapted to cause the physics cut-off frequency be
Constant, such as 0.5Hz.A function w(fc, fs can be derived from filter transfer function), described function can be by thing
Reason cut-off frequency fc is mapped to w.When sample frequency (frame rate) fs changes, even if fc is constant, w also can change.Compare
In other wave filter, it is especially suitable for the cut-off frequency (changing w) immediately changed according to the wave filter of above-mentioned filtering equation.
In step S604, by using the displacement of filtered pixel from step S602 to move second frame (frame 2)
Image.In this way, in the image of second frame (relative to first frame) caused due to the motion of video camera 102
Motion will be attenuated.In other words, the pixel displacement filtered will be used to compensate for first that camera motion is caused
The motion in video signal between frame and second frame, thus stablizes described video signal.
The pixel displacement filtered is changed whole (round) and is helped pixel displacement (i.e. the displacement of integer pixel).Do so
Allow the image using simple method to move second frame.This image is the stride of the storage space with instruction image
(stride) value, multiple pixel values, the pointer of the position of first pixel of instruction image, and the width of instruction picture traverse
Value represents.The movement of image includes: adjust pointer and width value in the case of uncomfortable synchronizing amplitude.It can be seen that
Width value is unrelated with stride value, the most then allow to change picture traverse in the case of not affecting image stride.Thus, exist
The when that image being moved (and/or readjusting size), the storage space (such as in memorizer 108) of image is to be not required to
To be changed.This means described method to need not in memorizer 108 and copy data.This copies with by the pruning region of image
Normal image pruning method to new area of memory defines contrast.It is the most complicated that region pruned by copy,
This process is probably harmful, particularly when CPU 106 can process resource may realize in limited mobile device
The when of described method.Utilize method described herein, owing to width value is independent of stride value, by change pointer and width
Keeping stride constant, image new, that be moved through can be created.
Image can be represented by multiple planes of delineation, such as luminance plane (Y) and two colorimetric planes (U and V).Pass through
Simply change sensing brightness and the pointer of colorimetric plane, the plane of delineation of input picture can be moved and readjust this figure
The size of image plane, thus revises the width of the plane of delineation while keeping stride constant.The plane of delineation is then by mobile identical
Amount, in order to guaranteeing can be by the plane of delineation being moved through together for representing the image of movement.
Moving process to realize this image, the plane of delineation needs corresponding pointer, say, that they can not be all
Represent with same pointer.In addition as it has been described above, described image must have independent width and stride value.
Fig. 7 is the expression of the image before and after movement and pruning modes.Original image represents with 702, and through moving
Image that is dynamic and that prune represents with 704.It can be seen that the stride value of image keeps constant, the width of image is then
It is reduced.Additionally, original pointer points to the top left corner pixel of original image, the pointer being adjusted then point to by mobile and
The upper left corner (it is in the position different from the top left corner pixel of original image) of the image pruned.In this way, simply
Ground changes width value and pointer i.e. may move and prunes described image.
As the summary of said method, before with video encoder encoded video signal, achieve in preprocessor
The image of video signal frame is stablized with the next stage:
1. estimate the video camera 102 angular displacement (step S306) between frame 1 and frame 2；
2. estimated angular displacement is mapped to the pixel displacement (step S308) of the image of frame 2；
3. remove frame 2 by wave filter being applied to pixel displacement sequence (or as above accumulation pixel displacement)
Image in unintended movements (step S602)；And
4. the position calculated by image moves to wave filter creates the stable image (step 604) of frame 2.For frame
The described frame dimension respective dimensions equal to or less than the original image of frame 2 stablizing image of 2.In other words, video signal
Stablize image be by cut away video camera 102 capture video signal original image within moving boundary and be constructed.
Back with reference to Fig. 2, after stabilizing step S208, this process advances to step S210.
In step S210, wherein will consider the next frame captured in step s 302, in order to determine equipment 100 whether
It is still in vibrations.It is to say, the angular displacement that video camera 102 is between two frames (frame 2 and frame 3) will be determined and be mapped to picture
Element displacement, wherein said pixel displacement represents the motion in the image of the video signal frame that the motion of video camera 102 is caused.?
Above by reference to step S306 and S308, this is described in detail.
In step S210, caused by the motion of video camera 102, fortune after filtering in the image of video signal frame
Dynamic amplitude will be compared with predetermined threshold, in order to determine whether equipment 100 is still in vibrations.
In step S210, if it is determined that equipment 100 is still in vibrations, then this process returns to step S208.In step
S208, removes the figure of frame 3 by wave filter is applied to pixel displacement sequence (or accumulation pixel displacement as above)
Unintended movements in Xiang, and the stable image of frame 3 is created by image being moved to the position of wave filter calculating,
The most this is described in detail.Once perform this operation, then the intervalometer that realizes on the appliance 100 (
Not display in Fig. 1 b) will be reset.
In step S210, if it is determined that equipment 100 does not shake, then this process advances to step S212, in this step
Middle can determine whether starts whether to have already been through a period when last reset intervalometer.
In step S212, if it is determined that not yet through this period, then this process returns to stabilizing step S208, and at this
The position that can calculate by image moves to wave filter in step creates the stable image of frame 3.
In step S212, if it is determined that have already been through this period, then this process returns to monitor step S208, thus
Frame 3 is not realized pruning modes.
Will be appreciated that by using timeout value (period that definition is above mentioned) to change in step S206 and S210
It is apt to the seriality of the video signal transmitted.
The predetermined threshold being used for opening stabilized treatment in step S204 is by by acceleration change and a threshold value phase
Comparison (after the filtering) calculates.In step S210, it is by by pixel for closing the predetermined threshold of stabilized treatment
Motion vector and pixel threshold value in each direction is compared and is relatively calculated.
Will be appreciated that above-described threshold value and time-out can be specific for implementation, depend on sensor
Data accuracy and the complexity of required change resolution.
Will be appreciated that above-mentioned process will persistent loop, in order to consider whether each frame of video camera 102 capture needs
Stablize.Only needs when, enabling stable processes, the most only to use the process described above with reference to Fig. 2 ensure that
Needs when, just prune video signal, thus keep the ultimate resolution of video signal.
In the above-described embodiments, motion sensor 110 is the gyro of the sampling producing the rotary motion representing equipment 100
Instrument.In other embodiments, motion sensor 110 can sense other kinds of motion, such as translational motion, and can produce
The sampling of the raw translational motion representing equipment 100.These samplings can be in the way of using and describing above in conjunction with rotary motion
Identical mode stablizes video signal.However as described above, for translational motion, pixel displacement will depend upon which image
In the distance of object, therefore, it must be accounted for determining pixel displacement when.Give an example, multiple acceleration
Meter can estimate rotary motion, and in this case, described accelerometer can in the case of not making other amendments quilt
Use.For more generally translating stabilized treatment, owing to the zones of different in image moves different pixel total amounts, because of
This, it is achieved method described herein may become increasingly difficult.But, if the distance to object is constant (and known)
, then described method is realized for translational motion and would is that simply.Even the distance to object be not constant (but
It is still known), it is also possible to described method is realized for translational motion, but is determining that video camera 102 translational motion is caused
Pixel displacement during will add extra complexity.
After stabilizing video signal, Video coding is used to process encoded video signal.For example, encoded
Video signal by the part as the video call for another user or can transmit as broadcast singal.By
This, for video signal, it is of great importance that (namely can postponed the least situation by real-time stabilization and coding
Under), in order to the communication thing that the perception of signal delay is perfectly clear by event or other those users at video call etc
Part uses.Alternatively, encoded video signal can preserve on the appliance 100, such as, be saved in memorizer 108
In.
Method step S200-S212 can realize with software or hardware on the appliance 100.Such as, CPU 106 can run
Processing component realizes step S200-S212.Give an example, can be carried for stablizing the computer program of video signal
Confession, runs during wherein this product can be saved in memorizer 108 and by CPU 106.Described computer program product micromicro is to be joined
It is set to when running on CPU 106 when perform method step S200-S212.Alternatively, it is possible to realize in device 100
Hardware component realizes step S200-S212.
Although additionally, be particularly shown and describe the present invention with reference to preferred embodiment, but those skilled in the art should
Understand, the amendment in the case of the scope of the present invention defined without departing from claims, in terms of various forms and details
It is all feasible.
Claims (17)
1. the method transmitting the video signal from subscriber equipment, the method includes:
Video camera is used to capture multiple video signal frame on a user device；
The motion sensor being associated with video camera is used to produce the multiple samplings representing described camera motion；
Described sampling is used to determine the displacement of described video camera between the successive frame that described video camera captures；
Determine motion in the video signal represented between the successive frame that caused by displacement determined by described video camera
Pixel displacement；
Described pixel displacement is compared with threshold value；
If described pixel displacement exceeds described threshold value, then video signal described in real-time stabilization, stablize described video signal and include:
In the following way described pixel displacement is filtered: be based upon pixel determined by second frame of described successive frame
Displacement determines accumulation pixel displacement；And be based upon accumulation pixel displacement that second frame determine with for the of described successive frame
The weighted sum of the accumulation pixel displacement filtered of one frame determines the accumulation pixel position filtered for second frame
Move；And
The figure of at least one in first frame of described successive frame and second frame is moved according to the pixel displacement filtered
Picture.
Method the most according to claim 1, the motion of wherein said video camera is rotary motion, and wherein said motion sensor is
Rotational motion sensor, and the displacement of wherein said video camera is the angular displacement of described video camera.
Method the most according to claim 1, wherein said subscriber equipment includes forward direction video camera and backward video camera.
Method the most according to claim 1, wherein uses the sample rate of described sampling produced by described motion sensor to be higher than
The frame rate of described video signal.
Method the most according to claim 1, wherein said subscriber equipment includes mobile device, and described video camera and described fortune
Dynamic sensor is placed in described mobile device.
Method the most according to claim 1, also includes adding time migration to multiple frames captured and produced multiple
At least one in sampling, so that the timing of the timing of the multiple frames captured and produced multiple samplings matches.
Method the most according to claim 1, also includes: if described pixel displacement is beyond described threshold value, then use intervalometer
Determine whether to have already been through the predetermined period, and only just stable having already been through this period when described in regard
Frequently signal.
Method the most according to claim 7, also includes: if described pixel displacement is not above described threshold value, then reset described
Intervalometer.
9., for stablizing an equipment for video signal, this equipment includes:
It is configured to capture the video camera of multiple frames of video signal；
It is associated with described video camera and is configured to generate the motion-sensing of multiple samplings of the motion representing described video camera
Device；
It is configured to use described sampling and determines the pixel position of described video camera between the successive frame that described video camera captures
The pixel displacement moved determines block, and described pixel displacement represents and regards described between the successive frame caused by described camera motion
Frequently the motion in signal；
It is configured to the comparison block described pixel displacement and threshold value compared；
Be configured by stablize in the case of exceeding described threshold value at described pixel displacement described video signal stablize described in regard
Frequently the motion compensation block of signal, stablizes described video signal and includes:
In the following way described pixel displacement is filtered: be based upon pixel determined by second frame of described successive frame
Displacement determines accumulation pixel displacement；And be based upon accumulation pixel displacement that second frame determine with for the of described successive frame
The weighted sum of the accumulation pixel displacement filtered of one frame determines the accumulation pixel position filtered for second frame
Move；And
The figure of at least one in first frame of described successive frame and second frame is moved according to the pixel displacement filtered
Picture.
10., for transmitting an equipment for the video signal from subscriber equipment, this equipment includes:
For using video camera to capture the device of multiple video signal frame on a user device；
For using the motion sensor being associated with video camera to produce the device of the multiple samplings representing camera motion；
For using described sampling to determine the device of the displacement of described video camera between the successive frame that described video camera captures；
For determining the fortune in the video signal represented between the successive frame caused by displacement determined by described video camera
The device of dynamic pixel displacement；
For the device that described pixel displacement and threshold value are compared；
For at described pixel displacement beyond the device of then video signal described in real-time stabilization in the case of described threshold value, for steady
The device of fixed described video signal includes:
Device in the following way described pixel displacement being filtered: be based upon second frame institute of described successive frame
The pixel displacement determined determines accumulation pixel displacement；And be based upon accumulation pixel displacement that second frame determine with for described
The weighted sum of the accumulation pixel displacement filtered of first frame of successive frame determines having filtered for second frame
Accumulation pixel displacement；And
At least one in first frame moving described successive frame according to the pixel displacement filtered and second frame
The device of image.
11. equipment according to claim 10, the motion of wherein said video camera is rotary motion, wherein said motion sensor
It is rotational motion sensor, and the displacement of wherein said video camera is the angular displacement of described video camera.
12. equipment according to claim 10, wherein said subscriber equipment includes forward direction video camera and backward video camera.
13. equipment according to claim 10, the sample rate wherein using described sampling produced by described motion sensor is high
Frame rate in described video signal.
14. equipment according to claim 10, wherein said subscriber equipment includes mobile device, and described video camera and described
Motion sensor is placed in described mobile device.
15. equipment according to claim 10, also include for time migration being added to the multiple frames captured and being produced
Multiple samplings at least one so that the timing of the timing of the multiple frames captured and produced multiple samplings is mutually
The device joined.
16. equipment according to claim 10, also include: if for pixel displacement beyond described threshold value, then using intervalometer
Determine whether to have already been through the predetermined period, and only just stable described having already been through this period when
The device of video signal.
17. equipment according to claim 16, also include in the case of being not above described threshold value at described pixel displacement
Reset the device of described intervalometer.
Applications Claiming Priority (4)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| GB1116566.9 | 2011-09-26 | ||
| GBGB1116566.9A GB201116566D0 (en) | 2011-09-26 | 2011-09-26 | Video stabilisation |
| US13/307,800 US8723966B2 (en) | 2011-09-26 | 2011-11-30 | Video stabilization |
| US13/307800 | 2011-11-30 |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102957869A CN102957869A (en) | 2013-03-06 |
| CN102957869B true CN102957869B (en) | 2016-08-17 |
Family
ID=47143266
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210363053.0A Active CN102957869B (en) | 2011-09-26 | 2012-09-26 | Video stabilization |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102957869B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR101613176B1 (en) * | 2013-03-18 | 2016-04-29 | 포토내이션 리미티드 | A method and apparatus for motion estimation |
| CN105306804B (en) * | 2014-07-31 | 2018-08-21 | 北京展讯高科通信技术有限公司 | Intelligent terminal and its video image stabilization method and device |
| US10708571B2 (en) * | 2015-06-29 | 2020-07-07 | Microsoft Technology Licensing, Llc | Video frame processing |
| JP6667268B2 (en) * | 2015-11-26 | 2020-03-18 | キヤノン株式会社 | Motion vector detection device, control method therefor, and imaging device |
| CN107743190A (en) * | 2016-08-10 | 2018-02-27 | 成都鼎桥通信技术有限公司 | Video anti-fluttering method |
| CN109711421A (en) * | 2017-10-25 | 2019-05-03 | 腾讯科技（深圳）有限公司 | A kind of data processing method and device |
| EP3883234B1 (en) * | 2020-03-17 | 2022-02-02 | Axis AB | Wearable camera and a method for power consumption optimization in the wearable camera |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US4864409A (en) * | 1986-10-09 | 1989-09-05 | Deutsche Thomson-Brandt Gmbh | Television camera acceleration compensation apparatus |
| EP1377040A1 (en) * | 2002-06-19 | 2004-01-02 | STMicroelectronics S.r.l. | Method of stabilizing an image sequence |
| CN1819626A (en) * | 2004-12-28 | 2006-08-16 | 精工爱普生株式会社 | Imaging apparatus and portable device and portable telephone using the same |
| CN101897174A (en) * | 2007-12-14 | 2010-11-24 | 三洋电机株式会社 | Imaging device and image reproduction device |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7880769B2 (en) * | 2004-02-13 | 2011-02-01 | Qualcomm Incorporated | Adaptive image stabilization |
| US7693405B2 (en) * | 2005-11-25 | 2010-04-06 | Seiko Epson Corporation | Image pickup device, method of controlling image pickup device, and recording medium |
-
2012
- 2012-09-26 CN CN201210363053.0A patent/CN102957869B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US4864409A (en) * | 1986-10-09 | 1989-09-05 | Deutsche Thomson-Brandt Gmbh | Television camera acceleration compensation apparatus |
| EP1377040A1 (en) * | 2002-06-19 | 2004-01-02 | STMicroelectronics S.r.l. | Method of stabilizing an image sequence |
| CN1819626A (en) * | 2004-12-28 | 2006-08-16 | 精工爱普生株式会社 | Imaging apparatus and portable device and portable telephone using the same |
| CN101897174A (en) * | 2007-12-14 | 2010-11-24 | 三洋电机株式会社 | Imaging device and image reproduction device |
Non-Patent Citations (1)
| Title |
|---|
| Using sensor for Efficient Video Coding in Hand-held Device;Andy L. Lin等;《http://scien.stanford.edu/pages/labsite/2010/ee398a/projects/reports/andy.pdf》;20100312;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102957869A (en) | 2013-03-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102957869B (en) | Video stabilization | |
| EP2742680B1 (en) | Video stabilisation | |
| CN106101566B (en) | Video stabilization | |
| US9762799B2 (en) | Received video stabilization | |
| EP3449624B1 (en) | Electronic image stabilization frequency estimator | |
| WO2012132168A1 (en) | Information processing device, information processing method, and data structure of location information | |
| US8941743B2 (en) | Preventing motion artifacts by intelligently disabling video stabilization | |
| Bell et al. | A non-linear filter for gyroscope-based video stabilization | |
| US10911675B2 (en) | Method for providing shake correction, signal processing device performing the method, and imaging device including the signal processing device | |
| EP2898665B1 (en) | Preventing motion artifacts by intelligently disabling video stabilization | |
| CN115706856A (en) | Video anti-shake method and device, electronic equipment and storage medium | |
| JP2015061252A (en) | Motion detection device and motion detection program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20200330 Address after: Redmond, Washington, USA Patentee after: MICROSOFT TECHNOLOGY LICENSING, LLC Address before: Ai Erlandubailin Patentee before: Skype |
|
| TR01 | Transfer of patent right |