CN113014823A - Anti-shake processing method, system, device and storage medium for image pickup apparatus - Google Patents
Anti-shake processing method, system, device and storage medium for image pickup apparatus Download PDFInfo
- Publication number
- CN113014823A CN113014823A CN202110321357.XA CN202110321357A CN113014823A CN 113014823 A CN113014823 A CN 113014823A CN 202110321357 A CN202110321357 A CN 202110321357A CN 113014823 A CN113014823 A CN 113014823A
- Authority
- CN
- China
- Prior art keywords
- image
- processed
- shooting
- time
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
- H04N23/681—Motion detection
- H04N23/6815—Motion detection by distinguishing pan or tilt from motion
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Studio Devices (AREA)
Abstract
The invention provides an anti-shake processing method, system, equipment and storage medium of a camera device, wherein the method comprises the following steps: acquiring a video frame to be processed as an image to be processed; acquiring motion data of the camera device corresponding to the shooting time of the image to be processed; calculating position transformation information and/or cutting information of the image to be processed according to the motion data; and rotating and/or cutting the image to be processed according to the position conversion information and/or the cutting information to obtain a processed video frame. According to the anti-shake processing method of the camera device, the video frame is extracted to serve as the image to be processed, the motion data is obtained, the position transformation information and/or the cutting information are/is calculated based on the motion data, and the image is rotated and/or cut based on the information obtained through calculation, so that the anti-shake processing of the video frame to be processed based on the motion data of the camera device is realized, and the video shooting effect is improved.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to an anti-shake processing method, system, device, and storage medium for an image capturing apparatus.
Background
The law enforcement instrument is also called video and audio law enforcement instrument for police or field law enforcement instrument, integrates the functions of digital camera shooting, digital camera shooting and interphone transmitter, can carry out digital record on dynamic and static field conditions in the law enforcement process, and is convenient for law enforcement in various environments. When the law enforcement instrument collects videos or video frames, the shooting effect can be affected by jitter.
Disclosure of Invention
In view of the problems in the prior art, an object of the present invention is to provide an anti-shake processing method, system, device and storage medium for an image capturing apparatus, which perform anti-shake processing on a video frame to be processed based on motion data of the image capturing apparatus, so as to improve a video capturing effect.
The embodiment of the invention provides an anti-shake processing method of a camera device, which comprises the following steps:
acquiring a video frame to be processed as an image to be processed;
acquiring motion data of the camera device corresponding to the shooting time of the image to be processed;
calculating position transformation information and/or cutting information of the image to be processed according to the motion data;
and rotating and/or cutting the image to be processed according to the position conversion information and/or the cutting information to obtain a processed video frame.
According to the anti-shake processing method of the camera device, the video frame is extracted to serve as the image to be processed, the motion data is obtained, the position transformation information and/or the cutting information are/is calculated based on the motion data, and the image is rotated and/or cut based on the information obtained through calculation, so that the anti-shake processing of the video frame to be processed based on the motion data of the camera device is realized, and the video shooting effect is improved.
In some embodiments, acquiring motion data of the image capturing device corresponding to the capturing time of the image to be processed includes the steps of:
acquiring the shooting start time and the shooting end time of the image to be processed;
dividing the image to be processed into a plurality of line blocks, wherein each line block comprises at least one continuous image line;
calculating the shooting time of each line block according to the shooting start time and the shooting end time of the image to be processed;
and acquiring motion data of the camera corresponding to the shooting time of each line block.
In some embodiments, calculating the photographing time for each line block according to the photographing start time and the photographing end time of the image to be processed includes the steps of:
calculating the shooting time delta t-tb according to the shooting start time ta and the shooting end time tb of the image to be processed;
calculating the average shooting time consumption delta t/n of each line block, wherein n is the total number of the line blocks;
and calculating the shooting start time tc ═ ta + (i-1) × Δ t/n of the ith line block, the shooting end time td ═ ta + i × Δ t/n, and the shooting time of each line block comprises the shooting start time tc and the shooting end time td.
In some embodiments, acquiring motion data of the image capturing device corresponding to the capturing time of the image to be processed includes the steps of:
acquiring rotation data acquired by a gyroscope of the camera device according to the shooting time of the image to be processed, wherein the shooting time of the image to be processed comprises shooting start time and shooting end time;
and converting the rotation data acquired by the gyroscope into a motion matrix of the camera device, and using the motion matrix as the motion data of the shooting time of the image to be processed.
In some embodiments, calculating the position transformation information of the image to be processed according to the motion data comprises the following steps:
acquiring a reference video frame as a reference image;
calculating a transformation matrix for transforming the position of one point in the image to be processed to the position of the same point in the reference image according to the motion matrix, and taking the transformation matrix of each point in the image to be processed as position transformation information of the image to be processed;
and performing rotation processing on the image to be processed according to the position transformation information, and the method comprises the following steps:
and transforming the positions of all points in the image to be processed according to the transformation matrix.
In some embodiments, the reference video frame and the to-be-processed video frame are two consecutive video frames, and calculating a transformation matrix for transforming a position of a point in the to-be-processed image to a position of the same point in the reference image includes calculating the transformation matrix by using the following formula:
W(t1，t2)＝KR′(t1)RT(t2)K-1。
wherein, W (t)1，t2) A transformation matrix representing the transformation of the position of a point in the image to be processed taken at the moment t2 to the position of the same point in the reference image taken at the moment t1, K being the camera eigen matrix of the image capturing device, K-1Is the inverse of the camera eigen matrix, R' (t)1) Is the product of the x-, y-and z-axis motion matrices of the camera at time t1, RT(t2) The transpose of the product of the x-axis, y-axis and z-axis motion matrices for the camera at time t 2.
In some embodiments, calculating cropping information for the image to be processed from the motion data comprises the steps of:
calculating a scaling coefficient of the image to be processed according to the motion data, and taking the scaling coefficient as clipping information of the image to be processed;
and cutting the image to be processed according to the cutting information, comprising the following steps:
and zooming the image to be processed according to the zooming coefficient, and removing the part of the zoomed image which exceeds the range of a preset image.
In some embodiments, calculating the scaling factor of the image to be processed from the motion data comprises calculating the scaling factor of the image to be processed using the following formula:
α′＝Kpxωx+Kpyωy+Kpzωz+Kix∑ωx+Kiy∑ωy+Kiz∑ωz+Kdx(ωx-ωx′)+Kdy(ωy-ωy′)+Kdz(ωz-ωz′)
wherein α is a scaling coefficient of the image to be processed, α' is ωx、ωy、ωzRepresenting the values of the motion error, ω, in the x, y and z directions, respectively, at time tx′、ωy′、ωz′Respectively representing the values of the motion error, K, in the x, y and z directions at time t + Deltatpx、Kpy、KpzRespectively representing proportional gains in the x, y and z directions, Kix、Kiy、KizDenotes the integral gain in the x, y and z directions, respectively, Kdx、Kdy、KdzRespectively representing differential gains in the x, y and z directions, alphamax、αminRespectively representing a preset maximum scaling coefficient value and a preset minimum scaling coefficient value.
An embodiment of the present invention further provides an anti-shake processing system for an image capturing apparatus, which is applied to the anti-shake processing method for the image capturing apparatus, and the system includes:
the image acquisition module is used for acquiring a video frame to be processed as an image to be processed;
the motion acquisition module is used for acquiring motion data of the shooting time of the image to be processed corresponding to the image pickup device;
the information calculation module is used for calculating the position transformation information and/or the cutting information of the image to be processed according to the motion data;
and the image processing module is used for rotating and/or cutting the image to be processed according to the position transformation information and/or the cutting information to obtain a processed video frame.
According to the anti-shake processing system of the camera device, the video frame is extracted through the image acquisition module to serve as the image to be processed, the motion data is acquired through the motion acquisition module, the position transformation information and/or the cutting information are calculated through the information calculation module based on the motion data, and the image is rotated and/or cut through the image processing module based on the information obtained through calculation, so that the anti-shake processing of the video frame to be processed based on the motion data of the camera device is realized, and the video shooting effect is improved.
An embodiment of the present invention further provides an anti-shake processing apparatus for an image pickup device, including:
a processor;
a memory having stored therein executable instructions of the processor;
wherein the processor is configured to execute the steps of the anti-shake processing method of the image pickup apparatus via execution of the executable instructions.
By adopting the anti-shake processing device of the image pickup apparatus provided by the invention, the processor executes the anti-shake processing method of the image pickup apparatus when executing the executable instruction, thereby being capable of obtaining the beneficial effects of the anti-shake processing method of the image pickup apparatus.
An embodiment of the present invention further provides a computer-readable storage medium, which is used for storing a program, and when the program is executed by a processor, the method for processing an image capture device according to the present invention is implemented.
By adopting the computer-readable storage medium provided by the invention, the stored program realizes the steps of the anti-shake processing method of the image pickup apparatus when being executed, thereby the beneficial effects of the anti-shake processing method of the image pickup apparatus can be obtained.
Drawings
Other features, objects and advantages of the present invention will become more apparent upon reading of the following detailed description of non-limiting embodiments thereof, with reference to the accompanying drawings.
Fig. 1 is a flowchart of an anti-shake processing method of an image capturing apparatus according to an embodiment of the present invention;
FIG. 2 is a schematic diagram comparing a law enforcement instrument device and a cell phone coordinate system according to an embodiment of the present invention;
FIG. 3 is a diagram of an embodiment of the present invention for dividing an image to be processed into a plurality of line blocks;
fig. 4 is a schematic diagram of an anti-shake processing system of an image capturing apparatus according to an embodiment of the present invention;
fig. 5 is a schematic structural view of an anti-shake processing apparatus of an image pickup device according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of a computer storage medium according to an embodiment of the present invention.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar structures, and thus their repetitive description will be omitted.
As shown in fig. 1, in an embodiment, the present invention provides an anti-shake processing method for an image capturing apparatus, including the steps of:
s100: acquiring a video frame to be processed as an image to be processed;
s200: acquiring motion data of the camera device corresponding to the shooting time of the image to be processed;
s300: calculating position transformation information and/or cutting information of the image to be processed according to the motion data;
s400: and rotating and/or cutting the image to be processed according to the position conversion information and/or the cutting information to obtain a processed video frame.
According to the anti-shake processing method of the camera device, the video frame is extracted as the image to be processed through the step S100, the motion data is obtained through the step S200, the position transformation information and/or the cutting information are calculated based on the motion data through the step S300, and the image is rotated and/or cut based on the calculated information through the step S400, so that the anti-shake processing of the video frame to be processed based on the motion data of the camera device is realized, and the video shooting effect is improved.
This embodiment will be described by taking the imaging apparatus as a law enforcement instrument. It is understood that the present invention is not limited thereto, and the anti-shake processing method of the present invention can also be applied to other types of image capturing devices, such as mobile phones, cameras, etc., and all of them fall within the protection scope of the present invention. The video recording process of the law enforcement instrument generally includes several parts of video capture, ISP (Image Signal Processing) effect Processing, and Image output. The invention further adds anti-shake processing between ISP effect processing and image output, thereby realizing stable image output.
The anti-shake Processing method of the image pickup apparatus of the present invention can be realized by using a motion sensor, a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). Specifically, the motion sensor may be used to collect motion data of the camera, the CPU may be used to perform steps S100 to S300, and the GPU may be used to perform step S400, but the present invention is not limited thereto.
In this embodiment, the step S100: and acquiring a video frame to be processed as an image to be processed, wherein the video frame to be processed is extracted from the video shot by the camera device and is used as the image to be processed.
In this embodiment, the image pickup apparatus is an image pickup apparatus using a Rolling Shutter (Rolling Shutter), that is, an image is exposed line by line at the time of shooting. The shooting time of each video frame includes a shooting start time and a shooting end time of the video frame.
In order to reduce image distortion due to the inconsistency of the time stamps of the exposures of the respective lines, it is necessary to acquire corresponding motion data for the respective lines. However, since the amount of calculation for each line is too large, the image is divided into a plurality of line blocks, each line block includes at least one continuous image line, and the average time is calculated as the time stamp of the line block. The calculation result of this step is the time t corresponding to the motion data acquired by each row of blocks.
Specifically, the step S200: the method for acquiring the motion data of the camera device corresponding to the shooting time of the image to be processed comprises the following steps:
acquiring the shooting start time and the shooting end time of the image to be processed;
dividing the image to be processed into a plurality of line blocks, wherein each line block comprises at least one continuous image line;
calculating the shooting time of each line block according to the shooting start time and the shooting end time of the image to be processed;
and acquiring motion data of the camera corresponding to the shooting time of each line block.
When acquiring motion data in units of line blocks, the position conversion information and/or cropping information of the image is acquired in subsequent steps S300 and S400, and the image is rotated and/or cropped in units of line blocks.
In this embodiment, calculating the photographing time for each line block according to the photographing start time and the photographing end time of the image to be processed includes the steps of:
calculating the shooting time delta t-tb according to the shooting start time ta and the shooting end time tb of the image to be processed;
calculating the average shooting time consumption delta t/n of each line block, wherein n is the total number of the line blocks;
and calculating the shooting start time tc ═ ta + (i-1) × Δ t/n of the ith line block, the shooting end time td ═ ta + i × Δ t/n, and the shooting time of each line block comprises the shooting start time tc and the shooting end time td.
For example, as shown in fig. 3, the image is divided into 10 line blocks, a time stamp is calculated for each line block, the motion data of the corresponding motion sensor is queried according to the time stamp, and the position conversion information and/or the cropping information are/is calculated, so that the phenomenon that the calibrated image has a serious jelly effect due to the deformation of the rolling shutter can be effectively prevented. In fig. 3, frame represents a video frame, time represents time, and gyro sampler represents a data acquisition time range of a motion sensor corresponding to each line block, and the motion sensor is taken as a gyroscope for example.
In the steps S200 to S400, the motion data of the motion sensor is captured according to the time stamp of each line block, and the motion data is filtered and the accumulated error is corrected. As shown in fig. 2, the law enforcement instrument is rotated by 90 degrees compared with a general mobile phone on the installation of the image sensor, so that the coordinate system of the image sensor is synchronized by performing a coordinate system conversion before the motion data filtering process of the motion sensor. I.e. first a camera model matrix is built from camera parameters.
Specifically, the camera model is introduced as follows:
x＝KX，and X＝λK-1x
wherein, X is the position of the point X on the image in the world coordinate system, K is the camera inherent matrix of the image pickup device, and λ is a scale factor, which can also be regarded as a scaling factor, and is a constant. With respect to a specific camera model. K-1Being the inverse of the camera eigen matrix, can be described by:
wherein (o)x，oy) Is the origin of the xy axis of the camera in the image plane and f is the input equivalent focal length, where 1 on the diagonal is based on the assumption that the image is square.
For the convenience of subsequent calculation, K is defined here-1The 3x3 matrix is expanded into a 4x4 matrix as follows:
in this embodiment, the motion sensor is a gyroscope. The step S200: the method for acquiring the motion data of the camera device corresponding to the shooting time of the image to be processed comprises the following steps:
acquiring rotation data acquired by a gyroscope of the camera device according to the shooting time of the image to be processed, wherein the shooting time of the image to be processed comprises shooting start time and shooting end time, and when the rotation data acquired by the gyroscope is acquired, the rotation data of the gyroscope in a time range between the shooting start time and the shooting end time of the image to be processed is acquired;
when an image to be processed is divided into a plurality of line blocks, acquiring rotation data of a gyroscope in a time range between a shooting start time and a shooting end time of each line block for each line block;
the rotation data collected by the gyroscope includes position conversion information in three directions, expressed in euler angles, that is, includes
Theta, psi. Euler angle is a set of 3 independent angle parameters for determining the position of fixed-point rotating rigid body, consisting of nutation angle theta, precession angle psi and rotation angle
And (4) forming.
And calculating a motion matrix R (t) of x, y and z axes according to the rotation data acquired by the gyroscope, wherein the motion matrix R (t) is used as the motion data of the shooting time of the image to be processed. The x-axis motion matrix Rx (t) is as follows:
the motion matrix in the y-axis direction ry (t) is as follows:
the motion matrix rz (t) in the z-axis direction is as follows:
in this embodiment, the step S300 of calculating the position transformation information of the image to be processed according to the motion data includes the following steps:
acquiring a reference video frame as a reference image, wherein in the embodiment, the reference video frame and the to-be-processed video frame are two continuous video frames, for example, the reference image is a jth frame video frame, and the to-be-processed image is an ith frame video frame;
with the camera eigen matrix and motion matrix described above, the same point in the ith and jth frames can be represented as follows:
xi＝KR(t(i，yi))X，and xj＝KR(t(j，yj) X, here yiAnd yjRespectively representing the ith line block of the ith frame and the jth line block of the jth frame, and in an embodiment, the image is divided into 10 line blocks, and the numerical range of y is 1-10.
t () represents a unit of time, t (i, y)i) Denotes the exposure time of the ith frame, the y-th line block, R (t (i, y)i) I.e. the motion matrix for the exposure time, t (j, y)j) Denotes the exposure time of the jth frame, the y row block, R (t (j, y)j) I.e. the motion matrix for the exposure time.
That is, for a point X in space, the coordinates of the ith frame and the jth frame on the picture are X respectivelyi，xjWherein x isiAnd xjIs a 4x1 vector.
And calculating a transformation matrix for transforming the position of one point in the image to be processed to the position of the same point in the reference image according to the motion matrix, and taking the transformation matrix as position transformation information of the image to be processed.
In the step S400, the rotating the image to be processed according to the position transformation information includes rotating the image to be processed according to the position transformation information, and specifically includes the following steps:
and transforming the positions of all points in the image to be processed according to the transformation matrix.
The purpose of the anti-shake algorithm is to put x into the consecutive j, i two framesiConversion to x by a transformation matrixjAt the position of (2), the picture is stabilized. Thus, the transformation matrix W is defined as follows:
W(t1，t2)＝KR′(t1)RT(t2)K-1。
wherein, W (t)1，t2) A transformation matrix representing the transformation of the position of a point in the image to be processed taken at the moment t2 to the position of the same point in the reference image taken at the moment t1, K being the camera eigen matrix of the image capturing device, K-1Is a transposed matrix of a camera intrinsic matrix of the image pickup apparatus, R' (t)1) The product R of the motion matrix of the x-axis, y-axis and z-axis of the camera at time t1x(t1)*Ry(t1)*Rz(t1)，RT(t2) The product R of the motion matrix of the x-axis, y-axis and z-axis of the camera at time t2x(t2)*Ry(t2)*Rz(t2) The transposed matrix of (2). .
In step S400, the position x of each point in the image to be processed is determinediThe transformation equations for the transformations are as follows, i.e. all transformations to xjThe position of (2):
xj＝W(t(j，yj)，t(i，yi))xi，where R′＝R.
further, in this embodiment, in the step S400, after the transformation matrix is obtained, the frame of the image to be processed may be directly transformed by using a GPU through a vertex shader.
In this embodiment, in step S300, the cropping information of the image to be processed is calculated according to the motion data, and the cropping information may also be obtained by using the motion matrix, so that the calculation time is shorter than that of a cropping method using a graphics-related algorithm, and the method can be more effectively applied to a real-time scene. Specifically, the step of calculating the cropping information of the image to be processed includes the following steps:
and calculating a scaling coefficient of the image to be processed according to the motion data, and taking the scaling coefficient as the clipping information of the image to be processed.
In the step S400, performing cropping processing on the image to be processed according to the cropping information, including performing cropping processing on the image to be processed according to the cropping information, specifically including the following steps:
and zooming the image to be processed according to the zooming coefficient, and removing the part of the zoomed image which exceeds the range of a preset image.
In this embodiment, calculating the scaling factor of the image to be processed according to the motion data includes the following steps:
first, the following formula is defined:
x′＝x/α
wherein x is the coordinate of any point after the transformed image center is defined as the origin (i.e. the coordinate of any point in the image before zooming), x' is the coordinate of the point in the final output image (i.e. the coordinate of the point in the image after zooming), and α is the defined zoom factor.
According to the above method for generating the transformation matrix, it is considered that the value of the scaling factor α should be correlated with the instantaneous angular velocity of the gyroscope output, and the value of α is controlled by a PID method, which is proportional-integral-derivative, and the formula is as follows:
wherein, each parameter is defined as follows:
MV denotes Manipulated Variable, MV (t) and u (t) are the adjusted parameters, which, when applied to the adjustment of the scaling factor α, correspond to the value of the output scaling factor α;
kp: proportional gain, which is an adaptation parameter;
ki: integral gain, also an adaptation parameter;
kd: differential gain, also an adaptation parameter;
e: the error is a set value (SP) -a feedback value (PV), the set value SP is a set ideal angular velocity value (an angular velocity value of the gyroscope is obtained by placing the camera device in an environment without shaking, and is set as SP), the SP (set point) value is a constant, and the feedback value PV (process value) is an instantaneous angular velocity value of the gyroscope obtained through a PID algorithm, namely an angular velocity value acquired in real time through the gyroscope;
t: current time of day
τ: the variable is integrated, and the value is from 0 to the current time t.
Since the scaling factor α is related to the gyroscope data, which is three-dimensional, i.e. all three directions x, y, and z have values, the error e in the PID equation is an error value including three directions x, y, and z, and therefore, the PID equation can be further adjusted as follows:
α′＝Kpxωx+Kpyωy+Kpzωz+Kix∑ωx+Kiy∑ωy+Kiz∑ωz+Kdx(ωx+ωx′)+Kdy(ωy+ωy′)+Kdz(ωz-ωz′)
wherein, K is a parameter that needs to be actually debugged, and specifically includes:
Kpx，Kpy，Kpzand respectively represent proportional gains of the gyroscopes in x, y and z directions.
wx＝wxsp-wxpvWherein w isxspSetting value, w, representing the x-direction of the gyroscopexpvRepresenting the feedback value in the x-direction of the gyroscope.
Kix，Kiy，KizAnd respectively represent the integral gains of the gyroscopes in x, y and z directions.
Kdx，Kdy，KdzAnd respectively represent differential gains of the gyroscopes in x, y and z directions.
wx，wy，wzThe values of the motion error, ω, in the x, y and z directions at time tx′、ωy′、ωz′The motion error values in the x direction, the y direction and the z direction at a time t + Δ t, where the time t is a current time and Δ t is a preset time variation, may be further defined as follows:
wx＝wxsp-wxpvwherein w isxspSetting value, w, representing the x-direction of the gyroscopexpvRepresenting the feedback value in the x-direction of the gyroscope.
wy＝wysp-wypvWherein w isyspSet value, w, representing the y-direction of the gyroscopeypvRepresenting the feedback value in the y-direction of the gyroscope.
wz＝wzsp-wzpvWherein w iszspSet value, w, representing the z-direction of the gyroscopezpvRepresenting the feedback value in the z-direction of the gyroscope.
Alpha is in the range of [0, 1]Since a very small scaling factor is obviously not reasonable, a threshold value can be set by debugging as the minimum value of α. Since the range of alpha is [0, 1 ]]Therefore, α does not exceed 1 at maximum, and the value of α can be further located by calibration, and a possible method is: respectively counting alpha values obtained by calculation under different jitter degrees, and taking the maximum value as alphamax. Since the maximum value is related to the actual product and application scenario, it is not a constant value.
Specifically, in step S400, the image to be processed is scaled according to the scaling factor, a portion of the scaled image that exceeds a preset image range is removed, a processed primitive can be output at a specific image coordinate by a fragment shader using a GPU, and the primitive that exceeds the image range can be automatically ignored.
As shown in fig. 4, an embodiment of the present invention further provides an anti-shake processing system for an image capturing apparatus, which is applied to the anti-shake processing method for the image capturing apparatus, and the system includes:
an image obtaining module M100, configured to obtain a video frame to be processed as an image to be processed;
a motion obtaining module M200, configured to obtain motion data of the image capturing apparatus corresponding to the shooting time of the image to be processed;
an information calculating module M300, configured to calculate position transformation information and/or cropping information of the image to be processed according to the motion data;
and the image processing module M400 is configured to rotate and/or crop the image to be processed according to the position transformation information and/or the cropping information, so as to obtain a processed video frame.
According to the anti-shake processing system of the camera device, the video frame is extracted as the image to be processed through the image acquisition module M100, the motion data is acquired through the motion acquisition module M200, the position transformation information and/or the cutting information are calculated through the information calculation module M300 based on the motion data, and the image is subjected to position transformation and/or cutting processing through the image processing module M400 based on the information obtained through calculation, so that the anti-shake processing of the video frame to be processed based on the motion data of the camera device is realized, and the video shooting effect is improved.
The embodiment of the invention also provides anti-shake processing equipment of the camera device, which comprises a processor; a memory having stored therein executable instructions of the processor; wherein the processor is configured to execute the steps of the anti-shake processing method of the image pickup apparatus via execution of the executable instructions.
As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or program product. Thus, various aspects of the invention may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" system.
An electronic device 600 according to this embodiment of the invention is described below with reference to fig. 5. The electronic device 600 shown in fig. 5 is only an example and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 5, the electronic device 600 is embodied in the form of a general purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 that connects the various system components (including the storage unit 620 and the processing unit 610), a display unit 640, and the like.
Wherein the storage unit stores program code executable by the processing unit 610 to cause the processing unit 610 to perform steps according to various exemplary embodiments of the present invention described in the above-mentioned electronic prescription flow processing method section of the present specification. For example, the processing unit 610 may perform the steps as shown in fig. 1.
The storage unit 620 may include readable media in the form of volatile memory units, such as a random access memory unit (RAM)6201 and/or a cache memory unit 6202, and may further include a read-only memory unit (ROM) 6203.
The memory unit 620 may also include a program/utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.
The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 600, and/or with any devices (e.g., router, modem, etc.) that enable the electronic device 600 to communicate with one or more other computing devices. Such communication may occur via an input/output (I/O) interface 650. Also, the electronic device 600 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via the network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via the bus 630. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.
By adopting the anti-shake processing device of the image pickup apparatus provided by the invention, the processor executes the anti-shake processing method of the image pickup apparatus when executing the executable instruction, thereby being capable of obtaining the beneficial effects of the anti-shake processing method of the image pickup apparatus.
An embodiment of the present invention further provides a computer-readable storage medium, which is used for storing a program, and when the program is executed by a processor, the method for processing an image capture device according to the present invention is implemented. In some possible embodiments, aspects of the present invention may also be implemented in the form of a program product comprising program code for causing a terminal device to perform the steps according to various exemplary embodiments of the present invention described in the above-mentioned electronic prescription flow processing method section of this specification, when the program product is run on the terminal device.
Referring to fig. 6, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited in this regard and, in the present document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The computer readable storage medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable storage medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or cluster. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
By adopting the computer-readable storage medium provided by the invention, the stored program realizes the steps of the anti-shake processing method of the image pickup apparatus when being executed, thereby the beneficial effects of the anti-shake processing method of the image pickup apparatus can be obtained.
The foregoing is a more detailed description of the invention in connection with specific preferred embodiments and it is not intended that the invention be limited to these specific details. For those skilled in the art to which the invention pertains, several simple deductions or substitutions can be made without departing from the spirit of the invention, and all shall be considered as belonging to the protection scope of the invention.
Claims (11)
1. An anti-shake processing method for an imaging apparatus, comprising:
acquiring a video frame to be processed as an image to be processed;
acquiring motion data of the camera device corresponding to the shooting time of the image to be processed;
calculating position transformation information and/or cutting information of the image to be processed according to the motion data;
and rotating and/or cutting the image to be processed according to the position conversion information and/or the cutting information to obtain a processed video frame.
2. The anti-shake processing method for an image pickup apparatus according to claim 1, wherein acquiring motion data of the image pickup apparatus corresponding to a shooting time of the image to be processed, comprises:
acquiring the shooting start time and the shooting end time of the image to be processed;
dividing the image to be processed into a plurality of line blocks, wherein each line block comprises at least one continuous image line;
calculating the shooting time of each line block according to the shooting start time and the shooting end time of the image to be processed;
and acquiring motion data of the camera corresponding to the shooting time of each line block.
3. The anti-shake processing method for an image pickup apparatus according to claim 2, wherein the step of calculating the shooting time for each line block from the shooting start time and the shooting end time of the image to be processed comprises:
calculating the shooting time delta t-tb according to the shooting start time ta and the shooting end time tb of the image to be processed;
calculating the average shooting time consumption delta t/n of each line block, wherein n is the total number of the line blocks;
and calculating the shooting start time tc ═ ta + (i-1) × Δ t/n of the ith line block, the shooting end time td ═ ta + i × Δ t/n, and the shooting time of each line block comprises the shooting start time tc and the shooting end time td.
4. The anti-shake processing method for an image pickup apparatus according to claim 1, wherein acquiring motion data of the image pickup apparatus corresponding to a shooting time of the image to be processed, comprises:
acquiring rotation data acquired by a gyroscope of the camera device according to the shooting time of the image to be processed, wherein the shooting time of the image to be processed comprises shooting start time and shooting end time;
and converting the rotation data acquired by the gyroscope into a motion matrix of the camera device, and using the motion matrix as the motion data of the shooting time of the image to be processed.
5. The anti-shake processing method for an image pickup apparatus according to claim 4, wherein calculating position conversion information of the image to be processed from the motion data includes:
acquiring a reference video frame as a reference image;
calculating a transformation matrix for transforming the position of one point in the image to be processed to the position of the same point in the reference image according to the motion matrix, and taking the transformation matrix of each point in the image to be processed as position transformation information of the image to be processed;
and performing rotation processing on the image to be processed according to the position transformation information, and the method comprises the following steps:
and transforming the positions of all points in the image to be processed according to the transformation matrix.
6. The method according to claim 5, wherein the reference video frame and the to-be-processed video frame are two consecutive video frames, and the calculating of the transformation matrix for transforming the position of a point in the to-be-processed image to the position of the same point in the reference image comprises calculating the transformation matrix using the following formula:
W(t1，t2)＝KR′(t1)RT(t2)K-1。
wherein, W (t)1，t2) A transformation matrix representing the transformation of the position of a point in the image to be processed taken at the moment t2 to the position of the same point in the reference image taken at the moment t1, K being the camera eigen matrix of the image capturing device, K-1Is the inverse of the camera eigen matrix, R' (t)1) Moments of motion in the x, y and z axes of the camera at time t1Product of arrays, RT(t2) The transpose of the product of the x-axis, y-axis and z-axis motion matrices for the camera at time t 2.
7. The anti-shake processing method for an image pickup apparatus according to claim 1, wherein the calculating of cropping information of the image to be processed from the motion data includes:
calculating a scaling coefficient of the image to be processed according to the motion data, and taking the scaling coefficient as clipping information of the image to be processed;
and cutting the image to be processed according to the cutting information, comprising the following steps:
and zooming the image to be processed according to the zooming coefficient, and removing the part of the zoomed image which exceeds the range of a preset image.
8. The anti-shake processing method for an image pickup apparatus according to claim 7, wherein calculating the scaling factor of the image to be processed from the motion data includes calculating the scaling factor of the image to be processed using the following formula:
α′＝Kpxωx+Kpyωy+Kpzωz+Kix∑ωx+Kiy∑ωy+Kiz∑ωz+Kdx(ωω-ωx′)+Kdy(ωy-ωy′)+Kdz(ωz-ωz′)
wherein α is a scaling coefficient of the image to be processed, α' is ωx、ωy、ωzRepresenting the values of the motion error, ω, in the x, y and z directions, respectively, at time tx′、ωy′、ωz′Respectively represent xThe motion error values of the direction, y-direction and z-direction at the time t + Δ t, Kpx、Kpy、KpzRespectively representing proportional gains in the x, y and z directions, Kix、Kiy、KizDenotes the integral gain in the x, y and z directions, respectively, Kdx、Kdy、KdzRespectively representing differential gains in the x, y and z directions, alphamax、αminRespectively representing a preset maximum scaling coefficient value and a preset minimum scaling coefficient value.
9. An anti-shake processing system of an image pickup apparatus, applied to the anti-shake processing method of the image pickup apparatus according to any one of claims 1 to 8, the system comprising:
the image acquisition module is used for acquiring a video frame to be processed as an image to be processed;
the motion acquisition module is used for acquiring motion data of the shooting time of the image to be processed corresponding to the image pickup device;
the information calculation module is used for calculating the position transformation information and/or the cutting information of the image to be processed according to the motion data;
and the image processing module is used for rotating and/or cutting the image to be processed according to the position transformation information and/or the cutting information to obtain a processed video frame.
10. An anti-shake processing apparatus of an image pickup device, comprising:
a processor;
a memory having stored therein executable instructions of the processor;
wherein the processor is configured to execute the steps of the anti-shake processing method of the image pickup apparatus according to any one of claims 1 to 8 via execution of the executable instructions.
11. A computer-readable storage medium storing a program, wherein the program realizes the steps of the anti-shake processing method of the image pickup apparatus according to any one of claims 1 to 8 when executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110321357.XA CN113014823B (en) | 2021-03-25 | 2021-03-25 | Anti-shake processing method, system, equipment and storage medium of image pickup device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110321357.XA CN113014823B (en) | 2021-03-25 | 2021-03-25 | Anti-shake processing method, system, equipment and storage medium of image pickup device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113014823A true CN113014823A (en) | 2021-06-22 |
| CN113014823B CN113014823B (en) | 2023-06-06 |
Family
ID=76407261
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110321357.XA Active CN113014823B (en) | 2021-03-25 | 2021-03-25 | Anti-shake processing method, system, equipment and storage medium of image pickup device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113014823B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105069753A (en) * | 2015-07-30 | 2015-11-18 | 华中科技大学 | Mobile-terminal-oriented method for restoring blurred image caused by jitter |
| CN106027852A (en) * | 2016-06-24 | 2016-10-12 | 西北工业大学 | Video image stabilization method for micro/nano-satellite |
| WO2018223381A1 (en) * | 2017-06-09 | 2018-12-13 | 厦门美图之家科技有限公司 | Video shake-prevention method and mobile device |
| CN110213479A (en) * | 2019-04-30 | 2019-09-06 | 北京迈格威科技有限公司 | A kind of video capture anti-fluttering method and device |
-
2021
- 2021-03-25 CN CN202110321357.XA patent/CN113014823B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105069753A (en) * | 2015-07-30 | 2015-11-18 | 华中科技大学 | Mobile-terminal-oriented method for restoring blurred image caused by jitter |
| CN106027852A (en) * | 2016-06-24 | 2016-10-12 | 西北工业大学 | Video image stabilization method for micro/nano-satellite |
| WO2018223381A1 (en) * | 2017-06-09 | 2018-12-13 | 厦门美图之家科技有限公司 | Video shake-prevention method and mobile device |
| CN110213479A (en) * | 2019-04-30 | 2019-09-06 | 北京迈格威科技有限公司 | A kind of video capture anti-fluttering method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113014823B (en) | 2023-06-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR102227583B1 (en) | Method and apparatus for camera calibration based on deep learning | |
| EP2574038B1 (en) | Image capturing apparatus, image processing apparatus, image processing method, and image processing program | |
| US8508651B2 (en) | Image stabilizing apparatus, image pick-up apparatus and image stabilizing method | |
| US11222409B2 (en) | Image/video deblurring using convolutional neural networks with applications to SFM/SLAM with blurred images/videos | |
| CN113556464B (en) | Shooting method and device and electronic equipment | |
| CN109561254B (en) | Method and device for preventing panoramic video from shaking and portable terminal | |
| US20150009345A1 (en) | Image pickup apparatus and control method for same | |
| WO2019171984A1 (en) | Signal processing device, signal processing method, and program | |
| WO2021253186A1 (en) | Information processing method and device, and imaging system | |
| CN113436113A (en) | Anti-shake image processing method, device, electronic equipment and storage medium | |
| EP4221181A1 (en) | Method for generating rotation direction of gyroscope and computer device | |
| CN110958363B (en) | Image processing method and device, computer readable medium and electronic device | |
| CN109688327B (en) | Method and device for preventing panoramic video from shaking and portable terminal | |
| CN109561253B (en) | Method and device for preventing panoramic video from shaking, portable terminal and storage medium | |
| CN113014823B (en) | Anti-shake processing method, system, equipment and storage medium of image pickup device | |
| CN114785957A (en) | Shooting method and device thereof | |
| CN111656763B (en) | Image acquisition control method, image acquisition control device and movable platform | |
| CN116137025A (en) | Video image correction method and device, computer readable medium and electronic equipment | |
| CN109963085B (en) | Method and device for adjusting shutter speed and robot | |
| CN115278086B (en) | Electronic anti-shake method for gyroscope | |
| CN116797497B (en) | Image processing method, device, equipment and storage medium | |
| CN113938605B (en) | Photographing method, photographing device, photographing equipment and photographing medium | |
| CN115297260A (en) | Image processing method and device | |
| CN117934325A (en) | Image processing method and device, calibration method and device and electronic equipment | |
| CN117745528A (en) | Image processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |