WO2019119597A1 - Method for implementing planar recording and panoramic recording by coordination between mobile terminal and lens assembly and lens assembly - Google Patents

Abstract

The present invention is applicable in the field of video recording; provided thereby is a method for implementing planar recording and panoramic recording by coordination between a mobile terminal and a lens assembly, and also provided thereby is a lens assembly. The lens assembly comprises a mounting bracket, a front lens and a rear lens, wherein both the front and back of the mounting bracket are provided with a through hole, the front lens is embedded into the through hole on the front of the mounting bracket, the rear lens is embedded into the through hole on the back of the mounting bracket, and the mounting bracket is detachably sleeved over the outer side of a camera area of a mobile terminal. When the lens assembly is installed on the mobile terminal, the front lens covers a front camera of the mobile terminal, and the rear lens covers a rear camera of the mobile terminal, wherein the front lens and the rear lens are both wide-angle lenses or fish-eye lenses; The front camera or rear camera of the mobile terminal is controlled to record video by means of a planar recording application program installed on the mobile terminal, thus planar video recording may be achieved by coordination with the front lens or the rear lens, which is thus low cost as a professional camera is not required.

Description

Mobile terminal and lens assembly cooperate to realize plane shooting, panoramic shooting method and lens assembly Technical field

The invention belongs to the field of video shooting, and in particular relates to a method and a lens assembly for realizing plane shooting, panoramic shooting, and a combination of a mobile terminal and a lens assembly.

Background technique

At present, wide-angle flat video is shot by a professional camera with a wide-angle lens. However, the cost of professional cameras is very high, and few people specialize in buying professional cameras for occasional wide-angle flat video. At present, panoramic pictures and videos are taken by a dedicated panoramic camera. The panoramic camera is composed of multiple lenses. The panoramic camera stores the pictures and videos taken by multiple lenses separately, and then exports them to a computer for synthesizing panoramic pictures and videos. . However, the cost of a dedicated panoramic camera is high, and fewer people specifically buy a panoramic camera for occasional panoramic pictures and videos. Therefore, it is highly desirable to provide a lens assembly that is low in cost and that can be used with a mobile terminal to achieve wide-angle flat video shooting or panoramic shooting.

technical problem

An object of the present invention is to provide a method and a lens assembly for a plane shooting, a panoramic shooting, and a lens assembly, which are intended to solve the problem of shooting a wide-angle flat video or a dedicated panoramic camera through a professional camera with a wide-angle lens. Shooting panoramic images and videos is a costly issue.

Technical solution

In a first aspect, the present invention provides a lens assembly including a mounting bracket, a front lens, and a rear lens, wherein the front and back sides of the mounting bracket are provided with through holes, and the front lens is embedded in the mounting bracket The front through hole, the rear lens is embedded in the through hole of the back of the mounting bracket, the mounting bracket is detachably sleeved outside the camera area of the mobile terminal, and the front lens covers the mobile terminal when the lens assembly is mounted to the mobile terminal Front camera, rear lens covers the rear camera of the mobile terminal, front lens and rear lens are wide-angle lens or fisheye lens; control the front camera or rear of the mobile terminal through the plane shooting application installed in the mobile terminal The camera is used for shooting, and the front lens or the rear lens is used for plane video shooting, or the front camera and the rear camera of the mobile terminal are simultaneously controlled by the panoramic shooting application installed in the mobile terminal, and the camera is matched. The front and rear lenses are used for panoramic shooting.

Further, the mobile terminal is a mobile phone or a tablet; the lens component does not block the display screen of the mobile terminal.

Further, the front lens is coaxial or substantially parallel to the optical axis of the front camera of the mobile terminal, and the rear lens is coaxial or substantially parallel to the optical axis of the rear camera of the mobile terminal.

Further, the front lens is embedded in the through hole of the front surface of the mounting bracket through the front lens sleeve, and the rear lens is embedded into the through hole of the back surface of the mounting bracket through the rear lens sleeve.

Further, the front lens sleeve and the rear lens sleeve are integrated with the mounting bracket, or the front lens sleeve and the rear lens sleeve are respectively detachably fixed to the mounting bracket, or The front lens sleeve and the rear lens sleeve are fixedly coupled to the mounting bracket, respectively.

Further, the mounting bracket has an annular side wall forming a hollow cavity, the shape of the cavity matching the camera area of the mobile terminal.

Further, a front surface of the mounting bracket is further provided with a through hole corresponding to a position of a horn of the mobile terminal.

In a second aspect, the present invention further provides a method for implementing planar shooting by a mobile terminal and a lens assembly, the method comprising:

The mobile terminal launches a plane shooting application installed in the mobile terminal, the mobile terminal being mounted with the lens assembly as described above;

The mobile terminal acquires a current state time stamp, an acceleration count value, and an angular velocity value of the gyroscope in the mobile terminal in real time;

The mobile terminal uses the extended Kalman filter combined with the acceleration count value and the angular velocity value to estimate the amount of rotation of the mobile terminal to the world coordinate system;

Controlling the front camera of the mobile terminal to acquire a planar video frame via a front lens of the lens assembly, or controlling a rear camera of the mobile terminal to acquire a planar video frame via a rear lens of the lens assembly;

The mobile terminal synchronizes the gyroscope time stamp with the time stamp of the planar video frame;

The mobile terminal performs quaternion interpolation on the state of the gyroscope to obtain a rotation matrix corresponding to the plane video frame;

The mobile terminal rotates the planar video frame according to the current rotation matrix to generate a stable planar video frame.

In a third aspect, the present invention further provides a method for implementing planar shooting by using a mobile terminal and a lens assembly, the method comprising:

The mobile terminal launches a plane shooting application installed in the mobile terminal, the mobile terminal being mounted with the lens assembly as described above;

The mobile terminal acquires the current state time stamp, the acceleration count value, and the angular velocity value of the mobile terminal in real time;

The mobile terminal estimates the rotation vector of the current state by using the extended Kalman filter combined with the acceleration count value and the angular velocity value;

The mobile terminal calculates a current rotation matrix by using a Rodrigue rotation formula according to a rotation vector of the current state;

Controlling the front camera of the mobile terminal to acquire a planar video frame via a front lens of the lens assembly, or controlling a rear camera of the mobile terminal to acquire a planar video frame via a rear lens of the lens assembly;

The mobile terminal rotates the planar video frame according to the current rotation matrix to generate a stable planar video frame.

In a fourth aspect, the present invention further provides a method for implementing panoramic shooting by using a mobile terminal and a lens assembly, the method comprising:

The mobile terminal launches a panoramic shooting application installed in the mobile terminal, the mobile terminal being mounted with the lens assembly as described above;

The mobile terminal pre-establishes a spherical panoramic mathematical model, wherein the spherical panoramic mathematical model is centered on the center of the spatial position of the front camera and the rear camera of the mobile terminal, and can be collected by the front camera and the rear camera. The spherical crowns obtained by the angles of view are combined into a spherical surface;

The front camera controlling the mobile terminal collects an image through the front lens of the lens assembly, and controls the rear camera of the mobile terminal to collect an image through the rear lens of the lens assembly;

According to the correspondence between the lens imaging height of the front lens and the rear lens and the angle of the field of view, the two images are mapped into the spherical panoramic mathematical model;

The overlapping regions of the two images in the spherical panoramic mathematical model are fused to splicing the two images into a spherical panoramic image covering all the viewing angles of the horizontal and vertical directions.

Beneficial effect

In the present invention, since the lens assembly includes a mounting bracket, a front lens, and a rear lens, when the lens assembly is mounted to the mobile terminal, the front lens covers the front camera of the mobile terminal, and the rear lens covers the rear camera of the mobile terminal. Both the front lens and the rear lens are wide-angle lenses or fisheye lenses. Therefore, the front camera or the rear camera of the mobile terminal is controlled by the plane shooting application installed in the mobile terminal to perform shooting, and the front camera or the rear lens is used for plane video shooting, which is low in cost and does not require a professional camera. The front camera and the rear camera of the mobile terminal are simultaneously controlled by the panoramic shooting application installed in the mobile terminal, and the front lens and the rear lens can be used for panoramic shooting, and the cost is low, and a dedicated panoramic camera is not required. .

Since the mobile terminal is equipped with the lens assembly provided by the present invention, and the method for implementing planar imaging by the mobile terminal and the lens assembly of the present invention, the state of the gyroscope is interpolated to obtain the rotation matrix of the corresponding planar video frame, Can get a more accurate rotation matrix. A planar video frame is then rotated according to the current rotation matrix to generate a stable planar video frame. Therefore, the planar video frame that can finally stabilize the jitter is robust to large noise scenes and most motion scenes.

In addition, because the estimated angle of the acceleration count value is susceptible to interference (such as walking, walking, running, etc.), the accumulated error of the angular velocity becomes larger and larger as time passes. In the method for implementing planar shooting by the mobile terminal and the lens assembly of the present invention, since the extended Kalman filter is combined with the acceleration count value and the angular velocity value, the rotation vector of the current state is estimated, and Rodrigue rotation is performed according to the rotation vector of the current state. The formula calculates the current rotation matrix and then rotates the planar video frame, thus ultimately stabilizing the dithered planar video frame.

Since the mobile terminal is equipped with the lens assembly provided by the embodiment of the present invention, the method for implementing the panoramic shooting by the mobile terminal and the lens assembly of the present invention can control the front camera of the mobile terminal to collect an image through the front lens of the lens assembly, and simultaneously The rear camera controlling the mobile terminal collects an image through the rear lens of the lens assembly, and then maps the two images to the sphere according to the correspondence between the lens imaging height of the front lens and the rear lens and the angle of the field of view. In the panoramic mathematical model, the overlapping regions of the two images in the spherical panoramic mathematical model are fused, thereby splicing the two images into a spherical panoramic image covering all the viewing angles of the horizontal and vertical directions. The invention can collect the complete panoramic image at one time, can record the panoramic video, can obtain the panoramic image more completely and efficiently, and greatly simplifies the user operation process and reduces the cost of the panoramic shooting; in addition, based on the spherical panoramic mathematical model Therefore, the information collection capability of the camera is maximized, so that it has the ability to save the scene on the spot.

DRAWINGS

1 is a front elevational view of a lens assembly according to an embodiment of the present invention.

2 is a rear elevational view of a lens assembly according to an embodiment of the present invention.

3 is a right side view of a lens assembly according to an embodiment of the present invention.

4 is a top plan view of a lens assembly according to an embodiment of the present invention.

FIG. 5 is a bottom view of a lens assembly according to an embodiment of the present invention.

Figure 6 is a cross-sectional view of a lens assembly according to an embodiment of the present invention.

FIG. 7 is a front view of a lens assembly and a mobile terminal according to an embodiment of the present invention.

FIG. 8 is a rear view of a lens assembly and a mobile terminal according to an embodiment of the present invention.

FIG. 9 is a top plan view of a lens assembly and a mobile terminal according to an embodiment of the present invention.

FIG. 10 is a right side view of the lens assembly and the mobile terminal provided by the embodiment of the present invention.

FIG. 11 is a schematic diagram of a second lens unit for performing panoramic shooting with a mobile terminal (Iphone 6, Iphone 6S, Iphone 7, Iphone 8) according to an embodiment of the present invention.

FIG. 12 is a schematic diagram of a third lens assembly for implementing panoramic shooting with a mobile terminal (Iphone 6Plus, Iphone 6S Plus, Iphone 7Plus, Iphone 8Plus) according to an embodiment of the present invention.

FIG. 13 is a schematic diagram of a fourth lens unit for performing panoramic shooting with a mobile terminal (Iphone X) according to an embodiment of the present invention.

FIG. 14 is a flowchart of a method for implementing planar shooting by using a mobile terminal and a lens assembly according to an embodiment of the present invention.

FIG. 15 is a flowchart of S103 in a method for implementing plane shooting by using a mobile terminal and a lens assembly according to an embodiment of the present invention.

FIG. 16 is a flowchart of a method for implementing planar shooting by a mobile terminal and a lens assembly according to another embodiment of the present invention.

FIG. 17 is a flowchart of S203 in a method for implementing planar shooting by a mobile terminal and a lens assembly according to another embodiment of the present invention.

FIG. 18 is a flowchart of a method for implementing panoramic shooting by using a mobile terminal and a lens assembly according to an embodiment of the present invention.

Embodiments of the invention

The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

In order to explain the technical solution described in the present invention, the following description will be made by way of specific embodiments.

Referring to FIG. 1 to FIG. 10 , the lens assembly 100 of the embodiment of the present invention includes a mounting bracket 101 , a front lens 102 , and a rear lens 103 . The front and back sides of the mounting bracket 101 are provided with through holes (not shown). The front lens 102 is embedded in the through hole of the front surface of the mounting bracket 101, the rear lens 103 is embedded in the through hole of the back surface of the mounting bracket 101, and the mounting bracket 101 is detachably sleeved outside the camera area of the mobile terminal 200. When the lens assembly 100 is mounted to the mobile terminal 200, the front lens 102 covers the front camera of the mobile terminal 200, the rear lens 103 covers the rear camera of the mobile terminal 200, and the lens assembly 100 does not block the display screen of the mobile terminal 200.

In the embodiment of the present invention, in order to make the photographing effect better, the front lens 102 is coaxial or substantially parallel with the optical axis of the front camera of the mobile terminal 200. The rear lens 103 is coaxial or substantially parallel to the optical axis of the rear camera of the mobile terminal 200.

In the embodiment of the present invention, the front lens 102 is embedded in the through hole of the front surface of the mounting bracket 101 through the front lens sleeve 104, and the rear lens 103 is inserted into the through hole of the rear surface of the mounting bracket 101 through the rear lens sleeve 105. . The front lens sleeve 104 and the rear lens sleeve 105 may be integral with the mounting bracket 101, or the front lens sleeve 104 and the rear lens sleeve 105 may be detachably fixed to the mounting bracket, respectively, or The lens barrel 104 and the rear lens barrel 105 are fixedly coupled to the mounting bracket 101, respectively. The mounting bracket 101 has an annular side wall 106 forming a hollow cavity, the shape of which matches the camera area of the mobile terminal 200, so that the mounting bracket 101 can be fixed to the camera area of the mobile terminal 200.

The mounting bracket 101 can have a top wall or no top wall, shown as a mounting bracket without a top wall (see Figure 10). As shown in FIGS. 6 and 10, since the top of the mobile terminal is generally curved, in order to better fit the side wall 106 of the mounting bracket 101 with the mobile terminal 200, the front and back side walls 106 of the mounting bracket 101 are oriented. The middle of the top portion extends a predetermined distance to form the fixing portion 107, so that the mounting bracket 101 does not easily slip. Referring to FIG. 5, the bottom end of the side wall 106 of the mounting bracket 101 mating with the side of the mobile terminal 200 is provided with a protrusion 108 to better secure the mounting bracket 101 to the mobile terminal 200. Referring to FIG. 7 , the front surface of the mounting bracket 101 is further provided with a through hole 109 corresponding to the horn position of the mobile terminal 200 , so that the lens assembly 100 does not affect the horn playing effect of the mobile terminal.

In the embodiment of the present invention, the mobile terminal 200 may be a mobile phone, a tablet computer, or the like. Both the front lens 102 and the rear lens 103 may be wide-angle lenses or fisheye lenses (ie, super wide-angle lenses).

1 to FIG. 10 is only a lens assembly used in conjunction with one of the mobile terminals. For different mobile terminals, especially mobile terminals having different camera positions, the structure of the lens assembly provided by the embodiment of the present invention can also be adapted. Modified as shown in Figures 11, 12 and 13.

After the lens assembly provided by the embodiment of the present invention is fixed to the mobile terminal, when a plane video needs to be captured, the front camera or the rear camera of the mobile terminal can be controlled by starting a plane shooting application installed in the mobile terminal. Plane shooting with the front or rear lens. Since the front camera and the rear camera are covered by the front lens and the rear lens, the images captured by the front camera and the rear camera are actually pre-positioned. The flat video frames captured by the lens and rear lens, because the front and rear lenses are wide-angle lenses or fisheye lenses, can achieve a 180-degree viewing angle.

When the lens assembly provided by the embodiment of the present invention is fixed to the mobile terminal, when the panoramic picture or video needs to be captured, the front camera and the rear position of the mobile terminal can be controlled simultaneously by starting the panoramic shooting application installed in the mobile terminal. The camera is shooting. Since the front camera and the rear camera are covered by the front lens and the rear lens, respectively, the images captured by the front camera and the rear camera are actually images captured by the front lens and the rear lens. The front and rear lenses are wide-angle lenses or fisheye lenses that can achieve a 180-degree viewing angle, so two images captured by the front lens, rear lens, front camera, and rear camera; two images can be mapped In the spherical panoramic mathematical model, the overlapping regions of the two images in the spherical panoramic mathematical model are fused, thereby splicing the two images into a spherical panoramic image covering all the viewing angles of the horizontal and vertical directions.

Referring to FIG. 14 , an embodiment of the present invention further provides a method for implementing a plane shooting by using a mobile terminal and a lens component, where the method includes:

S101. The mobile terminal starts a plane shooting application installed in the mobile terminal, where the mobile terminal is installed with a lens assembly according to an embodiment of the present invention;

In the embodiment of the present invention, after S101, the method may further include the following steps:

The mobile terminal receives an instruction selected by the user to activate the anti-shake function.

S102. The mobile terminal acquires, in real time, a current state timestamp, an acceleration count value, and an angular velocity value of the gyroscope in the mobile terminal.

In the embodiment of the present invention,

The real-time acquisition of the acceleration count value of the gyroscope in the mobile terminal may specifically be: reading the triaxial acceleration count value by using the gravity sensor.

The real-time acquisition of the angular velocity value of the gyroscope in the mobile terminal may specifically be: reading the triaxial angular velocity value by using the angular velocity sensor.

In the embodiment of the present invention, the following steps may be further included after S102:

The acceleration count value is denoised by low-pass filtering. Specifically, the following steps may be included:

The acceleration count value is subjected to low-pass filtering and noise reduction processing by the formula d' _i =α·d _i +(1 - α)·R _i ·d' _i-1 , where d' _i represents the low-pass filtering at the ith time After the acceleration count value, d _i represents the acceleration count value at the i-th moment, and R _i is the relative rotation amount of the ith ith frame video.

ω _i represents the angular velocity value at the _i- th moment, d' _i-1 represents the filtered acceleration count value at the i-1th time, and α represents the smoothing factor.

Where f _c represents the cutoff frequency of the low pass filter, Rc represents the time constant, and Δt represents the sampling time interval of the gyroscope data.

S103. The mobile terminal uses the extended Kalman filter to combine the acceleration count value and the angular velocity value to estimate the rotation amount of the mobile terminal to the world coordinate system;

Extended Kalman filtering linearizes the nonlinear system and then performs Kalman filtering. Kalman filtering is a highly efficient recursive filter that estimates the state of a dynamic system from a series of measurements that do not completely contain noise. .

Referring to FIG. 15, in the embodiment of the present invention, S103 may specifically include the following steps:

S1031, initial state rotation amount

Where d ₀ is the initial measured acceleration value, g is the world coordinate system gravity vector; initial process covariance

S1032, using the angular velocity value ω _{k to} calculate the state transition matrix Φ(ω _k ) at the Kth time;

Φ(ω _k )=exp(−[ω _k ·Δt] _× ), where ω _k is the angular velocity value at the Kth moment, and Δt represents the sampling time interval of the gyro data.

S1033. Calculating the covariance matrix Q _{k of the} state noise, updating the state rotation prior estimator

And process covariance a priori estimation matrix

Q _k is a covariance matrix of state noise;

among them,

Is the state rotation after-test estimator at time K-1;

among them,

Is the process covariance posterior estimation matrix at time K-1;

S1034, updating the observed noise variance matrix R _k from the acceleration value d _k , calculating the observation transfer Jacobian matrix H _k , and calculating the current observation measurement and the estimation observation error e _k ;

among them,

α is the smoothing factor of the amount of acceleration change, and β is the influence factor of the acceleration mode length;

Where h is the observation function, h(q,v)=q·g+v _k , g is the gravity vector in the world coordinate system, q is the state quantity, ie the amount of rotation from the world coordinate system to the gyro coordinate system, v _k is Measuring noise;

S1035, updating the optimal Kalman gain matrix K _k at the kth time;

S1036. Update the rotational posterior estimator of the mobile terminal to the world coordinate system according to the optimal Kalman gain matrix _Kk and the observation error _ek .

And process covariance posterior estimation matrix

S104. The front camera controlling the mobile terminal collects a plane video frame via a front lens of the lens component, or controls a rear camera of the mobile terminal to collect a plane video frame via a rear lens of the lens component;

S105. The mobile terminal synchronizes the gyroscope timestamp with a timestamp of the plane video frame.

In the embodiment of the present invention, S105 may specifically be:

The mobile terminal synchronizes the timestamp of the gyroscope with the time stamp of the planar video frame such that t _k ≥ t _j > t _k-1 , where t _j is the time stamp of the planar video frame and t _k is the time stamp of the Kth frame of the gyroscope. t _k-1 is the time stamp of the K-1 frame of the gyroscope.

S106. The mobile terminal performs quaternion interpolation on the state of the gyroscope to obtain a rotation matrix corresponding to the plane video frame.

In the embodiment of the present invention, S106 may specifically include the following steps:

The mobile terminal calculates the relative amount of rotation of the neighboring gyroscope time stamp,

Where r _k is the relative amount of rotation at the Kth moment,

with

The state posterior estimator for the kth and k-1th moments, that is, the amount of rotation of the world coordinate system to the gyro coordinate system;

The mobile terminal performs quaternion interpolation to obtain the relative rotation amount of the plane video frame to the kth frame, R _j =γ·I+(1−γ)·r _k , where R _j is the relative rotation amount of the kth frame,

The mobile terminal calculates a rotation matrix of the jth frame video in the plane video frame

S107. The mobile terminal rotates the planar video frame according to the current rotation matrix to generate a stable planar video frame.

In the embodiment of the present invention, S107 may specifically include the following steps:

The mobile terminal maps grid points on the latitude and longitude two-dimensional image to spherical coordinates;

The mobile terminal traverses all points on the unit ball, and uses the current rotation matrix to rotate all points on the unit ball to generate a stable planar video frame;

Among them, using the current rotation matrix to rotate all points on the unit ball, the following formula is used:

Where [x, y, z] ^T represents the spherical coordinate before the unit circle is rotated, [x _new , y _new , z _new ] ^T represents the spherical coordinate after rotation, Q _j represents the current rotation matrix, and t represents the displacement vector. t=[0,0,0] ^T .

Referring to FIG. 16 , another embodiment of the present invention further provides a method for implementing a plane shooting by a mobile terminal and a lens assembly, where the method includes:

S201. The mobile terminal starts a plane shooting application installed in the mobile terminal, where the mobile terminal is installed with a lens assembly according to an embodiment of the present invention;

In the embodiment of the present invention, after S201, the method may further include the following steps:

The mobile terminal receives an instruction selected by the user to activate the anti-shake function.

S202. The mobile terminal acquires, in real time, a current state timestamp, an acceleration count value, and an angular velocity value of the mobile terminal.

In the embodiment of the present invention,

The acceleration count value of the mobile terminal can be obtained in real time by using a gravity sensor to read the three-axis acceleration count value.

The real-time acquisition of the angular velocity value of the mobile terminal may specifically be: reading the triaxial angular velocity value by using the angular velocity sensor.

In the embodiment of the present invention, the following steps may be further included after S202:

The acceleration count value and the angular velocity value are denoised by low-pass filtering. Specifically, the following steps may be included:

The acceleration count value and the angular velocity value are respectively subjected to low-pass filtering and noise reduction processing by the formula d' _i =α·d _i +(1 - α)·d' _i-1 , wherein d _i represents the acceleration count value at the i-th moment Or angular velocity value; d' _i represents the acceleration count value or angular velocity value after the low-pass filtering at the i-th time; d' _i-1 represents the filtered acceleration count value or angular velocity value at the i-1th time; α represents the smoothing factor ,

Where f _c represents the cutoff frequency of the low pass filter, Rc represents the time constant, and Δt represents the sampling time interval.

S203. The mobile terminal uses the extended Kalman filter to combine the acceleration count value and the angular velocity value to estimate a rotation vector of the current state.

Extended Kalman filtering linearizes the nonlinear system and then performs Kalman filtering. Kalman filtering is a highly efficient recursive filter that estimates the state of a dynamic system from a series of measurements that do not completely contain noise. .

Referring to FIG. 17, in the embodiment of the present invention, S203 may specifically include the following steps:

S2031, the mobile terminal using the angular velocity numerical state transition matrix at time k _k F.; Value using an accelerometer, in conjunction with the rotation matrix on the gravity vector g and a state in the reference frame to calculate the current time prediction residual

In the embodiment of the present invention, S2031 may specifically include the following steps:

The mobile terminal initializes an initial state transition matrix, an initial prediction covariance matrix, and an initial observation matrix, wherein the initial state transition matrix

Initial prediction covariance matrix

Initial observation matrix

The mobile terminal calculates the state transition matrix at time k

Computing observation information matrix

Where x _k-1 represents the state estimation of the mobile terminal at time k-1, and x _k represents the state estimation of the mobile terminal at time k,

Indicates a partial differential symbol, f denotes a state equation function, x denotes the state of the mobile terminal, that is, a rotation angle in three axial directions, and h denotes an observation equation function,

x _k-2 represents the state of the mobile terminal at time k-2, u _k-1 represents the angular velocity value at time k-1, and w _k-1 represents the process noise at time k-1,

Indicates that the k-2 time is used to predict the estimated state of the mobile terminal at time k-1, x _k-1 represents the state of the mobile terminal at time k-1, u _k represents the angular velocity value at time k, and w _k represents the kth Process noise at the moment,

Representing the k-1 time to predict the estimated state of the mobile terminal at time k, x _k-2 =[X _k-2 , Y _k-2 , Z _k-2 ] ^T , where X _k-2 , Y _{k- 2} , Z _k-2 represents the rotation angle of the reference system coordinate system on the X-axis, Y-axis, and Z-axis at the _k- 2th time, x _k-1 = [X _k-1 , Y _k-1 , Z _k-1 ] ^T , where X _k-1 , Y _k-1 , Z _k-1 represent the rotation angle of the reference frame coordinate system on the X-axis, the Y-axis, and the Z-axis at the k-1th time, and T represents the transposition;

The mobile terminal projects the vertical downward gravity acceleration under the reference frame coordinate system to the rigid body coordinate system, and passes the formula

Calculate the forecasting margin

Where z _k is the acceleration count value after noise reduction processing using low-pass filtering at time k, and H _k is an observation information matrix, indicating that the observation equation z _k =h(x _k ,g,v _k ) is calculated using the current estimated state A Jacobian matrix, where g represents the vertical downward gravity vector in the reference coordinate system, g = [0, 0, -9.81] ^T , and v _{k is} expressed as a measurement error.

S2032, the mobile terminal using the estimation error covariance on a state of the matrix _{P k-1 | k-1} , the current state of the state transition matrix F _k and the process noise Q estimation error covariance of the current state of the matrix P _{k | k-1;}

In the embodiment of the present invention, the S2032 may specifically use a formula.

The calculated state prediction estimates a covariance matrix P _k|k-1 , where P _k-1|k-1 represents an estimated covariance matrix of k-1 state, and Q _k represents a covariance matrix of process noise,

Dt represents the sampling interval time of the gyroscope data, and F _k represents the state transition matrix at time k,

Indicates the transpose of F _k .

S2033. The mobile terminal calculates an optimal Kalman gain matrix _Kk of the current state by using the estimated current covariance matrix _Pk|k-1 , the observation matrix _Hk, and the noise variance matrix R of the current state.

In the embodiment of the present invention, S2033 may specifically include the following steps:

Using the state prediction estimation covariance matrix P _k|k-1 to calculate the optimal Kalman gain matrix K _k at time _k ,

R represents the noise covariance matrix,

σ ² represents the noise variance, generally σ = 0.75, and H _k represents the observational information Jacobian matrix at time k.

Indicates the transpose of H _k .

S2034. The optimal Kalman gain matrix K _{k of the} current terminal according to the current state and the current time prediction margin

Update current state estimate rotation vector

In the embodiment of the present invention, S2034 may specifically include the following steps:

The update state estimate obtains the rotation vector of the current state obtained by merging the acceleration count value and the angular velocity value at time k.

Update the estimated covariance matrix P _k|k , P _k|k =(IK _k ·H _k )P _k|k-1 , where I is the identity matrix and P _k|k is the estimated error covariance matrix needed at the next moment P _k-1|k-1 .

S204. The mobile terminal calculates a current rotation matrix by using a Rodrigue rotation formula according to a rotation vector of the current state.

The Rodrigue rotation formula is a calculation formula for calculating a new vector obtained by rotating a given angle around a rotation axis in a three-dimensional space. This formula uses the original vector, the rotation axis and their cross product as the frame to represent the vector after the rotation.

S205. The front camera that controls the mobile terminal collects a plane video frame through a front lens of the lens component, or controls a rear camera of the mobile terminal to collect a plane video frame via a rear lens of the lens component;

S206. The mobile terminal rotates the planar video frame according to the current rotation matrix to generate a stable planar video frame.

In the embodiment of the present invention, S206 may specifically include the following steps:

Mapping points on the warp and weft image to points of the spherical image;

Traversing all points on the unit sphere, using the current rotation matrix to rotate all points on the unit sphere to produce a stable planar video frame.

Wherein, using the current rotation matrix to rotate all points on the unit ball, the following formula can be used:

Where x, y, z represent the spherical coordinates before the unit circle is rotated, x _new , y _new , z _new represents the spherical coordinates after rotation, M _k represents the current rotation matrix, t represents the displacement vector, t = [0, 0 , 0] ^T .

In the present invention, since the lens assembly includes a mounting bracket, a front lens, and a rear lens, when the lens assembly is mounted to the mobile terminal, the front lens covers the front camera of the mobile terminal, and the rear lens covers the rear camera of the mobile terminal. The front lens and the rear lens are wide-angle lenses or fisheye lenses; therefore, the front camera or the rear camera of the mobile terminal is controlled by a plane shooting application installed in the mobile terminal to cooperate with the front lens or the rear lens. The lens achieves flat video shooting at a low cost and does not require a professional camera.

Since the mobile terminal is equipped with the lens assembly provided by the present invention, and the method for implementing planar imaging by the mobile terminal and the lens assembly of the present invention, the state of the gyroscope is interpolated to obtain the rotation matrix of the corresponding planar video frame, Can get a more accurate rotation matrix. A planar video frame is then rotated according to the current rotation matrix to generate a stable planar video frame. Therefore, the planar video frame that can finally stabilize jitter is robust to large noise scenes and most motion scenes.

In addition, because the estimated angle of the acceleration count value is susceptible to interference (such as walking, walking, running, etc.), the accumulated error of the angular velocity becomes larger and larger as time passes. In the method for implementing planar shooting by the mobile terminal and the lens assembly of the present invention, since the extended Kalman filter is combined with the acceleration count value and the angular velocity value, the rotation vector of the current state is estimated, and Rodrigue rotation is performed according to the rotation vector of the current state. The formula calculates the current rotation matrix and then rotates the planar video frame, thus ultimately stabilizing the dithered planar video frame.

Referring to FIG. 18, an embodiment of the present invention further provides a method for implementing panoramic shooting by using a mobile terminal and a lens assembly, where the method includes:

S301, the mobile terminal starts a panoramic shooting application installed in the mobile terminal, and the mobile terminal is installed with the lens component provided by the embodiment of the present invention;

S302: The mobile terminal pre-establishes a spherical panoramic mathematical model, wherein the spherical panoramic mathematical model uses a center symmetry point of a spatial position of the front camera and the rear camera of the mobile terminal as a center of the ball, and the front camera and the rear camera are The spherical crowns obtained by the acquired angles of view are combined into a spherical surface.

S303. The front camera that controls the mobile terminal collects an image through the front lens of the lens component, and controls the rear camera of the mobile terminal to collect an image through the rear lens of the lens component;

S304. Map the two images into the spherical panoramic mathematical model according to the correspondence between the lens imaging height of the front lens and the rear lens and the angle of the field of view;

In the embodiment of the present invention, S304 may specifically include the following steps:

Converting the pixel points of each position in the two images into corresponding field angle positions in the spherical panoramic mathematical model according to the correspondence between the lens imaging height of the front lens and the rear lens and the angle of the field of view;

Combining the angle ω of the pixel at each position with respect to the center of the circle in the image, the position of the pixel in the original image is uniquely determined on the spherical panoramic mathematical model of any radius;

Each view angle position is filled with pixel data of the corresponding position on the spherical panoramic mathematical model.

S305: merging the overlapping regions of the two images in the spherical panoramic mathematical model, thereby splicing the two images into a spherical panoramic image covering all the viewing directions of the horizontal and vertical directions.

In the embodiment of the present invention, S305 may specifically be:

In the overlapping area of the two images in the spherical panoramic mathematical model, the pixel value of the point with the largest gray value is taken as the fused pixel value, thereby splicing the two images into a spherical surface covering all the viewing angles of the horizontal and vertical directions. Panoramic image.

The method may further include the step of anti-shake the spherical panoramic image, and specifically may include the following steps:

The mobile terminal acquires a current state time stamp, an acceleration count value, and an angular velocity value of the gyroscope in the mobile terminal in real time;

The mobile terminal uses the extended Kalman filter combined with the acceleration count value and the angular velocity value to estimate the amount of rotation of the mobile terminal to the world coordinate system;

The mobile terminal synchronizes the gyroscope time stamp with the time stamp of the spherical panoramic image;

The mobile terminal performs quaternion interpolation on the state of the gyroscope to obtain a rotation matrix corresponding to the spherical panoramic image;

The mobile terminal rotates the spherical panoramic image according to the current rotation matrix to generate a stable spherical panoramic image.

The step of performing anti-shake on the spherical panoramic image may further include the following steps:

The mobile terminal acquires the current state time stamp, the acceleration count value, and the angular velocity value of the mobile terminal in real time;

The mobile terminal estimates the rotation vector of the current state by using the extended Kalman filter combined with the acceleration count value and the angular velocity value;

The mobile terminal calculates the current rotation matrix by the Rodrigue rotation formula according to the rotation vector of the current state;

The mobile terminal rotates the spherical panoramic image according to the current rotation matrix to generate a stable spherical panoramic image.

Since the mobile terminal is equipped with the lens assembly provided by the embodiment of the present invention, the front camera of the mobile terminal can be controlled to collect an image through the front lens of the lens assembly, and the rear camera of the mobile terminal is controlled to pass through the rear lens of the lens assembly. Acquiring an image, and then mapping the two images into a spherical panoramic mathematical model according to the correspondence between the lens imaging height of the front lens and the rear lens and the angle of the field of view, two of the spherical panoramic mathematical models The overlapping regions of the image are fused to splicing the two images into a spherical panoramic image covering all viewing angles in the horizontal and vertical directions. The invention can collect the complete panoramic image at one time, can record the panoramic video, can obtain the panoramic image more completely and efficiently, and greatly simplifies the user operation process and reduces the cost of the panoramic shooting; in addition, based on the spherical panoramic mathematical model Therefore, the information collection capability of the camera is maximized, so that it has the ability to save the scene on the spot.

In the present invention, since the lens assembly includes a mounting bracket, a front lens, and a rear lens, when the lens assembly is mounted to the mobile terminal, the front lens covers the front camera of the mobile terminal, and the rear lens covers the rear camera of the mobile terminal. Both the front lens and the rear lens are wide-angle lenses or fisheye lenses; therefore, the front camera and the rear camera of the mobile terminal are simultaneously controlled by the panoramic shooting application installed in the mobile terminal, and the front lens and the rear are matched. The lens can achieve panoramic shooting at a low cost and does not require a dedicated panoramic camera.

The above is only the preferred embodiment of the present invention, and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection of the present invention. Within the scope.

Claims (18)

1. A lens assembly, characterized in that the lens assembly comprises a mounting bracket, a front lens and a rear lens, wherein the front and the back of the mounting bracket are provided with through holes, and the front lens is embedded in the front surface of the mounting bracket The hole and the rear lens are embedded in the through hole of the back of the mounting bracket, and the mounting bracket is detachably sleeved outside the camera area of the mobile terminal. When the lens assembly is mounted to the mobile terminal, the front lens covers the front camera of the mobile terminal. The rear lens covers the rear camera of the mobile terminal, and the front lens and the rear lens are wide-angle lenses or fisheye lenses; the front camera or the rear camera of the mobile terminal is controlled by a plane shooting application installed in the mobile terminal to shoot Planing video shooting with the front lens or the rear lens, or simultaneously controlling the front camera and the rear camera of the mobile terminal through a panoramic shooting application installed in the mobile terminal, together with the front lens Panoramic shooting with rear lens.
2. The lens assembly of claim 1 wherein said mobile terminal is a cell phone or tablet; said lens assembly does not obscure the display of the mobile terminal.
3. The lens assembly according to claim 1, wherein said front lens is coaxial or substantially parallel to an optical axis of a front camera of said mobile terminal, and said optical axis of said rear camera and said rear camera of said mobile terminal Coaxial or substantially parallel.
4. The lens assembly according to claim 1, wherein the front lens is embedded in a through hole of a front surface of the mounting bracket through a front lens sleeve, and the rear lens is embedded in a rear surface of the mounting bracket through a rear lens sleeve Through hole.
5. The lens assembly according to claim 1, wherein the front lens sleeve and the rear lens sleeve are integral with the mounting bracket, or the front lens sleeve and the rear lens sleeve are respectively Removably attached to the mounting bracket, or the front lens sleeve and the rear lens sleeve are fixedly coupled to the mounting bracket, respectively.
6. The lens assembly of claim 1 wherein said mounting bracket has an annular side wall defining a hollow cavity, the cavity having a shape that matches the camera area of the mobile terminal.
7. The lens assembly according to claim 1, wherein the front surface of the mounting bracket is further provided with a through hole corresponding to a position of a horn of the mobile terminal.
8. A method for implementing planar shooting by using a mobile terminal and a lens assembly, wherein the method includes:

   The mobile terminal launches a plane shooting application installed in the mobile terminal, the mobile terminal being mounted with the lens assembly according to any one of claims 1 to 7;

   The mobile terminal acquires a current state time stamp, an acceleration count value, and an angular velocity value of the gyroscope in the mobile terminal in real time;

   The mobile terminal uses the extended Kalman filter combined with the acceleration count value and the angular velocity value to estimate the amount of rotation of the mobile terminal to the world coordinate system;

   Controlling the front camera of the mobile terminal to acquire a planar video frame via a front lens of the lens assembly, or controlling a rear camera of the mobile terminal to acquire a planar video frame via a rear lens of the lens assembly;

   The mobile terminal synchronizes the gyroscope time stamp with the time stamp of the planar video frame;

   The mobile terminal performs quaternion interpolation on the state of the gyroscope to obtain a rotation matrix corresponding to the plane video frame;

   The mobile terminal rotates the planar video frame according to the current rotation matrix to generate a stable planar video frame.
9. The method according to claim 8, wherein the mobile terminal uses the extended Kalman filter in combination with the acceleration count value and the angular velocity value to estimate the amount of rotation of the mobile terminal to the world coordinate system, including:

   S1031, initial state rotation amount

   

   Where d ₀ is the initial measured acceleration value, g is the world coordinate system gravity vector; initial process covariance

   

   S1032, using the angular velocity value ω _{k to} calculate the state transition matrix Φ(ω _k ) at the Kth time;

   Φ(ω _k )=exp(−[ω _k ·Δt] _× ), where ω _k is the angular velocity value at the Kth moment, and Δt represents the sampling time interval of the gyro data;

   S1033. Calculating the covariance matrix Q _{k of the} state noise, updating the state rotation prior estimator

   

   And process covariance a priori estimation matrix

   

   ;

   

   Q _k is a covariance matrix of state noise;

   

   among them,

   

   Is the state rotation after-test estimator at time K-1;

   

   among them,

   

   Is the process covariance posterior estimation matrix at time K-1;

   S1034, updating the observed noise variance matrix R _k from the acceleration value d _k , calculating the observation transfer Jacobian matrix H _k , and calculating the current observation measurement and the estimation observation error e _k ;

   

   among them,

   

   

   α is the smoothing factor of the amount of acceleration change, and β is the influence factor of the acceleration mode length;

   

   Where h is the observation function, h(q,v)=q·g+v _k , g is the gravity vector in the world coordinate system, q is the state quantity, ie the amount of rotation from the world coordinate system to the gyro coordinate system, v _k is Measuring noise;

   

   S1035, updating the optimal Kalman gain matrix K _k at the kth time;

   

   S1036. Update the rotational posterior estimator of the mobile terminal to the world coordinate system according to the optimal Kalman gain matrix _Kk and the observation error _ek .

   

   And process covariance posterior estimation matrix

   

   

   
10. The method according to claim 9, wherein the time slot of the mobile terminal synchronization gyroscope time stamp and the planar video frame is specifically:

    The mobile terminal synchronizes the timestamp of the gyroscope with the time stamp of the planar video frame such that t _k ≥ t _j > t _k-1 , where t _j is the time stamp of the planar video frame and t _k is the time stamp of the Kth frame of the gyroscope. t _k-1 is the time stamp of the K-1 frame of the gyroscope.
11. The method of claim 10, wherein the telnet interpolation of the state of the gyro to obtain the rotation matrix of the corresponding planar video frame comprises:

    The mobile terminal calculates the relative amount of rotation of the neighboring gyroscope time stamp,

    

    Where r _k is the relative amount of rotation at the Kth moment,

    

    with

    

    The state posterior estimator for the kth and k-1th moments, that is, the amount of rotation of the world coordinate system to the gyro coordinate system;

    The mobile terminal performs quaternion interpolation to obtain the relative rotation amount of the plane video frame to the kth frame, R _j =γ·I+(1−γ)·r _k , where R _j is the relative rotation amount of the kth frame,

    

    The mobile terminal calculates a rotation matrix of the jth frame video in the plane video frame

    
12. The method of claim 11, wherein the mobile terminal rotates the planar video frame according to the current rotation matrix, and the generating the stable planar video frame comprises:

    The mobile terminal maps grid points on the latitude and longitude two-dimensional image to spherical coordinates;

    The mobile terminal traverses all points on the unit ball, and uses the current rotation matrix to rotate all points on the unit ball to generate a stable planar video frame;

    Among them, using the current rotation matrix to rotate all points on the unit ball, the following formula is used:

    

    Where [x, y, z] ^T represents the spherical coordinate before the unit circle is rotated, [x _new , y _new , z _new ] ^T represents the spherical coordinate after rotation, Q _j represents the current rotation matrix, and t represents the displacement vector. t=[0,0,0] ^T .
13. A method for implementing planar shooting by using a mobile terminal and a lens assembly, wherein the method includes:

    The mobile terminal launches a plane shooting application installed in the mobile terminal, the mobile terminal being mounted with the lens assembly according to any one of claims 1 to 7;

    The mobile terminal acquires the current state time stamp, the acceleration count value, and the angular velocity value of the mobile terminal in real time;

    The mobile terminal estimates the rotation vector of the current state by using the extended Kalman filter combined with the acceleration count value and the angular velocity value;

    The mobile terminal calculates a current rotation matrix by using a Rodrigue rotation formula according to a rotation vector of the current state;

    Controlling the front camera of the mobile terminal to acquire a planar video frame via a front lens of the lens assembly, or controlling a rear camera of the mobile terminal to acquire a planar video frame via a rear lens of the lens assembly;

    The mobile terminal rotates the planar video frame according to the current rotation matrix to generate a stable planar video frame.
14. The method according to claim 13, wherein the mobile terminal uses the extended Kalman filter in combination with the acceleration count value and the angular velocity value, and estimating the rotation vector of the current state specifically includes:

    The mobile terminal uses the angular velocity value to calculate the state transition matrix F k at time _k ; using the acceleration count value, combined with the gravity vector g in the reference coordinate system and the rotation matrix of the previous state to calculate the current time prediction margin

    

    A mobile terminal using the estimated state error covariance matrix _{P k-1 | k-1} , the current state of the state transition matrix F _k and Q process noise estimate the current state error covariance matrix P _{k | k-1;}

    The mobile terminal calculates the current state's optimal Kalman gain matrix _Kk using the estimated current state error covariance matrix _Pk|k-1 , the observation matrix _Hk, and the noise variance matrix R;

    The optimal Kalman gain matrix K _{k of the} mobile terminal according to the current state and the current time prediction margin

    

    Update current state estimate rotation vector

    
15. The method according to claim 14, wherein said mobile terminal calculates a state transition matrix F _k at time k using an angular velocity value; and uses an acceleration count value in combination with a gravity vector g in a reference coordinate system and a rotation matrix of a previous state; Calculate the current forecasting margin

    

    Specifically, the following steps are included:

    The mobile terminal initializes an initial state transition matrix, an initial prediction covariance matrix, and an initial observation matrix, wherein the initial state transition matrix

    

    Initial prediction covariance matrix

    

    Initial observation matrix

    

    The mobile terminal calculates the state transition matrix at time k

    

    Computing observation information matrix

    

    Where x _k-1 represents the state estimation of the mobile terminal at time k-1, and x _k represents the state estimation of the mobile terminal at time k,

    

    Indicates a partial differential symbol, f denotes a state equation function, x denotes the state of the mobile terminal, that is, a rotation angle in three axial directions, and h denotes an observation equation function,

    

    x _k-2 represents the state of the mobile terminal at time k-2, u _k-1 represents the angular velocity value at time k-1, and w _k-1 represents the process noise at time k-1,

    

    Indicates that the k-2 time is used to predict the estimated state of the mobile terminal at time k-1, x _k-1 represents the state of the mobile terminal at time k-1, u _k represents the angular velocity value at time k, and w _k represents the kth Process noise at the moment,

    

    Representing the k-1 time to predict the estimated state of the mobile terminal at time k, x _k-2 =[X _k-2 , Y _k-2 , Z _k-2 ] ^T , where X _k-2 , Y _{k- 2} , Z _k-2 represents the rotation angle of the reference system coordinate system on the X-axis, Y-axis, and Z-axis at the _k- 2th time, x _k-1 = [X _k-1 , Y _k-1 , Z _k-1 ] ^T , where X _k-1 , Y _k-1 , Z _k-1 represent the rotation angle of the reference frame coordinate system on the X-axis, the Y-axis, and the Z-axis at the k-1th time, and T represents the transposition;

    The mobile terminal projects the vertical downward gravity acceleration under the reference frame coordinate system to the rigid body coordinate system, and passes the formula

    

    Calculate the forecasting margin

    

    Where z _k is the acceleration count value after noise reduction processing using low-pass filtering at time k, and H _k is an observation information matrix, indicating that the observation equation z _k =h(x _k ,g,v _k ) is calculated using the current estimated state a Jacobian matrix, where g represents a vertically downward gravity vector in a reference coordinate system, g = [0, 0, -9.81] ^T , v _{k is} expressed as a measurement error;

    The use of the previous state estimate error covariance matrix _{P k-1 | k-1} , the current state of the state transition matrix F _k and Q process noise estimate the current state error covariance matrix P _{k | k-1} is specifically:

    Using formula

    

    The calculated state prediction estimates a covariance matrix P _k|k-1 , where P _k-1|k-1 represents an estimated covariance matrix of k-1 state, and Q _k represents a covariance matrix of process noise,

    

    Dt represents the sampling interval time of the gyroscope data, and F _k represents the state transition matrix at time k,

    

    Represents the transpose of F _k ;

    The calculating the optimal Kalman gain matrix K _k of the current state by using the estimated current state error covariance matrix P _k|k-1 , the observation matrix H _k and the noise variance matrix R specifically includes the following steps:

    Using the state prediction estimation covariance matrix P _k|k-1 to calculate the optimal Kalman gain matrix K _k at time _k ,

    

    R represents the noise covariance matrix,

    

    σ ² represents the noise variance, generally σ = 0.75, and H _k represents the observational information Jacobian matrix at time k.

    

    Represents the transpose of H _k ;

    The optimal Kalman gain matrix K _k according to the current state and the current time prediction margin

    

    Update current state estimate rotation vector

    

    Specifically, the following steps are included:

    The update state estimate obtains the rotation vector of the current state obtained by merging the acceleration count value and the angular velocity value at time k.

    

    Update the estimated covariance matrix P _k|k , P _k|k =(IK _k ·H _k )P _k|k-1 , where I is the identity matrix and P _k|k is the estimated error covariance matrix needed at the next moment P _k-1|k-1 .
16. A method for implementing panoramic shooting in cooperation with a lens assembly, wherein the method comprises:

    The mobile terminal launches a panoramic shooting application installed in the mobile terminal, the mobile terminal being mounted with the lens assembly according to any one of claims 1 to 7;

    The mobile terminal pre-establishes a spherical panoramic mathematical model, wherein the spherical panoramic mathematical model is centered on the center of the spatial position of the front camera and the rear camera of the mobile terminal, and can be collected by the front camera and the rear camera. The spherical crowns obtained by the angles of view are combined into a spherical surface;

    The front camera controlling the mobile terminal collects an image through the front lens of the lens assembly, and controls the rear camera of the mobile terminal to collect an image through the rear lens of the lens assembly;

    According to the correspondence between the lens imaging height of the front lens and the rear lens and the angle of the field of view, the two images are mapped into the spherical panoramic mathematical model;

    The overlapping regions of the two images in the spherical panoramic mathematical model are fused to splicing the two images into a spherical panoramic image covering all the viewing angles of the horizontal and vertical directions.
17. The method according to claim 16, wherein the mapping between the two images to the spherical panoramic mathematical model is based on the correspondence between the lens imaging height of the front lens and the rear lens and the angle of the field of view. include:

    Converting the pixel points of each position in the two images into corresponding field angle positions in the spherical panoramic mathematical model according to the correspondence between the lens imaging height of the front lens and the rear lens and the angle of the field of view;

    Combining the angle of each pixel at each position with respect to the center of the circle, the position of the pixel in the original image is uniquely determined on a spherical panoramic mathematical model of arbitrary radius;

    Each view angle position is filled with pixel data of the corresponding position on the spherical panoramic mathematical model.
18. The method according to claim 17, wherein said merging of overlapping regions of two images in a spherical panoramic mathematical model, thereby splicing the two images into a spherical surface covering all viewing angles in horizontal and vertical directions The panoramic image is specifically:

    In the overlapping area of the two images in the spherical panoramic mathematical model, the pixel value of the point with the largest gray value is taken as the fused pixel value, thereby splicing the two images into a spherical surface covering all the viewing angles of the horizontal and vertical directions. Panoramic image.

Images