WO2021035525A1 - 图像处理方法和装置、电子设备、计算机可读存储介质 - Google Patents

图像处理方法和装置、电子设备、计算机可读存储介质 Download PDF

Info

Publication number
WO2021035525A1
WO2021035525A1 PCT/CN2019/102801 CN2019102801W WO2021035525A1 WO 2021035525 A1 WO2021035525 A1 WO 2021035525A1 CN 2019102801 W CN2019102801 W CN 2019102801W WO 2021035525 A1 WO2021035525 A1 WO 2021035525A1
Authority
WO
WIPO (PCT)
Prior art keywords
information
current
camera
image
posture
Prior art date
Application number
PCT/CN2019/102801
Other languages
English (en)
French (fr)
Inventor
贾玉虎
Original Assignee
Oppo广东移动通信有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Oppo广东移动通信有限公司 filed Critical Oppo广东移动通信有限公司
Priority to CN201980096840.3A priority Critical patent/CN113875221A/zh
Priority to PCT/CN2019/102801 priority patent/WO2021035525A1/zh
Priority to EP19943025.7A priority patent/EP4013030A4/en
Publication of WO2021035525A1 publication Critical patent/WO2021035525A1/zh
Priority to US17/673,249 priority patent/US20220174217A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/73—Deblurring; Sharpening
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H04N23/681—Motion detection
    • H04N23/6811—Motion detection based on the image signal
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H04N23/681—Motion detection
    • H04N23/6812—Motion detection based on additional sensors, e.g. acceleration sensors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H04N23/682—Vibration or motion blur correction
    • H04N23/683—Vibration or motion blur correction performed by a processor, e.g. controlling the readout of an image memory
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20172—Image enhancement details
    • G06T2207/20201—Motion blur correction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30244—Camera pose

Definitions

  • This application relates to the field of imaging, in particular to an image processing method and device, electronic equipment, and computer-readable storage media.
  • the embodiments of the present application provide an image processing method, device, electronic device, and computer-readable storage medium, which can improve the overall anti-shake effect.
  • An image processing method including:
  • a method for acquiring in-depth information including:
  • the target pose information and the first depth information of each pixel in the image to be processed in the current pose determine the second depth information of each pixel in the image to be processed in the target pose.
  • An image processing device including:
  • An attitude acquisition module configured to acquire current attitude information of the camera, where the current attitude information includes current angular velocity information
  • a conversion module for converting the current posture information into target posture information
  • the depth information acquisition module is used to acquire the first depth information of each pixel in the image to be processed in the current posture
  • the determining module is configured to determine the second depth information of each pixel in the image to be processed in the target pose according to the target pose information and the first depth information of each pixel in the image to be processed in the current pose In-depth information
  • the target image determination module is used to obtain the first internal parameter information of the camera, according to the current posture information, the first depth information, the target posture information, the second depth information, and the first internal parameter information Re-projection processing is performed on the image to be processed to obtain a target image.
  • a depth information acquisition device including:
  • An attitude acquisition module configured to acquire current attitude information of the camera, where the current attitude information includes current angular velocity information
  • a conversion module for converting the current posture information into target posture information
  • the depth information acquisition module is used to acquire the first depth information of each pixel in the image to be processed in the current posture
  • the determining module is configured to determine the second depth information of each pixel in the image to be processed in the target pose according to the target pose information and the first depth information of each pixel in the image to be processed in the current pose In-depth information.
  • An electronic device includes a memory and a processor, and a computer program is stored in the memory.
  • the processor causes the processor to perform the operations of the image processing method or the depth information acquisition method.
  • a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, realizes the operations of the above-mentioned image processing method or depth information acquisition method.
  • the above-mentioned image processing method and device, electronic equipment, and computer-readable storage medium are used to obtain the current posture information of the camera, the current posture information includes the current angular velocity information, the current posture information is converted into target posture information, and the current posture to be processed is obtained
  • the first depth information of each pixel in the image according to the target pose information and the first depth information of each pixel in the image to be processed in the current pose, determine the second depth information of each pixel in the image to be processed in the target pose , Acquire the first internal parameter information of the camera, and perform reprojection processing on the image to be processed according to the current posture information, first depth information, target posture information, second depth information, and first internal parameter information to obtain the target image, which can be realized for each pixel Point for targeted anti-shake, making the image anti-shake effect more stable.
  • Figure 1 is a block diagram of the internal structure of an electronic device in an embodiment
  • FIG. 2 is a flowchart of an image processing method in an embodiment
  • Figure 3 is a schematic diagram of a small hole imaging in an embodiment
  • FIG. 4 is a flowchart of an operation of obtaining the first depth information of each pixel in the image to be processed in the current pose in an embodiment
  • FIG. 5 is a flowchart of the operation of obtaining the first internal parameter information of the camera in an embodiment
  • FIG. 6 is a flowchart of the operation of obtaining the first internal parameter information of the camera in another embodiment
  • FIG. 7 is a schematic diagram of optical image stabilization in an embodiment
  • Figure 8 is a schematic diagram of an image processing method in an embodiment
  • Fig. 9 is a flowchart of a method for acquiring depth information in an embodiment
  • Figure 10 is a structural block diagram of an image processing device in an embodiment
  • Fig. 11 is a structural block diagram of a depth information acquiring device in an embodiment
  • Fig. 12 is a schematic diagram of the internal structure of an electronic device in an embodiment.
  • the image processing method and depth information acquisition method in the embodiments of the present application can be applied to electronic devices.
  • the electronic device may be a computer device with a camera, a personal digital assistant, a tablet computer, a smart phone, a wearable device, etc.
  • the above-mentioned electronic device may include an image processing circuit, which may be implemented by hardware and/or software components, and may include various processing units that define an ISP (Image Signal Processing, image signal processing) pipeline.
  • Fig. 1 is a schematic diagram of an image processing circuit in an embodiment. As shown in FIG. 1, for ease of description, only various aspects of the image processing technology related to the embodiments of the present application are shown.
  • the image processing circuit includes a first ISP processor 130, a second ISP processor 140, and a control logic 150.
  • the first camera 110 includes one or more first lenses 112 and a first image sensor 114.
  • the first image sensor 114 may include a color filter array (such as a Bayer filter).
  • the first image sensor 114 may acquire the light intensity and wavelength information captured by each imaging pixel of the first image sensor 114, and provide information that can be obtained by the first ISP.
  • the second camera 120 includes one or more second lenses 122 and a second image sensor 124.
  • the second image sensor 124 may include a color filter array (such as a Bayer filter).
  • the second image sensor 124 may acquire the light intensity and wavelength information captured by each imaging pixel of the second image sensor 124, and provide information that can be used by the second ISP.
  • a set of image data processed by the processor 140 includes one or more first lenses 112 and a first image sensor 114.
  • the first image collected by the first camera 110 is transmitted to the first ISP processor 130 for processing.
  • the statistical data of the first image (such as image brightness, image contrast value) , The color of the image, etc.) are sent to the control logic 150.
  • the control logic 150 can determine the control parameters of the first camera 110 according to the statistical data, so that the first camera 110 can perform operations such as auto-focusing and auto-exposure according to the control parameters.
  • the first image can be stored in the image memory 160 after being processed by the first ISP processor 130, and the first ISP processor 130 can also read the image stored in the image memory 160 for processing.
  • the first image can be directly sent to the display 170 for display after being processed by the ISP processor 130, and the display 170 can also read the image in the image memory 160 for display.
  • the first ISP processor 130 processes image data pixel by pixel in multiple formats.
  • each image pixel may have a bit depth of 8, 10, 12, or 14 bits, and the first ISP processor 130 may perform one or more image processing operations on the image data and collect statistical information about the image data.
  • the image processing operations can be performed with the same or different bit depth accuracy.
  • the image memory 160 may be a part of a memory device, a storage device, or an independent dedicated memory in an electronic device, and may include DMA (Direct Memory Access) features.
  • DMA Direct Memory Access
  • the first ISP processor 130 may perform one or more image processing operations, such as temporal filtering.
  • the processed image data can be sent to the image memory 160 for additional processing before being displayed.
  • the first ISP processor 130 receives the processed data from the image memory 160, and performs image data processing in the RGB and YCbCr color spaces on the processed data.
  • the image data processed by the first ISP processor 130 may be output to the display 170 for viewing by the user and/or further processed by a graphics engine or a GPU (Graphics Processing Unit, graphics processor).
  • the output of the first ISP processor 130 can also be sent to the image memory 160, and the display 170 can read image data from the image memory 160.
  • the image memory 160 may be configured to implement one or more frame buffers.
  • the statistical data determined by the first ISP processor 130 may be sent to the control logic 150.
  • the statistical data may include first image sensor 114 statistical information such as automatic exposure, automatic white balance, automatic focus, flicker detection, black level compensation, and shading correction of the first lens 112.
  • the control logic 150 may include a processor and/or a microcontroller that executes one or more routines (such as firmware). The one or more routines can determine the control parameters and the first camera 110 of the first camera 110 based on the received statistical data.
  • the control parameters of the first camera 110 may include gain, integration time of exposure control, anti-shake parameters, flash control parameters, control parameters of the first lens 112 (for example, focal length for focusing or zooming), or a combination of these parameters.
  • the ISP control parameters may include gain levels and color correction matrices for automatic white balance and color adjustment (for example, during RGB processing), and the first lens 112 shading correction parameters.
  • the second image captured by the second camera 120 is transmitted to the second ISP processor 140 for processing.
  • the statistical data of the second image (such as image brightness, image The contrast value of the image, the color of the image, etc.) are sent to the control logic 150, and the control logic 150 can determine the control parameters of the second camera 120 according to the statistical data, so that the second camera 120 can perform operations such as auto focus and auto exposure according to the control parameters .
  • the second image can be stored in the image memory 160 after being processed by the second ISP processor 140, and the second ISP processor 140 can also read the image stored in the image memory 160 for processing.
  • the second image can be directly sent to the display 170 for display after being processed by the ISP processor 140, and the display 170 can also read the image in the image memory 160 for display.
  • the second camera 120 and the second ISP processor 140 may also implement the processing procedures described by the first camera 110 and the first ISP processor 130.
  • the first camera 110 may be a color camera
  • the second camera 120 may be a TOF (Time Of Flight) camera or a structured light camera.
  • the TOF camera can obtain the TOF depth map
  • the structured light camera can obtain the structured light depth map.
  • the first camera 110 and the second camera 120 may both be color cameras. Obtain binocular depth maps through two color cameras.
  • the first ISP processor 130 and the second ISP processor 140 may be the same ISP processor.
  • the image to be processed can be obtained through the preview screen, and the image to be processed is sent to the ISP processor.
  • the ISP processor can obtain the current posture information of the camera when shooting, the current posture information includes the current angular velocity information; then the current posture information is converted into target posture information; the first position of each pixel in the image to be processed in the current posture is obtained Depth information, according to the target pose information and the first depth information of each pixel in the image to be processed in the current pose, determine the second depth information of each pixel in the image to be processed in the target pose; obtain the first internal parameter information of the camera According to the current posture information, the first depth information, the target posture information, the second depth information, and the first internal reference information, the image to be processed is reprojected to obtain the target image.
  • the accurate depth information of each pixel in the current pose and converting the accurate depth information in the current pose to the depth information in the target pose, it is possible to perform targeted reprojection processing on each pixel to improve the image
  • Fig. 2 is a flowchart of an image processing method in an embodiment. As shown in Figure 2, the image processing method includes:
  • Operation 202 Acquire current posture information of the camera, where the current posture information includes current angular velocity information.
  • the current posture information refers to information that characterizes the current posture of the camera, and the current posture information includes current angular velocity information.
  • the current angular velocity information can be converted into the rotation matrix of the camera in the world coordinate system. Therefore, the rotation matrix can be used to characterize the current posture information of the camera.
  • the ISP processor or the central processing unit of the electronic device can obtain the three-axis angular velocity of the camera through the gyroscope, and the three-axis angular velocity is corrected and integrated in the time domain to output the three-axis angular velocity information.
  • the current posture information is converted into target posture information.
  • the target posture refers to the posture of the camera in a stable state after reprojecting the image to be processed taken in the current posture.
  • the target posture information refers to the information that can characterize the target posture.
  • the ISP processor or central processing unit of the electronic device obtains the current posture information of the camera, it can make predictions based on the current posture, and determine the target posture corresponding to the current posture. Further, the current posture information can be converted into target posture information through the target posture prediction algorithm.
  • first depth information of each pixel in the image to be processed in the current posture is obtained.
  • the first depth information refers to accurate depth information corresponding to each pixel in the image to be processed in the current posture.
  • the image to be processed may be a complete image or a partial image in an image.
  • the ISP processor or the central processing unit of the electronic device can obtain the first depth information corresponding to each pixel in the image to be processed in the current posture.
  • Operation 208 according to the target posture information and the first depth information of each pixel in the image to be processed in the current posture, determine the second depth information of each pixel in the image to be processed in the target posture.
  • the second depth information refers to the depth information corresponding to each pixel of the image to be processed in the target pose.
  • the ISP processor or central processor of the electronic device converts the current posture information into target posture information to obtain the target posture corresponding to the current posture.
  • the first depth information of each pixel in the image to be processed under the current posture can be transformed into the second depth information of each pixel in the image to be processed under the target posture.
  • the first internal parameter information of the camera is obtained, and the image to be processed is reprojected according to the current posture information, the first depth information, the target posture information, the second depth information, and the first internal parameter information to obtain the target image.
  • the first internal parameter information refers to the real-time internal parameter information of the camera acquired in the current posture.
  • the ISP processor or the central processing unit of the electronic device can obtain the first internal parameter information of the camera in the current posture. Then, input the current posture information, the first depth information, the target posture information, the second depth information, and the first internal parameter information into the reprojection mapping algorithm, and the reprojection of each pixel of the image to be processed can be obtained. Pixel coordinates.
  • the target image can be obtained by outputting the image according to the reprojected pixel coordinates of each pixel of the image to be processed.
  • the image processing method in this embodiment obtains the current posture information of the camera, the current posture information includes the current angular velocity information, converts the current posture information into target posture information, and obtains the first position of each pixel in the image to be processed in the current posture.
  • a depth information according to the target pose information and the first depth information of each pixel in the image to be processed in the current pose, determine the second depth information of each pixel in the image to be processed in the target pose, and obtain the first internal parameters of the camera Information, according to the current posture information, the first depth information, the target posture information, the second depth information and the first internal reference information to reproject the image to be processed to obtain the target image, which can achieve targeted anti-shake for each pixel , Which makes the image stabilization effect more stable.
  • f is the focal length of the camera
  • c x and c y are the offset of the camera center coordinates
  • K is the camera internal parameter matrix including the parameters f
  • c x and c y is the world coordinate of the point p in the three-dimensional space
  • the coordinates under the system (u, v) are the pixel coordinates of the pixel point p′ of p
  • R and T are the rotation matrix and translation matrix of the camera in the world coordinate system, which represent the current posture information of the camera
  • Z c is the three-dimensional
  • mapping relationship between three-dimensional space points and image pixel coordinates in the imaging process is as follows:
  • the image to be processed in the current posture can be reprojected to the posture of the camera with the posture of R′ and T′ (that is, the target posture), and then the corresponding Z c ′ can be calculated according to the target posture.
  • the process of projection can be described as:
  • the current posture information further includes at least one of current altitude information, current orientation information, and current geographic location information;
  • the acquiring current posture information of the camera includes:
  • Obtain the current angular velocity information of the camera and obtain at least one of the current altitude information, current position information, and current geographic location information of the camera; the current altitude information, the current position information, and the current geographic location At least one of the information is fused with the current angular velocity information to obtain the current posture information of the camera.
  • the current height information refers to the height of the camera from the ground plane in the current posture.
  • the current azimuth information refers to the direction and position of the camera in the current posture, for example: east, south, west, north and other azimuths.
  • the current geographic location information refers to the geographic coordinates of the camera in the current posture.
  • the ISP processor or central processing unit of the electronic device can obtain the angular velocity information of the camera in the current attitude through the gyroscope, the height of the camera from the ground plane in the current attitude can be detected by the altimeter, and the current position of the camera can be acquired through the compass.
  • the position in the attitude can also be obtained through GPS (Global Positioning System, Global Positioning System) positioning to obtain the geographic coordinates of the camera in the current attitude.
  • GPS Global Positioning System, Global Positioning System
  • the ISP processor or central processing unit of the electronic device can select at least one of the current altitude information, the current position information, and the current geographic location information, and perform fusion processing on the selected information and the current angular velocity information to obtain the The current posture information of the camera.
  • the current altitude information and the current angular velocity information are fused; the current azimuth information and the current angular velocity information are fused; the current geographic location information and the current angular velocity information are fused; the current altitude information is fused .
  • the current azimuth information and the current angular velocity information are fused; the current altitude information, the current geographic location information, and the current angular velocity information are fused; the current location information, the current geographic location information, and the current angular velocity information Perform fusion processing; fusion processing of current altitude information, current azimuth information, current geographic location information, and current angular velocity information can obtain the current posture information of the camera.
  • the selected information and the current angular velocity information can be processed by Kalman filtering to realize the fusion of the information and obtain the current posture information of the camera.
  • the Kalman filter is essentially a data fusion algorithm that fuses data with the same purpose, from different sensors, and with different units to obtain a more accurate measurement value of the purpose.
  • the image processing method in this embodiment by acquiring the current angular velocity information of the camera, and acquiring at least one of the current height information, current orientation information, and current geographic location information of the camera, a variety of characterization cameras can be obtained. Different information about the current posture. At least one of the current altitude information, the current orientation information, and the current geographic location information is fused with the current angular velocity information to obtain the current posture information of the camera, which provides multiple ways to obtain the current posture information of the camera. Through the fusion of different information, the current posture information of the camera can be obtained more accurately.
  • the current posture information includes current acceleration information; the method further includes: acquiring current acceleration information of the camera;
  • At least one of the current height information, current position information, and current geographic location information is fused with the current angular velocity information to obtain the current posture information of the camera, including: combining the current height information and current position information of the camera Perform fusion processing with at least one of the current geographic location information, the current angular velocity information, and the current acceleration information to obtain the current posture information of the camera.
  • the current acceleration information refers to the acceleration of the camera in the current posture.
  • the current acceleration information can be converted into the translation matrix of the camera in the world coordinate system. Therefore, the translation matrix can be used as a component of the current posture information of the camera.
  • the ISP processor or the central processing unit of the electronic device can obtain the three-axis acceleration through an accelerometer, and the three-axis acceleration is corrected and integrated in the two time domains to obtain the three-axis position information of the camera.
  • the three-axis position information can be converted into the translation matrix of the camera in the world coordinate system.
  • at least one can be selected from the current height information of the camera, the current position information, and the current geographic location information, and the selected information can be fused with the current angular velocity information and the current acceleration information.
  • the fusion processing can be realized by fusion algorithms such as Kalman filtering algorithm.
  • the current altitude information, the current acceleration information and the current angular velocity information are fused; the current azimuth information, the current acceleration information, and the current angular velocity information are fused; the current geographic location information, the current acceleration Information and current angular velocity information are fused; current altitude information, current azimuth information, current acceleration information, and current angular velocity information are fused; current altitude information, current geographic location information, and current acceleration information Perform fusion processing with current angular velocity information; merge current azimuth information, current geographic location information, current acceleration information, and current angular velocity information; combine current altitude information, current azimuth information, and current geographic location information , The current acceleration information and the current angular velocity information are fused to obtain the current posture information of the camera.
  • the image processing method in this embodiment by acquiring current acceleration information, at least one of the current height information, current orientation information, and current geographic location information of the camera is combined with the current angular velocity information, and current acceleration information. Fusion processing provides a variety of ways to obtain the current posture information of the camera.
  • the current acceleration information is also used as a reference for obtaining the current posture information, so that the current posture information of the camera can be obtained more accurately.
  • the ISP processor or central processing unit of the electronic device can also obtain the acceleration of gravity through the gravity sensor, and select at least one of the current height information, current position information, and current geographic location information of the camera and the acceleration of gravity. , And the current angular velocity information are fused to obtain the current posture information of the camera.
  • converting the current posture information into target posture information includes:
  • Acquire current sports scene information determine target posture information according to the current sports scene information and the current posture information.
  • the current motion scene information refers to the motion state of the camera during the shooting process. For example, when the user uses the camera to move the camera horizontally to shoot, the camera is in a state of horizontal movement.
  • the ISP processor or the central processing unit of the electronic device can determine the movement state of the camera during the shooting process according to the current orientation information of the camera and the current geographic location information. Then, according to the current motion state of the camera, determine the direction in which the jitter needs to be eliminated under the current posture. For example, when shooting horizontally, it is necessary to eliminate the jitter in the roll and pitch directions as much as possible, while retaining the jitter in the yaw direction.
  • pitch is the direction of rotation around the x-axis
  • the angle of rotation is the pitch angle.
  • Yaw is the direction of rotation around the y axis
  • the angle of rotation is the yaw angle
  • Roll is the direction of rotation around the z axis
  • the angle of rotation is the roll angle.
  • the information in the direction in which the jitter needs to be eliminated in the current posture information can be subjected to low-pass filtering to remove the high frequency and retain the low frequency.
  • the information in the direction where the jitter does not need to be eliminated may not be processed, and the posture information for eliminating jitter according to the current motion state of the camera is output, and the target posture information can be obtained.
  • the traditional target pose prediction method only obtains the target pose by predicting the current pose.
  • the direction in which the jitter is to be eliminated and the direction in which the jitter does not need to be eliminated in the current posture can be determined.
  • Targeted processing of the information in all directions in the current posture information makes the predicted target posture information more accurate and more in line with the actual shooting scene.
  • determining the target posture information according to the current motion scene information and the current posture information includes: converting the current posture information to a frequency domain space; according to the current motion scene information, converting the frequency domain The current attitude information in the space is processed by low-pass filtering; the current attitude information after the low-pass filtering is converted to the time domain space to obtain the target attitude information.
  • the time domain is the time domain, which describes the relationship of mathematical functions or physical signals to time.
  • the time domain waveform of a signal can express the change of the signal over time.
  • the frequency domain is the frequency domain
  • the independent variable of the frequency domain is the frequency, that is, the horizontal axis is the frequency
  • the vertical axis is the amplitude of the frequency signal, that is, the spectrogram.
  • the spectrogram describes the frequency structure of the signal and the relationship between the frequency and the amplitude of the frequency signal.
  • the ISP processor or the central processing unit obtains the angular velocity information through the gyroscope, and integrates the angular velocity information in the time domain to obtain the current attitude information in the time domain space. Then, the ISP processor or the central processing unit can convert the current attitude information in the time domain space into the frequency domain space.
  • the ISP processor or the central processor determines the direction of the jitter to be eliminated in the current posture information according to the motion state of the camera, and performs low-pass filtering on the information in the direction of jitter to be eliminated in the current posture information to remove the current posture information
  • the amount of high frequency in the information in the direction in which the jitter is to be eliminated, and the amount of low frequency is retained.
  • the information in the direction in which the jitter does not need to be eliminated in the current posture information may not be processed, and the current posture information of the low frequency in the frequency domain space after low-pass filtering processing is obtained.
  • the current attitude information of the low-frequency quantity after the low-pass filter processing of the ISP processor or the central processing unit is converted to the time domain space, and the information after the low-pass filter processing is converted from the frequency domain space to the time domain space to obtain the attitude
  • the information is the target posture information.
  • the image processing method in this embodiment converts the current posture information to the frequency domain space, and performs low-pass filtering processing on the current posture information in the frequency domain space according to the current motion scene information of the camera, so that the jitter can be eliminated in the direction to be eliminated.
  • Information is processed in a targeted manner.
  • the current posture information can be accurately converted into target posture information according to the current motion scene information, so that the posture of the image to be processed in a stable state can be accurately predicted, that is, the target posture.
  • the acquiring current sports scene information includes:
  • the ISP processor or central processing unit of the electronic device can obtain the current position information of the camera through a compass, and obtain the current geographic location information of the camera through a GPS positioning system.
  • the current position information of the camera and the current geographic location information are processed by Kalman filtering to realize the fusion of the two kinds of information, thereby obtaining the current motion scene information.
  • the change in the direction and location of the camera and the geographic coordinates can be determined.
  • the current azimuth information and the current geographic location information are fused and processed, and the movement state of the camera maintained during the shooting process can be intuitively and accurately determined according to the changes in the direction and location of the camera and the geographic coordinates.
  • the acquiring the first depth information of each pixel in the image to be processed in the current pose includes:
  • the initial depth information is the depth information of each pixel in the image to be processed in the current posture acquired by the depth camera;
  • the third depth information is the depth information of each pixel in the image to be processed in the current attitude obtained through the dual-camera parallax ranging method;
  • the fourth depth information is acquired through the phase focusing method in the current attitude. Processing the depth information of each pixel in the image; performing fusion correction processing on the acquired at least two types of depth information to obtain the first depth information of each pixel in the image to be processed in the current posture.
  • the fusion correction processing refers to the analysis and selection of different information of the same image to be processed obtained in different ways, so as to fuse different information obtained in different ways to the same image.
  • the first depth information refers to depth information with higher precision for each pixel in the image to be processed in the current posture.
  • the ISP processor or central processing unit of the electronic device can obtain the initial depth information of each pixel in the image to be processed in the current posture through a depth camera, and obtain the initial depth information of each pixel in the current posture through a dual-camera parallax ranging method.
  • the third depth information of each pixel in the image to be processed may also be obtained by phase focusing in the current posture of the fourth depth information of each pixel in the image to be processed.
  • the ISP processor or the central processing unit of the electronic device can obtain the depth information of each pixel in the same image to be processed in the current posture by at least two of the above three methods to obtain the initial At least two of depth information, third depth information, and fourth depth information.
  • the ISP processor or the central processing unit of the electronic device can merge and superimpose the depth information obtained in at least two ways, so as to integrate the at least two kinds of depth information obtained.
  • the depth information of each pixel obtained after the fusion correction processing is the first depth information of each pixel in the image to be processed in the current posture.
  • the depth information of the same image to be processed in the current posture is obtained in at least two ways to obtain at least two types of depth information.
  • the depth information of the same image to be processed in the current posture is obtained in at least two ways to obtain at least two types of depth information.
  • the initial depth information of the image to be processed can be acquired through a TOF camera, or through a dual-camera camera, or through a structured light camera, an infrared camera, etc., to acquire the initial depth information of the image to be processed.
  • the initial image can be captured by the TOF camera, and the low-resolution area in the initial image can be interpolated, or the entire initial image can be interpolated to obtain a higher resolution than the initial image.
  • Interpolation processing methods include, but are not limited to, bi-square interpolation and bi-cubic interpolation.
  • the depth information includes a depth value; and performing fusion correction processing on the acquired at least two types of depth information to obtain the first depth information of each pixel in the image to be processed in the current posture includes:
  • the ISP processor or the central processing unit of the electronic device can obtain the depth information of each pixel in the same image to be processed in the current posture in three ways, and can obtain three kinds of depth information corresponding to each pixel. That is, each pixel in the image to be processed corresponds to initial depth information, third depth information, and fourth depth information.
  • the ISP processor or the central processing unit of the electronic device can obtain at least two types of depth information corresponding to each pixel, and calculate the average value of the obtained at least two types of depth information to obtain the average value corresponding to each pixel.
  • the average value corresponding to each pixel point is used as the first depth value of each pixel point in the image to be processed in the current posture.
  • the depth information selected by all pixels is the depth information obtained in the same way, for example, each pixel obtains its own corresponding initial depth information and third depth information; each pixel obtains its corresponding initial depth Information and fourth depth information; each pixel obtains its corresponding third depth information and fourth depth information; each pixel obtains its corresponding initial depth information, third depth information, and fourth depth information.
  • the image processing method in this embodiment determines the average value of at least two types of depth information corresponding to each pixel, and uses the average value corresponding to each pixel as the first depth value of each pixel in the image to be processed in the current posture, Provides a variety of ways to get the first depth value. Through the fusion of different depth information, richer detailed information of the image to be processed can be obtained, so that the calculated depth information of each pixel in the image to be processed is more accurate.
  • the acquiring first depth information of each pixel in the image to be processed in the current pose includes:
  • initial depth information of each pixel in the image to be processed in the current posture is obtained.
  • the ISP processor or central processing unit of the electronic device can shoot the image to be processed of the same scene in the current posture through the depth camera, and can directly obtain the initial depth information of each pixel in the image to be processed.
  • the depth camera may be a TOF camera, a dual-camera camera, a structured light camera, and the like.
  • the ISP processor or central processing unit of the electronic device can directly capture the image to be processed in the current posture through the TOF camera, dual-camera camera and structured light camera and other image acquisition devices. No other conversion processing is required, and the image to be processed can be obtained simply and quickly. In the depth information of each pixel, thereby improving the speed of image processing.
  • the ISP processor or central processing unit of the electronic device can determine the same scene currently being shot, and after optical focusing is performed by driving the lens through a motor, the focus position (ie, focus area) in the current posture can be determined, which can be based on a preset
  • the corresponding relationship between the focus position and the focus value, and the mapping relationship between the focus value and the depth information can obtain the depth information corresponding to each pixel in the focus area in the current posture.
  • the initial depth information and the depth information of the focus area in the current attitude are fused and corrected to obtain the first depth information of each pixel in the image to be processed in the current attitude.
  • the accuracy of the depth information of the image to be processed directly captured by the depth camera is not as high as the accuracy of obtaining the depth information during optical focusing. That is, the accuracy of the initial depth information of each pixel in the image to be processed in the current attitude acquired by the depth camera is lower than the accuracy of the depth information of the focus area in the current attitude after optical focusing.
  • the ISP processor or central processor of the electronic device can merge and superimpose the depth information of the focus area in the current attitude and the depth information of the image to be processed directly captured by the depth camera, so that each pixel in the image to be processed in the current attitude
  • the depth information of can reach the accuracy of the depth information of the focus area, so as to obtain the first depth information of each pixel in the image to be processed in the current posture.
  • the image processing method in this embodiment by acquiring the initial depth information of each pixel in the image to be processed in the current attitude, and acquiring the depth information of the focus area in the current attitude, two types of the same image to be processed in the current attitude can be obtained. Depth information of different precisions.
  • the initial depth information and the depth information of the focus area in the current posture are fused and corrected to obtain the first depth information of each pixel in the image to be processed under the current posture.
  • High-precision local depth information can be applied to Each part of the image to be processed is thus obtained more accurate depth information of the image to be processed, that is, the first depth information of each pixel in the image to be processed in the current posture is obtained.
  • the depth information includes a depth value
  • the initial depth information and the depth information of the focus area in the current pose are fused and corrected to obtain the first pixel value of each pixel in the image to be processed in the current pose.
  • Depth information including: determining the pixel points in the image to be processed in the current posture that match the pixels of the focus area in the current posture; determining the depth value of the focus area and the pixels that match in the image to be processed The difference or ratio between the initial depth values of the points; according to the initial depth information of each pixel in the image to be processed in the current attitude and the difference or ratio, it is determined that each pixel in the image to be processed in the current attitude The first depth information of the pixel.
  • the ISP processor or the central processing unit of the electronic device can obtain the depth information of the focus area in the current posture, and obtain the partial depth information of the partial image in the focus area in the image to be processed.
  • the ISP processor or central processing unit of the electronic device can further determine the depth value of the focus area. Next, match each pixel in the partial image of the focus area with each pixel in the image to be processed in the current posture, and determine the pixels of the partial image in the focus area and the corresponding pixels in the image to be processed under the current posture point.
  • each pixel there are multiple pixels in the focus area, and the depth value of each pixel is the same.
  • the multiple pixels refer to at least two pixels.
  • the ISP processor or central processing unit of the electronic device obtains the initial depth value of the successfully matched pixel in the image to be processed in the current posture, and obtains the depth value of the focus area, and calculates the difference between the two successfully matched pixels. Difference value, the difference value corresponding to each successfully matched pixel can be obtained. Or calculate the ratio between the two successfully matched pixels to obtain the ratio corresponding to each successfully matched pixel.
  • the ISP processor or the central processing unit of the electronic device can determine the first adjustment value according to the difference corresponding to each pixel that is successfully matched.
  • the initial depth value of each pixel in the image to be processed in the current posture is added to the first adjustment value to obtain the first depth value of each pixel in the image to be processed in the current posture.
  • the first depth value of is the first depth information corresponding to each pixel in the image to be processed in the current pose.
  • the ISP processor or the central processing unit of the electronic device may determine the second adjustment value according to the ratio corresponding to each successfully matched pixel. Multiply the initial depth value of each pixel in the image to be processed in the current posture by the second adjustment value to obtain the first depth value of each pixel in the image to be processed in the current posture.
  • the first depth value of is the first depth information corresponding to each pixel in the image to be processed in the current pose.
  • determining the first adjustment value according to the difference value corresponding to each successfully matched pixel includes: determining the middle value or the maximum value or the minimum value of each difference value corresponding to the matched pixel; Either the value or the maximum value or the minimum value is used as the first adjustment value.
  • determining the second adjustment value according to the ratio corresponding to each successfully matched pixel includes: determining the middle value or the maximum value or the minimum value of each difference corresponding to the matched pixel; Either the maximum value or the minimum value is used as the second adjustment value.
  • the difference or ratio between the depth value of the focus area and the initial depth value of the matching pixel in the image to be processed can be determined to determine the depth value of the focus area and the The difference between the initial depth values of the matched pixels in the image.
  • the depth information of the partial image of the focus area and the initial depth information in the image to be processed can be fused to obtain the current
  • the accurate depth information of each pixel in the image to be processed in the posture improves the accuracy of the depth information of the image to be processed.
  • the acquiring depth information of the focus area in the current attitude includes:
  • Determine the focus area in the current posture obtain the corresponding focus value according to the focus area; obtain the corresponding depth information according to the focus value, and use the depth information corresponding to the focus value as the depth information of the focus area in the current posture.
  • the focus area is the focus position of the motor-driven lens after focusing.
  • the ISP processor or central processing unit of the electronic device can determine the focus position of the current motor-driven lens for focusing, and obtain the corresponding focus area in the current posture according to the preset correspondence between the focus area and the focus value.
  • the focal length value of the focus area in the current attitude is the focal length value of the lens in the current attitude. Then, through the preset correspondence between the focal length value of the lens and the depth information, the depth information corresponding to the focal length value of the lens in the current posture is obtained, and the depth information corresponding to the focal length value of the lens under the current posture is taken as the current posture Depth information of the lower focus area.
  • the image processing method in this embodiment can obtain the focal length value corresponding to the focus area in the current posture according to the preset correspondence relationship between the focus area and the focal length value, and obtain the current posture according to the preset correspondence relation between the focal length value and the depth information
  • the depth information corresponding to the focal length value of the lower lens can be indirectly obtained the depth information corresponding to the focus area in the current posture.
  • acquiring the corresponding focal length value according to the focus area includes:
  • mapping relationship between the focus area and the focus value is acquired, and the focus value corresponding to the focus area is acquired according to the mapping relationship.
  • the mapping relationship between the focus area and the focal length value is preset.
  • the ISP processor or the central processing unit of the electronic device can obtain the mapping relationship, and determine the focus area that is the same as the focus area in the current posture from the mapping relationship. And obtain the focal length value corresponding to the same focus area in the mapping relationship, then the focal length value is the focal length value corresponding to the focus area in the current attitude, so that the preset mapping relationship between the focus area and the focal length value can be quickly obtained The focal length value corresponding to the focus area in the current attitude.
  • the mapping relationship between the focus area and the focal length value may be embodied by a relationship mapping table.
  • the obtaining the mapping relationship between the focus area and the focal length value, and obtaining the focal length value corresponding to the focus area according to the mapping relationship includes :
  • a relationship mapping table is obtained, the focal length value corresponding to the focus area is obtained from the relationship mapping table, and the mapping relationship between the focus area and the focal length value is recorded in the relationship mapping table.
  • the relationship mapping table is a table of a preset correspondence relationship between the focus area and the focus value, and is used to record the mapping relationship between the focus area and the focus value.
  • the ISP processor or the central processing unit of the electronic device can obtain the relationship mapping table, compare the focus area in the current attitude with the focus area in the relationship map one by one, and determine the relationship between the focus area in the relationship mapping table and the current attitude.
  • the focus area is the same as the focus area.
  • the focal length value corresponding to the same focus area in the relationship mapping table can be obtained, and the focal length value is the focal length value corresponding to the focus area in the current posture.
  • the focal length value corresponding to the focus area in the current posture can be quickly and simply obtained.
  • acquiring corresponding depth information according to the focal length value includes:
  • a focus curve is acquired, and depth information corresponding to the focus value is acquired from the focus curve, and the focus curve is a correlation curve between the focus value and the depth information.
  • the focus curve is the correlation curve between the focal length value of the current lens and the depth information of the focus area.
  • the focus curve is a curve established in advance based on the focal length value and the depth information, and each focal length value corresponds to one piece of depth information.
  • the ISP processor or central processing unit of the electronic device can obtain the focus curve, and according to the corresponding relationship between the focal length value of the recording lens and the depth information of the focus area in the focus curve, the focal length value corresponding to the focus area in the current posture can be obtained.
  • the depth information of the mapping relationship is used as the depth information of the focus area in the current posture.
  • the depth information corresponding to the focal length value of the focus area in the current attitude can be quickly and simply obtained through the focus curve, so that the depth information corresponding to the focus area in the current attitude can be obtained indirectly.
  • the second depth information of each pixel in the image to be processed in the target pose is determined according to the target pose information and the first depth information of each pixel in the image to be processed in the current pose ,include:
  • the first three-dimensional coordinates corresponding to each pixel in the image to be processed in the current posture use a coordinate conversion algorithm to convert the first three-dimensional coordinates to the second three-dimensional coordinates according to the target posture information to obtain the The second depth information of each pixel in the image to be processed, where the second three-dimensional coordinate is a three-dimensional coordinate corresponding to each pixel in the image to be processed in the target pose.
  • the ISP processor or central processing unit of the electronic device acquires the three-dimensional coordinates of each pixel in the image to be processed in the current posture in the world coordinate system, that is, the first three-dimensional coordinates, and converts the current posture to the current posture through a coordinate conversion algorithm.
  • the first three-dimensional coordinate corresponding to each pixel point of is converted into the second three-dimensional coordinate in the target posture.
  • the second three-dimensional coordinate is the three-dimensional coordinate of each pixel in the image to be processed in the world coordinate system under the target pose.
  • the coordinate conversion algorithm can convert the three-dimensional coordinates of each pixel in the image to be processed from the current posture to the target posture, thereby determining the reprojection mode of each pixel.
  • the coordinate conversion algorithm is:
  • (x, y, z) is the first three-dimensional coordinate corresponding to each pixel in the current pose
  • (x', y', z') is the second three-dimensional coordinate corresponding to each pixel in the target pose
  • R and T It is the rotation matrix and translation matrix of the camera in the current posture in the world coordinate system, which represents the current posture information of the camera.
  • R′ and T′ are the rotation matrix and translation matrix of the camera in the world coordinate system under the target pose, which represent the target pose information of the camera.
  • the ISP processor or central processing unit of the electronic device obtains the (x, y, z) of each pixel in the image to be processed in the current posture, and can obtain R, T, R′ and R′ and R′ according to the current posture information and target posture information. T', substituting these data into the above coordinate conversion algorithm, you can calculate (x', y', z') corresponding to each pixel.
  • the image processing method in this embodiment uses a coordinate conversion algorithm to convert the coordinates of each pixel in the image to be processed in the current pose to the corresponding coordinate in the target pose, so that each pixel can be reprojected in a targeted manner Processing to achieve targeted anti-shake for each pixel.
  • acquiring the first internal parameter information of the camera includes:
  • the initial internal parameter information of the camera is obtained.
  • the initial internal parameter information is preset camera internal parameter information at a specific focusing distance.
  • the camera needs to be calibrated before it leaves the factory, and the initial internal reference information is the internal parameter information preset at a specific focus distance before the camera leaves the factory.
  • the ISP processor or the central processing unit of the electronic device obtains the initial internal parameter information of the camera, and the initial internal parameter information includes the focal length value and the offset of the center coordinate of the camera.
  • the focal length value of the lens will change at different focusing positions, and the ISP processor or central processing unit of the electronic device can obtain the real-time focal length value of the camera lens according to the focus area in the current posture.
  • the initial internal parameter information of the camera is updated according to the focal length value of the camera in the current posture to obtain the first internal parameter information of the camera.
  • the first internal parameter information is internal parameter information obtained after updating at least one of the focal length value and the offset in the initial internal parameter information.
  • the ISP processor or central processing unit of the electronic device can replace the focal length value in the initial internal parameter information of the camera with the focal length value of the lens in the current posture, thereby updating the internal parameter information, and the updated internal parameter information is the first internal parameter information of the camera.
  • One internal reference information One internal reference information.
  • the image processing method in this embodiment obtains the real-time focal length value of the camera in the current posture to update the internal parameter information of the camera, which can solve the problem of using the same focal length value for each pixel in the image to be processed in the traditional image processing method.
  • the points are all projected to the same unit plane, resulting in the problem that the true depth information of each pixel of the image cannot be detected.
  • the image processing method in this embodiment improves the accuracy of depth information detection.
  • the method further includes:
  • the Hall value in the current posture is obtained through the Hall sensor.
  • Operation 604 Determine an offset of the camera in the current posture based on the Hall value.
  • the Hall sensor is a magnetic field sensor made according to the Hall effect.
  • the Hall effect is essentially the deflection caused by the Lorentz force of moving charged particles in a magnetic field. When charged particles (electrons or holes) are confined in the solid material, this deflection results in the accumulation of positive and negative charges in the direction of the vertical current and magnetic field, thereby forming an additional lateral electric field.
  • the electronic device can record the offset scale of the camera lens on the XY plane through the Hall sensor, and while recording the offset scale, it can also record the direction of the offset, according to the distance corresponding to each scale, and the offset Direction, and then get the lens offset (c x , c y ).
  • the magnitude of the Hall value collected by the Hall sensor is known, and the magnitude of the lens offset at the current moment can be uniquely determined.
  • the angular velocity information collected by the gyroscope sensor corresponds to the Hall value collected by the Hall sensor in time sequence.
  • the updating the initial internal parameter information of the camera according to the focal length value of the camera in the current posture to obtain the first internal parameter information of the camera includes:
  • the initial internal parameter information of the camera is updated according to the focal length value of the camera in the current posture and the offset of the camera to obtain the first internal parameter information of the camera.
  • the initial internal parameter information of the camera includes the focal length f of the camera, the offset c x of the camera on the X plane, and the offset c y of the camera on the X plane.
  • the ISP processor or central processing unit of the electronic device can replace the offset and focal length value in the initial internal parameter information with the offset of the camera and the focal length of the current lens collected in the current posture, thereby obtaining the first camera's offset and focal length.
  • the image processing method in this embodiment obtains the Hall value in the current posture through a Hall sensor, and determines the offset of the camera in the current posture based on the Hall value, thereby obtaining the real-time camera offset,
  • the initial internal parameter information of the camera is updated according to the focal length value of the camera and the offset of the camera in the current posture, which can better meet the anti-shake requirements of shooting under zooming conditions.
  • FIG. 7 it is a schematic diagram of optical image stabilization in an embodiment.
  • Optical image stabilization generally compensates for the posture difference caused by the rotation of the camera by moving the lens on the x and y axes.
  • the posture difference is the difference between the current posture of the camera and the target posture.
  • FIG. 8 it is a schematic diagram of an image processing method in an embodiment.
  • the IPS processor or central processing unit of the electronic device obtains the current three-axis angular velocity information through the gyroscope, the current three-axis acceleration information through the accelerometer, the current altitude information through the altimeter, and the current position information through the compass.
  • the GPS positioning system obtains the current geographic location information. Then, the current three-axis angular velocity information, the current three-axis acceleration information, the current altitude information, the current azimuth information, and the current geographic location information are fused to obtain the current posture information of the camera. And according to the current position information and the current geographic location information, the current motion scene information of the camera is obtained.
  • the IPS processor or central processor of the electronic device can also obtain the depth information of each pixel in the same image to be processed in the current posture through the TOF camera, the dual-camera parallax ranging method, and the phase focusing method.
  • the three kinds of depth information obtained in the three ways are merged to obtain the accurate depth information of each pixel in the image to be processed in the current posture.
  • the IPS processor or central processing unit of the electronic device also drives the lens to focus through a motor, and updates the initial internal parameter information of the camera according to the position, focal length, and offset of the lens in the X and Y directions.
  • the IPS processor or central processing unit of the electronic device can reproject the image to be processed according to the current posture information of the camera, the current motion scene information, the updated internal parameter information and the accurate depth information of each pixel, and the output is stable.
  • the target image can be reproject the image to be processed according to the current posture information of the camera, the current motion scene information, the updated internal parameter information and the accurate depth information of each pixel, and the output is stable.
  • an image processing method including:
  • Operation (b1) is to obtain the current angular velocity information and current acceleration information of the camera, and obtain at least one of the current height information, current azimuth information, and current geographic location information of the camera.
  • Operation (b3) is to perform fusion processing on the current position information and the current geographic location information to obtain current sports scene information.
  • Operation (b4) according to the current motion scene information and the current posture information, determine the target posture information.
  • the initial depth information is the depth information of each pixel in the image to be processed in the current posture obtained by the depth camera.
  • the third depth information is the depth information of each pixel in the image to be processed in the current posture obtained by a dual-camera parallax ranging method.
  • the fourth depth information is the depth information of each pixel in the image to be processed in the current posture obtained through a phase focusing method.
  • the depth information includes a depth value, and the average value of the at least two kinds of depth information corresponding to each pixel point is determined.
  • the average value corresponding to each pixel is used as the first depth value of each pixel in the image to be processed in the current posture.
  • Operation (b10) is to obtain the first three-dimensional coordinates corresponding to each pixel in the image to be processed in the current posture.
  • the coordinate conversion algorithm is used to convert the first three-dimensional coordinates to the second three-dimensional coordinates according to the target posture information to obtain the second depth information of each pixel in the image to be processed under the target posture, and the second The three-dimensional coordinates are the three-dimensional coordinates corresponding to each pixel in the image to be processed under the target pose.
  • Operate (b12) to obtain the initial internal parameter information of the camera Obtain the focal length value of the camera in the current posture.
  • Operation (b13) obtain the Hall value in the current posture through the Hall sensor.
  • Operation (b14) is to determine the offset of the camera in the current posture based on the Hall value.
  • the initial internal parameter information of the camera is updated according to the focal length value of the camera in the current posture and the offset of the camera to obtain the first internal parameter information of the camera.
  • the image processing method in this embodiment at least one of the current height information, current position information, and current geographic location information of the camera is fused with the current angular velocity information, and current acceleration information, based on multiple types of information
  • the current posture information of the camera obtained by fusion is more accurate.
  • the depth information of each pixel in the same image to be processed in the current posture is obtained through the TOF camera, the dual-camera parallax ranging method, and the phase focusing method.
  • the three kinds of depth information obtained by the three methods are merged to obtain richer and more comprehensive detail information, which makes the depth information of each pixel in the image to be processed in the current pose more accurate.
  • Converting the coordinates of each pixel in the current posture to the corresponding coordinates under the target posture can determine the three-dimensional coordinates of the image to be processed in a stable state.
  • the initial internal parameter information is updated according to the focal length value and lens offset of the camera lens obtained in the current posture, so that the object distance in the shooting screen is closer to the object distance of the actual scene.
  • the image to be processed is reprojected, and the pixel coordinates of each pixel in the target posture can be obtained, so that the anti-shake effect can be output Consistent target image.
  • the image processing method in this embodiment realizes the advantages of optical anti-shake and electronic anti-shake, provides a theoretical basis, data path, and algorithm prototype for cooperation between the two, and can realize anti-shake before and after image sensor imaging.
  • a method for acquiring depth information including:
  • Operation 902 Acquire current posture information of the camera, where the current posture information includes current angular velocity information.
  • the ISP processor or the central processing unit of the electronic device can obtain the three-axis angular velocity of the camera through the gyroscope, and the three-axis angular velocity is corrected and integrated in the time domain to output the three-axis angular velocity information.
  • the current posture information is converted into target posture information.
  • the ISP processor or central processing unit of the electronic device obtains the current posture information of the camera, it can make predictions based on the current posture, and determine the target posture corresponding to the current posture. Further, the current posture information can be converted into target posture information through the target posture prediction algorithm.
  • Operation 906 Acquire first depth information of each pixel in the image to be processed in the current posture.
  • the ISP processor or the central processing unit of the electronic device can obtain the first depth information corresponding to each pixel in the image to be processed in the current posture.
  • Operation 908 according to the target pose information and the first depth information of each pixel in the image to be processed in the current pose, determine the second depth information of each pixel in the image to be processed in the target pose.
  • the ISP processor or central processor of the electronic device converts the current posture information into target posture information to obtain the target posture corresponding to the current posture.
  • the first depth information of each pixel in the image to be processed under the current posture can be transformed into the second depth information of each pixel in the image to be processed under the target posture.
  • the depth information acquisition method in this embodiment acquires the current posture information of the camera, the current posture information includes the current angular velocity information, converts the current posture information into target posture information, and obtains the information of each pixel in the image to be processed in the current posture
  • the first depth information according to the target pose information and the first depth information of each pixel in the image to be processed in the current pose, determine the second depth information of each pixel in the image to be processed in the target pose, and the available to be processed The accurate depth information of the image in the target pose.
  • the current posture information further includes at least one of current altitude information, current orientation information, and current geographic location information;
  • the acquiring current posture information of the camera includes:
  • Obtain the current angular velocity information of the camera and obtain at least one of the current altitude information, current position information, and current geographic location information of the camera; the current altitude information, the current position information, and the current geographic location At least one of the information is fused with the current angular velocity information to obtain the current posture information of the camera.
  • the ISP processor or central processing unit of the electronic device can obtain the angular velocity information of the camera in the current attitude through the gyroscope, the height of the camera from the ground plane in the current attitude can be detected by the altimeter, and the current position of the camera can be acquired through the compass.
  • the position in the attitude can also be used to obtain the geographic coordinates of the camera in the current attitude through GPS.
  • the ISP processor or central processing unit of the electronic device can select at least one of the current altitude information, the current position information, and the current geographic location information, and perform fusion processing on the selected information and the current angular velocity information to obtain the The current posture information of the camera.
  • the selected information and the current angular velocity information can be processed by Kalman filtering to realize the fusion of the information and obtain the current posture information of the camera.
  • the Kalman filter is essentially a data fusion algorithm that fuses data with the same purpose, from different sensors, and with different units to obtain a more accurate measurement value of the purpose.
  • the image processing method in this embodiment by acquiring the current angular velocity information of the camera, and acquiring at least one of the current height information, current orientation information, and current geographic location information of the camera, a variety of characterization cameras can be obtained. Different information about the current posture. At least one of the current altitude information, the current orientation information, and the current geographic location information is fused with the current angular velocity information to obtain the current posture information of the camera, which provides multiple ways to obtain the current posture information of the camera. Through the fusion of different information, the current posture information of the camera can be obtained more accurately.
  • the current posture information further includes current acceleration information; the method further includes: acquiring current acceleration information of the camera;
  • the fusion processing of at least one of the current altitude information, the current orientation information, and the current geographic location information with the current angular velocity information to obtain the current posture information of the camera includes: At least one of the altitude information, the current orientation information, and the current geographic location information is fused with the current angular velocity information and the current acceleration information to obtain the current posture information of the camera.
  • the ISP processor or the central processing unit of the electronic device can obtain the three-axis acceleration through an accelerometer, and the three-axis acceleration is corrected and integrated in the two time domains to obtain the three-axis position information of the camera.
  • the three-axis position information can be converted into the translation matrix of the camera in the world coordinate system.
  • at least one can be selected from the current height information of the camera, the current position information, and the current geographic location information, and the selected information can be fused with the current angular velocity information and the current acceleration information.
  • the fusion processing can be realized by fusion algorithms such as Kalman filtering algorithm.
  • the image processing method in this embodiment by acquiring current acceleration information, at least one of the current height information, current orientation information, and current geographic location information of the camera is combined with the current angular velocity information, and current acceleration information. Fusion processing provides a variety of ways to obtain the current posture information of the camera.
  • the current acceleration information is also used as a reference for obtaining the current posture information, so that the current posture information of the camera can be obtained more accurately.
  • Fig. 10 is a structural block diagram of an image processing apparatus according to an embodiment.
  • the image processing apparatus includes: a posture acquisition module 1002, a conversion module 1004, a depth information acquisition module 1006, a determination module 1008, and a target image determination module 1010. among them,
  • the posture acquisition module 1002 is used to obtain current posture information of the camera, and the current posture information includes current angular velocity information.
  • the conversion module 1004 is used to convert the current posture information into target posture information.
  • the depth information acquisition module 1006 is used to acquire the first depth information of each pixel in the image to be processed in the current posture.
  • the determining module 1008 is configured to determine the second depth information of each pixel in the image to be processed in the target pose according to the target pose information and the first depth information of each pixel in the image to be processed in the current pose.
  • the target image determination module 1010 is configured to obtain the first internal parameter information of the camera, and perform the processing of the image to be processed according to the current posture information, the first depth information, the target posture information, the second depth information, and the first internal parameter information. Perform re-projection processing to obtain the target image.
  • the image processing device in this embodiment obtains the current posture information of the camera, the current posture information includes the current angular velocity information, converts the current posture information into target posture information, and obtains the first position of each pixel in the image to be processed in the current posture.
  • a depth information according to the target pose information and the first depth information of each pixel in the image to be processed in the current pose, determine the second depth information of each pixel in the image to be processed in the target pose, and obtain the first internal parameters of the camera Information, according to the current posture information, the first depth information, the target posture information, the second depth information and the first internal reference information to reproject the image to be processed to obtain the target image, which can achieve targeted anti-shake for each pixel , Which makes the image stabilization effect more stable.
  • the current posture information further includes at least one of current altitude information, current position information, and current geographic location information;
  • the posture acquisition module 1002 is also used to: acquire the current angular velocity information of the camera, and acquire at least one of the current altitude information, current orientation information, and current geographic location information of the camera; At least one of the azimuth information and the current geographic location information is fused with the current angular velocity information to obtain the current posture information of the camera.
  • the current angular velocity information of the camera By acquiring the current angular velocity information of the camera, and acquiring at least one of the current height information, current orientation information, and current geographic location information of the camera, a variety of different information used to characterize the current posture of the camera can be obtained. At least one of the current altitude information, the current orientation information, and the current geographic location information is fused with the current angular velocity information to obtain the current posture information of the camera, which provides multiple ways to obtain the current posture information of the camera. Through the fusion of different information, the current posture information of the camera can be obtained more accurately.
  • the current posture information includes current acceleration information
  • the posture acquisition module 1002 is also used to: acquire current acceleration information of the camera; at least one of the current height information, the current orientation information, and the current geographic position information of the camera is related to the current angular velocity information, and the current angular velocity information.
  • the acceleration information of the camera is fused to obtain the current posture information of the camera.
  • the current acceleration information By acquiring the current acceleration information, at least one of the current height information, current azimuth information, and current geographic location information of the camera is fused with the current angular velocity information and current acceleration information to provide a variety of acquisition cameras The mode of current posture information.
  • the current acceleration information is also used as a reference for obtaining the current posture information, so that the current posture information of the camera can be obtained more accurately.
  • the conversion module 1004 is further configured to: obtain current sports scene information; and determine target posture information according to the current sports scene information and the current posture information.
  • the direction in which the jitter is to be eliminated in the current posture and the direction in which the jitter does not need to be eliminated can be determined, so as to perform targeted processing on the information in all directions in the current posture information, so as to predict
  • the obtained target posture information is more accurate and more in line with the actual shooting scene
  • the conversion module 1004 is also used to: obtain the current position information and current geographic position information of the camera; perform fusion processing on the current position information and the current geographic position information to obtain the current sports scene information.
  • the conversion module 1004 is also used to: obtain the current position information and current geographic position information of the camera; perform fusion processing on the current position information and the current geographic position information to obtain the current sports scene information.
  • the change of the camera's direction position and geographic coordinates can be determined.
  • the current azimuth information and the current geographic location information are fused and processed, and the movement state of the camera maintained during the shooting process can be intuitively and accurately determined according to the changes in the direction and location of the camera and the geographic coordinates.
  • the depth information acquisition module 1006 is further configured to: acquire at least two types of depth information among the initial depth information, the third depth information, and the fourth depth information; wherein, the initial depth information is acquired through a depth camera.
  • the third depth information is the depth information of each pixel in the image to be processed in the current posture obtained by dual-camera parallax ranging;
  • the fourth depth The information is the depth information of each pixel in the image to be processed in the current posture acquired by phase focusing; the at least two kinds of acquired depth information are fused and corrected to obtain the image in the current posture.
  • the first depth information of each pixel is further configured to: acquire at least two types of depth information among the initial depth information, the third depth information, and the fourth depth information; wherein, the initial depth information is acquired through a depth camera.
  • the depth information of each pixel in the image to be processed in the current posture is the depth information of each pixel in the image to be processed in the current posture
  • the image processing device in this embodiment obtains the depth information of the same image to be processed in the current posture in at least two ways to obtain at least two types of depth information. By fusing and superimposing at least two kinds of acquired depth information with each other, richer and more comprehensive detail information can be obtained, and thus more accurate depth information of each pixel in the image to be processed can be obtained.
  • the depth information includes a depth value; the depth information acquisition module 1006 is further configured to: determine the mean value of the at least two kinds of depth information corresponding to each pixel; and use the mean value corresponding to each pixel as the current The first depth value of each pixel in the image to be processed in the posture.
  • the image processing device in this embodiment determines the average value of at least two types of depth information corresponding to each pixel point, and uses the average value corresponding to each pixel point as the first depth value of each pixel point in the image to be processed in the current posture, Provides a variety of ways to get the first depth value. Through the fusion of different depth information, richer detailed information of the image to be processed can be obtained, so that the calculated depth information of each pixel in the image to be processed is more accurate.
  • the depth information acquisition module 1006 is further configured to: acquire the initial depth information of each pixel in the image to be processed in the current attitude; acquire the depth information of the focus area in the current attitude; and combine the initial depth information with The depth information of the focus area in the current posture is fused and corrected to obtain the first depth information of each pixel in the image to be processed under the current posture.
  • the image processing device in this embodiment obtains the initial depth information of each pixel in the image to be processed in the current posture, and obtains the depth information of the focus area in the current posture, and can obtain two types of the same image to be processed in the current posture. Depth information of different precisions.
  • the initial depth information and the depth information of the focus area in the current posture are fused and corrected to obtain the first depth information of each pixel in the image to be processed under the current posture.
  • High-precision local depth information can be applied to Each part of the image to be processed is thus obtained more accurate depth information of the image to be processed, that is, the first depth information of each pixel in the image to be processed in the current posture is obtained.
  • the depth information acquisition module 1006 is further configured to: determine the focus area in the current posture, and obtain the corresponding focal length value according to the focus area; obtain the corresponding depth information according to the focal length value, which will correspond to the focal length value
  • the depth information of is used as the depth information of the focus area in the current posture.
  • the depth information acquiring module 1006 is further configured to acquire a focus curve, and acquire depth information corresponding to the focus value from the focus curve, and the focus curve is an associated curve between the focus value and the depth information.
  • the depth information corresponding to the focal length value of the focus area in the current attitude can be quickly and simply obtained through the focus curve, so that the depth information corresponding to the focus area in the current attitude can be obtained indirectly.
  • the determining module 1008 is further configured to: obtain the first three-dimensional coordinate corresponding to each pixel in the image to be processed in the current posture; use a coordinate conversion algorithm to obtain the first three-dimensional coordinate according to the target posture information Converted to a second three-dimensional coordinate to obtain the second depth information of each pixel in the image to be processed in the target pose, and the second three-dimensional coordinate is the three-dimensional coordinate corresponding to each pixel in the image to be processed in the target pose .
  • the coordinate conversion algorithm to convert the coordinates of each pixel in the image to be processed in the current pose to the corresponding coordinate in the target pose, each pixel can be reprojected in a targeted manner to achieve the goal of each pixel. Anti-shake.
  • the determining module 1008 is further used to: obtain the initial internal parameter information of the camera; obtain the focal length value of the camera in the current posture; update the initial camera focal length value according to the focal length value of the camera in the current posture Internal reference information to obtain the first internal reference information of the camera.
  • the determining module 1008 is further configured to: obtain the Hall value in the current posture through the Hall sensor; determine the offset of the camera in the current posture based on the Hall value; and according to the current posture
  • the focal length value of the camera and the offset of the camera below update the initial internal parameter information of the camera to obtain the first internal parameter information of the camera.
  • the value and the offset of the camera update the initial internal reference information of the camera, which can better meet the anti-shake requirements of shooting under zooming conditions.
  • a depth information acquisition device including:
  • the posture acquisition module 1102 is used to obtain current posture information of the camera, and the current posture information includes current angular velocity information.
  • the conversion module 1104 is used to convert the current posture information into target posture information.
  • the depth information acquisition module 1106 is used to acquire the first depth information of each pixel in the image to be processed in the current posture.
  • the determining module 1108 is configured to determine the second depth information of each pixel in the image to be processed in the target pose according to the target pose information and the first depth information of each pixel in the image to be processed in the current pose.
  • the depth information acquisition device in this embodiment acquires the current posture information of the camera, the current posture information includes the current angular velocity information, converts the current posture information into target posture information, and obtains the information of each pixel in the image to be processed in the current posture
  • the first depth information according to the target pose information and the first depth information of each pixel in the image to be processed in the current pose, determine the second depth information of each pixel in the image to be processed in the target pose, and the available to be processed The accurate depth information of the image in the target pose.
  • the current posture information further includes at least one of current altitude information, current orientation information, and current geographic location information; the posture acquisition module 1102 is also used to: obtain the current angular velocity information of the camera, and Obtain at least one of the current altitude information, current position information, and current geographic location information of the camera; the current altitude information, the current location information, and at least one of the current geographic location information are combined with the current altitude information, the current location information, and the current geographic location information.
  • the current angular velocity information is fused to obtain the current posture information of the camera.
  • the current angular velocity information of the camera By acquiring the current angular velocity information of the camera, and acquiring at least one of the current height information, current orientation information, and current geographic location information of the camera, a variety of different information used to characterize the current posture of the camera can be obtained. At least one of the current altitude information, the current orientation information, and the current geographic location information is fused with the current angular velocity information to obtain the current posture information of the camera, which provides multiple ways to obtain the current posture information of the camera. Through the fusion of different information, the current posture information of the camera can be obtained more accurately.
  • the current posture information further includes current acceleration information; the posture acquisition module 1102 is also used to: obtain the current acceleration information of the camera; the current height information of the camera, the current orientation information, and the current acceleration information. At least one of the geographic location information is fused with the current angular velocity information and the current acceleration information to obtain the current posture information of the camera.
  • the current acceleration information By acquiring the current acceleration information, at least one of the current height information, current azimuth information, and current geographic location information of the camera is fused with the current angular velocity information and current acceleration information to provide a variety of acquisition cameras The mode of current posture information.
  • the current acceleration information is also used as a reference for obtaining the current posture information, so that the current posture information of the camera can be obtained more accurately.
  • the division of the modules in the above image processing device is only for illustration. In other embodiments, the image processing device and the depth information acquisition device can be divided into different modules as needed to complete the above image processing device and depth information acquisition device. All or part of the function.
  • Fig. 12 is a schematic diagram of the internal structure of an electronic device in an embodiment.
  • the electronic device includes a processor and a memory connected through a system bus.
  • the processor is used to provide computing and control capabilities to support the operation of the entire electronic device.
  • the memory may include a non-volatile storage medium and internal memory.
  • the non-volatile storage medium stores an operating system and a computer program.
  • the computer program can be executed by a processor to implement an image processing method and a depth information acquisition method provided in the following embodiments.
  • the internal memory provides a cached operating environment for the operating system computer program in the non-volatile storage medium.
  • the electronic device can be a mobile phone, a tablet computer, or a personal digital assistant or a wearable device.
  • each module in the image processing device and the depth information acquisition device provided in the embodiments of the present application may be in the form of a computer program.
  • the computer program can be run on a terminal or a server.
  • the program module composed of the computer program can be stored in the memory of the terminal or the server.
  • the embodiment of the present application also provides a computer-readable storage medium.
  • One or more non-volatile computer-readable storage media containing computer-executable instructions when the computer-executable instructions are executed by one or more processors, cause the processors to perform the operations of the image processing method.
  • Any reference to memory, storage, database, or other media used in the embodiments of the present application may include non-volatile and/or volatile memory.
  • Suitable non-volatile memory may include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory.
  • Volatile memory may include random access memory (RAM), which acts as external cache memory.
  • RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous Link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
  • SRAM static RAM
  • DRAM dynamic RAM
  • SDRAM synchronous DRAM
  • DDR SDRAM double data rate SDRAM
  • ESDRAM enhanced SDRAM
  • SLDRAM synchronous Link (Synchlink) DRAM
  • Rambus direct RAM
  • DRAM direct memory bus dynamic RAM
  • RDRAM memory bus dynamic RAM

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Studio Devices (AREA)

Abstract

一种图像处理方法包括：将摄像头的当前姿态信息转换为目标姿态信息，当前姿态信息包括当前的角速度信息；获取在当前姿态下待处理图像中各像素点的第一深度信息；根据第一深度信息确定目标姿态下各像素点的第二深度信息；根据当前姿态信息、第一深度信息、目标姿态信息、第二深度信息和摄像头的第一内参信息，得到目标图像。

Description

图像处理方法和装置、电子设备、计算机可读存储介质 技术领域
本申请涉及影像领域，特别是涉及一种图像处理方法和装置、电子设备、计算机可读存储介质。
背景技术
随着影像技术的发展，人们越来越经常通过电子设备上的摄像头等图像采集设备拍摄图像或视频，记录各种信息。在进行拍摄的过程中，由于外界的抖动，会带来拍摄画面的抖动，造成图像的运动模糊和视频的不稳定。为了保证拍摄的质量，需要对拍摄过程进行防抖。
但是，传统的防抖方案大多采用简化处理，认为拍摄画面中的所有物体均处于单位平面上，因此造成拍摄画面的防抖效果不稳定。
发明内容
本申请实施例提供一种图像处理方法、装置、电子设备、计算机可读存储介质，可以提高整体的防抖效果。
一种图像处理方法，包括：
获取摄像头的当前姿态信息，所述当前姿态信息包括当前的角速度信息；
将所述当前姿态信息转换为目标姿态信息；
获取在当前姿态下待处理图像中各像素点的第一深度信息；
根据所述目标姿态信息和在所述当前姿态下所述待处理图像中各像素点的第一深度信息，确定在目标姿态下所述待处理图像中各像素点的第二深度信息；
获取所述摄像头的第一内参信息，根据所述当前姿态信息、所述第一深度信息、所述目标姿态信息、所述第二深度信息和所述第一内参信息对所述待处理图像进行重投影处理，得到目标图像。
一种深度信息获取方法，包括：
获取摄像头的当前姿态信息，所述当前姿态信息包括当前的角速度信息；
将所述当前姿态信息转换为目标姿态信息；
获取在当前姿态下待处理图像中各像素点的第一深度信息；
根据所述目标姿态信息和在所述当前姿态下所述待处理图像中各像素点的第一深度信息，确定在目标姿态下所述待处理图像中各像素点的第二深度信息。
一种图像处理装置，包括：
姿态获取模块，用于获取摄像头的当前姿态信息，所述当前姿态信息包括当前的角速度信息；
转换模块，用于将所述当前姿态信息转换为目标姿态信息；
深度信息获取模块，用于获取在当前姿态下待处理图像中各像素点的第一深度信息；
确定模块，用于根据所述目标姿态信息和在所述当前姿态下所述待处理图像中各像素点的第一深度信息，确定在目标姿态下所述待处理图像中各像素点的第二深度信息；
目标图像确定模块，用于获取所述摄像头的第一内参信息，根据所述当前姿态信息、所述第一深度信息、所述目标姿态信息、所述第二深度信息和所述第一内参信息对所述待处理图像进行重投影处理，得到目标图像。
一种深度信息获取装置，包括：
姿态获取模块，用于获取摄像头的当前姿态信息，所述当前姿态信息包括当前的角速度信息；
转换模块，用于将所述当前姿态信息转换为目标姿态信息；
深度信息获取模块，用于获取在当前姿态下待处理图像中各像素点的第一深度信息；
确定模块，用于根据所述目标姿态信息和在所述当前姿态下所述待处理图像中各像素点的第一深度信息，确定在目标姿态下所述待处理图像中各像素点的第二深度信息。
一种电子设备，包括存储器及处理器，所述存储器中储存有计算机程序，所述计算机程序被所述处理器执行时，使得所述处理器执行上述图像处理方法或深度信息获取方法的操作。
一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现上述图像处理方法或深度信息获取方法的操作。
上述图像处理方法和装置、电子设备、计算机可读存储介质，获取摄像头的当前姿态信息，该当前姿态信息包括当前的角速度信息，将当前姿态信息转换为目标姿态信息，获取在当前姿态下待处理图像中各像素点的第一深度信息，根据目标姿态信息和在当前姿态下待处理图像中各像素点的第一深度信息，确定在目标姿态下待处理图像中各像素点的第二深度信息，获取摄像头的第一内参信息，根据当前姿态信息、第一深度信息、目标姿态信息、第二深度信息和第一内参信息对待处理图像进行重投影处理，得到目标图像，可以实现对每个像素点进行针对性的防抖，使得拍摄的图像防抖效果更稳定。
附图说明
为了更清楚地说明本申请实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为一个实施例中电子设备的内部结构框图；
图2为一个实施例中图像处理方法的流程图；
图3为一个实施例中小孔成像的原理图；
图4为一个实施例中获取在当前姿态下待处理图像中各像素点的第一深度信息的操作的流程图；
图5为一个实施例中获取摄像头第一内参信息的操作的流程图；
图6为另一个实施例中获取摄像头第一内参信息的操作的流程图；
图7为一个实施例中光学防抖的原理图；
图8为一个实施例中图像处理方法的原理图；
图9为一个实施例中深度信息获取方法的流程图；
图10为一个实施例中图像处理装置的结构框图；
图11为一个实施例中深度信息获取装置的结构框图；
图12为一个实施例中电子设备的内部结构示意图。
具体实施方式
为了使本申请的目的、技术方案及优点更加清楚明白，以下结合附图及实施例，对本申请进行进一步详细说明。应当理解，此处所描述的具体实施例仅仅用以解释本申请，并不用于限定本申请。
本申请实施例中的图像处理方法、深度信息获取方法可应用于电子设备。该电子设备可为带有摄像头的计算机设备、个人数字助理、平板电脑、智能手机、穿戴式设备等。
在一个实施例中，上述电子设备中可包括图像处理电路，图像处理电路可以利用硬件 和/或软件组件实现，可包括定义ISP(Image Signal Processing，图像信号处理)管线的各种处理单元。图1为一个实施例中图像处理电路的示意图。如图1所示，为便于说明，仅示出与本申请实施例相关的图像处理技术的各个方面。
如图1所示，图像处理电路包括第一ISP处理器130、第二ISP处理器140和控制逻辑器150。第一摄像头110包括一个或多个第一透镜112和第一图像传感器114。第一图像传感器114可包括色彩滤镜阵列(如Bayer滤镜)，第一图像传感器114可获取用第一图像传感器114的每个成像像素捕捉的光强度和波长信息，并提供可由第一ISP处理器130处理的一组图像数据。第二摄像头120包括一个或多个第二透镜122和第二图像传感器124。第二图像传感器124可包括色彩滤镜阵列(如Bayer滤镜)，第二图像传感器124可获取用第二图像传感器124的每个成像像素捕捉的光强度和波长信息，并提供可由第二ISP处理器140处理的一组图像数据。
第一摄像头110采集的第一图像传输给第一ISP处理器130进行处理，第一ISP处理器130处理第一图像后，可将第一图像的统计数据(如图像的亮度、图像的反差值、图像的颜色等)发送给控制逻辑器150，控制逻辑器150可根据统计数据确定第一摄像头110的控制参数，从而第一摄像头110可根据控制参数进行自动对焦、自动曝光等操作。第一图像经过第一ISP处理器130进行处理后可存储至图像存储器160中，第一ISP处理器130也可以读取图像存储器160中存储的图像以对进行处理。另外，第一图像经过ISP处理器130进行处理后可直接发送至显示器170进行显示，显示器170也可以读取图像存储器160中的图像以进行显示。
其中，第一ISP处理器130按多种格式逐个像素地处理图像数据。例如，每个图像像素可具有8、10、12或14比特的位深度，第一ISP处理器130可对图像数据进行一个或多个图像处理操作、收集关于图像数据的统计信息。其中，图像处理操作可按相同或不同的位深度精度进行。
图像存储器160可为存储器装置的一部分、存储设备、或电子设备内的独立的专用存储器，并可包括DMA(Direct Memory Access，直接直接存储器存取)特征。
当接收到来自第一图像传感器114接口时，第一ISP处理器130可进行一个或多个图像处理操作，如时域滤波。处理后的图像数据可发送给图像存储器160，以便在被显示之前进行另外的处理。第一ISP处理器130从图像存储器160接收处理数据，并对所述处理数据进行RGB和YCbCr颜色空间中的图像数据处理。第一ISP处理器130处理后的图像数据可输出给显示器170，以供用户观看和/或由图形引擎或GPU(Graphics Processing Unit，图形处理器)进一步处理。此外，第一ISP处理器130的输出还可发送给图像存储器160，且显示器170可从图像存储器160读取图像数据。在一个实施例中，图像存储器160可被配置为实现一个或多个帧缓冲器。
第一ISP处理器130确定的统计数据可发送给控制逻辑器150。例如，统计数据可包括自动曝光、自动白平衡、自动聚焦、闪烁检测、黑电平补偿、第一透镜112阴影校正等第一图像传感器114统计信息。控制逻辑器150可包括执行一个或多个例程(如固件)的处理器和/或微控制器，一个或多个例程可根据接收的统计数据，确定第一摄像头110的控制参数及第一ISP处理器130的控制参数。例如，第一摄像头110的控制参数可包括增益、曝光控制的积分时间、防抖参数、闪光控制参数、第一透镜112控制参数(例如聚焦或变焦用焦距)、或这些参数的组合等。ISP控制参数可包括用于自动白平衡和颜色调整(例如，在RGB处理期间)的增益水平和色彩校正矩阵，以及第一透镜112阴影校正参数。
同样地，第二摄像头120采集的第二图像传输给第二ISP处理器140进行处理，第二ISP处理器140处理第一图像后，可将第二图像的统计数据(如图像的亮度、图像的反差值、图像的颜色等)发送给控制逻辑器150，控制逻辑器150可根据统计数据确定第二摄像头120的控制参数，从而第二摄像头120可根据控制参数进行自动对焦、自动曝光等操 作。第二图像经过第二ISP处理器140进行处理后可存储至图像存储器160中，第二ISP处理器140也可以读取图像存储器160中存储的图像以对进行处理。另外，第二图像经过ISP处理器140进行处理后可直接发送至显示器170进行显示，显示器170也可以读取图像存储器160中的图像以进行显示。第二摄像头120和第二ISP处理器140也可以实现如第一摄像头110和第一ISP处理器130所描述的处理过程。
在一个实施例中，第一摄像头110可为彩色摄像头，第二摄像头120可为TOF(Time Of Flight，飞行时间)摄像头或结构光摄像头。TOF摄像头可获取TOF深度图，结构光摄像头可获取结构光深度图。第一摄像头110和第二摄像头120可均为彩色摄像头。通过两个彩色摄像头获取双目深度图。第一ISP处理器130和第二ISP处理器140可为同一ISP处理器。
第一摄像头110进行拍摄时，可通过预览画面得到待处理图像，将待处理图像发送给ISP处理器。ISP处理器可获取摄像头拍摄时的当前姿态信息，该当前姿态信息包括当前的角速度信息；然后将当前姿态信息转换为目标姿态信息；获取在当前姿态下该待处理图像中各像素点的第一深度信息，根据目标姿态信息和在当前姿态下待处理图像中各像素点的第一深度信息，确定在目标姿态下待处理图像中各像素点的第二深度信息；获取摄像头的第一内参信息，根据当前姿态信息、第一深度信息、目标姿态信息、第二深度信息和第一内参信息对待处理图像进行重投影处理，得到目标图像。通过确定每个像素点在当前姿态下的准确深度信息，并将当前姿态下的准确深度信息转换为目标姿态下的深度信息，从而可对每个像素点进行针对性的重投影处理，提高图像的整体防抖效果。
图2为一个实施例中图像处理方法的流程图。如图2所示，图像处理方法包括：
操作202，获取摄像头的当前姿态信息，该当前姿态信息包括当前的角速度信息。
其中，当前姿态信息是指表征摄像头当前姿态的信息，当前姿态信息包括当前的角速度信息。当前的角速度信息可转换为摄像头在世界坐标系下的旋转矩阵。因此，旋转矩阵可用于表征摄像头的当前姿态信息。
具体地，电子设备的ISP处理器或中央处理器可通过陀螺仪获取摄像头的三轴角速度，将该三轴角速度经过校正和在时间域上的积分处理，输出三轴角速度信息。
操作204，将当前姿态信息转换为目标姿态信息。
其中，目标姿态是指将在当前姿态下拍摄得到的待处理图像进行重投影后处于稳定状态时摄像头的姿态。目标姿态信息是指能够表征目标姿态的信息。
具体地，电子设备的ISP处理器或中央处理器得到摄像头的当前姿态信息后，可根据当前姿态进行预测，确定当前姿态对应的目标姿态。进一步地，可通过目标姿态预测算法将当前姿态信息转换为目标姿态信息。
操作206，获取在当前姿态下待处理图像中各像素点的第一深度信息。
其中，第一深度信息是指当前姿态下待处理图像中各像素点对应的准确深度信息。该待处理图像可以是一张完整的图像，也可以是一张图像中的部分图像。
具体地，电子设备的ISP处理器或中央处理器可获取当前姿态下的待处理图像中每个像素点对应的第一深度信息。
操作208，根据该目标姿态信息和在当前姿态下该待处理图像中各像素点的第一深度信息，确定在目标姿态下该待处理图像中各像素点的第二深度信息。
其中，第二深度信息是指目标姿态下待处理图像的各像素点对应的深度信息。
具体地，电子设备的ISP处理器或中央处理器将该当前姿态信息转换为目标姿态信息后，得到当前姿态对应的目标姿态。可根据当前姿态和目标姿态，将在当前姿态下待处理图像中各像素点的第一深度信息经过坐标变换，转换为目标姿态下待处理图像中各像素点的第二深度信息。
操作210，获取该摄像头的第一内参信息，根据当前姿态信息、第一深度信息、目标 姿态信息、第二深度信息和第一内参信息对该待处理图像进行重投影处理，得到目标图像。
其中，第一内参信息是指在当前姿态下获取的摄像头的实时内参信息。
具体地，电子设备的ISP处理器或中央处理器可获取在当前姿态下摄像头的第一内参信息。接着，将当前姿态信息、该第一深度信息、该目标姿态信息、该第二深度信息和该第一内参信息输入到重投影映射算法中，可得到待处理图像的每个像素点重投影的像素坐标。根据待处理图像的每个像素点重投影的像素坐标输出图像，即可得到目标图像。
本实施例中的图像处理方法，获取摄像头的当前姿态信息，该当前姿态信息包括当前的角速度信息，将当前姿态信息转换为目标姿态信息，获取在当前姿态下待处理图像中各像素点的第一深度信息，根据目标姿态信息和在当前姿态下待处理图像中各像素点的第一深度信息，确定在目标姿态下待处理图像中各像素点的第二深度信息，获取摄像头的第一内参信息，根据当前姿态信息、第一深度信息、目标姿态信息、第二深度信息和第一内参信息对待处理图像进行重投影处理，得到目标图像，可以实现对每个像素点进行针对性的防抖，使得拍摄的图像防抖效果更稳定。
如图3所示，为一个实施例中小孔成像的原理图。其中，f为摄像头焦距，c x及c y为摄像头中心坐标偏移，K为包括参数f、c x和c y的摄像头内参矩阵，(x,y,z)为三维空间点p在世界坐标系下的坐标，(u,v)为p的像素点p′的像素坐标，R和T为摄像头在世界坐标系下的旋转矩阵和平移矩阵，表征了摄像头的当前姿态信息，Z c为三维空间点p在摄像头坐标系下的物距。不同摄像头姿态下，摄像头坐标系发生变化，Z c也会发生变化。
成像过程中三维空间点到图像像素坐标的映射关系如公式(1)：
Figure PCTCN2019102801-appb-000001
为了获得稳定输出的图像，可将当前姿态下的待处理图像重投影到摄像头姿态为R′和T′的姿态下(即目标姿态)，再根据目标姿态计算出对应的Z c′，则重投影的过程可以描述为：
Figure PCTCN2019102801-appb-000002
在一个实施例中，该当前姿态信息还包括当前的高度信息、当前的方位信息和当前的地理位置信息中的至少一种；该获取摄像头的当前姿态信息，包括：
获取摄像头当前的角速度信息，并获取该摄像头当前的高度信息、当前的方位信息和当前的地理位置信息中的至少一种；将该当前的高度信息、该当前的方位信息和该当前的地理位置信息中的至少一种和该当前的角速度信息进行融合处理，得到该摄像头的当前姿态信息。
其中，当前的高度信息是指在当前姿态下摄像头距离地平面的高度。当前的方位信息是指在当前姿态下摄像头的方向位置，例如：东、南、西、北等方位。当前的地理位置信息指在当前姿态下摄像头的地理坐标。
具体地，电子设备的ISP处理器或中央处理器可通过陀螺仪获取摄像头在当前姿态下的角速度信息，可通过高度表检测摄像头在当前姿态下距离地平面的高度，可通过指南针获取摄像头在当前姿态下的方位，还可通过GPS(Global Positioning System，全球定位系统)定位获取摄像头在当前姿态下的地理坐标。
接着，电子设备的ISP处理器或中央处理器可选取当前的高度信息、当前的方位信息和当前的地理位置信息中的至少一种，将选取的信息和当前的角速度信息进行融合处理，得到该摄像头的当前姿态信息。
例如，将当前的高度信息和当前的角速度信息进行融合处理；将当前的方位信息和当前的角速度信息进行融合处理；将当前的地理位置信息和当前的角速度信息进行融合处理；将当前的高度信息、当前的方位信息和当前的角速度信息进行融合处理；将当前的高度信息、当前的地理位置信息和当前的角速度信息进行融合处理；将当前的方位信息、当前的地理位置信息和当前的角速度信息进行融合处理；将当前的高度信息、当前的方位信息、当前的地理位置信息和当前的角速度信息进行融合处理，均可获得该摄像头的当前姿态信息。
进一步地，可将选取的信息和当前的角速度信息经过卡尔曼滤波处理，以实现信息的融合，得到该摄像头的当前姿态信息。卡尔曼滤波本质上是一种数据融合算法，将具有相同目的、来自不同传感器、具有不同单位的数据融合在一起，得到一个更精确的目的测量值。
本实施例中的图像处理方法，通过获取摄像头当前的角速度信息，并获取该摄像头当前的高度信息、当前的方位信息和当前的地理位置信息中的至少一种，可得到多种用于表征摄像头当前姿态的不同信息。将当前的高度信息、当前的方位信息和当前的地理位置信息中的至少一种和当前的角速度信息进行融合处理，得到该摄像头的当前姿态信息，提供了多种获取摄像头当前姿态信息的方式。通过不同信息的融合，能够更准确得到摄像头的当前姿态信息。
在一个实施例中，该当前姿态信息包括还包括当前的加速度信息；该方法还包括：获取摄像头当前的加速度信息；
将当前的高度信息、当前的方位信息和当前的地理位置信息中的至少一种和当前的角速度信息进行融合处理，得到摄像头的当前姿态信息，包括：将摄像头当前的高度信息、当前的方位信息和当前的地理位置信息中的至少一种与当前的角速度信息，以及当前的加速度信息进行融合处理，得到摄像头的当前姿态信息。
其中，当前的加速度信息是指摄像头在当前姿态下的加速度。当前的加速度信息可转换为摄像头在世界坐标系下的平移矩阵。因此，平移矩阵可作为表征摄像头当前的姿态信息的一个分量。
具体地，电子设备的ISP处理器或中央处理器可通过加速度计获取三轴加速度，将该三轴加速度经过校正处理以及在两次时间域上的积分，可得到摄像头的三轴位置信息。三轴位置信息可转换为摄像头在世界坐标系下的平移矩阵。接着，可从摄像头当前的高度信息、当前的方位信息和当前的地理位置信息中选取至少一种，并将选取的信息与当前的角速度信息和当前的加速度信息进行融合处理。该融合处理可通过卡尔曼滤波算法等融合算法实现。
例如，将当前的高度信息、当前的加速度信息和当前的角速度信息进行融合处理；将当前的方位信息、当前的加速度信息和当前的角速度信息进行融合处理；将当前的地理位置信息、当前的加速度信息和当前的角速度信息进行融合处理；将当前的高度信息、当 前的方位信息、当前的加速度信息和当前的角速度信息进行融合处理；将当前的高度信息、当前的地理位置信息、当前的加速度信息和当前的角速度信息进行融合处理；将当前的方位信息、当前的地理位置信息、当前的加速度信息和当前的角速度信息进行融合处理；将当前的高度信息、当前的方位信息、当前的地理位置信息、当前的加速度信息和当前的角速度信息进行融合处理，均可获得该摄像头的当前姿态信息。
本实施例中的图像处理方法，通过获取当前的加速度信息，将摄像头当前的高度信息、当前的方位信息和当前的地理位置信息中的至少一种与当前的角速度信息，以及当前的加速度信息进行融合处理，提供了多种获取摄像头当前姿态信息的方式。并且将当前的加速度信息也作为获取当前姿态信息的一个参考量，能够更准确得到摄像头的当前姿态信息。
在本实施例中，电子设备的ISP处理器或中央处理器也可通过重力传感器获取重力加速度，将摄像头当前的高度信息、当前的方位信息和当前的地理位置信息中选取至少一种和重力加速度，以及当前的角速度信息进行融合，得到摄像头的当前姿态信息。
在一个实施例中，该将该当前姿态信息转换为目标姿态信息，包括：
获取当前的运动场景信息；根据该当前的运动场景信息和该当前姿态信息，确定目标姿态信息。
其中，当前的运动场景信息是指在拍摄过程中摄像头的运动状态。例如，用户使用摄像头横向移动拍摄时，摄像头处于横向运动的状态。
具体地，电子设备的ISP处理器或中央处理器可根据摄像头当前的方位信息和当前的地理位置信息确定在拍摄过程中摄像头的运动状态。接着，根据摄像头当前的运动状态确定在当前姿态下需要消除抖动的方向。例如，横向移动拍摄时需要尽可能的消除roll和pitch方向的抖动，而保留yaw方向上的抖动。其中，pitch为围绕x轴旋转的方向，旋转的角度为俯仰角。Yaw为围绕y轴旋转的方向，旋转的角度为偏航角，Roll为围绕z轴旋转的方向，旋转的角度为翻滚角。用户手持电子设备保持位置不变的拍摄时，需要尽可能消除所有方向上抖动，以实现三脚架的效果。
接着可将当前姿态信息中需要消除抖动的方向上的信息进行低通滤波处理，去除高频量，保留低频量。对于不需要消除抖动的方向上的信息可不做处理，从而输出根据摄像头当前的运动状态消除抖动的姿态信息，即可得到目标姿态信息。
传统的目标姿态预测方法仅通过对当前姿态的预测得到目标姿态，而本实施例中根据摄像头当前的运动状态息，可确定当前姿态上的待消除抖动的方向和不需要消除抖动的方向，以对当前姿态信息中各方向上的信息进行针对性处理，使得预测得到的目标姿态信息更加准确，更符合实际拍摄场景。
在一个实施例中，根据该当前的运动场景信息和该当前姿态信息，确定目标姿态信息包括：将所述当前姿态信息转换到频域空间；根据该当前的运动场景信息，将所述频域空间的当前姿态信息进行低通滤波处理；将经过低通滤波处理后的当前姿态信息转换到时域空间，得到目标姿态信息。
其中，时域即时间域，是描述数学函数或物理信号对时间的关系，例如一个信号的时域波形可以表达信号随时间的变化。频域即频率域，频域的自变量是频率，即横轴是频率，纵轴是该频率信号的幅度，也就是频谱图。频谱图描述了信号的频率结构及频率与该频率信号幅度的关系。
具体地，ISP处理器或中央处理器通过陀螺仪获取角速度信息，并将角速度信息通过时域上的积分，得到时域空间中的当前姿态信息。接着，ISP处理器或中央处理器可将时域空间上的当前姿态信息转换到频域空间。
接着，ISP处理器或中央处理器根据摄像头的运动状态确定当前姿态信息中待消除抖动的方向，并将该当前姿态信息中待消除抖动的方向上的信息进行低通滤波，去除该当前姿态信息中待消除抖动的方向上的信息中的高频量，保留低频量。对于当前姿态信息中不 需要消除抖动的方向上的信息可不做处理，得到频域空间中经过低通滤波处理后的低频量的当前姿态信息。
接着，ISP处理器或中央处理器经过低通滤波处理后的低频量的当前姿态信息再转换到时域空间，经过低通滤波处理后的信息从频域空间转换到时域空间后得到的姿态信息即为目标姿态信息。
本实施例中的图像处理方法，通过将该当前姿态信息转换到频域空间，根据摄像头当前的运动场景信息对该频域空间的当前姿态信息进行低通滤波处理，可对待消除抖动的方向上信息进行针对性处理。可根据当前的运动场景信息将当前姿态信息准确转换为目标姿态信息，从而可以准确预测出待处理图像在稳定状态下的姿态，即目标姿态。
在一个实施例中，该获取当前的运动场景信息，包括：
获取该摄像头当前的方位信息和当前的地理位置信息；将该当前的方位信息和该当前的地理位置信息进行融合处理，得到当前的运动场景信息。
具体地，电子设备的ISP处理器或中央处理器可通过指南针获取该摄像头当前的方位信息，并通过GPS定位系统获取摄像头当前的地理位置信息。将摄像头当前的方位信息和当前的地理位置信息通过卡尔曼滤波处理，实现两种信息的融合，从而得到当前的运动场景信息。
本实施例中的图像处理方法，通过获取该摄像头当前的方位信息和当前的地理位置信息，可确定摄像头方向位置和地理坐标的变动情况。将该当前的方位信息和该当前的地理位置信息进行融合处理，根据该摄像头方向位置和地理坐标的变动情况，可直观准确的确定该摄像头在拍摄过程中所保持的运动状态。
在一个实施例中，该获取在当前姿态下待处理图像中各像素点的第一深度信息，包括：
获取初始深度信息、第三深度信息和第四深度信息中的至少两种深度信息；其中，该初始深度信息是通过深度摄像头获取的在当前姿态下待处理图像中各像素点的深度信息；该第三深度信息是通过双摄视差测距方式获取的在该当前姿态下该待处理图像中各像素点的深度信息；该第四深度信息是通过相位对焦方式获取的在该当前姿态下该待处理图像中各像素点的深度信息；将该获取的至少两种深度信息进行融合修正处理，得到在该当前姿态下该待处理图像中各像素点的第一深度信息。
其中，融合修正处理是指根据不同的方式获得的同一待处理图像的不同信息经过分析、选择，以将不同方式获得的不同信息融合到同一张图像上。第一深度信息是指当前姿态下该待处理图像中各像素点精度更高的深度信息。
具体地，电子设备的ISP处理器或中央处理器可通过深度摄像头获取在当前姿态下待处理图像中各像素点的初始深度信息，通过双摄视差测距方式获取在所述当前姿态下所述待处理图像中各像素点的第三深度信息，还可以通过相位对焦方式获取在所述当前姿态下所述待处理图像中各像素点的第四深度信息。而在本实施例中，电子设备的ISP处理器或中央处理器可通过上述三种方式中的至少两种方式获取在当前姿态下的同一张待处理图像中各像素点的深度信息，得到初始深度信息、第三深度信息和第四深度信息中的至少两种。
接着，电子设备的ISP处理器或中央处理器可将至少两种方式获取的得到的深度信息相互融合叠加，从而将获取到的至少两种深度信息进行整合。融合修正处理后得到的各像素点的深度信息即为该当前姿态下该待处理图像中各像素点的第一深度信息。
本实施例中的图像处理方法，通过至少两种方式获取当前姿态下同一待处理图像的深度信息，得到至少两种深度信息。将获取的至少两种深度信息相互融合叠加，可得到更丰富更全面的细节信息，从而得到更准确的待处理图像中各像素点的深度信息。
在本实施例中，可以通过TOF摄像头获取待处理图像的初始深度信息，也可以通过双摄摄像头获取，还可通过结构光摄像头、红外摄像头等获取该待处理图像的初始深度信息。
在本实施例中，可通过TOF摄像头采集初始图像，并对初始图像中分辨率低的区域进行插值处理，或者对整张初始图像进行插值处理，从而得到比该初始图像的分辨率高的待处理图像。插值处理方式包括但不限于双平方插值方式、双立方插值方式。
在一个实施例中，该深度信息包括深度值；该将该获取的至少两种深度信息进行融合修正处理，得到在该当前姿态下该待处理图像中各像素点的第一深度信息，包括：
确定各像素点分别对应的至少两种深度信息的均值；将该各像素点对应的均值作为在该当前姿态下该待处理图像中各像素点的第一深度值。
具体地，电子设备的ISP处理器或中央处理器可通过三种方式获取在当前姿态下的同一张待处理图像中各像素点的深度信息，可得到每个像素点对应三种深度信息。即待处理图像中的每个像素点均对应初始深度信息、第三深度信息和第四深度信息。电子设备的ISP处理器或中央处理器可获取每个像素点分别对应的至少两种深度信息，并计算所获取的至少两种深度信息的均值，可得到各像素点分别对应的均值。将该各像素点对应的均值作为在该当前姿态下该待处理图像中各像素点的第一深度值。
可以理解的是，所有像素点选取的深度信息为使用相同方式获取的深度信息，例如，每个像素点获取各自对应的初始深度信息和第三深度信息；每个像素点获取各自对应的初始深度信息和第四深度信息；每个像素点获取各自对应的第三深度信息和第四深度信息；每个像素点获取各自对应的初始深度信息、第三深度信息和第四深度信息。
本实施例中的图像处理方法，确定各像素点分别对应的至少两种深度信息的均值，将该各像素点对应的均值作为在当前姿态下待处理图像中各像素点的第一深度值，提供了多种得到第一深度值的方式。通过不同的深度信息的融合，可获取待处理图像更丰富的细节信息，使得计算得到的待处理图像中各像素点的深度信息更准确。
在一个实施例中，如图4所示，该获取在当前姿态下待处理图像中各像素点的第一深度信息，包括：
操作402，获取在当前姿态下待处理图像中各像素点的初始深度信息。
具体地，电子设备的ISP处理器或中央处理器可通过深度摄像头在当前姿态下拍摄同一场景的待处理图像，可直接得到该待处理图像中各像素点的初始深度信息。
在本实施例中，该深度摄像头可以是TOF摄像头，双摄摄像头和结构光摄像头等。电子设备的ISP处理器或中央处理器可通过TOF摄像头，双摄摄像头和结构光摄像头等图像采集设备直接拍摄得到当前姿态下的待处理图像，不需要其它转化处理，可简单快速得到待处理图像中各像素点的深度信息，从而提高图像处理的速度。
操作404，获取该当前姿态下对焦区域的深度信息。
具体地，电子设备的ISP处理器或中央处理器可确定当前拍摄的同一场景，通过马达驱动镜头进行光学对焦后，可确定当前姿态下的对焦位置(即对焦区域)，即可根据预设的对焦位置与焦距值的对应关系，以及焦距值与深度信息的对映射关系，可得到当前姿态下的对焦区域中每个像素点对应的深度信息。
操作406，将该初始深度信息和该当前姿态下对焦区域的深度信息进行融合修正处理，得到在该当前姿态下该待处理图像中各像素点的第一深度信息。
具体地，通过深度摄像头直接拍摄得到的待处理图像的深度信息的精度没有光学对焦时获得深度信息的精度高。即通过深度摄像头获取的在当前姿态下待处理图像中各像素点的初始深度信息的精度，比光学对焦后该当前姿态下对焦区域的深度信息的精度低。则电子设备的ISP处理器或中央处理器可将当前姿态下对焦区域的深度信息和通过深度摄像头直接拍摄得到的待处理图像的深度信息相互融合叠加，使得当前姿态下待处理图像中各像素点的深度信息均可达到对焦区域的深度信息的精度，从而得到在该当前姿态下该待处理图像中各像素点的第一深度信息。
本实施例中的图像处理方法，通过获取在当前姿态下待处理图像中各像素点的初始深 度信息，获取该当前姿态下对焦区域的深度信息，可得到当前姿态下同一待处理图像的两种不同精度的深度信息。将该初始深度信息和该当前姿态下对焦区域的深度信息进行融合修正处理，得到在该当前姿态下该待处理图像中各像素点的第一深度信息，可将高精度的局部深度信息应用于待处理图像的各个部分，从而得到该待处理图像更准确的深度信息，即得到在该当前姿态下该待处理图像中各像素点的第一深度信息。
在一个实施例中，该深度信息包括深度值，该将该初始深度信息和该当前姿态下对焦区域的深度信息进行融合修正处理，得到在该当前姿态下该待处理图像中各像素点的第一深度信息，包括：确定在该当前姿态下待处理图像中与该当前姿态下对焦区域的各像素点相匹配的像素点；确定该对焦区域的深度值和该待处理图像中相匹配的像素点的初始深度值之间的差值或比值；根据在该当前姿态下该待处理图像中各像素点的初始深度信息和该差值或比值，确定在该当前姿态下该待处理图像中各像素点的第一深度信息。
具体地，电子设备的ISP处理器或中央处理器可获取当前姿态下的对焦区域的深度信息，得到待处理图像中处于对焦区域的部分图像的局部深度信息。电子设备的ISP处理器或中央处理器可进一步确定该对焦区域的深度值。接着，将对焦区域的部分图像中的每个像素点与当前姿态下待处理图像中的各像素进行匹配，确定对焦区域的部分图像的像素点和当前姿态下待处理图像中具有对应关系的像素点。
在本实施例中，对焦区域存在多个像素点，各像素点的深度值相同。该多个像素点指至少两个像素点。
接着，电子设备的ISP处理器或中央处理器获取当前姿态下待处理图像中匹配成功的像素点的初始深度值，并获取对焦区域的深度值，并计算匹配成功的两个像素点之间的差值，可得到每个匹配成功的像素点对应的差值。或者计算匹配成功的两个像素点之间的比值，可得到每个匹配成功的像素点对应的比值。
接着，电子设备的ISP处理器或中央处理器可根据每个匹配成功的像素点对应的差值确定第一调整值。将在该当前姿态下该待处理图像中各像素点的初始深度值加上该第一调整值，得到在该当前姿态下该待处理图像中各像素点的第一深度值，该各像素点的第一深度值即为该当前姿态下该待处理图像中各像素点对应的第一深度信息。
或者，电子设备的ISP处理器或中央处理器可根据每个匹配成功的像素点对应的比值确定第二调整值。将在该当前姿态下该待处理图像中各像素点的初始深度值乘以该第二调整值，得到在该当前姿态下该待处理图像中各像素点的第一深度值，该各像素点的第一深度值即为该当前姿态下该待处理图像中各像素点对应的第一深度信息。
在本实施例中，根据每个匹配成功的像素点对应的差值确定第一调整值,包括：确定相匹配的像素点对应的各差值的中间值或最大值或最小值；将该中间值或最大值或最小值中的任一个作为第一调整值。
在本实施例中，根据每个匹配成功的像素点对应的比值确定第二调整值,包括：确定相匹配的像素点对应的各差值的中间值或最大值或最小值；将该中间值或最大值或最小值中的任一个作为第二调整值。
本实施例中的图像处理方法，确定该对焦区域的深度值和该待处理图像中相匹配的像素点的初始深度值之间的差值或比值，可确定对焦区域的深度值与该待处理图像中相匹配的像素点的初始深度值之间的差距。根据该差值或比值和在该当前姿态下该待处理图像中各像素点的初始深度信息，可将对焦区域的局部图像的深度信息和待处理图像中的初始深度信息进行融合，得到该当前姿态下该待处理图像中各像素点准确的深度信息，从而提高了待处理图像的深度信息的精度。
在一个实施例中，该获取该当前姿态下对焦区域的深度信息，包括：
确定该当前姿态下的对焦区域，根据该对焦区域获取对应的焦距值；根据该焦距值获取对应的深度信息，将与该焦距值对应的深度信息作为该当前姿态下对焦区域的深度信 息。
其中，对焦区域为马达驱动镜头进行对焦后的对焦位置。
具体地，电子设备的ISP处理器或中央处理器可确定当前马达驱动镜头进行对焦后的对焦位置，并根据预设的对焦区域与焦距值之间的对应关系，获取当前姿态下的对焦区域对应的焦距值，该当前姿态下的对焦区域对应的焦距值即为当前姿态下镜头的焦距值。接着，再通过预设的镜头的焦距值与深度信息之间的对应关系，获取当前姿态下镜头的焦距值对应的深度信息，将与当前姿态下镜头的焦距值对应的深度信息作为该当前姿态下对焦区域的深度信息。
本实施例中的图像处理方法，可根据预设的对焦区域与焦距值的对应关系得到当前姿态下的对焦区域对应的焦距值，并根据预设的焦距值与深度信息的对应关系得到当前姿态下镜头的焦距值对应的深度信息，从而可间接得到当前姿态下的对焦区域对应的深度信息。
在一个实施例中，该根据该对焦区域获取对应的焦距值，包括：
获取对焦区域与焦距值之间的映射关系，根据该映射关系获取对焦区域对应的焦距值。
具体地，对焦区域与焦距值之间的映射关系为预先设置的。电子设备的ISP处理器或中央处理器可获取该映射关系，从该映射关系中确定与该当前姿态下的对焦区域相同的对焦区域。并获取映射关系中相同的对焦区域对应的焦距值，则该焦距值即为当前姿态下的对焦区域对应的焦距值，从而可通过预先设置的对焦区域与焦距值之间的映射关系，快速得到当前姿态下的对焦区域对应的焦距值。
在一个实施例中，该对焦区域与焦距值之间的映射关系可通过关系映射表体现，该获取对焦区域与焦距值之间的映射关系，根据该映射关系获取对焦区域对应的焦距值，包括：
获取关系映射表，从该关系映射表中获取该对焦区域对应的焦距值，该关系映射表中记录对焦区域和焦距值之间的映射关系。
其中，关系映射表为预先设置的对焦区域与对焦值之间的对应关系的表，用于记录对焦区域和焦距值之间的映射关系。
具体地，电子设备的ISP处理器或中央处理器可获取关系映射表，将该当前姿态下的对焦区域与关系映射表中的对焦区域一一对比，确定该关系映射表中的与当前姿态下的对焦区域相同的对焦区域。接着，可获取该关系映射表中相同的对焦区域对应的焦距值，则该焦距值即为当前姿态下的对焦区域对应的焦距值。通过关系映射表可快速简单的得到当前姿态下的对焦区域对应的焦距值。
在一个实施例中，该根据该焦距值获取对应的深度信息，包括：
获取对焦曲线，从该对焦曲线中获取该焦距值对应的深度信息，该对焦曲线为焦距值与深度信息之间的关联曲线。
其中，对焦曲线是当前镜头的焦距值和对焦区域的深度信息之间的关联曲线。
具体地，该对焦曲线为预先根据焦距值和深度信息建立的曲线，每个焦距值均对应一个深度信息。电子设备的ISP处理器或中央处理器可获取对焦曲线，根据该对焦曲线中记录镜头的焦距值和对焦区域的深度信息的对应关系，可得到与该当前姿态下的对焦区域对应的焦距值具有映射关系的深度信息，并将该深度信息作为当前姿态下对焦区域的深度信息。通过对焦曲线可快速简单的得到当前姿态下的对焦区域的焦距值对应的深度信息，从而可间接得到当前姿态下的对焦区域对应的深度信息。
在一个实施例中，该根据该目标姿态信息和在该当前姿态下该待处理图像中各像素点的第一深度信息，确定在目标姿态下该待处理图像中各像素点的第二深度信息，包括：
获取在该当前姿态下该待处理图像中各像素点对应的第一三维坐标；采用坐标转换算法根据该目标姿态信息将该第一三维坐标转换为第二三维坐标，得到在该目标姿态下该待 处理图像中各像素点的第二深度信息，该第二三维坐标为在该目标姿态下该待处理图像中各像素点对应的三维坐标。
具体地，电子设备的ISP处理器或中央处理器获取在该当前姿态下该待处理图像中各像素点在世界坐标系中的三维坐标，即第一三维坐标，通过坐标转换算法将当前姿态下的各像素点对应的第一三维坐标转换为目标姿态下的第二三维坐标。进一步地，该第二三维坐标为在该目标姿态下该待处理图像中各像素点在世界坐标系中的三维坐标。通过坐标转换算法可将待处理图像中的每个像素点的三维坐标从当前姿态转换到目标姿态，从而确定每个像素点的重投影方式。
例如，坐标转换算法为：
Figure PCTCN2019102801-appb-000003
其中，(x,y,z)为当前姿态下各像素点对应的第一三维坐标，(x′,y′,z′)为目标姿态下各像素点对应的第二三维坐标，R和T为在当前姿态下的摄像头在世界坐标系下的旋转矩阵和平移矩阵，表征了摄像头的当前姿态信息。R′和T′为在目标姿态下的摄像头在世界坐标系下的旋转矩阵和平移矩阵，表征了摄像头的目标姿态信息。电子设备的ISP处理器或中央处理器获取在该当前姿态下该待处理图像中各像素点的(x,y,z)，根据当前姿态信息和目标姿态信息可得到R、T、R′和T′，将这些数据代入上述坐标转换算法，即可计算出各像素点对应的(x′,y′,z′)。
本实施例中的图像处理方法，通过采用坐标转换算法将当前姿态下的待处理图像中的各像素点的坐标转换目标姿态下的对应坐标，从而可针对性的对每个像素点进行重投影处理，实现每个像素点的针对性防抖。
在一个实施例中，如图5所示，该获取该摄像头的第一内参信息，包括：
操作502，获取该摄像头的初始内参信息。
操作504，获取该当前姿态下的该摄像头的焦距值。
其中，初始内参信息为预先设置的在特定对焦距离下的摄像头内参信息。
具体地，摄像头出厂前需要进行标定，初始内参信息是摄像头出厂前预先设置的在特定对焦距离下的内参信息。电子设备的ISP处理器或中央处理器获取摄像头的初始内参信息，该初始内参信息中包括焦距值和摄像头的中心坐标偏移量。对焦时，在不同的对焦位置，镜头的焦距值会发生改变，电子设备的ISP处理器或中央处理器可根据当前姿态下的对焦区域获取该摄像头镜头的实时焦距值。
操作506，根据该当前姿态下的该摄像头的焦距值更新该摄像头的初始内参信息，得到该摄像头的第一内参信息。
其中，第一内参信息为更新初始内参信息中的焦距值和偏移量中的至少一个后所得到的内参信息。
具体地，电子设备的ISP处理器或中央处理器可用该当前姿态下的镜头的焦距值替换该摄像头的初始内参信息中的焦距值，从而更新该内参信息，更新后的内参信息为摄像头的第一内参信息。
本实施例中的图像处理方法，通过获取该当前姿态下的该摄像头的实时焦距值，以更新摄像头的内参信息，能够解决传统图像处理方法中使用相同的焦距值将待处理图像中的各像素点均投影到同一单位平面，导致无法检测出图像各像素点的真实深度信息的问题，本实施例中的图像处理方法提高了深度信息检测的准确性。
在一个实施例中，如图6所示，该方法还包括：
操作602，通过霍尔传感器获取该当前姿态下的霍尔值。
操作604，基于该霍尔值确定该当前姿态下的该摄像头的偏移量。
其中，霍尔传感器(Hall sensor)是根据霍尔效应制作的一种磁场传感器，霍尔效应从本质上讲是运动的带电粒子在磁场中受洛仑兹力作用引起的偏转。当带电粒子(电子或空穴)被约束在固体材料中，这种偏转就导致在垂直电流和磁场的方向上产生正负电荷的聚积，从而形成附加的横向电场。
具体地，电子设备可以通过霍尔传感器记录摄像头的镜头在XY平面上的偏移刻度，并记录偏移刻度的同时，还可以记录偏移的方向，根据每个刻度对应的距离，以及偏移方向，继而得到镜头偏移(c x,c y)。在本申请实施例中，已知霍尔传感器采集的霍尔值的大小，即可唯一确定出当前时刻该镜头偏移的大小。其中，陀螺仪传感器采集的角速度信息与霍尔传感器采集的霍尔值在时序上相对应。
该根据该当前姿态下的该摄像头的焦距值更新该摄像头的初始内参信息，得到该摄像头的第一内参信息，包括：
操作606，根据该当前姿态下的该摄像头的焦距值和该摄像头的偏移量更新该摄像头的初始内参信息，得到该摄像头的第一内参信息。
具体地，摄像头的初始内参信息包括摄像头的焦距值f、摄像头在X平面上的偏移量c x和摄像头在X平面上的偏移量c y。电子设备的ISP处理器或中央处理器可将在当前姿态下采集到的摄像头的偏移量和当前镜头的焦距值，替换初始内参信息中的偏移量和焦距值，从而得到该摄像头的第一内参信息。
本实施例中的图像处理方法，通过霍尔传感器获取该当前姿态下的霍尔值，基于该霍尔值确定该当前姿态下的该摄像头的偏移量，从而获取实时的摄像头偏移量，根据当前姿态下的该摄像头的焦距值和摄像头的偏移量更新该摄像头的初始内参信息，更能够满足在变焦情况下进行拍摄的防抖需求。
如图7所示，为一个实施例中光学防抖的原理图。光学防抖一般是通过在x，y轴上移动镜头，来补偿摄像头转动造成的姿态差。姿态差为摄像头当前姿态与目标姿态之间的差异。
如图8所示，为一个实施例中图像处理方法的原理图。
电子设备的IPS处理器或中央处理器通过陀螺仪获取当前的三轴角速度信息，通过加速度计获取当前的三轴加速度信息，通过高度表获取当前的高度信息，通过指南针获取当前的方位信息，通过GPS定位系统获取当前的地理位置信息。接着，将当前的三轴角速度信息、当前的三轴加速度信息、当前的高度信息、当前的方位信息和当前的地理位置信息进行融合得到摄像头的当前姿态信息。并根据当前的方位信息和当前的地理位置信息，得到摄像头当前的运动场景信息。电子设备的IPS处理器或中央处理器还可通过TOF摄像头、通过双摄视差测距方式、相位对焦方式分别获取在当前姿态下同一待处理图像中各像素点的深度信息。并将三种方式获取得到的三种深度信息进行融合，得到当前姿态下待处理图像中各像素点准确的深度信息。电子设备的IPS处理器或中央处理器还通过马达驱动镜头对焦，根据镜头的位置、焦距和在X、Y方向的偏移量更新摄像头的初始内参信息。 接着，电子设备的IPS处理器或中央处理器可根据摄像头的当前姿态信息、当前的运动场景信息、更新后的内参信息和各像素点的准确深度信息等对待处理图像进行重投影处理，输出稳定的目标图像。
在一个实施例中，提供了一种图像处理方法，包括：
操作(b1)，获取摄像头当前的角速度信息和当前的加速度信息，并获取该摄像头当前的高度信息、当前的方位信息和当前的地理位置信息中的至少一种。
操作(b2)，将该摄像头当前的高度信息、该当前的方位信息和该当前的地理位置信息中的至少一种与该当前的角速度信息，以及当前的加速度信息进行融合处理，得到摄像头的当前姿态信息。
操作(b3)，将该当前的方位信息和该当前的地理位置信息进行融合处理，得到当前的运动场景信息。
操作(b4)，根据该当前的运动场景信息和该当前姿态信息，确定目标姿态信息。
操作(b5)，获取初始深度信息、第三深度信息和第四深度信息中的至少两种深度信息。其中，该初始深度信息是通过深度摄像头获取的在当前姿态下待处理图像中各像素点的深度信息。
操作(b6)，该第三深度信息是通过双摄视差测距方式获取的在该当前姿态下该待处理图像中各像素点的深度信息。
操作(b7)，该第四深度信息是通过相位对焦方式获取的在该当前姿态下该待处理图像中各像素点的深度信息。
操作(b8)，该深度信息包括深度值，确定各像素点分别对应的该至少两种深度信息的均值。
操作(b9)，将该各像素点对应的均值作为在该当前姿态下该待处理图像中各像素点的第一深度值。
操作(b10)，获取在该当前姿态下该待处理图像中各像素点对应的第一三维坐标。
操作(b11)，采用坐标转换算法根据该目标姿态信息将该第一三维坐标转换为第二三维坐标，得到在该目标姿态下该待处理图像中各像素点的第二深度信息，该第二三维坐标为在该目标姿态下该待处理图像中各像素点对应的三维坐标。
操作(b12)，获取该摄像头的初始内参信息。获取该当前姿态下的该摄像头的焦距值。
操作(b13)，通过霍尔传感器获取该当前姿态下的霍尔值。
操作(b14)，基于该霍尔值确定该当前姿态下的该摄像头的偏移量。
操作(b15)，根据该当前姿态下的该摄像头的焦距值和该摄像头的偏移量更新该摄像头的初始内参信息，得到该摄像头的第一内参信息。
操作(b16)，根据该当前姿态信息、该第一深度信息、该目标姿态信息、该第二深度信息和该第一内参信息对该待处理图像进行重投影处理，得到目标图像。
本实施例中的图像处理方法，将摄像头当前的高度信息、当前的方位信息和当前的地理位置信息中的至少一种与当前的角速度信息，以及当前的加速度信息进行融合处理，根据多种信息融合得到摄像头的当前姿态信息更准确。通过TOF摄像头、通过双摄视差测距方式、相位对焦方式三种方式分别获取在当前姿态下同一待处理图像中各像素点的深度信息。并将三种方式获取得到的三种深度信息进行融合，得到更丰富更全面的细节信息，使得当前姿态下待处理图像中各像素点的深度信息的精度更高。将当前姿态下的各像素点的坐标转换为目标姿态下对应的坐标，可确定待处理图像在稳定状态下的三维坐标。根据当前姿态下获得的摄像头镜头的焦距值和镜头偏移量更新初始内参信息，使得拍摄画面内的物距更贴近实际场景的物距。根据当前姿态信息、第一深度信息、目标姿态信息、第二深度信息和第一内参信息对待处理图像进行重投影处理，可得到各像素点在目标姿态下的像素坐标，从而可输出防抖效果一致的目标图像。本实施例中的图像处理方法实现了光学 防抖，电子防抖的优势结合，提供了二者之间协作的理论依据、数据通路和算法原型，在图像传感器成像前后均能实现防抖。
在一个实施例中，如图9所示，提供了一种深度信息获取方法，包括：
操作902，获取摄像头的当前姿态信息，该当前姿态信息包括当前的角速度信息。
具体地，电子设备的ISP处理器或中央处理器可通过陀螺仪获取摄像头的三轴角速度，将该三轴角速度经过校正和在时间域上的积分处理，输出三轴角速度信息。
操作904，将该当前姿态信息转换为目标姿态信息。
具体地，电子设备的ISP处理器或中央处理器得到摄像头的当前姿态信息后，可根据当前姿态进行预测，确定当前姿态对应的目标姿态。进一步地，可通过目标姿态预测算法将当前姿态信息转换为目标姿态信息。
操作906，获取在当前姿态下待处理图像中各像素点的第一深度信息。
具体地，电子设备的ISP处理器或中央处理器可获取当前姿态下的待处理图像中每个像素点对应的第一深度信息。
操作908，根据该目标姿态信息和在该当前姿态下该待处理图像中各像素点的第一深度信息，确定在目标姿态下该待处理图像中各像素点的第二深度信息。
具体地，电子设备的ISP处理器或中央处理器将该当前姿态信息转换为目标姿态信息后，得到当前姿态对应的目标姿态。可根据当前姿态和目标姿态，将在当前姿态下待处理图像中各像素点的第一深度信息经过坐标变换，转换为目标姿态下待处理图像中各像素点的第二深度信息。
本实施例中的深度信息获取方法，获取摄像头的当前姿态信息，该当前姿态信息包括当前的角速度信息，将当前姿态信息转换为目标姿态信息，获取在当前姿态下待处理图像中各像素点的第一深度信息，根据目标姿态信息和在当前姿态下待处理图像中各像素点的第一深度信息，确定在目标姿态下待处理图像中各像素点的第二深度信息，可获得的待处理图像在目标姿态下的准确深度信息。
在一个实施例中，该当前姿态信息还包括当前的高度信息、当前的方位信息和当前的地理位置信息中的至少一种；该获取摄像头的当前姿态信息，包括：
获取摄像头当前的角速度信息，并获取该摄像头当前的高度信息、当前的方位信息和当前的地理位置信息中的至少一种；将该当前的高度信息、该当前的方位信息和该当前的地理位置信息中的至少一种和该当前的角速度信息进行融合处理，得到该摄像头的当前姿态信息。
具体地，电子设备的ISP处理器或中央处理器可通过陀螺仪获取摄像头在当前姿态下的角速度信息，可通过高度表检测摄像头在当前姿态下距离地平面的高度，可通过指南针获取摄像头在当前姿态下的方位，还可通过GPS获取摄像头在当前姿态下的地理坐标。
接着，电子设备的ISP处理器或中央处理器可选取当前的高度信息、当前的方位信息和当前的地理位置信息中的至少一种，将选取的信息和当前的角速度信息进行融合处理，得到该摄像头的当前姿态信息。
进一步地，可将选取的信息和当前的角速度信息经过卡尔曼滤波处理，以实现信息的融合，得到该摄像头的当前姿态信息。卡尔曼滤波本质上是一种数据融合算法，将具有相同目的、来自不同传感器、具有不同单位的数据融合在一起，得到一个更精确的目的测量值。
本实施例中的图像处理方法，通过获取摄像头当前的角速度信息，并获取该摄像头当前的高度信息、当前的方位信息和当前的地理位置信息中的至少一种，可得到多种用于表征摄像头当前姿态的不同信息。将当前的高度信息、当前的方位信息和当前的地理位置信息中的至少一种和当前的角速度信息进行融合处理，得到该摄像头的当前姿态信息，提供了多种获取摄像头当前姿态信息的方式。通过不同信息的融合，能够更准确得到摄像头的 当前姿态信息。
在一个实施例中，该当前姿态信息还包括当前的加速度信息；该方法还包括：获取摄像头当前的加速度信息；
该将该当前的高度信息、该当前的方位信息和该当前的地理位置信息中的至少一种和该当前的角速度信息进行融合处理，得到该摄像头的当前姿态信息，包括：将该摄像头当前的高度信息、该当前的方位信息和该当前的地理位置信息中的至少一种与该当前的角速度信息，以及当前的加速度信息进行融合处理，得到摄像头的当前姿态信息。
具体地，电子设备的ISP处理器或中央处理器可通过加速度计获取三轴加速度，将该三轴加速度经过校正处理以及在两次时间域上的积分，可得到摄像头的三轴位置信息。三轴位置信息可转换为摄像头在世界坐标系下的平移矩阵。接着，可从摄像头当前的高度信息、当前的方位信息和当前的地理位置信息中选取至少一种，并将选取的信息与当前的角速度信息和当前的加速度信息进行融合处理。该融合处理可通过卡尔曼滤波算法等融合算法实现。
本实施例中的图像处理方法，通过获取当前的加速度信息，将摄像头当前的高度信息、当前的方位信息和当前的地理位置信息中的至少一种与当前的角速度信息，以及当前的加速度信息进行融合处理，提供了多种获取摄像头当前姿态信息的方式。并且将当前的加速度信息也作为获取当前姿态信息的一个参考量，能够更准确得到摄像头的当前姿态信息。
应该理解的是，虽然图2-图9的流程图中的各个操作按照箭头的指示依次显示，但是这些操作并不是必然按照箭头指示的顺序依次执行。除非本文中有明确的说明，这些操作的执行并没有严格的顺序限制，这些操作可以以其它的顺序执行。而且，图2-图9中的至少一部分操作可以包括多个子操作或者多个阶段，这些子操作或者阶段并不必然是在同一时刻执行完成，而是可以在不同的时刻执行，这些子操作或者阶段的执行顺序也不必然是依次进行，而是可以与其它操作或者其它操作的子操作或者阶段的至少一部分轮流或者交替地执行。
图10为一个实施例的图像处理装置的结构框图。如图10所示，该图像处理装置包括：姿态获取模块1002、转换模块1004、深度信息获取模块1006、确定模块1008和目标图像确定模块1010。其中，
姿态获取模块1002，用于获取摄像头的当前姿态信息，该当前姿态信息包括当前的角速度信息。
转换模块1004，用于将该当前姿态信息转换为目标姿态信息。
深度信息获取模块1006，用于获取在当前姿态下待处理图像中各像素点的第一深度信息。
确定模块1008，用于根据该目标姿态信息和在该当前姿态下该待处理图像中各像素点的第一深度信息，确定在目标姿态下该待处理图像中各像素点的第二深度信息。
目标图像确定模块1010，用于获取该摄像头的第一内参信息，根据该当前姿态信息、该第一深度信息、该目标姿态信息、该第二深度信息和该第一内参信息对该待处理图像进行重投影处理，得到目标图像。
本实施例中的图像处理装置，获取摄像头的当前姿态信息，该当前姿态信息包括当前的角速度信息，将当前姿态信息转换为目标姿态信息，获取在当前姿态下待处理图像中各像素点的第一深度信息，根据目标姿态信息和在当前姿态下待处理图像中各像素点的第一深度信息，确定在目标姿态下待处理图像中各像素点的第二深度信息，获取摄像头的第一内参信息，根据当前姿态信息、第一深度信息、目标姿态信息、第二深度信息和第一内参信息对待处理图像进行重投影处理，得到目标图像，可以实现对每个像素点进行针对性的防抖，使得拍摄的图像防抖效果更稳定。
在一个实施例中，该当前姿态信息还包括当前的高度信息、当前的方位信息和当前的 地理位置信息中的至少一种；
该姿态获取模块1002还用于：获取摄像头当前的角速度信息，并获取该摄像头当前的高度信息、当前的方位信息和当前的地理位置信息中的至少一种；将该当前的高度信息、该当前的方位信息和该当前的地理位置信息中的至少一种和该当前的角速度信息进行融合处理，得到该摄像头的当前姿态信息。
通过获取摄像头当前的角速度信息，并获取该摄像头当前的高度信息、当前的方位信息和当前的地理位置信息中的至少一种，可得到多种用于表征摄像头当前姿态的不同信息。将当前的高度信息、当前的方位信息和当前的地理位置信息中的至少一种和当前的角速度信息进行融合处理，得到该摄像头的当前姿态信息，提供了多种获取摄像头当前姿态信息的方式。通过不同信息的融合，能够更准确得到摄像头的当前姿态信息。
在一个实施例中，该当前姿态信息包括还包括当前的加速度信息；
该姿态获取模块1002还用于：获取摄像头当前的加速度信息；将该摄像头当前的高度信息、该当前的方位信息和该当前的地理位置信息中的至少一种与该当前的角速度信息，以及当前的加速度信息进行融合处理，得到摄像头的当前姿态信息。
通过获取当前的加速度信息，将摄像头当前的高度信息、当前的方位信息和当前的地理位置信息中的至少一种与当前的角速度信息，以及当前的加速度信息进行融合处理，提供了多种获取摄像头当前姿态信息的方式。并且将当前的加速度信息也作为获取当前姿态信息的一个参考量，能够更准确得到摄像头的当前姿态信息。
在一个实施例中，该转换模块1004还用于：获取当前的运动场景信息；根据该当前的运动场景信息和该当前姿态信息，确定目标姿态信息。而本实施例中根据摄像头当前的运动状态息，可确定当前姿态上的待消除抖动的方向和不需要消除抖动的方向，以对当前姿态信息中各方向上的信息进行针对性处理，使得预测得到的目标姿态信息更加准确，更符合实际拍摄场景
在一个实施例中，该转换模块1004还用于：获取该摄像头当前的方位信息和当前的地理位置信息；将该当前的方位信息和该当前的地理位置信息进行融合处理，得到当前的运动场景信息。通过获取该摄像头当前的方位信息和当前的地理位置信息，可确定摄像头方向位置和地理坐标的变动情况。将该当前的方位信息和该当前的地理位置信息进行融合处理，根据该摄像头方向位置和地理坐标的变动情况，可直观准确的确定该摄像头在拍摄过程中所保持的运动状态。
在一个实施例中，深度信息获取模块1006还用于：获取初始深度信息、第三深度信息和第四深度信息中的至少两种深度信息；其中，该初始深度信息是通过深度摄像头获取的在当前姿态下待处理图像中各像素点的深度信息；该第三深度信息是通过双摄视差测距方式获取的在该当前姿态下该待处理图像中各像素点的深度信息；该第四深度信息是通过相位对焦方式获取的在该当前姿态下该待处理图像中各像素点的深度信息；将该获取的至少两种深度信息进行融合修正处理，得到在该当前姿态下该待处理图像中各像素点的第一深度信息。
本实施例中的图像处理装置，通过至少两种方式获取当前姿态下同一待处理图像的深度信息，得到至少两种深度信息。将获取的至少两种深度信息相互融合叠加，可得到更丰富更全面的细节信息，从而得到更准确的待处理图像中各像素点的深度信息。
在一个实施例中，该深度信息包括深度值；深度信息获取模块1006还用于：确定各像素点分别对应的该至少两种深度信息的均值；将该各像素点对应的均值作为在该当前姿态下该待处理图像中各像素点的第一深度值。
本实施例中的图像处理装置，确定各像素点分别对应的至少两种深度信息的均值，将该各像素点对应的均值作为在当前姿态下待处理图像中各像素点的第一深度值，提供了多种得到第一深度值的方式。通过不同的深度信息的融合，可获取待处理图像更丰富的细 节信息，使得计算得到的待处理图像中各像素点的深度信息更准确。
在一个实施例中，该深度信息获取模块1006还用于：获取在当前姿态下待处理图像中各像素点的初始深度信息；获取该当前姿态下对焦区域的深度信息；将该初始深度信息和该当前姿态下对焦区域的深度信息进行融合修正处理，得到在该当前姿态下该待处理图像中各像素点的第一深度信息。
本实施例中的图像处理装置，通过获取在当前姿态下待处理图像中各像素点的初始深度信息，获取该当前姿态下对焦区域的深度信息，可得到当前姿态下同一待处理图像的两种不同精度的深度信息。将该初始深度信息和该当前姿态下对焦区域的深度信息进行融合修正处理，得到在该当前姿态下该待处理图像中各像素点的第一深度信息，可将高精度的局部深度信息应用于待处理图像的各个部分，从而得到该待处理图像更准确的深度信息，即得到在该当前姿态下该待处理图像中各像素点的第一深度信息。
在一个实施例中，深度信息获取模块1006还用于：确定该当前姿态下的对焦区域，根据该对焦区域获取对应的焦距值；根据该焦距值获取对应的深度信息，将与该焦距值对应的深度信息作为该当前姿态下对焦区域的深度信息。根据预设的对焦区域与焦距值的对应关系得到当前姿态下的对焦区域对应的焦距值，并根据预设的焦距值与深度信息的对应关系得到当前姿态下镜头的焦距值对应的深度信息，从而可间接得到当前姿态下的对焦区域对应的深度信息。
在一个实施例中，该深度信息获取模块1006还用于：获取对焦曲线，从该对焦曲线中获取该焦距值对应的深度信息，该对焦曲线为焦距值与深度信息之间的关联曲线。通过对焦曲线可快速简单的得到当前姿态下的对焦区域的焦距值对应的深度信息，从而可间接得到当前姿态下的对焦区域对应的深度信息。
在一个实施例中，该确定模块1008还用于：获取在该当前姿态下该待处理图像中各像素点对应的第一三维坐标；采用坐标转换算法根据该目标姿态信息将该第一三维坐标转换为第二三维坐标，得到在该目标姿态下该待处理图像中各像素点的第二深度信息，该第二三维坐标为在该目标姿态下该待处理图像中各像素点对应的三维坐标。通过采用坐标转换算法将当前姿态下的待处理图像中的各像素点的坐标转换目标姿态下的对应坐标，从而可针对性的对每个像素点进行重投影处理，实现每个像素点的针对性防抖。
在一个实施例中，该确定模块1008还用于：获取该摄像头的初始内参信息；获取该当前姿态下的该摄像头的焦距值；根据该当前姿态下的该摄像头的焦距值更新该摄像头的初始内参信息，得到该摄像头的第一内参信息。通过获取该当前姿态下的该摄像头的实时焦距值，以更新摄像头的内参信息，能够解决传统图像处理方法中使用相同的焦距值将待处理图像中的各像素点均投影到同一单位平面，导致无法检测出图像各像素点的真实深度信息的问题，本实施例中的图像处理方法提高了深度信息检测的准确性。
在一个实施例中，该确定模块1008还用于：通过霍尔传感器获取该当前姿态下的霍尔值；基于该霍尔值确定该当前姿态下的该摄像头的偏移量；根据该当前姿态下的该摄像头的焦距值和该摄像头的偏移量更新该摄像头的初始内参信息，得到该摄像头的第一内参信息。通过霍尔传感器获取该当前姿态下的霍尔值，基于该霍尔值确定该当前姿态下的该摄像头的偏移量，从而获取实时的摄像头偏移量，根据当前姿态下的该摄像头的焦距值和摄像头的偏移量更新该摄像头的初始内参信息，更能够满足在变焦情况下进行拍摄的防抖需求。
在一个实施例中，如图11所示，提供了一种深度信息获取装置，包括：
姿态获取模块1102，用于获取摄像头的当前姿态信息，该当前姿态信息包括当前的角速度信息。
转换模块1104，用于将该当前姿态信息转换为目标姿态信息。
深度信息获取模块1106，用于获取在当前姿态下待处理图像中各像素点的第一深度 信息。
确定模块1108，用于根据该目标姿态信息和在该当前姿态下该待处理图像中各像素点的第一深度信息，确定在目标姿态下该待处理图像中各像素点的第二深度信息。
本实施例中的深度信息获取装置，获取摄像头的当前姿态信息，该当前姿态信息包括当前的角速度信息，将当前姿态信息转换为目标姿态信息，获取在当前姿态下待处理图像中各像素点的第一深度信息，根据目标姿态信息和在当前姿态下待处理图像中各像素点的第一深度信息，确定在目标姿态下待处理图像中各像素点的第二深度信息，可获得的待处理图像在目标姿态下的准确深度信息。
在一个实施例中，该当前姿态信息还包括当前的高度信息、当前的方位信息和当前的地理位置信息中的至少一种；该姿态获取模块1102还用于：获取摄像头当前的角速度信息，并获取该摄像头当前的高度信息、当前的方位信息和当前的地理位置信息中的至少一种；将该当前的高度信息、该当前的方位信息和该当前的地理位置信息中的至少一种和该当前的角速度信息进行融合处理，得到该摄像头的当前姿态信息。
通过获取摄像头当前的角速度信息，并获取该摄像头当前的高度信息、当前的方位信息和当前的地理位置信息中的至少一种，可得到多种用于表征摄像头当前姿态的不同信息。将当前的高度信息、当前的方位信息和当前的地理位置信息中的至少一种和当前的角速度信息进行融合处理，得到该摄像头的当前姿态信息，提供了多种获取摄像头当前姿态信息的方式。通过不同信息的融合，能够更准确得到摄像头的当前姿态信息。
在一个实施例中，该当前姿态信息还包括当前的加速度信息；该姿态获取模块1102还用于：获取摄像头当前的加速度信息；将该摄像头当前的高度信息、该当前的方位信息和该当前的地理位置信息中的至少一种与该当前的角速度信息，以及当前的加速度信息进行融合处理，得到摄像头的当前姿态信息。
通过获取当前的加速度信息，将摄像头当前的高度信息、当前的方位信息和当前的地理位置信息中的至少一种与当前的角速度信息，以及当前的加速度信息进行融合处理，提供了多种获取摄像头当前姿态信息的方式。并且将当前的加速度信息也作为获取当前姿态信息的一个参考量，能够更准确得到摄像头的当前姿态信息。
上述图像处装置中各个模块的划分仅用于举例说明，在其他实施例中，可将图像处理装置和深度信息获取装置按照需要划分为不同的模块，以完成上述图像处理装置和深度信息获取装置的全部或部分功能。
图12为一个实施例中电子设备的内部结构示意图。如图12所示，该电子设备包括通过系统总线连接的处理器和存储器。其中，该处理器用于提供计算和控制能力，支撑整个电子设备的运行。存储器可包括非易失性存储介质及内存储器。非易失性存储介质存储有操作系统和计算机程序。该计算机程序可被处理器所执行，以用于实现以下各个实施例所提供的一种图像处理方法和一种深度信息获取方法。内存储器为非易失性存储介质中的操作系统计算机程序提供高速缓存的运行环境。该电子设备可以是手机、平板电脑或者个人数字助理或穿戴式设备等。
本申请实施例中提供的图像处理装置和深度信息获取装置中的各个模块的实现可为计算机程序的形式。该计算机程序可在终端或服务器上运行。该计算机程序构成的程序模块可存储在终端或服务器的存储器上。该计算机程序被处理器执行时，实现本申请实施例中所描述方法的操作。
本申请实施例还提供了一种计算机可读存储介质。一个或多个包含计算机可执行指令的非易失性计算机可读存储介质，当所述计算机可执行指令被一个或多个处理器执行时，使得所述处理器执行图像处理方法的操作。本申请实施例所使用的对存储器、存储、数据库或其它介质的任何引用可包括非易失性和/或易失性存储器。合适的非易失性存储器可 包括只读存储器(ROM)、可编程ROM(PROM)、电可编程ROM(EPROM)、电可擦除可编程ROM(EEPROM)或闪存。易失性存储器可包括随机存取存储器(RAM)，它用作外部高速缓冲存储器。作为说明而非局限，RAM以多种形式可得，诸如静态RAM(SRAM)、动态RAM(DRAM)、同步DRAM(SDRAM)、双数据率SDRAM(DDR SDRAM)、增强型SDRAM(ESDRAM)、同步链路(Synchlink)DRAM(SLDRAM)、存储器总线(Rambus)直接RAM(RDRAM)、直接存储器总线动态RAM(DRDRAM)、以及存储器总线动态RAM(RDRAM)。
以上所述实施例仅表达了本申请的几种实施方式，其描述较为具体和详细，但并不能因此而理解为对本申请专利范围的限制。应当指出的是，对于本领域的普通技术人员来说，在不脱离本申请构思的前提下，还可以做出若干变形和改进，这些都属于本申请的保护范围。因此，本申请专利的保护范围应以所附权利要求为准。

Claims (20)

  1. 一种图像处理方法，包括：
    获取摄像头的当前姿态信息，所述当前姿态信息包括当前的角速度信息；
    将所述当前姿态信息转换为目标姿态信息；
    获取在当前姿态下待处理图像中各像素点的第一深度信息；
    根据所述目标姿态信息和在所述当前姿态下所述待处理图像中各像素点的第一深度信息，确定在目标姿态下所述待处理图像中各像素点的第二深度信息；
    获取所述摄像头的第一内参信息，根据所述当前姿态信息、所述第一深度信息、所述目标姿态信息、所述第二深度信息和所述第一内参信息对所述待处理图像进行重投影处理，得到目标图像。
  2. 根据权利要求1所述的方法，其特征在于，所述当前姿态信息还包括当前的高度信息、当前的方位信息和当前的地理位置信息中的至少一种；所述获取摄像头的当前姿态信息，包括：
    获取摄像头当前的角速度信息，并获取所述摄像头当前的高度信息、当前的方位信息和当前的地理位置信息中的至少一种；
    将所述当前的高度信息、所述当前的方位信息和所述当前的地理位置信息中的至少一种和所述当前的角速度信息进行融合处理，得到所述摄像头的当前姿态信息。
  3. 根据权利要求2所述的方法，其特征在于，所述当前姿态信息包括还包括当前的加速度信息；所述方法还包括：
    获取摄像头当前的加速度信息；
    所述将所述当前的高度信息、所述当前的方位信息和所述当前的地理位置信息中的至少一种和所述当前的角速度信息进行融合处理，得到所述摄像头的当前姿态信息，包括：
    将所述摄像头当前的高度信息、所述当前的方位信息和所述当前的地理位置信息中的至少一种与所述当前的角速度信息，以及当前的加速度信息进行融合处理，得到摄像头的当前姿态信息。
  4. 根据权利要求1所述的方法，其特征在于，所述将所述当前姿态信息转换为目标姿态信息，包括：
    获取当前的运动场景信息；
    根据所述当前的运动场景信息和所述当前姿态信息，确定目标姿态信息。
  5. 根据权利要求4所述的方法，其特征在于，所述获取当前的运动场景信息，包括：
    获取所述摄像头当前的方位信息和当前的地理位置信息；
    将所述当前的方位信息和所述当前的地理位置信息进行融合处理，得到当前的运动场景信息。
  6. 根据权利要求1所述的方法，其特征在于，所述获取在当前姿态下待处理图像中各像素点的第一深度信息，包括：
    获取初始深度信息、第三深度信息和第四深度信息中的至少两种深度信息；其中，所述初始深度信息是通过深度摄像头获取的在当前姿态下待处理图像中各像素点的深度信息；
    所述第三深度信息是通过双摄视差测距方式获取的在所述当前姿态下所述待处理图像中各像素点的深度信息；
    所述第四深度信息是通过相位对焦方式获取的在所述当前姿态下所述待处理图像中各像素点的深度信息；
    将所述获取的至少两种深度信息进行融合修正处理，得到在所述当前姿态下所述待处理图像中各像素点的第一深度信息。
  7. 根据权利要求6所述的方法，其特征在于，所述深度信息包括深度值；所述将所述获取的至少两种深度信息进行融合修正处理，得到在所述当前姿态下所述待处理图像中各像素点的第一深度信息，包括：
    确定各像素点分别对应的所述至少两种深度信息的均值；
    将所述各像素点对应的均值作为在所述当前姿态下所述待处理图像中各像素点的第一深度值。
  8. 根据权利要求1所述的方法，其特征在于，所述获取在当前姿态下待处理图像中各像素点的第一深度信息，包括：
    获取在当前姿态下待处理图像中各像素点的初始深度信息；
    获取所述当前姿态下对焦区域的深度信息；
    将所述初始深度信息和所述当前姿态下对焦区域的深度信息进行融合修正处理，得到在所述当前姿态下所述待处理图像中各像素点的第一深度信息。
  9. 根据权利要求8所述的方法，其特征在于，所述获取所述当前姿态下对焦区域的深度信息，包括：
    确定所述当前姿态下的对焦区域，根据所述对焦区域获取对应的焦距值；
    根据所述焦距值获取对应的深度信息，将与所述焦距值对应的深度信息作为所述当前姿态下对焦区域的深度信息。
  10. 根据权利要求8所述的方法，其特征在于，所述根据所述焦距值获取对应的深度信息，包括：
    获取对焦曲线，从所述对焦曲线中获取所述焦距值对应的深度信息，所述对焦曲线为焦距值与深度信息之间的关联曲线。
  11. 根据权利要求1所述的方法，其特征在于，所述根据所述目标姿态信息和在所述当前姿态下所述待处理图像中各像素点的第一深度信息，确定在目标姿态下所述待处理图像中各像素点的第二深度信息，包括：
    获取在所述当前姿态下所述待处理图像中各像素点对应的第一三维坐标；
    采用坐标转换算法根据所述目标姿态信息将所述第一三维坐标转换为第二三维坐标，得到在所述目标姿态下所述待处理图像中各像素点的第二深度信息，所述第二三维坐标为在所述目标姿态下所述待处理图像中各像素点对应的三维坐标。
  12. 根据权利要求1所述的方法，其特征在于，所述获取所述摄像头的第一内参信息，包括：
    获取所述摄像头的初始内参信息；
    获取所述当前姿态下的所述摄像头的焦距值；
    根据所述当前姿态下的所述摄像头的焦距值更新所述摄像头的初始内参信息，得到所述摄像头的第一内参信息。
  13. 根据权利要求12所述的方法，其特征在于，所述方法还包括：
    通过霍尔传感器获取所述当前姿态下的霍尔值；
    基于所述霍尔值确定所述当前姿态下的所述摄像头的偏移量；
    所述根据所述当前姿态下的所述摄像头的焦距值更新所述摄像头的初始内参信息，得到所述摄像头的第一内参信息，包括：
    根据所述当前姿态下的所述摄像头的焦距值和所述摄像头的偏移量更新所述摄像头的初始内参信息，得到所述摄像头的第一内参信息。
  14. 一种深度信息获取方法，包括：
    获取摄像头的当前姿态信息，所述当前姿态信息包括当前的角速度信息；
    将所述当前姿态信息转换为目标姿态信息；
    获取在当前姿态下待处理图像中各像素点的第一深度信息；
    根据所述目标姿态信息和在所述当前姿态下所述待处理图像中各像素点的第一深度信息，确定在目标姿态下所述待处理图像中各像素点的第二深度信息。
  15. 根据权利要求14所述的方法，其特征在于，所述当前姿态信息还包括当前的高度信息、当前的方位信息和当前的地理位置信息中的至少一种；所述获取摄像头的当前姿态信息，包括：
    获取摄像头当前的角速度信息，并获取所述摄像头当前的高度信息、当前的方位信息和当前的地理位置信息中的至少一种；
    将所述当前的高度信息、所述当前的方位信息和所述当前的地理位置信息中的至少一种和所述当前的角速度信息进行融合处理，得到所述摄像头的当前姿态信息。
  16. 根据权利要求15所述的方法，其特征在于，所述当前姿态信息还包括当前的加速度信息；所述方法还包括：
    获取摄像头当前的加速度信息；
    所述将所述当前的高度信息、所述当前的方位信息和所述当前的地理位置信息中的至少一种和所述当前的角速度信息进行融合处理，得到所述摄像头的当前姿态信息，包括：
    将所述摄像头当前的高度信息、所述当前的方位信息和所述当前的地理位置信息中的至少一种与所述当前的角速度信息，以及当前的加速度信息进行融合处理，得到摄像头的当前姿态信息。
  17. 一种图像处理装置，包括：
    姿态获取模块，用于获取摄像头的当前姿态信息，所述当前姿态信息包括当前的角速度信息；
    转换模块，用于将所述当前姿态信息转换为目标姿态信息；
    深度信息获取模块，用于获取在当前姿态下待处理图像中各像素点的第一深度信息；
    确定模块，用于根据所述目标姿态信息和在所述当前姿态下所述待处理图像中各像素点的第一深度信息，确定在目标姿态下所述待处理图像中各像素点的第二深度信息；
    目标图像确定模块，用于获取所述摄像头的第一内参信息，根据所述当前姿态信息、所述第一深度信息、所述目标姿态信息、所述第二深度信息和所述第一内参信息对所述待处理图像进行重投影处理，得到目标图像。
  18. 一种深度信息获取装置，包括：
    姿态获取模块，用于获取摄像头的当前姿态信息，所述当前姿态信息包括当前的角速度信息；
    转换模块，用于将所述当前姿态信息转换为目标姿态信息；
    深度信息获取模块，用于获取在当前姿态下待处理图像中各像素点的第一深度信息；
    确定模块，用于根据所述目标姿态信息和在所述当前姿态下所述待处理图像中各像素点的第一深度信息，确定在目标姿态下所述待处理图像中各像素点的第二深度信息。
  19. 一种电子设备，包括存储器及处理器，所述存储器中储存有计算机程序，所述计算机程序被所述处理器执行时，使得所述处理器执行如权利要求1至16中任一项所述的方法的步骤。
  20. 一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现如权利要求1至16中任一项所述的方法的步骤。
PCT/CN2019/102801 2019-08-27 2019-08-27 图像处理方法和装置、电子设备、计算机可读存储介质 WO2021035525A1 (zh)

Priority Applications (4)

Application Number Priority Date Filing Date Title
CN201980096840.3A CN113875221A (zh) 2019-08-27 2019-08-27 图像处理方法和装置、电子设备、计算机可读存储介质
PCT/CN2019/102801 WO2021035525A1 (zh) 2019-08-27 2019-08-27 图像处理方法和装置、电子设备、计算机可读存储介质
EP19943025.7A EP4013030A4 (en) 2019-08-27 2019-08-27 IMAGE PROCESSING METHOD AND DEVICE, AND ELECTRONIC DEVICE AND COMPUTER READABLE STORAGE MEDIUM
US17/673,249 US20220174217A1 (en) 2019-08-27 2022-02-16 Image processing method and device, electronic device, and computer-readable storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2019/102801 WO2021035525A1 (zh) 2019-08-27 2019-08-27 图像处理方法和装置、电子设备、计算机可读存储介质

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/673,249 Continuation US20220174217A1 (en) 2019-08-27 2022-02-16 Image processing method and device, electronic device, and computer-readable storage medium

Publications (1)

Publication Number Publication Date
WO2021035525A1 true WO2021035525A1 (zh) 2021-03-04

Family

ID=74684941

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/102801 WO2021035525A1 (zh) 2019-08-27 2019-08-27 图像处理方法和装置、电子设备、计算机可读存储介质

Country Status (4)

Country Link
US (1) US20220174217A1 (zh)
EP (1) EP4013030A4 (zh)
CN (1) CN113875221A (zh)
WO (1) WO2021035525A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9892564B1 (en) * 2017-03-30 2018-02-13 Novarad Corporation Augmenting real-time views of a patient with three-dimensional data

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103745474A (zh) * 2014-01-21 2014-04-23 南京理工大学 基于惯性传感器和摄像机的图像配准方法
JP2014089168A (ja) * 2012-10-30 2014-05-15 Hal-Design Laboratory Co Ltd カメラのキャリブレーション方法及びカメラのキャリブレーション装置
CN109005355A (zh) * 2018-09-28 2018-12-14 维沃移动通信有限公司 一种拍摄方法及移动终端
CN109141433A (zh) * 2018-09-20 2019-01-04 江阴市雷奥机器人技术有限公司 一种机器人室内定位系统及定位方法

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060195858A1 (en) * 2004-04-15 2006-08-31 Yusuke Takahashi Video object recognition device and recognition method, video annotation giving device and giving method, and program
JP4958610B2 (ja) * 2007-04-06 2012-06-20 キヤノン株式会社 画像防振装置、撮像装置及び画像防振方法
US8823813B2 (en) * 2011-06-06 2014-09-02 Apple Inc. Correcting rolling shutter using image stabilization
JP5977591B2 (ja) * 2012-06-20 2016-08-24 オリンパス株式会社 画像処理装置及びそれを備えた撮像装置、画像処理方法、並びに画像処理プログラムを記録したコンピュータ読み取り可能な記録媒体
JP6047008B2 (ja) * 2012-12-25 2016-12-21 キヤノン株式会社 画像処理装置およびそれを備えた撮像装置、ならびに画像処理装置の制御方法
US9734872B2 (en) * 2016-01-19 2017-08-15 Htc Corporation Method and electronic apparatus for generating time-lapse video and recording medium using the method
JP6904843B2 (ja) * 2017-08-03 2021-07-21 キヤノン株式会社 撮像装置およびその制御方法
JP7027064B2 (ja) * 2017-08-10 2022-03-01 キヤノン株式会社 画像処理装置および画像処理方法、撮像装置、プログラム
CN109544620B (zh) * 2018-10-31 2021-03-30 Oppo广东移动通信有限公司 图像处理方法和装置、计算机可读存储介质和电子设备
CN112102386A (zh) * 2019-01-22 2020-12-18 Oppo广东移动通信有限公司 图像处理方法、装置、电子设备和计算机可读存储介质
CN109714536B (zh) * 2019-01-23 2021-02-23 Oppo广东移动通信有限公司 图像校正方法、装置、电子设备及计算机可读存储介质

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2014089168A (ja) * 2012-10-30 2014-05-15 Hal-Design Laboratory Co Ltd カメラのキャリブレーション方法及びカメラのキャリブレーション装置
CN103745474A (zh) * 2014-01-21 2014-04-23 南京理工大学 基于惯性传感器和摄像机的图像配准方法
CN109141433A (zh) * 2018-09-20 2019-01-04 江阴市雷奥机器人技术有限公司 一种机器人室内定位系统及定位方法
CN109005355A (zh) * 2018-09-28 2018-12-14 维沃移动通信有限公司 一种拍摄方法及移动终端

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4013030A4 *

Also Published As

Publication number Publication date
EP4013030A4 (en) 2022-08-31
US20220174217A1 (en) 2022-06-02
EP4013030A1 (en) 2022-06-15
CN113875221A (zh) 2021-12-31

Similar Documents

Publication Publication Date Title
CN111147741B (zh) 基于对焦处理的防抖方法和装置、电子设备、存储介质
CN109194876B (zh) 图像处理方法、装置、电子设备和计算机可读存储介质
CN108769528B (zh) 图像补偿方法和装置、计算机可读存储介质和电子设备
KR102526794B1 (ko) 카메라 모듈, 고체 촬상 소자, 전자 기기 및 촬상 방법
CN111246089B (zh) 抖动补偿方法和装置、电子设备、计算机可读存储介质
Karpenko et al. Digital video stabilization and rolling shutter correction using gyroscopes
CN110610465B (zh) 图像校正方法和装置、电子设备、计算机可读存储介质
US8767036B2 (en) Panoramic imaging apparatus, imaging method, and program with warning detection
CN113875219B (zh) 图像处理方法和装置、电子设备、计算机可读存储介质
WO2017020150A1 (zh) 一种图像处理方法、装置及摄像机
JP2019510234A (ja) 奥行き情報取得方法および装置、ならびに画像取得デバイス
JP2018189730A (ja) 像ブレ補正装置及びその制御方法、プログラム、記憶媒体
CN107615744B (zh) 一种图像拍摄参数的确定方法及摄像装置
CN109598764A (zh) 摄像头标定方法和装置、电子设备、计算机可读存储介质
CN109660718A (zh) 图像处理方法和装置、电子设备、计算机可读存储介质
CN110493522A (zh) 防抖方法和装置、电子设备、计算机可读存储介质
CN112204946A (zh) 数据处理方法、装置、可移动平台及计算机可读存储介质
CN111246100A (zh) 防抖参数的标定方法、装置和电子设备
CN110177213B (zh) 陀螺仪参数调整方法和装置、终端、计算机可读存储介质
US20220174217A1 (en) Image processing method and device, electronic device, and computer-readable storage medium
CN109671028A (zh) 图像处理方法和装置、电子设备、计算机可读存储介质
US11128814B2 (en) Image processing apparatus, image capturing apparatus, video reproducing system, method and program
KR102131369B1 (ko) 천체를 촬영한 복수의 이미지를 합성하는 방법 및 장치
CN110519513B (zh) 防抖方法和装置、电子设备、计算机可读存储介质
CN110233969B (zh) 图像处理方法和装置、电子设备、计算机可读存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19943025

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2019943025

Country of ref document: EP

Effective date: 20220311