WO2020192209A1 - 一种基于Dual Camera+TOF的大光圈虚化方法 - Google Patents

一种基于Dual Camera+TOF的大光圈虚化方法 Download PDF

Info

Publication number
WO2020192209A1
WO2020192209A1 PCT/CN2019/127944 CN2019127944W WO2020192209A1 WO 2020192209 A1 WO2020192209 A1 WO 2020192209A1 CN 2019127944 W CN2019127944 W CN 2019127944W WO 2020192209 A1 WO2020192209 A1 WO 2020192209A1
Authority
WO
WIPO (PCT)
Prior art keywords
depth
data
binocular
area
error
Prior art date
Application number
PCT/CN2019/127944
Other languages
English (en)
French (fr)
Inventor
邓稀佳
刘苑文
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from CN201910330861.9A external-priority patent/CN111741283A/zh
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to CN201980096705.9A priority Critical patent/CN114365482A/zh
Priority to EP19921752.2A priority patent/EP3930321A4/en
Publication of WO2020192209A1 publication Critical patent/WO2020192209A1/zh
Priority to US17/484,093 priority patent/US12096134B2/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/95—Computational photography systems, e.g. light-field imaging systems
    • H04N23/958—Computational photography systems, e.g. light-field imaging systems for extended depth of field imaging
    • H04N23/959—Computational photography systems, e.g. light-field imaging systems for extended depth of field imaging by adjusting depth of field during image capture, e.g. maximising or setting range based on scene characteristics
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/239—Image signal generators using stereoscopic image cameras using two 2D image sensors having a relative position equal to or related to the interocular distance
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
    • G01S17/06—Systems determining position data of a target
    • G01S17/08—Systems determining position data of a target for measuring distance only
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G06T7/593—Depth or shape recovery from multiple images from stereo images
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/128—Adjusting depth or disparity
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/271—Image signal generators wherein the generated image signals comprise depth maps or disparity maps
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/45—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/63—Control of cameras or camera modules by using electronic viewfinders
    • H04N23/633—Control of cameras or camera modules by using electronic viewfinders for displaying additional information relating to control or operation of the camera
    • H04N23/635—Region indicators; Field of view indicators
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/67—Focus control based on electronic image sensor signals
    • H04N23/671—Focus control based on electronic image sensor signals in combination with active ranging signals, e.g. using light or sound signals emitted toward objects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/2621—Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G06T2207/10012—Stereo images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G06T2207/30201—Face
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N2013/0074—Stereoscopic image analysis
    • H04N2013/0081—Depth or disparity estimation from stereoscopic image signals

Definitions

  • This application relates to the field of image processing, in particular to an image processing apparatus and method.
  • the large aperture blur effect is a special photo effect, the starting point of the effect is to simulate the effect of a SLR camera.
  • the SLR camera can keep the focused object clear and the non-focused object blurred, so that the focused object is more prominent.
  • the target object and the object with the same or similar depth as the target object remain clear, and the other objects are blurred.
  • the binocular system can determine the depth of objects in the captured image.
  • the depth map determined by the binocular system has many pixels and high resolution. And the binocular system is consistent with the human senses, and the application range is wider. However, the accuracy and stability of the depth map determined by the binocular system is poor.
  • the binocular system can be used to determine the depth of the object to perform blur processing, but due to the limitation of the accuracy and stability of the depth estimation result of the binocular system, blur error may be caused.
  • This application provides an image processing method.
  • the problem that the binocular system determines the depth of the object and performs the blur processing may cause blur errors.
  • an image processing method including: acquiring binocular image data of a scene and TOF data of the time of flight of the scene, the binocular image data including first image data and first image data obtained from different cameras Two image data; determine the error area according to the binocular image data and the binocular depth data; the binocular depth data is determined according to the binocular image data; according to the TOF data, the error The depth in the region is corrected to determine the corrected binocular depth data; according to the corrected binocular depth data, the first image data is blurred, wherein the first image data is The image data to be displayed.
  • the error area can be corrected according to the TOF data, which effectively avoids the error of the depth estimation of some specific areas caused by the poor accuracy and stability of the binocular depth estimation method.
  • the advantages of TOF data accuracy and stability, and the high resolution of binocular images it realizes accurate, stable and high-resolution depth estimation, thereby reducing or even eliminating blur errors in images with large aperture blur effects.
  • the step of determining the error area based on binocular image data and binocular depth data includes: according to the binocular depth data and the first image data Or the second image data determines the area of the first image data corresponding to an area where at least one of accuracy or stability in the binocular depth data does not satisfy a preset range.
  • the error area includes a first error area;
  • the corrected binocular depth data includes: correcting the depth of the first error region of the binocular dense depth data according to the TOF data to determine the corrected binocular depth data.
  • the depth of the first error region of the binocular dense depth data is corrected according to the TOF data to determine the corrected
  • the binocular depth data includes: taking the depth of the first error region corresponding to the TOF data as the depth of the first error region; performing densification processing on the depth of the first error region; The depth of the area outside the first error area corresponding to the binocular dense depth data is used as the depth of the area outside the corresponding first error area to obtain the corrected binocular depth data.
  • the error area includes a first error area and a second error area
  • the binocular depth data includes binocular dense depth data
  • correcting the depth in the error area to determine the corrected binocular depth data includes: according to the binocular dense depth Data and the first image data to determine the first error area; according to the TOF data, correct the depth of the first error area of the binocular dense depth data to determine the first corrected dense depth Data; according to the first corrected dense depth data, the TOF data, the first image data, determine the second error area; according to the TOF data, the first corrected dense depth data
  • the depth of the second error region is corrected to determine the corrected binocular depth data.
  • the depth of the first error region of the binocular dense depth data is corrected according to the TOF data to determine the first corrected dense depth data , Including: taking the depth of the first error area corresponding to the TOF data as the depth of the first error area; performing densification processing on the depth of the first error area; densifying the binocular
  • the depth of the area outside the first error area corresponding to the depth data is used as the depth of the area outside the corresponding first error area to obtain the first corrected dense depth data.
  • the error region includes a first error region;
  • the binocular depth data includes binocular sparse depth data;
  • the binocular image data and the binocular Determining the error area according to the depth data of the binocular, including: determining a first error area according to the sparse binocular depth data, the first image data, and the second image data; the determining the first error area according to the TOF data and the double Correcting the depth in the error area to determine the corrected binocular depth data, including: according to the TOF data and the binocular sparse depth data, correcting the depth of the first error area The depth is corrected to determine the corrected binocular depth data.
  • the depth of the first error region of the binocular sparse depth data is corrected according to the TOF data to determine the corrected Binocular depth data includes: taking the depth of the first error region corresponding to the TOF data as the depth of the first error region; and setting the binocular sparse depth data outside the first error region The depth of the area is taken as the depth of the area outside the corresponding first error area; the first error area and the depth of the area outside the first error area are densified to obtain the Corrected binocular depth data.
  • the error area includes a first error area and a second error area
  • the binocular depth data includes binocular sparse depth data
  • Data and the binocular depth data to determine an error region including: correcting the depth in the error region according to the TOF data and the binocular depth data to determine the corrected binocular depth
  • the data includes: determining the first error area according to the binocular sparse depth data, the first image data, and the second image data; according to the TOF data and the binocular sparse depth data, correcting Correcting the depth of the first error area to determine first corrected dense depth data; determining the second error area according to the first corrected dense depth data, the TOF data, and the first image data;
  • the TOF data and the first corrected dense depth data are used to correct the depth of the second error region to determine the corrected binocular depth data.
  • the depth of the first error region of the binocular sparse depth data is corrected according to the TOF data to determine the first corrected dense depth data , Including: using the depth of the first error region corresponding to the TOF data as the depth of the first error region; and using the binocular sparse depth data corresponding to the depth of the region outside the first error region As the corresponding depth of the area outside the first error area; densify the depth of the first error area and the area outside the first error area to obtain the first corrected dense depth data.
  • the depth of the second error region of the first corrected dense depth data is corrected according to the TOF data to determine the corrected
  • the binocular depth data includes: using the depth of the second error area corresponding to the TOF data as the depth of the second error area; performing densification processing on the depth of the second error area; The depth of the area outside the second error area corresponding to the first corrected dense depth data is used as the depth of the area outside the corresponding second error area to obtain the corrected binocular depth data .
  • the second error region includes a depth jump region in the first modified dense depth data.
  • the first error area includes at least one of the following areas: a repeated texture area, an area with a color difference less than a first preset value, and a depth gradient area.
  • the method before correcting the depth in the error region according to the TOF data to determine the corrected binocular depth data, the method further includes: adjusting The depth of the TOF data and/or the binocular depth data is such that the systematic depth error between the TOF data and the binocular data is smaller than a second preset value.
  • the method before correcting the depth in the error region according to the TOF data to determine the corrected binocular depth data, the method further includes: adjusting The relative position of the TOF data and/or the binocular depth data is such that the systematic position error of the TOF data and the binocular data is smaller than a third preset value.
  • an image processing device including: an acquisition module for acquiring binocular image data of a scene and TOF data of the scene, the binocular image data including first image data and second image data Image data; a determination module for determining an error area based on the binocular image data and the binocular depth data; the binocular depth data is determined based on the binocular image data; a correction module for The TOF data corrects the depth in the error area to determine the corrected binocular depth data; the blurring processing module is used to correct the first binocular depth data according to the corrected binocular depth data One image data is blurred.
  • the error area includes a first error area
  • the binocular depth data includes binocular dense depth data
  • the determining module is configured to, according to the binocular dense depth data, Depth data and the first image data to determine the first error area
  • the correction module is configured to correct the depth of the first error area according to the TOF data and the binocular dense depth data To determine the corrected binocular depth data.
  • the correction module is configured to: use the depth of the first error region corresponding to the TOF data as the depth of the first error region; Performing densification processing on the depth of an error area; taking the depth of the area outside the first error area corresponding to the binocular dense depth data as the depth of the area outside the first error area, To obtain the corrected binocular depth data.
  • the error area includes a first error area and a second error area
  • the binocular depth data includes binocular dense depth data
  • the determining module is configured to: The binocular dense depth data and the first image data determine the first error area
  • the correction module is configured to correct the first error based on the TOF data and the binocular dense depth data
  • the depth of the region is corrected to determine the first corrected dense depth data
  • the determining module is further configured to determine the second corrected dense depth data according to the first corrected dense depth data, the TOF data, and the first image data Error area
  • the correction module is also used to correct the depth of the second error area according to the TOF data and the first corrected dense depth data to determine the corrected binocular depth data .
  • the correction module is configured to: use the depth of the first error region corresponding to the TOF data as the depth of the first error region; Performing densification processing on the depth of an error area; taking the depth of the area outside the first error area corresponding to the binocular dense depth data as the depth of the area outside the first error area, To obtain the first modified dense depth data.
  • the error area includes a first error area;
  • the binocular depth data includes binocular sparse depth data;
  • the determining module is configured to, according to the binocular sparse Depth data, the first image data, and the second image data to determine the first error area;
  • the correction module is configured to perform the correction on the first error region according to the TOF data and the binocular sparse depth data The depth of an error area is corrected to determine the corrected binocular depth data.
  • the correction module is configured to: use the depth of the first error region corresponding to the TOF data as the depth of the first error region; The depth of the area outside the first error area corresponding to the sparse depth data is taken as the depth of the area outside the corresponding first error area; for the first error area and the area outside the first error area The depth of the region is densified to obtain the corrected binocular depth data.
  • the error area includes a first error area and a second error area
  • the binocular depth data includes binocular sparse depth data
  • the determining module is configured to: The binocular sparse depth data, the first image data, and the second image data determine the first error region
  • the correction module is configured to, according to the TOF data and the binocular sparse depth data , Correcting the depth of the first error region to determine first corrected dense depth data
  • the determining module is further configured to, according to the first corrected dense depth data, the TOF data, and the first image data , Determine the second error area
  • the correction module is further configured to correct the depth of the second error area according to the TOF data and the first corrected dense depth data to determine the corrected After the binocular depth data.
  • the correction module is configured to: use the depth of the first error region corresponding to the TOF data as the depth of the first error region; The depth of the area outside the first error area corresponding to the sparse depth data is taken as the depth of the area outside the corresponding first error area; for the first error area and the area outside the first error area Densification processing is performed on the depth of the region to obtain the first modified density depth data.
  • the correction module is configured to: use the depth of the second error region corresponding to the TOF data as the depth of the second error region; Perform densification processing on the depth of the second error region; take the depth of the region outside the second error region corresponding to the first corrected dense depth data as the depth of the region outside the second error region To obtain the corrected binocular depth data.
  • the second error region includes a depth jump region in the first modified dense depth data.
  • the first error area includes at least one of the following areas: a repeated texture area, an area with a color difference less than a first preset value, or a depth gradient area.
  • the image processing device further includes a first adjustment module for correcting the depth in the error region according to the TOF data to determine Before the corrected binocular depth data, the depth of the TOF data and/or the binocular depth data is adjusted so that the systematic depth error between the TOF data and the binocular data is smaller than a second preset value.
  • the image processing device further includes a second adjustment module, configured to correct the depth in the error region according to the TOF data to determine Before the corrected binocular depth data, adjust the relative position of the TOF data and/or the binocular depth data so that the system position error between the TOF data and the binocular data is less than a third preset value .
  • an image processing method including: acquiring binocular image data of a scene and TOF data of the time of flight of the scene, the binocular image data including first image data and second image data obtained from different cameras 2.
  • Image data correcting the binocular depth data according to the TOF data to obtain corrected binocular depth data, the binocular depth data being determined according to the binocular image data; according to the corrected binocular depth data
  • the binocular depth data of, the first image data is blurred, and the first image data is the image data to be displayed.
  • the error area can be corrected according to the TOF data, which effectively avoids the error of the depth estimation of some specific areas caused by the poor accuracy and stability of the binocular depth estimation method.
  • the advantages of TOF data accuracy and stability, and the high resolution of binocular images it realizes accurate, stable and high-resolution depth estimation, thereby reducing or even eliminating blur errors in images with large aperture blur effects.
  • the correcting the binocular depth data according to the TOF data to obtain the corrected binocular depth data includes: according to the binocular image data and the binocular The depth data is used to determine the error area; according to the TOF data, the depth in the error area is corrected to determine the corrected binocular depth data.
  • the step of determining the error region based on binocular image data and binocular depth data includes: according to the binocular depth data and the first image data Or the second image data determines the area of the first image data corresponding to an area where at least one of accuracy or stability in the binocular depth data does not satisfy a preset range.
  • the error area includes a first error area;
  • the area of the range corresponds to the area of the first image data;
  • the correcting the depth in the error area according to the TOF data to determine the corrected binocular depth data includes: according to the TOF Data to correct the depth of the first error region of the binocular dense depth data to determine the corrected binocular depth data.
  • the depth of the first error region of the binocular dense depth data is corrected according to the TOF data to determine the corrected binocular
  • the depth data includes: taking the depth of the first error area corresponding to the TOF data as the depth of the first error area; performing densification processing on the depth of the first error area; The depth of the area outside the first error area corresponding to the dense depth data is used as the depth of the area outside the corresponding first error area to obtain the corrected binocular depth data.
  • the error area includes a first error area and a second error area; the second error area includes a partial area outside the first error area;
  • the binocular depth data includes binocular dense depth data; said determining the accuracy or stability of the binocular depth data according to the binocular depth data and the first image data or the second image data At least one area of the first image data corresponding to an area that does not meet the preset range includes: determining the first error area according to the binocular dense depth data and the first image data, and the first error area
  • An error area is an area of the first image data corresponding to an area where at least one of accuracy or stability in the binocular depth data does not meet the preset range; according to the TOF data, the error area is Correcting the depth of the binocular to determine the corrected binocular depth data includes: correcting the depth of the first error region of the binocular dense depth data according to the TOF data to determine the first corrected dense Depth data; determining the second error area according to the first corrected
  • the depth of the first error region of the binocular dense depth data is corrected according to the TOF data to determine the first corrected dense depth data , Including: taking the depth of the first error area corresponding to the TOF data as the depth of the first error area; performing densification processing on the depth of the first error area; densifying the binocular The depth of the area outside the first error area corresponding to the depth data is used as the depth of the area outside the corresponding first error area to obtain the first corrected dense depth data.
  • the error area includes a first error area;
  • the depth of the first error region of the binocular sparse depth data is corrected according to the TOF data to determine the corrected Binocular depth data includes: taking the depth of the first error region corresponding to the TOF data as the depth of the first error region; and setting the binocular sparse depth data outside the first error region The depth of the area is taken as the depth of the area outside the corresponding first error area; the first error area and the depth of the area outside the first error area are densified to obtain the Corrected binocular depth data.
  • the error area includes a first error area and a second error area; the second error area includes a partial area outside the first error area;
  • the binocular depth data includes binocular sparse depth data; said determining the accuracy or stability of the binocular depth data according to the binocular depth data and the first image data or the second image data At least one area of the first image data corresponding to an area that does not meet the preset range includes: determining the first image data according to the binocular sparse depth data, the first image data, and the second image data An error area, where the first error area is an area of the first image data corresponding to an area in which at least one of accuracy or stability in the binocular depth data does not satisfy a preset range; the according to TOF data , Correcting the depth in the error region to determine the corrected binocular depth data, including: correcting the depth of the first error region of the binocular sparse depth data according to the TOF data , To determine the first corrected dense depth data
  • the depth of the first error region of the binocular sparse depth data is corrected according to the TOF data to determine the first corrected dense depth data , Including: using the depth of the first error region corresponding to the TOF data as the depth of the first error region; and using the binocular sparse depth data corresponding to the depth of the region outside the first error region As the corresponding depth of the area outside the first error area; densify the depth of the first error area and the area outside the first error area to obtain the first corrected dense depth data.
  • the depth of the second error region of the first corrected dense depth data is corrected according to the TOF data to determine the corrected
  • the binocular depth data includes: using the depth of the second error area corresponding to the TOF data as the depth of the second error area; performing densification processing on the depth of the second error area; The depth of the area outside the second error area corresponding to the first corrected dense depth data is used as the depth of the area outside the corresponding second error area to obtain the corrected binocular depth data .
  • the second error region includes a depth jump region in the first modified dense depth data.
  • the first error area includes at least one of the following areas: a repeated texture area, an area with a color difference less than a first preset value, or a depth gradient area.
  • the method before correcting the depth in the error region according to the TOF data to determine the corrected binocular depth data, the method further includes: adjusting The depth of the TOF data and/or the binocular depth data is such that the systematic depth error between the TOF data and the binocular data is smaller than a second preset value.
  • the method before correcting the depth in the error region according to the TOF data to determine the corrected binocular depth data, the method further includes: adjusting The relative position of the TOF data and/or the binocular depth data is such that the systematic position error of the TOF data and the binocular data is smaller than a third preset value.
  • an image processing device including: an acquisition module for acquiring binocular image data of a scene, and TOF data of the time of flight of the scene, the binocular image data including the first image data obtained from different cameras Image data and second image data; a correction module for correcting the binocular depth data according to the TOF data to obtain corrected binocular depth data, the binocular depth data being based on the binocular image data Determined; a blurring processing module, configured to perform blurring processing on the first image data according to the corrected binocular depth data, wherein the first image data is image data to be displayed.
  • the correction module includes a determining unit, configured to determine an error area according to the binocular image data and binocular depth data; and a correction unit, configured to determine an error region according to the TOF Data to correct the depth in the error region to determine the corrected binocular depth data.
  • the determining unit is specifically configured to: determine the binocular depth data according to the binocular depth data and the first image data or the second image data An area of the first image data corresponding to an area where at least one of accuracy or stability in the depth data does not satisfy the preset range.
  • the error area includes a first error area
  • the binocular depth data includes binocular dense depth data
  • the determining unit is specifically configured to, according to the binocular The dense depth data and the first image data determine the first error area
  • the first error area corresponds to an area in which at least one of accuracy or stability in the binocular depth data does not meet a preset range Region of the first image data
  • the correction unit is configured to correct the depth of the first error region of the binocular dense depth data according to the TOF data to determine the corrected Binocular depth data.
  • the correction unit is configured to: use the depth of the first error region corresponding to the TOF data as the depth of the first error region; Performing densification processing on the depth of an error area; taking the depth of the area outside the first error area corresponding to the binocular dense depth data as the depth of the area outside the first error area, To obtain the corrected binocular depth data.
  • the error area includes a first error area and a second error area; the second error area includes a partial area outside the first error area;
  • the binocular depth data includes binocular dense depth data;
  • the determining unit is configured to determine the first error area based on the binocular dense depth data and the first image data, where the first error area is The area of the first image data corresponding to the area where at least one of accuracy or stability in the binocular depth data does not meet the preset range;
  • the correction unit is configured to, according to the TOF data, perform the correction of the double Correcting the depth of the first error region of the target dense depth data to determine the first corrected dense depth data;
  • the determining unit is further configured to, according to the first corrected dense depth data, the TOF data, and the First image data, determining the second error area, the second error area being the first image corresponding to an area where at least one of accuracy or stability in the binocular depth data does not meet a preset range Data area;
  • the correction unit is further configured to,
  • the correction unit is configured to: use the depth of the first error region corresponding to the TOF data as the depth of the first error region; Performing densification processing on the depth of an error area; taking the depth of the area outside the first error area corresponding to the binocular dense depth data as the depth of the area outside the first error area, To obtain the first modified dense depth data.
  • the error area includes a first error area
  • the binocular depth data includes binocular sparse depth data
  • the determining unit is configured to, according to the binocular sparse Depth data, the first image data, and the second image data, determining the first error area, the first error area being that at least one of accuracy or stability in the binocular depth data is not satisfied
  • the area of the preset range corresponds to the area of the first image data
  • the correction unit is configured to correct the depth of the first error area of the binocular sparse depth data according to the TOF data to Determine the corrected binocular depth data.
  • the correction unit is configured to: use the depth of the first error region corresponding to the TOF data as the depth of the first error region; The depth of the area outside the first error area corresponding to the sparse depth data is taken as the depth of the area outside the corresponding first error area; for the first error area and the area outside the first error area The depth of the region is densified to obtain the corrected binocular depth data.
  • the error area includes a first error area and a second error area; the second error area includes a partial area outside the first error area;
  • the binocular depth data includes binocular sparse depth data;
  • the determining unit is configured to determine the first error region according to the binocular sparse depth data, the first image data, and the second image data, so
  • the first error area is an area of the first image data corresponding to an area where at least one of accuracy or stability in the binocular depth data does not meet the preset range;
  • the correction unit is configured to, according to the TOF data is used for correcting the depth of the first error region of the binocular sparse depth data to determine first corrected dense depth data;
  • the determining unit is also used for, according to the first corrected dense depth data,
  • the TOF data and the first image data determine the second error area, and the second error area corresponds to an area where at least one of accuracy or stability in the binocular depth data does not meet a preset range
  • the correction unit is configured to: use the depth of the first error region corresponding to the TOF data as the depth of the first error region; The depth of the area outside the first error area corresponding to the sparse depth data is taken as the depth of the area outside the corresponding first error area; for the first error area and the area outside the first error area Densification processing is performed on the depth of the region to obtain the first modified density depth data.
  • the correction unit is configured to: use the depth of the second error region corresponding to the TOF data as the depth of the second error region; Perform densification processing on the depth of the second error region; take the depth of the region outside the second error region corresponding to the first corrected dense depth data as the depth of the region outside the second error region To obtain the corrected binocular depth data.
  • the second error region includes a depth jump region in the first modified dense depth data.
  • the first error area includes at least one of the following areas: a repeated texture area, an area with a color difference less than a first preset value, or a depth gradient area.
  • a first adjustment module configured to correct the depth in the error region according to the TOF data to determine the corrected depth Before the binocular depth data, the depth of the TOF data and/or the binocular depth data is adjusted so that the systematic depth error between the TOF data and the binocular data is smaller than a second preset value.
  • a second adjustment module configured to correct the depth in the error region according to the TOF data to determine the corrected depth Before the binocular depth data, the relative position of the TOF data and/or the binocular depth data is adjusted so that the systematic position error of the TOF data and the binocular data is smaller than a third preset value.
  • an image processing device including: a memory, configured to store code; a processor, configured to read the code in the memory to execute the above-mentioned first aspect or the above-mentioned third aspect and possible The method in the embodiment.
  • a computer program storage medium has program instructions, and when the program instructions are executed, the above-mentioned first aspect or the above-mentioned third aspect and possible implementations thereof The method is executed.
  • a terminal device in a seventh aspect, the chip includes: a binocular system for collecting binocular image data; a time-of-flight TOF device for collecting TOF data; at least one processor, when the program instructions are at least When executed in a processor, the method in the foregoing first aspect or the foregoing third aspect and possible implementation manners thereof are executed.
  • a terminal device including the device in the foregoing second aspect or the foregoing fourth aspect and possible implementation manners thereof.
  • Figure 1 is a schematic structural diagram of a terminal device.
  • Figure 2 is a block diagram of the software structure of a terminal device.
  • Fig. 3 is a schematic structural diagram of another terminal device.
  • Figure 4 is a schematic flow chart of depth calculation based on a binocular system.
  • Figure 5 is a schematic diagram of the process and image of a depth-based image blurring method.
  • Fig. 6 is a schematic diagram of an example of an image screen captured by the main camera.
  • Fig. 7 is a schematic diagram of another example of an image screen captured by the main camera.
  • Fig. 8 is a schematic flowchart of a method for depth calculation of a binocular system.
  • FIG. 9 is a schematic flowchart of an image processing method provided by an embodiment of the present application.
  • FIG. 10 is a schematic flowchart of an image processing method according to another embodiment of the present application.
  • FIG. 11 is a schematic flowchart of an image processing method according to another embodiment of the present application.
  • FIG. 12 is a schematic flowchart of an image processing method according to another embodiment of the present application.
  • FIG. 13 is a schematic flowchart of a deep fusion method proposed by an embodiment of the present application.
  • Figure 14 is a schematic diagram of an optical flow neural network (flownet) operation.
  • Figure 15 is a schematic structural diagram of an encoding-decoding network.
  • FIG. 16 is a schematic flowchart of an image processing method provided by another embodiment of the present application.
  • FIG. 17 is a schematic flowchart of an image processing method proposed by another embodiment of the present application.
  • FIG. 18 is a schematic flowchart of an image processing method proposed by another embodiment of the present application.
  • FIG. 19 is a schematic flowchart of an image processing method provided by another embodiment of the present application.
  • FIG. 20 is a schematic structural diagram of an image processing apparatus proposed by an embodiment of the present application.
  • FIG. 21 is a schematic structural diagram of an image processing apparatus proposed by an embodiment of the present application.
  • FIG. 22 is a schematic structural diagram of a terminal device proposed by an embodiment of the present application.
  • Large aperture blur is a characteristic of the image acquisition unit.
  • the large-aperture blur effect is a special photo effect. After the user selects the target object, the target object and the object with the same attribute (such as the same depth level, etc.) as the target object remain clear, and the other objects are blurred. The starting point of this effect is to simulate the effect of a SLR camera to keep the focused object clear and the non-focused object blurred.
  • the binocular system includes two image acquisition units, and the two image acquisition units will collect images at the same time during framing.
  • FIG. 1 is a schematic diagram of the structure of an electronic device 100.
  • the electronic device 110 may be a terminal device.
  • the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, and an antenna 2.
  • Mobile communication module 150 wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone jack 170D, sensor module 180, buttons 190, motor 191, indicator 192, camera 193, display screen 194, and Subscriber identification module (subscriber identification module, SIM) card interface 195, etc.
  • SIM Subscriber identification module
  • the sensor module 180 may include pressure sensor 180A, gyroscope sensor 180B, air pressure sensor 180C, magnetic sensor 180D, acceleration sensor 180E, distance sensor 180F, proximity light sensor 180G, fingerprint sensor 180H, temperature sensor 180J, touch sensor 180K, ambient light Sensor 180L, bone conduction sensor 180M, time of flight (time of flight, TOF) sensor 180N, etc.
  • the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100.
  • the electronic device 100 may include more or fewer components than shown, or combine certain components, or split certain components, or arrange different components.
  • the illustrated components can be implemented in hardware, software, or a combination of software and hardware.
  • the processor 110 may include one or more processing units.
  • the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), and an image signal processor. (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural-network processing unit (NPU), etc.
  • AP application processor
  • modem processor modem processor
  • GPU graphics processing unit
  • image signal processor image signal processor
  • ISP image signal processor
  • controller video codec
  • digital signal processor digital signal processor
  • DSP digital signal processor
  • NPU neural-network processing unit
  • the different processing units may be independent devices or integrated in one or more processors.
  • the controller can generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.
  • a memory may also be provided in the processor 110 to store instructions and data.
  • the memory in the processor 110 is a cache memory.
  • the memory can store instructions or data that have just been used or recycled by the processor 110. If the processor 110 needs to use the instruction or data again, it can be directly called from the memory. Repeated accesses are avoided, the waiting time of the processor 110 is reduced, and the efficiency of the system is improved.
  • the processor 110 may include one or more interfaces.
  • the interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuit sound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, and a universal asynchronous transmitter (universal asynchronous transmitter) interface.
  • I2C integrated circuit
  • I2S integrated circuit built-in audio
  • PCM pulse code modulation
  • PCM pulse code modulation
  • UART universal asynchronous transmitter
  • MIPI mobile industry processor interface
  • GPIO general-purpose input/output
  • SIM subscriber identity module
  • USB Universal Serial Bus
  • the I2C interface is a two-way synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL).
  • the processor 110 may include multiple sets of I2C buses.
  • the processor 110 may be coupled to the touch sensor 180K, charger, flash, camera 193, etc. through different I2C bus interfaces.
  • the processor 110 may couple the touch sensor 180K through an I2C interface, so that the processor 110 and the touch sensor 180K communicate through an I2C bus interface to implement the touch function of the electronic device 100.
  • the I2S interface can be used for audio communication.
  • the processor 110 may include multiple sets of I2S buses.
  • the processor 110 may be coupled with the audio module 170 through an I2S bus to realize communication between the processor 110 and the audio module 170.
  • the audio module 170 may transmit audio signals to the wireless communication module 160 through an I2S interface, so as to realize the function of answering calls through a Bluetooth headset.
  • the PCM interface can also be used for audio communication to sample, quantize and encode analog signals.
  • the audio module 170 and the wireless communication module 160 may be coupled through a PCM bus interface.
  • the audio module 170 may also transmit audio signals to the wireless communication module 160 through the PCM interface, so as to realize the function of answering calls through the Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.
  • the UART interface is a universal serial data bus used for asynchronous communication.
  • the bus can be a two-way communication bus. It converts the data to be transmitted between serial communication and parallel communication.
  • the UART interface is generally used to connect the processor 110 and the wireless communication module 160.
  • the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function.
  • the audio module 170 may transmit audio signals to the wireless communication module 160 through a UART interface, so as to realize the function of playing music through a Bluetooth headset.
  • the MIPI interface can be used to connect the processor 110 with the display screen 194, the camera 193 and other peripheral devices.
  • the MIPI interface includes camera serial interface (camera serial interface, CSI), display serial interface (display serial interface, DSI), etc.
  • the processor 110 and the camera 193 communicate through a CSI interface to implement the shooting function of the electronic device 100.
  • the processor 110 and the display screen 194 communicate through a DSI interface to realize the display function of the electronic device 100.
  • the GPIO interface can be configured through software.
  • the GPIO interface can be configured as a control signal or as a data signal.
  • the GPIO interface can be used to connect the processor 110 with the camera 193, the display screen 194, the wireless communication module 160, the audio module 170, the sensor module 180, and so on.
  • GPIO interface can also be configured as I2C interface, I2S interface, UART interface, MIPI interface, etc.
  • the USB interface 130 is an interface that complies with the USB standard specification, and specifically may be a Mini USB interface, a Micro USB interface, a USB Type C interface, and so on.
  • the USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and peripheral devices. It can also be used to connect headphones and play audio through the headphones. This interface can also be used to connect other electronic devices, such as AR devices.
  • the interface connection relationship between the modules illustrated in the embodiment of the present invention is merely illustrative and does not constitute a structural limitation of the electronic device 100.
  • the electronic device 100 may also adopt different interface connection modes in the foregoing embodiments, or a combination of multiple interface connection modes.
  • the charging management module 140 is used to receive charging input from the charger.
  • the charger can be a wireless charger or a wired charger.
  • the charging management module 140 may receive the charging input of the wired charger through the USB interface 130.
  • the charging management module 140 may receive the wireless charging input through the wireless charging coil of the electronic device 100. While the charging management module 140 charges the battery 142, it can also supply power to the electronic device through the power management module 141.
  • the power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110.
  • the power management module 141 receives input from the battery 142 and/or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160.
  • the power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle times, and battery health status (leakage, impedance).
  • the power management module 141 may also be provided in the processor 110.
  • the power management module 141 and the charging management module 140 may also be provided in the same device.
  • the wireless communication function of the electronic device 100 can be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor.
  • the antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals.
  • Each antenna in the electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization.
  • antenna 1 can be multiplexed as a diversity antenna of a wireless local area network.
  • the antenna can be used in combination with a tuning switch.
  • the mobile communication module 150 can provide a wireless communication solution including 2G/3G/4G/5G and the like applied to the electronic device 100.
  • the mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc.
  • the mobile communication module 150 can receive electromagnetic waves by the antenna 1, and perform processing such as filtering, amplifying and transmitting the received electromagnetic waves to the modem processor for demodulation.
  • the mobile communication module 150 can also amplify the signal modulated by the modem processor, and convert it into electromagnetic waves for radiation via the antenna 1.
  • at least part of the functional modules of the mobile communication module 150 may be provided in the processor 110.
  • at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 may be provided in the same device.
  • the modem processor may include a modulator and a demodulator.
  • the modulator is used to modulate the low frequency baseband signal to be sent into a medium and high frequency signal.
  • the demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. Then the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing.
  • the low-frequency baseband signal is processed by the baseband processor and then passed to the application processor.
  • the application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.), or displays an image or video through the display screen 194.
  • the modem processor may be an independent device.
  • the modem processor may be independent of the processor 110 and be provided in the same device as the mobile communication module 150 or other functional modules.
  • the wireless communication module 160 can provide applications on the electronic device 100 including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), and global navigation satellites.
  • WLAN wireless local area networks
  • BT wireless fidelity
  • GNSS global navigation satellite system
  • FM frequency modulation
  • NFC near field communication technology
  • infrared technology infrared, IR
  • the wireless communication module 160 may be one or more devices integrating at least one communication processing module.
  • the wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110.
  • the wireless communication module 160 can also receive the signal to be sent from the processor 110, perform frequency modulation, amplify it, and convert it into electromagnetic wave radiation via the antenna 2.
  • the antenna 1 of the electronic device 100 is coupled with the mobile communication module 150, and the antenna 2 is coupled with the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology.
  • the wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), broadband Code division multiple access (wideband code division multiple access, WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC , FM, and/or IR technology, etc.
  • the GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), quasi-zenith satellite system (quasi -zenith satellite system, QZSS) and/or satellite-based augmentation systems (SBAS).
  • GPS global positioning system
  • GLONASS global navigation satellite system
  • BDS Beidou navigation satellite system
  • QZSS quasi-zenith satellite system
  • SBAS satellite-based augmentation systems
  • the electronic device 100 implements a display function through a GPU, a display screen 194, and an application processor.
  • the GPU is a microprocessor for image processing, connected to the display 194 and the application processor.
  • the GPU is used to perform mathematical and geometric calculations for graphics rendering.
  • the processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.
  • the display screen 194 is used to display images, videos, etc.
  • the display screen 194 includes a display panel.
  • the display panel can adopt liquid crystal display (LCD), organic light-emitting diode (OLED), active-matrix organic light-emitting diode or active-matrix organic light-emitting diode (active-matrix organic light-emitting diode).
  • LCD liquid crystal display
  • OLED organic light-emitting diode
  • active-matrix organic light-emitting diode active-matrix organic light-emitting diode
  • AMOLED flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diode (QLED), etc.
  • the electronic device 100 may include one or N display screens 194, and N is a positive integer greater than one.
  • the electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.
  • the ISP is used to process the data fed back from the camera 193. For example, when taking a picture, the shutter is opened, the light is transmitted to the photosensitive element of the camera through the lens, the light signal is converted into an electrical signal, and the photosensitive element of the camera transfers the electrical signal to the ISP for processing and is converted into an image visible to the naked eye.
  • ISP can also optimize the image noise, brightness, and skin color. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene.
  • the ISP may be provided in the camera 193.
  • the camera 193 is used to capture still images or videos.
  • the object generates an optical image through the lens and projects it to the photosensitive element.
  • the photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor.
  • CMOS complementary metal-oxide-semiconductor
  • the photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal.
  • ISP outputs digital image signals to DSP for processing.
  • DSP converts digital image signals into standard RGB, YUV and other formats.
  • the electronic device 100 may include 1 or N cameras 193, and N is a positive integer greater than 1.
  • the electronic device 100 may include a binocular system.
  • the binocular system can include two cameras. Both cameras in the binocular system can be used to collect image data. In other words, both cameras in the binocular system can be used to capture still images or videos.
  • Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the electronic device 100 selects the frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point.
  • Video codecs are used to compress or decompress digital video.
  • the electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos in a variety of encoding formats, such as: moving picture experts group (MPEG) 1, MPEG2, MPEG3, MPEG4, and so on.
  • MPEG moving picture experts group
  • NPU is a neural-network (NN) computing processor.
  • NN neural-network
  • the NPU can realize applications such as intelligent cognition of the electronic device 100, such as image recognition, face recognition, voice recognition, text understanding, and so on.
  • the external memory interface 120 may be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100.
  • the external memory card communicates with the processor 110 through the external memory interface 120 to realize the data storage function. For example, save music, video and other files in an external memory card.
  • the internal memory 121 may be used to store computer executable program code, where the executable program code includes instructions.
  • the internal memory 121 may include a storage program area and a storage data area.
  • the storage program area can store an operating system, at least one application program (such as a sound playback function, an image playback function, etc.) required by at least one function.
  • the data storage area can store data (such as audio data, phone book, etc.) created during the use of the electronic device 100.
  • the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), etc.
  • the processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121 and/or instructions stored in a memory provided in the processor.
  • the electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor. For example, music playback, recording, etc.
  • the audio module 170 is used to convert digital audio information into an analog audio signal for output, and is also used to convert an analog audio input into a digital audio signal.
  • the audio module 170 can also be used to encode and decode audio signals.
  • the audio module 170 may be provided in the processor 110, or part of the functional modules of the audio module 170 may be provided in the processor 110.
  • the speaker 170A also called a “speaker” is used to convert audio electrical signals into sound signals.
  • the electronic device 100 can listen to music through the speaker 170A, or listen to a hands-free call.
  • the receiver 170B also called “earpiece” is used to convert audio electrical signals into sound signals.
  • the electronic device 100 answers a call or voice message, it can receive the voice by bringing the receiver 170B close to the human ear.
  • the microphone 170C also called “microphone”, “microphone”, is used to convert sound signals into electrical signals.
  • the user can approach the microphone 170C through the mouth to make a sound, and input the sound signal to the microphone 170C.
  • the electronic device 100 may be provided with at least one microphone 170C. In other embodiments, the electronic device 100 may be provided with two microphones 170C, which can implement noise reduction functions in addition to collecting sound signals. In some other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify sound sources, and realize directional recording functions.
  • the earphone interface 170D is used to connect wired earphones.
  • the earphone interface 170D may be a USB interface 130, or a 3.5mm open mobile terminal platform (OMTP) standard interface, and a cellular telecommunications industry association (cellular telecommunications industry association of the USA, CTIA) standard interface.
  • OMTP open mobile terminal platform
  • CTIA cellular telecommunications industry association of the USA, CTIA
  • the pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal.
  • the pressure sensor 180A may be provided on the display screen 194. Pressure sensor 180A
  • the capacitive pressure sensor may include at least two parallel plates with conductive material.
  • the electronic device 100 determines the intensity of the pressure according to the change in capacitance.
  • the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180A.
  • the electronic device 100 may also calculate the touched position according to the detection signal of the pressure sensor 180A.
  • touch operations that act on the same touch location but have different touch operation strengths may correspond to different operation instructions.
  • the gyro sensor 180B may be used to determine the movement posture of the electronic device 100.
  • the angular velocity of the electronic device 100 around three axes ie, x, y, and z axes
  • the gyro sensor 180B can be used for image stabilization.
  • the gyro sensor 180B detects the shake angle of the electronic device 100, calculates the distance that the lens module needs to compensate according to the angle, and allows the lens to counteract the shake of the electronic device 100 through reverse movement to achieve anti-shake.
  • the gyro sensor 180B can also be used for navigation and somatosensory game scenes.
  • the air pressure sensor 180C is used to measure air pressure.
  • the electronic device 100 calculates the altitude based on the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.
  • the magnetic sensor 180D includes a Hall sensor.
  • the electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip holster.
  • the electronic device 100 can detect the opening and closing of the flip according to the magnetic sensor 180D.
  • features such as automatic unlocking of the flip cover are set.
  • the acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of electronic devices, and used in applications such as horizontal and vertical screen switching, pedometers, etc.
  • the electronic device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.
  • the proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector such as a photodiode.
  • the light emitting diode may be an infrared light emitting diode.
  • the electronic device 100 emits infrared light to the outside through the light emitting diode.
  • the electronic device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100.
  • the electronic device 100 can use the proximity light sensor 180G to detect that the user holds the electronic device 100 close to the ear to talk, so as to automatically turn off the screen to save power.
  • the proximity light sensor 180G can also be used in leather case mode, and the pocket mode will automatically unlock and lock the screen.
  • the ambient light sensor 180L is used to sense the brightness of the ambient light.
  • the electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived brightness of the ambient light.
  • the ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures.
  • the ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in the pocket to prevent accidental touch.
  • the fingerprint sensor 180H is used to collect fingerprints.
  • the electronic device 100 can use the collected fingerprint characteristics to realize fingerprint unlocking, access application locks, fingerprint photographs, fingerprint answering calls, etc.
  • the temperature sensor 180J is used to detect temperature.
  • the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold value, the electronic device 100 executes to reduce the performance of the processor located near the temperature sensor 180J, so as to reduce power consumption and implement thermal protection.
  • the electronic device 100 when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 due to low temperature.
  • the electronic device 100 boosts the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.
  • Touch sensor 180K also called “touch device”.
  • the touch sensor 180K may be disposed on the display screen 194, and the touch screen is composed of the touch sensor 180K and the display screen 194, which is also called a “touch screen”.
  • the touch sensor 180K is used to detect touch operations acting on or near it.
  • the touch sensor can pass the detected touch operation to the application processor to determine the type of touch event.
  • the visual output related to the touch operation can be provided through the display screen 194.
  • the touch sensor 180K may also be disposed on the surface of the electronic device 100, which is different from the position of the display screen 194.
  • the bone conduction sensor 180M can acquire vibration signals.
  • the bone conduction sensor 180M can obtain the vibration signal of the vibrating bone mass of the human voice.
  • the bone conduction sensor 180M can also contact the human pulse and receive the blood pressure pulse signal.
  • the bone conduction sensor 180M may also be provided in the earphone, combined with the bone conduction earphone.
  • the audio module 170 can parse the voice signal based on the vibration signal of the vibrating bone block of the voice obtained by the bone conduction sensor 180M, and realize the voice function.
  • the application processor may analyze the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M, and realize the heart rate detection function.
  • the TOF sensor 180N may include a transmitting module and a receiving module.
  • the transmitting module of the TOF sensor 180N can send light pulses, and the receiving module of the TOF sensor 180N can receive light reflected by objects.
  • the distance between the object and the TOF sensor 180N can be obtained by detecting the flight (round trip) time of the light pulse.
  • the light signal emitted by the transmitting module of the TOF sensor 180N is reflected after encountering an object, and the receiving module of the TOF sensor 180N can receive the light signal reflected by the object.
  • the distance of the photographed scene can be calculated to generate depth information.
  • the button 190 includes a power button, a volume button, and so on.
  • the button 190 may be a mechanical button. It can also be a touch button.
  • the electronic device 100 may receive key input, and generate key signal input related to user settings and function control of the electronic device 100.
  • the motor 191 can generate vibration prompts.
  • the motor 191 can be used for incoming call vibration notification, and can also be used for touch vibration feedback.
  • touch operations applied to different applications can correspond to different vibration feedback effects.
  • Acting on touch operations in different areas of the display screen 194, the motor 191 can also correspond to different vibration feedback effects.
  • Different application scenarios for example: time reminding, receiving information, alarm clock, games, etc.
  • the touch vibration feedback effect can also support customization.
  • the indicator 192 may be an indicator light, which may be used to indicate the charging status, power change, or to indicate messages, missed calls, notifications, and so on.
  • the SIM card interface 195 is used to connect to the SIM card.
  • the SIM card can be inserted into the SIM card interface 195 or pulled out from the SIM card interface 195 to achieve contact and separation with the electronic device 100.
  • the electronic device 100 may support 1 or N SIM card interfaces, and N is a positive integer greater than 1.
  • the SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc.
  • the same SIM card interface 195 can insert multiple cards at the same time. The types of the multiple cards can be the same or different.
  • the SIM card interface 195 can also be compatible with different types of SIM cards.
  • the SIM card interface 195 may also be compatible with external memory cards.
  • the electronic device 100 interacts with the network through the SIM card to implement functions such as call and data communication.
  • the electronic device 100 adopts an eSIM, that is, an embedded SIM card.
  • the eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.
  • the software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture.
  • the embodiment of the present invention takes an Android system with a layered architecture as an example to exemplify the software structure of the electronic device 100.
  • FIG. 2 is a block diagram of the software structure of the electronic device 100 according to an embodiment of the present application.
  • the layered architecture divides the software into several layers, and each layer has a clear role and division of labor. Communication between layers through software interface.
  • the Android system is divided into four layers, from top to bottom, the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.
  • the application layer can include a series of application packages.
  • the application package can include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.
  • the application framework layer provides application programming interfaces (application programming interface, API) and programming frameworks for applications in the application layer.
  • the application framework layer includes some predefined functions.
  • the application framework layer can include a window manager, a content provider, a view system, a phone manager, a resource manager, and a notification manager.
  • the window manager is used to manage window programs.
  • the window manager can obtain the size of the display, determine whether there is a status bar, lock the screen, take a screenshot, etc.
  • the content provider is used to store and retrieve data and make these data accessible to applications.
  • the data may include video, image, audio, phone calls made and received, browsing history and bookmarks, phone book, etc.
  • the view system includes visual controls, such as controls that display text and controls that display pictures.
  • the view system can be used to build applications.
  • the display interface can be composed of one or more views.
  • a display interface that includes a short message notification icon may include a view that displays text and a view that displays pictures.
  • the phone manager is used to provide the communication function of the electronic device 100. For example, the management of the call status (including connecting, hanging up, etc.).
  • the resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, etc.
  • the notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages, and it can disappear automatically after a short stay without user interaction.
  • the notification manager is used to notify the download completion, message reminder, etc.
  • the notification manager can also be a notification that appears in the status bar at the top of the system in the form of a chart or scroll bar text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, text messages are prompted in the status bar, prompt sounds, electronic devices vibrate, and indicator lights flash.
  • Android Runtime includes core libraries and virtual machines. Android runtime is responsible for the scheduling and management of the Android system.
  • the core library consists of two parts: one part is the function functions that the java language needs to call, and the other part is the core library of Android.
  • the application layer and the application framework layer run in a virtual machine.
  • the virtual machine executes the java files of the application layer and the application framework layer as binary files.
  • the virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.
  • the system library can include multiple functional modules. For example: surface manager (surface manager), media library (Media Libraries), three-dimensional graphics processing library (for example: OpenGL ES), 2D graphics engine (for example: SGL), etc.
  • the surface manager is used to manage the display subsystem and provides a combination of 2D and 3D layers for multiple applications.
  • the media library supports playback and recording of a variety of commonly used audio and video formats, as well as still image files.
  • the media library can support multiple audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.
  • the 3D graphics processing library is used to realize 3D graphics drawing, image rendering, synthesis, and layer processing.
  • the 2D graphics engine is a drawing engine for 2D drawing.
  • the kernel layer is the layer between hardware and software.
  • the kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.
  • the corresponding hardware interrupt is sent to the kernel layer.
  • the kernel layer processes touch operations into original input events (including touch coordinates, time stamps of touch operations, etc.).
  • the original input events are stored in the kernel layer.
  • the application framework layer obtains the original input event from the kernel layer, and identifies the control corresponding to the input event. Taking the touch operation as a touch click operation, and the control corresponding to the click operation is the control of the camera application icon as an example, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer.
  • the camera 193 captures still images or videos.
  • FIG. 3 is a schematic structural diagram of a terminal device 300.
  • the structure of the terminal device 300 can refer to FIG. 1.
  • the terminal device 300 may include more or fewer components than the electronic device 100.
  • the software system of the terminal device 300 may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture.
  • the software system of the terminal device 300 can refer to FIG. 2.
  • the terminal device 300 may be a binocular system 310.
  • the binocular system 310 may include two cameras 193, namely a main camera and a secondary camera.
  • the binocular system is sometimes called a dual camera system. Both cameras in the binocular system can be used to collect image data.
  • the image data collected by the two cameras can be used to achieve the same or different functions.
  • the image collected by one of the cameras can be used for the image display of the terminal.
  • This camera can be called the main camera or the main camera; the image collected by the other camera can be used to calculate the depth or realize other functions.
  • the camera can be called the secondary camera. camera.
  • the binocular system can use dual cameras to estimate the depth of the object, and then blur the full clear image based on the depth information to achieve the final large-aperture blur effect.
  • the terminal device 300 may include a TOF device 320.
  • the TOF device 320 may also be referred to as a TOF system, a TOF sensor, or the like.
  • the TOF device 320 may include a transmitting module and a receiving module.
  • the transmitting module of the TOF device 320 can send light pulses, and the receiving module of the TOF device 320 can receive light reflected by the object.
  • the distance between the object and the TOF device 320 can be obtained by detecting the flight (round trip) time of the light pulse. This distance can also be called depth.
  • the terminal device 300 may include a flash 330.
  • Flash 330 is also called electronic flash, high-speed flash.
  • the flash lamp 330 stores high-voltage electricity through a capacitor, and the pulse trigger discharges the flash tube to complete an instant flash. In dark places, the flash can make the scene brighter.
  • the terminal device 300 may also include a telephoto camera (not shown).
  • the binocular system 310 and the TOF device 320 may be arranged adjacently.
  • the main camera and the secondary camera may be arranged in parallel with the edge 340 of the terminal device 300.
  • the TOF device 320 may be located between the two cameras in the binocular system 310, or may be located in other locations around the main camera and the sub-camera in the binocular system 310.
  • the receiver and the main camera in the TOF device 320 may be arranged parallel to the edge 340 or perpendicular to the edge 340. It can be understood that, in the position relationship in the embodiments of the present application, parallel may include approximately parallel, and vertical may include approximately vertical.
  • the terminal device 300 may include a processor 110.
  • the processor 110 may be used to determine binocular dense depth data.
  • the processor 110 may include one or more processing units.
  • the processor may include a central processing unit (CPU), a neural-network processing unit (NPU), an image signal processor (ISP), and a digital signal processor (digital signal processor). processor, DSP), etc.
  • the processor can also be used to determine binocular sparse depth data.
  • the terminal device 300 may include a dedicated depth map application pipeline (DMAP) chip, and the DMAP chip may be used to determine a binocular sparse depth map.
  • the DMAP chip may include a processor for determining the binocular sparse depth map.
  • Terminal equipment can also be called user equipment.
  • the terminal equipment can communicate with one or more core networks (core networks, CN) via the access network equipment.
  • Terminal equipment may sometimes also be referred to as an access terminal, terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, wireless network device, user agent, or user device.
  • User equipment can be a cellular phone, a cordless phone, a session initiation protocol (SIP) phone, a wireless local loop (WLL) station, a personal digital assistant (PDA), a wireless communication function Handheld devices, computing devices or other devices connected to wireless modems, in-vehicle devices, wearable devices or the Internet of Things, terminal devices in the Internet of Vehicles, and any form of user equipment in the future network.
  • SIP session initiation protocol
  • WLL wireless local loop
  • PDA personal digital assistant
  • Figure 4 is a schematic flow chart of depth calculation based on a binocular system.
  • a calibration parameter (calibration parameter) is obtained.
  • the correction parameters can be internal and external parameters of the binocular system.
  • step S402 image rectification is performed on the binocular image data according to the correction parameters. Place the image in a coordinate system, and the correction can be to change the coordinate point of the image.
  • step S403 depth calculation is performed to determine binocular depth data.
  • step S404 through depth optimization, the binocular optimization depth data is determined.
  • the depth calculation result obtained in step S403 can be optimized to obtain binocular optimized depth data.
  • a large aperture effect can be achieved, that is, out-of-focus imaging.
  • step S405 may be performed.
  • step S405 automatic focus (AF) determination (automatic focus, AF) is performed according to the binocular depth data.
  • the binocular image data can be obtained by image collection by a binocular system. It may also be obtained by two image acquisitions of a certain scene by one acquisition unit at two close positions.
  • Figure 5 is a schematic diagram of the process and image of a depth-based image blurring method.
  • step S501 the main image is captured. All objects in the image are basically clear.
  • step S502 a depth map corresponding to the main image is acquired.
  • the pixel value of each point represents the depth value of the point from the main camera.
  • a large aperture special effect result map is determined.
  • the image taken by the main camera is a portrait, and the points with the same depth value as the portrait in the large aperture effect result map remain clear, and the points with the depth value of the portrait are blurred.
  • the depth calculation based on the binocular system may be wrong, resulting in a false blur effect.
  • the binocular system to calculate the depth it is mainly based on the three primary colors of red, green, and blue (red, green, blue, RGB) to match the main and auxiliary images.
  • the reliability of depth calculation is unstable and the accuracy is not high.
  • Fig. 6 shows an example of an image screen captured by the main camera.
  • the repeated texture area shown in 601 such as blinds, stickers, etc.
  • Fig. 7 shows another example of an image screen captured by the main camera.
  • the weak texture areas shown in 701 such as wall corners, doors and handles of the same color
  • the RGB features are not obvious, or there are no features at all, that is, the color difference in this area is small, which makes the main and sub-images unable to match, and may also cause depth Calculation error. Due to the depth error, it may lead to errors in the blurring result map, failing to achieve the expected blurring effect, and poor user experience.
  • this application proposes an image processing method based on a binocular system and a time of flight (TOF) system.
  • TOF time of flight
  • the binocular system can determine the dense depth map, that is, the depth map has more pixels and high resolution. And the binocular system is consistent with the human senses, and the application range is wider. However, the accuracy and stability of the depth map determined by the binocular system is poor. For example, it is easy to cause blur error in repeated texture or weak texture area. In addition, because the depth accuracy of the binocular system is low, for the collected region of interest (ROI) images (such as a depth gradient plane or curved surface such as a human face), the blur hierarchy is poor. Both the depth jump area and the depth jump area in the embodiments of the present application may refer to the portrait area.
  • ROI region of interest
  • TOF technology is a technology that uses lasers to calculate depth.
  • the depth can be calculated based on the difference between laser emission and reception time, regardless of the RGB characteristics.
  • the depth calculation accuracy based on the TOF system is high and the stability is good.
  • the TOF system can only determine the sparse depth map, and the resolution of the depth map is low.
  • the TOF system cannot collect the depth of the remote point, and there will be no data points in the depth map. Since the laser signal may be disturbed, an error transition point of the depth will appear in the depth map.
  • Fig. 8 is a schematic flowchart of a method for depth calculation of a binocular system.
  • Binocular depth calculation can be divided into four steps: image correction, sparse depth map calculation, dense depth map calculation, and post-processing.
  • the image correction uses the preset internal and external parameters of the binocular system to correct the input binocular image and obtain the image after line alignment. According to the calibration values of the internal and external parameters of the binocular system, the binocular data collected by the binocular system is corrected to obtain the corrected main image and the corrected secondary image. For example, the binocular image can be corrected.
  • the internal parameters of the binocular system can include focal length, coordinate axis tilt parameters, principal point coordinates, etc., as well as radial and tangential distortion parameters.
  • the external parameters of the binocular system can include rotation parameters, translation parameters, and so on.
  • the ROI area can be an image area selected from the image, and the characteristics of this area can be the focus of image analysis. Some areas in the image may be irrelevant, such as sky, walls, grass, etc.
  • the ROI area may include, for example, a depth gradient area such as a human face.
  • the sparse depth map can also be determined.
  • the sparse depth map can be obtained by sparse depth map calculation.
  • the sparse depth map calculation can be performed in one or more existing or possible future methods.
  • the depth calculation engineering (DCE) module algorithm may be executed based on software, or may be performed based on a depth calculation hardware chip.
  • the depth calculation hardware chip may be, for example, a dedicated DMAP chip.
  • the sparse depth map can be obtained by calculating the sparse depth map on the corrected main and auxiliary images.
  • the sparse depth map can be a sparse depth map based on the main image, or a sparse depth map based on a secondary image.
  • the sparse depth map may also be called binocular sparse depth map or binocular sparse depth data.
  • the dense depth map can be determined according to the corrected main image, the corrected auxiliary image, the ROI area, and the sparse depth map.
  • the dense depth map can be obtained in many ways, for example, the dense depth map can be determined by a depth densification algorithm.
  • the deep densification algorithm may be, for example, an optical flow deep densification algorithm.
  • different parameters can be used to calculate the inside and outside of the ROI area.
  • the calculation parameters outside the ROI area can make the depth calculation result outside the ROI area smoother, and the calculation parameters inside the ROI area It can make the depth change in the ROI area more prominent.
  • the dense depth map may also be called binocular dense depth map or binocular dense depth data.
  • Depth map post-processing according to the main image, ROI area, dense depth map and other information, optimize the edge of the depth map, correct the internal and external depth errors of the portrait, and determine the final binocular depth map.
  • the binocular depth map can also be called binocular dense depth map or binocular dense depth data.
  • the ROI area can be a portrait area in the main image.
  • the binocular depth map determined according to the method in Fig. 8 has a high resolution and is consistent with the human senses, but the accuracy and stability are poor.
  • the embodiment of the present application combines TOF depth data to improve the method in FIG. 8.
  • TOF uses the time difference between laser emission and reception to obtain depth results.
  • a dedicated depth calculation chip or image signal processing (ISP) algorithm can be used to obtain TOF depth data.
  • ISP image signal processing
  • the TOF sparse depth map can be obtained from the system.
  • the image processing based on the binocular system and the TOF system can combine the advantages of the binocular depth estimation principle and the TOF depth estimation principle to obtain an ideal blurred image result.
  • FIG. 9 is a schematic flowchart of an image processing method provided by an embodiment of the present application.
  • step S901 binocular image data of the scene and TOF data of the time of flight of the scene are acquired, the binocular image data including first image data and second image data.
  • Obtaining binocular image data of the scene may be by collecting binocular image data through a binocular system, or by acquiring binocular image data from a storage device or an image acquisition device.
  • the TOF data can be collected through a TOF device, or the TOF data can be obtained from a storage device or a TOF data collection device.
  • the TOF data can be TOF time data or TOF depth data.
  • the signal emitted by the transmitter is reflected by the objects in the scene and then received by the receiver.
  • TOF time data may include the round trip time of the signal.
  • TOF time data is processed, and TOF depth data can include the depth of the scene.
  • TOF time data includes TOF time collected by TOF equipment.
  • TOF depth data includes TOF depth determined according to TOF time data. In TOF data, there may be no data points due to the large depth of the objects in the scene, the receiver does not receive the signal.
  • the depth corresponding to the no data point can also be determined by the no data point.
  • the first image data and the second image data are respectively image data collected by two image acquisition units (such as cameras) in the binocular system on the scene.
  • the first image data is image data collected by the main camera
  • the second image data is image data collected by the secondary camera.
  • the first image data may be image data to be displayed, such as image data displayed on a screen, a display, or a display panel.
  • the depth of the TOF data and/or the binocular depth data may be adjusted, so that the systematic depth error between the TOF data and the binocular data is smaller than a second preset value.
  • the relative position of the TOF data and/or the binocular depth data may be adjusted, so that the systematic position error of the TOF data and the binocular data is smaller than a third preset value.
  • the TOF sparse depth map and the binocular depth map can be matched by offline calibration or online calibration, that is, coordinate alignment.
  • the coordinate alignment can be row alignment and/or column alignment.
  • the difference matching parameters of the two are obtained through offline calibration, so as to match the TOF depth value to the binocular depth map.
  • the relative position of the TOF data and/or the binocular depth data may be adjusted so that the system coordinate error of the TOF data and the binocular depth data is smaller than a second preset value.
  • the TOF data and/or binocular depth data can be adjusted to reduce or even eliminate the depth error caused by the system between the TOF data and the binocular depth data.
  • the depth of the TOF data and/or the binocular depth data may be adjusted so that the systematic depth error between the TOF data and the binocular depth data is smaller than a third preset value.
  • the system depth error is caused by the difference in the calculation principle of TOF data and binocular depth data.
  • the TOF sparse depth map and the binocular depth map are matched.
  • the embodiment of the present application does not limit the sequence of position and depth adjustment.
  • the relative position of the TOF data and/or the binocular depth data may be adjusted first, and then the depth of the TOF data and/or the binocular depth data may be adjusted. It can also be done in the reverse order or simultaneously.
  • the above method includes: correcting the binocular depth data according to the TOF data to obtain corrected binocular depth data, the binocular depth data being determined according to the binocular image data .
  • an error area is determined according to the binocular image data and binocular depth data.
  • the error area may be, for example, an area where accuracy and/or stability do not meet requirements.
  • the error area is an area in the depth data determined by binocular depth estimation that does not meet the requirements for accuracy and/or stability. Error areas can also be called error-prone areas.
  • the error area may be an error-prone area in the depth map calculated by binocular data.
  • the error area can be determined based on the first image data and the binocular depth data.
  • the error area can also be determined based on the second image data and binocular depth data.
  • the result of image blur based on the binocular system may be incorrect in areas such as repeated textures and weak textures. That is, in areas such as repeated textures and weak textures, the stability of the binocular depth data may not meet the requirements.
  • the weak texture area can also be referred to as an area where the color difference is not obvious, that is, the color difference of the area is less than a preset value.
  • the accuracy requirements for the depth data can be different in different areas of the same image data.
  • the depth of the background area in the image may have low accuracy requirements; while the ROI areas such as portraits may have high accuracy requirements. Therefore, for the depth gradient area, the accuracy of the binocular depth data determined based on the binocular data may not meet the requirements.
  • the depth gradient area may be, for example, an area where the depth change amount is less than or equal to a certain preset value.
  • the binocular depth data is determined based on the binocular image data.
  • the binocular depth data may be binocular dense depth data or binocular sparse depth data.
  • the binocular depth data may be a depth map determined according to the binocular image data.
  • the binocular dense depth data may be performed based on the software execution of the DCE module algorithm, or may be performed based on a depth calculation hardware chip.
  • the depth calculation hardware chip may be a dedicated DMAP chip, for example.
  • the binocular dense depth data there are many effective information and high pixel accuracy.
  • the binocular dense depth data can be performed based on a software-executed depth densification algorithm, and the depth densification algorithm may be, for example, an optical flow densification algorithm.
  • features in the binocular image data can be extracted.
  • the extracted features can include, for example, people, fingers, cars, car windows, white walls, repeated curtain grids, sky and so on.
  • the error area may be an area corresponding to one or more of all the extracted features.
  • step S903 the depth in the error area is corrected according to the TOF data to determine the corrected binocular depth data of the scene.
  • Correcting the depth in the error area may include: for the error area, determining the corrected binocular depth data of the error area according to the TOF depth data of the error area.
  • steps S902-S903 may include the following steps:
  • the first error area is determined according to the binocular dense depth data and the first image data.
  • the error area includes the first error area.
  • the binocular depth data includes binocular dense depth data.
  • the result of correcting the depth of the first error region can be used as the corrected binocular depth data for blurring processing.
  • the binocular dense depth data may be the dense depth map in FIG. 8 or the binocular depth map in FIG. 8.
  • the corrected binocular depth data of the first error area can be determined according to the TOF depth data of the error area.
  • a densification algorithm can be used to determine the corrected binocular depth data of the first error region.
  • the binocular dense depth data of the area outside the first error area may be used as the corrected binocular depth data of the area outside the first error area.
  • the corrected binocular depth data may be dense depth data.
  • the depth of the first error area corresponding to the TOF data may be used as the depth of the first error area; the depth of the first error area may be densified; the double The depth of the area outside the first error area corresponding to the dense depth data is used as the depth of the area outside the corresponding first error area to obtain the corrected binocular depth data.
  • the result of correcting the depth of the first error region can also be used as the first corrected dense depth data for further correction.
  • the first corrected dense depth data of the first error area can be determined according to the TOF depth data of the error area.
  • a densification algorithm can be used to determine the first corrected density depth data of the first error region.
  • the binocular dense depth data of the area outside the first error area may be used as the first corrected dense depth data of the area outside the first error area.
  • the first modified dense depth data may be dense depth data.
  • the depth of the first error area corresponding to the TOF data may be used as the depth of the first error area; the depth of the first error area may be densified; the double The depth of the area outside the first error area corresponding to the mesh dense depth data is used as the depth of the area outside the corresponding first error area to obtain the first corrected dense depth data.
  • steps S902-S903 may include the following steps:
  • a first error area is determined.
  • the depth of the first error region is corrected.
  • the error area includes the first error area.
  • the binocular depth data includes binocular sparse depth data.
  • the result of correcting the depth of the first error region can be used as the corrected binocular depth data for blurring processing.
  • the corrected binocular depth data of the first error area can be determined according to the TOF depth data of the error area.
  • a densification algorithm can be used to determine the corrected binocular depth data of the first error region.
  • the corrected binocular depth data of the area outside the first error area may be determined based on the binocular sparse depth data of the area outside the first error area.
  • the corrected binocular depth data may be dense depth data.
  • the depth of the first error area corresponding to the TOF data may be used as the depth of the first error area; the area outside the first error area corresponding to the binocular sparse depth data As the corresponding depth of the area outside the first error area; densify the depth of the first error area and the area outside the first error area to obtain the corrected Binocular depth data.
  • the result of correcting the depth of the first error region can also be used as the first corrected dense depth data for further correction.
  • the first corrected dense depth data of the first error area can be determined according to the TOF depth data of the error area.
  • a densification algorithm can be used to determine the first corrected density depth data of the first error region.
  • the first corrected dense depth data of the area outside the first error area may be determined according to the binocular thinning depth data of the area outside the first error area.
  • a densification algorithm can be used to determine the first corrected dense depth data of the area outside the first error area.
  • the first modified dense depth data may be dense depth data.
  • the depth of the first error area corresponding to the TOF data is taken as the depth of the first error area; the binocular sparse depth data corresponds to the area outside the first error area The depth is taken as the corresponding depth of the area outside the first error area; the first error area and the depth of the area outside the first error area are densified to obtain the first corrected density In-depth data.
  • the further correction may include the following steps: determining the second error area according to the first corrected dense depth data, the TOF data, and the first image data.
  • the depth of the second error region is corrected to determine the corrected binocular depth data.
  • the error area includes the second error area.
  • the first error area may include at least one of the following areas: a repeated texture area, an area with a color difference less than a preset value, and a depth gradient area.
  • the first error area is determined according to the main image and the binocular depth image. Determine the first error-prone area according to the texture information of the main image; determine the first error area from the first error-prone area according to the binocular depth map; in the binocular depth map, the depth of the first error area The difference between the value and the surrounding depth value is greater than or equal to the first preset value.
  • the second error region includes the depth jump region in the first modified dense depth data.
  • a second error area is determined.
  • the red, green, and blue RGB information of the main image determine the edge area of the object in the main image; according to the binocular depth map, determine the second error area from the edge area; the edge area includes the second error area; In the binocular depth map, the difference between the depth value of the second error region and the surrounding depth value is greater than or equal to a second preset value.
  • the corrected binocular depth data of the second error area can be determined according to the TOF depth data of the error area.
  • a densification algorithm can be used to determine the corrected binocular depth data of the second error area based on the TOF depth data of the error area.
  • the first corrected dense depth data of the area outside the second error area may be used as the corrected binocular depth data of the area outside the first error area.
  • the depth of the second error region corresponding to the TOF data may be used as the depth of the second error region; the depth of the second error region may be densified; and the first error region may be densified; The depth of the area outside the second error area corresponding to a corrected dense depth data is used as the depth of the area outside the corresponding second error area to obtain the corrected binocular depth data.
  • the error area may include a portrait area.
  • the portrait area of the main image can be determined by the RGB information in the main image.
  • the depth value of the TOF depth map corresponding to the portrait region may be used to correct the depth value of the binocular depth map corresponding to the error region through a neural network.
  • the threshold can be used to determine whether the TOF data is valid and remove possible abnormal points in the TOF data.
  • Abnormal points may include deep error transition points and/or no data points.
  • the TOF determination data can be valid TOF depth data, or it can be a flag indicating whether the TOF depth data is valid or not, such as "0" or "1".
  • the embodiment of the present application does not limit the way the TOF determines data. Take TOF data as TOF depth data as an example. For example, data whose depth meets the threshold condition in TOF depth data can be recorded as valid.
  • the threshold condition may be that the depth satisfies a certain numerical range, for example, greater than or less than a certain preset value.
  • determining the second error area may include: determining the second error region according to the first modified dense depth data, TOF data, TOF determination data, and first image data Wrong area. That is, the second error area does not include the area corresponding to the invalid TOF depth data. That is, in the further correction, only the depth of the region corresponding to the valid TOF depth data may be corrected, and the depth of the region corresponding to the invalid TOF depth data may not be corrected.
  • step S904 perform blurring processing on the first image data according to the corrected binocular depth data.
  • FIG. 10 is a schematic flowchart of an image processing method provided by an embodiment of the present application.
  • step S1001 binocular image data of the scene and TOF data of the time of flight of the scene are acquired, and the binocular image data includes the main image obtained by the main camera.
  • step S1002 determine an error area in the main image according to the main image and the binocular depth map; the binocular depth map is determined according to the binocular image data;
  • step S1003 a TOF depth map is acquired according to the TOF data
  • step S1004 the depth value of the TOF depth map corresponding to the error area is used to correct the depth value of the binocular depth map corresponding to the error area through the neural network to obtain the corrected binocular A depth map
  • the modified binocular depth map includes depth values corresponding to the error area, and depth values corresponding to other areas, and the other areas are areas in the main image excluding the error area
  • the depth values corresponding to other regions are obtained according to the depth values corresponding to the other regions in the binocular depth map;
  • the depth value corresponding to the first error area in the TOF depth map may be used to correct the depth value corresponding to the first error area in the binocular depth map through a neural network to obtain the first error area A corrected binocular depth map; the depth value of the TOF depth map corresponding to the second error region can be used to correct the first corrected binocular depth map corresponding to the first error region through a neural network Second, the depth value of the error region to obtain the corrected binocular depth map.
  • the binocular depth map is a binocular dense depth map; the depth values corresponding to other regions are obtained according to the depth values corresponding to the other regions in the binocular depth map, including: The depth value of the other region corresponds to the depth value of the binocular dense depth map.
  • the binocular depth map is a binocular sparse depth map; the depth values corresponding to other regions are obtained according to the depth values corresponding to the other regions in the binocular depth map, including: The depth values of the other regions are obtained by densifying the depth values of the binocular sparse depth map corresponding to the other regions.
  • the densification treatment may be, for example, a method of densification treatment such as optical flow densification treatment.
  • the method further includes: determining the portrait area according to the main image; the second error area is located outside the first error area and the portrait area; the TOF depth map may be used The depth values corresponding to the first error area and the portrait area in the middle, and the depth values corresponding to the first error area and the portrait area in the binocular depth map are corrected by a neural network to obtain the first A corrected binocular depth map; the depth value of the TOF depth map corresponding to the second error region can be used to correct the first corrected binocular depth map corresponding to the second error region through a neural network The depth value of the error area to obtain the corrected binocular depth map.
  • step S1005 the main image is blurred according to the corrected binocular depth map.
  • the error area may include a first error area and/or a second error area.
  • the first error area may include a repeated texture area and/or an area where the color difference is less than a preset value.
  • the first error area can be determined according to the main image and the binocular depth image. Determine the first error-prone area according to the texture information of the main image; determine the first error area from the first error-prone area according to the binocular depth map; in the binocular depth map, the depth of the first error area The difference between the value and the surrounding depth value is greater than or equal to the first preset value.
  • the second error area may be a depth jump area.
  • the second error area can be determined according to the main image and the binocular depth image. According to the red, green, and blue RGB information of the main image, determine the edge area of the object in the main image; according to the binocular depth map, determine the second error area from the edge area; the edge area includes the second error area; In the binocular depth map, the difference between the depth value of the second error region and the surrounding depth value is greater than or equal to a second preset value.
  • the second error area may also be the entire depth jump area in the main image, or the depth jump area around the portrait area in the main image.
  • the second error area can be determined according to the main image, binocular depth image, and portrait area. Determine the edge area of the object in the main image according to the red, green, and blue RGB information of the main image; determine the second error area around the portrait area from the edge area according to the binocular depth map; the edge area includes the second error Area; In the binocular depth map, the difference between the depth value of the second error area and the surrounding depth value is greater than or equal to a second preset value.
  • the portrait area can be determined according to the RGB information of the main image.
  • the portrait area may be, for example, the entire area of the human body in the main image, or may be a part of the human body, such as a human face area, a human hand area, and the like.
  • the portrait area can also be understood as the error area.
  • the correction of the first error area, the second error area, and the portrait area can be performed in the same or different processors.
  • the correction of the first error area, the second error area, and the portrait area can be performed at the same time or at different times.
  • the first error area and the portrait area can be corrected in the first level network
  • the second error area can be corrected in the second level network.
  • the first level network and the second level network can be neural networks.
  • the embodiment of the present application does not limit the relationship between the first error area and the portrait area.
  • the first error region does not include the portrait region, or includes all or part of the portrait region.
  • the second error area includes a depth jump area in the binocular depth map.
  • the first error area includes at least one of the following areas: a repeated texture area, an area with a color difference less than a first preset value, or a depth gradient area.
  • the method before correcting the depth in the error region according to the TOF depth map to determine the corrected binocular depth map, the method further includes: adjusting the TOF depth map and/or The depth of the binocular depth map is such that the systematic depth error between the TOF depth map and the binocular depth map is smaller than a second preset value.
  • the method before correcting the depth in the error region according to the TOF depth map to determine the corrected binocular depth map, the method further includes: adjusting the TOF depth map and/or The relative position of the binocular depth map is such that the system position error of the TOF depth map and the binocular depth map is smaller than a third preset value.
  • FIG. 11 is a schematic flowchart of an image processing method proposed by an embodiment of the present application.
  • step S1101 the binocular image data of the scene and the TOF data of the flight time of the scene are acquired, and the binocular image data includes the main image acquired by the main camera.
  • step S1102 the binocular sparse depth map is obtained by using the binocular image data.
  • step S1103 a TOF depth map is acquired according to the TOF data.
  • step S1104 an error region located in the main image is determined according to the binocular sparse depth map and the binocular image data.
  • step S1105 the depth value corresponding to the error region in the TOF depth map is used to correct the depth value of the binocular sparse depth map corresponding to the error region through a neural network.
  • step S1106 using the binocular image data to perform densification processing on the binocular sparse depth map to obtain depth values corresponding to other regions to obtain a corrected binocular depth map; the corrected binocular depth map
  • the latter binocular depth map includes depth values corresponding to the error area and depth values corresponding to other areas, and the other areas are areas in the main image excluding the error area.
  • the portrait area may be determined according to the main image; the second error area is located outside the first error area and the portrait area; the TOF depth map may be used to correspond to the The depth values of the first error region and the portrait region are corrected by the neural network to correct the depth values of the binocular depth map corresponding to the first error region and the portrait region to obtain the first corrected binocular Depth map; the first corrected binocular depth map includes depth values corresponding to the first error region and the portrait region, and depth values corresponding to the first other region; the first other region is the The area in the main image excluding the first error area and the portrait area.
  • the binocular image data may be used to perform densification processing on the binocular sparse depth map to obtain depth values corresponding to other regions.
  • the binocular image data may be used to perform densification processing on the binocular sparse depth map; to obtain the first corrected binocular depth map corresponding to the first other depth value.
  • the depth value corresponding to the second error region in the TOF depth map may be used to correct the depth value corresponding to the second error region in the first corrected binocular depth map through a neural network to obtain The corrected binocular depth map; the corrected binocular depth map includes a depth value corresponding to the second error area, and a depth value corresponding to another area; the depth value of the other area Is the first modified binocular depth map corresponding to the other depth values.
  • the depth value of the TOF depth map corresponding to the first error region may be used to correct the depth value of the binocular depth map corresponding to the first error region through a neural network to obtain the first error region.
  • the binocular image data may be used to perform densification processing on the binocular sparse depth map to obtain depth values corresponding to other regions;
  • the binocular image data may be used to perform densification processing on the binocular sparse depth map corresponding to the first other depth value; to obtain the first corrected binocular depth map corresponding to the The first other depth value;
  • the depth value corresponding to the second error region in the TOF depth map may be used to correct the depth value corresponding to the second error region in the first corrected binocular depth map through a neural network to obtain The corrected binocular depth map; the corrected binocular depth map includes a depth value corresponding to the second error area, and a depth value corresponding to another area; the depth value of the other area Is the first modified binocular depth map corresponding to the other depth values.
  • step S1107 the main image is blurred according to the corrected binocular depth map.
  • the method before correcting the depth in the error region according to the TOF depth map to determine the corrected binocular depth map, the method further includes: adjusting the TOF depth map and/or The depth of the binocular depth map is such that the systematic depth error between the TOF depth map and the binocular depth map is smaller than a second preset value.
  • the method before correcting the depth in the error region according to the TOF depth map to determine the corrected binocular depth map, the method further includes: adjusting the TOF depth map and/or The relative position of the binocular depth map is such that the system position error of the TOF depth map and the binocular depth map is smaller than a third preset value.
  • FIG. 12 is a schematic flowchart of an image processing method proposed by an embodiment of the present application.
  • the binocular depth map is calculated.
  • the TOF depth data is calculated.
  • Image fusion uses a specific algorithm to combine two or more images into a new image.
  • the main image collected by the main image acquisition unit in the binocular system is blurred to obtain the blurred result image.
  • Deep fusion can be performed using a two-level convolutional neural network (CNN) network.
  • CNN convolutional neural network
  • FIG. 13 is a schematic flowchart of a deep fusion method proposed in an embodiment of the present application.
  • the binocular depth map can be the binocular depth map obtained after the depth post-processing in FIG. 8 or the dense depth map without the depth post-processing. Deep fusion can be performed using a two-level convolutional neural network (CNN) network.
  • CNN convolutional neural network
  • the binocular depth map and the TOF sparse depth map obtained from the TOF device may be certain system differences between the binocular depth map and the TOF sparse depth map obtained from the TOF device, such as a difference in position and/or a difference in depth.
  • the systematic error is caused by the system difference in the collection and calculation of the binocular depth map and the TOF depth map.
  • the position difference can be reflected by the coordinates in the same coordinate system.
  • the TOF sparse depth map and the binocular depth map can be matched by offline calibration or online calibration, that is, coordinate alignment.
  • the coordinate alignment can be row alignment and/or column alignment.
  • the difference matching parameters of the two are obtained through offline calibration, so as to match the TOF depth value to the binocular depth map.
  • the position of the TOF sparse depth map and/or the binocular depth map may be adjusted so that the system position error of the TOF sparse depth map and the binocular depth map is less than or equal to a preset value.
  • the TOF sparse depth map and the binocular depth map can be adjusted to the same size.
  • the system position error is caused by the difference between the position of the TOF device collecting TOF data and the binocular system collecting binocular image data.
  • the depth error of TOF data and binocular depth data can be reduced or even eliminated by adjusting the depth of TOF data.
  • the depth of the TOF data is adjusted so that the systematic depth error between the TOF data and the binocular depth data is less than or equal to a preset value.
  • the system depth error is caused by the difference in the calculation principle of TOF data and binocular depth data.
  • the TOF sparse depth map and/or the binocular depth map can be adjusted to reduce or even eliminate the depth error caused by the system between the TOF data and the binocular depth data.
  • the depth of the TOF sparse depth map and/or the binocular depth map may be adjusted so that the systematic depth error of the TOF sparse depth map and the binocular depth map is smaller than a third preset value.
  • the system depth error is caused by the difference between the calculation principles of the TOF sparse depth map and the binocular depth map.
  • the TOF sparse depth map and the binocular depth map are matched.
  • the first-level CNN network can determine the first error area based on the corrected main image and the binocular depth map.
  • the first-level CNN network can correct the first error area in the binocular depth map according to the TOF sparse depth map, that is, adjust the depth in the first error area of the binocular depth map according to the TOF depth, so as to get the first Correct the dense depth map.
  • the first modified dense depth map can be used as the final fusion depth map.
  • the first error area may be an error-prone area in the depth map calculated by binocular data, for example, it may be a repeated texture area, an area with a color difference less than a preset value, or a depth gradient area.
  • the second-level CNN network can further modify the first modified dense depth map.
  • the second-level CNN network can determine the second error area based on the corrected main image and the first corrected dense depth map.
  • the second-level CNN network can correct the second error area in the first corrected dense depth map according to the TOF sparse depth map, that is, adjust the depth in the second error area of the first corrected dense depth map according to the TOF depth, thereby The fusion depth map can be obtained.
  • the second error area may be an area where the depth jumps in the corrected dense depth map. Correcting the depth of the second error region can be used to improve the accuracy of the depth of the edge region of the object in the main image data, and can also be used to remove the wrong depth jump.
  • the main picture collected by the main picture acquisition unit in the binocular system is blurred to obtain the blurred result picture.
  • Figure 14 is a schematic diagram of an optical flow neural network (flownet) operation.
  • the binocular data obtained by the binocular system can extract image features through a network structure consisting of only convolutional layers and multiple vector convolution operations (convolution, conv) to perform dense depth calculation.
  • Binocular data can also use other network structures, such as a network structure that first extracts the features of two pictures independently, and then matches these features, so as to perform dense depth calculation.
  • the two-stage CNN network for the deep fusion process can use an encoder-decoder network.
  • Figure 15 is a schematic structural diagram of an encoding-decoding network.
  • the encoder can analyze the input data.
  • the encoder can be used for image recognition.
  • the encoder can be used to extract features of the image.
  • the extracted features as the output of the encoder can be transmitted to the decoder through skip connection.
  • the extracted features can be, for example, people, fingers, cars, car windows, white walls, repeated curtain grids, sky, etc.
  • the decoder can determine the error region in the binocular depth data according to the features extracted by the encoder, and correct the depth of the region.
  • the error area may be an area corresponding to one or more of all the features extracted by the encoder.
  • the error area may include a repeated texture area, an area with a color difference less than a preset value, a depth gradient area, or a depth jump area, etc.
  • the error area can be an error-prone area for
  • FIG. 16 is a schematic flowchart of an image processing method proposed by an embodiment of the present application.
  • the first-level network equipment that performs the deep fusion process can determine and correct the dense depth map based on the main image data in the binocular data, the dense depth data determined by the binocular data, and the TOF depth data.
  • the first-level network equipment can be an encoding-decoding network or a network of other structures.
  • the first-level network device can be a CNN network device or other artificial intelligence (AI) network.
  • the binocular dense depth data may be the dense depth map data as an intermediate result in FIG. 8 or the binocular depth map data as the final result in FIG. 8.
  • the first-level network device can correct the depth of the first error region in the binocular dense depth data according to the TOF depth data, and determine the corrected dense depth map.
  • the first-level network device may, for example, determine the first error area based on the main image data and binocular dense depth data.
  • the first error area may be an error-prone area in the depth map calculated by binocular data, for example, it may be a repeated texture area, an area with a color difference less than a preset value, or a depth gradient area.
  • the encoder 1 in the first-level network device can perform feature extraction based on the main image data and binocular dense depth data.
  • the decoder 1 in the first-level network device can determine the first error area based on the extracted features and combined with the main image data and/or binocular dense data.
  • the correction of the depth of the first error region may be performed by the decoder 1 in the first-level network device.
  • the decoder 1 in the first-level network device can correct the depth of the first error region in the dense depth data according to the TOF depth data, and determine the corrected dense depth map.
  • the modified dense depth map output by the first-level network can be used as the input of the second-level network.
  • the modified dense depth map can be used as the final fusion depth map.
  • the second-level CNN network for the deep fusion process can determine the second dense depth map based on the modified dense depth map, main image data and TOF depth data.
  • the second-level network equipment can correct the depth of the second error region in the corrected dense depth map to determine the final fusion depth map.
  • the second-level network equipment can be an encoding-decoding network or a network of other structures.
  • the second-level network device can be a CNN network device or other artificial intelligence (AI) network.
  • the final fusion depth map can be a dense depth map.
  • the second-level network device can determine the second error area, for example, it can determine the second error area based on the main image data, the modified dense depth map, and the TOF determination data.
  • the second error area may be an edge area of the object in the main image data, for example, it may be an area where the depth of the modified dense depth image has jumped. Correcting the depth of the second error region can be used to improve the accuracy of the depth of the edge region of the object in the main image data, and can also be used to remove the wrong depth jump.
  • the threshold value can be used to determine whether the TOF data is valid, and to remove possible abnormal points in the TOF data. Abnormal points may include deep error transition points and/or no data points.
  • the TOF determination data can be valid TOF depth data, or it can be a flag indicating whether the TOF depth data is valid or not, such as "0" or "1". The embodiment of the present application does not limit the way the TOF determines data.
  • the TOF data may be TOF depth data. In the TOF depth data, the data whose depth meets the threshold condition can be recorded as valid.
  • the threshold condition may be that the depth satisfies a certain numerical range, for example, greater than or less than a certain preset value.
  • the decoder 2 of the second-level network device can correct the depth of the second error region in the dense depth map based on the TOF data, or based on the TOF data and the TOF determination data. According to the TOF determination data, the second-level network device may only correct the depth of the second error region corresponding to the valid TOF depth data, and not correct the depth of the second error region corresponding to the invalid TOF depth data.
  • the main picture collected by the main picture acquisition unit in the binocular system is blurred to obtain the blurred result picture.
  • the dense depth map of the binocular system is first calculated, and then the final fusion depth map is obtained through CNN network optimization. Since the CNN network can calculate and optimize the depth map end-to-end, it can take the corrected main and auxiliary images and the TOF depth map as input, and obtain the final fusion depth map through the CNN network.
  • FIG. 17 is a schematic flowchart of an image processing method proposed by an embodiment of the present application.
  • depth calculation is performed to determine the fusion depth map.
  • the main picture collected by the main picture acquisition unit in the binocular system is blurred to obtain the blurred result picture.
  • FIG. 18 is a schematic flowchart of an image processing method proposed by an embodiment of the present application. Obtaining the fusion depth map can be divided into four steps: image correction, sparse depth map calculation, dense depth map calculation, and post-processing.
  • the binocular data and TOF data are input into the depth calculation module at the same time, and the depth calculation module performs optimization calculations based on the input three-way data, generates a depth map that meets the requirements, and finally enters a high-quality blurring result map.
  • the image correction uses the preset internal and external parameters of the binocular system to correct the input binocular image and obtain the image after line alignment. According to the calibration values of the internal and external parameters of the binocular system, the binocular data collected by the binocular system is corrected to obtain the corrected main image and the corrected secondary image.
  • the corrected main image and the corrected secondary image perform ROI segmentation to determine the ROI area.
  • the binocular sparse depth map can also be determined.
  • the binocular sparse depth map can be obtained by sparse depth map calculation.
  • the sparse depth map calculation can be performed in a variety of ways existing or in the future.
  • the DCE algorithm can be executed based on software, or it can be based on a deep computing hardware chip, such as a DMAP chip.
  • a binocular sparse depth map based on the main image can be obtained.
  • the dense depth map can be determined according to the corrected main image, the corrected auxiliary image, the ROI area, the sparse depth map, and the TOF depth map.
  • the TOF sparse depth map can be aligned with the binocular depth map to obtain the adjusted TOF sparse depth map.
  • the TOF sparse depth map and the binocular depth map can be matched by offline calibration or online calibration, that is, coordinate alignment.
  • the coordinate alignment can be row alignment and/or column alignment. For example, the difference matching parameters of the two are obtained through offline calibration, so as to match the TOF depth value to the binocular depth map.
  • the depth error of TOF data and binocular depth data can be reduced or even eliminated by adjusting the depth of TOF data.
  • the depth of the TOF data is adjusted so that the systematic depth error between the TOF data and the binocular depth data is less than or equal to a preset value.
  • the system depth error is caused by the difference in the calculation principle of TOF data and binocular depth data.
  • the dense depth map can be obtained by the depth density algorithm. Obtain the dense depth map according to the corrected main and auxiliary images, binocular sparse depth map, adjusted TOF sparse depth map, etc.
  • the depth densification algorithm may be, for example, a densification algorithm based on the AI depth filling algorithm.
  • the depth of the first error region in the binocular depth map is corrected according to the TOF data.
  • the binocular depth map can be a sparse depth map or a binocular dense depth map.
  • the binocular dense depth map can be determined according to the corrected main image, the corrected secondary image, the ROI area, and the sparse depth map.
  • the first error area may include at least one of the following areas in the first image data: a repeated texture area, an area with a color difference less than a preset value, and a depth gradient area.
  • the depth gradient area can be a portrait area.
  • the portrait area of the main image can be determined by the corrected main image.
  • the portrait area can be determined by the RGB characteristics of the main image.
  • different parameters can be used to calculate the inside and outside of the ROI area.
  • the calculation parameters in the ROI area can make the depth calculation result in the ROI area smoother.
  • the depth value of one or more points of the depth jump in the ROI area can be removed.
  • Depth jumps can be abnormal points of depth.
  • the first error area of the main image can be determined by the texture information of the main image, and the depth value corresponding to the first error area can be corrected using TOF data.
  • the TOF data can be used to correct the depth value of the portrait area.
  • the dense depth map can be post-processed. According to the main image or the corrected main image, ROI area, dense depth map and other information, optimize the edge of the ROI area in the dense depth map, the depth jump area, correct obvious depth errors, and determine the final binocular depth map.
  • the main picture collected by the main picture acquisition unit in the binocular system is blurred to obtain the blurred result picture.
  • FIG. 19 is a schematic flowchart of an image processing method proposed by an embodiment of the present application.
  • the first-level network device can determine the first modified dense depth map based on the binocular data, the sparse depth map determined by the binocular data, and the TOF depth data.
  • the first-level network device may determine the first modified density depth map according to the AI-based densification algorithm.
  • the first-level network device can be a CNN network device or other artificial intelligence (AI) network.
  • the first-level network device can determine the first error area based on the binocular data and the sparse depth map.
  • the first error area may be an error-prone area in the depth map calculated by binocular data, for example, it may be a repeated texture area, an area with a color difference less than a preset value, or a depth gradient area.
  • the first-level network device may correct the first error region in the sparse depth data according to the TOF data, thereby determining to correct the dense depth data.
  • the first-level network device corrects the first error region in the sparse depth data according to the TOF data, and the method for determining and correcting the dense depth data is not limited.
  • the first-level network device may determine the dense depth data of the area outside the first error area based on the binocular data, or based on the binocular data and the sparse depth map.
  • the first-level network device can determine the dense depth data of the first error region based on TOF data and binocular data, or based on TOF data, binocular data, and sparse depth map.
  • the corrected dense depth data may include dense depth data of an area outside the first error area and dense depth data of the first error area.
  • the first-level network device may modify the depth of the first error region in the sparse depth map according to the TOF data to obtain a modified sparse depth map.
  • the first-level network equipment can determine the modified dense depth data based on the modified sparse depth map and binocular data.
  • the modified dense depth data can be used as a fusion depth map.
  • the fusion depth map can also be obtained after the revised dense depth data is revised by the second-level network equipment.
  • the second-level network device can determine the second dense depth map based on the modified dense depth map, main image data, and TOF depth data.
  • the second-level network equipment can correct the depth of the second error region in the corrected dense depth map to determine the final fusion depth map.
  • the second-level network equipment can be an encoding-decoding network or a network of other structures.
  • the second-level network equipment can be CNN network equipment or other AI networks.
  • the final fusion depth map can be a dense depth map.
  • the second-level network device can determine the second error area, for example, it can determine the second error area based on the main image data, the modified dense depth map, and the TOF determination data.
  • the second error area may be an edge area of the object in the main image data, for example, it may be an area where the depth of the modified dense depth image has jumped. Correcting the depth of the second error region can be used to improve the accuracy of the depth of the edge region of the object in the main image data, and can also be used to remove the wrong depth jump.
  • the decoder 2 of the second-level network device can correct the depth of the second error region in the dense depth map based on the TOF data and/or TOF determination data. According to the TOF determination data, the second-level network device may only correct the depth of the second error region corresponding to the valid TOF depth data, and not correct the depth of the second error region corresponding to the invalid TOF depth data.
  • the threshold value can be used to determine whether the TOF data is valid, and to remove possible abnormal points in the TOF data. Abnormal points may include deep error transition points and/or no data points.
  • the TOF determination data can be valid TOF depth data, or it can be a flag indicating whether the TOF depth data is valid or not, such as "0" or "1". The embodiment of the present application does not limit the way the TOF determines data.
  • the TOF data may be TOF depth data. In the TOF depth data, the data whose depth meets the threshold condition can be recorded as valid.
  • the threshold condition may be that the depth satisfies a certain numerical range, for example, greater than or less than a certain preset value.
  • the main picture collected by the main picture acquisition unit in the binocular system is blurred to obtain the blurred result picture.
  • FIG. 20 is an image processing apparatus 1800 provided by an embodiment of the present application, including:
  • the acquiring module 1810 is configured to acquire binocular image data of a scene and TOF data of the time of flight of the scene.
  • the binocular image data includes first image data and second image data obtained according to different cameras.
  • the determining module 1820 is configured to determine the error area according to the binocular image data and the binocular depth data; the binocular depth data is determined according to the binocular image data.
  • the correction module 1830 is configured to correct the depth in the error area according to the TOF data to determine the corrected binocular depth data of the scene.
  • the blurring processing module 1840 is configured to perform blurring processing on the first image data according to the corrected binocular depth data, where the first image data is image data to be displayed.
  • the error area includes a first error area; the binocular depth data includes binocular dense depth data.
  • the determining module 1820 is configured to determine, according to the binocular depth data, and the first image data or the second image data, that at least one of accuracy or stability in the binocular depth data is not satisfied
  • the area of the preset range corresponds to the area of the first image data.
  • the determining module 1820 is configured to determine the first error area according to the binocular dense depth data and the first image data.
  • the correction module 1830 is configured to correct the depth of the first error region according to the TOF data and the binocular dense depth data to determine the corrected binocular depth data.
  • the correction module 1830 is configured to: use the depth of the first error region corresponding to the TOF data as the depth of the first error region; and perform densification processing on the depth of the first error region ; Use the depth of the area outside the first error area corresponding to the binocular dense depth data as the depth of the area outside the corresponding first error area to obtain the corrected binocular depth data.
  • the error area includes a first error area and a second error area;
  • the binocular depth data includes binocular dense depth data.
  • the determining module 1820 is configured to determine the first error area according to the binocular dense depth data and the first image data.
  • the correction module 1830 is configured to correct the depth of the first error region according to the TOF data and the binocular dense depth data to determine the first corrected dense depth data.
  • the determining module 1820 is further configured to determine the second error area according to the first corrected dense depth data, the TOF data, and the first image data.
  • the correction module 1830 is further configured to correct the depth of the second error region according to the TOF data and the first corrected dense depth data to determine the corrected binocular depth data.
  • the correction module 1830 is configured to: use the depth of the first error region corresponding to the TOF data as the depth of the first error region; and perform densification processing on the depth of the first error region Use the depth of the region outside the first error region corresponding to the binocular dense depth data as the depth of the region outside the corresponding first error region to obtain the first corrected dense depth data.
  • the error area includes a first error area;
  • the binocular depth data includes binocular sparse depth data.
  • the determining module 1820 is configured to determine a first error area according to the binocular sparse depth data, the first image data, and the second image data;
  • the correction module 1830 is configured to correct the depth of the first error region according to the TOF data and the binocular sparse depth data to determine the corrected binocular depth data.
  • the correction module 1830 is configured to: use the depth of the first error region corresponding to the TOF data as the depth of the first error region; and use the first error region corresponding to the binocular sparse depth data The depth of the area outside the area is taken as the corresponding depth of the area outside the first error area; the first error area and the depth of the area outside the first error area are densified to obtain The corrected binocular depth data.
  • the error area includes a first error area and a second error area
  • the binocular depth data includes binocular sparse depth data
  • the determining module 1820 is configured to determine the first error region according to the binocular sparse depth data, the first image data, and the second image data.
  • the correction module 1830 is configured to correct the depth of the first error region according to the TOF data and the binocular sparse depth data, and determine the first corrected dense depth data.
  • the determining module 1820 is further configured to determine the second error area according to the first corrected dense depth data, the TOF data, and the first image data.
  • the correction module 1830 is further configured to correct the depth of the second error region according to the TOF data and the first corrected dense depth data to determine the corrected binocular depth data.
  • the correction module 1830 is configured to: use the depth of the first error region corresponding to the TOF data as the depth of the first error region; and use the first error region corresponding to the binocular sparse depth data The depth of the area outside the area is taken as the corresponding depth of the area outside the first error area; the first error area and the depth of the area outside the first error area are densified to obtain The first modified dense depth data.
  • the correction module 1830 is configured to: use the depth of the second error area corresponding to the TOF data as the depth of the second error area; perform densification processing on the depth of the second error area ; Taking the depth of the area outside the second error area corresponding to the first corrected dense depth data as the depth of the area outside the corresponding second error area to obtain the corrected binocular In-depth data.
  • the second error region includes a depth jump region in the first modified dense depth data.
  • the first error area includes at least one of the following areas: a repeated texture area, an area with a color difference less than a first preset value, and a depth gradient area.
  • the image processing apparatus 1800 further includes a first adjustment module for correcting the depth in the error region according to the TOF data to determine the corrected binocular of the scene Before depth data, the depth of the TOF data and/or the binocular depth data is adjusted so that the systematic depth error between the TOF data and the binocular data is smaller than a second preset value.
  • the image processing apparatus 1800 further includes a second adjustment module, configured to correct the depth in the error area according to the TOF data to determine the corrected binocular of the scene Before depth data, the relative position of the TOF data and/or the binocular depth data is adjusted so that the system position error of the TOF data and the binocular data is smaller than a third preset value.
  • a second adjustment module configured to correct the depth in the error area according to the TOF data to determine the corrected binocular of the scene Before depth data, the relative position of the TOF data and/or the binocular depth data is adjusted so that the system position error of the TOF data and the binocular data is smaller than a third preset value.
  • FIG. 21 is an image processing apparatus 2100 provided by an embodiment of the present application, including:
  • the obtaining module 2110 is configured to obtain binocular image data of the scene and TOF data of the time of flight of the scene, the binocular image data including the main image obtained by the main camera;
  • the determining module 2120 is configured to determine an error area in the main image according to the main image and the binocular depth map; the binocular depth map is determined according to the binocular image data;
  • the correction module 2130 uses the depth value corresponding to the error area in the TOF depth map to correct the depth value corresponding to the error area in the binocular depth map through a neural network to obtain the corrected binocular A depth map, the modified binocular depth map includes depth values corresponding to the error area, and depth values corresponding to other areas, and the other areas are areas in the main image excluding the error area
  • the depth value corresponding to the other area is obtained according to the depth value of the binocular depth map corresponding to the other area;
  • the TOF depth map is obtained according to the TOF data;
  • the blur processing module 2140 performs blur processing on the main image according to the corrected binocular depth map.
  • the error area includes a first error area and a second error area, and the second error area is located outside the first error area and the portrait area; the portrait area is obtained according to the main image ;
  • the correction module 2130 is configured to use the depth values corresponding to the first error region and the portrait region in the TOF depth map to correct the binocular depth map corresponding to the first error region and the portrait region through a neural network.
  • the depth value of the portrait area to obtain the first corrected binocular depth map;
  • the correction module 2130 is configured to use the depth value of the TOF depth map corresponding to the second error region to correct the first corrected binocular depth map corresponding to the second error region through a neural network Depth value to obtain the corrected binocular depth map.
  • the error area includes a first error area and a second error area, and the second error area is located outside the first error area;
  • the correction module 2130 is configured to use the depth value of the TOF depth map corresponding to the first error region to correct the depth value of the binocular depth map corresponding to the first error region through a neural network to obtain The first revised binocular depth map;
  • the correction module 2130 is configured to use the depth value of the TOF depth map corresponding to the second error region to correct the first corrected binocular depth map corresponding to the second error region through a neural network Depth value to obtain the corrected binocular depth map.
  • the determining unit 2120 is configured to determine the first error-prone area according to the texture information of the main image; determine the first error-prone area from the first error-prone area according to the binocular depth map; In the target depth map, the difference between the depth value of the first error region and the surrounding depth value is greater than or equal to a first preset value.
  • the second error area includes a depth jump area in the binocular depth map.
  • the first error area includes at least one of the following areas: a repeated texture area and an area with a color difference less than a first preset value.
  • the device 2100 further includes: an adjustment module for adjusting the depth of the TOF depth map and/or the binocular depth map so that the system depth error between the TOF depth map and the binocular depth map Less than the second preset value.
  • the adjustment module is configured to adjust the relative position of the TOF depth map and/or the binocular depth map so that the system position error of the TOF depth map and the binocular depth map is smaller than a third prediction. Set value.
  • FIG. 22 is a terminal device 1900 proposed by an embodiment of the present application, including:
  • the binocular system 1910 is used to collect binocular image data, where the binocular image data includes first image data and second image data.
  • TOF device 1920 used to collect TOF data.
  • the processor 1930 when the program instructions are executed by the at least one processor, the processor 1930 is configured to perform the following operations: determine an error area according to the binocular image data and binocular depth data; the binocular depth data Is determined according to the binocular image data; according to the TOF data, the depth in the error area is corrected to determine the corrected binocular depth data of the scene; according to the corrected binocular depth data; The binocular depth data performs blurring processing on the first image data.
  • the processor 1930 may also be used to obtain the binocular image data and the TOF data.
  • the error area includes a first error area;
  • the binocular depth data includes binocular dense depth data;
  • the processor 1930 is configured to perform the following operations: determine the first error area according to the binocular dense depth data and the first image data; according to the TOF data and the binocular dense depth data, perform the following operations: The depth of the first error region is corrected to determine the corrected binocular depth data.
  • the processor 1930 is configured to: use the depth of the first error region corresponding to the TOF data as the depth of the first error region; and perform densification processing on the depth of the first error region ; Use the depth of the area outside the first error area corresponding to the binocular dense depth data as the depth of the area outside the corresponding first error area to obtain the corrected binocular depth data.
  • the error area includes a first error area and a second error area
  • the binocular depth data includes binocular dense depth data
  • the processor 1930 is configured to perform the following operations: according to the binocular dense depth data and Determining the first error region by the first image data; correcting the depth of the first error region according to the TOF data and the binocular dense depth data to determine the first corrected dense depth data; According to the first corrected dense depth data, the TOF data, and the first image data, the second error area is determined; according to the TOF data and the first corrected dense depth data, the second The depth of the error area is corrected to determine the corrected binocular depth data.
  • the processor 1930 is configured to: use the depth of the first error region corresponding to the TOF data as the depth of the first error region; and perform densification processing on the depth of the first error region Use the depth of the region outside the first error region corresponding to the binocular dense depth data as the depth of the region outside the corresponding first error region to obtain the first corrected dense depth data.
  • the error area includes a first error area
  • the binocular depth data includes binocular sparse depth data
  • the processor 1930 is configured to perform the following operations: according to the binocular sparse depth data, the first image Data and the second image data to determine the first error area; according to the TOF data and the binocular sparse depth data, the depth of the first error area is corrected to determine the corrected Binocular depth data.
  • the processor 1930 is configured to: use the depth of the first error region corresponding to the TOF data as the depth of the first error region; and use the first error region corresponding to the binocular sparse depth data
  • the depth of the area outside the area is taken as the corresponding depth of the area outside the first error area; the first error area and the depth of the area outside the first error area are densified to obtain The corrected binocular depth data.
  • the error area includes a first error area and a second error area
  • the binocular depth data includes binocular sparse depth data
  • the processor 1930 is configured to perform the following operations: according to the binocular sparse depth data, The first image data and the second image data determine the first error area; according to the TOF data and the binocular sparse depth data, the depth of the first error area is corrected to determine the first error area A modified dense depth data; determining the second error area according to the first modified dense depth data, the TOF data, and the first image data; according to the TOF data and the first modified dense depth data , Correcting the depth of the second error region to determine the corrected binocular depth data.
  • the processor 1930 is configured to: use the depth of the first error region corresponding to the TOF data as the depth of the first error region; and use the first error region corresponding to the binocular sparse depth data
  • the depth of the area outside the area is taken as the corresponding depth of the area outside the first error area; the first error area and the depth of the area outside the first error area are densified to obtain The first modified dense depth data.
  • the processor 1930 is configured to: use the depth of the second error region corresponding to the TOF data as the depth of the second error region; and perform densification processing on the depth of the second error region ; Taking the depth of the area outside the second error area corresponding to the first corrected dense depth data as the depth of the area outside the corresponding second error area to obtain the corrected binocular In-depth data.
  • the second error region includes a depth jump region in the first corrected dense depth data.
  • the first error area includes at least one of the following areas: a repeated texture area, an area with a color difference less than a first preset value, or a depth gradient area.
  • the processor 1930 is configured to: adjust the TOF before correcting the depth in the error region according to the TOF data to determine the corrected binocular depth data of the scene The depth of the data and/or the binocular depth data, so that the systematic depth error of the TOF data and the binocular data is smaller than a second preset value.
  • the processor 1930 is configured to: adjust the TOF before correcting the depth in the error region according to the TOF data to determine the corrected binocular depth data of the scene The relative position of the data and/or the binocular depth data, so that the system position error of the TOF data and the binocular data is smaller than a third preset value.
  • An embodiment of the application further provides an image processing device, including: an acquisition module for acquiring binocular image data of a scene and TOF data of the time of flight of the scene.
  • the binocular image data includes the first image data obtained from different cameras. One image data and second image data; a correction module for correcting the binocular depth data according to the TOF data to obtain corrected binocular depth data, the binocular depth data is based on the binocular image Data determined; a blurring processing module, configured to perform blurring processing on the first image data according to the corrected binocular depth data, wherein the first image data is image data to be displayed.
  • the correction module includes a determining unit for determining an error area based on the binocular image data and binocular depth data; and a correction unit for determining the error area based on the TOF data Correct the depth in the middle to determine the corrected binocular depth data.
  • the determining unit is specifically configured to: determine the accuracy or stability of the binocular depth data according to the binocular depth data, and the first image data or the second image data At least one item in does not meet the preset range and corresponds to the area of the first image data.
  • the error area includes a first error area;
  • the binocular depth data includes binocular dense depth data;
  • the determining unit is specifically configured to, according to the binocular dense depth data and the first Image data, determining the first error area, the first error area being the first image data corresponding to an area where at least one of accuracy or stability in the binocular depth data does not meet a preset range
  • the correction unit is used to correct the depth of the first error area of the binocular dense depth data according to the TOF data to determine the corrected binocular depth data.
  • the correction unit is configured to: use the depth of the first error region corresponding to the TOF data as the depth of the first error region; Perform densification processing; take the depth of the area outside the first error area corresponding to the binocular dense depth data as the depth of the area outside the corresponding first error area to obtain the corrected Binocular depth data.
  • the error area includes a first error area and a second error area; the second error area includes a partial area outside the first error area; the binocular depth data includes binocular Dense depth data; the determining unit is configured to determine the first error area based on the binocular dense depth data and the first image data, where the first error area is the accuracy in the binocular depth data Or the area of the first image data corresponding to the area where at least one of the stability does not meet the preset range; the correction unit is configured to, according to the TOF data, perform correction on the binocular dense depth data The depth of the first error region is corrected to determine the first corrected dense depth data; the determining unit is further configured to determine the first corrected dense depth data, the TOF data, and the first image data.
  • the second error area is an area of the first image data corresponding to an area where at least one of accuracy or stability in the binocular depth data does not satisfy a preset range; and the correction The unit is further configured to correct the depth of the second error region of the first corrected dense depth data according to the TOF data to determine the corrected binocular depth data.
  • the correction unit is configured to: use the depth of the first error region corresponding to the TOF data as the depth of the first error region; Perform densification processing; use the depth of the area outside the first error area corresponding to the binocular dense depth data as the depth of the area outside the corresponding first error area to obtain the first correction Dense depth data.
  • the error area includes a first error area
  • the binocular depth data includes binocular sparse depth data
  • the determining unit is configured to, according to the binocular sparse depth data, the first Image data and the second image data, determining the first error area, the first error area being the area corresponding to the area where at least one of accuracy or stability in the binocular depth data does not meet a preset range
  • the correction unit is configured to correct the depth of the first error area of the binocular sparse depth data according to the TOF data to determine the corrected Binocular depth data.
  • the correction unit is configured to: use the depth of the first error region corresponding to the TOF data as the depth of the first error region; and set the depth of the first error region corresponding to the binocular sparse depth data.
  • the depth of the area outside the first error area is taken as the depth of the area outside the corresponding first error area; the depth of the first error area and the area outside the first error area is densified Processing to obtain the corrected binocular depth data.
  • the error area includes a first error area and a second error area; the second error area includes a partial area outside the first error area; the binocular depth data includes binocular Sparse depth data; the determining unit is configured to determine the first error area based on the binocular sparse depth data, the first image data, and the second image data, where the first error area is The area of the first image data corresponding to the area where at least one of accuracy or stability in the binocular depth data does not meet the preset range; the correction unit is configured to, according to the TOF data, perform the correction of the double Correcting the depth of the first error region of the sparse depth data to determine the first corrected dense depth data; the determining unit is further configured to: according to the first corrected dense depth data, the TOF data, and the first Image data, determining the second error area, the second error area being the first image data corresponding to an area where at least one of accuracy or stability in the binocular depth data does not meet a preset range
  • the correction unit is also used to correct the
  • the correction unit is configured to: use the depth of the first error region corresponding to the TOF data as the depth of the first error region; and set the depth of the first error region corresponding to the binocular sparse depth data.
  • the depth of the area outside the first error area is taken as the depth of the area outside the corresponding first error area; the depth of the first error area and the area outside the first error area is densified Processing to obtain the first modified dense depth data.
  • the correction unit is configured to: use the depth of the second error region corresponding to the TOF data as the depth of the second error region; and determine the depth of the second error region Perform densification processing; take the depth of the area outside the second error area corresponding to the first corrected dense depth data as the depth of the area outside the corresponding second error area to obtain the corrected After the binocular depth data.
  • the second error region includes a depth jump region in the first modified dense depth data.
  • the first error area includes at least one of the following areas: a repeated texture area, an area with a color difference less than a first preset value, or a depth gradient area.
  • it further includes: a first adjustment module, configured to adjust the depth in the error region according to the TOF data to determine the corrected binocular depth data.
  • the depth of the TOF data and/or the binocular depth data is such that the systematic depth error between the TOF data and the binocular data is smaller than a second preset value.
  • it further includes: a second adjustment module, configured to adjust the depth in the error region according to the TOF data to determine the corrected binocular depth data.
  • the relative position of the TOF data and/or the binocular depth data is such that the systematic position error of the TOF data and the binocular data is smaller than a third preset value.
  • An embodiment of the present application further provides an image processing device, including: a memory, configured to store code; and a processor, configured to read the code in the memory to execute the above method.
  • the embodiment of the present application also provides a computer program storage medium, the computer program storage medium has program instructions, and when the program instructions are executed, the method in the foregoing is executed.
  • An embodiment of the present application also provides a chip system, the chip system includes at least one processor, and when a program instruction is executed in the at least one processor, the method in the foregoing is executed.
  • At least one refers to one or more
  • multiple refers to two or more.
  • And/or describes the association relationship of the associated objects, indicating that there can be three types of relationships, for example, A and/or B, which can mean the existence of A alone, A and B at the same time, and B alone. Among them, A and B can be singular or plural.
  • the character “/” generally indicates that the associated objects are in an “or” relationship.
  • “The following at least one item” and similar expressions refer to any combination of these items, including any combination of single items or plural items.
  • At least one of a, b, and c can represent: a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.
  • the disclosed system, device, and method may be implemented in other ways.
  • the device embodiments described above are only illustrative.
  • the division of the units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components can be combined or It can be integrated into another system, or some features can be ignored or not implemented.
  • the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.
  • each unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.
  • the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium.
  • the technical solution of this application essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present application.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disk and other media that can store program code .

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Signal Processing (AREA)
  • Multimedia (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • Computational Linguistics (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Data Mining & Analysis (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Electromagnetism (AREA)
  • Remote Sensing (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Human Computer Interaction (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Processing (AREA)

Abstract

本申请提供了一种图像处理的方法，包括：在获取双摄像头产生的拍摄场景的双目图像数据后，利用该双目图像数据确定双目深度数据，以确定图像中的错误区域，并根据与上述场景对应的飞行时间TOF数据，对该错误区域的深度进行修正，最后将被修正后的双目深度数据用于对双目图像数据中的其中一目图像数据进行虚化处理。由于在对错误区域的深度进行修正时考虑了TOF，上述方法可以有效避免双目深度估计方式的精度和稳定性较差造成的深度估计错误，提高了深度估计的精度和稳定性。

Description

一种基于Dual Camera+TOF的大光圈虚化方法
本申请要求于2019年3月25日提交中国国家知识产权局、申请号为201910229288.2、申请名称为“一种基于Dual Camera+TOF的大光圈虚化方法”的中国专利申请的优先权，于2019年4月23日提交中国专利局、申请号为201910330861.9、申请名称为“图像处理的装置和方法”的中国专利申请的优先权，以及于2019年8月14日提交中国专利局、申请号为201910749592.X、申请名称为“图像处理的装置和方法”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及图像处理领域，具体涉及一种图像处理的装置和方法。
背景技术
大光圈虚化效果是一种特殊的拍照效果，该效果的出发点是模拟单反相机效果。单反相机可以保持对焦物体清晰，非对焦物体模糊，从而使对焦物体更加醒目突出。具有大光圈虚化效果的图像中，目标物体和同目标物体具有相同或相近的深度的物体保持清晰，其余物体被模糊。
通过双目系统可以确定采集的图像中物体的深度。通过双目系统确定的深度图像素点较多，分辨率高。且双目系统与人眼感官相符，应用范围更广。但是双目系统确定的深度图的精度和稳定性较差。
为了实现大光圈虚化效果，可以利用双目系统确定物体深度从而进行虚化处理，但是由于双目系统深度估计结果的精度和稳定性的限制，可能导致虚化错误。
发明内容
本申请提供一种图像处理的方法。通过该方法，可以解决双目系统确定物体深度从而进行虚化处理可能导致虚化错误的问题。结合TOF数据对根据双目系统确定的深度数据进行修正，从而减少甚至消除具有大光圈虚化效果的图像中虚化错误。
第一方面，提供一种图像处理的方法，包括：获取场景的双目图像数据，以及所述场景的飞行时间TOF数据，所述双目图像数据包括根据不同摄像头得到的第一图像数据和第二图像数据；根据所述双目图像数据和所述双目深度数据，确定错误区域；所述双目深度数据是根据所述双目图像数据确定的；根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据；根据所述被修正后的双目深度数据，对所述第一图像数据进行虚化处理，其中所述第一图像数据为待显示的图像数据。
通过上述方法，可以根据TOF数据对错误区域进行修正，有效避免双目深度估计方式的精度和稳定性较差的造成的对一些特定区域的深度估计错误。结合TOF数据精度和稳定性的优势，以及双目图像分辨率高的特点，实现精确、稳定、高分辨率的深度估计，从而减少甚至消除具有大光圈虚化效果的图像中虚化错误。
结合第一方面，在一种可能的实施方式中，所述根据双目图像数据和双目深度数据，确定错误区域的步骤，包括：根据所述双目深度数据，以及所述第一图像数据或所述第二图像数据，确定所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域。
结合第一方面，在一种可能的实施方式中，所述错误区域包括第一错误区域；所述双目深度数据包括双目稠密深度数据；所述根据所述双目深度数据，以及所述第一图像数据或所述第二图像数据，确定所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域，包括：根据所述双目稠密深度数据和所述第一图像数据，确定所述第一错误区域；所述根据所述TOF数据，对所述错误区域中的深度进行修正，以确定所述预设场景的被修正后的双目深度数据，包括：根据所述TOF数据，对所述双目稠密深度数据的所述第一错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
结合第一方面，在一种可能的实施方式中，所述根据所述TOF数据，对所述双目稠密深度数据的所述第一错误区域的深度进行修正，以确定所述被修正后的双目深度数据，包括：将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；对所述第一错误区域的所述深度进行稠密化处理；将所述双目稠密深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度，以获得所述被修正后的双目深度数据。
结合第一方面，在一种可能的实施方式中，所述错误区域包括第一错误区域和第二错误区域；所述双目深度数据包括双目稠密深度数据；所述根据所述双目图像数据和所述双目深度数据，确定错误区域；根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据，包括：根据所述双目稠密深度数据和所述第一图像数据，确定所述第一错误区域；根据所述TOF数据，对所述双目稠密深度数据的所述第一错误区域的深度进行修正，以确定第一修正稠密深度数据；根据所述第一修正稠密深度数据、所述TOF数据、所述第一图像数据，确定所述第二错误区域；根据所述TOF数据，对所述第一修正稠密深度数据的所述第二错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
结合第一方面，在一种可能的实施方式中，所述根据所述TOF数据，对所述双目稠密深度数据的所述第一错误区域的深度进行修正，以确定第一修正稠密深度数据，包括：将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；对所述第一错误区域的所述深度进行稠密化处理；将所述双目稠密深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度，以获得所述第一修正稠密深度数据。
结合第一方面，在一种可能的实施方式中，所述错误区域包括第一错误区域；所述双目深度数据包括双目稀疏深度数据；所述根据所述双目图像数据和所述双目深度数据，确定错误区域，包括：根据所述双目稀疏深度数据、所述第一图像数据和所述第二图像数据，确定第一错误区域；所述根据所述TOF数据和所述双目深度数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据，包括：根据所述TOF数据和所述双目稀疏深度数据，对所述第一错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
结合第一方面，在一种可能的实施方式中，所述根据所述TOF数据，对所述双目稀疏 深度数据的所述第一错误区域的深度进行修正，以确定所述被修正后的双目深度数据，包括：将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；将所述双目稀疏深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度；对所述第一错误区域和所述第一错误区域之外的区域的深度进行稠密化处理，以获得所述被修正后的双目深度数据。
结合第一方面，在一种可能的实施方式中，所述错误区域包括第一错误区域和第二错误区域；所述双目深度数据包括双目稀疏深度数据；所述根据所述双目图像数据和所述双目深度数据，确定错误区域，包括：所述根据所述TOF数据和所述双目深度数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据，包括：根据所述双目稀疏深度数据、所述第一图像数据、所述第二图像数据，确定所述第一错误区域；根据所述TOF数据和所述双目稀疏深度数据，对所述第一错误区域的深度进行修正，确定第一修正稠密深度数据；根据所述第一修正稠密深度数据、所述TOF数据、所述第一图像数据，确定所述第二错误区域；根据所述TOF数据和所述第一修正稠密深度数据，对所述第二错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
结合第一方面，在一种可能的实施方式中，所述根据所述TOF数据，对所述双目稀疏深度数据的所述第一错误区域的深度进行修正，以确定第一修正稠密深度数据，包括：将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；将所述双目稀疏深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度；对所述第一错误区域和所述第一错误区域之外的区域的深度进行稠密化处理，以获得所述第一修正稠密深度数据。
结合第一方面，在一种可能的实施方式中，所述根据所述TOF数据，对所述第一修正稠密深度数据的所述第二错误区域的深度进行修正，以确定所述被修正后的双目深度数据，包括：将所述TOF数据对应的所述第二错误区域的深度作为所述第二错误区域的深度；对所述第二错误区域的所述深度进行稠密化处理；将所述第一修正稠密深度数据对应的所述第二错误区域之外的区域的深度作为对应的所述第二错误区域之外的区域的深度，以获得所述被修正后的双目深度数据。
结合第一方面，在一种可能的实施方式中，所述第二错误区域包括所述第一修正稠密深度数据中的深度跳变区域。
结合第一方面，在一种可能的实施方式中，所述第一错误区域包括以下区域中至少一种：重复纹理区域、色差小于第一预设值的区域、深度渐变区域。
结合第一方面，在一种可能的实施方式中，在所述根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据之前，还包括：调整所述TOF数据和/或所述双目深度数据的深度，以使得所述TOF数据与所述双目数据的系统深度误差小于第二预设值。
结合第一方面，在一种可能的实施方式中，在所述根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据之前，还包括：调整所述TOF数据和/或所述双目深度数据的相对位置，以使得所述TOF数据与所述双目数据的系统位置误差小于第三预设值。
第二方面，提供一种图像处理装置，包括：获取模块，用于获取场景的双目图像数据，以及所述场景的飞行时间TOF数据，所述双目图像数据包括第一图像数据和第二图像数 据；确定模块，用于根据所述双目图像数据和所述双目深度数据，确定错误区域；所述双目深度数据是根据所述双目图像数据确定的；修正模块，用于根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据；虚化处理模块，用于根据所述被修正后的双目深度数据，对所述第一图像数据进行虚化处理。
结合第二方面，在一种可能的实施方式中，所述错误区域包括第一错误区域；所述双目深度数据包括双目稠密深度数据；所述确定模块用于，根据所述双目稠密深度数据和所述第一图像数据，确定所述第一错误区域；所述修正模块用于，根据所述TOF数据和所述双目稠密深度数据，对所述第一错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
结合第二方面，在一种可能的实施方式中，所述修正模块用于：将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；对所述第一错误区域的所述深度进行稠密化处理；将所述双目稠密深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度，以获得所述被修正后的双目深度数据。
结合第二方面，在一种可能的实施方式中，所述错误区域包括第一错误区域和第二错误区域；所述双目深度数据包括双目稠密深度数据；所述确定模块用于，根据所述双目稠密深度数据和所述第一图像数据，确定所述第一错误区域；所述修正模块用于，根据所述TOF数据和所述双目稠密深度数据，对所述第一错误区域的深度进行修正，以确定第一修正稠密深度数据；所述确定模块还用于，根据所述第一修正稠密深度数据、所述TOF数据、所述第一图像数据，确定所述第二错误区域；所述修正模块还用于，根据所述TOF数据和所述第一修正稠密深度数据，对所述第二错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
结合第二方面，在一种可能的实施方式中，所述修正模块用于：将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；对所述第一错误区域的所述深度进行稠密化处理；将所述双目稠密深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度，以获得所述第一修正稠密深度数据。
结合第二方面，在一种可能的实施方式中，所述错误区域包括第一错误区域；所述双目深度数据包括双目稀疏深度数据；所述确定模块用于，根据所述双目稀疏深度数据、所述第一图像数据和所述第二图像数据，确定所述第一错误区域；所述修正模块用于，根据所述TOF数据和所述双目稀疏深度数据，对所述第一错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
结合第二方面，在一种可能的实施方式中，所述修正模块用于：将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；将所述双目稀疏深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度；对所述第一错误区域和所述第一错误区域之外的区域的深度进行稠密化处理，以获得所述被修正后的双目深度数据。
结合第二方面，在一种可能的实施方式中，所述错误区域包括第一错误区域和第二错误区域；所述双目深度数据包括双目稀疏深度数据；所述确定模块用于，根据所述双目稀疏深度数据、所述第一图像数据、所述第二图像数据，确定所述第一错误区域；所述修正模块用于，根据所述TOF数据和所述双目稀疏深度数据，对所述第一错误区域的深度进行修正，确定第一修正稠密深度数据；所述确定模块还用于，根据所述第一修正稠密深度数 据、所述TOF数据、所述第一图像数据，确定所述第二错误区域；所述修正模块还用于，根据所述TOF数据和所述第一修正稠密深度数据，对所述第二错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
结合第二方面，在一种可能的实施方式中，所述修正模块用于：将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；将所述双目稀疏深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度；对所述第一错误区域和所述第一错误区域之外的区域的深度进行稠密化处理，以获得所述第一修正稠密深度数据。
结合第二方面，在一种可能的实施方式中，所述修正模块用于：将所述TOF数据对应的所述第二错误区域的深度作为所述第二错误区域的深度；对所述第二错误区域的所述深度进行稠密化处理；将所述第一修正稠密深度数据对应的所述第二错误区域之外的区域的深度作为对应的所述第二错误区域之外的区域的深度，以获得所述被修正后的双目深度数据。
结合第二方面，在一种可能的实施方式中，所述第二错误区域包括所述第一修正稠密深度数据中的深度跳变区域。
结合第二方面，在一种可能的实施方式中，所述第一错误区域包括以下区域中至少一种：重复纹理区域、色差小于第一预设值的区域或深度渐变区域。
结合第二方面，在一种可能的实施方式中，所述图像处理装置还包括第一调整模块，用于在所述根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据之前，调整所述TOF数据和/或所述双目深度数据的深度，以使得所述TOF数据与所述双目数据的系统深度误差小于第二预设值。
结合第二方面，在一种可能的实施方式中，所述图像处理装置还包括第二调整模块，用于在所述根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据之前，调整所述TOF数据和/或所述双目深度数据的相对位置，以使得所述TOF数据与所述双目数据的系统位置误差小于第三预设值。
第三方面，提供一种图像处理的方法，包括：获取场景的双目图像数据，以及所述场景的飞行时间TOF数据，所述双目图像数据包括根据不同摄像头得到的第一图像数据和第二图像数据；根据所述TOF数据修正所述双目深度数据以得到被修正后的双目深度数据，所述双目深度数据是根据所述双目图像数据确定的；根据所述被修正后的双目深度数据，对所述第一图像数据进行虚化处理，其中所述第一图像数据为待显示的图像数据。
通过上述方法，可以根据TOF数据对错误区域进行修正，有效避免双目深度估计方式的精度和稳定性较差的造成的对一些特定区域的深度估计错误。结合TOF数据精度和稳定性的优势，以及双目图像分辨率高的特点，实现精确、稳定、高分辨率的深度估计，从而减少甚至消除具有大光圈虚化效果的图像中虚化错误。
结合第三方面，在一种可能的实施方式中，所述根据所述TOF数据修正所述双目深度数据以得到被修正后的双目深度数据包括：根据所述双目图像数据和双目深度数据，确定错误区域；根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据。
结合第三方面，在一种可能的实施方式中，所述根据双目图像数据和双目深度数据， 确定错误区域的步骤，包括：根据所述双目深度数据，以及所述第一图像数据或所述第二图像数据，确定所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域。
结合第三方面，在一种可能的实施方式中，所述错误区域包括第一错误区域；所述双目深度数据包括双目稠密深度数据；所述根据所述双目深度数据，以及所述第一图像数据或所述第二图像数据，确定所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域，包括：根据所述双目稠密深度数据和所述第一图像数据，确定所述第一错误区域，所述第一错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；所述根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据，包括：根据所述TOF数据，对所述双目稠密深度数据的所述第一错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
结合第三方面，在一种可能的实施方式中，所述根据所述TOF数据，对所述双目稠密深度数据的所述第一错误区域的深度进行修正，以确定被修正后的双目深度数据，包括：将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；对所述第一错误区域的所述深度进行稠密化处理；将所述双目稠密深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度，以获得所述被修正后的双目深度数据。
结合第三方面，在一种可能的实施方式中，所述错误区域包括第一错误区域和第二错误区域；所述第二错误区域包括所述第一错误区域之外的部分区域；所述双目深度数据包括双目稠密深度数据；所述根据所述双目深度数据，以及所述第一图像数据或所述第二图像数据，确定所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域，包括：根据所述双目稠密深度数据和所述第一图像数据，确定所述第一错误区域，所述第一错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；所述根据TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据，包括：根据所述TOF数据，对所述双目稠密深度数据的所述第一错误区域的深度进行修正，以确定第一修正稠密深度数据；根据所述第一修正稠密深度数据、所述TOF数据、所述第一图像数据，确定所述第二错误区域，所述第二错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；根据所述TOF数据，对所述第一修正稠密深度数据的所述第二错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
结合第三方面，在一种可能的实施方式中，所述根据所述TOF数据，对所述双目稠密深度数据的所述第一错误区域的深度进行修正，以确定第一修正稠密深度数据，包括：将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；对所述第一错误区域的所述深度进行稠密化处理；将所述双目稠密深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度，以获得所述第一修正稠密深度数据。
结合第三方面，在一种可能的实施方式中，所述错误区域包括第一错误区域；所述双目深度数据包括双目稀疏深度数据；所述根据所述双目深度数据，以及所述第一图像数据或所述第二图像数据，确定所述双目深度数据中精度或稳定性中的至少一项不满足预设范 围的区域对应的所述第一图像数据的区域，包括：根据所述双目稀疏深度数据、所述第一图像数据和所述第二图像数据，确定所述第一错误区域，所述第一错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；所述根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据，包括：根据所述TOF数据，对所述双目稀疏深度数据的所述第一错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
结合第三方面，在一种可能的实施方式中，所述根据所述TOF数据，对所述双目稀疏深度数据的所述第一错误区域的深度进行修正，以确定所述被修正后的双目深度数据，包括：将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；将所述双目稀疏深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度；对所述第一错误区域和所述第一错误区域之外的区域的深度进行稠密化处理，以获得所述被修正后的双目深度数据。
结合第三方面，在一种可能的实施方式中，所述错误区域包括第一错误区域和第二错误区域；所述第二错误区域包括所述第一错误区域之外的部分区域；所述双目深度数据包括双目稀疏深度数据；所述根据所述双目深度数据，以及所述第一图像数据或所述第二图像数据，确定所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域，包括：根据所述双目稀疏深度数据、所述第一图像数据、所述第二图像数据，确定所述第一错误区域，所述第一错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；所述根据TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据，包括：根据所述TOF数据，对所述双目稀疏深度数据的所述第一错误区域的深度进行修正，以确定第一修正稠密深度数据；根据所述第一修正稠密深度数据、所述TOF数据、所述第一图像数据，确定所述第二错误区域，所述第二错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；根据所述TOF数据和所述第一修正稠密深度数据，对所述第二错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
结合第三方面，在一种可能的实施方式中，所述根据所述TOF数据，对所述双目稀疏深度数据的所述第一错误区域的深度进行修正，以确定第一修正稠密深度数据，包括：将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；将所述双目稀疏深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度；对所述第一错误区域和所述第一错误区域之外的区域的深度进行稠密化处理，以获得所述第一修正稠密深度数据。
结合第三方面，在一种可能的实施方式中，所述根据所述TOF数据，对所述第一修正稠密深度数据的所述第二错误区域的深度进行修正，以确定所述被修正后的双目深度数据，包括：将所述TOF数据对应的所述第二错误区域的深度作为所述第二错误区域的深度；对所述第二错误区域的所述深度进行稠密化处理；将所述第一修正稠密深度数据对应的所述第二错误区域之外的区域的深度作为对应的所述第二错误区域之外的区域的深度，以获得所述被修正后的双目深度数据。
结合第三方面，在一种可能的实施方式中，所述第二错误区域包括所述第一修正稠密深度数据中的深度跳变区域。
结合第三方面，在一种可能的实施方式中，所述第一错误区域包括以下区域中至少一种：重复纹理区域、色差小于第一预设值的区域或深度渐变区域。
结合第三方面，在一种可能的实施方式中，在所述根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据之前，还包括：调整所述TOF数据和/或所述双目深度数据的深度，以使得所述TOF数据与所述双目数据的系统深度误差小于第二预设值。
结合第三方面，在一种可能的实施方式中，在所述根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据之前，还包括：调整所述TOF数据和/或所述双目深度数据的相对位置，以使得所述TOF数据与所述双目数据的系统位置误差小于第三预设值。
第四方面，提供一种图像处理装置，包括：获取模块，用于获取场景的双目图像数据，以及所述场景的飞行时间TOF数据，所述双目图像数据包括根据不同摄像头得到的第一图像数据和第二图像数据；修正模块，用于根据所述TOF数据修正所述双目深度数据以得到被修正后的双目深度数据，所述双目深度数据是根据所述双目图像数据确定的；虚化处理模块，用于根据所述被修正后的双目深度数据，对所述第一图像数据进行虚化处理，其中所述第一图像数据为待显示的图像数据。
结合第四方面，在一种可能的实施方式中，所述修正模块包括确定单元，用于根据所述双目图像数据和双目深度数据，确定错误区域；修正单元，用于根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据。
结合第四方面，在一种可能的实施方式中，所述确定单元具体用于：根据所述双目深度数据，以及所述第一图像数据或所述第二图像数据，确定所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域。
结合第四方面，在一种可能的实施方式中，所述错误区域包括第一错误区域；所述双目深度数据包括双目稠密深度数据；所述确定单元具体用于，根据所述双目稠密深度数据和所述第一图像数据，确定所述第一错误区域，所述第一错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；所述修正单元用于，根据所述TOF数据，对所述双目稠密深度数据的所述第一错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
结合第四方面，在一种可能的实施方式中，所述修正单元用于：将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；对所述第一错误区域的所述深度进行稠密化处理；将所述双目稠密深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度，以获得所述被修正后的双目深度数据。
结合第四方面，在一种可能的实施方式中，所述错误区域包括第一错误区域和第二错误区域；所述第二错误区域包括所述第一错误区域之外的部分区域；所述双目深度数据包括双目稠密深度数据；所述确定单元用于，根据所述双目稠密深度数据和所述第一图像数据，确定所述第一错误区域，所述第一错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；所述修正单元用于，根据所述TOF数据，对所述双目稠密深度数据的所述第一错误区域的深度进行修正，以确定第一修正稠密深度数据；所述确定单元还用于，根据所述第一修正稠密深度数据、所述TOF数据、所述第一图像数据，确定所述第二错误区域，所述第二错误区域为所述双目深度数 据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；所述修正单元还用于，根据所述TOF数据，对所述第一修正稠密深度数据的所述第二错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
结合第四方面，在一种可能的实施方式中，所述修正单元用于：将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；对所述第一错误区域的所述深度进行稠密化处理；将所述双目稠密深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度，以获得所述第一修正稠密深度数据。
结合第四方面，在一种可能的实施方式中，所述错误区域包括第一错误区域；所述双目深度数据包括双目稀疏深度数据；所述确定单元用于，根据所述双目稀疏深度数据、所述第一图像数据和所述第二图像数据，确定所述第一错误区域，所述第一错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；所述修正单元用于，根据所述TOF数据，对所述双目稀疏深度数据的所述第一错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
结合第四方面，在一种可能的实施方式中，所述修正单元用于：将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；将所述双目稀疏深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度；对所述第一错误区域和所述第一错误区域之外的区域的深度进行稠密化处理，以获得所述被修正后的双目深度数据。
结合第四方面，在一种可能的实施方式中，所述错误区域包括第一错误区域和第二错误区域；所述第二错误区域包括所述第一错误区域之外的部分区域；所述双目深度数据包括双目稀疏深度数据；所述确定单元用于，根据所述双目稀疏深度数据、所述第一图像数据、所述第二图像数据，确定所述第一错误区域，所述第一错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；所述修正单元用于，根据所述TOF数据，对所述双目稀疏深度数据的所述第一错误区域的深度进行修正，确定第一修正稠密深度数据；所述确定单元还用于，根据所述第一修正稠密深度数据、所述TOF数据、所述第一图像数据，确定所述第二错误区域，所述第二错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；所述修正单元还用于，根据所述TOF数据，对所述第一修正稠密深度数据的所述第二错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
结合第四方面，在一种可能的实施方式中，所述修正单元用于：将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；将所述双目稀疏深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度；对所述第一错误区域和所述第一错误区域之外的区域的深度进行稠密化处理，以获得所述第一修正稠密深度数据。
结合第四方面，在一种可能的实施方式中，所述修正单元用于：将所述TOF数据对应的所述第二错误区域的深度作为所述第二错误区域的深度；对所述第二错误区域的所述深度进行稠密化处理；将所述第一修正稠密深度数据对应的所述第二错误区域之外的区域的深度作为对应的所述第二错误区域之外的区域的深度，以获得所述被修正后的双目深度数据。
结合第四方面，在一种可能的实施方式中，所述第二错误区域包括所述第一修正稠密 深度数据中的深度跳变区域。
结合第四方面，在一种可能的实施方式中，所述第一错误区域包括以下区域中至少一种：重复纹理区域、色差小于第一预设值的区域或深度渐变区域。
结合第四方面，在一种可能的实施方式中，还包括：第一调整模块，用于在所述根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据之前，调整所述TOF数据和/或所述双目深度数据的深度，以使得所述TOF数据与所述双目数据的系统深度误差小于第二预设值。
结合第四方面，在一种可能的实施方式中，还包括：第二调整模块，用于在所述根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据之前，调整所述TOF数据和/或所述双目深度数据的相对位置，以使得所述TOF数据与所述双目数据的系统位置误差小于第三预设值。
第五方面，提供一种图像处理装置，包括：存储器，用于存储代码；处理器，用于读取所述存储器中的代码，以执行如上述第一方面或上述第三方面及其可能的实施方式中的方法。
第六方面，提供一种计算机程序存储介质，所述计算机程序存储介质具有程序指令，当所述程序指令被执行时，使得如上述第一方面或上述第三方面及其可能的实施方式中的方法被执行。
第七方面，提供一种终端设备，所述芯片包括：双目系统，用于采集双目图像数据；飞行时间TOF器件，用于采集TOF数据；至少一个处理器，当程序指令被所述至少一个处理器中执行时，使得如上述第一方面或上述第三方面及其可能的实施方式中的方法被执行。
第八方面，提供一种终端设备，包括如上述第二方面或上述第四方面及其可能的实施方式中的装置。
附图说明
图1是一种终端设备的示意性结构图。
图2是一种终端设备的软件结构框图。
图3是另一种终端设备的示意性结构图。
图4是一种基于双目系统的深度计算的示意性流程图。
图5是一种基于深度的图像虚化方法的流程和图像的示意图。
图6是主摄像头采集的图像画面的一例的示意图。
图7是主摄像头采集的图像画面的另一例的示意图。
图8是一种双目系统深度计算的方法的示意性流程图。
图9是本申请一个实施例提供的一种图像处理的方法的示意性流程图。
图10是本申请另一个实施例提供的一种图像处理的方法的示意性流程图。
图11是本申请另一个实施例提供的一种图像处理的方法的示意性流程图。
图12是本申请另一个实施例提供的一种图像处理的方法的示意性流程图。
图13是本申请一个实施例提出的一种深度融合的方法的示意性流程图。
图14是一种光流神经网络(flownet)的运算的示意图。
图15是一种编码-解码式网络的示意性结构图。
图16是本申请又一个实施例提出的一种图像处理的方法的示意性流程图。
图17是本申请又一个实施例提出的一种图像处理的方法的示意性流程图。
图18是本申请又一个实施例提出的一种图像处理的方法的示意性流程图。
图19是本申请又一个实施例提出的一种图像处理的方法的示意性流程图。
图20是本申请一个实施例提出的一种图像处理的装置的示意性结构图。
图21是本申请一个实施例提出的一种图像处理的装置的示意性结构图。
图22是本申请一个实施例提出的一种终端设备的示意性结构图。
具体实施方式
下面将结合本发明实施例中的副图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
大光圈虚化是一种图像采集单元的特性。大光圈虚化效果是一种特殊的拍照效果，用户选定目标物体后，目标物体和同目标物体相同属性(如相同深度层次等)的物体保持清晰，其余物体被模糊。该效果的出发点是模拟单反相机效果，保持对焦物体清晰，非对焦物体模糊。
目前，市面上大部分智能终端设备都具备了大光圈虚化的特性效果，而该效果的实现主要是基于双目系统(dual camera)。双目系统包含两个图像采集单元，在进行取景时，两个图像采集单元同时会采集图像。
图1是一种电子设备100的结构示意图。电子设备110可以是终端设备。
电子设备100可以包括处理器110，外部存储器接口120，内部存储器121，通用串行总线(universal serial bus，USB)接口130，充电管理模块140，电源管理模块141，电池142，天线1，天线2，移动通信模块150，无线通信模块160，音频模块170，扬声器170A，受话器170B，麦克风170C，耳机接口170D，传感器模块180，按键190，马达191，指示器192，摄像头193，显示屏194，以及用户标识模块(subscriber identification module，SIM)卡接口195等。其中传感器模块180可以包括压力传感器180A，陀螺仪传感器180B，气压传感器180C，磁传感器180D，加速度传感器180E，距离传感器180F，接近光传感器180G，指纹传感器180H，温度传感器180J，触摸传感器180K，环境光传感器180L，骨传导传感器180M，飞行时间(time of flight，TOF)传感器180N等。
可以理解的是，本发明实施例示意的结构并不构成对电子设备100的具体限定。在本申请另一些实施例中，电子设备100可以包括比图示更多或更少的部件，或者组合某些部件，或者拆分某些部件，或者不同的部件布置。图示的部件可以以硬件，软件或软件和硬件的组合实现。
处理器110可以包括一个或多个处理单元，例如：处理器110可以包括应用处理器(application processor，AP)，调制解调处理器，图形处理器(graphics processing unit，GPU)，图像信号处理器(image signal processor，ISP)，控制器，视频编解码器，数字信号处理器(digital signal processor，DSP)，基带处理器，和/或神经网络处理器(neural-network processing unit，NPU)等。其中，不同的处理单元可以是独立的器件，也可以集成在一个或多个处理器中。
控制器可以根据指令操作码和时序信号，产生操作控制信号，完成取指令和执行指令的控制。
处理器110中还可以设置存储器，用于存储指令和数据。在一些实施例中，处理器110中的存储器为高速缓冲存储器。该存储器可以保存处理器110刚用过或循环使用的指令或数据。如果处理器110需要再次使用该指令或数据，可从所述存储器中直接调用。避免了重复存取，减少了处理器110的等待时间，因而提高了系统的效率。
在一些实施例中，处理器110可以包括一个或多个接口。接口可以包括集成电路(inter-integrated circuit，I2C)接口，集成电路内置音频(inter-integrated circuit sound，I2S)接口，脉冲编码调制(pulse code modulation，PCM)接口，通用异步收发传输器(universal asynchronous receiver/transmitter，UART)接口，移动产业处理器接口(mobile industry processor interface，MIPI)，通用输入输出(general-purpose input/output，GPIO)接口，用户标识模块(subscriber identity module，SIM)接口，和/或通用串行总线(universal serial bus，USB)接口等。
I2C接口是一种双向同步串行总线，包括一根串行数据线(serial data line，SDA)和一根串行时钟线(derail clock line，SCL)。在一些实施例中，处理器110可以包含多组I2C总线。处理器110可以通过不同的I2C总线接口分别耦合触摸传感器180K，充电器，闪光灯，摄像头193等。例如：处理器110可以通过I2C接口耦合触摸传感器180K，使处理器110与触摸传感器180K通过I2C总线接口通信，实现电子设备100的触摸功能。
I2S接口可以用于音频通信。在一些实施例中，处理器110可以包含多组I2S总线。处理器110可以通过I2S总线与音频模块170耦合，实现处理器110与音频模块170之间的通信。在一些实施例中，音频模块170可以通过I2S接口向无线通信模块160传递音频信号，实现通过蓝牙耳机接听电话的功能。
PCM接口也可以用于音频通信，将模拟信号抽样，量化和编码。在一些实施例中，音频模块170与无线通信模块160可以通过PCM总线接口耦合。在一些实施例中，音频模块170也可以通过PCM接口向无线通信模块160传递音频信号，实现通过蓝牙耳机接听电话的功能。所述I2S接口和所述PCM接口都可以用于音频通信。
UART接口是一种通用串行数据总线，用于异步通信。该总线可以为双向通信总线。它将要传输的数据在串行通信与并行通信之间转换。在一些实施例中，UART接口通常被用于连接处理器110与无线通信模块160。例如：处理器110通过UART接口与无线通信模块160中的蓝牙模块通信，实现蓝牙功能。在一些实施例中，音频模块170可以通过UART接口向无线通信模块160传递音频信号，实现通过蓝牙耳机播放音乐的功能。
MIPI接口可以被用于连接处理器110与显示屏194，摄像头193等外围器件。MIPI接口包括摄像头串行接口(camera serial interface，CSI)，显示屏串行接口(display serial interface，DSI)等。在一些实施例中，处理器110和摄像头193通过CSI接口通信，实现电子设备100的拍摄功能。处理器110和显示屏194通过DSI接口通信，实现电子设备100的显示功能。
GPIO接口可以通过软件配置。GPIO接口可以被配置为控制信号，也可被配置为数据信号。在一些实施例中，GPIO接口可以用于连接处理器110与摄像头193，显示屏194，无线通信模块160，音频模块170，传感器模块180等。GPIO接口还可以被配置为I2C接口，I2S接口，UART接口，MIPI接口等。
USB接口130是符合USB标准规范的接口，具体可以是Mini USB接口，Micro USB接口，USB Type C接口等。USB接口130可以用于连接充电器为电子设备100充电，也可以用于电子设备100与外围设备之间传输数据。也可以用于连接耳机，通过耳机播放音频。该接口还可以用于连接其他电子设备，例如AR设备等。
可以理解的是，本发明实施例示意的各模块间的接口连接关系，只是示意性说明，并不构成对电子设备100的结构限定。在本申请另一些实施例中，电子设备100也可以采用上述实施例中不同的接口连接方式，或多种接口连接方式的组合。
充电管理模块140用于从充电器接收充电输入。其中，充电器可以是无线充电器，也可以是有线充电器。在一些有线充电的实施例中，充电管理模块140可以通过USB接口130接收有线充电器的充电输入。在一些无线充电的实施例中，充电管理模块140可以通过电子设备100的无线充电线圈接收无线充电输入。充电管理模块140为电池142充电的同时，还可以通过电源管理模块141为电子设备供电。
电源管理模块141用于连接电池142，充电管理模块140与处理器110。电源管理模块141接收电池142和/或充电管理模块140的输入，为处理器110，内部存储器121，显示屏194，摄像头193，和无线通信模块160等供电。电源管理模块141还可以用于监测电池容量，电池循环次数，电池健康状态(漏电，阻抗)等参数。在其他一些实施例中，电源管理模块141也可以设置于处理器110中。在另一些实施例中，电源管理模块141和充电管理模块140也可以设置于同一个器件中。
电子设备100的无线通信功能可以通过天线1，天线2，移动通信模块150，无线通信模块160，调制解调处理器以及基带处理器等实现。
天线1和天线2用于发射和接收电磁波信号。电子设备100中的每个天线可用于覆盖单个或多个通信频带。不同的天线还可以复用，以提高天线的利用率。例如：可以将天线1复用为无线局域网的分集天线。在另外一些实施例中，天线可以和调谐开关结合使用。
移动通信模块150可以提供应用在电子设备100上的包括2G/3G/4G/5G等无线通信的解决方案。移动通信模块150可以包括至少一个滤波器，开关，功率放大器，低噪声放大器(low noise amplifier，LNA)等。移动通信模块150可以由天线1接收电磁波，并对接收的电磁波进行滤波，放大等处理，传送至调制解调处理器进行解调。移动通信模块150还可以对经调制解调处理器调制后的信号放大，经天线1转为电磁波辐射出去。在一些实施例中，移动通信模块150的至少部分功能模块可以被设置于处理器110中。在一些实施例中，移动通信模块150的至少部分功能模块可以与处理器110的至少部分模块被设置在同一个器件中。
调制解调处理器可以包括调制器和解调器。其中，调制器用于将待发送的低频基带信号调制成中高频信号。解调器用于将接收的电磁波信号解调为低频基带信号。随后解调器将解调得到的低频基带信号传送至基带处理器处理。低频基带信号经基带处理器处理后，被传递给应用处理器。应用处理器通过音频设备(不限于扬声器170A，受话器170B等)输出声音信号，或通过显示屏194显示图像或视频。在一些实施例中，调制解调处理器可以是独立的器件。在另一些实施例中，调制解调处理器可以独立于处理器110，与移动通信模块150或其他功能模块设置在同一个器件中。
无线通信模块160可以提供应用在电子设备100上的包括无线局域网(wireless local area networks，WLAN)(如无线保真(wireless fidelity，Wi-Fi)网络)，蓝牙(bluetooth，BT)， 全球导航卫星系统(global navigation satellite system，GNSS)，调频(frequency modulation，FM)，近距离无线通信技术(near field communication，NFC)，红外技术(infrared，IR)等无线通信的解决方案。无线通信模块160可以是集成至少一个通信处理模块的一个或多个器件。无线通信模块160经由天线2接收电磁波，将电磁波信号调频以及滤波处理，将处理后的信号发送到处理器110。无线通信模块160还可以从处理器110接收待发送的信号，对其进行调频，放大，经天线2转为电磁波辐射出去。
在一些实施例中，电子设备100的天线1和移动通信模块150耦合，天线2和无线通信模块160耦合，使得电子设备100可以通过无线通信技术与网络以及其他设备通信。所述无线通信技术可以包括全球移动通讯系统(global system for mobile communications，GSM)，通用分组无线服务(general packet radio service，GPRS)，码分多址接入(code division multiple access，CDMA)，宽带码分多址(wideband code division multiple access，WCDMA)，时分码分多址(time-division code division multiple access，TD-SCDMA)，长期演进(long term evolution，LTE)，BT，GNSS，WLAN，NFC，FM，和/或IR技术等。所述GNSS可以包括全球卫星定位系统(global positioning system，GPS)，全球导航卫星系统(global navigation satellite system，GLONASS)，北斗卫星导航系统(beidou navigation satellite system，BDS)，准天顶卫星系统(quasi-zenith satellite system，QZSS)和/或星基增强系统(satellite based augmentation systems，SBAS)。
电子设备100通过GPU，显示屏194，以及应用处理器等实现显示功能。GPU为图像处理的微处理器，连接显示屏194和应用处理器。GPU用于执行数学和几何计算，用于图形渲染。处理器110可包括一个或多个GPU，其执行程序指令以生成或改变显示信息。
显示屏194用于显示图像，视频等。显示屏194包括显示面板。显示面板可以采用液晶显示屏(liquid crystal display，LCD)，有机发光二极管(organic light-emitting diode，OLED)，有源矩阵有机发光二极体或主动矩阵有机发光二极体(active-matrix organic light emitting diode的，AMOLED)，柔性发光二极管(flex light-emitting diode，FLED)，Miniled，MicroLed，Micro-oLed，量子点发光二极管(quantum dot light emitting diodes，QLED)等。在一些实施例中，电子设备100可以包括1个或N个显示屏194，N为大于1的正整数。
电子设备100可以通过ISP，摄像头193，视频编解码器，GPU，显示屏194以及应用处理器等实现拍摄功能。
ISP用于处理摄像头193反馈的数据。例如，拍照时，打开快门，光线通过镜头被传递到摄像头感光元件上，光信号转换为电信号，摄像头感光元件将所述电信号传递给ISP处理，转化为肉眼可见的图像。ISP还可以对图像的噪点，亮度，肤色进行算法优化。ISP还可以对拍摄场景的曝光，色温等参数优化。在一些实施例中，ISP可以设置在摄像头193中。
摄像头193用于捕获静态图像或视频。物体通过镜头生成光学图像投射到感光元件。感光元件可以是电荷耦合器件(charge coupled device，CCD)或互补金属氧化物半导体(complementary metal-oxide-semiconductor，CMOS)光电晶体管。感光元件把光信号转换成电信号，之后将电信号传递给ISP转换成数字图像信号。ISP将数字图像信号输出到DSP加工处理。DSP将数字图像信号转换成标准的RGB，YUV等格式的图像信号。在一些实施例中，电子设备100可以包括1个或N个摄像头193，N为大于1的正整数。例如，电子设备100可以包括双目系统。双目系统可以包括两个摄像头。双目系统中的两个摄像头 均可以用于采集图像数据。也就是说，双目系统中的两个摄像头均可以用于捕获静态图像或视频。
数字信号处理器用于处理数字信号，除了可以处理数字图像信号，还可以处理其他数字信号。例如，当电子设备100在频点选择时，数字信号处理器用于对频点能量进行傅里叶变换等。
视频编解码器用于对数字视频压缩或解压缩。电子设备100可以支持一种或多种视频编解码器。这样，电子设备100可以播放或录制多种编码格式的视频，例如：动态图像专家组(moving picture experts group，MPEG)1，MPEG2，MPEG3，MPEG4等。
NPU为神经网络(neural-network，NN)计算处理器，通过借鉴生物神经网络结构，例如借鉴人脑神经元之间传递模式，对输入信息快速处理，还可以不断的自学习。通过NPU可以实现电子设备100的智能认知等应用，例如：图像识别，人脸识别，语音识别，文本理解等。
外部存储器接口120可以用于连接外部存储卡，例如Micro SD卡，实现扩展电子设备100的存储能力。外部存储卡通过外部存储器接口120与处理器110通信，实现数据存储功能。例如将音乐，视频等文件保存在外部存储卡中。
内部存储器121可以用于存储计算机可执行程序代码，所述可执行程序代码包括指令。内部存储器121可以包括存储程序区和存储数据区。其中，存储程序区可存储操作系统，至少一个功能所需的应用程序(比如声音播放功能，图像播放功能等)等。存储数据区可存储电子设备100使用过程中所创建的数据(比如音频数据，电话本等)等。此外，内部存储器121可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件，闪存器件，通用闪存存储器(universal flash storage，UFS)等。处理器110通过运行存储在内部存储器121的指令，和/或存储在设置于处理器中的存储器的指令，执行电子设备100的各种功能应用以及数据处理。
电子设备100可以通过音频模块170，扬声器170A，受话器170B，麦克风170C，耳机接口170D，以及应用处理器等实现音频功能。例如音乐播放，录音等。
音频模块170用于将数字音频信息转换成模拟音频信号输出，也用于将模拟音频输入转换为数字音频信号。音频模块170还可以用于对音频信号编码和解码。在一些实施例中，音频模块170可以设置于处理器110中，或将音频模块170的部分功能模块设置于处理器110中。
扬声器170A，也称“喇叭”，用于将音频电信号转换为声音信号。电子设备100可以通过扬声器170A收听音乐，或收听免提通话。
受话器170B，也称“听筒”，用于将音频电信号转换成声音信号。当电子设备100接听电话或语音信息时，可以通过将受话器170B靠近人耳接听语音。
麦克风170C，也称“话筒”，“传声器”，用于将声音信号转换为电信号。当拨打电话或发送语音信息时，用户可以通过人嘴靠近麦克风170C发声，将声音信号输入到麦克风170C。电子设备100可以设置至少一个麦克风170C。在另一些实施例中，电子设备100可以设置两个麦克风170C，除了采集声音信号，还可以实现降噪功能。在另一些实施例中，电子设备100还可以设置三个，四个或更多麦克风170C，实现采集声音信号，降噪，还可以识别声音来源，实现定向录音功能等。
耳机接口170D用于连接有线耳机。耳机接口170D可以是USB接口130，也可以是 3.5mm的开放移动电子设备平台(open mobile terminal platform，OMTP)标准接口，美国蜂窝电信工业协会(cellular telecommunications industry association of the USA，CTIA)标准接口。
压力传感器180A用于感受压力信号，可以将压力信号转换成电信号。在一些实施例中，压力传感器180A可以设置于显示屏194。压力传感器180A
的种类很多，如电阻式压力传感器，电感式压力传感器，电容式压力传感器等。电容式压力传感器可以是包括至少两个具有导电材料的平行板。当有力作用于压力传感器180A，电极之间的电容改变。电子设备100根据电容的变化确定压力的强度。当有触摸操作作用于显示屏194，电子设备100根据压力传感器180A检测所述触摸操作强度。电子设备100也可以根据压力传感器180A的检测信号计算触摸的位置。在一些实施例中，作用于相同触摸位置，但不同触摸操作强度的触摸操作，可以对应不同的操作指令。例如：当有触摸操作强度小于第一压力阈值的触摸操作作用于短消息应用图标时，执行查看短消息的指令。当有触摸操作强度大于或等于第一压力阈值的触摸操作作用于短消息应用图标时，执行新建短消息的指令。
陀螺仪传感器180B可以用于确定电子设备100的运动姿态。在一些实施例中，可以通过陀螺仪传感器180B确定电子设备100围绕三个轴(即，x，y和z轴)的角速度。陀螺仪传感器180B可以用于拍摄防抖。示例性的，当按下快门，陀螺仪传感器180B检测电子设备100抖动的角度，根据角度计算出镜头模组需要补偿的距离，让镜头通过反向运动抵消电子设备100的抖动，实现防抖。陀螺仪传感器180B还可以用于导航，体感游戏场景。
气压传感器180C用于测量气压。在一些实施例中，电子设备100通过气压传感器180C测得的气压值计算海拔高度，辅助定位和导航。
磁传感器180D包括霍尔传感器。电子设备100可以利用磁传感器180D检测翻盖皮套的开合。在一些实施例中，当电子设备100是翻盖机时，电子设备100可以根据磁传感器180D检测翻盖的开合。进而根据检测到的皮套的开合状态或翻盖的开合状态，设置翻盖自动解锁等特性。
加速度传感器180E可检测电子设备100在各个方向上(一般为三轴)加速度的大小。当电子设备100静止时可检测出重力的大小及方向。还可以用于识别电子设备姿态，应用于横竖屏切换，计步器等应用。
距离传感器180F，用于测量距离。电子设备100可以通过红外或激光测量距离。在一些实施例中，拍摄场景，电子设备100可以利用距离传感器180F测距以实现快速对焦。
接近光传感器180G可以包括例如发光二极管(LED)和光检测器，例如光电二极管。发光二极管可以是红外发光二极管。电子设备100通过发光二极管向外发射红外光。电子设备100使用光电二极管检测来自附近物体的红外反射光。当检测到充分的反射光时，可以确定电子设备100附近有物体。当检测到不充分的反射光时，电子设备100可以确定电子设备100附近没有物体。电子设备100可以利用接近光传感器180G检测用户手持电子设备100贴近耳朵通话，以便自动熄灭屏幕达到省电的目的。接近光传感器180G也可用于皮套模式，口袋模式自动解锁与锁屏。
环境光传感器180L用于感知环境光亮度。电子设备100可以根据感知的环境光亮度自适应调节显示屏194亮度。环境光传感器180L也可用于拍照时自动调节白平衡。环境光传感器180L还可以与接近光传感器180G配合，检测电子设备100是否在口袋里，以防 误触。
指纹传感器180H用于采集指纹。电子设备100可以利用采集的指纹特性实现指纹解锁，访问应用锁，指纹拍照，指纹接听来电等。
温度传感器180J用于检测温度。在一些实施例中，电子设备100利用温度传感器180J检测的温度，执行温度处理策略。例如，当温度传感器180J上报的温度超过阈值，电子设备100执行降低位于温度传感器180J附近的处理器的性能，以便降低功耗实施热保护。在另一些实施例中，当温度低于另一阈值时，电子设备100对电池142加热，以避免低温导致电子设备100异常关机。在其他一些实施例中，当温度低于又一阈值时，电子设备100对电池142的输出电压执行升压，以避免低温导致的异常关机。
触摸传感器180K，也称“触控器件”。触摸传感器180K可以设置于显示屏194，由触摸传感器180K与显示屏194组成触摸屏，也称“触控屏”。触摸传感器180K用于检测作用于其上或附近的触摸操作。触摸传感器可以将检测到的触摸操作传递给应用处理器，以确定触摸事件类型。可以通过显示屏194提供与触摸操作相关的视觉输出。在另一些实施例中，触摸传感器180K也可以设置于电子设备100的表面，与显示屏194所处的位置不同。
骨传导传感器180M可以获取振动信号。在一些实施例中，骨传导传感器180M可以获取人体声部振动骨块的振动信号。骨传导传感器180M也可以接触人体脉搏，接收血压跳动信号。在一些实施例中，骨传导传感器180M也可以设置于耳机中，结合成骨传导耳机。音频模块170可以基于所述骨传导传感器180M获取的声部振动骨块的振动信号，解析出语音信号，实现语音功能。应用处理器可以基于所述骨传导传感器180M获取的血压跳动信号解析心率信息，实现心率检测功能。
TOF传感器180N可以包括发射模组和接收模组。TOF传感器180N的发射模组可以发送光脉冲，TOF传感器180N的接收模组可以接收物体反射的光。可以通过检测光脉冲的飞行(往返)时间获得物体与TOF传感器180N的距离。TOF传感器180N的发射模组发出的光信号遇物体后反射，TOF传感器180N的接收模组可以接收物体反射的光信号。通过计算TOF传感器180N发射和接收光信号的时间差或相位差，可以计算被拍摄景物的距离，以产生深度信息。
按键190包括开机键，音量键等。按键190可以是机械按键。也可以是触摸式按键。电子设备100可以接收按键输入，产生与电子设备100的用户设置以及功能控制有关的键信号输入。
马达191可以产生振动提示。马达191可以用于来电振动提示，也可以用于触摸振动反馈。例如，作用于不同应用(例如拍照，音频播放等)的触摸操作，可以对应不同的振动反馈效果。作用于显示屏194不同区域的触摸操作，马达191也可对应不同的振动反馈效果。不同的应用场景(例如：时间提醒，接收信息，闹钟，游戏等)也可以对应不同的振动反馈效果。触摸振动反馈效果还可以支持自定义。
指示器192可以是指示灯，可以用于指示充电状态，电量变化，也可以用于指示消息，未接来电，通知等。
SIM卡接口195用于连接SIM卡。SIM卡可以通过插入SIM卡接口195，或从SIM卡接口195拔出，实现和电子设备100的接触和分离。电子设备100可以支持1个或N个SIM卡接口，N为大于1的正整数。SIM卡接口195可以支持Nano SIM卡，Micro SIM卡， SIM卡等。同一个SIM卡接口195可以同时插入多张卡。所述多张卡的类型可以相同，也可以不同。SIM卡接口195也可以兼容不同类型的SIM卡。SIM卡接口195也可以兼容外部存储卡。电子设备100通过SIM卡和网络交互，实现通话以及数据通信等功能。在一些实施例中，电子设备100采用eSIM，即：嵌入式SIM卡。eSIM卡可以嵌在电子设备100中，不能和电子设备100分离。
电子设备100的软件系统可以采用分层架构，事件驱动架构，微核架构，微服务架构，或云架构。本发明实施例以分层架构的Android系统为例，示例性说明电子设备100的软件结构。
图2是本申请实施例的电子设备100的软件结构框图。
分层架构将软件分成若干个层，每一层都有清晰的角色和分工。层与层之间通过软件接口通信。在一些实施例中，将Android系统分为四层，从上至下分别为应用程序层，应用程序框架层，安卓运行时(Android runtime)和系统库，以及内核层。
应用程序层可以包括一系列应用程序包。
如图2所示，应用程序包可以包括相机，图库，日历，通话，地图，导航，WLAN，蓝牙，音乐，视频，短信息等应用程序。
应用程序框架层为应用程序层的应用程序提供应用编程接口(application programming interface，API)和编程框架。应用程序框架层包括一些预先定义的函数。
如图2所示，应用程序框架层可以包括窗口管理器，内容提供器，视图系统，电话管理器，资源管理器，通知管理器等。
窗口管理器用于管理窗口程序。窗口管理器可以获取显示屏大小，判断是否有状态栏，锁定屏幕，截取屏幕等。
内容提供器用来存放和获取数据，并使这些数据可以被应用程序访问。所述数据可以包括视频，图像，音频，拨打和接听的电话，浏览历史和书签，电话簿等。
视图系统包括可视控件，例如显示文字的控件，显示图片的控件等。视图系统可用于构建应用程序。显示界面可以由一个或多个视图组成的。例如，包括短信通知图标的显示界面，可以包括显示文字的视图以及显示图片的视图。
电话管理器用于提供电子设备100的通信功能。例如通话状态的管理(包括接通，挂断等)。
资源管理器为应用程序提供各种资源，比如本地化字符串，图标，图片，布局文件，视频文件等等。
通知管理器使应用程序可以在状态栏中显示通知信息，可以用于传达告知类型的消息，可以短暂停留后自动消失，无需用户交互。比如通知管理器被用于告知下载完成，消息提醒等。通知管理器还可以是以图表或者滚动条文本形式出现在系统顶部状态栏的通知，例如后台运行的应用程序的通知，还可以是以对话窗口形式出现在屏幕上的通知。例如在状态栏提示文本信息，发出提示音，电子设备振动，指示灯闪烁等。
Android Runtime包括核心库和虚拟机。Android runtime负责安卓系统的调度和管理。
核心库包含两部分：一部分是java语言需要调用的功能函数，另一部分是安卓的核心库。
应用程序层和应用程序框架层运行在虚拟机中。虚拟机将应用程序层和应用程序框架层的java文件执行为二进制文件。虚拟机用于执行对象生命周期的管理，堆栈管理，线程 管理，安全和异常的管理，以及垃圾回收等功能。
系统库可以包括多个功能模块。例如：表面管理器(surface manager)，媒体库(Media Libraries)，三维图形处理库(例如：OpenGL ES)，2D图形引擎(例如：SGL)等。
表面管理器用于对显示子系统进行管理，并且为多个应用程序提供了2D和3D图层的融合。
媒体库支持多种常用的音频，视频格式回放和录制，以及静态图像文件等。媒体库可以支持多种音视频编码格式，例如:MPEG4，H.264，MP3，AAC，AMR，JPG，PNG等。
三维图形处理库用于实现三维图形绘图，图像渲染，合成，和图层处理等。
2D图形引擎是2D绘图的绘图引擎。
内核层是硬件和软件之间的层。内核层至少包含显示驱动，摄像头驱动，音频驱动，传感器驱动。
下面结合捕获拍照场景，示例性说明电子设备100软件以及硬件的工作流程。
当触摸传感器180K接收到触摸操作，相应的硬件中断被发给内核层。内核层将触摸操作加工成原始输入事件(包括触摸坐标，触摸操作的时间戳等信息)。原始输入事件被存储在内核层。应用程序框架层从内核层获取原始输入事件，识别该输入事件所对应的控件。以该触摸操作是触摸单击操作，该单击操作所对应的控件为相机应用图标的控件为例，相机应用调用应用框架层的接口，启动相机应用，进而通过调用内核层启动摄像头驱动，通过摄像头193捕获静态图像或视频。
图3是一种终端设备300的示意性结构图。终端设备300的结构可以参考图1。终端设备300可以包括比电子设备100更多或更少的部件。终端设备300的软件系统可以采用分层架构，事件驱动架构，微核架构，微服务架构，或云架构。终端设备300的软件系统可以参考图2。
终端设备300可以双目系统310。双目系统310可以包括两个摄像头193，即主摄像头和副摄像头。双目系统有时也称为双摄像头系统，双目系统中的两个摄像头均可以用于采集图像数据。两个摄像头采集的图像数据可以用于实现相同或不同的功能。例如，其中一个摄像头采集的图像可以用于终端的图像显示，该摄像头可以称为主摄像头，或主摄；另外一个摄像头采集的图像可以用于计算深度或实现其他功能，该摄像头可以称为副摄像头。双目系统可以利用双摄像头估计物体深度，然后根据深度信息对全清晰图进行虚化处理，达成最终的大光圈虚化效果。
终端设备300可以包括TOF器件320。TOF器件320也可以称为TOF系统、TOF传感器等。TOF器件320可以包括发射模组和接收模组。TOF器件320的发射模组可以发送光脉冲，TOF器件320的接收模组可以接收物体反射的光。可以通过检测光脉冲的飞行(往返)时间获得物体与TOF器件320的距离。该距离也可以称为深度。
终端设备300可以包括闪光灯330。闪光灯330又称电子闪光灯，高速闪光灯。闪光灯330通过电容器存储高压电，脉冲触发使闪光管放电，完成瞬间闪光。在昏暗的地方，通过闪光灯可以使得景物更明亮。
终端设备300还可以包括长焦摄像头(未示出)。
双目系统310与TOF器件320可以临近布置。主摄像头和副摄像头可以与终端设备300的边沿340平行布置。TOF器件320可以位于双目系统310中的两个摄像头之间，也可以位于双目系统310中的主摄像头和副摄像头周围的其他位置。TOF器件320中的接收 器与主摄像头可以平行于边沿340或垂直于边沿340布置。可以理解，本申请实施例中的位置关系，平行可以包括近似平行，垂直可以包括近似垂直。
终端设备300可以包括处理器110。处理器110可以用于确定双目稠密深度数据。处理器110可以包括一个或多个处理单元。例如，处理器可以包括中央处理器(central processing unit，CPU)、神经网络处理器(neural-network processing unit，NPU)、图像信号处理器(image signal processor，ISP)、数字信号处理器(digital signal processor，DSP)等。处理器也可以用于确定双目稀疏深度数据。
终端设备300可以包括专用深度图应用流水线(depth map application pipeline，DMAP)芯片，DMAP芯片可以用于确定双目稀疏深度图。DMAP芯片可以包括用于确定双目稀疏深度图的处理器。
终端设备也可称为用户设备。终端设备可以经接入网设备与一个或多个核心网(core network，CN)进行通信。终端设备有时也可称为接入终端、终端、用户单元、用户站、移动站、移动台、远方站、远程终端、移动设备、用户终端、无线网络设备、用户代理或用户装置。用户设备可以是蜂窝电话、无绳电话、会话启动协议(session initiation protocol，SIP)电话、无线本地环路(wireless localloop，WLL)站、个人数字处理(personal digital assistant，PDA)、具有无线通信功能的手持设备、计算设备或连接到无线调制解调器的其它设备、车载设备、可穿戴设备或物联网、车联网中的终端设备以及未来网络中的任意形态的用户设备等。
图4是一种基于双目系统的深度计算的示意性流程图。
在步骤S401，获取校正参数(calibration parameter)。校正参数可以是双目系统的内参数和外参数。
在步骤S402，根据校正参数，对双目图像数据进行图像矫正(rectification)。将图像置于一个坐标系中，矫正可以是改变图像的坐标点。
在步骤S403，进行深度计算(depth calculation)，确定双目深度数据。
在步骤S404，通过深度优化(depth optimization)，确定双目优化深度数据。经过步骤S404，可以对步骤S403得到的深度计算的结果进行优化，得到双目优化深度数据。根据双目优化深度数据，可以实现大光圈效果，也就是焦外成像。
在一些实施例中，在步骤S403之后，还可以进行步骤S405。
在步骤S405，根据双目深度数据，进行自动对焦(automatic focus，AF)确定(determination)。双目图像数据可以是双目系统进行图像采集得到的。也可以是一个采集单元对某一场景在两个距离较近位置进行两次图像采集得到的。
图5是一种基于深度的图像虚化方法的流程和图像的示意图。
在步骤S501，主摄拍摄图像。图像中的所有物体基本都是清晰的。
在步骤S502，获取主摄图像对应的深度图。在深度图中，每一点的像素值表示该点距离主摄像头的深度值。
在步骤S503，确定大光圈特效结果图。例如主摄拍摄的图像为人像，大光圈特效结果图中与人像的深度值相同的点保持清晰，与人像的深度值不同的点被模糊处理。
在一些情况下，基于双目系统的深度计算可能出现错误，导致错误的虚化效果。基于双目系统计算深度，主要基于红绿蓝(red、green、blue，RGB)三原色特征在主副图上进行匹配，深度计算的可靠性不稳定，准确度不高。
图6示出了主摄像头采集的图像画面的一例。在601所示的重复纹理区域，如百叶窗，贴纸等，由于RGB特征重复出现，无法精确区分主副图是应该利用哪一个位置点的特征进行匹配，可能造成深度计算错误。图7示出了主摄像头采集的图像画面的另一例。在701所示的弱纹理区域，如墙角、同一颜色的门和把手等，由于RGB特征不明显，或者根本没有特征，即该区域的色差较小，导致主副图无法匹配，也可能造成深度计算错误。由于深度错误，可能会导致虚化结果图出现错误，不能达到预期的虚化效果，用户体验较差。
根据双目深度估计原理，基于双目系统的图像虚化结果，在重复纹理，弱纹理等区域可能出现虚化结果错误。本申请为了解决上述问题，提出了一种基于双目系统和飞行时间(time of flight，TOF)系统的图像处理的方法。
双目系统可以确定稠密深度图，即深度图的像素点较多，分辨率高。且双目系统与人眼感官相符，应用范围更广。但是双目系统确定的深度图的精度和稳定性较差。例如在重复纹理或弱纹理区域容易导致虚化错误。并且由于双目系统的深度精度较低，对于采集的兴趣区域(region of interest，ROI)图像(如人脸等深度渐变平面或曲面)，虚化层次性较差。本申请实施例中的深度跳变区域、深度跳变区域均可以指人像区域。
TOF技术是一种利用激光计算深度的技术。利用TOF技术，可以根据激光发射和接收时间差异来计算深度，和RGB特征无关。基于TOF系统的深度计算精度高，稳定性好。但是由于采样数量的限制，TOF系统仅能够确定稀疏深度图，深度图的分辨率较低。并且由于远距点无反馈信息，TOF系统无法采集远距离点的深度，深度图中会出现无数据点。由于激光信号可能受到干扰，深度图中会出现深度的错误跳变点。
图8是一种双目系统深度计算的方法的示意性流程图。
双目深度计算主要可以分为图像矫正，稀疏深度图计算，稠密深度图计算，后处理四个步骤。
图像矫正利用预设的双目系统内外参数，矫正输入的双目图像，获取行对齐矫正后的图像。根据双目系统内外参数标定值，对双目系统采集的双目数据进行行对其图像校正，得到矫正后的主图和矫正后的副图。例如可以对双目图像进行行对其图像校正。
双目系统的内参数可以包括焦距、坐标轴倾斜参数、主点坐标等，还可以包括径向和切向畸变参数等。双目系统的外参数可以包括旋转参数、平移参数等。
根据矫正后的主图和矫正后的副图，进行ROI分割，可以确定ROI区域。ROI区域可以是从图像中选择的一个图像区域，这个区域的特征可以是图像分析所关注的重点。图像中的有些区域可能是无关紧要的，例如天空，墙壁，草地等。ROI区域例如可以包括人脸等深度渐变的区域。
根据矫正后的主图和矫正后的副图，还可以确定稀疏深度图。稀疏深度图可以通过稀疏深度图计算获得。稀疏深度图计算，可以通过现有的或未来可能出现的一种或多种方式进行。可以基于软件执行工程化深度计算(depth calculation engineering，DCE)模块算法进行，也可以基于深度计算硬件芯片进行，深度计算硬件芯片例如可以是专用DMAP芯片。通过对矫正后的主副图进行稀疏深度图计算，可以得到稀疏深度图。稀疏深度图可以是基于主图的稀疏深度图，也可以是基于副图的稀疏深度图。稀疏深度图也可以称为双目稀疏深度图或双目稀疏深度数据。
确定ROI区域和稀疏深度图数据后，可以根据矫正后主图、矫正后副图、ROI区域和稀疏深度图，确定稠密深度图。稠密深度图可以通过多种方式得到，例如可以通过深度稠 密化算法确定稠密深度图。深度稠密化算法例如可以是光流深度稠密化算法。稠密深度计算，可以利用矫正后的主图和副图，稀疏深度图等，基于光流等稠密化深度计算方法，获取稠密深度图。可选地，在深度稠密化算法中，可以对ROI区域内外采用不同的参数进行计算，例如，ROI区域外的计算参数可以使得ROI区域外的深度计算结果更为平滑，ROI区域内的计算参数可以使得ROI区域内的深度变化更为突出。该稠密深度图也可以称为双目稠密深度图或双目稠密深度数据。
深度图后处理，根据主图，ROI区域，稠密深度图等信息，优化深度图的边缘，纠正人像内外深度错误，确定最终的双目深度图。双目深度图也可以称为双目稠密深度图或双目稠密深度数据。ROI区域可以是主图中的人像区域。
根据图8的方法确定的双目深度图，分辨率高，与人眼感官相符，但是精度和稳定性较差。本申请实施例结合TOF深度数据，对图8中的方法进行改进。
TOF是利用激光的发射和接收时间差来获取深度结果。可以利用专用深度计算芯片或图像信号处理(image signal processing，ISP)算法获取TOF深度数据。示例性地，可以从系统中获取TOF稀疏深度图。
基于双目系统和TOF系统进行图像处理，可以结合双目深度估计原理与TOF深度估计原理的优点，能够获取理想的虚化图像结果。
图9是本申请实施例提供的一种图像处理的方法的示意性流程图。
在步骤S901，获取场景的双目图像数据，以及所述场景的飞行时间TOF数据，所述双目图像数据包括第一图像数据和第二图像数据。
获取场景的双目图像数据，可以是通过双目系统采集双目图像数据，也可以是从存储装置或图像采集装置获取双目图像数据。
获取TOF数据，可以是通过TOF设备采集TOF数据，也可以是从存储装置或TOF数据采集装置获取TOF数据。TOF数据可以是TOF时间数据，也可以是TOF深度数据。发射器发射的信号经场景中的物体反射后被接收器接收。TOF时间数据可以包括信号的往返时间。TOF时间数据进行处理，TOF深度数据可以包括场景的深度。TOF时间数据包括TOF设备采集的TOF时间。TOF深度数据包括根据TOF时间数据确定的TOF深度。TOF数据中，可能由于场景中的物体的深度较大，接收器未接收信号，存在无数据点。根据TOF数据的无数据点可以确定该无数据点对应的场景中的物体的深度较大，例如深度大于预设值。因此，通过无数据点也可以确定无数据点对应的深度。
第一图像数据和第二图像数据分别是双目系统中两个图像采集单元(例如摄像头)对场景采集的图像数据。例如，第一图像数据为主摄像头采集的图像数据，而第二图像数据为副摄像头采集的图像数据。第一图像数据可以为待显示的图像数据，例如显示在屏幕、显示器或显示面板上的图像数据。
在步骤S902之前，可以调整所述TOF数据和/或所述双目深度数据的深度，以使得所述TOF数据与所述双目数据的系统深度误差小于第二预设值。
在步骤S902之前，可以调整所述TOF数据和/或所述双目深度数据的相对位置，以使得所述TOF数据与所述双目数据的系统位置误差小于第三预设值。
由于测量方法和原理不同，根据双目图像数据确定的双目深度数据和根据TOF器件获取的TOF数据可能会存在一定的系统差异，例如场景中的各个物体在图片中位置的差异，深度的差异等。系统误差是由于双目深度图与TOF深度图采集与计算过程中系统的差别造 成的。
可以通过离线标定或在线标定等方式可以将TOF稀疏深度图与双目深度图进行匹配，即进行坐标对齐。坐标对齐可以是行对齐和/或列对齐。例如，通过离线标定获取二者的差异匹配参数，从而将TOF深度值匹配到双目深度图中。例如，可以调整所述TOF数据和/或所述双目深度数据的相对位置，以使得所述TOF数据与所述双目深度数据的系统坐标误差小于第二预设值。
可以通过对TOF数据和/或双目深度数据进行调整，减小甚至消除TOF数据与双目深度数据的由于系统造成的深度误差。例如，可以调整所述TOF数据和/或所述双目深度数据的深度，以使得所述TOF数据与双目深度数据的系统深度误差小于第三预设值。系统深度误差是由于TOF数据与双目深度数据的计算原理的差异造成的。
通过对TOF数据和双目深度数据的位置和深度进行调整，的实现了TOF稀疏深度图和双目深度图匹配。
本申请实施例对于位置和深度调整的顺序不作限定。可以先调整调整所述TOF数据和/或所述双目深度数据的相对位置，再调整所述TOF数据和/或所述双目深度数据的深度。也可以按照相反的顺序，或同时进行。
在一种实施方式中，上述方法包括：根据所述TOF数据修正所述双目深度数据以得到被修正后的双目深度数据，所述双目深度数据是根据所述双目图像数据确定的。具体地，在步骤S902，根据所述双目图像数据和双目深度数据，确定错误区域。
错误区域例如可以是精度和/或稳定性不满足要求的区域。也就是说，错误区域是通过双目深度估计确定的深度数据中精度和/或稳定性不满足要求的区域。错误区域也可以称为易错区域。错误区域可以是通过双目数据计算的深度图中易错区域。
可以根据第一图像数据和双目深度数据，确定错误区域。也可以根据第二图像数据和双目深度数据，确定错误区域。
根据双目深度估计原理，基于双目系统的图像虚化结果，在重复纹理，弱纹理等区域可能出现虚化结果错误。即，重复纹理，弱纹理等区域，双目深度数据稳定性可能不满足要求。通过双目图像数据确定双目深度数据时，需要对第一图像数据和第二图像数据进行匹配。由于重复纹理，弱纹理等区域图像的匹配较为困难，图像匹配的错误可能导致该区域的深度错误。弱纹理区域也可以称为色差不明显的区域，即该区域的色差小于预设值。
为了实现大光圈虚化的效果，在同一图像数据的不同区域，对深度数据的精度要求可以不同。例如，图像中背景区域的深度，精度要求可能较低；而人像等ROI区域，精度要求可能较高。因此，对于深度渐变区域，根据双目数据确定的双目深度数据的精度可能不满足要求。深度渐变区域例如可以是深度变化量小于等于某一预设值的区域。
双目深度数据是根据双目图像数据确定的。双目深度数据可以是双目稠密深度数据，也可以是双目稀疏深度数据。双目深度数据可以是根据双目图像数据确定的一幅深度图。
双目稠密深度数据中，有效信息少，像素精度低。双目稠密深度数据可以基于软件执行DCE模块算法进行，也可以基于深度计算硬件芯片进行，深度计算硬件芯片例如可以是专用DMAP芯片。
双目稠密深度数据中，有效信息多，像素精度高。双目稠密深度数据可以基于软件执行深度稠密化算法进行，深度稠密化算法例如可以是光流稠密化算法。
根据所述双目图像数据和所述双目深度数据，可以提取双目图像数据中的特征。提取 的特征例如可以包括人、手指、汽车、车窗、白墙、重复的窗帘格、天空等。错误区域可以是提取的所有特征中的一个或多个特征对应的区域。
在步骤S903，根据所述TOF数据，对所述错误区域中的深度进行修正，以确定所述场景的被修正后的双目深度数据。
对错误区域中的深度进行修正可以包括：对于错误区域，根据所述错误区域的TOF深度数据，确定所述错误区域的被修正后的双目深度数据。
通过步骤S902-S903，可以对双目深度数据中错误区域的深度进行修正。仅对错误区域的深度进行修正，可以减小计算量，同时提高修正的深度的准确性。对双目深度数据中错误区域的深度进行修正可以包括：将错误区域的TOF深度数据作为错误区域的的深度数据，通过稠密化算法确定错误区域的被修正后的双目深度数据。
在一个示例中，步骤S902-S903可以包括以下步骤：
根据所述双目稠密深度数据和所述第一图像数据，确定所述第一错误区域。
根据所述TOF数据和所述双目稠密深度数据，对所述第一错误区域的深度进行修正。错误区域包括第一错误区域。
双目深度数据包括双目稠密深度数据。对所述第一错误区域的深度进行修正的结果可以作为被修正后的双目深度数据，用于虚化处理。参考图8，双目稠密深度数据可以是图8中的稠密深度图，也可以是图8中的双目深度图。
也就是说，对于第一错误区域，可以根据所述错误区域的TOF深度数据，确定第一错误区域的被修正后的双目深度数据。例如，可以通过稠密化算法，确定第一错误区域的被修正后的双目深度数据。对于第一错误区域之外的区域，可以将第一错误区域之外的区域的双目稠密深度数据作为第一错误区域之外的区域的被修正后的双目深度数据。被修正后的双目深度数据可以是稠密深度数据。换句话说，可以将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；对所述第一错误区域的所述深度进行稠密化处理；将所述双目稠密深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度，以获得所述被修正后的双目深度数据。
对所述第一错误区域的深度进行修正的结果也可以作为第一修正稠密深度数据，用于进一步的修正。
也就是说，对于第一错误区域，可以根据所述错误区域的TOF深度数据，确定第一错误区域的第一修正稠密深度数据。例如，可以通过稠密化算法，确定第一错误区域的第一修正稠密深度数据。对于第一错误区域之外的区域，可以将第一错误区域之外的区域的双目稠密深度数据作为第一错误区域之外的区域的第一修正稠密深度数据。第一修正稠密深度数据可以是稠密深度数据。换句话说，可以将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；对所述第一错误区域的所述深度进行稠密化处理；将所述双目稠密深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度，以获得所述第一修正稠密深度数据。
在另一个示例中，步骤S902-S903可以包括以下步骤：
根据所述双目稀疏深度数据、所述第一图像数据和所述第二图像数据，确定第一错误区域。
根据所述TOF数据和所述双目稀疏深度数据，对所述第一错误区域的深度进行修正。
错误区域包括第一错误区域。双目深度数据包括双目稀疏深度数据。
对所述第一错误区域的深度进行修正的结果可以作为被修正后的双目深度数据，用于虚化处理。
也就是说，对于第一错误区域，可以根据所述错误区域的TOF深度数据，确定第一错误区域的被修正后的双目深度数据。例如，可以通过稠密化算法，确定第一错误区域的被修正后的双目深度数据。对于第一错误区域之外的区域，可以根据第一错误区域之外的区域的双目稀疏深度数据，确定第一错误区域之外的区域的被修正后的双目深度数据。被修正后的双目深度数据可以是稠密深度数据。换句话说，可以将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；将所述双目稀疏深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度；对所述第一错误区域和所述第一错误区域之外的区域的深度进行稠密化处理，以获得所述被修正后的双目深度数据。
对所述第一错误区域的深度进行修正的结果也可以作为第一修正稠密深度数据，用于进一步的修正。
也就是说，对于第一错误区域，可以根据所述错误区域的TOF深度数据，确定第一错误区域的第一修正稠密深度数据。例如，可以通过稠密化算法，确定第一错误区域的第一修正稠密深度数据。对于第一错误区域之外的区域，可以根据第一错误区域之外的区域的双目稀疏深度数据，确定第一错误区域之外的区域的第一修正稠密深度数据。例如，可以通过稠密化算法，确定第一错误区域之外的区域的第一修正稠密深度数据。第一修正稠密深度数据可以是稠密深度数据。换句话说，将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；将所述双目稀疏深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度；对所述第一错误区域和所述第一错误区域之外的区域的深度进行稠密化处理，以获得所述第一修正稠密深度数据。
在上述两个示例中，进一步的修正可以包括以下步骤：根据所述第一修正稠密深度数据、所述TOF数据、所述第一图像数据，确定所述第二错误区域。
根据所述TOF数据和所述第一修正稠密深度数据，对所述第二错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
错误区域包括第二错误区域。
第一错误区域可以包括以下区域中至少一种：重复纹理区域、色差小于预设值的区域、深度渐变区域。
可以述根据所述主图和所述双目深度图，确定第一错误区域。根据主图的纹理信息，确定第一易错区域；根据双目深度图，从第一易错区域中确定第一错误区域；在所述双目深度图中，所述第一错误区域的深度值与其周围的深度值的差值大于或等于第一预设值。
第二错误区域包括所述第一修正稠密深度数据中的深度跳变区域。
根据所述主图和所述双目深度图，确定第二错误区域。根据主图的红绿蓝RGB信息，确定主图中的物体的边缘区域；根据双目深度图，从边缘区域中确定第二错误区域；所述边缘区域包括所述第二错误区域；在所述双目深度图中，所述第二错误区域的深度值与其周围的深度值的差值大于或等于第二预设值。
对于第二错误区域，可以根据所述错误区域的TOF深度数据，确定第二错误区域的被修正后的双目深度数据。例如，可以通过稠密化算法，根据所述错误区域的TOF深度数据，确定第二错误区域的被修正后的双目深度数据。对于第二错误区域之外的区域，可以将第 二错误区域之外的区域的第一修正稠密深度数据作为第一错误区域之外的区域的被修正后的双目深度数据。也就是说，可以将所述TOF数据对应的所述第二错误区域的深度作为所述第二错误区域的深度；对所述第二错误区域的所述深度进行稠密化处理；将所述第一修正稠密深度数据对应的所述第二错误区域之外的区域的深度作为对应的所述第二错误区域之外的区域的深度，以获得所述被修正后的双目深度数据。
错误区域可以包括人像区域。
可以通过主图中的RGB信息确定主图的人像区域。可以利用所述TOF深度图中对应于所述人像区域的深度值，通过神经网络修正所述双目深度图中对应于所述错误区域的深度值。
在进一步的修正中，可以通过阈值判定的方式，判断TOF数据是否有效，去除TOF数据中可能的异常点。异常点可以包括深度的错误跳变点和/或无数据点。TOF判定数据可以是有效的TOF深度数据，也可以是“0”“1”等指示TOF深度数据是否有效的标识。本申请实施例对TOF判定数据的方式不作限定。以TOF数据是TOF深度数据为例进行说明。例如，TOF深度数据中深度满足阈值条件的数据可以记为有效。阈值条件可以是深度满足一定数值范围，例如大于或小于某一预设值。
根据第一修正稠密深度数据、TOF数据、第一图像数据，确定第二错误区域，可以包括：根据第一修正稠密深度数据、TOF数据、TOF判定数据、第一图像数据，确定所述第二错误区域。即，第二错误区域不包括无效的TOF深度数据对应的区域。也就是说，在进一步的修正中，可以仅对有效的TOF深度数据对应的区域的深度进行修正，对于无效的TOF深度数据对应的区域的深度不进行修正。
在步骤S904，根据所述被修正后的双目深度数据，对所述第一图像数据进行虚化处理。
图10是本申请实施例提供的一种图像处理的方法的示意性流程图。
在步骤S1001，获取场景的双目图像数据，以及所述场景的飞行时间TOF数据，所述双目图像数据包括利用主摄像头获取的主图。
在步骤S1002，根据所述主图和双目深度图，确定位于所述主图中的错误区域；所述双目深度图是根据所述双目图像数据确定的；
在步骤S1003，根据所述TOF数据获取TOF深度图；
在步骤S1004，利用所述TOF深度图中对应于所述错误区域的深度值，通过神经网络修正所述双目深度图中对应于所述错误区域的深度值，以得到被修正后的双目深度图，所述修改正后的双目深度图包括对应于错误区域的深度值，以及对应于其他区域的深度值，所述其他区域为所述主图中除所述错误区域之外的区域；所述对应于其他区域的深度值是根据所述双目深度图中对应于所述其他区域的深度值得到的；
可选地，可以利用所述TOF深度图中对应于所述第一错误区域的深度值，通过神经网络修正所述双目深度图中对应于所述第一错误区域的深度值，以得到第一修正后的双目深度图；可以利用所述TOF深度图中对应于所述第二错误区域的深度值，通过神经网络修正所述第一修正后的双目深度图中对应于所述第二错误区域的深度值，以得到所述被修正后的双目深度图。
可选地，所述双目深度图为双目稠密深度图；所述对应于其他区域的深度值是根据所述双目深度图中对应于所述其他区域的深度值得到的，包括：所述其他区域的深度值为所述双目稠密深度图对应于所述其他区域的深度值。
可选地，所述双目深度图为双目稀疏深度图；所述对应于其他区域的深度值是根据所述双目深度图中对应于所述其他区域的深度值得到的，包括：所述其他区域的深度值是对所述双目稀疏深度图对应于所述其他区域的深度值进行稠密化处理得到的。稠密化处理例如可以是光流稠密化处理等稠密化处理的方法。
可选地，所述方法还包括：根据所述主图，确定所述人像区域；所述第二错误区域位于所述第一错误区域和所述人像区域之外；可以利用所述TOF深度图中对应于所述第一错误区域和所述人像区域的深度值，通过神经网络修正所述双目深度图中对应于所述第一错误区域和所述人像区域的深度值，以得到第一修正后的双目深度图；可以利用所述TOF深度图中对应于所述第二错误区域的深度值，通过神经网络修正所述第一修正后的双目深度图中对应于所述第二错误区域的深度值，以得到所述被修正后的双目深度图。
在步骤S1005，根据所述被修正后的双目深度图被修正后的双目深度图，对所述主图进行虚化处理。
错误区域可以包括第一错误区域和/或第二错误区域。
第一错误区域可以包括重复纹理区域和/或色差小于预设值的区域。
可以根据所述主图和所述双目深度图，确定第一错误区域。根据主图的纹理信息，确定第一易错区域；根据双目深度图，从第一易错区域中确定第一错误区域；在所述双目深度图中，所述第一错误区域的深度值与其周围的深度值的差值大于或等于第一预设值。
第二错误区域可以是深度跳变区域。
可以根据所述主图、双目深度图，确定第二错误区域。根据主图的红绿蓝RGB信息，确定主图中的物体的边缘区域；根据双目深度图，从边缘区域中确定第二错误区域；所述边缘区域包括所述第二错误区域；在所述双目深度图中，所述第二错误区域的深度值与其周围的深度值的差值大于或等于第二预设值。
第二错误区域也可以是主图中全部的深度跳变区域，也可以是主图中人像区域周围的深度跳变区域。
可以根据所述主图、双目深度图和人像区域，确定第二错误区域。根据主图的红绿蓝RGB信息，确定主图中的物体的边缘区域；根据双目深度图，从边缘区域中确定人像区域周围的第二错误区域；所述边缘区域包括所述第二错误区域；在所述双目深度图中，所述第二错误区域的深度值与其周围的深度值的差值大于或等于第二预设值。
可以根据主图的RGB信息确定人像区域。人像区域例如可以是主图中人体的全部的区域，也可以是人体的一部分的区域，例如人脸区域、人手区域等。人像区域也可以理解为错误区域。
对第一错误区域、第二错误区域、人像区域的修正可以相同或不同的处理器中进行。第一错误区域、第二错误区域、人像区域的修正可以同时或不同时进行。
例如，可以在第一级网络中进行对第一错误区域和人像区域的修正，在第二级网络中进行对第二错误区域的修正，第一级网络和第二级网络可以是神经网络。
本申请实施例对第一错误区域和人像区域的相互关系不作限定。例如，第一错误区域不包括人像区域，或包括人像区域中的全部或部分区域。
可选地，所述第二错误区域包括所述双目深度图中的深度跳变区域。
可选地，所述第一错误区域包括以下区域中至少一种：重复纹理区域、色差小于第一预设值的区域或深度渐变区域。
可选地，在所述根据所述TOF深度图，对所述错误区域中的深度进行修正，以确定被修正后的双目深度图之前，还包括：调整所述TOF深度图和/或所述双目深度图的深度，以使得所述TOF深度图与所述双目深度图的系统深度误差小于第二预设值。
可选地，在所述根据所述TOF深度图，对所述错误区域中的深度进行修正，以确定被修正后的双目深度图之前，还包括：调整所述TOF深度图和/或所述双目深度图的相对位置，以使得所述TOF深度图与所述双目深度图的系统位置误差小于第三预设值。
图11是本申请实施例提出的一种图像处理的方法的示意性流程图。
在步骤S1101，获取场景的双目图像数据，以及所述场景的飞行时间TOF数据，所述双目图像数据包括利用主摄像头获取的主图。
在步骤S1102，利用所述双目图像数据获取双目稀疏深度图。
在步骤S1103，根据所述TOF数据获取TOF深度图。
在步骤S1104，根据所述双目稀疏深度图和所述双目图像数据，确定位于所述主图中的错误区域。
在步骤S1105，利用所述TOF深度图中对应于所述错误区域的深度值，通过神经网络修正所述双目稀疏深度图中对应于所述错误区域的深度值。
在步骤S1106，利用所述双目图像数据，对所述双目稀疏深度图进行稠密化处理，以得到对应于其他区域的深度值，以得到被修正后的双目深度图；所述被修正后的双目深度图包括对应于错误区域的深度值以及对应于其他区域的深度值，所述其他区域为所述主图中除所述错误区域之外的区域。
示例性地，可以根据所述主图，确定所述人像区域；所述第二错误区域位于所述第一错误区域和所述人像区域之外；可以利用所述TOF深度图中对应于所述第一错误区域和所述人像区域的深度值，通过神经网络修正所述双目深度图中对应于所述第一错误区域和所述人像区域的深度值，以得到第一修正后的双目深度图；第一修正后的双目深度图包括对应于所述第一错误区域和所述人像区域的深度值，以及对应于第一其他区域的深度值；所述第一其他区域为所述主图中除所述第一错误区域和所述人像区域之外的区域。
可以利用所述双目图像数据，对所述双目稀疏深度图进行稠密化处理，以得到对应于其他区域的深度值。
可以利用所述双目图像数据，对所述双目稀疏深度图进行稠密化处理；以得到所述第一修正后的双目深度图中对应于所述第一其他的深度值。
可以利用所述TOF深度图中对应于所述第二错误区域的深度值，通过神经网络修正所述第一修正后的双目深度图中对应于所述第二错误区域的深度值，以得到所述被修正后的双目深度图；所述被修正后的双目深度图包括对应于所述第二错误区域的深度值，以及对应于其他区域的深度值；所述其他区域的深度值为所述第一修正后的双目深度图中对应于所述其他的深度值。
示例性地，可以利用所述TOF深度图中对应于所述第一错误区域的深度值，通过神经网络修正所述双目深度图中对应于所述第一错误区域的深度值，以得到第一修正后的双目深度图；第一修正后的双目深度图包括对应于所述第一错误区域的深度值，以及对应于第一其他区域的深度值；所述第一其他区域为所述主图中除所述第一错误区域之外的区域；
可以利用所述双目图像数据，对所述双目稀疏深度图进行稠密化处理，以得到对应于其他区域的深度值；
可以利用所述双目图像数据，对所述双目稀疏深度图对应于所述第一其他的深度值进行稠密化处理；以得到所述第一修正后的双目深度图中对应于所述第一其他的深度值；
可以利用所述TOF深度图中对应于所述第二错误区域的深度值，通过神经网络修正所述第一修正后的双目深度图中对应于所述第二错误区域的深度值，以得到所述被修正后的双目深度图；所述被修正后的双目深度图包括对应于所述第二错误区域的深度值，以及对应于其他区域的深度值；所述其他区域的深度值为所述第一修正后的双目深度图中对应于所述其他的深度值。
在步骤S1107，根据所述被修正后的双目深度图被修正后的双目深度图，对所述主图进行虚化处理。
可选地，在所述根据所述TOF深度图，对所述错误区域中的深度进行修正，以确定被修正后的双目深度图之前，还包括：调整所述TOF深度图和/或所述双目深度图的深度，以使得所述TOF深度图与所述双目深度图的系统深度误差小于第二预设值。
可选地，在所述根据所述TOF深度图，对所述错误区域中的深度进行修正，以确定被修正后的双目深度图之前，还包括：调整所述TOF深度图和/或所述双目深度图的相对位置，以使得所述TOF深度图与所述双目深度图的系统位置误差小于第三预设值。
图12是本申请实施例提出的一种图像处理的方法的示意性流程图。
首先，分别计算出双目系统和TOF系统的深度图。基于双目系统的双目图像数据，计算双目深度图。根据TOF器件采集的TOF数据，计算TOF深度数据。
之后，根据双目深度图和TOF深度图，进行深度图的图像融合，确定融合深度图。图像融合(image fusion)是用特定的算法将两幅或多幅图像综合成一幅新的图像。
最后，根据融合深度图，对双目系统中的主图采集单元采集的主图进行虚化处理，获得虚化结果图。
深度融合可以利用两级卷积神经网络(convolutional neural network，CNN)网络进行。
图13是本申请实施例提出的一种深度融合的方法的示意性流程图。
根据主图或矫正后的主图、双目深度图和TOF稀疏深度图，可以进行深度融合，得到最终深度图。双目深度图可以是图8中经过深度后处理得到的双目深度图，也可以是未经过深度后处理的稠密深度图。深度融合可以利用两级卷积神经网络(convolutional neural network，CNN)网络进行。
由于测量方法和原理不同，双目深度图和根据TOF器件获取的TOF稀疏深度图可能会存在一定的系统差异，例如位置的差异和/或深度的差异。系统误差是由于双目深度图与TOF深度图采集与计算过程中系统的差别造成的。位置的差异可以通过在同一坐标系中的坐标体现。
由于测量方法和原理不同，根据双目图像数据确定的双目深度数据和根据TOF器件获取的TOF数据可能会存在一定的系统差异，例如场景中的各个物体在图片中位置的差异，深度的差异等。系统误差是由于双目深度图与TOF深度图采集与计算过程中系统的差别造成的。
可以通过离线标定或在线标定等方式可以将TOF稀疏深度图与双目深度图进行匹配，即进行坐标对齐。坐标对齐可以是行对齐和/或列对齐。例如，通过离线标定获取二者的差异匹配参数，从而将TOF深度值匹配到双目深度图中。例如，可以调整所述TOF稀疏深度图和/或所述双目深度图的位置，以使得所述TOF稀疏深度图与所述双目深度图的系统 位置误差小于或等于预设值。基于双目系统的标定参数，可以使TOF稀疏深度图与双目深度图调整为相同的尺寸。系统位置误差是由于采集TOF数据的TOF器件与采集双目图像数据的双目系统位置的差异造成的。
可以通过对TOF数据的深度进行调整，减小甚至消除TOF数据与双目深度数据的由于系统造成的深度误差。调整所述TOF数据的深度，以使得所述TOF数据与双目深度数据的系统深度误差小于或等于预设值。系统深度误差是由于TOF数据与双目深度数据的计算原理的差异造成的。
可以通过对TOF稀疏深度图和/或双目深度图进行调整，减小甚至消除TOF数据与双目深度数据的由于系统造成的深度误差。例如，可以调整所述TOF稀疏深度图和/或所述双目深度图的深度，以使得所述TOF稀疏深度图与双目深度图的系统深度误差小于第三预设值。系统深度误差是由于TOF稀疏深度图与双目深度图的计算原理的差异造成的。
通过对TOF稀疏深度图和双目深度图的位置和深度进行调整，的实现了TOF稀疏深度图和双目深度图匹配。
第一级CNN网络可以根据矫正后主图和双目深度图，确定第一错误区域。第一级CNN网络可以根据TOF稀疏深度图对双目深度图中的第一错误区域的进行修正，即根据TOF深度对双目深度图第一错误区域中的深度进行调整，从而可以得到第一修正稠密深度图。第一修正稠密深度图可以作为最终的融合深度图。
第一错误区域可以是通过双目数据计算的深度图中易错区域，例如可以是重复纹理区域，色差小于预设值的区域，或深度渐变区域。
第二级CNN网络可以对第一修正稠密深度图进行进一步的修正。第二级CNN网络可以根据矫正后主图和第一修正稠密深度图，确定第二错误区域。第二级CNN网络可以根据TOF稀疏深度图对第一修正稠密深度图中的第二错误区域的进行修正，即根据TOF深度对第一修正稠密深度图第二错误区域中的深度进行调整，从而可以得到融合深度图。
第二错误区域可以是修正稠密深度图中深度发生跳变的区域。对第二错误区域的深度进行修正，可以用于提高主图数据中的物体的边缘区域的深度的精度，也可以用于去除错误的深度跳变。
根据融合深度图，对双目系统中的主图采集单元采集的主图进行虚化处理，可以获得虚化结果图。
图14是一种光流神经网络(flownet)的运算的示意图。双目系统获取的双目数据通过一个只有卷积层组成的网络结构，多次向量卷积运算(convolution，conv)，可以提取出图像特征，进行稠密化深度计算。双目数据也可以通过其他的网络结构，例如一种网络结构先独立的提取两张图片的特征，再对这些特征进行匹配，从而可以进行稠密化深度计算。
进行深度融合过程的两级CNN网络可以利用编码-解码(encoder-decoder)式网络。图15是一种编码-解码式网络的示意性结构图。编码器可以对输入数据进行分析。编码器可以用于图像的识别。编码器可以用于提取图像的特征。提取的特征作为编码器的输出可以通过跳跃式传递(skip connection)的方式传输至解码器。提取的特征例如可以是人、手指、车、车窗、白墙、重复的窗帘格、天空等。解码器可以根据编码器提取的特征确定双目深度数据中的错误区域，并对该区域的深度进行修正。错误区域可以是编码器提取的所有特征中的一个或多个特征对应的区域。错误区域可以包括重复纹理区域，色差小于预设值的区域，深度渐变区域，或深度跳变区域等。错误区域可以是双目深度计算易错区域。
图16是本申请实施例提出的一种图像处理的方法的示意性流程图。
进行深度融合过程的第一级网络设备，可以根据双目数据中的主图数据、双目数据确定的稠密深度数据、TOF深度数据，确定修正稠密深度图。第一级网络设备可以是编码-解码式网络，也可以是其他结构的网络。第一级网络设备可以是CNN网络设备，也可以是其他人工智能(artificial intelligence，AI)网络。双目稠密深度数据可以是图8中作为中间结果的稠密深度图数据，也可以是图8中的作为最终结果的双目深度图数据。第一级网络设备可以根据TOF深度数据对双目稠密深度数据中第一错误区域的深度进行修正，确定修正稠密深度图。第一级网络设备例如可以根据主图数据和双目稠密深度数据，确定第一错误区域。第一错误区域可以是通过双目数据计算的深度图中易错区域，例如可以是重复纹理区域，色差小于预设值的区域，或深度渐变区域。第一级网络设备中的编码器1可以根据主图数据和双目稠密深度数据，进行特征提取。第一级网络设备中的解码器1可以根据提取的特征，并结合主图数据和/或双目稠密数据，确定第一错误区域。第一错误区域的深度的修正，可以由第一级网络设备中的解码器1进行。第一级网络设备中的解码器1，可以根据TOF深度数据，对稠密深度数据中，第一错误区域的深度进行修正，确定修正稠密深度图。第一级网络输出的修正稠密深度图，可以作为第二级网络的输入。在一些实施例中，修正稠密深度图可以作为最终的融合深度图。
进行深度融合过程的第二级CNN网络，可以根据修正稠密深度图、主图数据和TOF深度数据，确定第二稠密深度图。第二级网络设备可以对修正稠密深度图中，第二错误区域的深度进行修正，确定最终的融合深度图。第二级网络设备可以是编码-解码式网络，也可以是其他结构的网络。第二级网络设备可以是CNN网络设备，也可以是其他人工智能(artificial intelligence，AI)网络。最终的融合深度图可以是稠密深度图。
第二级网络设备可以确定第二错误区域，例如可以根据主图数据、修正稠密深度图、TOF判定数据确定第二错误区域。第二错误区域可以是主图数据中的物体的边缘区域，例如可以是修正稠密深度图中深度发生跳变的区域。对第二错误区域的深度进行修正，可以用于提高主图数据中的物体的边缘区域的深度的精度，也可以用于去除错误的深度跳变。
可以通过阈值判定的方式，判断TOF数据是否有效，去除TOF数据中可能的异常点。异常点可以包括深度的错误跳变点和/或无数据点。TOF判定数据可以是有效的TOF深度数据，也可以是“0”“1”等指示TOF深度数据是否有效的标识。本申请实施例对TOF判定数据的方式不作限定。TOF数据可以是TOF深度数据。TOF深度数据中深度满足阈值条件的数据可以记为有效。阈值条件可以是深度满足一定数值范围，例如大于或小于某一预设值。
第二级网络设备的解码器2可以根据TOF数据，或根据TOF数据和TOF判定数据，修正稠密深度图中的第二错误区域的深度。根据TOF判定数据，第二级网络设备可以仅对有效的TOF深度数据对应的第二错误区域的深度进行修正，对于无效的TOF深度数据对应的第二错误区域的深度不进行修正。
根据融合深度图，对双目系统中的主图采集单元采集的主图进行虚化处理，可以获得虚化结果图。
在上述方法中，首先计算了双目系统的稠密深度图，再通过CNN网络优化获取最终的融合深度图。由于CNN网络可以端到端的计算和优化深度图，因此可以将矫正后的主副图，TOF深度图作为输入，通过CNN网络获取最终的融合深度图。
图17是本申请实施例提出的一种图像处理的方法的示意性流程图。
根据双目图像数据和TOF数据，进行深度计算，确定融合深度图。
根据融合深度图，对双目系统中的主图采集单元采集的主图进行虚化处理，可以获得虚化结果图。
图18是本申请实施例提出的一种图像处理的方法的示意性流程图。获取融合深度图主要可以分为图像矫正，稀疏深度图计算，稠密深度图计算，后处理四个步骤。
将双目数据和TOF数据同时输入深度计算模块，深度计算模块根据输入的三路数据进行优化计算，生成满足需要要求的深度图，最终输入高质量虚化结果图。
图像矫正利用预设的双目系统内外参数，矫正输入的双目图像，获取行对齐矫正后的图像。根据双目系统内外参数标定值，对双目系统采集的双目数据进行行对其图像校正，得到矫正后的主图和矫正后的副图。
根据矫正后的主图和矫正后的副图，进行ROI分割，可以确定ROI区域。
根据矫正后的主图和矫正后的副图，还可以确定双目稀疏深度图。双目稀疏深度图可以通过稀疏深度图计算获得。稀疏深度图计算，可以通过现有的或未来可能出现的多种方式进行。可以基于软件执行DCE算法进行，也可以基于深度计算硬件芯片进行，例如DMAP芯片。通过对矫正后的主副图进行稀疏深度图计算，可以得到基于主图的双目稀疏深度图。
确定ROI区域和稀疏深度图数据后，可以根据矫正后主图、矫正后副图、ROI区域、稀疏深度图、TOF深度图，确定稠密深度图。
由于测量方法和原理不同，双目深度图和TOF获取的深度可能会存在一定的系统差异例如场景中的各个物体在图片中位置的差异，深度的差异等。系统误差是由于双目深度图与TOF深度图采集与计算过程中系统的差别造成的。
可以对TOF稀疏深度图与双目深度图进行坐标对齐，获取调整后的TOF稀疏深度图。可以通过离线标定或在线标定等方式可以将TOF稀疏深度图与双目深度图进行匹配，即进行坐标对齐。坐标对齐可以是行对齐和/或列对齐。例如，通过离线标定获取二者的差异匹配参数，从而将TOF深度值匹配到双目深度图中。
可以通过对TOF数据的深度进行调整，减小甚至消除TOF数据与双目深度数据的由于系统造成的深度误差。调整所述TOF数据的深度，以使得所述TOF数据与双目深度数据的系统深度误差小于或等于预设值。系统深度误差是由于TOF数据与双目深度数据的计算原理的差异造成的。
稠密深度图可以通过深度稠密化算法得到。根据矫正后的主副图、双目稀疏深度图、调整后的TOF稀疏深度图等，获取稠密深度图。深度稠密化算法例如可以是基于AI深度填充算法等稠密化算法。在稠密深度图确定的过程中，根据TOF数据，对双目深度图中的第一错误区域的深度进行修正。双目深度图可以是稀疏深度图，也可以是双目稠密深度图。示例性地，根据矫正后主图、矫正后副图、ROI区域、稀疏深度图，可以确定双目稠密深度图。第一错误区域可以包括所述第一图像数据中以下区域中至少一种：重复纹理区域、色差小于预设值的区域、深度渐变区域。深度渐变区域可以是人像区域。可以通过矫正后的主图确定主图的人像区域。可以通过主图的RGB特征确定人像区域。
可选地，在深度稠密化算法中，可以对ROI区域内外采用不同的参数进行计算，例如，ROI区域内的计算参数可以使得ROI区域内的深度计算结果更为平滑。例如，可以去除ROI区域内深度跳变的一个或多个点的深度值。深度跳变可以是深度的异常点。可以通过 主图的纹理信息确定主图的第一错误区域，利用TOF数据对第一错误区域对应的深度值进行修正。可以利用TOF数据对人像区域的深度值进行修正。
可以对稠密深度图进行后处理。根据主图或矫正后的主图、ROI区域、稠密深度图等信息，优化稠密深度图中的ROI区域的边缘、深度跳变的区域，纠正明显的深度错误，确定最终的双目深度图。
根据融合深度图，对双目系统中的主图采集单元采集的主图进行虚化处理，可以获得虚化结果图。
图19是本申请实施例提出的一种图像处理的方法的示意性流程图。
第一级网络设备可以根据双目数据、双目数据确定的稀疏深度图和TOF深度数据，确定第一修正稠密深度图。第一级网络设备可以根据基于AI的稠密化算法确定第一修正稠密深度图。第一级网络设备可以是CNN网络设备，也可以是其他人工智能(artificial intelligence，AI)网络。
第一级网络设备可以根据双目数据和稀疏深度图，确定第一错误区域。第一错误区域可以是通过双目数据计算的深度图中易错区域，例如可以是重复纹理区域，色差小于预设值的区域，或深度渐变区域。
第一级网络设备可以根据所述TOF数据，对稀疏深度数据中的所述第一错误区域进行修正，从而确定修正稠密深度数据。
本申请实施例对第一级网络设备根据所述TOF数据对稀疏深度数据中的第一错误区域进行修正，确定修正稠密深度数据的方式不做限定。
在一些实施例中，第一级网络设备可以根据双目数据，或根据双目数据和稀疏深度图，确定第一错误区域之外的区域的稠密深度数据。第一级网络设备可以根据TOF数据和双目数据，或根据TOF数据、双目数据和稀疏深度图，确定第一错误区域的稠密深度数据。修正稠密深度数据可以包括第一错误区域之外的区域的稠密深度数据与第一错误区域的稠密深度数据。
在一些实施例中，第一级网络设备可以根据TOF数据修改稀疏深度图中第一错误区域的的深度得到修改后的稀疏深度图。第一级网络设备可以根据修改后的稀疏深度图和双目数据，确定修正稠密深度数据。
修正稠密深度数据可以作为融合深度图。融合深度图也可以是修正稠密深度数据经过第二级网络设备的修订后得到的。
第二级网络设备可以根据修正稠密深度图、主图数据和TOF深度数据，确定第二稠密深度图。第二级网络设备可以对修正稠密深度图中，第二错误区域的深度进行修正，确定最终的融合深度图。第二级网络设备可以是编码-解码式网络，也可以是其他结构的网络。第二级网络设备可以是CNN网络设备，也可以是其他AI网络。最终的融合深度图可以是稠密深度图。
第二级网络设备可以确定第二错误区域，例如可以根据主图数据、修正稠密深度图、TOF判定数据确定第二错误区域。第二错误区域可以是主图数据中的物体的边缘区域，例如可以是修正稠密深度图中深度发生跳变的区域。对第二错误区域的深度进行修正，可以用于提高主图数据中的物体的边缘区域的深度的精度，也可以用于去除错误的深度跳变。
第二级网络设备的解码器2可以根据TOF数据和/或TOF判定数据，修正稠密深度图中的第二错误区域的深度。根据TOF判定数据，第二级网络设备可以仅对有效的TOF深 度数据对应的第二错误区域的深度进行修正，对于无效的TOF深度数据对应的第二错误区域的深度不进行修正。
可以通过阈值判定的方式，判断TOF数据是否有效，去除TOF数据中可能的异常点。异常点可以包括深度的错误跳变点和/或无数据点。TOF判定数据可以是有效的TOF深度数据，也可以是“0”“1”等指示TOF深度数据是否有效的标识。本申请实施例对TOF判定数据的方式不作限定。TOF数据可以是TOF深度数据。TOF深度数据中深度满足阈值条件的数据可以记为有效。阈值条件可以是深度满足一定数值范围，例如大于或小于某一预设值。
根据融合深度图，对双目系统中的主图采集单元采集的主图进行虚化处理，可以获得虚化结果图。
上文结合图1至图19的描述了本申请实施例的方法实施例，下面结合图20至图22，描述本申请实施例的装置实施例。应理解，方法实施例的描述与装置实施例的描述相互对应，因此，未详细描述的部分可以参见前面方法实施例。
图20是本申请实施例提供的一种图像处理的装置1800，包括：
获取模块1810，用于获取场景的双目图像数据，以及所述场景的飞行时间TOF数据，所述双目图像数据包括根据不同摄像头得到的第一图像数据和第二图像数据。
确定模块1820，用于根据所述双目图像数据和双目深度数据，确定错误区域；所述双目深度数据是根据所述双目图像数据确定的。
修正模块1830，用于根据所述TOF数据，对所述错误区域中的深度进行修正，以确定所述场景的被修正后的双目深度数据。
虚化处理模块1840，用于根据所述被修正后的双目深度数据，对所述第一图像数据进行虚化处理，其中所述第一图像数据为待显示的图像数据。
可选地，所述错误区域包括第一错误区域；所述双目深度数据包括双目稠密深度数据。
具体地，确定模块1820用于根据所述双目深度数据，以及所述第一图像数据或所述第二图像数据，确定所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域。
确定模块1820用于，根据所述双目稠密深度数据和所述第一图像数据，确定所述第一错误区域。
修正模块1830用于，根据所述TOF数据和所述双目稠密深度数据，对所述第一错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
可选地，修正模块1830用于：将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；对所述第一错误区域的所述深度进行稠密化处理；将所述双目稠密深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度，以获得所述被修正后的双目深度数据。
可选地，所述错误区域包括第一错误区域和第二错误区域；所述双目深度数据包括双目稠密深度数据。
确定模块1820用于，根据所述双目稠密深度数据和所述第一图像数据，确定所述第一错误区域。
修正模块1830用于，根据所述TOF数据和所述双目稠密深度数据，对所述第一错误区域的深度进行修正，以确定第一修正稠密深度数据。
确定模块1820还用于，根据所述第一修正稠密深度数据、所述TOF数据、所述第一图像数据，确定所述第二错误区域。
修正模块1830还用于，根据所述TOF数据和所述第一修正稠密深度数据，对所述第二错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
可选地，修正模块1830用于：将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；对所述第一错误区域的所述深度进行稠密化处理；将所述双目稠密深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度，以获得所述第一修正稠密深度数据。
可选地，所述错误区域包括第一错误区域；所述双目深度数据包括双目稀疏深度数据。
确定模块1820用于，根据所述双目稀疏深度数据、所述第一图像数据和所述第二图像数据，确定第一错误区域；
修正模块1830用于，根据所述TOF数据和所述双目稀疏深度数据，对所述第一错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
可选地，修正模块1830用于：将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；将所述双目稀疏深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度；对所述第一错误区域和所述第一错误区域之外的区域的深度进行稠密化处理，以获得所述被修正后的双目深度数据。
可选地，所述错误区域包括第一错误区域和第二错误区域；所述双目深度数据包括双目稀疏深度数据。
确定模块1820用于，根据所述双目稀疏深度数据、所述第一图像数据、所述第二图像数据，确定所述第一错误区域。
修正模块1830用于，根据所述TOF数据和所述双目稀疏深度数据，对所述第一错误区域的深度进行修正，确定第一修正稠密深度数据。
确定模块1820还用于，根据所述第一修正稠密深度数据、所述TOF数据、所述第一图像数据，确定所述第二错误区域。
修正模块1830还用于，根据所述TOF数据和所述第一修正稠密深度数据，对所述第二错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
可选地，修正模块1830用于：将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；将所述双目稀疏深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度；对所述第一错误区域和所述第一错误区域之外的区域的深度进行稠密化处理，以获得所述第一修正稠密深度数据。
可选地，修正模块1830用于：将所述TOF数据对应的所述第二错误区域的深度作为所述第二错误区域的深度；对所述第二错误区域的所述深度进行稠密化处理；将所述第一修正稠密深度数据对应的所述第二错误区域之外的区域的深度作为对应的所述第二错误区域之外的区域的深度，以获得所述被修正后的双目深度数据。
可选地，第二错误区域包括所述第一修正稠密深度数据中的深度跳变区域。
可选地，第一错误区域包括以下区域中至少一种：重复纹理区域、色差小于第一预设值的区域、深度渐变区域。
可选地，图像处理的装置1800还包括第一调整模块，用于在所述根据所述TOF数据，对所述错误区域中的深度进行修正，以确定所述场景的被修正后的双目深度数据之前，调 整所述TOF数据和/或所述双目深度数据的深度，以使得所述TOF数据与所述双目数据的系统深度误差小于第二预设值。
可选地，图像处理的装置1800还包括第二调整模块，用于在所述根据所述TOF数据，对所述错误区域中的深度进行修正，以确定所述场景的被修正后的双目深度数据之前，调整所述TOF数据和/或所述双目深度数据的相对位置，以使得所述TOF数据与所述双目数据的系统位置误差小于第三预设值。
图21是本申请实施例提供的一种图像处理的装置2100，包括：
获取模块2110，用于获取场景的双目图像数据，以及所述场景的飞行时间TOF数据，所述双目图像数据包括利用主摄像头获取的主图；
确定模块2120，用于根据所述主图和双目深度图，确定位于所述主图中的错误区域；所述双目深度图是根据所述双目图像数据确定的；
修正模块2130，利用所述TOF深度图中对应于所述错误区域的深度值，通过神经网络修正所述双目深度图中对应于所述错误区域的深度值，以得到被修正后的双目深度图，所述修改正后的双目深度图包括对应于错误区域的深度值，以及对应于其他区域的深度值，所述其他区域为所述主图中除所述错误区域之外的区域；所述对应于其他区域的深度值是根据所述双目深度图中对应于所述其他区域的深度值得到的；所述TOF深度图是根据所述TOF数据得到的；
虚化处理模块2140，根据所述被修正后的双目深度图，对所述主图进行虚化处理。
可选地，所述错误区域包括第一错误区域和第二错误区域，所述第二错误区域位于所述第一错误区域和人像区域之外；所述人像区域是根据所述主图得到的；
修正模块2130用于，利用所述TOF深度图中对应于所述第一错误区域和所述人像区域的深度值，通过神经网络修正所述双目深度图中对应于所述第一错误区域和所述人像区域的深度值，以得到第一修正后的双目深度图；
修正模块2130用于，利用所述TOF深度图中对应于所述第二错误区域的深度值，通过神经网络修正所述第一修正后的双目深度图中对应于所述第二错误区域的深度值，以得到所述被修正后的双目深度图。
可选地，所述错误区域包括第一错误区域和第二错误区域，所述第二错误区域位于所述第一错误区域之外；
修正模块2130用于，利用所述TOF深度图中对应于所述第一错误区域的深度值，通过神经网络修正所述双目深度图中对应于所述第一错误区域的深度值，以得到第一修正后的双目深度图；
修正模块2130用于，利用所述TOF深度图中对应于所述第二错误区域的深度值，通过神经网络修正所述第一修正后的双目深度图中对应于所述第二错误区域的深度值，以得到所述被修正后的双目深度图。
可选地，确定单元2120用于，根据主图的纹理信息，确定第一易错区域；根据双目深度图，从第一易错区域中确定第一错误区域；易错区域在所述双目深度图中，所述第一错误区域的深度值与其周围的深度值的差值大于或等于第一预设值。
可选地，所述第二错误区域包括所述双目深度图中的深度跳变区域。
可选地，所述第一错误区域包括以下区域中至少一种：重复纹理区域、色差小于第一预设值的区域。
可选地，装置2100还包括：调整模块，用于调整所述TOF深度图和/或所述双目深度图的深度，以使得所述TOF深度图与所述双目深度图的系统深度误差小于第二预设值。
可选地，调整模块用于，调整所述TOF深度图和/或所述双目深度图的相对位置，以使得所述TOF深度图与所述双目深度图的系统位置误差小于第三预设值。
图22是本申请实施例提出的一种终端设备1900，包括：
双目系统1910，用于采集双目图像数据，所述双目图像数据包括第一图像数据和第二图像数据。
TOF器件1920，用于采集TOF数据。
处理器1930，当程序指令被所述至少一个处理器中执行时，处理器1930用于执行以下操作：根据所述双目图像数据和双目深度数据，确定错误区域；所述双目深度数据是根据所述双目图像数据确定的；根据所述TOF数据，对所述错误区域中的深度进行修正，以确定所述场景的被修正后的双目深度数据；根据所述被修正后的双目深度数据，对所述第一图像数据进行虚化处理。
可选地，处理器1930，还可以用于获取所述双目图像数据和所述TOF数据。
可选地，所述错误区域包括第一错误区域；所述双目深度数据包括双目稠密深度数据；
处理器1930用于执行以下操作：根据所述双目稠密深度数据和所述第一图像数据，确定所述第一错误区域；根据所述TOF数据和所述双目稠密深度数据，对所述第一错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
可选地，处理器1930用于：将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；对所述第一错误区域的所述深度进行稠密化处理；将所述双目稠密深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度，以获得所述被修正后的双目深度数据。
可选地，所述错误区域包括第一错误区域和第二错误区域；所述双目深度数据包括双目稠密深度数据；处理器1930用于执行以下操作：根据所述双目稠密深度数据和所述第一图像数据，确定所述第一错误区域；根据所述TOF数据和所述双目稠密深度数据，对所述第一错误区域的深度进行修正，以确定第一修正稠密深度数据；根据所述第一修正稠密深度数据、所述TOF数据、所述第一图像数据，确定所述第二错误区域；根据所述TOF数据和所述第一修正稠密深度数据，对所述第二错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
可选地，处理器1930用于：将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；对所述第一错误区域的所述深度进行稠密化处理；将所述双目稠密深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度，以获得所述第一修正稠密深度数据。
可选地，所述错误区域包括第一错误区域；所述双目深度数据包括双目稀疏深度数据；处理器1930用于执行以下操作：根据所述双目稀疏深度数据、所述第一图像数据和所述第二图像数据，确定所述第一错误区域；根据所述TOF数据和所述双目稀疏深度数据，对所述第一错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
可选地，处理器1930用于：将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；将所述双目稀疏深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度；对所述第一错误区域和所述第一错 误区域之外的区域的深度进行稠密化处理，以获得所述被修正后的双目深度数据。
可选地，所述错误区域包括第一错误区域和第二错误区域；所述双目深度数据包括双目稀疏深度数据；处理器1930用于执行以下操作：根据所述双目稀疏深度数据、所述第一图像数据、所述第二图像数据，确定所述第一错误区域；根据所述TOF数据和所述双目稀疏深度数据，对所述第一错误区域的深度进行修正，确定第一修正稠密深度数据；根据所述第一修正稠密深度数据、所述TOF数据、所述第一图像数据，确定所述第二错误区域；根据所述TOF数据和所述第一修正稠密深度数据，对所述第二错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
可选地，处理器1930用于：将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；将所述双目稀疏深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度；对所述第一错误区域和所述第一错误区域之外的区域的深度进行稠密化处理，以获得所述第一修正稠密深度数据。
可选地，处理器1930用于：将所述TOF数据对应的所述第二错误区域的深度作为所述第二错误区域的深度；对所述第二错误区域的所述深度进行稠密化处理；将所述第一修正稠密深度数据对应的所述第二错误区域之外的区域的深度作为对应的所述第二错误区域之外的区域的深度，以获得所述被修正后的双目深度数据。
可选地，所述第二错误区域包括所述第一修正稠密深度数据中的深度跳变区域。
可选地，所述第一错误区域包括以下区域中至少一种：重复纹理区域、色差小于第一预设值的区域或深度渐变区域。
可选地，处理器1930用于：在所述根据所述TOF数据，对所述错误区域中的深度进行修正，以确定所述场景的被修正后的双目深度数据之前，调整所述TOF数据和/或所述双目深度数据的深度，以使得所述TOF数据与所述双目数据的系统深度误差小于第二预设值。
可选地，处理器1930用于：在所述根据所述TOF数据，对所述错误区域中的深度进行修正，以确定所述场景的被修正后的双目深度数据之前，调整所述TOF数据和/或所述双目深度数据的相对位置，以使得所述TOF数据与所述双目数据的系统位置误差小于第三预设值。
本申请实施例还提供一种图像处理装置，包括：获取模块，用于获取场景的双目图像数据，以及所述场景的飞行时间TOF数据，所述双目图像数据包括根据不同摄像头得到的第一图像数据和第二图像数据；修正模块，用于根据所述TOF数据修正所述双目深度数据以得到被修正后的双目深度数据，所述双目深度数据是根据所述双目图像数据确定的；虚化处理模块，用于根据所述被修正后的双目深度数据，对所述第一图像数据进行虚化处理，其中所述第一图像数据为待显示的图像数据。
在一种实施方式中，所述修正模块包括确定单元，用于根据所述双目图像数据和双目深度数据，确定错误区域；修正单元，用于根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据。
在一种实施方式中，所述确定单元具体用于：根据所述双目深度数据，以及所述第一图像数据或所述第二图像数据，确定所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域。
在一种实施方式中，所述错误区域包括第一错误区域；所述双目深度数据包括双目稠 密深度数据；所述确定单元具体用于，根据所述双目稠密深度数据和所述第一图像数据，确定所述第一错误区域，所述第一错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；所述修正单元用于，根据所述TOF数据，对所述双目稠密深度数据的所述第一错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
在一种实施方式中，所述修正单元用于：将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；对所述第一错误区域的所述深度进行稠密化处理；将所述双目稠密深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度，以获得所述被修正后的双目深度数据。
在一种实施方式中，所述错误区域包括第一错误区域和第二错误区域；所述第二错误区域包括所述第一错误区域之外的部分区域；所述双目深度数据包括双目稠密深度数据；所述确定单元用于，根据所述双目稠密深度数据和所述第一图像数据，确定所述第一错误区域，所述第一错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；所述修正单元用于，根据所述TOF数据，对所述双目稠密深度数据的所述第一错误区域的深度进行修正，以确定第一修正稠密深度数据；所述确定单元还用于，根据所述第一修正稠密深度数据、所述TOF数据、所述第一图像数据，确定所述第二错误区域，所述第二错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；所述修正单元还用于，根据所述TOF数据，对所述第一修正稠密深度数据的所述第二错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
在一种实施方式中，所述修正单元用于：将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；对所述第一错误区域的所述深度进行稠密化处理；将所述双目稠密深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度，以获得所述第一修正稠密深度数据。
在一种实施方式中，所述错误区域包括第一错误区域；所述双目深度数据包括双目稀疏深度数据；所述确定单元用于，根据所述双目稀疏深度数据、所述第一图像数据和所述第二图像数据，确定所述第一错误区域，所述第一错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；所述修正单元用于，根据所述TOF数据，对所述双目稀疏深度数据的所述第一错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
在一种实施方式中，所述修正单元用于：将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；将所述双目稀疏深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度；对所述第一错误区域和所述第一错误区域之外的区域的深度进行稠密化处理，以获得所述被修正后的双目深度数据。
在一种实施方式中，所述错误区域包括第一错误区域和第二错误区域；所述第二错误区域包括所述第一错误区域之外的部分区域；所述双目深度数据包括双目稀疏深度数据；所述确定单元用于，根据所述双目稀疏深度数据、所述第一图像数据、所述第二图像数据，确定所述第一错误区域，所述第一错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；所述修正单元用于，根据所 述TOF数据，对所述双目稀疏深度数据的所述第一错误区域的深度进行修正，确定第一修正稠密深度数据；所述确定单元还用于，根据所述第一修正稠密深度数据、所述TOF数据、所述第一图像数据，确定所述第二错误区域，所述第二错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；所述修正单元还用于，根据所述TOF数据，对所述第一修正稠密深度数据的所述第二错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
在一种实施方式中，所述修正单元用于：将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；将所述双目稀疏深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度；对所述第一错误区域和所述第一错误区域之外的区域的深度进行稠密化处理，以获得所述第一修正稠密深度数据。
在一种实施方式中，所述修正单元用于：将所述TOF数据对应的所述第二错误区域的深度作为所述第二错误区域的深度；对所述第二错误区域的所述深度进行稠密化处理；将所述第一修正稠密深度数据对应的所述第二错误区域之外的区域的深度作为对应的所述第二错误区域之外的区域的深度，以获得所述被修正后的双目深度数据。
在一种实施方式中，所述第二错误区域包括所述第一修正稠密深度数据中的深度跳变区域。
在一种实施方式中，所述第一错误区域包括以下区域中至少一种：重复纹理区域、色差小于第一预设值的区域或深度渐变区域。
在一种实施方式中，还包括：第一调整模块，用于在所述根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据之前，调整所述TOF数据和/或所述双目深度数据的深度，以使得所述TOF数据与所述双目数据的系统深度误差小于第二预设值。
在一种实施方式中，还包括：第二调整模块，用于在所述根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据之前，调整所述TOF数据和/或所述双目深度数据的相对位置，以使得所述TOF数据与所述双目数据的系统位置误差小于第三预设值。
本申请实施例还提供一种图像处理装置，包括：存储器，用于存储代码；处理器，用于读取所述存储器中的代码，以执行前文中的方法。
本申请实施例还提供一种计算机程序存储介质，所述计算机程序存储介质具有程序指令，当所述程序指令被执行时，使得前文中的方法被执行。
本申请实施例还提供一种芯片系统，所述芯片系统包括至少一个处理器，当程序指令在所述至少一个处理器中执行时，使得前文中的方法被执行。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
本申请实施例中，“至少一个”是指一个或者多个，“多个”是指两个或两个以上。“和/或”，描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示单 独存在A、同时存在A和B、单独存在B的情况。其中A，B可以是单数或者复数。字符“/”一般表示前后关联对象是一种“或”的关系。“以下至少一项”及其类似表达，是指的这些项中的任意组合，包括单项或复数项的任意组合。例如，a，b和c中的至少一项可以表示：a,b,c,a-b,a-c,b-c,或a-b-c，其中a,b,c可以是单个，也可以是多个。所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统、装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。
所述功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(read-only memory，ROM)、随机存取存储器(random access memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准。

Claims (65)

  1. 一种图像处理的方法，其特征在于，包括：
    获取场景的双目图像数据，以及所述场景的飞行时间TOF数据，所述双目图像数据包括根据不同摄像头得到的第一图像数据和第二图像数据；
    根据所述双目图像数据和双目深度数据，确定错误区域，所述双目深度数据是根据所述双目图像数据确定的；
    根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据；
    根据所述被修正后的双目深度数据，对所述第一图像数据进行虚化处理，其中所述第一图像数据为待显示的图像数据。
  2. 根据权利要求1所述的方法，其特征在于，所述根据双目图像数据和双目深度数据，确定错误区域的步骤，包括：
    根据所述双目深度数据，以及所述第一图像数据或所述第二图像数据，确定所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域。
  3. 根据权利要求2所述的方法，其特征在于，所述错误区域包括第一错误区域；所述双目深度数据包括双目稠密深度数据；
    所述根据所述双目深度数据，以及所述第一图像数据或所述第二图像数据，确定所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域，包括：
    根据所述双目稠密深度数据和所述第一图像数据，确定所述第一错误区域，所述第一错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；
    所述根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据，包括：根据所述TOF数据，对所述双目稠密深度数据的所述第一错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
  4. 根据权利要求3所述的方法，其特征在于，所述根据所述TOF数据，对所述双目稠密深度数据的所述第一错误区域的深度进行修正，以确定被修正后的双目深度数据，包括：
    将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；
    对所述第一错误区域的所述深度进行稠密化处理；
    将所述双目稠密深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度，以获得所述被修正后的双目深度数据。
  5. 根据权利要求2所述的方法，其特征在于，所述错误区域包括第一错误区域和第二错误区域；所述第二错误区域包括所述第一错误区域之外的部分区域；所述双目深度数据包括双目稠密深度数据；
    所述根据所述双目深度数据，以及所述第一图像数据或所述第二图像数据，确定所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域，包括：
    根据所述双目稠密深度数据和所述第一图像数据，确定所述第一错误区域，所述第一 错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；
    所述根据TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据，包括：
    根据所述TOF数据，对所述双目稠密深度数据的所述第一错误区域的深度进行修正，以确定第一修正稠密深度数据；
    根据所述第一修正稠密深度数据、所述TOF数据、所述第一图像数据，确定所述第二错误区域，所述第二错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；
    根据所述TOF数据，对所述第一修正稠密深度数据的所述第二错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
  6. 根据权利要求5所述的方法，其特征在于，所述根据所述TOF数据，对所述双目稠密深度数据的所述第一错误区域的深度进行修正，以确定第一修正稠密深度数据，包括：
    将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；
    对所述第一错误区域的所述深度进行稠密化处理；
    将所述双目稠密深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度，以获得所述第一修正稠密深度数据。
  7. 根据权利要求2所述的方法，其特征在于，所述错误区域包括第一错误区域；所述双目深度数据包括双目稀疏深度数据；
    所述根据所述双目深度数据，以及所述第一图像数据或所述第二图像数据，确定所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域，包括：
    根据所述双目稀疏深度数据、所述第一图像数据和所述第二图像数据，确定所述第一错误区域，所述第一错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；
    所述根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据，包括：根据所述TOF数据，对所述双目稀疏深度数据的所述第一错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
  8. 根据权利要求7所述的方法，其特征在于，所述根据所述TOF数据，对所述双目稀疏深度数据的所述第一错误区域的深度进行修正，以确定所述被修正后的双目深度数据，包括：
    将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；
    将所述双目稀疏深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度；
    对所述第一错误区域和所述第一错误区域之外的区域的深度进行稠密化处理，以获得所述被修正后的双目深度数据。
  9. 根据权利要求2所述的方法，其特征在于，所述错误区域包括第一错误区域和第二错误区域；所述第二错误区域包括所述第一错误区域之外的部分区域；所述双目深度数据包括双目稀疏深度数据；
    所述根据所述双目深度数据，以及所述第一图像数据或所述第二图像数据，确定所述 双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域，包括：
    根据所述双目稀疏深度数据、所述第一图像数据、所述第二图像数据，确定所述第一错误区域，所述第一错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；
    所述根据TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据，包括：
    根据所述TOF数据，对所述双目稀疏深度数据的所述第一错误区域的深度进行修正，以确定第一修正稠密深度数据；
    根据所述第一修正稠密深度数据、所述TOF数据、所述第一图像数据，确定所述第二错误区域，所述第二错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；
    根据所述TOF数据和所述第一修正稠密深度数据，对所述第二错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
  10. 根据权利要求9所述的方法，其特征在于，所述根据所述TOF数据，对所述双目稀疏深度数据的所述第一错误区域的深度进行修正，以确定第一修正稠密深度数据，包括：
    将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；
    将所述双目稀疏深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度；
    对所述第一错误区域和所述第一错误区域之外的区域的深度进行稠密化处理，以获得所述第一修正稠密深度数据。
  11. 根据权利要求5、6、9、或10中任一项所述的方法，其特征在于，所述根据所述TOF数据，对所述第一修正稠密深度数据的所述第二错误区域的深度进行修正，以确定所述被修正后的双目深度数据，包括：
    将所述TOF数据对应的所述第二错误区域的深度作为所述第二错误区域的深度；
    对所述第二错误区域的所述深度进行稠密化处理；
    将所述第一修正稠密深度数据对应的所述第二错误区域之外的区域的深度作为对应的所述第二错误区域之外的区域的深度，以获得所述被修正后的双目深度数据。
  12. 根据权利要求5、6、或9-11中任一项所述的方法，其特征在于，所述第二错误区域包括所述第一修正稠密深度数据中的深度跳变区域。
  13. 根据权利要求3-12中任一项所述的方法，其特征在于，所述第一错误区域包括以下区域中至少一种：重复纹理区域、色差小于第一预设值的区域或深度渐变区域。
  14. 根据权利要求1-13中任一项所述的方法，其特征在于，在所述根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据之前，还包括：
    调整所述TOF数据和/或所述双目深度数据的深度，以使得所述TOF数据与所述双目数据的系统深度误差小于第二预设值。
  15. 根据权利要求1-14中任一项所述的方法，其特征在于，在所述根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据之前，还包括：
    调整所述TOF数据和/或所述双目深度数据的相对位置，以使得所述TOF数据与所述双目数据的系统位置误差小于第三预设值。
  16. 一种图像处理装置，其特征在于，包括：
    获取模块，用于获取场景的双目图像数据，以及所述场景的飞行时间TOF数据，所述双目图像数据包括根据不同摄像头得到的第一图像数据和第二图像数据；
    确定模块，用于根据所述双目图像数据和双目深度数据，确定错误区域；所述双目深度数据是根据所述双目图像数据确定的；
    修正模块，用于根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据；
    虚化处理模块，用于根据所述被修正后的双目深度数据，对所述第一图像数据进行虚化处理，其中所述第一图像数据为待显示的图像数据。
  17. 根据权利要求16所述的图像处理装置，其特征在于，所述确定模块具体用于：
    根据所述双目深度数据，以及所述第一图像数据或所述第二图像数据，确定所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域。
  18. 根据权利要求17所述的图像处理装置，其特征在于，所述错误区域包括第一错误区域；所述双目深度数据包括双目稠密深度数据；
    所述确定模块具体用于，根据所述双目稠密深度数据和所述第一图像数据，确定所述第一错误区域，所述第一错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；
    所述修正模块用于，根据所述TOF数据，对所述双目稠密深度数据的所述第一错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
  19. 根据权利要求18所述的图像处理装置，其特征在于，所述修正模块用于：
    将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；
    对所述第一错误区域的所述深度进行稠密化处理；
    将所述双目稠密深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度，以获得所述被修正后的双目深度数据。
  20. 根据权利要求17所述的图像处理装置，其特征在于，所述错误区域包括第一错误区域和第二错误区域；所述第二错误区域包括所述第一错误区域之外的部分区域；所述双目深度数据包括双目稠密深度数据；
    所述确定模块用于，根据所述双目稠密深度数据和所述第一图像数据，确定所述第一错误区域，所述第一错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；
    所述修正模块用于，根据所述TOF数据，对所述双目稠密深度数据的所述第一错误区域的深度进行修正，以确定第一修正稠密深度数据；
    所述确定模块还用于，根据所述第一修正稠密深度数据、所述TOF数据、所述第一图像数据，确定所述第二错误区域，所述第二错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；
    所述修正模块还用于，根据所述TOF数据，对所述第一修正稠密深度数据的所述第二错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
  21. 根据权利要求20所述的图像处理装置，其特征在于，所述修正模块用于：
    将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；
    对所述第一错误区域的所述深度进行稠密化处理；
    将所述双目稠密深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度，以获得所述第一修正稠密深度数据。
  22. 根据权利要求17所述的图像处理装置，其特征在于，所述错误区域包括第一错误区域；所述双目深度数据包括双目稀疏深度数据；
    所述确定模块用于，根据所述双目稀疏深度数据、所述第一图像数据和所述第二图像数据，确定所述第一错误区域，所述第一错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；
    所述修正模块用于，根据所述TOF数据，对所述双目稀疏深度数据的所述第一错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
  23. 根据权利要求22所述的图像处理装置，其特征在于，所述修正模块用于：
    将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；
    将所述双目稀疏深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度；
    对所述第一错误区域和所述第一错误区域之外的区域的深度进行稠密化处理，以获得所述被修正后的双目深度数据。
  24. 根据权利要求17所述的图像处理装置，其特征在于，所述错误区域包括第一错误区域和第二错误区域；所述第二错误区域包括所述第一错误区域之外的部分区域；所述双目深度数据包括双目稀疏深度数据；
    所述确定模块用于，根据所述双目稀疏深度数据、所述第一图像数据、所述第二图像数据，确定所述第一错误区域，所述第一错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；
    所述修正模块用于，根据所述TOF数据，对所述双目稀疏深度数据的所述第一错误区域的深度进行修正，确定第一修正稠密深度数据；
    所述确定模块还用于，根据所述第一修正稠密深度数据、所述TOF数据、所述第一图像数据，确定所述第二错误区域，所述第二错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；
    所述修正模块还用于，根据所述TOF数据，对所述第一修正稠密深度数据的所述第二错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
  25. 根据权利要求24所述的图像处理装置，其特征在于，所述修正模块用于：
    将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；
    将所述双目稀疏深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度；
    对所述第一错误区域和所述第一错误区域之外的区域的深度进行稠密化处理，以获得所述第一修正稠密深度数据。
  26. 根据权利要求20、21、24、或25中任一项所述的方法，其特征在于，所述修正模块用于：
    将所述TOF数据对应的所述第二错误区域的深度作为所述第二错误区域的深度；
    对所述第二错误区域的所述深度进行稠密化处理；
    将所述第一修正稠密深度数据对应的所述第二错误区域之外的区域的深度作为对应 的所述第二错误区域之外的区域的深度，以获得所述被修正后的双目深度数据。
  27. 根据权利要求20、21、或24-26中任一项所述的图像处理装置，其特征在于，所述第二错误区域包括所述第一修正稠密深度数据中的深度跳变区域。
  28. 根据权利要求18-27中任一项所述的图像处理装置，其特征在于，所述第一错误区域包括以下区域中至少一种：重复纹理区域、色差小于第一预设值的区域或深度渐变区域。
  29. 根据权利要求16-28中任一项所述的图像处理装置，其特征在于，还包括：
    第一调整模块，用于在所述根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据之前，调整所述TOF数据和/或所述双目深度数据的深度，以使得所述TOF数据与所述双目数据的系统深度误差小于第二预设值。
  30. 根据权利要求16-29中任一项所述的图像处理装置，其特征在于，还包括：
    第二调整模块，用于在所述根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据之前，调整所述TOF数据和/或所述双目深度数据的相对位置，以使得所述TOF数据与所述双目数据的系统位置误差小于第三预设值。
  31. 一种图像处理的方法，其特征在于，包括：
    获取场景的双目图像数据，以及所述场景的飞行时间TOF数据，所述双目图像数据包括根据不同摄像头得到的第一图像数据和第二图像数据；
    根据所述TOF数据修正所述双目深度数据以得到被修正后的双目深度数据，所述双目深度数据是根据所述双目图像数据确定的；
    根据所述被修正后的双目深度数据，对所述第一图像数据进行虚化处理，其中所述第一图像数据为待显示的图像数据。
  32. 根据权利要求31所述的方法，其特征在于，所述根据所述TOF数据修正所述双目深度数据以得到被修正后的双目深度数据包括：
    根据所述双目图像数据和双目深度数据，确定错误区域；
    根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据。
  33. 根据权利要求32所述的方法，其特征在于，所述根据双目图像数据和双目深度数据，确定错误区域的步骤，包括：
    根据所述双目深度数据，以及所述第一图像数据或所述第二图像数据，确定所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域。
  34. 根据权利要求33所述的方法，其特征在于，所述错误区域包括第一错误区域；所述双目深度数据包括双目稠密深度数据；
    所述根据所述双目深度数据，以及所述第一图像数据或所述第二图像数据，确定所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域，包括：
    根据所述双目稠密深度数据和所述第一图像数据，确定所述第一错误区域，所述第一错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；
    所述根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目 深度数据，包括：根据所述TOF数据，对所述双目稠密深度数据的所述第一错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
  35. 根据权利要求34所述的方法，其特征在于，所述根据所述TOF数据，对所述双目稠密深度数据的所述第一错误区域的深度进行修正，以确定被修正后的双目深度数据，包括：
    将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；
    对所述第一错误区域的所述深度进行稠密化处理；
    将所述双目稠密深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度，以获得所述被修正后的双目深度数据。
  36. 根据权利要求33所述的方法，其特征在于，所述错误区域包括第一错误区域和第二错误区域；所述第二错误区域包括所述第一错误区域之外的部分区域；所述双目深度数据包括双目稠密深度数据；
    所述根据所述双目深度数据，以及所述第一图像数据或所述第二图像数据，确定所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域，包括：
    根据所述双目稠密深度数据和所述第一图像数据，确定所述第一错误区域，所述第一错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；
    所述根据TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据，包括：
    根据所述TOF数据，对所述双目稠密深度数据的所述第一错误区域的深度进行修正，以确定第一修正稠密深度数据；
    根据所述第一修正稠密深度数据、所述TOF数据、所述第一图像数据，确定所述第二错误区域，所述第二错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；
    根据所述TOF数据，对所述第一修正稠密深度数据的所述第二错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
  37. 根据权利要求36所述的方法，其特征在于，所述根据所述TOF数据，对所述双目稠密深度数据的所述第一错误区域的深度进行修正，以确定第一修正稠密深度数据，包括：
    将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；
    对所述第一错误区域的所述深度进行稠密化处理；
    将所述双目稠密深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度，以获得所述第一修正稠密深度数据。
  38. 根据权利要求33所述的方法，其特征在于，所述错误区域包括第一错误区域；所述双目深度数据包括双目稀疏深度数据；
    所述根据所述双目深度数据，以及所述第一图像数据或所述第二图像数据，确定所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域，包括：
    根据所述双目稀疏深度数据、所述第一图像数据和所述第二图像数据，确定所述第一 错误区域，所述第一错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；
    所述根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据，包括：根据所述TOF数据，对所述双目稀疏深度数据的所述第一错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
  39. 根据权利要求38所述的方法，其特征在于，所述根据所述TOF数据，对所述双目稀疏深度数据的所述第一错误区域的深度进行修正，以确定所述被修正后的双目深度数据，包括：
    将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；
    将所述双目稀疏深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度；
    对所述第一错误区域和所述第一错误区域之外的区域的深度进行稠密化处理，以获得所述被修正后的双目深度数据。
  40. 根据权利要求33所述的方法，其特征在于，所述错误区域包括第一错误区域和第二错误区域；所述第二错误区域包括所述第一错误区域之外的部分区域；所述双目深度数据包括双目稀疏深度数据；
    所述根据所述双目深度数据，以及所述第一图像数据或所述第二图像数据，确定所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域，包括：
    根据所述双目稀疏深度数据、所述第一图像数据、所述第二图像数据，确定所述第一错误区域，所述第一错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；
    所述根据TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据，包括：
    根据所述TOF数据，对所述双目稀疏深度数据的所述第一错误区域的深度进行修正，以确定第一修正稠密深度数据；
    根据所述第一修正稠密深度数据、所述TOF数据、所述第一图像数据，确定所述第二错误区域，所述第二错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；
    根据所述TOF数据和所述第一修正稠密深度数据，对所述第二错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
  41. 根据权利要求40所述的方法，其特征在于，所述根据所述TOF数据，对所述双目稀疏深度数据的所述第一错误区域的深度进行修正，以确定第一修正稠密深度数据，包括：
    将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；
    将所述双目稀疏深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度；
    对所述第一错误区域和所述第一错误区域之外的区域的深度进行稠密化处理，以获得所述第一修正稠密深度数据。
  42. 根据权利要求36、37、40、或41中任一项所述的方法，其特征在于，所述根据 所述TOF数据，对所述第一修正稠密深度数据的所述第二错误区域的深度进行修正，以确定所述被修正后的双目深度数据，包括：
    将所述TOF数据对应的所述第二错误区域的深度作为所述第二错误区域的深度；
    对所述第二错误区域的所述深度进行稠密化处理；
    将所述第一修正稠密深度数据对应的所述第二错误区域之外的区域的深度作为对应的所述第二错误区域之外的区域的深度，以获得所述被修正后的双目深度数据。
  43. 根据权利要求36、37、或40-42中任一项所述的方法，其特征在于，所述第二错误区域包括所述第一修正稠密深度数据中的深度跳变区域。
  44. 根据权利要求34-43中任一项所述的方法，其特征在于，所述第一错误区域包括以下区域中至少一种：重复纹理区域、色差小于第一预设值的区域或深度渐变区域。
  45. 根据权利要求31-44中任一项所述的方法，其特征在于，在所述根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据之前，还包括：
    调整所述TOF数据和/或所述双目深度数据的深度，以使得所述TOF数据与所述双目数据的系统深度误差小于第二预设值。
  46. 根据权利要求31-45中任一项所述的方法，其特征在于，在所述根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据之前，还包括：
    调整所述TOF数据和/或所述双目深度数据的相对位置，以使得所述TOF数据与所述双目数据的系统位置误差小于第三预设值。
  47. 一种图像处理装置，其特征在于，包括：
    获取模块，用于获取场景的双目图像数据，以及所述场景的飞行时间TOF数据，所述双目图像数据包括根据不同摄像头得到的第一图像数据和第二图像数据；
    修正模块，用于根据所述TOF数据修正所述双目深度数据以得到被修正后的双目深度数据，所述双目深度数据是根据所述双目图像数据确定的；以及
    虚化处理模块，用于根据所述被修正后的双目深度数据，对所述第一图像数据进行虚化处理，其中所述第一图像数据为待显示的图像数据。
  48. 根据权利要求47所述的图像处理装置，其特征在于，所述修正模块包括：
    确定单元，用于根据所述双目图像数据和双目深度数据，确定错误区域；以及
    修正单元，用于根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据。
  49. 根据权利要求48所述的图像处理装置，其特征在于，所述确定单元具体用于：
    根据所述双目深度数据，以及所述第一图像数据或所述第二图像数据，确定所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域。
  50. 根据权利要求49所述的图像处理装置，其特征在于，所述错误区域包括第一错误区域；所述双目深度数据包括双目稠密深度数据；
    所述确定单元具体用于，根据所述双目稠密深度数据和所述第一图像数据，确定所述第一错误区域，所述第一错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；
    所述修正单元用于，根据所述TOF数据，对所述双目稠密深度数据的所述第一错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
  51. 根据权利要求50所述的图像处理装置，其特征在于，所述修正单元用于：
    将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；
    对所述第一错误区域的所述深度进行稠密化处理；
    将所述双目稠密深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度，以获得所述被修正后的双目深度数据。
  52. 根据权利要求49所述的图像处理装置，其特征在于，所述错误区域包括第一错误区域和第二错误区域；所述第二错误区域包括所述第一错误区域之外的部分区域；所述双目深度数据包括双目稠密深度数据；
    所述确定单元用于，根据所述双目稠密深度数据和所述第一图像数据，确定所述第一错误区域，所述第一错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；
    所述修正单元用于，根据所述TOF数据，对所述双目稠密深度数据的所述第一错误区域的深度进行修正，以确定第一修正稠密深度数据；
    所述确定单元还用于，根据所述第一修正稠密深度数据、所述TOF数据、所述第一图像数据，确定所述第二错误区域，所述第二错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；
    所述修正单元还用于，根据所述TOF数据，对所述第一修正稠密深度数据的所述第二错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
  53. 根据权利要求52所述的图像处理装置，其特征在于，所述修正单元用于：
    将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；
    对所述第一错误区域的所述深度进行稠密化处理；
    将所述双目稠密深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度，以获得所述第一修正稠密深度数据。
  54. 根据权利要求49所述的图像处理装置，其特征在于，所述错误区域包括第一错误区域；所述双目深度数据包括双目稀疏深度数据；
    所述确定单元用于，根据所述双目稀疏深度数据、所述第一图像数据和所述第二图像数据，确定所述第一错误区域，所述第一错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；
    所述修正单元用于，根据所述TOF数据，对所述双目稀疏深度数据的所述第一错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
  55. 根据权利要求54所述的图像处理装置，其特征在于，所述修正单元用于：
    将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；
    将所述双目稀疏深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度；
    对所述第一错误区域和所述第一错误区域之外的区域的深度进行稠密化处理，以获得所述被修正后的双目深度数据。
  56. 根据权利要求49所述的图像处理装置，其特征在于，所述错误区域包括第一错误区域和第二错误区域；所述第二错误区域包括所述第一错误区域之外的部分区域；所述双目深度数据包括双目稀疏深度数据；
    所述确定单元用于，根据所述双目稀疏深度数据、所述第一图像数据、所述第二图像 数据，确定所述第一错误区域，所述第一错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；
    所述修正单元用于，根据所述TOF数据，对所述双目稀疏深度数据的所述第一错误区域的深度进行修正，确定第一修正稠密深度数据；
    所述确定单元还用于，根据所述第一修正稠密深度数据、所述TOF数据、所述第一图像数据，确定所述第二错误区域，所述第二错误区域为所述双目深度数据中精度或稳定性中的至少一项不满足预设范围的区域对应的所述第一图像数据的区域；
    所述修正单元还用于，根据所述TOF数据，对所述第一修正稠密深度数据的所述第二错误区域的深度进行修正，以确定所述被修正后的双目深度数据。
  57. 根据权利要求56所述的图像处理装置，其特征在于，所述修正单元用于：
    将所述TOF数据对应的所述第一错误区域的深度作为所述第一错误区域的深度；
    将所述双目稀疏深度数据对应的所述第一错误区域之外的区域的深度作为对应的所述第一错误区域之外的区域的深度；
    对所述第一错误区域和所述第一错误区域之外的区域的深度进行稠密化处理，以获得所述第一修正稠密深度数据。
  58. 根据权利要求52、53、56、或57中任一项所述的方法，其特征在于，所述修正单元用于：
    将所述TOF数据对应的所述第二错误区域的深度作为所述第二错误区域的深度；
    对所述第二错误区域的所述深度进行稠密化处理；
    将所述第一修正稠密深度数据对应的所述第二错误区域之外的区域的深度作为对应的所述第二错误区域之外的区域的深度，以获得所述被修正后的双目深度数据。
  59. 根据权利要求52、53、或56-58中任一项所述的图像处理装置，其特征在于，所述第二错误区域包括所述第一修正稠密深度数据中的深度跳变区域。
  60. 根据权利要求50-59中任一项所述的图像处理装置，其特征在于，所述第一错误区域包括以下区域中至少一种：重复纹理区域、色差小于第一预设值的区域或深度渐变区域。
  61. 根据权利要求47-60中任一项所述的图像处理装置，其特征在于，还包括：
    第一调整模块，用于在所述根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据之前，调整所述TOF数据和/或所述双目深度数据的深度，以使得所述TOF数据与所述双目数据的系统深度误差小于第二预设值。
  62. 根据权利要求47-61中任一项所述的图像处理装置，其特征在于，还包括：
    第二调整模块，用于在所述根据所述TOF数据，对所述错误区域中的深度进行修正，以确定被修正后的双目深度数据之前，调整所述TOF数据和/或所述双目深度数据的相对位置，以使得所述TOF数据与所述双目数据的系统位置误差小于第三预设值。
  63. 一种图像处理装置，其特征在于，包括：
    存储器，用于存储代码；
    处理器，用于读取所述存储器中的代码，以执行如权利要求1至15中任一项所述的方法，或执行如权利要求31至46中任一项所述的方法。
  64. 一种计算机程序存储介质，其特征在于，所述计算机程序存储介质具有程序指令， 当所述程序指令被执行时，使得如权利要求1至15中任一项所述的方法被执行，或执行如权利要求31至46中任一项所述的方法。
  65. 一种终端设备，其特征在于，包括：
    双目系统，用于采集双目图像数据；
    飞行时间TOF器件，用于采集TOF数据；
    至少一个处理器，当程序指令被所述至少一个处理器中执行时，使得如权利要求1至15中任一项所述的方法，或如权利要求31至46中任一项所述的方法被执行。
PCT/CN2019/127944 2019-03-25 2019-12-24 一种基于Dual Camera+TOF的大光圈虚化方法 WO2020192209A1 (zh)

Priority Applications (3)

Application Number Priority Date Filing Date Title
CN201980096705.9A CN114365482A (zh) 2019-03-25 2019-12-24 一种基于Dual Camera+TOF的大光圈虚化方法
EP19921752.2A EP3930321A4 (en) 2019-03-25 2019-12-24 DUAL CAMERA + TOF LARGE APERTURE BLUR PROCESS
US17/484,093 US12096134B2 (en) 2019-03-25 2021-09-24 Big aperture blurring method based on dual cameras and TOF

Applications Claiming Priority (6)

Application Number Priority Date Filing Date Title
CN201910229288.2 2019-03-25
CN201910229288 2019-03-25
CN201910330861.9 2019-04-23
CN201910330861.9A CN111741283A (zh) 2019-03-25 2019-04-23 图像处理的装置和方法
CN201910749592.XA CN111741284A (zh) 2019-03-25 2019-08-14 图像处理的装置和方法
CN201910749592.X 2019-08-14

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/484,093 Continuation US12096134B2 (en) 2019-03-25 2021-09-24 Big aperture blurring method based on dual cameras and TOF

Publications (1)

Publication Number Publication Date
WO2020192209A1 true WO2020192209A1 (zh) 2020-10-01

Family

ID=72609647

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/127944 WO2020192209A1 (zh) 2019-03-25 2019-12-24 一种基于Dual Camera+TOF的大光圈虚化方法

Country Status (4)

Country Link
US (1) US12096134B2 (zh)
EP (1) EP3930321A4 (zh)
CN (1) CN114365482A (zh)
WO (1) WO2020192209A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113052890A (zh) * 2021-03-31 2021-06-29 奥比中光科技集团股份有限公司 一种深度真值获取方法、装置、系统及深度相机
CN117560480A (zh) * 2024-01-09 2024-02-13 荣耀终端有限公司 一种图像深度估计方法及电子设备

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11546568B1 (en) * 2020-03-06 2023-01-03 Nvidia Corporation View synthesis for dynamic scenes
JP2021150738A (ja) * 2020-03-17 2021-09-27 キヤノン株式会社 分割パターン決定装置、及び、それを用いた画像符号化装置、及び、学習装置、並びに、分割パターン決定装置及び学習装置の制御方法、及び、プログラム
US11449968B2 (en) * 2020-12-31 2022-09-20 Samsung Electronics Co., Ltd. System and method for synthetic depth-of-field effect rendering for videos
US20220270273A1 (en) * 2021-02-19 2022-08-25 Advanced Micro Devices, Inc. Machine learning-based object-centric approach to image manipulation
US20230196504A1 (en) * 2021-12-17 2023-06-22 Memery Inc. Augmented reality alignment with a global positioning system and mobile device
WO2024113023A1 (en) * 2022-12-01 2024-06-06 Axiiio Pty Ltd Enhanced image capture
CN116704572B (zh) * 2022-12-30 2024-05-28 荣耀终端有限公司 一种基于深度摄像头的眼动追踪方法和装置
CN116990830B (zh) * 2023-09-27 2023-12-29 锐驰激光(深圳)有限公司 基于双目和tof的距离定位方法、装置、电子设备及介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140112574A1 (en) * 2012-10-23 2014-04-24 Electronics And Telecommunications Research Institute Apparatus and method for calibrating depth image based on relationship between depth sensor and color camera
CN106772431A (zh) * 2017-01-23 2017-05-31 杭州蓝芯科技有限公司 一种结合tof技术和双目视觉的深度信息获取装置及其方法
CN108234858A (zh) * 2017-05-19 2018-06-29 深圳市商汤科技有限公司 图像虚化处理方法、装置、存储介质及电子设备
CN109040556A (zh) * 2018-08-22 2018-12-18 Oppo广东移动通信有限公司 成像装置及电子设备
CN109274957A (zh) * 2018-10-31 2019-01-25 维沃移动通信有限公司 一种深度图像拍摄方法及移动终端

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP4016981A3 (en) * 2013-12-24 2022-09-21 Sony Depthsensing Solutions A time-of-flight camera system
CN106612387B (zh) 2015-10-15 2019-05-21 杭州海康威视数字技术股份有限公司 一种组合深度图获得方法及深度相机
CN105869167A (zh) 2016-03-30 2016-08-17 天津大学 基于主被动融合的高分辨率深度图获取方法
US10762651B2 (en) * 2016-09-30 2020-09-01 Magic Leap, Inc. Real time calibration for time-of-flight depth measurement
CN106993112B (zh) 2017-03-09 2020-01-10 Oppo广东移动通信有限公司 基于景深的背景虚化方法及装置和电子装置
US10755428B2 (en) * 2017-04-17 2020-08-25 The United States Of America, As Represented By The Secretary Of The Navy Apparatuses and methods for machine vision system including creation of a point cloud model and/or three dimensional model
CN110476185B (zh) * 2017-06-02 2023-04-04 上海科技大学 景深信息估算方法和装置
KR101889886B1 (ko) * 2017-12-22 2018-08-21 세명대학교 산학협력단 심도 정보 생성 방법 및 장치
JP7418340B2 (ja) * 2018-03-13 2024-01-19 マジック リープ， インコーポレイテッド 機械学習を使用した画像増強深度感知
CN109615652B (zh) 2018-10-23 2020-10-27 西安交通大学 一种深度信息获取方法及装置
EP3876528A4 (en) * 2018-11-02 2022-02-23 Guangdong Oppo Mobile Telecommunications Corp., Ltd. METHOD FOR TAKING DEPTH IMAGE, DEVICE FOR TAKING DEPTH IMAGE, AND ELECTRONIC DEVICE
US10860889B2 (en) * 2019-01-11 2020-12-08 Google Llc Depth prediction from dual pixel images

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140112574A1 (en) * 2012-10-23 2014-04-24 Electronics And Telecommunications Research Institute Apparatus and method for calibrating depth image based on relationship between depth sensor and color camera
CN106772431A (zh) * 2017-01-23 2017-05-31 杭州蓝芯科技有限公司 一种结合tof技术和双目视觉的深度信息获取装置及其方法
CN108234858A (zh) * 2017-05-19 2018-06-29 深圳市商汤科技有限公司 图像虚化处理方法、装置、存储介质及电子设备
CN109040556A (zh) * 2018-08-22 2018-12-18 Oppo广东移动通信有限公司 成像装置及电子设备
CN109274957A (zh) * 2018-10-31 2019-01-25 维沃移动通信有限公司 一种深度图像拍摄方法及移动终端

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113052890A (zh) * 2021-03-31 2021-06-29 奥比中光科技集团股份有限公司 一种深度真值获取方法、装置、系统及深度相机
CN117560480A (zh) * 2024-01-09 2024-02-13 荣耀终端有限公司 一种图像深度估计方法及电子设备
CN117560480B (zh) * 2024-01-09 2024-05-31 荣耀终端有限公司 一种图像深度估计方法及电子设备

Also Published As

Publication number Publication date
EP3930321A4 (en) 2022-04-20
EP3930321A1 (en) 2021-12-29
CN114365482A (zh) 2022-04-15
US20220086360A1 (en) 2022-03-17
US12096134B2 (en) 2024-09-17

Similar Documents

Publication Publication Date Title
WO2020192209A1 (zh) 一种基于Dual Camera+TOF的大光圈虚化方法
WO2020168956A1 (zh) 一种拍摄月亮的方法和电子设备
WO2021136050A1 (zh) 一种图像拍摄方法及相关装置
WO2021104485A1 (zh) 一种拍摄方法及电子设备
WO2022127787A1 (zh) 一种图像显示的方法及电子设备
CN111741284A (zh) 图像处理的装置和方法
WO2022100685A1 (zh) 一种绘制命令处理方法及其相关设备
WO2021180089A1 (zh) 界面切换方法、装置和电子设备
CN113797530B (zh) 图像的预测方法、电子设备和存储介质
CN112087649B (zh) 一种设备搜寻方法以及电子设备
WO2021057626A1 (zh) 图像处理方法、装置、设备及计算机存储介质
CN113542580B (zh) 去除眼镜光斑的方法、装置及电子设备
WO2022001258A1 (zh) 多屏显示方法、装置、终端设备及存储介质
CN113660408B (zh) 一种视频拍摄防抖方法与装置
WO2023284715A1 (zh) 一种物体重建方法以及相关设备
CN113452898A (zh) 一种拍照方法及装置
WO2022206589A1 (zh) 一种图像处理方法以及相关设备
WO2023005298A1 (zh) 基于多摄像头的图像内容屏蔽方法和装置
US20240193945A1 (en) Method for determining recommended scenario and electronic device
CN115150542B (zh) 一种视频防抖方法及相关设备
WO2024078275A1 (zh) 一种图像处理的方法、装置、电子设备及存储介质
CN116708751B (zh) 一种拍照时长的确定方法、装置和电子设备
CN114283195B (zh) 生成动态图像的方法、电子设备及可读存储介质
WO2023030067A1 (zh) 遥控方法、遥控设备和被控制设备
WO2022033344A1 (zh) 视频防抖方法、终端设备和计算机可读存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19921752

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2019921752

Country of ref document: EP

Effective date: 20210922