WO2022022136A1 - 深度图像生成方法及装置、参考图像生成方法及装置、电子设备以及计算机可读存储介质 - Google Patents

深度图像生成方法及装置、参考图像生成方法及装置、电子设备以及计算机可读存储介质 Download PDF

Info

Publication number
WO2022022136A1
WO2022022136A1 PCT/CN2021/100350 CN2021100350W WO2022022136A1 WO 2022022136 A1 WO2022022136 A1 WO 2022022136A1 CN 2021100350 W CN2021100350 W CN 2021100350W WO 2022022136 A1 WO2022022136 A1 WO 2022022136A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
target
depth
target object
effective pixel
Prior art date
Application number
PCT/CN2021/100350
Other languages
English (en)
French (fr)
Inventor
洪哲鸣
王军
王少鸣
郭润增
Original Assignee
腾讯科技（深圳）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 腾讯科技（深圳）有限公司 filed Critical 腾讯科技（深圳）有限公司
Priority to EP21849480.5A priority Critical patent/EP4033757A4/en
Priority to JP2022548830A priority patent/JP2023517830A/ja
Publication of WO2022022136A1 publication Critical patent/WO2022022136A1/zh
Priority to US17/736,899 priority patent/US20220262026A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/254—Image signal generators using stereoscopic image cameras in combination with electromagnetic radiation sources for illuminating objects
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B11/00—Measuring arrangements characterised by the use of optical techniques
    • G01B11/24—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures
    • G01B11/25—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object
    • G01B11/2513—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object with several lines being projected in more than one direction, e.g. grids, patterns
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/88—Lidar systems specially adapted for specific applications
    • G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
    • G01S7/497—Means for monitoring or calibrating
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • G06T7/337—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/521—Depth or shape recovery from laser ranging, e.g. using interferometry; from the projection of structured light
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/90—Determination of colour characteristics
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/271—Image signal generators wherein the generated image signals comprise depth maps or disparity maps
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10048—Infrared image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G06T2207/30201—Face
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30204—Marker
    • G06T2207/30208—Marker matrix

Definitions

  • the present application relates to the technical field of depth imaging, and in particular, to a depth image generation method, a depth image generation apparatus, a reference image generation method, a reference image generation method apparatus, an electronic device, and a computer-readable storage medium.
  • some depth images may have black borders, which leads to their application in some fields.
  • a face depth image can be introduced to improve the accuracy of the authentication result; however, if a part of the user's face is located in the black border of the above-mentioned depth image, the face authentication may fail.
  • a depth image generation method a depth image generation apparatus, a reference image generation method, a reference image generation method apparatus, an electronic device, and a computer-readable storage medium are provided.
  • a method for generating a depth image comprising: emitting structured light to a reference plane, and imaging the reference plane with first and second effective pixels of an image sensor to obtain a reference image; The structured light is emitted to a target object, and the first effective pixel is used to image the target object to obtain a target image; and a depth image of the target object is obtained according to the target image and the reference image.
  • a method for generating a depth image comprising: emitting structured light to a target object, and imaging the target object with a first effective pixel of an image sensor to obtain a target image; and The image and the reference image are used to obtain a depth image of the target object; wherein, the reference image is obtained by imaging a reference plane by using the first effective pixels and the second effective pixels of the image sensor.
  • a method for generating a reference image comprising: emitting structured light to a reference plane, and imaging the reference plane with first and second effective pixels of an image sensor to obtain a reference image;
  • the first effective pixel is an effective pixel used when performing depth imaging on the target object.
  • a depth image generation device comprising: a reference image generation module configured to emit structured light to a reference plane, and use first effective pixels and second effective pixels of an image sensor for the reference plane imaging to obtain a reference image; a target image acquisition module, used for emitting the structured light to a target object, and imaging the target object by using the first effective pixel to obtain a target image; and a depth image generation module, with to obtain a depth image of the target object according to the target image and the reference image.
  • a depth image generation device comprising: a target image acquisition module, configured to emit structured light to a target object, and use the first effective pixel of an image sensor to image the target object to obtain a target object an image; and a depth image generation module for obtaining a depth image of the target object according to the target image and the reference image; wherein the reference image is a pair of first and second effective pixels using the image sensor Obtained by reference plane imaging.
  • a reference image generation device comprising: a reference image generation module, configured to emit structured light to a reference plane, and use first effective pixels and second effective pixels of an image sensor for the reference plane imaging to obtain a reference image; wherein, the first effective pixel is an effective pixel used when performing depth imaging on the target object.
  • an electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute by executing the executable instructions The method of any of the above.
  • a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements any one of the methods described above.
  • a computer program product or computer program comprising computer instructions stored in a computer readable storage medium.
  • the processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the methods provided in the various optional implementations described above.
  • the first effective pixel is used, and the second effective pixel that is not used in the prior art is also used, so that the The size of the generated reference image can be increased.
  • the main reason for the black edge phenomenon is that the pixels at the edge of the target image do not have corresponding pixels in the reference image; however, in the depth image generation method provided by the example embodiments of this application, since the size of the reference image is increased, then Part or all of the pixels at the edge of the target image can be matched to the corresponding pixels in the reference image, thereby reducing or avoiding the occurrence of black borders in the depth image to a certain extent.
  • FIG. 1 shows a schematic structural diagram of a depth imaging system to which an embodiment of the present application can be applied
  • FIG. 2 schematically shows an application scene diagram of a depth image generation method according to an embodiment of the present application
  • FIG. 3 shows a schematic diagram of the principle of the generation of the black edge phenomenon according to an embodiment of the present application
  • FIG. 4 schematically shows a flowchart of a depth image generation method according to an embodiment of the present application
  • FIG. 5 schematically shows a block diagram of an image sensor according to an embodiment of the present application
  • FIG. 6 schematically shows a structural diagram of an image sensor according to an embodiment of the present application.
  • FIG. 7 schematically shows a flowchart of steps of obtaining a depth image according to an embodiment of the present application
  • FIG. 8 schematically shows a schematic diagram of a block matching method according to an embodiment of the present application.
  • FIG. 9 schematically shows a schematic diagram of calculating depth information corresponding to a target pixel according to an embodiment of the present application.
  • FIG. 10 schematically shows a schematic diagram of the generation principle of the black border phenomenon according to an embodiment of the present application.
  • FIG. 11 schematically shows a flowchart of a depth image generation method according to an embodiment of the present application
  • FIG. 12 schematically shows a block diagram of a depth image generating apparatus according to an embodiment of the present application.
  • FIG. 13 shows a schematic structural diagram of a computer system suitable for implementing the electronic device according to the embodiment of the present application.
  • Example embodiments will now be described more fully with reference to the accompanying drawings.
  • Example embodiments can be embodied in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art.
  • the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
  • numerous specific details are provided in order to give a thorough understanding of the embodiments of the present application.
  • those skilled in the art will appreciate that the technical solutions of the present application may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be employed.
  • well-known solutions have not been shown or described in detail to avoid obscuring aspects of the application.
  • FIG. 1 it is a schematic structural diagram of a depth imaging system 100 provided by the inventor.
  • the depth imaging system 100 shown in FIG. 1 mainly includes a structured light projection module 101, an image sensor 102 and a processing module 103; the depth imaging system can be used to perform depth imaging on the target object 104 to obtain a corresponding depth image; the target object 104 may be a human face or other object to be imaged.
  • the structured light projection module 101 may include a light source and optical components.
  • the light source may be a laser diode or a semiconductor laser, etc., or an edge-emitting laser, a vertical cavity surface laser transmitter, or a corresponding array laser, etc.; the wavelength of the light emitted from the light source may be infrared or ultraviolet.
  • the optical component is used to modulate the light beam emitted by the light source and then emit structured light; the optical component can be a refractive optical element, a diffractive optical element, or a combination of the two.
  • the structured light beam may be a structured light beam in the form of a speckle, a spot, a stripe, or a two-dimensional pattern or other encoding.
  • the image sensor 102 may be a charge-coupled device image sensor (Charge Coupled Device, CCD) or a complementary metal-oxide-semiconductor (Complementary Metal-Oxide-Semiconductor, CMOS) image sensor or the like.
  • CCD Charge Coupled Device
  • CMOS complementary metal-oxide-semiconductor
  • optical components such as a filter, a micro lens array (Micro Lens Array, MLA) and the like may also be provided.
  • the filter may be a Bayer filter or an infrared filter or the like.
  • the above-mentioned filter can be set to filter only the light beam with the wavelength ⁇ to pass through, thereby improving the quality of subsequent images.
  • the connection line between the structured light projection module 101 and the image sensor 102 is called the baseline, for example, the direction of the baseline may be the x-axis direction shown in FIG. 1 .
  • the optical axes of the structured light projection module 101 and the image sensor 102 may be parallel or may form a certain angle. In the present exemplary embodiment, the optical axes of the structured light projection module 101 and the image sensor 102 are arranged in parallel. With this arrangement, the computational complexity in the subsequent generation of a depth image can be simplified.
  • the processing module 103 may include one or more processors and one or more memories, and may be used to control the structured light projection module 101 and the image sensor 102 and receive and process related data.
  • the processor may include, for example, a digital signal processor (Digital Signal Processing, DSP), an application processor (Multimedia Application Processor, MAP), a Field-Programmable Gate Array (Field-Programmable Gate Array, FPGA), an application-specific integrated circuit (Application One or a combination of Specific Integrated Circuit, ASIC), etc.
  • the memory may include one of random access memory (Random Access Memory, RAM), read only memory (Read Only Memory, ROM), flash memory (Flash), etc. or combination.
  • the control and data processing instructions executed by the processing module 103 can be stored in the memory in the form of software, firmware, etc. and called by the processor when needed, or the instructions can be directly solidified into the circuit to form a dedicated circuit (or dedicated processor) to Executing the corresponding instructions can also be implemented in the form of a combination of software and dedicated circuits.
  • the processing module 103 may also include input/output interfaces, and/or network interfaces that support network communications.
  • the processed data may be transmitted to other devices or other units in the system, such as a display unit or an external terminal device, through an interface.
  • the above-mentioned structured light projection module 101 , image sensor 102 , and processing module 103 may be independently distributed, or may be partially or fully integrated into one electronic device.
  • the structured light projection module 101, the image sensor 102 and the processing module 103 can all be integrated into a smartphone, tablet computer, notebook computer or camera; with reference to FIG. 2, the structured light projection module 101, the image sensor 102 and the processing module 103 are all integrated in the smart phone 201 , and the target object 104 can be deeply imaged by the smart phone 201 .
  • the structured light projection module 101 and the image sensor 102 may be integrated into one electronic device, and the processing module 103 may be a cloud server or other third-party electronic devices.
  • the structured light projection module 101 can be used to emit structured light to the reference plane, and the image sensor 102 can be used to image the reference plane to obtain a reference image in the calibration stage before the terminal equipment leaves the factory.
  • the structured light projection module 101 emits the structured light to the target object, and uses the image sensor 102 to image the target object to obtain the target image; further, the relative value of each pixel in the target image to the reference image can be calculated. and calculate the depth information corresponding to each pixel according to the offset, so as to generate the depth image according to the depth information corresponding to each pixel.
  • FIG. 3 it is a schematic diagram of the principle of the black edge phenomenon:
  • the range of the outgoing light of the structured light projection module 101 is the area 301, and the range of the incident light of the image sensor 102 is the area 302; if the target object is located in the overlapping range of the area 301 and the area 302, a depth image can theoretically be generated.
  • a depth image can theoretically be generated.
  • the reference image 303 since there is a certain distance between the structured light projection module 101 and the image sensor 102, and the size of the reference image 303 is fixed, when the plane where the target object is located is far away from the reference plane, some pixels in the target image 304 may not be able to be seen in the reference plane.
  • the corresponding pixels in the image 303 are matched, so that the corresponding depth information cannot be determined; in the depth image, for the pixels whose corresponding depth information cannot be determined, 0 will be used to complement, that is, the pixel value of which the corresponding depth information cannot be determined is set to 0, which leads to the appearance of black borders.
  • the pixels in the area A of the target image 304 and the area A of the target image 305 in FIG. 3 can be matched to the corresponding pixels in the reference image 303, that is, the corresponding depth information can be determined;
  • the pixels in the B area of the target image 304 and the B area of the target image 305 do not exist in the image 303, that is, the corresponding depth information cannot be determined; therefore, in the final generated depth image, the B area of the target image 304 and the target image 305.
  • the corresponding positions of the B area are all black borders. Generally speaking, the closer the distance between the plane where the target object is and the reference plane, the smaller the range of the black border, and the farther the distance between the plane where the target object is and the reference plane, the larger the range of the black border.
  • the existence of the black edge phenomenon will affect the field of view (FOV, Field angle of View) during face authentication; at the same time, the size of the black edge in the depth image may change with the distance, which further increases. Uncertainty in the certification process. Specifically, when face authentication is performed at different distances, the black border area in the depth image is different, resulting in different effective areas for face authentication; and if the user's face is in the black border area of the depth image, it will affect the result of this face authentication, thereby reducing the user experience. In addition, the range of the black border area in the depth image will change with the distance, and it is difficult to design an effective strategy to avoid it in practical applications.
  • FOV Field angle of View
  • an example embodiment of the present application provides a new method for generating a depth image, which is described by taking the application of the method in an electronic device as an example.
  • the depth image generation method may include the following steps S410 to S430. in:
  • step S410 the structured light is emitted to the reference plane, and the reference plane is imaged by using the first effective pixel and the second effective pixel of the image sensor to obtain a reference image.
  • the structured light projection module 101 in the electronic device can be used to emit structured light beams in the form of encoding such as speckles, spots, stripes, or two-dimensional patterns to the reference plane.
  • Structured light is a projection of light in a known spatial direction. gather.
  • the reference plane can be, for example, a flat plate at a preset distance (eg, 0.5 meters, 0.6 meters, etc.) from the imaging system.
  • the electronic device can image the reference plane by using the first effective pixel and the second effective pixel of the image sensor, so as to obtain a reference image, wherein the number of the second effective pixel is at least one, for example, a part of the second effective pixel can be used.
  • the reference image can be obtained by using the effective pixels, or the reference image can be obtained by using all the second effective pixels.
  • the reference image may be pre-stored in a designated storage location; for example, stored in the memory of the terminal device or stored in a cloud server.
  • step S410 is usually performed in the calibration phase before the terminal equipment leaves the factory, but may also be performed in the terminal equipment use phase or other time nodes, which is not specifically limited in this application.
  • the first effective pixel and the second effective pixel of the image sensor will be described in detail.
  • the image sensor 500 includes an effective pixel area 501 located in the center and a dummy pixel area 502 located around the effective pixel area 501 .
  • FIG. 6 it is a schematic structural diagram of a common image sensor 600 .
  • the image sensor 600 has 1344 columns and 1136 rows of pixels, ie, a total of 1526784 (1344 ⁇ 1136) pixels.
  • the effective pixels are all pixels that can participate in imaging; the dummy pixels cannot participate in imaging, but can be used for black level signal correction and interpolation of pixels at the edge of the image.
  • the image window area is generally defined by four parameters, namely HS (Horizontal Start, horizontal start), HE (Horizontal End, horizontal end), VS (Vertical Start, vertical start) and VE (Vertical End, vertical end); of course, it can also be defined by fewer parameters, such as only by HS (Horizontal Start, horizontal start) and VE (Vertical End, vertical end) and so on.
  • the areas where the pixels from the 169th row to the 968th row and the 33rd column to the 1312th column are located are configured as the image window area, through which the image sensor output can be made Images with a resolution of 1280 ⁇ 800.
  • the first effective pixels are effective pixels used for actual imaging when the user uses the terminal device
  • the second effective pixels are effective pixels that are not used for actual imaging when the user uses the terminal device.
  • the first effective pixel is the effective pixel located in the designated area of the image sensor (image window area)
  • the second effective pixel is the effective pixel located outside the designated area of the image sensor.
  • the second effective pixel It can also be a part of the effective pixels located outside the designated area of the image sensor.
  • the first effective pixel is also the pixel of the actual imaging area
  • the second effective pixel is the first effective pixel
  • the second effective pixel may also be a part of the effective pixel other than the first effective pixel; this also belongs to the protection scope of the present application.
  • the first effective pixel of the image sensor When generating a reference image in the technical field, the first effective pixel of the image sensor is usually used; on the one hand, this is a long-standing default practice in the technical field, and has become a common operating habit of those skilled in the art; on the other hand , the size of the reference image formed in this way is the same as the size of the target image formed subsequently, and it is more convenient to perform related processing and operations.
  • the inventor overcomes these technical prejudices and creatively utilizes the first effective pixel while also utilizing the second effective pixel to participate in the generation of the reference image, which can effectively increase the cost without increasing the cost. Refer to the size of the image, so that the black edge phenomenon of the depth image can be reduced or avoided.
  • step S420 the structured light is emitted to a target object, and the target object is imaged by using the first effective pixel to obtain a target image.
  • the above-mentioned target object may be a human face or other object to be imaged; the electronic device may use the above-mentioned structured light projection module 101 to emit a structure in the form of encoding such as speckle, spot, stripe or two-dimensional pattern to the target object Light beam; the structured light beam emitted to the target object needs to be consistent with the structured light beam used to generate the reference image.
  • the target object may be imaged using the first effective pixel of the image sensor. It should be noted that, since the image window area is usually preconfigured before the terminal device leaves the factory, during use, the terminal device can automatically determine the above-mentioned first effective pixel by reading the configuration file related to the image window area.
  • step S430 a depth image of the target object is obtained according to the target image and the reference image.
  • the electronic device may read the reference image generated in step S410 from a designated storage location, for example, read the reference image from the memory of the electronic device or from a cloud server. After the electronic device acquires the reference image, it can refer to steps S710 to S730 as shown in FIG. 7 to obtain the depth image of the target object. in:
  • step S710 the target pixels in the target image are matched in the reference image.
  • the electronic device may determine the matching pixels corresponding to the target pixels in the target image in the reference image by using methods such as block matching or matrix matching.
  • a pixel block Bij of size m ⁇ n centered on the target pixel Pij can be extracted as the search pixel block; then in the reference image , taking the position corresponding to the target pixel Pij as the center and in the search window Vij of size W ⁇ H, according to the preconfigured search strategy and similarity evaluation index, find the matching pixel block corresponding to the search pixel block Bij.
  • the pixel block B'k1 is determined to be the same as the search pixel block.
  • i and k are positive integers, indicating the row where the pixel is located; j, l are positive integers, indicating the column where the pixel is located; m, n, W, and H are all positive integers, and W>m, H>n.
  • SGBM Semi-Global Block Matching
  • the reference image and the target image may be binarized first; however, those skilled in the art can easily understand that the binarization process Optional step only.
  • the reference image and the The target image is binarized, so that the value of each pixel in the image is 0 or 1.
  • the electronic device can take the average value of the brightness of each pixel in the entire image area as the threshold value, and set the pixel greater than the threshold value as 1, and the pixel less than the threshold value as 0; the global threshold method is used in applications. The effect is better when generating indoor depth images.
  • the brightness average can be calculated for the (1, 1, 100, 100) area first, and the pixels larger than the threshold in this area are taken as 1, Pixels smaller than the threshold value are 0.
  • the local threshold method works well when applied to outdoor depth image generation.
  • step S720 the offset of the target pixel is determined according to the position of the matched pixel in the reference image, and the depth information corresponding to the target pixel is calculated according to the offset of the target pixel.
  • the matching pixel corresponding to the target pixel Pij in the reference image is P'k1.
  • the offset (that is, the aberration) x of the target pixel can be calculated; then, combined with the triangulation method, the target object can be calculated
  • the depth H of the imaging point 901 on the target object that is, the distance between the imaging point 901 on the target object and the terminal device.
  • H R/(1-(R ⁇ x)/(f ⁇ d)); wherein, d is the length of the baseline (ie, the connection line 903 between the structured light projection module 101 and the image sensor 102 ), and R is The distance between the reference plane 902 and the baseline, f is the focal length of the image sensor 102; since d, R, and f are all fixed constants, after determining the offset x of the target pixel, the electronic device can calculate the target pixel Corresponding depth information H.
  • the depth information corresponding to the target pixel may also be calculated in other ways.
  • a mapping table between the offset of each pixel and the depth information can be established in advance, and after the offset x of the target pixel is determined, the depth information corresponding to the target pixel can be obtained by querying the mapping table; these also belong to this The scope of protection applied for.
  • step S730 the depth image is generated using the depth information corresponding to each of the target pixels.
  • the depth information can be converted into a depth image; for example, the depth image can be a grayscale image, and the larger the grayscale value of the pixel in the depth image, the greater the depth value.
  • the depth image may also exist in an image channel or in other forms, which also belong to the protection scope of the present application.
  • the OV9286 image sensor which is commonly used in depth cameras, as an example, with a total of 1328 columns and 1120 rows of effective pixels. Since the industry usually needs a depth image with a resolution of 640 ⁇ 400 or 1280 ⁇ 800, in the prior art, in the production and calibration stage of the depth camera, 1280 ⁇ 800 effective pixels in the image sensor are used to generate a reference image ;In actual use, after the structured light projection module emits structured light beams to the target object, the target object is imaged by the 1280 ⁇ 800 effective pixels in the image sensor to obtain a target image with a resolution of 1280 ⁇ 800; then, according to the target image (with a resolution of 1280 ⁇ 800) and a reference image (with a resolution of 1280 ⁇ 800) to generate a depth image.
  • the electronic device uses 1328 ⁇ 1120 effective pixels in the image sensor to generate a reference image during the production and calibration stage of the depth camera; in actual use, the structured light projection module of the electronic device emits light to the target object. After structuring the light beam, the target object with 1280 ⁇ 800 effective pixels in the image sensor is imaged to obtain a target image with a resolution of 1280 ⁇ 800; 1328 ⁇ 1120) to generate a depth image to reduce or remove black borders.
  • FIG. 10 which is a schematic diagram of the principle of the depth image generation method in this example embodiment: similar to FIG. 3 above, the outgoing light range of the structured light projection module 101 is area 301 , and the incident light range of the image sensor 102 is area 302 .
  • the pixels in the area A' of the target image 304 and the area A' of the target image 305 can be matched to the corresponding pixels in the reference image 1003, that is, the corresponding depth information can be determined; Compared with the area A in FIG.
  • the pixels in the area B' of the target image 304 and the area B' of the target image 305 do not exist in the reference image 1003, that is, the corresponding depth information cannot be determined; therefore,
  • the positions corresponding to the B' area of the target image 304 and the B' area of the target image 305 are both black borders; among them, the B' area is much smaller than the B area in FIG. 3 . .
  • all areas of the target image can be matched to the corresponding pixels in the reference image 1003, that is, the corresponding depth information can be determined. Therefore, in In the area corresponding to the dotted frame 1004, the black border phenomenon is completely avoided.
  • the depth image generation method in this example embodiment can indeed reduce or avoid the black edge phenomenon of the depth image to a certain extent. Furthermore, the application of depth images in some fields can be optimized. Taking an applied depth camera or other terminal device with a depth imaging function as an example, the depth image generation method in this example embodiment can increase the field of view during depth image acquisition to a certain extent, and improve the quality of the depth image; If the above-mentioned depth camera or terminal device with depth camera function is used for face authentication, the field of view during face authentication can be increased to a certain extent, and the authentication failure caused by the black edge of the depth image can be reduced, thereby improving the user experience. .
  • the existing second effective pixels are used in the depth image generation method provided by this example embodiment, no modification to the hardware is required, that is, the depth image generation method provided by the example embodiment of the present application does not Additional optimization costs are incurred; at the same time, the scope of application of the depth image generation method provided by this example embodiment is also increased.
  • the electronic device may use the first effective pixels of the image sensor and the second effective pixels of different numbers to image the reference plane to obtain a plurality of reference images of different sizes, wherein a plurality of reference images are obtained.
  • Reference images of different sizes refer to at least two reference images of different sizes.
  • the pixels in the 33rd row to the 1312th row and the 169th row to the 968th row are the first effective pixels; the 9th row to the 32nd row and the 1313th row
  • the pixels from the 1336th column, the 9th row to the 168th row, and the 969th row to the 1128th row are the second effective pixels.
  • the pixels from columns 9 to 1336 and rows 9 to 1128, that is, all the first and second effective pixels can be used to generate the first reference image;
  • a selected reference image may be determined from the multiple reference images according to a specified rule, and a depth image of the target object may be obtained according to the target image and the selected reference image. For example, the distance from the target object to the reference plane may be acquired first; then, the selected reference image is determined from the plurality of reference images according to the distance from the target object to the reference plane. For example:
  • the distance between the target object and the reference terminal device can be determined by using the laser ranging module or other distance sensor set in the terminal device; since the terminal device is connected to the reference plane The distance of is a fixed value. After the distance between the target object and the reference terminal device is determined, the distance from the target object to the reference plane can be roughly determined. 3 and 10, it can be known that the closer the distance between the target object plane and the reference plane, the smaller the black border range, and the farther the target object plane is from the reference plane, the larger the black border range.
  • the determined size of the selected reference image may be positively correlated with the distance from the target object to the reference plane; that is, the greater the distance from the target object to the reference plane, the greater the distance from the target object to the reference plane, the greater the selected reference image.
  • the size is correspondingly larger.
  • the above-mentioned first reference image can be determined as the selected reference image
  • the above-mentioned second reference image can be determined as the selected reference image.
  • the image is determined to be the selected reference image. This can effectively reduce the amount of computation during matching.
  • multiple reference images may also be generated in other manners.
  • the pixels from columns 9 to 1312 and rows 169 to 968 can be used to generate the first reference image; Generate a second reference image with pixels at row 968; generate a third reference image with pixels at columns 33-1312, rows 9-968; and generate a third reference image with pixels at columns 33-1312, rows 169-1
  • the 1128 rows of pixels generate the fourth reference image.
  • the selected reference image may also be determined from the plurality of reference images in other ways.
  • the position of the target object in the target image may be determined first; for example, the position of the target object in the target image may be determined by foreground image extraction or other methods.
  • a selected reference image may be determined from a plurality of the reference images according to the position of the target object in the target image. For example:
  • the above-mentioned first reference image can be determined as the selected reference image.
  • the above-mentioned second reference image can be determined as the selected reference image; if the position of the target object in the target image is high, the above-mentioned third reference image can be determined as the selected reference image.
  • the selected reference image It is determined as the selected reference image; if the position of the target object is lower in the target image, the above-mentioned fourth reference image can be determined as the selected reference image.
  • the algorithm complexity during matching can also be effectively reduced, thereby reducing the amount of computation.
  • the present application also provides a depth image generation method.
  • the method may include the following steps S1110 and S1120. in:
  • step S1110 the structured light is emitted to the target object, and the first effective pixel of the image sensor is used to image the target object to obtain a target image;
  • step S1120 a depth image of the target object is obtained according to the target image and the reference image; wherein, the reference image is obtained by imaging a reference plane by using the first effective pixels and the second effective pixels of the image sensor.
  • the present application also provides a reference image generation method.
  • the method may include the steps of: emitting structured light to a reference plane, and imaging the reference plane with first and second effective pixels of an image sensor to obtain a reference image; wherein, the first effective pixel is a target Effective pixels to use when depth imaging of the object.
  • the depth image generation apparatus 1200 may include a reference image generation module 1210 , a target image acquisition module 1220 and a depth image generation module 1230 . in:
  • the reference image generation module 1201 may be configured to emit structured light to a reference plane, and image the reference plane by using the first effective pixels and the second effective pixels of the image sensor to obtain a reference image.
  • the target image acquisition module 1220 may be configured to emit the structured light to a target object, and use the first effective pixels to image the target object to obtain a target image.
  • the depth image generation module 1230 may be configured to obtain a depth image of the target object according to the target image and the reference image.
  • the first effective pixel is an effective pixel located in a designated area of the image sensor; the second effective pixel is an effective pixel located outside the designated area of the image sensor.
  • the reference image generation module 1210 uses the first effective pixels and all the second effective pixels of the image sensor to image the reference plane.
  • the reference image generation module 1210 uses the first effective pixels of the image sensor and the second effective pixels of different numbers to image the reference plane to obtain at least two different sizes reference image.
  • the depth image generation module 1230 determines a selected reference image from the at least two reference images of different sizes, and obtains a depth image of the target object according to the target image and the selected reference image.
  • the depth image generation module 1230 determines the selected reference image from the at least two reference images of different sizes in the following manner: acquiring the target object to the a distance from the reference plane; and determining a selected reference image from the at least two reference images of different sizes according to the distance from the target object to the reference plane.
  • the size of the selected reference image is positively related to the distance from the target object to the reference plane.
  • the depth image generation module 1230 specifically determines the selected reference image from the at least two reference images of different sizes through the following steps: determining that the target object is in the a position in the target image; and determining a selected reference image from the at least two reference images of different sizes according to the position of the target object in the target image.
  • the apparatus further includes: an image storage module, configured to pre-store the reference image in a designated storage location.
  • the depth image generation module 1230 reads the reference image from the designated storage location, and obtains a depth image of the target object according to the target image and the read reference image.
  • the depth image generation module 1230 specifically obtains the depth image of the target object through the following steps: matching the target pixels in the target image with the reference image ; Determine the offset of the target pixel according to the position of the matched pixel in the reference image; Calculate the depth information corresponding to the target pixel according to the offset of the target pixel; Use the corresponding depth information of each target pixel The depth information generates the depth image.
  • the depth image generation module 1230 before matching the target pixels in the target image in the reference image, the depth image generation module 1230 further performs the matching on the reference image and the target image. Binarization processing.
  • a depth image generation device comprising: a target image acquisition module, configured to emit structured light to a target object, and use the first effective pixel of an image sensor to image the target object to obtain a target object an image; a depth image generation module for obtaining a depth image of the target object according to the target image and the reference image; wherein, the reference image is a pair of reference images using the first effective pixels and the second effective pixels of the image sensor obtained by planar imaging.
  • a reference image generation device comprising: a reference image generation module, configured to emit structured light to a reference plane, and use first effective pixels and second effective pixels of an image sensor for the reference plane imaging to obtain a reference image; wherein, the first effective pixel is an effective pixel used when performing depth imaging on the target object.
  • FIG. 13 shows a schematic structural diagram of a computer system suitable for implementing the electronic device according to the embodiment of the present application.
  • a computer system 1300 includes a central processing unit (CPU) 1301, which can be loaded into a random access memory (RAM) 1303 according to a program stored in a read only memory (ROM) 1302 or a program from a storage section 1308 Instead, various appropriate actions and processes are performed.
  • RAM random access memory
  • ROM read only memory
  • various programs and data required for system operation are also stored.
  • the CPU 1301, the ROM 1302, and the RAM 1303 are connected to each other through a bus 1304.
  • An input/output (I/O) interface 1305 is also connected to bus 1304 .
  • the following components are connected to the I/O interface 1305: an input section 1306 including a keyboard, a mouse, etc.; an output section 1307 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1308 including a hard disk, etc. ; and a communication section 1309 including a network interface card such as a LAN card, a modem, and the like.
  • the communication section 1309 performs communication processing via a network such as the Internet.
  • Drivers 1310 are also connected to I/O interface 1305 as needed.
  • a removable medium 1311 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 1310 as needed so that a computer program read therefrom is installed into the storage section 1308 as needed.
  • embodiments of the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart.
  • the computer program may be downloaded and installed from the network via the communication portion 1309, and/or installed from the removable medium 1311.
  • CPU central processing unit
  • the present application also provides a computer-readable medium.
  • the computer-readable medium may be included in the electronic device described in the above embodiments; it may also exist alone without being assembled into the electronic device. middle.
  • the above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by an electronic device, causes the electronic device to implement the methods described in the following embodiments. For example, the electronic device can implement the steps shown in FIG. 4 to FIG. 9 and the like.
  • the computer-readable medium shown in this application may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two.
  • the computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), fiber optics, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
  • a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
  • a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.
  • a computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device .
  • Program code embodied on a computer readable medium may be transmitted using any suitable medium including, but not limited to, wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
  • each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logical functions for implementing the specified functions executable instructions.
  • the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.

Abstract

本申请提供一种深度图像生成方法及装置、参考图像生成方法、电子设备；涉及深度成像技术领域。所述深度图像生成方法包括：向参考平面发射结构光，并利用图像传感器的第一有效像素和第二有效像素对所述参考平面成像，以得到参考图像；向目标对象发射所述结构光，并利用所述第一有效像素对所述目标对象成像，以得到目标图像；根据所述目标图像和所述参考图像得到所述目标对象的深度图像。本申请可以在一定程度上减少或者避免深度图像出现黑边现象。

Description

深度图像生成方法及装置、参考图像生成方法及装置、电子设备以及计算机可读存储介质
本申请要求于2020年07月28日提交中国专利局，申请号为2020107396664，申请名称为“深度图像生成方法及装置、参考图像生成方法、电子设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及深度成像技术领域，具体而言，涉及一种深度图像生成方法、深度图像生成装置、参考图像生成方法、参考图像生成方法装置、电子设备以及计算机可读存储介质。
背景技术
随着计算机视觉的发展，基于二维彩色图像处理的传统计算机视觉技术已无法满足人们将计算机视觉应用于三维物理世界中的需求。深度图像作为可以直接反映物体距离信息的图像，得到了越来越多的应用。
但在现有技术中，部分深度图像可能会存在黑边的现象，进而到其在某些领域的应用。例如，在面部认证过程中，可以引入面部深度图像，从而提高认证结果的准确性；但如果用户面部正好有部分位于上述深度图像的黑边中，则可能会导致面部认证失败。
需要说明的是，在上述背景技术部分公开的信息仅用于加强对本申请的背景的理解，因此可以包括不构成对本领域普通技术人员已知的现有技术的信息。
发明内容
根据本申请提供的各种实施例，提供一种深度图像生成方法、深度图像生成装置、参考图像生成方法、参考图像生成方法装置、电子设备以及计算机可读存储介质。
根据本申请的一个方面，提供一种深度图像生成方法，包括：向参考平面发射结构光，并利用图像传感器的第一有效像素和第二有效像素对所述参考平面成像，以得到参考图像；向目标对象发射所述结构光，并利用所述第一有效像素对所述目标对象成像，以得到目标图像；及根据所述目标图像和所述参考图像得到所述目标对象的深度图像。
根据本申请的一个方面，提供一种深度图像生成方法，包括：向目标对象发射结构光，并利用图像传感器的第一有效像素对所述目标对象成像，以得到目标图像；及根据所述目标图像和参考图像得到所述目标对象的深度图像；其中，所述参考图像是利用所述图像传感器的第一有效像素和第二有效像素对参考平面成像获得。
根据本申请的一个方面，提供一种参考图像生成方法，包括：向参考平面发射结构光，并利用图像传感器的第一有效像素和第二有效像素对所述参考平面成像，以得到参考图像；其中，所述第一有效像素为对目标对象进行深度成像时被使用的有效像素。
根据本申请的一个方面，提供一种深度图像生成装置，包括：参考图像生成模块，用于向参考平面发射结构光，并利用图像传感器的第一有效像素和第二有效像素对所述参考平面成像，以得到参考图像；目标图像获取模块，用于向目标对象发射所述结构光，并利用所述第一有效像素对所述目标对象成像，以得到目标图像；及深度图像生成模块，用于根据所述目标图像和所述参考图像得到所述目标对象的深度图像。
根据本申请的一个方面，提供一种深度图像生成装置，包括：目标图像获取模块，用于向目标对象发射结构光，并利用图像传感器的第一有效像素对所述目标对象成像，以得到目标图像；及深度图像生成模块，用于根据所述目标图像和参考图像得到所述目标对象的深度图像；其中，所述参考图像是利用所述图像传感器的第一有效像素和第二有效像素对参考平面成像获得。
根据本申请的一个方面，提供一种参考图像生成装置，包括：参考图像生成模块，用于向参考平面发射结构光，并利用图像传感器的第一有效像素和第二有效像素对所述参考平面成像，以得到参考图像；其中，所述第一有效像素 为对目标对象进行深度成像时被使用的有效像素。
根据本申请的一个方面，提供一种电子设备，包括：处理器；以及存储器，用于存储所述处理器的可执行指令；其中，所述处理器配置为经由执行所述可执行指令来执行上述任意一项所述的方法。
根据本申请的一个方面，提供一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现上述任意一项所述的方法。
根据本申请的一个方面，提供一种计算机程序产品或计算机程序，该计算机程序产品或计算机程序包括计算机指令，该计算机指令存储在计算机可读存储介质中。计算机设备的处理器从计算机可读存储介质读取该计算机指令，处理器执行该计算机指令，使得该计算机设备执行上述的各种可选实现方式中提供的方法。
本申请示例性实施例可以具有以下部分或全部有益效果：
基于本申请示例实施方式所提供的深度图像生成方法，生成参考图像的过程中，在利用第一有效像素的同时，还利用了在现有技术中并不被使用的第二有效像素，这样则可以增加生成的参考图像的尺寸。而黑边现象产生的主要原因是，目标图像边缘的像素在参考图像中不存在相对应的像素；而本申请示例实施方式所提供的深度图像生成方法中，由于增加了参考图像的尺寸，则可以使得目标图像边缘的部分或全部像素在参考图像匹配到相对应的像素，从而可以在一定程度上减少或者避免深度图像出现黑边现象。
应当理解的是，以上的一般描述和后文的细节描述仅是示例性和解释性的，并不能限制本申请。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，示出了符合本申请的实施例，并与说明书一起用于解释本申请的原理。显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1示出了可以应用本申请实施例的一种深度成像系统结构示意图；
图2示意性示出了根据本申请的一个实施例的深度图像生成方法的应用场景图；
图3示出了根据本申请的一个实施例的黑边现象产生的原理示意图；
图4示意性示出了根据本申请的一个实施例的深度图像生成方法的流程图；
图5示意性示出了根据本申请的一个实施例的图像传感器的框图；
图6示意性示出了根据本申请的一个实施例的图像传感器的结构图；
图7示意性示出了根据本申请的一个实施例的得到深度图像步骤的流程图；
图8示意性示出了根据本申请的一个实施例的块匹配方法原理图；
图9示意性示出了根据本申请的一个实施例的计算目标像素对应的深度信息的原理图；
图10示意性示出了根据本申请的一个实施例的黑边现象产生原理示意图；
图11示意性示出了根据本申请的一个实施例的深度图像生成方法的流程图；
图12示意性示出了根据本申请的一个实施例的深度图像生成装置的框图；及
图13示出了适于用来实现本申请实施例的电子设备的计算机系统的结构示意图。
具体实施方式
现在将参考附图更全面地描述示例实施方式。然而，示例实施方式能够以多种形式实施，且不应被理解为限于在此阐述的范例；相反，提供这些实施方式使得本申请将更加全面和完整，并将示例实施方式的构思全面地传达给本领域的技术人员。所描述的特征、结构或特性可以以任何合适的方式结合在一个或更多实施方式中。在下面的描述中，提供许多具体细节从而给出对本申请的实施方式的充分理解。然而，本领域技术人员将意识到，可以实践本申请的技术方案而省略所述特定细节中的一个或更多，或者可以采用其它的方法、组元、装置、步骤等。在其它情况下，不详细示出或描述公知技术方案以避免喧宾夺主而使得本申请的各方面变得模糊。
此外，附图仅为本申请的示意性图解，并非一定是按比例绘制。图中相同的附图标记表示相同或类似的部分，因而将省略对它们的重复描述。附图中所示的一些方框图是功能实体，不一定必须与物理或逻辑上独立的实体相对应。可以采用软件形式来实现这些功能实体，或在一个或多个硬件模块或集成电路中实现这些功能实体，或在不同网络和/或处理器装置和/或微控制器装置中实现这些功能实体。
参考图1所示，为发明人提供的一种深度成像系统100的结构示意图。图1所示的深度成像系统100中，主要包括结构光投影模块101、图像传感器102以及处理模块103；通过深度成像系统可以用于对目标对象104进行深度成像，得到对应的深度图像；目标对象104可以为人脸或者其他即待成像的对象。
结构光投影模块101可以包括光源以及光学组件。其中，光源可为激光二极管或者半导体激光器等，还可以为边发射激光器、垂直腔面激光发射器或者相应的阵列激光器等；光源的出射光的波长可以是红外或者紫外等。光学组件用于将光源发出的光束进行调制后向外发射出结构光；光学组件可以是折射型光学元件、衍射型光学元件或者两者的组合等。本示例实施方式中，结构光光束可以是散斑、斑点、条纹或者二维图案等编码形式的结构光光束。
图像传感器102可以为电荷藕合器件图像传感器(Charge Coupled Device，CCD)或互补金属氧化物半导体(Complementary Metal-0xide-Semiconductor，CMOS)图像传感器等。此外，为了便于采集入射光，在图像传感器102的入射光光路上，还可以设置如滤光片、微透镜阵列(Micro Lens Array，MLA)等光学组件。其中，滤光片可以为拜尔滤光片或者红外滤光片等。此外，当结构光投影模块101所投射结构光波长为λ时，上述滤光片可以设置为仅过滤波长为λ的光束通过，从而提升后续图像质量。
结构光投影模块101与图像传感器102之间的连线被称为基线，比如基线方向可以为图1中所示的x轴方向。结构光投影模块101与图像传感器102的光轴可以平行也可以形成一定的夹角。本示例实施方式中，结构光投影模块101与图像传感器102的光轴为平行设置，通过这种设置，可以简化后续生成深度图像时的计算复杂度。
处理模块103可以包含一个或多个处理器以及一个或多个存储器，可以用于对结构光投影模块101与图像传感器102进行控制以及接受相关的数据并进行处理。其中，处理器可以包含如数字信号处理器(Digital Signal Processing，DSP)、应用处理器(Multimedia Application Processor，MAP)、现场可编程门阵列(Field-ProgrammableGate Array，FPGA)、特定用途集成电路(Application Specific Integrated Circuit，ASIC)等中的一种或组合，存储器可以包含如随机存取存储器(Random Access Memory，RAM)、只读存储器(Read Only Memory，ROM)、闪存(Flash)等中的一种或组合。处理模块103所执行的控制、数据处理指令可以以软件、固件等形式保存在存储器中并在需要时被处理器调用，也可以直接将指令固化到电路中形成专用电路(或专用处理器)以执行相应的指令，也可以通过软件以及专用电路组合的形式来实现。处理模块103还可以包含输入/输出接口，和/或支持网络通信的网络接口。在本申请一些实施例中，可以通过接口将处理后的数据传输至其他设备或者系统中的其他单元，比如显示单元或者外部终端设备等。
本示例实施方式中，上述结构光投影模块101、图像传感器102以及处理模块103可以独立分散设置，也可以部分或者全部集成在一个电子设备之中。例如，结构光投影模块101、图像传感器102以及处理模块103均可以集成在智能手机、平板电脑、笔记本电脑或者摄像机之中；参考图2所示，结构光投影模块101、图像传感器102以及处理模块103均集成在智能手机201，通过智能手机201即可对目标对象104进行深度成像。又例如，结构光投影模块101、图像传感器102可以集成在一个电子设备，处理模块103可以为云服务器或者其他第三方电子设备等。
基于上述深度成像系统100，则可以在终端设备出厂前的标定阶段，利用结构光投影模块101向参考平面发射结构光，并利用图像传感器102对参考平面成像，以得到参考图像。用户在使用过程中，结构光投影模块101向目标对象发射所述结构光，并利用图像传感器102对所述目标对象成像，以得到目标图像；进而，可以计算目标图像中各像素相对于参考图像中对应像素的偏移量，并根据偏移量计算所述各像素对应的深度信息，从而根据各像素对应的深度信 息生成所述深度图像。
但基于上述深度图像生成方法得到的部分深度图像可能会存在黑边的现象，造成深度图像的一些区域无效，进而影响到深度图像在某些领域的应用。参考图3所示，为黑边现象产生的原理示意图：
其中，结构光投影模块101的出射光范围为区域301，图像传感器102的入射光范围为区域302；如果目标对象位于区域301和区域302重合的范围内，理论上都可以生成深度图像。但由于结构光投影模块101和图像传感器102之间存在一定的距离，且参考图像303大小固定，因此当目标对象所在平面与参考平面距离较远时，目标图像304中的部分像素可能无法在参考图像303中匹配到相对应的像素，导致无法确定对应的深度信息；在深度图像中，对于无法确定对应的深度信息的像素，会使用0补足，即将无法确定对应的深度信息的像素值设置为0，这样导致了黑边现象的出现。
举例而言，图3中的目标图像304的A区域和目标图像305的A区域中的像素，均可以在参考图像303中匹配到相对应的像素，即均可以确定对应的深度信息；但参考图像303中并不存在目标图像304的B区域和目标图像305的B区域中的像素，即无法确定对应的深度信息；因此，最终生成的深度图像中，目标图像304的B区域和目标图像305的B区域对应的位置均为黑边。通常而言，目标对象所在平面与参考平面距离越近则黑边范围越小，目标对象所在平面与参考平面距离越远则黑边范围越大。
以面部认证为例，黑边现象的存在会影响到面部认证时的视场角(FOV，Field angle of View)；同时，深度图像中黑边的大小可能会随着距离而变化，进一步增加了认证过程中的不确定性。具体而言，在不同距离进行面部认证，深度图像中黑边区域不同，导致面部认证有效区域不同；且如果用户面部正好处于深度图像黑边区域，则会影响本次面部认证的结果，从而降低了用户体验。此外，深度图像中黑边区域的范围会随着距离变化，在实际应用中也难以设计有效的策略进行规避。
基于上述一个或多个问题，本申请示例实施方式中提供了一种新的深度图像生成方法，以该方法应用于电子设备中为例进行说明。参考图4所示，该深 度图像生成方法可以包括下述步骤S410至步骤S430。其中：
在步骤S410中，向参考平面发射结构光，并利用图像传感器的第一有效像素和第二有效像素对所述参考平面成像，以得到参考图像。
本示例实施方式中，可以利用电子设备中结构光投影模块101向参考平面发射如散斑、斑点、条纹或者二维图案等编码形式的结构光光束,结构光是已知空间方向的投影光线的集合。参考平面例如可以是距离成像系统预设距离(如0.5米、0.6米等)的平板。接着，电子设备可以利用图像传感器的第一有效像素和第二有效像素对所述参考平面成像，从而得到参考图像，其中，第二有效像素的数量为至少一个，比如，可以使用一部分的第二有效像素来得到参考图像，也可以使用全部的第二有效像素来得到参考图像。本示例实施方式中，在得到参考图像之后，可以将参考图像预先存储在指定存储位置；例如，存储在终端设备的存储器或者存在云端服务器等。需要说明的是，步骤S410通常是在终端设备出厂前的标定阶段执行，但也可以是在终端设备使用阶段或者其他时间节点执行，本申请对此不做特殊限定。下面，将对图像传感器的第一有效像素和第二有效像素进行详细说明。
一般而言，参考图5所示，图像传感器500包括位于中心的有效像素区域501以及位于有效像素区域501周边的虚设像素区域502。参考图6所示，为一种常用图像传感器600的结构示意图。图像传感器600具有1344列、1136行像素，即共1526784(1344×1136)个像素。所述1526784个像素中，中心的1328列、1120行像素位于有效像素区域，即共计1487360(1328×1120)个有效像素(Active Pixel)；所述1526784个像素中，上部8行、下部8行、左部8列以及右部8列位于周边虚设像素区域，即共计39424个虚设像素(Dummy Pixel)。其中，有效像素均为可以参与成像的像素；虚设像素无法参与成像，但可以用于进行黑阶信号校正以及对图像边缘的像素进行插值。
虽然有效像素均可以参与成像，但在实际应用中，为了适配具体终端的图像分辨率要求，通常仅配置部分有效像素(如下述图像窗口区域的像素)参与实际成像，其他有效像素则不参与实际成像。具体而言，在本技术领域，通常需要对图像传感器进行“开窗”操作，从而实现对于图像传感器的图像窗口区 域(Windowing)的配置；图像窗口区域一般由四个参数定义，即HS(Horizontal Start，水平开始)、HE(Horizontal End，水平结束)、VS(Vertical Start，垂直开始)以及VE(Vertical End，垂直结束)；当然，也可以通过更少的参数定义，如仅通过HS(Horizontal Start，水平开始)以及VE(Vertical End，垂直结束)表示等。
继续参考图6所示，在图像传感器600中，第169行至第968行、第33列至第1312列像素所在的区域被配置为图像窗口区域，通过该图像窗口区域，可以使得图像传感器输出分辨率为1280×800的图像。本示例实施方式中，第一有效像素为用户使用终端设备时，用于实际成像的有效像素，第二有效像素为用户使用终端设备时，不用于实际成像的有效像素。结合上述图像窗口区域，第一有效像素即位于图像传感器指定区域(图像窗口区域)的有效像素，第二有效像素即位于图像传感器指定区域外的有效像素，在一个实施例中，第二有效像素也可以是位于图像传感器指定区域外的部分有效像素。但容易理解的是，在本申请的其他示例性实施例中，如果实际成像区域是根据其他规则配置，则第一有效像素则同样为实际成像区域的像素，第二有效像素为第一有效像素之外的有效像素，在一个实施例中，第二有效像素也可以为第一有效像素之外的部分有效像素；这同样属于本申请的保护范围。
本技术领域在生成参考图像时，通常利用的都是图像传感器的第一有效像素；一方面，这是本技术领域长久以来的默认做法，已经成为本领域技术人员的普遍操作习惯；另一方面，这样形成的参考图像和后续形成的目标图像的尺寸一致，进行相关处理以及运算时会更加方便。在本申请中，发明人则克服了这些技术偏见，创造性的在利用第一有效像素的同时，还利用第二有效像素参与参考图像的生成，这样则可以在不增加成本的同时，有效的增加参考图像的尺寸，从而可以减少或者避免深度图像出现黑边现象。
在步骤S420中，向目标对象发射所述结构光，并利用所述第一有效像素对所述目标对象成像，以得到目标图像。
本示例实施方式中，上述目标对象可以为人脸或者其他即待成像的对象；电子设备可以利用上述结构光投影模块101向目标对象发射如散斑、斑点、条 纹或者二维图案等编码形式的结构光光束；向目标对象发射的结构光光束需要与生成参考图像时使用的结构光光束保持一致。接着，可以利用图像传感器的第一有效像素对所述目标对象成像。需要说明的是，由于图像窗口区域通常是在终端设备出厂前预先配置的，因此在使用过程中，终端设备读取图像窗口区域相关的配置文件即可自动确定上述的第一有效像素。
在步骤S430中，根据所述目标图像和所述参考图像得到所述目标对象的深度图像。
本示例实施方式中，电子设备可以从指定存储位置读取步骤S410中生成的参考图像，例如，从电子设备的存储器或者从云端服务器读取参考图像等。电子设备获取参考图像之后，可以参考如图7所示的步骤S710至步骤S730，得到所述目标对象的深度图像。其中：
在步骤S710中，对所述目标图像中的目标像素在所述参考图像中进行匹配。
本示例实施方式中，电子设备可以通过块匹配或者矩阵匹配等方法，确定目标图像中的目标像素在参考图像中对应的匹配像素。参考图8所示，以块匹配方法为例，对于目标图像中的目标像素Pij，可以提取以目标像素Pij为中心，大小为m×n的像素块Bij作为搜索像素块；然后在参考图像中，以目标像素Pij所对应的位置为中心，大小为W×H的搜索窗口Vij内，按照预先配置的搜索策略和相似度评价指标来寻找与搜索像素块Bij相对应的匹配像素块。例如，如果搜索窗口Vij内，像素块B’kl与搜索像素块Bij的相似度评价指标w(Bij,B’kl)相比于其他像素块最大，则确定像素块B’kl为与搜索像素块Bij匹配的像素块；进而可以确定目标像素Pij在参考图像中对应的匹配像素为P’kl。其中，i、k均为正整数，表示像素所在行；j、l均为正整数，表示像素所在列；m、n、W、H均为正整数，且W>m、H>n。此外，根据需求，也可以采用如半全局块匹配(Semi-Global Block Matching，SGBM)等其他方法实现像素匹配，本示例性实施例中对此不做特殊限定。
此外，为了更方便进行匹配以及方便后续运算，本示例实施方式中还可以首先对所述参考图像以及所述目标图像进行二值化处理；但本领域技术人员容易理解的是，二值化处理仅为可选的步骤。举例而言，本申请中可以通过全局 阈值方法、局部阈值方法、动态阈值方法、Niblack算法、P-分位数方法、迭代方法、熵方法、最大类间方差算法等技术手段，对参考图像以及目标图像进行二值化处理，使得图像中各像素的取值为0或者1。
以全局阈值方法为例，电子设备可以取整幅图像区域中各像素的亮度平均值为阈值，并将大于阈值的像素取值为1、小于阈值的像素取值为0；全局阈值方法在应用于室内的深度图像生成时，效果较好。以局部阈值方法为例，对于如1280×800个像素的目标图像，可以首先对(1、1、100、100)区域求亮度平均值，并将该区域内大于阈值的像素取值为1、小于阈值的像素取值为0。再顺序对(1、101、100、200)区域、(1、201、100、300)区域直至(1201、701、1280、800)区域做同样的处理，从而完成对整幅图像的二值化；局部阈值方法在应用于室外的深度图像生成时，效果较好。
在步骤S720中，根据匹配到的像素在所述参考图像的位置确定所述目标像素的偏移量，并根据所述目标像素的偏移量计算所述目标像素对应的深度信息。
举例而言，参考图9所示，目标像素Pij在参考图像中对应的匹配像素为P’kl。根据匹配像素P’kl的位置和目标像素Pij在参考图像中对应的位置，可以计算出目标像素的偏移量(也即像差)x；接着，结合三角测距法，可以计算出目标对象上的成像点901的深度H，也即目标对象上的成像点901和终端设备之间的距离。例如，H＝R/(1-(R·x)/(f·d))；其中，d为基线(即结构光投影模块101和图像传感器102之间的连线903)的长度，R为参考平面902与基线之间的距离，f为图像传感器102的焦距；由于d、R、f均为固定常量，因此，在确定目标像素的偏移量x之后，电子设备即可计算得到目标像素对应的深度信息H。
当然，在本申请的其他示例性实施例中，也可以通过其他方式计算所述目标像素对应的深度信息。例如，可以预先建立各像素的偏移量与深度信息之间的映射表，在确定目标像素的偏移量x之后，即可在映射表中查询得到目标像素对应的深度信息；这些同样属于本申请的保护范围。
在步骤S730中，利用各所述目标像素对应的深度信息生成所述深度图像。
电子设备通过上述步骤S720得到各目标像素对应的深度信息之后，即可将 深度信息转换为深度图像；例如，深度图像可以为灰度图，深度图像中像素的灰度值越大则表示深度值H越小，灰度值越小则表示深度值H越大；也可以是深度图像中像素的灰度值越大则表示深度值H越大，灰度值越小则表示深度值H越小，本示例性实施例中对此不做特殊限定。在本申请的其他示例性实施例中，所述深度图像也可以是图像通道或者其他形式的存在，这同样属于本申请的保护范围。
以在深度相机中较为普遍使用的OV9286型号的图像传感器为例，其有效像素共计1328列、1120行。由于行业内通常需要的是分辨率为640×400或者1280×800的深度图像，因此，现有技术中，在深度相机生产标定阶段，均采用图像传感器中的1280×800个有效像素生成参考图像；在实际使用时，结构光投影模块向目标对象发射结构光光束后，通过图像传感器中的1280×800个有效像素目标对象成像，得到分辨率为1280×800的目标图像；然后，根据目标图像(分辨率为1280×800)与参考图像(分辨率为1280×800)生成深度图像。而在本示例实施方式中则可以是，电子设备在深度相机生产标定阶段，采用图像传感器中的1328×1120个有效像素生成参考图像；在实际使用时，电子设备结构光投影模块向目标对象发射结构光光束后，通过图像传感器中的1280×800个有效像素目标对象成像，得到分辨率为1280×800的目标图像；然后，根据目标图像(分辨率为1280×800)与参考图像(分辨率为1328×1120)生成深度图像，从而减少或者去除黑边。
参考图10所示，为本示例实施方式中深度图像生成方法的原理示意图：与图上述图3类似，结构光投影模块101的出射光范围为区域301，图像传感器102的入射光范围为区域302。其中，目标图像304的A’区域和目标图像305的A’区域中的像素，均可以在参考图像1003中匹配到相对应的像素，即均可以确定对应的深度信息；其中，A’区域相比于图3中的A区域，均扩大了很多；参考图像1003中并不存在目标图像304的B’区域和目标图像305的B’区域中的像素，即无法确定对应的深度信息；因此，最终生成的深度图像中，目标图像304的B’区域和目标图像305的B’区域对应的位置均为黑边；其中，B’区域相比于图3中的B区域，均减小了很多。此外，在部分区域，例如图10中 的虚线框1004对应的区域内，目标图像的所有区域均可以在参考图像1003中匹配到相对应的像素，即均可以确定对应的深度信息，因此，在虚线框1004对应的区域内，则完全避免了黑边现象的出现。
由上可知，本示例实施方式中的深度图像生成方法确实可以在一定程度上减少或者避免深度图像出现黑边现象。进而，可以优化深度图像在一些领域的应用。以应用的深度相机或者其他具有深度摄像功能的终端设备为例，通过本示例实施方式中的深度图像生成方法，可以在一定程度上增加深度图像采集时的视场角，提升深度图像的质量；如果使用上述深度相机或者具有深度摄像功能的终端设备进行面部认证，则可以在一定程度上增加面部认证时的视场角，减少了由于深度图像黑边而导致的认证失败，从而提高了用户体验。
此外，由于本示例实施方式所提供的深度图像生成方法中，使用的是已有的第二有效像素，无需对硬件进行改动，也即本申请示例实施方式所提供的深度图像生成方法并不会产生额外的优化成本；同时，也增加了本示例实施方式所提供的深度图像生成方法的适用范围。
在本申请的另一示例性实施例中，为了减少上述步骤S710中的匹配运算量，在上述深度图像生成方法的基础上进行了进一步的改进。具体而言，电子设备可以在上述步骤S410中，利用图像传感器的第一有效像素和不同数量的所述第二有效像素对所述参考平面成像，得到多个不同大小的参考图像，其中多个不同大小的参考图像是指至少二个不同大小的参考图像。
举例而言，以图6中的图像传感器为例，其中，第33列至第1312列、第169行至第968行的像素为第一有效像素；第9列至第32列、第1313列至第1336列、第9行至第168行、第969行至第1128行的像素为第二有效像素。本示例实施方式中，例如可以利用第9列至第1336列、第9行至第1128行的像素，即全部的第一有效像素和第二有效像素生成第一参考图像；利用第21列至第1324列、第89行至第1048行的像素，即部分的第一有效像素和第二有效像素生成第二参考图像。
在生成多个参考图像之后，则可以按照指定规则，从多个参考图像中确定被选参考图像，以及，根据所述目标图像和所述被选参考图像得到所述目标对 象的深度图像。例如，可以首先获取所述目标对象到所述参考平面的距离；接着，根据所述目标对象到所述参考平面的距离，从多个所述参考图像中确定被选参考图像。举例而言：
本示例实施方式中，在对目标图像进行深度信息采集时，可以利用终端设备中设置的激光测距模块或者其他距离传感器，确定目标对象与参考终端设备之间的距离；由于终端设备到参考平面的距离是固定值，在确定目标对象与参考终端设备之间的距离之后，即可大致确定目标对象到所述参考平面的距离。参考上述图3以及图10可以得知，目标对象所在平面与参考平面距离越近则黑边范围越小，目标对象所在平面与参考平面距离越远则黑边范围越大。因此，本示例实施方式中，确定的所述被选参考图像的大小可以与所述目标对象到所述参考平面的距离正相关；即目标对象到参考平面的距离越大，则被选参考图像尺寸相应越大。
因此，当目标对象所在平面与参考平面距离较大时，例如可以将上述第一参考图像中确定为被选参考图像；当目标对象所在平面与参考平面距离较小时，例如可以将上述第二参考图像中确定为被选参考图像。这样则可以有效的降低匹配时的运算量。以块匹配方法为例，其算法复杂度正比于O(N,M,W,H,m,n)，其中N和M分别为参考图像像素行数和列数(其他参数指代同上文)；因此，相比于全部使用第一参考图像，在利用尺寸较小的第二参考图像时，可以有效的降低匹配时的算法复杂度，从而降低运算量。
在本申请的其他示例性实施例中，也可以通过其他方式的生成多个参考图像。仍以图6中的图像传感器为例，例如可以利用第9列至第1312列、第169行至第968行的像素生成第一参考图像；利用第33列至第1336列、第169行至第968行的像素生成第二参考图像；利用第33列至第1312列、第9行至第968行的像素生成第三参考图像；以及利用第33列至第1312列、第169行至第1128行的像素生成第四参考图像。
同时，也可以通过其他方式从多个参考图像中确定被选参考图像。例如，可以首先确定所述目标对象在所述目标图像中的位置；如可通过前景图像提取或者其他方法，确定目标对象在目标图像中的位置。接着，可以根据所述目标 对象在所述目标图像中的位置，从多个所述参考图像中确定被选参考图像。举例而言：
本示例实施方式中，在获取目标图像之后，如果目标对象在目标图像中的位置偏左，则如果深度图像的左边缘出现黑边现象，会影响较大，其他位置即使出现黑边，也可能影响较小；基于此，则可以将上述第一参考图像确定为被选参考图像。同样的，如果目标对象在目标图像中的位置偏右，则可以将上述第二参考图像确定为被选参考图像；如果目标对象在目标图像中的位置偏上，则可以将上述第三参考图像确定为被选参考图像；如果目标对象在目标图像中的位置偏下，则可以将上述第四参考图像确定为被选参考图像。相比于全部使用最大尺寸参考图像，在利用尺寸较小的第一至第四参考图像时，同样可以有效的降低匹配时的算法复杂度，从而降低运算量。
本申请中还提供了一种深度图像生成方法。参考图11所示，该方法可以包括下述步骤S1110以及步骤S1120。其中：
在步骤S1110中，向目标对象发射结构光，并利用图像传感器的第一有效像素对所述目标对象成像，以得到目标图像；
在步骤S1120中，根据所述目标图像和参考图像得到所述目标对象的深度图像；其中，所述参考图像是利用所述图像传感器的第一有效像素和第二有效像素对参考平面成像获得。
本申请中还提供了一种参考图像生成方法。该方法可以包括步骤：向参考平面发射结构光，并利用图像传感器的第一有效像素和第二有效像素对所述参考平面成像，以得到参考图像；其中，所述第一有效像素为对目标对象进行深度成像时被使用的有效像素。
上述深度图像生成方法以及参考图像生成方法中各步骤的具体细节已经在上文的示例性实施例中进行了详细的描述，因此此处不再赘述。
应当注意，尽管在附图中以特定顺序描述了本申请中方法的各个步骤，但是，这并非要求或者暗示必须按照该特定顺序来执行这些步骤，或是必须执行全部所示的步骤才能实现期望的结果。附加的或备选的，可以省略某些步骤，将多个步骤合并为一个步骤执行，以及/或者将一个步骤分解为多个步骤执行 等。
进一步的，本示例实施方式中，还提供了一种深度图像生成装置。参考图12所示，该深度图像生成装置1200可以包括参考图像生成模块1210、目标图像获取模块1220以及深度图像生成模块1230。其中：
参考图像生成模块1201可以用于向参考平面发射结构光，并利用图像传感器的第一有效像素和第二有效像素对所述参考平面成像，以得到参考图像。目标图像获取模块1220可以用于向目标对象发射所述结构光，并利用所述第一有效像素对所述目标对象成像，以得到目标图像。深度图像生成模块1230可以用于根据所述目标图像和所述参考图像得到所述目标对象的深度图像。
在本申请的一种示例性实施例中：所述第一有效像素为位于所述图像传感器指定区域的有效像素；所述第二有效像素为位于所述图像传感器指定区域外的有效像素。
在本申请的一种示例性实施例中，所述参考图像生成模块1210利用图像传感器的第一有效像素和全部第二有效像素对所述参考平面成像。
在本申请的一种示例性实施例中，所述参考图像生成模块1210利用图像传感器的第一有效像素和不同数量的所述第二有效像素对所述参考平面成像，得到至少二个不同大小的参考图像。所述深度图像生成模块1230从所述至少二个不同大小的参考图像中确定被选参考图像，以及，根据所述目标图像和所述被选参考图像得到所述目标对象的深度图像。
在本申请的一种示例性实施例中，所述深度图像生成模块1230具体通过下述方式从所述至少二个不同大小的参考图像中确定被选参考图像：获取所述目标对象到所述参考平面的距离；以及根据所述目标对象到所述参考平面的距离，从所述至少二个不同大小的参考图像中确定被选参考图像。
在本申请的一种示例性实施例中，所述被选参考图像的大小与所述目标对象到所述参考平面的距离正相关。
在本申请的一种示例性实施例中，所述深度图像生成模块1230具体通过下述步骤从所述至少二个不同大小的参考图像中确定被选参考图像：确定所述目标对象在所述目标图像中的位置；以及根据所述目标对象在所述目标图像中的 位置，从所述至少二个不同大小的参考图像中确定被选参考图像。
在本申请的一种示例性实施例中，所述装置还包括：图像存储模块，用于将所述参考图像预先存储在指定存储位置。所述深度图像生成模块1230从所述指定存储位置读取所述参考图像，以及，根据所述目标图像和读取的所述参考图像得到所述目标对象的深度图像。
在本申请的一种示例性实施例中，所述深度图像生成模块1230具体通过下述步骤得到所述目标对象的深度图像：对所述目标图像中的目标像素在所述参考图像中进行匹配；根据匹配到的像素在所述参考图像的位置确定所述目标像素的偏移量；根据所述目标像素的偏移量计算所述目标像素对应的深度信息；利用各所述目标像素对应的深度信息生成所述深度图像。
在本申请的一种示例性实施例中，对所述目标图像中的目标像素在所述参考图像中进行匹配之前，所述深度图像生成模块1230还对所述参考图像以及所述目标图像进行二值化处理。
根据本申请的一个方面，提供一种深度图像生成装置，包括：目标图像获取模块，用于向目标对象发射结构光，并利用图像传感器的第一有效像素对所述目标对象成像，以得到目标图像；深度图像生成模块，用于根据所述目标图像和参考图像得到所述目标对象的深度图像；其中，所述参考图像是利用所述图像传感器的第一有效像素和第二有效像素对参考平面成像获得。
根据本申请的一个方面，提供一种参考图像生成装置，包括：参考图像生成模块，用于向参考平面发射结构光，并利用图像传感器的第一有效像素和第二有效像素对所述参考平面成像，以得到参考图像；其中，所述第一有效像素为对目标对象进行深度成像时被使用的有效像素。
上述深度图像生成装置以及参考图像生成装置中各模块或单元的具体细节已经在对应的深度图像生成方法中进行了详细的描述，因此此处不再赘述。
图13示出了适于用来实现本申请实施例的电子设备的计算机系统的结构示意图。
需要说明的是，图13示出的电子设备的计算机系统1300仅是一个示例，不应对本申请实施例的功能和使用范围带来任何限制。
如图13所示，计算机系统1300包括中央处理单元(CPU)1301，其可以根据存储在只读存储器(ROM)1302中的程序或者从储存部分1308加载到随机访问存储器(RAM)1303中的程序而执行各种适当的动作和处理。在RAM 1303中，还存储有系统操作所需的各种程序和数据。CPU 1301、ROM 1302以及RAM 1303通过总线1304彼此相连。输入/输出(I/O)接口1305也连接至总线1304。
以下部件连接至I/O接口1305：包括键盘、鼠标等的输入部分1306；包括诸如阴极射线管(CRT)、液晶显示器(LCD)等以及扬声器等的输出部分1307；包括硬盘等的储存部分1308；以及包括诸如LAN卡、调制解调器等的网络接口卡的通信部分1309。通信部分1309经由诸如因特网的网络执行通信处理。驱动器1310也根据需要连接至I/O接口1305。可拆卸介质1311，诸如磁盘、光盘、磁光盘、半导体存储器等等，根据需要安装在驱动器1310上，以便于从其上读出的计算机程序根据需要被安装入储存部分1308。
特别地，根据本申请的实施例，下文参考流程图描述的过程可以被实现为计算机软件程序。例如，本申请的实施例包括一种计算机程序产品，其包括承载在计算机可读介质上的计算机程序，该计算机程序包含用于执行流程图所示的方法的程序代码。在这样的实施例中，该计算机程序可以通过通信部分1309从网络上被下载和安装，和/或从可拆卸介质1311被安装。在该计算机程序被中央处理单元(CPU)1301执行时，执行本申请的方法和装置中限定的各种功能。
作为另一方面，本申请还提供了一种计算机可读介质，该计算机可读介质可以是上述实施例中描述的电子设备中所包含的；也可以是单独存在，而未装配入该电子设备中。上述计算机可读介质承载有一个或者多个程序，当上述一个或者多个程序被一个该电子设备执行时，使得该电子设备实现如下述实施例中所述的方法。例如，所述的电子设备可以实现如图4～图9所示的各个步骤等。
需要说明的是，本申请所示的计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机 访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(EPROM或闪存)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本申请中，计算机可读存储介质可以是任何包含或存储程序的有形介质，该程序可以被指令执行系统、装置或者器件使用或者与其结合使用。而在本申请中，计算机可读的信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的程序代码。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读的信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。计算机可读介质上包含的程序代码可以用任何适当的介质传输，包括但不限于：无线、电线、光缆、RF等等，或者上述的任意合适的组合。
附图中的流程图和框图，图示了按照本申请各种实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分，上述模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意，在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个接连地表示的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图或流程图中的每个方框、以及框图或流程图中的方框的组合，可以用执行规定的功能或操作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。
应当理解的是，本申请并不局限于上面已经描述并在附图中示出的精确结构，并且可以在不脱离其范围进行各种修改和改变。本申请的范围仅由所附的权利要求来限制。

Claims (20)

  1. 一种深度图像生成方法，其特征在于，由电子设备执行，包括：
    向参考平面发射结构光，并利用图像传感器的第一有效像素和第二有效像素对所述参考平面成像，以得到参考图像；
    向目标对象发射所述结构光，并利用所述第一有效像素对所述目标对象成像，以得到目标图像；及
    根据所述目标图像和所述参考图像得到所述目标对象的深度图像；
    其中，所述第一有效像素为位于所述图像传感器指定区域的有效像素，所述第二有效像素为位于所述指定区域外的有效像素。
  2. 根据权利要求1所述的方法，其特征在于，所述指定区域为所述图像传感器的图像窗口区域。
  3. 根据权利要求1所述的方法，其特征在于，所述利用图像传感器的第一有效像素和第二有效像素对所述参考平面成像，包括：
    利用图像传感器的第一有效像素和全部第二有效像素对所述参考平面成像。
  4. 根据权利要求1所述的方法，其特征在于，所述利用图像传感器的第一有效像素和第二有效像素对所述参考平面成像，包括：
    利用图像传感器的第一有效像素和不同数量的所述第二有效像素对所述参考平面成像，得到至少二个不同大小的参考图像；
    根据所述目标图像和所述参考图像得到所述目标对象的深度图像包括：
    从所述至少二个不同大小的参考图像中确定被选参考图像，以及，根据所述目标图像和所述被选参考图像得到所述目标对象的深度图像。
  5. 根据权利要求4所述的方法，其特征在于，从所述至少二个不同大小的参考图像中确定被选参考图像，包括：
    获取所述目标对象到所述参考平面的距离；以及
    根据所述目标对象到所述参考平面的距离，从所述至少二个不同大小的参考图像中确定被选参考图像。
  6. 根据权利要求5所述的方法，其特征在于，所述被选参考图像的大小与 所述目标对象到所述参考平面的距离正相关。
  7. 根据权利要求4所述的方法，其特征在于，从所述至少二个不同大小的参考图像中确定被选参考图像，包括：
    确定所述目标对象在所述目标图像中的位置；以及
    根据所述目标对象在所述目标图像中的位置，从所述至少二个不同大小的参考图像中确定被选参考图像。
  8. 根据权利要求1所述的方法，其特征在于，所述方法还包括：将所述参考图像预先存储在指定存储位置；
    根据所述目标图像和所述参考图像得到所述目标对象的深度图像包括：
    从所述指定存储位置读取所述参考图像，以及，根据所述目标图像和读取的所述参考图像得到所述目标对象的深度图像。
  9. 根据权利要求1～7任意一项所述的方法，其特征在于，根据所述目标图像和所述参考图像得到所述目标对象的深度图像，包括：
    对所述目标图像中的目标像素在所述参考图像中进行匹配；
    根据匹配到的像素在所述参考图像的位置确定所述目标像素的偏移量；
    根据所述目标像素的偏移量计算所述目标像素对应的深度信息；及
    利用各所述目标像素对应的深度信息生成所述深度图像。
  10. 根据权利要求8所述的方法，其特征在于，对所述目标图像中的目标像素在所述参考图像中进行匹配之前，所述方法还包括：
    对所述参考图像以及所述目标图像进行二值化处理。
  11. 一种深度图像生成方法，其特征在于，包括：
    向目标对象发射结构光，并利用图像传感器的第一有效像素对所述目标对象成像，以得到目标图像；及
    根据所述目标图像和参考图像得到所述目标对象的深度图像；
    其中，所述参考图像是利用所述图像传感器的第一有效像素和第二有效像素对参考平面成像获得。
  12. 一种参考图像生成方法，其特征在于，包括：
    向参考平面发射结构光，并利用图像传感器的第一有效像素和第二有效像 素对所述参考平面成像，以得到参考图像；
    其中，所述第一有效像素为对目标对象进行深度成像时被使用的有效像素。
  13. 一种深度图像生成装置，其特征在于，包括：
    参考图像生成模块，用于向参考平面发射结构光，并利用图像传感器的第一有效像素和第二有效像素对所述参考平面成像，以得到参考图像；
    目标图像获取模块，用于向目标对象发射所述结构光，并利用所述第一有效像素对所述目标对象成像，以得到目标图像；及
    深度图像生成模块，用于根据所述目标图像和所述参考图像得到所述目标对象的深度图像。
  14. 根据权利要求13所述的装置，其特征在于，所述第一有效像素为位于所述图像传感器指定区域的有效像素；所述第二有效像素为位于所述图像传感器指定区域外的有效像素。
  15. 根据权利要求13所述的装置，其特征在于，所述参考图像生成模块利用图像传感器的第一有效像素和全部第二有效像素对所述参考平面成像。
  16. 根据权利要求13所述的装置，其特征在于，所述参考图像生成模块利用图像传感器的第一有效像素和不同数量的所述第二有效像素对所述参考平面成像，得到至少二个不同大小的参考图像；
    所述深度图像生成模块从所述至少二个不同大小的参考图像中确定被选参考图像，以及，根据所述目标图像和所述被选参考图像得到所述目标对象的深度图像。
  17. 一种深度图像生成装置，其特征在于，包括：
    目标图像获取模块，用于向目标对象发射结构光，并利用图像传感器的第一有效像素对所述目标对象成像，以得到目标图像；及
    深度图像生成模块，用于根据所述目标图像和参考图像得到所述目标对象的深度图像；其中，所述参考图像是利用所述图像传感器的第一有效像素和第二有效像素对参考平面成像获得。
  18. 一种参考图像生成装置，其特征在于，包括：
    参考图像生成模块，用于向参考平面发射结构光，并利用图像传感器的第 一有效像素和第二有效像素对所述参考平面成像，以得到参考图像；其中，所述第一有效像素为对目标对象进行深度成像时被使用的有效像素。
  19. 一种计算机可读存储介质，其上存储有计算机程序，其特征在于，所述计算机程序被处理器执行时实现权利要求1-12任一项所述的方法。
  20. 一种电子设备，其特征在于，包括：
    处理器；以及
    存储器，用于存储所述处理器的可执行指令；
    其中，所述处理器配置为经由执行所述可执行指令来执行权利要求1-12任一项所述的方法。
PCT/CN2021/100350 2020-07-28 2021-06-16 深度图像生成方法及装置、参考图像生成方法及装置、电子设备以及计算机可读存储介质 WO2022022136A1 (zh)

Priority Applications (3)

Application Number Priority Date Filing Date Title
EP21849480.5A EP4033757A4 (en) 2020-07-28 2021-06-16 DEPTH IMAGING METHOD AND APPARATUS, REFERENCE IMAGING METHOD AND APPARATUS, ELECTRONIC DEVICE, AND COMPUTER READABLE STORAGE MEDIUM
JP2022548830A JP2023517830A (ja) 2020-07-28 2021-06-16 深度画像生成方法及び装置、基準画像生成方法及び装置、電子機器、ならびにコンピュータプログラム
US17/736,899 US20220262026A1 (en) 2020-07-28 2022-05-04 Depth image generation method and apparatus, reference image generation method and apparatus, electronic device, and computer-readable storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010739666.4A CN112752088B (zh) 2020-07-28 2020-07-28 深度图像生成方法及装置、参考图像生成方法、电子设备
CN202010739666.4 2020-07-28

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/736,899 Continuation US20220262026A1 (en) 2020-07-28 2022-05-04 Depth image generation method and apparatus, reference image generation method and apparatus, electronic device, and computer-readable storage medium

Publications (1)

Publication Number Publication Date
WO2022022136A1 true WO2022022136A1 (zh) 2022-02-03

Family

ID=75645391

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/100350 WO2022022136A1 (zh) 2020-07-28 2021-06-16 深度图像生成方法及装置、参考图像生成方法及装置、电子设备以及计算机可读存储介质

Country Status (5)

Country Link
US (1) US20220262026A1 (zh)
EP (1) EP4033757A4 (zh)
JP (1) JP2023517830A (zh)
CN (1) CN112752088B (zh)
WO (1) WO2022022136A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112752088B (zh) * 2020-07-28 2023-03-28 腾讯科技（深圳）有限公司 深度图像生成方法及装置、参考图像生成方法、电子设备

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103024419A (zh) * 2012-12-31 2013-04-03 青岛海信信芯科技有限公司 一种视频图像处理方法及系统
CN105306922A (zh) * 2014-07-14 2016-02-03 联想(北京)有限公司 一种深度相机参考图的获取方法和装置
US9361698B1 (en) * 2014-11-12 2016-06-07 Amazon Technologies, Inc. Structure light depth sensor
CN106875435A (zh) * 2016-12-14 2017-06-20 深圳奥比中光科技有限公司 获取深度图像的方法及系统
CN108711167A (zh) * 2018-05-15 2018-10-26 深圳奥比中光科技有限公司 深度成像系统及其温度误差校正方法
CN109961406A (zh) * 2017-12-25 2019-07-02 深圳市优必选科技有限公司 一种图像处理的方法、装置及终端设备
CN112752088A (zh) * 2020-07-28 2021-05-04 腾讯科技（深圳）有限公司 深度图像生成方法及装置、参考图像生成方法、电子设备

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106878697A (zh) * 2016-06-29 2017-06-20 鲁班嫡系机器人 一种拍摄方法及其成像方法、装置和设备
JP6736539B2 (ja) * 2017-12-15 2020-08-05 キヤノン株式会社 撮像装置及びその駆動方法
CN108924408B (zh) * 2018-06-15 2020-11-03 深圳奥比中光科技有限公司 一种深度成像方法及系统
CN110088563B (zh) * 2019-03-13 2021-03-19 深圳市汇顶科技股份有限公司 图像深度的计算方法、图像处理装置及三维测量系统

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103024419A (zh) * 2012-12-31 2013-04-03 青岛海信信芯科技有限公司 一种视频图像处理方法及系统
CN105306922A (zh) * 2014-07-14 2016-02-03 联想(北京)有限公司 一种深度相机参考图的获取方法和装置
US9361698B1 (en) * 2014-11-12 2016-06-07 Amazon Technologies, Inc. Structure light depth sensor
CN106875435A (zh) * 2016-12-14 2017-06-20 深圳奥比中光科技有限公司 获取深度图像的方法及系统
CN109961406A (zh) * 2017-12-25 2019-07-02 深圳市优必选科技有限公司 一种图像处理的方法、装置及终端设备
CN108711167A (zh) * 2018-05-15 2018-10-26 深圳奥比中光科技有限公司 深度成像系统及其温度误差校正方法
CN112752088A (zh) * 2020-07-28 2021-05-04 腾讯科技（深圳）有限公司 深度图像生成方法及装置、参考图像生成方法、电子设备

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4033757A4 *

Also Published As

Publication number Publication date
EP4033757A4 (en) 2023-01-25
CN112752088B (zh) 2023-03-28
JP2023517830A (ja) 2023-04-27
US20220262026A1 (en) 2022-08-18
CN112752088A (zh) 2021-05-04
EP4033757A1 (en) 2022-07-27

Similar Documents

Publication Publication Date Title
US11625896B2 (en) Face modeling method and apparatus, electronic device and computer-readable medium
US10334151B2 (en) Phase detection autofocus using subaperture images
US20170059305A1 (en) Active illumination for enhanced depth map generation
US20190164257A1 (en) Image processing method, apparatus and device
KR101903619B1 (ko) 구조화된 스테레오
JP6657214B2 (ja) 画像ベースの深さ検知システムの精度測定
WO2020119467A1 (zh) 高精度稠密深度图像的生成方法和装置
CN107808398B (zh) 摄像头参数算出装置以及算出方法、程序、记录介质
CN107241592B (zh) 一种成像设备及成像方法
Wilm et al. Accurate and simple calibration of DLP projector systems
CN113724391A (zh) 三维模型构建方法、装置、电子设备和计算机可读介质
WO2022022136A1 (zh) 深度图像生成方法及装置、参考图像生成方法及装置、电子设备以及计算机可读存储介质
US20220012905A1 (en) Image processing device and three-dimensional measuring system
KR20180000696A (ko) 적어도 하나의 라이트필드 카메라를 사용하여 입체 이미지 쌍을 생성하는 방법 및 장치
CN113706692B (zh) 三维图像重构方法、装置、电子设备以及存储介质
US11295421B2 (en) Image processing method, image processing device and electronic device
CN111757086A (zh) 有源双目相机、rgb-d图像确定方法及装置
US11195290B2 (en) Apparatus and method for encoding in structured depth camera system
CN112750157B (zh) 一种深度图像生成方法及装置
JP2018081378A (ja) 画像処理装置、撮像装置、画像処理方法および画像処理プログラム
CN113052884A (zh) 信息处理方法、信息处理装置、存储介质与电子设备
KR101632069B1 (ko) 양안 기반에서 굴절 매질을 이용한 깊이 지도 생성 방법 및 장치
TWI831552B (zh) 圖像識別模型訓練方法、圖像深度識別方法及相關設備
CN116366986A (zh) 感光芯片接收光强的确定方法及相关设备
WO2022016331A1 (en) Method of compensating tof depth map and electronic device

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21849480

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021849480

Country of ref document: EP

Effective date: 20220421

ENP Entry into the national phase

Ref document number: 2022548830

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE