WO2019184183A1 - Target image acquisition system and method - Google Patents

Target image acquisition system and method Download PDF

Info

Publication number
WO2019184183A1
WO2019184183A1 PCT/CN2018/099301 CN2018099301W WO2019184183A1 WO 2019184183 A1 WO2019184183 A1 WO 2019184183A1 CN 2018099301 W CN2018099301 W CN 2018099301W WO 2019184183 A1 WO2019184183 A1 WO 2019184183A1
Authority
WO
WIPO (PCT)
Prior art keywords
target
image
structured light
acquisition
camera
Prior art date
Application number
PCT/CN2018/099301
Other languages
French (fr)
Chinese (zh)
Inventor
许星
钟亮洪
Original Assignee
深圳奥比中光科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳奥比中光科技有限公司 filed Critical 深圳奥比中光科技有限公司
Publication of WO2019184183A1 publication Critical patent/WO2019184183A1/en

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60Control of cameras or camera modules
    • GPHYSICS
    • G01MEASURING; TESTING
    • G01BMEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B11/00Measuring arrangements characterised by the use of optical techniques
    • G01B11/24Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures
    • G01B11/25Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object
    • G01B11/2513Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object with several lines being projected in more than one direction, e.g. grids, patterns
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/50Depth or shape recovery
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/50Depth or shape recovery
    • G06T7/521Depth or shape recovery from laser ranging, e.g. using interferometry; from the projection of structured light
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/10Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/56Cameras or camera modules comprising electronic image sensors; Control thereof provided with illuminating means
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/10Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
    • H04N25/11Arrangement of colour filter arrays [CFA]; Filter mosaics
    • H04N25/13Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
    • H04N25/131Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements including elements passing infrared wavelengths
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/10Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
    • H04N25/11Arrangement of colour filter arrays [CFA]; Filter mosaics
    • H04N25/13Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
    • H04N25/133Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements including elements passing panchromatic light, e.g. filters passing white light
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/10Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
    • H04N25/11Arrangement of colour filter arrays [CFA]; Filter mosaics
    • H04N25/13Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
    • H04N25/135Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements based on four or more different wavelength filter elements
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00Details of television systems
    • H04N5/222Studio circuitry; Studio devices; Studio equipment
    • H04N5/2224Studio circuitry; Studio devices; Studio equipment related to virtual studio applications
    • H04N5/2226Determination of depth image, e.g. for foreground/background separation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00Indexing scheme for image data processing or generation, in general
    • G06T2200/04Indexing scheme for image data processing or generation, in general involving 3D image data
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10024Color image
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10048Infrared image

Definitions

  • the present invention relates to acquisition of a target image, and more particularly to a target image acquisition system and method.
  • 3D modeling 3D modeling and face recognition often require depth cameras to output high-resolution depth images to improve the accuracy of modeling and face recognition algorithms.
  • the depth camera's output frame rate has higher requirements, and the high frame rate depth image will reduce the delay and bring a better user experience.
  • the present invention provides a target image acquisition system and method.
  • a target image acquisition system includes: an acquisition camera for acquiring an image of a target area; a floodlight illumination unit for providing flood illumination to the target area; and a structured light projector for projecting to the target area a structured light image; a processor, coupled to the acquisition camera, the floodlighting unit, and the structured light projector, configured to: control the collection camera to collect target floodlights under illumination of the floodlighting unit An image; identifying a foreground object in the target flood image; controlling the acquisition camera to acquire a target structured light image on a pixel corresponding to the foreground object under the projected light projector projection.
  • the present invention also provides a target image acquisition system, the floodlighting unit is a floodlight illuminator; or an illumination source independent of the target image acquisition system; the floodlighting unit and the structured light projector are cross-activated
  • the acquisition camera performs exposure within the activation interval and acquires a target flood image or a structured light image; the activation time of the structured light projection unit is longer than the activation time of the flood illumination unit.
  • the present invention further provides a target image acquisition system, wherein the target floodlight image is acquired by the acquisition camera in a low resolution mode; and the target structured light image is acquired by the acquisition camera in a cropping mode;
  • the processor is further configured to calculate a target depth image using the target structured light image.
  • the present invention provides a target image acquisition method, including: S1: acquiring a target flood image of a target area; S2: identifying a foreground target in the target flood image; S3: collecting a pixel corresponding to the foreground target Target structure light image.
  • step S1 the acquisition camera acquires the target flood image in a low resolution mode; in step S3, the acquisition camera acquires the target structured light image in a crop mode.
  • the acquisition of the target flood image is performed in conjunction with the acquisition of the target structured light image.
  • the invention also provides a target image acquisition method, which further comprises the following steps:
  • the invention has the beneficial effects of providing a target image acquisition system and method, which first acquires a target floodlight image, then extracts a foreground target in the target floodlight image, and then collects a target structure on a pixel corresponding to the foreground target.
  • the light image is processed into a target structured light image before the depth image is acquired.
  • the target structured light image is subjected to depth calculation to obtain a target depth image. Since the amount of data is small relative to the full resolution, the depth is The algorithm also runs faster, ensuring high frame rate output for depth images.
  • FIG. 1 is a schematic diagram of a target image acquisition system in accordance with one embodiment of the present invention.
  • FIG. 2 is a timing diagram of a floodlight illuminator, a structured light projector, and a acquisition camera, in accordance with one embodiment of the present invention.
  • FIG. 3 is a schematic diagram of a target image acquisition method according to an embodiment of the present invention.
  • FIG. 4 is a schematic diagram of a target image acquisition method according to a second embodiment of the present invention.
  • FIG. 5 is a schematic diagram of an acquisition camera image acquisition principle according to an embodiment of the present invention.
  • FIG. 6 is a schematic diagram of an acquisition camera image acquisition principle according to still another embodiment of the present invention.
  • FIG. 7 is a schematic diagram of a target image acquisition method according to a third embodiment of the present invention.
  • FIG. 8 is a schematic diagram of a target image acquisition method according to a fourth embodiment of the present invention.
  • FIG. 9 is a schematic diagram of a target image acquisition system in accordance with still another embodiment of the present invention.
  • FIG. 10 is a schematic diagram of a target image acquisition method according to a fifth embodiment of the present invention.
  • FIG. 11 is a schematic diagram of a target image acquisition method according to a sixth embodiment of the present invention.
  • 10-processor 11-flood illuminator, 12-structured light projector, 13-collector camera, 71-first acquisition camera, 72-structured light projector, 73-second acquisition camera.
  • the target image acquisition system includes a processor 10 and a structured light projector 12 coupled thereto, and a collection camera 13 for projecting a structured light beam into the space, when the structured light beam is incident on the object, A corresponding structured light image is formed which is then acquired by the acquisition camera 13 and forms a structured light image of the object, and the processor 10 further calculates a depth image based on the structured light image.
  • the single structured light projector 12 and the single acquisition camera 13 form a monocular structured light depth imaging system, and the processor 10 will calculate a depth image based on the monocular structure optical trigonometry principle.
  • the processor 10 performs a matching calculation on the currently acquired object structured light image and the pre-stored reference structured light image to obtain a deviation value of the pixel between the two images, and further calculates the depth according to the deviation value.
  • the reference structured light image referred to herein is a plane placed at a known depth distance, which is acquired by the acquisition camera 13 or other acquisition camera when the structured light projector 12 projects a structured light beam onto the plane.
  • two or more acquisition cameras 13 may also be included that form a binocular or multi-view structured light depth imaging system with the structured light projector 12.
  • a binocular structured light system composed of two acquisition cameras 13 and a single structured light projector 12 as an example, when the structured light projector 12 projects a structured light beam into space, the two acquisition cameras 13 collect left and right two.
  • the processor 10 can also obtain the depth image by matching the left and right structured light images based on the binocular vision algorithm; or calculate the left and right structured light images and the corresponding reference structured light images respectively.
  • the left and right acquisition modules can be set to have different parameters, such as different resolutions, focal lengths, etc., so that simultaneous acquisitions can have different a structured light image of a resolution, an angle of view, or the like.
  • a depth image of a different resolution, an angle of view, or the like can be simultaneously acquired; in one embodiment, the acquired plurality of depth images can be merged into one frame and more A depth image of multiple information.
  • the matching calculation refers to selecting a sub-region of a certain size centering on a certain pixel on the current structured light image (or reference structured light image), such as a sub-region of 7 ⁇ 7, 11 ⁇ 11 pixel size, and then in the reference structure.
  • the light image (or the current structured light image) is searched for the sub-region most similar to the sub-region, and the difference between the pixel coordinates of the two sub-regions on the two images is the deviation value; secondly, the difference between the deviation value and the depth value is used.
  • the depth value can be calculated based on the deviation value, and the depth value of a plurality of pixels constitutes a depth image.
  • the principle of matching calculation between two or more structured light images of the left and right is similar to the above principle.
  • the target image acquisition system further includes a floodlight illuminator 11 coupled to the processor 10, the floodlight illuminator 11 being used as a floodlighting illumination unit to provide floodlight illumination.
  • the processor 10 controls the flood illuminator 11, the structured light projector 12, and the acquisition camera 13 through a bus or the like, and can also be connected through some data transmission interfaces, such as an interface connected to the acquisition camera 13 through MIPI, VGA, etc., to receive The image captured by the camera 13 is acquired.
  • floodlight illuminator 11 and structured light projector 12 are used to emit light beams of the same wavelength, such as infrared light, while acquisition camera 13 is comprised of pixels for collecting light beams of that wavelength.
  • the processor 10 can realize the collection of different images by controlling the timing between the three, and specifically can control the target camera to collect the target flood image under the illumination of the floodlighting unit; and identify the foreground in the target flood image.
  • Target controlling the acquisition camera to acquire a target structured light image on a pixel corresponding to the foreground object under the projection of the structured light projector.
  • the floodlighting unit can also be other light sources in the environment, such as ambient light can be used as floodlighting. That is, the floodlight illumination may be active light emitted by a light source such as an infrared light source, or may be ambient light.
  • the processor 10 may be implemented by a depth calculation processor disposed inside the system, which may be a dedicated processor such as a SOC, an FPGA, or the like, or may be a general purpose processor.
  • an external computing device such as a computer, a mobile terminal, a server, or the like, may be utilized.
  • the external computing device receives the structured light image from the acquisition module 13 and performs depth calculation, and the obtained depth image may be directly used for the Other applications for the device.
  • the functions implemented by the processor may be implemented by a processor or an application in the terminal.
  • the depth calculation function is stored in the memory in the form of a software module, and is called by a processor in the terminal to implement depth calculation. It is to be understood that the target image acquisition system provided by the present invention and/or the target image acquisition device employing the target image acquisition method provided by the present invention should be considered as the scope of protection of the present invention.
  • the structured light image may be a stripe pattern, a two-dimensional pattern, a speckle pattern (speckle pattern), or the like
  • the structured light wavelength may be a visible light wavelength, an infrared light wavelength, an ultraviolet light wavelength, or the like.
  • FIG. 2 shows a schematic diagram of the timing control of a floodlight illuminator, a structured light projector, and a acquisition camera.
  • the timing diagrams 20, 21, and 22 correspond to the flood illuminator 11, the structured light projector 12, and the acquisition camera 13, respectively, and the raised portions in the figure indicate that the corresponding device is in an active state, such as the flood illuminator 11 and the structured light projection.
  • the instrument 12 is in a projection state and the acquisition camera is in an exposure state.
  • the processor 10 controls the flood illuminator 11 and the structured light projector 12 to perform cross-activation, while controlling the acquisition camera to perform exposure in each activation interval and collect corresponding images.
  • the floodlight image A is acquired under the illumination of the floodlight illuminator 11, and the structured light image B is acquired by the structured light projector 12, and the floodlight image A and the structured light image B are sequentially output to the processor for processing.
  • a reasonable setting of the activation time of the flood illuminator 11 and the structured light projector 12 to acquire a higher quality image, such as setting the activation time of the structured light projector 12 to be longer,
  • the activation order of the floodlight illuminator 11 and the structured light projector 12 may also be set to other forms, depending on the actual application needs. For example, after the floodlight illuminator 11 is activated twice, the structured light projector 12 is activated.
  • FIG. 1 is a schematic diagram of a method of acquiring a target image, which is executed by the processor 10 to implement a corresponding function, in accordance with one embodiment of the present invention.
  • the acquisition camera 13 is controlled to acquire a target floodlight image under the illumination of the floodlighting unit;
  • the target floodlight image referred to herein refers to a target floodlight image containing the target.
  • the output depth image resolution is often VGA, that is, 640x480, or lower resolution. Therefore, in the present invention, the HD resolution 1280x960 is taken as an example, and It is understood that other resolutions are also suitable for use in the present invention.
  • the processor 10 applies a synchronization trigger signal to the floodlight illuminator 11 and the acquisition camera 13 to use the floodlight image A captured by the acquisition camera 13 to the target area while the floodlight illuminator 11 provides floodlight illumination.
  • the acquisition camera 13 may be a full resolution output, that is, output a 1280 ⁇ 960 resolution floodlight image A.
  • the acquisition camera 13 may also be controlled to use a binning mode or a skipping mode.
  • the low resolution mode acquires a low resolution image of the full field of view. Under the premise that the output frame rate requirement is high and the output interface transmission speed is constant, if the full-resolution image cannot achieve high frame rate output, the low-resolution output mode as described above can be adopted.
  • the floodlight image contains foreground objects of interest, such as faces, human bodies, objects, etc., and also contains some background objects, such as the scene in which the person is located.
  • foreground objects of interest such as faces, human bodies, objects, etc.
  • background objects such as the scene in which the person is located.
  • the foreground and background in the floodlight image need to be segmented, and many image segmentation algorithms can be applied to this step, such as threshold segmentation, mean shift, clustering, and the like.
  • image segmentation algorithm it is necessary to balance the calculation efficiency and the calculation accuracy, especially the calculation efficiency.
  • the slow image segmentation speed will reduce the final image output frame rate (depth image output frame rate).
  • the acquisition camera is controlled to acquire a target structured light image on the pixel corresponding to the foreground object under the projection of the structured light projector. Since the foreground pixel area is acquired in the previous step, in this step, the acquisition camera will only sample the pixel corresponding to the foreground area in the cropping mode, that is, output only the foreground image corresponding to the foreground area. Since the structured light projector is in the on state at this time, the acquired foreground image is the target structured light image. It should be noted that for a dynamic target, such as a moving human body, the pixels corresponding to the target between the two images will be different. Therefore, when the pixel corresponding to the foreground region is selected, the camera can be moved according to the moving speed of the human body. The parameters appropriately expand the pixel area. In fact, in the case of a large frame rate (such as 30fps, 60fps, etc.), the foreground areas in adjacent frame images are nearly identical.
  • the processor 10 will acquire the target structured light image required by the current application, although the structured light image includes only a small field of view, yet has a higher resolution.
  • a depth image is calculated based on the structured light image to obtain a target depth image. Since the amount of data is small relative to the full resolution, the depth algorithm may be faster. Thereby, a high frame rate output of the depth image can be ensured.
  • the acquisition camera can output images up to 1280x960@60fps. If the acquisition camera is used as the structured light image acquisition, only 1280x960 can be realized due to the limitations of the depth calculation algorithm and hardware. The depth image output of @10fps is too low for the depth image frame rate to meet the needs of some applications.
  • the acquisition camera can acquire a flood image of 1280 ⁇ 960@30 fps, and combine the high-speed image segmentation algorithm to identify the foreground target.
  • the target structured light image of 640x480@30fps can be obtained, and the real-time pair can be satisfied according to the current depth calculation algorithm and related hardware.
  • the target structured light image of 640x480@30fps is processed and a depth image of 640x480@30fps is output.
  • the depth image acquired in this embodiment only includes the target, and the detailed information is more abundant, and the step of image segmentation is omitted.
  • the flood illuminator 11 and the structured light projector 12 project light beams of the same wavelength
  • the acquisition camera is configured to respectively acquire the flood image A and the structured light image at different timings.
  • the acquisition camera can simultaneously acquire beams of two wavelengths, which have W pixels that are sensitive to white light (light of all wavelengths) and IR pixels that are sensitive to infrared light.
  • the floodlighting unit is ambient light
  • the acquisition camera can simultaneously collect the floodlight image and the structured light image, except that the effective pixels of the floodlight image and the structured light image are lower than the overall pixels of the acquisition camera.
  • the effective pixel is half of the whole pixel.
  • the acquired depth image has more detailed information.
  • the acquisition camera can simultaneously acquire color images (RGB) and infrared images.
  • RGB color images
  • infrared images For example, a color flood image and an infrared structure light image can be simultaneously acquired, or a color structure light image and an infrared flood image.
  • FIG. 5 and FIG. 6 are only used as an example to illustrate that when the wavelengths of the beams emitted by the flood illuminator 11 and the structured light projector 12 are different, the acquisition cameras that simultaneously sensitize the two wavelengths can be simultaneously acquired.
  • the target flood image and the structured light image are not limited to the examples of FIGS. 5 and 6 in actual use.
  • 7 is a schematic diagram of a target image acquisition method for simultaneously acquiring target floodlight images and structured light images of different wavelengths based on a collection camera, the method being executed by the processor 10 to implement a corresponding function.
  • the acquisition camera is controlled to collect the target flood image and the structured light image under the illumination of the flood illuminator 11 and the structured light projector 12; the flood illuminator 11 and the structured light projector 12 can be always turned on. It can also be turned on according to the frequency and pulsed with a certain gap, and its frequency should be consistent with the exposure frequency of the acquisition camera. Assume that the acquisition camera can output an image of 1280x960@30fps. Under the illumination of the floodlight illuminator and the structured light projector, each image acquired contains the target flood image information and the structured light image information, as shown in FIG.
  • the floodlight image information and the structured light image information are each half of each image, and the acquisition camera then separately extracts the pixels corresponding to the respective images, and fills other blank pixels according to the upsampling algorithm, and finally obtains 1280x960@ A 30fps flood image and a structured light image of 1280x960@30fps, and there is no parallax between the flood image and the structured light image.
  • the foreground and background in the floodlight image need to be segmented, and many image segmentation algorithms can be applied to this step, such as threshold segmentation, mean shift, clustering, and the like.
  • image segmentation algorithm it is necessary to balance the calculation efficiency and the calculation accuracy, especially the calculation efficiency.
  • the slow image segmentation speed will reduce the final image output frame rate (depth image output frame rate).
  • a pixel region corresponding to the foreground object in the structured light image is extracted to obtain a target structured light image. Since there is no parallax between the target flood image and the structured light image, the foreground region in the target flood image recognized in the previous step is also the foreground region in the structured light image, and the structured light image in this region is extracted. The pixel is the target structured light image.
  • the processor 10 will acquire the target structured light image required by the current application, although the structured light image includes only a small field of view, yet has a higher resolution.
  • a depth image can be obtained based on the structured light image to obtain a target depth image. Since the amount of data is small relative to the full resolution, the operation speed of the depth algorithm is also Faster, thus ensuring high frame rate output of depth images.
  • FIG. 7 and FIG. 8 Compared with the methods shown in FIG. 3 and FIG. 4, there is a time difference between the target flood image and the structured light image in the methods of FIGS. 3 and 4, when the target is a moving object.
  • the algorithm may be invalidated.
  • the target floodlight image and the structured light image of Figure 7 and Figure 8 are synchronously acquired, so they can adapt to fast moving objects, but the structured light collected by the acquisition camera The image contains only a portion of the pixels, so the details of the resulting depth image are lost.
  • the target image acquisition system includes a processor 10 and a first acquisition camera 71, a second acquisition camera 73, and a structured light projector 72 and a floodlighting unit connected thereto, because the floodlighting unit uses ambient light in the system. So it is not shown in the picture.
  • the first acquisition camera 71 and the second acquisition camera 73 are respectively used to acquire images of different wavelengths.
  • the first acquisition camera is configured to acquire a target floodlight image of the first wavelength of the target region; the second acquisition camera is configured to acquire the structured light image of the target region.
  • a floodlight illuminator or an illumination source independent of the target image acquisition system may also be included. The following is an example in which ambient light is used as a floodlight.
  • the first acquisition camera is an RGB camera for acquiring RGB images
  • the second acquisition camera is an infrared camera for acquiring IR images
  • the structured light projector is for emitting infrared structured light images. Since there is parallax between the RGB camera and the infrared camera, it is necessary to calibrate the two cameras, which can be calibrated by any calibration method of the prior art. The purpose of the calibration is to obtain the mutual positional relationship between one camera and the other camera. (translation and placement matrix, R and T).
  • FIG. 10 is a schematic diagram of a target image acquisition method according to another embodiment of the present invention. The method is performed by processor 10 to implement the corresponding functions.
  • the RGB camera and the infrared camera are controlled to acquire RGB images and infrared structured light images.
  • the processor 10 controls the RGB camera and the infrared camera to extract RGB images and infrared structured light images at the same frame rate.
  • the resolutions of the RGB images and the infrared images may be the same or different.
  • the RGB cameras in the system are required to perform photographing. Such tasks, so RGB images have higher resolution, but in this embodiment, the acquired RGB images are used for foreground target recognition, so RGB images can be acquired in low resolution mode. It can improve the frame rate of image acquisition and reduce the difficulty of subsequent foreground target recognition.
  • the foreground object in the RGB image is identified; in this step, the foreground and the background in the RGB image need to be segmented, and many image segmentation algorithms can be applied to this step, such as threshold segmentation and mean shift (mean shift). ), clustering, and so on.
  • image segmentation algorithms such as threshold segmentation and mean shift (mean shift). ), clustering, and so on.
  • image segmentation algorithm it is necessary to balance the calculation efficiency and the calculation accuracy, especially the calculation efficiency.
  • the slow image segmentation speed will reduce the final image output frame rate (depth image output frame rate).
  • the target structured light image on the pixel corresponding to the foreground target on the infrared structured light image is extracted.
  • the region of the corresponding foreground target in the target structured light image can be located, and the pixel of the region can be further extracted as a target. Structured light image.
  • the processor 10 will acquire the target structured light image required by the current application, and then calculate the depth value of each pixel in the target structured light image by using a depth algorithm to generate a target depth image. .
  • the depth calculation can be performed in real time, thereby achieving high frame rate output.
  • the RGB image and the infrared image may also be acquired asynchronously, and the RGB image and the infrared image may be separately collected at a certain timing according to a form similar to the embodiment shown in FIG. 3 and FIG.
  • the infrared camera can acquire the infrared structured light image by using the cropping mode based on the foreground target area identified in the RGB image, thereby further reducing the amount of data to ensure high-speed output.

Abstract

Provided are a target image acquisition system and method. The system comprises: a collection camera for collecting an image of a target region; a floodlighting unit for providing the target region with floodlighting; a structure light projector for projecting a structure light image to the target region; and a processor connected to the collection camera, the floodlighting unit and the structure light projector, and used for controlling the collection camera to collect a target floodlight image under the lighting of the floodlighting unit, for identifying a foreground target in the target floodlight image, and for controlling the collection camera to collect a target structure light image on a pixel corresponding to the foreground target under the projection of the structure light projector. Before a depth image is acquired, a structure light image is processed to be a target structure light image, and depth calculation is then carried out on the target structure light image to obtain a target depth image, thereby ensuring the high frame rate output of the depth image.

Description

目标图像获取系统与方法Target image acquisition system and method 技术领域Technical field
本发明涉及目标图像的获取,尤其涉及一种目标图像获取系统与方法。The present invention relates to acquisition of a target image, and more particularly to a target image acquisition system and method.
背景技术Background technique
消费级深度相机的出现给众多领域带来了变革,比如3D建模、手势交互、人脸识别等。不同的应用场景对深度相机的性能要求也不一样,比如3D建模以及人脸识别等往往需要深度相机输出高分辨率的深度图像以提高建模以及人脸识别算法精度;而手势交互则对深度相机的输出帧率有较高的要求,高帧率的深度图像会降低延迟,带来更好的用户体验。The emergence of consumer-grade depth cameras has revolutionized many areas, such as 3D modeling, gesture interaction, and face recognition. Different application scenarios have different performance requirements for depth cameras. For example, 3D modeling and face recognition often require depth cameras to output high-resolution depth images to improve the accuracy of modeling and face recognition algorithms. The depth camera's output frame rate has higher requirements, and the high frame rate depth image will reduce the delay and bring a better user experience.
目前,消费级深度相机所面临的问题之一即是深度图像分辨率与输出帧率之间的矛盾,特别是对于基于结构光技术的深度相机,当其深度图像分辨率越高时,由于深度计算量的增加导致其输出帧率大幅下降,高分辨率与高帧率无法同时实现。At present, one of the problems faced by consumer-grade depth cameras is the contradiction between depth image resolution and output frame rate, especially for depth cameras based on structured light technology, when the depth image resolution is higher, due to depth The increase in the amount of calculation results in a large drop in the output frame rate, and high resolution and high frame rate cannot be achieved at the same time.
发明内容Summary of the invention
本发明为了解决现有技术中高分辨率与高帧率无法同时实现的问题,提供一种目标图像获取系统与方法。In order to solve the problem that the high resolution and the high frame rate cannot be simultaneously implemented in the prior art, the present invention provides a target image acquisition system and method.
为了解决上述问题,本发明采用的技术方案如下所述:In order to solve the above problems, the technical solution adopted by the present invention is as follows:
一种目标图像获取系统,包括:采集相机,用于采集目标区域的图像;泛光照明单元,用于对所述目标区域提供泛光照明;结构光投影仪,用于向所述目标区域投射结构光图像;处理器,与所述采集相机、所述泛光照明单元以及所述结构光投影仪连接,用于:控制所述采集相机采集在所述泛光照明单元照明下的目标泛光图像;识别所述目标泛光图像中的前景目标;控制所述采集相机采集在所述结构光投影仪投影下与所述前景目标对应的像素上的目标结构光图像。A target image acquisition system includes: an acquisition camera for acquiring an image of a target area; a floodlight illumination unit for providing flood illumination to the target area; and a structured light projector for projecting to the target area a structured light image; a processor, coupled to the acquisition camera, the floodlighting unit, and the structured light projector, configured to: control the collection camera to collect target floodlights under illumination of the floodlighting unit An image; identifying a foreground object in the target flood image; controlling the acquisition camera to acquire a target structured light image on a pixel corresponding to the foreground object under the projected light projector projection.
本发明还提供一种目标图像获取系统,所述泛光照明单元是泛光照明器;或,与所述目标图像获取系统独立的照明光源;所述泛光照明单元和结构光投影仪交叉激活;所述采集相机在激活区间内进行曝光并采集目标泛光图像或结构光图像;所述结构光投影单元的激活时间比所述泛光照明单元的激活时间长。The present invention also provides a target image acquisition system, the floodlighting unit is a floodlight illuminator; or an illumination source independent of the target image acquisition system; the floodlighting unit and the structured light projector are cross-activated The acquisition camera performs exposure within the activation interval and acquires a target flood image or a structured light image; the activation time of the structured light projection unit is longer than the activation time of the flood illumination unit.
本发明又提供一种目标图像获取系统,所述目标泛光图像是所述采集相机在低分辨率模式下采集的;所述目标结构光图像是所述采集相机在裁剪模式下采集 的;所述处理器还用于利用所述目标结构光图像计算出目标深度图像。The present invention further provides a target image acquisition system, wherein the target floodlight image is acquired by the acquisition camera in a low resolution mode; and the target structured light image is acquired by the acquisition camera in a cropping mode; The processor is further configured to calculate a target depth image using the target structured light image.
本发明提供一种目标图像获取方法,包括:S1:采集目标区域的目标泛光图像;S2:识别所述目标泛光图像中的前景目标;S3:采集与所述前景目标对应的像素上的目标结构光图像。步骤S1中,采集相机在低分辨率模式下采集所述目标泛光图像;步骤S3中,采集相机在裁剪模式下采集所述目标结构光图像。采集所述目标泛光图像与采集所述目标结构光图像是交叉进行的。The present invention provides a target image acquisition method, including: S1: acquiring a target flood image of a target area; S2: identifying a foreground target in the target flood image; S3: collecting a pixel corresponding to the foreground target Target structure light image. In step S1, the acquisition camera acquires the target flood image in a low resolution mode; in step S3, the acquisition camera acquires the target structured light image in a crop mode. The acquisition of the target flood image is performed in conjunction with the acquisition of the target structured light image.
本发明还提供一种目标图像获取方法,,还包括如下步骤:The invention also provides a target image acquisition method, which further comprises the following steps:
S4:利用所述目标结构光图像计算出目标深度图像。S4: Calculate the target depth image by using the target structured light image.
本发明的有益效果为:提供一种目标图像获取系统和方法,通过先获取目标泛光图像,然后提取目标泛光图像中的前景目标,再采集与所述前景目标对应的像素上的目标结构光图像,在深度图像获取之前对结构光图像处理使其成为目标结构光图像,此时再对目标结构光图像进行深度计算得到目标深度图像,由于数据量相对于全分辨率较小,因此深度算法的运算速度也会较快,从而可以确保深度图像的高帧率输出。The invention has the beneficial effects of providing a target image acquisition system and method, which first acquires a target floodlight image, then extracts a foreground target in the target floodlight image, and then collects a target structure on a pixel corresponding to the foreground target. The light image is processed into a target structured light image before the depth image is acquired. At this time, the target structured light image is subjected to depth calculation to obtain a target depth image. Since the amount of data is small relative to the full resolution, the depth is The algorithm also runs faster, ensuring high frame rate output for depth images.
附图说明DRAWINGS
图1是根据本发明一个实施例的目标图像获取系统示意图。1 is a schematic diagram of a target image acquisition system in accordance with one embodiment of the present invention.
图2是根据本发明一个实施例的泛光照明器、结构光投影仪以及采集相机的时序控制示意图。2 is a timing diagram of a floodlight illuminator, a structured light projector, and a acquisition camera, in accordance with one embodiment of the present invention.
图3是根据本发明一个实施例的目标图像获取方法示意图。FIG. 3 is a schematic diagram of a target image acquisition method according to an embodiment of the present invention.
图4是根据本发明第二个实施例的目标图像获取方法示意图4 is a schematic diagram of a target image acquisition method according to a second embodiment of the present invention;
图5是根据本发明一个实施例的采集相机图像采集原理示意图。FIG. 5 is a schematic diagram of an acquisition camera image acquisition principle according to an embodiment of the present invention.
图6是根据本发明又一个实施例的采集相机图像采集原理示意图。FIG. 6 is a schematic diagram of an acquisition camera image acquisition principle according to still another embodiment of the present invention.
图7是根据本发明第三个实施例的目标图像获取方法示意图。FIG. 7 is a schematic diagram of a target image acquisition method according to a third embodiment of the present invention.
图8是根据本发明第四个实施例的目标图像获取方法示意图。FIG. 8 is a schematic diagram of a target image acquisition method according to a fourth embodiment of the present invention.
图9是根据本发明又一个实施例的目标图像获取系统示意图。9 is a schematic diagram of a target image acquisition system in accordance with still another embodiment of the present invention.
图10是根据本发明第五个实施例的目标图像获取方法示意图。FIG. 10 is a schematic diagram of a target image acquisition method according to a fifth embodiment of the present invention.
图11是根据本发明第六个实施例的目标图像获取方法示意图。11 is a schematic diagram of a target image acquisition method according to a sixth embodiment of the present invention.
其中,10-处理器,11-泛光照明器,12-结构光投影仪,13-采集相机,71-第一采集相机,72-结构光投影仪,73-第二采集相机。Among them, 10-processor, 11-flood illuminator, 12-structured light projector, 13-collector camera, 71-first acquisition camera, 72-structured light projector, 73-second acquisition camera.
具体实施方式detailed description
下面结合附图通过具体实施例对本发明进行详细的介绍,以使更好的理解本发明,但下述实施例并不限制本发明范围。另外,需要说明的是,下述实施例中所提供的图示仅以示意方式说明本发明的基本构思,附图中仅显示与本发明中有关的组件而非按照实际实施时的组件数目、形状及尺寸绘制,其实际实施时各组件的形状、数量及比例可为一种随意的改变,且其组件布局形态也可能更为复杂。The present invention will be described in detail with reference to the accompanying drawings, in order to provide a better understanding of the invention. In addition, it should be noted that the illustrations provided in the following embodiments merely illustrate the basic concept of the present invention in a schematic manner, and only the components related to the present invention are shown in the drawings, instead of the number of components in actual implementation, The shape and size are drawn. In the actual implementation, the shape, number and proportion of each component can be a random change, and the component layout form may be more complicated.
图1是根据本发明一个实施例的目标图像获取系统示意图。目标图像获取系统包括处理器10以及与之连接的结构光投影仪12以及采集相机13,其中结构光投影仪12用于向空间中投射结构光光束,当结构光光束照射到物体上时,会形成相应的结构光图像,该图案随后被采集相机13采集并形成该物体的结构光图像,处理器10基于该结构光图像进一步计算出深度图像。1 is a schematic diagram of a target image acquisition system in accordance with one embodiment of the present invention. The target image acquisition system includes a processor 10 and a structured light projector 12 coupled thereto, and a collection camera 13 for projecting a structured light beam into the space, when the structured light beam is incident on the object, A corresponding structured light image is formed which is then acquired by the acquisition camera 13 and forms a structured light image of the object, and the processor 10 further calculates a depth image based on the structured light image.
单个结构光投影仪12与单个采集相机13组成单目结构光深度成像系统,处理器10将基于单目结构光三角法原理计算出深度图像。在一个实施例中,处理器10将当前采集到的物体结构光图像与预先保存的参考结构光图像进行匹配计算,以得到两幅图像之间像素的偏离值,根据该偏离值进一步计算出深度值,这里所说的参考结构光图像是在已知深度距离上放置一块平面,当结构光投影仪12向该平面投影出结构光光束后由采集相机13或其他采集相机采集得到的。The single structured light projector 12 and the single acquisition camera 13 form a monocular structured light depth imaging system, and the processor 10 will calculate a depth image based on the monocular structure optical trigonometry principle. In one embodiment, the processor 10 performs a matching calculation on the currently acquired object structured light image and the pre-stored reference structured light image to obtain a deviation value of the pixel between the two images, and further calculates the depth according to the deviation value. Value, the reference structured light image referred to herein is a plane placed at a known depth distance, which is acquired by the acquisition camera 13 or other acquisition camera when the structured light projector 12 projects a structured light beam onto the plane.
在一些实施例中,也可以包含两个及以上的采集相机13,其与结构光投影仪12组成双目或多目结构光深度成像系统。以两个采集相机13与单个结构光投影仪12组成的双目结构光系统为例进行说明,当结构光投影仪12向空间中投射结构光光束后,两个采集相机13采集左、右两幅结构光图像,处理器10基于双目视觉算法通过对左、右结构光图像的匹配计算也可以获取深度图像;也可以分别对左、右结构光图像与各自对应的参考结构光图像进行计算以获取两幅深度图像,这样做的好处在于,在一个实施例中可以将左、右采集模组设置成具有不同的参数,比如不同的分辨率、焦距等,由此可以同时采集具有比如不同分辨率、视场角等的结构光图像,进一步地,可以同时获取不同分辨率、视场角等的深度图像;在一个实施例中,还可以将获取的多个深度图像融合成一幅具备更多信息的深度图像。In some embodiments, two or more acquisition cameras 13 may also be included that form a binocular or multi-view structured light depth imaging system with the structured light projector 12. Taking a binocular structured light system composed of two acquisition cameras 13 and a single structured light projector 12 as an example, when the structured light projector 12 projects a structured light beam into space, the two acquisition cameras 13 collect left and right two. For the structured light image, the processor 10 can also obtain the depth image by matching the left and right structured light images based on the binocular vision algorithm; or calculate the left and right structured light images and the corresponding reference structured light images respectively. To obtain two depth images, the advantage of this is that in one embodiment, the left and right acquisition modules can be set to have different parameters, such as different resolutions, focal lengths, etc., so that simultaneous acquisitions can have different a structured light image of a resolution, an angle of view, or the like. Further, a depth image of a different resolution, an angle of view, or the like can be simultaneously acquired; in one embodiment, the acquired plurality of depth images can be merged into one frame and more A depth image of multiple information.
在一些实施例中,匹配计算指的是在当前结构光图像(或参考结构光图像)上 以某像素为中心选取一定大小的子区域,比如7x7、11x11像素大小的子区域,然后在参考结构光图像(或当前结构光图像)上搜索与子区域最为相似的子区域,两个子区域在两幅图像上像素坐标之间的差值即为偏离值;其次利用偏离值与深度值之间的对应关系,基于偏离值就可以计算出深度值,多个像素的深度值就构成了深度图像。对于左、右两幅或多幅结构光图像之间的匹配计算原理与上述原理类似。In some embodiments, the matching calculation refers to selecting a sub-region of a certain size centering on a certain pixel on the current structured light image (or reference structured light image), such as a sub-region of 7×7, 11×11 pixel size, and then in the reference structure. The light image (or the current structured light image) is searched for the sub-region most similar to the sub-region, and the difference between the pixel coordinates of the two sub-regions on the two images is the deviation value; secondly, the difference between the deviation value and the depth value is used. Correspondence relationship, the depth value can be calculated based on the deviation value, and the depth value of a plurality of pixels constitutes a depth image. The principle of matching calculation between two or more structured light images of the left and right is similar to the above principle.
在一些实施方式中,目标图像获取系统还包括与处理器10连接的泛光照明器11,泛光照明器11作为泛光照明单元用来提供泛光照明。处理器10通过总线等方式控制泛光照明器11、结构光投影仪12以及采集相机13,也可以通过一些数据传输接口进行连接,比如通过MIPI、VGA等接口与采集相机13连接,以接收由采集相机13采集到的图像。在一个实施例中,泛光照明器11与结构光投影仪12用于发射相同波长的光束,比如红外光,而采集相机13由包含了用于采集该波长光束的像素。处理器10可以通过对三者之间时序上的控制来实现对不同图像的采集,具体的可以控制采集相机采集在泛光照明单元照明下的目标泛光图像;识别目标泛光图像中的前景目标;控制采集相机采集在结构光投影仪投影下与前景目标对应的像素上的目标结构光图像。在一些实施例中,泛光照明单元也可以是环境中的其他光源,比如环境光可以作为泛光照明。即泛光照明可以是红外光源等光源发出的主动光,也可以是环境光。在下述具体实施例中,有些是以系统中包含泛光照明器的情况下进行的描述,有些是以环境光作为泛光照明单元进行的描述,应该可以理解的是,根据不同的情况,可以选择具体的泛光照明的形式,但是其方法是通用的,以下不作具体的区分。In some embodiments, the target image acquisition system further includes a floodlight illuminator 11 coupled to the processor 10, the floodlight illuminator 11 being used as a floodlighting illumination unit to provide floodlight illumination. The processor 10 controls the flood illuminator 11, the structured light projector 12, and the acquisition camera 13 through a bus or the like, and can also be connected through some data transmission interfaces, such as an interface connected to the acquisition camera 13 through MIPI, VGA, etc., to receive The image captured by the camera 13 is acquired. In one embodiment, floodlight illuminator 11 and structured light projector 12 are used to emit light beams of the same wavelength, such as infrared light, while acquisition camera 13 is comprised of pixels for collecting light beams of that wavelength. The processor 10 can realize the collection of different images by controlling the timing between the three, and specifically can control the target camera to collect the target flood image under the illumination of the floodlighting unit; and identify the foreground in the target flood image. Target; controlling the acquisition camera to acquire a target structured light image on a pixel corresponding to the foreground object under the projection of the structured light projector. In some embodiments, the floodlighting unit can also be other light sources in the environment, such as ambient light can be used as floodlighting. That is, the floodlight illumination may be active light emitted by a light source such as an infrared light source, or may be ambient light. In the following specific embodiments, some are described in the case where the system includes a floodlight illuminator, and some are described as ambient light as a floodlighting unit. It should be understood that, depending on the situation, The specific form of floodlighting is chosen, but the method is universal, and no specific distinction is made below.
处理器10可以是配置在系统内部的深度计算处理器来执行,该处理器可以专用处理器如SOC、FPGA等,也可以是通用处理器。在一些实施例中,也可以利用外部计算设备,如计算机、移动终端、服务器等设备,外部计算设备接收来自采集模组13的结构光图像后实施深度计算,得到的深度图像可直接用于该设备的其他应用。在一个实施例中,当系统作为嵌入式装置集成到其他计算终端时,如电脑、平板、手机、电视等目标图像获取装置,处理器所实现的功能可以由终端内的处理器或应用来完成,比如将深度计算功能以软件模块形式存储在存储器中,被终端内的处理器调用从而实现深度计算。可以理解的是,采用本发明提供 的目标图像获取系统和/或采用本发明提供的目标图像获取方法的目标图像获取装置都应该视为本发明保护的范围。The processor 10 may be implemented by a depth calculation processor disposed inside the system, which may be a dedicated processor such as a SOC, an FPGA, or the like, or may be a general purpose processor. In some embodiments, an external computing device, such as a computer, a mobile terminal, a server, or the like, may be utilized. The external computing device receives the structured light image from the acquisition module 13 and performs depth calculation, and the obtained depth image may be directly used for the Other applications for the device. In one embodiment, when the system is integrated as an embedded device to other computing terminals, such as a target image acquiring device such as a computer, a tablet, a mobile phone, or a television, the functions implemented by the processor may be implemented by a processor or an application in the terminal. For example, the depth calculation function is stored in the memory in the form of a software module, and is called by a processor in the terminal to implement depth calculation. It is to be understood that the target image acquisition system provided by the present invention and/or the target image acquisition device employing the target image acquisition method provided by the present invention should be considered as the scope of protection of the present invention.
结构光图像可以是条纹图案、二维图案、散斑图案(斑点图案)等,结构光波长可以是可见光波长、红外光波长、紫外光波长等。The structured light image may be a stripe pattern, a two-dimensional pattern, a speckle pattern (speckle pattern), or the like, and the structured light wavelength may be a visible light wavelength, an infrared light wavelength, an ultraviolet light wavelength, or the like.
图2所示的是泛光照明器、结构光投影仪以及采集相机的时序控制示意图。其中时序图20、21以及22分别对应泛光照明器11、结构光投影仪12以及采集相机13,图中凸起的部分表示相应的器件处于激活状态,比如泛光照明器11与结构光投影仪12位于投影状态、采集相机位于曝光状态。从图2所示可以看出,本实施例中,处理器10控制泛光照明器11以及结构光投影仪12进行交叉激活,同时控制采集相机在各个激活区间内进行曝光并采集相应的图像。在泛光照明器11照射下采集到泛光图像A,在结构光投影仪12投射下采集到结构光图像B,泛光图像A与结构光图像B依次输出至处理器进行处理。在一些实施例中,通过将泛光照明器11以及结构光投影仪12的激活时长的合理设置以采集到更高质量的图像,比如将结构光投影仪12的激活时间设置得更长一些,以确保满足足够的曝光时间以采集到更高质量的结构光图像;在一些实施例中,根据实际应用需要,泛光照明器11与结构光投影仪12的激活次序也可以设置成其他形式,比如激活两次泛光照明器11后激活一次结构光投影仪12等。Figure 2 shows a schematic diagram of the timing control of a floodlight illuminator, a structured light projector, and a acquisition camera. The timing diagrams 20, 21, and 22 correspond to the flood illuminator 11, the structured light projector 12, and the acquisition camera 13, respectively, and the raised portions in the figure indicate that the corresponding device is in an active state, such as the flood illuminator 11 and the structured light projection. The instrument 12 is in a projection state and the acquisition camera is in an exposure state. As can be seen from FIG. 2, in the present embodiment, the processor 10 controls the flood illuminator 11 and the structured light projector 12 to perform cross-activation, while controlling the acquisition camera to perform exposure in each activation interval and collect corresponding images. The floodlight image A is acquired under the illumination of the floodlight illuminator 11, and the structured light image B is acquired by the structured light projector 12, and the floodlight image A and the structured light image B are sequentially output to the processor for processing. In some embodiments, by arranging a reasonable setting of the activation time of the flood illuminator 11 and the structured light projector 12 to acquire a higher quality image, such as setting the activation time of the structured light projector 12 to be longer, To ensure that sufficient exposure time is met to acquire a higher quality structured light image; in some embodiments, the activation order of the floodlight illuminator 11 and the structured light projector 12 may also be set to other forms, depending on the actual application needs. For example, after the floodlight illuminator 11 is activated twice, the structured light projector 12 is activated.
在一些应用中,要求获取被测量目标的高分辨率深度图像,然而受限于深度计算算法以及处理器计算能力的限制,实现高分辨率深度图像采集往往需要花费较高的成本。本发明一个实施方式中将提供基于图1所示系统的获取高分辨率目标深度图像的方法。图3是根据本发明一个实施例的目标图像获取方法示意图,该方法被处理器10执行以实现相应的功能。In some applications, it is required to obtain a high resolution depth image of the measured object. However, limited by the depth calculation algorithm and the limitation of the computing power of the processor, achieving high resolution depth image acquisition often requires a high cost. A method of acquiring a high resolution target depth image based on the system of FIG. 1 will be provided in one embodiment of the present invention. 3 is a schematic diagram of a method of acquiring a target image, which is executed by the processor 10 to implement a corresponding function, in accordance with one embodiment of the present invention.
首先,控制采集相机13采集在泛光照明单元照明下的目标泛光图像;这里说的目标泛光图像是指含有目标的目标泛光图像。已有技术中,比如微软kinect、Intel realsense等深度相机,其输出深度图像分辨率往往为VGA,即640x480,或者更低分辨率,因此在本发明中以高清分辨率1280x960为例进行说明,可以理解的是,其他分辨率也适用于本发明。在本步骤中,处理器10给泛光照明器11以及采集相机13施加同步触发信号,以使用在泛光照明器11提供泛光照明的同时,采集相机13采集到目标区域的泛光图像A,此时采集相机13可以是全 分辨率输出,即输出1280x960分辨率的泛光图像A,在一个实施例中,也可以控制采集相机13使用binning mode(合并模式)或者skipping mode(抽取模式)等低分辨率模式获取全视场的低分辨率图像。在输出帧率要求较高以及输出接口传输速度一定的前提下,若全分辨率图像无法实现高帧率的输出,则可以采取如上所述的低分辨率输出模式。First, the acquisition camera 13 is controlled to acquire a target floodlight image under the illumination of the floodlighting unit; the target floodlight image referred to herein refers to a target floodlight image containing the target. In the prior art, for example, the depth camera of Microsoft kinect, Intel realsense, etc., the output depth image resolution is often VGA, that is, 640x480, or lower resolution. Therefore, in the present invention, the HD resolution 1280x960 is taken as an example, and It is understood that other resolutions are also suitable for use in the present invention. In this step, the processor 10 applies a synchronization trigger signal to the floodlight illuminator 11 and the acquisition camera 13 to use the floodlight image A captured by the acquisition camera 13 to the target area while the floodlight illuminator 11 provides floodlight illumination. At this time, the acquisition camera 13 may be a full resolution output, that is, output a 1280×960 resolution floodlight image A. In one embodiment, the acquisition camera 13 may also be controlled to use a binning mode or a skipping mode. The low resolution mode acquires a low resolution image of the full field of view. Under the premise that the output frame rate requirement is high and the output interface transmission speed is constant, if the full-resolution image cannot achieve high frame rate output, the low-resolution output mode as described above can be adopted.
一般地,泛光图像中含有感兴趣的前景目标,比如人脸、人体、物体等,同时也会含有一些背景目标,比如人所在的场景等。对于一些应用而言,如人脸识别、3D建模等,往往仅需要前景目标信息,而背景则需要去除。Generally, the floodlight image contains foreground objects of interest, such as faces, human bodies, objects, etc., and also contains some background objects, such as the scene in which the person is located. For some applications, such as face recognition, 3D modeling, etc., only foreground target information is needed, and the background needs to be removed.
其次,识别目标泛光图像中的前景目标。在这一步骤中,需要将泛光图像中的前景与背景进行分割,诸多图像分割算法均可以应用到本步骤中,如阈值分割法、均值法(mean shift)、聚类法等等。在选取图像分割算法时,需要兼顾计算效率与计算精度,特别是计算效率,图像分割速度慢会降低最终的图像输出帧率(深度图像输出帧率)。在前景区域被分割后,识别出前景区域,或者说识别出前景区域所在的前景像素区域。Second, identify foreground targets in the target flood image. In this step, the foreground and background in the floodlight image need to be segmented, and many image segmentation algorithms can be applied to this step, such as threshold segmentation, mean shift, clustering, and the like. When selecting the image segmentation algorithm, it is necessary to balance the calculation efficiency and the calculation accuracy, especially the calculation efficiency. The slow image segmentation speed will reduce the final image output frame rate (depth image output frame rate). After the foreground region is segmented, the foreground region is identified, or the foreground pixel region where the foreground region is located is identified.
最后,控制采集相机采集在结构光投影仪投影下与前景目标对应的像素上的目标结构光图像。由于在上一步骤中获取了前景像素区域,在本步骤中,采集相机将在裁剪模式下(cropping mode)仅对与前景区域对应的像素进行采样,即仅输出与前景区域对应的前景图像,由于此时结构光投影仪处于开启状态,因此获取的前景图像为目标结构光图像。需要注意的是,对于动态目标,比如移动的人体,前后两幅图像之间目标所对应的像素也会有区别,因此,在选取与前景区域对应的像素时,可以根据人体的移动速度与相机的参数适当扩大像素区域。事实上,在帧率较大(如30fps、60fps等)的情况下,相邻帧图像中前景区域近乎相同。Finally, the acquisition camera is controlled to acquire a target structured light image on the pixel corresponding to the foreground object under the projection of the structured light projector. Since the foreground pixel area is acquired in the previous step, in this step, the acquisition camera will only sample the pixel corresponding to the foreground area in the cropping mode, that is, output only the foreground image corresponding to the foreground area. Since the structured light projector is in the on state at this time, the acquired foreground image is the target structured light image. It should be noted that for a dynamic target, such as a moving human body, the pixels corresponding to the target between the two images will be different. Therefore, when the pixel corresponding to the foreground region is selected, the camera can be moved according to the moving speed of the human body. The parameters appropriately expand the pixel area. In fact, in the case of a large frame rate (such as 30fps, 60fps, etc.), the foreground areas in adjacent frame images are nearly identical.
在以上的各个步骤后,处理器10将获取当前应用所需要的目标结构光图像,虽然该结构光图像仅包括较小的视场角,然而却具有较高的分辨率。After each of the above steps, the processor 10 will acquire the target structured light image required by the current application, although the structured light image includes only a small field of view, yet has a higher resolution.
如图4所示,在本发明的变通实施例中,基于该结构光图像进行深度计算得到目标深度图像,由于数据量相对于全分辨率较小,因此深度算法的运算速度也会较快,从而可以确保深度图像的高帧率输出。As shown in FIG. 4, in an alternative embodiment of the present invention, a depth image is calculated based on the structured light image to obtain a target depth image. Since the amount of data is small relative to the full resolution, the depth algorithm may be faster. Thereby, a high frame rate output of the depth image can be ensured.
现在以一个更直观的实施例来说明以上各步骤,比如采集相机最高可以输出1280x960@60fps的图像,若利用该采集相机作为结构光图像采集,由于深度计 算算法以及硬件等限制,仅能实现1280x960@10fps的深度图像输出,由于深度图像帧率太低,导致无法满足一些应用的需要。而利用以上所述的方法,即在泛光图像以及结构光投影仪以交叉时序进行开启,采集相机则可以获取1280x960@30fps的泛光图像,结合高速度的图像分割算法,在识别到前景目标区域后(假设目标区域位于采集相机视场角中间且占整个视场的50%),由此则可以获取640x480@30fps的目标结构光图像,根据当前深度计算算法以及相关硬件,可以满足实时对640x480@30fps的目标结构光图像进行处理并输出640x480@30fps的深度图像。与直接采用640x480@30fps的采集相机相比,本实施例所获取的深度图像仅包含目标,细节信息更加丰富,同时省去了图像分割的步骤。Now use a more intuitive embodiment to illustrate the above steps. For example, the acquisition camera can output images up to 1280x960@60fps. If the acquisition camera is used as the structured light image acquisition, only 1280x960 can be realized due to the limitations of the depth calculation algorithm and hardware. The depth image output of @10fps is too low for the depth image frame rate to meet the needs of some applications. By using the method described above, that is, when the floodlight image and the structured light projector are turned on at the cross timing, the acquisition camera can acquire a flood image of 1280×960@30 fps, and combine the high-speed image segmentation algorithm to identify the foreground target. After the region (assuming that the target region is located in the middle of the field of view of the acquisition camera and accounts for 50% of the entire field of view), the target structured light image of 640x480@30fps can be obtained, and the real-time pair can be satisfied according to the current depth calculation algorithm and related hardware. The target structured light image of 640x480@30fps is processed and a depth image of 640x480@30fps is output. Compared with the acquisition camera directly adopting 640x480@30fps, the depth image acquired in this embodiment only includes the target, and the detailed information is more abundant, and the step of image segmentation is omitted.
图2与图3、图4所示的实施例中,泛光照明器11与结构光投影仪12投射相同波长的光束,采集相机用于在不同时序上分别获取泛光图像A与结构光图像B。In the embodiment shown in FIG. 2 and FIG. 3 and FIG. 4, the flood illuminator 11 and the structured light projector 12 project light beams of the same wavelength, and the acquisition camera is configured to respectively acquire the flood image A and the structured light image at different timings. B.
图5与图6所示的是根据本发明一些实施例的采集相机采集原理示意图。图5中,采集相机可以对两种波长的光束进行同步采集,其拥有对白光(所有波长的光)感光的W像素以及对红外光感光的IR像素,当泛光照明单元为环境光、结构光投影仪用于投射红外结构光时,采集相机可以同时采集泛光图像与结构光图像,只不过泛光图像与结构光图像的有效像素要低于采集相机整体像素。在本实施例中,有效像素为整体像素的一半,在其他实施例中,泛光图像与结构光图像像素也可以为其他比例,比如W:IR=1:3,由此可以保证结构光图像中拥有更多的图像细节,所获取的深度图像具有更精细的信息。在图6中,采集相机可以同时采集彩色图像(RGB)与红外图像,比如可以同时采集彩色泛光图像与红外结构光图像,也可以是彩色结构光图像与红外泛光图像。5 and 6 are schematic diagrams of acquisition camera acquisition principles in accordance with some embodiments of the present invention. In Figure 5, the acquisition camera can simultaneously acquire beams of two wavelengths, which have W pixels that are sensitive to white light (light of all wavelengths) and IR pixels that are sensitive to infrared light. When the floodlighting unit is ambient light, structure When the light projector is used to project infrared structured light, the acquisition camera can simultaneously collect the floodlight image and the structured light image, except that the effective pixels of the floodlight image and the structured light image are lower than the overall pixels of the acquisition camera. In this embodiment, the effective pixel is half of the whole pixel. In other embodiments, the flood image and the structured light image pixel may also be other ratios, such as W:IR=1:3, thereby ensuring the structured light image. With more image detail, the acquired depth image has more detailed information. In FIG. 6, the acquisition camera can simultaneously acquire color images (RGB) and infrared images. For example, a color flood image and an infrared structure light image can be simultaneously acquired, or a color structure light image and an infrared flood image.
可以理解的是,图5与图6仅作为示例来说明当泛光照明器11与结构光投影仪12所发出光束波长不同时,可以利用对两种波长同时感光的采集相机同时采集与之对应的目标泛光图像与结构光图像,实际使用中不限于图5和图6的示例。图7是基于采集相机同时采集不同波长的目标泛光图像和结构光图像的目标图像获取方法示意图,该方法被处理器10执行以实现相应的功能。It can be understood that FIG. 5 and FIG. 6 are only used as an example to illustrate that when the wavelengths of the beams emitted by the flood illuminator 11 and the structured light projector 12 are different, the acquisition cameras that simultaneously sensitize the two wavelengths can be simultaneously acquired. The target flood image and the structured light image are not limited to the examples of FIGS. 5 and 6 in actual use. 7 is a schematic diagram of a target image acquisition method for simultaneously acquiring target floodlight images and structured light images of different wavelengths based on a collection camera, the method being executed by the processor 10 to implement a corresponding function.
首先,控制所述采集相机采集在泛光照明器11以及所述结构光投影仪12照 明下的目标泛光图像与结构光图像;泛光照明器11与结构光投影仪12可以一直处于开启状态,也可以按照频率开启并以一定的间隙脉冲发光,其频率应与采集相机的曝光频率一致。假定采集相机可以输出1280x960@30fps的图像,在泛光照明器与结构光投影仪同时照明下,所获取的每幅图像均含有目标泛光图像信息与结构光图像信息,对于图5所示的采集相机而言,则每幅图像中泛光图像信息与结构光图像信息各占一半,采集相机随后会分别提取各自图像对应的像素,并根据上采样算法填充其他空白像素,最终可以得到1280x960@30fps的泛光图像以及1280x960@30fps的结构光图像,并且泛光图像与结构光图像之间没有视差。First, the acquisition camera is controlled to collect the target flood image and the structured light image under the illumination of the flood illuminator 11 and the structured light projector 12; the flood illuminator 11 and the structured light projector 12 can be always turned on. It can also be turned on according to the frequency and pulsed with a certain gap, and its frequency should be consistent with the exposure frequency of the acquisition camera. Assume that the acquisition camera can output an image of 1280x960@30fps. Under the illumination of the floodlight illuminator and the structured light projector, each image acquired contains the target flood image information and the structured light image information, as shown in FIG. In the case of the acquisition camera, the floodlight image information and the structured light image information are each half of each image, and the acquisition camera then separately extracts the pixels corresponding to the respective images, and fills other blank pixels according to the upsampling algorithm, and finally obtains 1280x960@ A 30fps flood image and a structured light image of 1280x960@30fps, and there is no parallax between the flood image and the structured light image.
其次,识别目标泛光图像中的前景目标。在这一步骤中,需要将泛光图像中的前景与背景进行分割,诸多图像分割算法均可以应用到本步骤中,如阈值分割法、均值法(mean shift)、聚类法等等。在选取图像分割算法时,需要兼顾计算效率与计算精度,特别是计算效率,图像分割速度慢会降低最终的图像输出帧率(深度图像输出帧率)。在前景区域被分割后,识别出前景区域,或者说识别出前景区域所在的前景像素区域。Second, identify foreground targets in the target flood image. In this step, the foreground and background in the floodlight image need to be segmented, and many image segmentation algorithms can be applied to this step, such as threshold segmentation, mean shift, clustering, and the like. When selecting the image segmentation algorithm, it is necessary to balance the calculation efficiency and the calculation accuracy, especially the calculation efficiency. The slow image segmentation speed will reduce the final image output frame rate (depth image output frame rate). After the foreground region is segmented, the foreground region is identified, or the foreground pixel region where the foreground region is located is identified.
最后,提取结构光图像中与前景目标对应的像素区域以得到目标结构光图像。由于目标泛光图像与结构光图像之间没有视差,因此上一步骤上识别到的目标泛光图像中的前景区域同样也是结构光图像中的前景区域,提取出这一区域中的结构光图像像素即为目标结构光图像。Finally, a pixel region corresponding to the foreground object in the structured light image is extracted to obtain a target structured light image. Since there is no parallax between the target flood image and the structured light image, the foreground region in the target flood image recognized in the previous step is also the foreground region in the structured light image, and the structured light image in this region is extracted. The pixel is the target structured light image.
在以上的各个步骤后,处理器10将获取当前应用所需要的目标结构光图像,虽然该结构光图像仅包括较小的视场角,然而却具有较高的分辨率。After each of the above steps, the processor 10 will acquire the target structured light image required by the current application, although the structured light image includes only a small field of view, yet has a higher resolution.
如图8所示,在本发明的一种变通实施例中,可以基于该结构光图像进行深度计算得到目标深度图像,由于数据量相对于全分辨率较小,因此深度算法的运算速度也会较快,从而可以确保深度图像的高帧率输出。As shown in FIG. 8, in an alternative embodiment of the present invention, a depth image can be obtained based on the structured light image to obtain a target depth image. Since the amount of data is small relative to the full resolution, the operation speed of the depth algorithm is also Faster, thus ensuring high frame rate output of depth images.
图7、图8所示的方法与图3、图4所示的方法相比,图3、图4方法中目标泛光图像与结构光图像之间存在时间上的差距,当目标为运动物体且运动速度较快时,可能会导致算法失效;图7、图8方法目标泛光图像与结构光图像由于是同步获取,因此可以适应快速运动的物体,但由于采集相机所采集到的结构光图像仅包含部分像素,因此所得到的深度图像的细节信息会有所丢失。7 and FIG. 8 Compared with the methods shown in FIG. 3 and FIG. 4, there is a time difference between the target flood image and the structured light image in the methods of FIGS. 3 and 4, when the target is a moving object. When the motion speed is fast, the algorithm may be invalidated. The target floodlight image and the structured light image of Figure 7 and Figure 8 are synchronously acquired, so they can adapt to fast moving objects, but the structured light collected by the acquisition camera The image contains only a portion of the pixels, so the details of the resulting depth image are lost.
图9是根据本发明又一个实施例的目标图像获取系统示意图。目标图像获取 系统包括处理器10以及与之连接的第一采集相机71、第二采集相机73以及结构光投影仪72以及泛光照明单元,因为在本系统中泛光照明单元采用的是环境光,所以图中未画出。第一采集相机71与第二采集相机73分别用于采集不同波长的图像。在一种实施例中,第一采集相机用于采集目标区域的第一波长的目标泛光图像;第二采集相机用于采集目标区域的所述结构光图像。可以理解的是,在一个实施例中,也可以包含泛光照明器,或者与目标图像获取系统独立的照明光源。以下以环境光为泛光为例进行说明。9 is a schematic diagram of a target image acquisition system in accordance with still another embodiment of the present invention. The target image acquisition system includes a processor 10 and a first acquisition camera 71, a second acquisition camera 73, and a structured light projector 72 and a floodlighting unit connected thereto, because the floodlighting unit uses ambient light in the system. So it is not shown in the picture. The first acquisition camera 71 and the second acquisition camera 73 are respectively used to acquire images of different wavelengths. In one embodiment, the first acquisition camera is configured to acquire a target floodlight image of the first wavelength of the target region; the second acquisition camera is configured to acquire the structured light image of the target region. It will be appreciated that in one embodiment, a floodlight illuminator or an illumination source independent of the target image acquisition system may also be included. The following is an example in which ambient light is used as a floodlight.
在一个实施例中,第一采集相机为RGB相机,用于采集RGB图像;第二采集相机为红外相机,用于采集IR图像;结构光投影仪用于发射红外结构光图像。由于RGB相机与红外相机之间存在视差,因此需要对两个相机进行标定,可以利用已有技术的任一标定方法进行标定,标定的目的是获取其中一个相机相对于另一相机的相互位置关系(平移与放置矩阵,R和T)。图10是根据本发明另一个实施例的目标图像获取方法示意图。该方法被处理器10执行以实现相应的功能。In one embodiment, the first acquisition camera is an RGB camera for acquiring RGB images; the second acquisition camera is an infrared camera for acquiring IR images; and the structured light projector is for emitting infrared structured light images. Since there is parallax between the RGB camera and the infrared camera, it is necessary to calibrate the two cameras, which can be calibrated by any calibration method of the prior art. The purpose of the calibration is to obtain the mutual positional relationship between one camera and the other camera. (translation and placement matrix, R and T). FIG. 10 is a schematic diagram of a target image acquisition method according to another embodiment of the present invention. The method is performed by processor 10 to implement the corresponding functions.
首先,控制RGB相机以及红外相机获取RGB图像与红外结构光图像。处理器10控制RGB相机与红外相机以相同的帧率提取RGB图像与红外结构光图像,RGB图像与红外图像的分辨率可以相同也可以不同,一般地,系统中的RGB相机需要用来执行拍照等任务,因此RGB图像拥有更高的分辨率,但在本实施例中,其采集的RGB图像是用于为前景目标识别应用的,因此可以在低分辨率模式下采集RGB图像,这样一方面可以提高图像获取的帧率,同时可以降低后续前景目标识别的难度。First, the RGB camera and the infrared camera are controlled to acquire RGB images and infrared structured light images. The processor 10 controls the RGB camera and the infrared camera to extract RGB images and infrared structured light images at the same frame rate. The resolutions of the RGB images and the infrared images may be the same or different. Generally, the RGB cameras in the system are required to perform photographing. Such tasks, so RGB images have higher resolution, but in this embodiment, the acquired RGB images are used for foreground target recognition, so RGB images can be acquired in low resolution mode. It can improve the frame rate of image acquisition and reduce the difficulty of subsequent foreground target recognition.
其次,识别RGB图像中的前景目标;在这一步骤中,需要将RGB图像中的前景与背景进行分割,诸多图像分割算法均可以应用到本步骤中,如阈值分割法、均值法(mean shift)、聚类法等等。在选取图像分割算法时,需要兼顾计算效率与计算精度,特别是计算效率,图像分割速度慢会降低最终的图像输出帧率(深度图像输出帧率)。在前景区域被分割后,识别出前景区域,或者说识别出前景区域所在的前景像素区域。Secondly, the foreground object in the RGB image is identified; in this step, the foreground and the background in the RGB image need to be segmented, and many image segmentation algorithms can be applied to this step, such as threshold segmentation and mean shift (mean shift). ), clustering, and so on. When selecting the image segmentation algorithm, it is necessary to balance the calculation efficiency and the calculation accuracy, especially the calculation efficiency. The slow image segmentation speed will reduce the final image output frame rate (depth image output frame rate). After the foreground region is segmented, the foreground region is identified, or the foreground pixel region where the foreground region is located is identified.
最后,基于RGB相机与红外相机的相对位置关系,提取红外结构光图像上与前景目标对应的像素上的目标结构光图像。在RGB图像中确认出前景目标所 在的区域后,根据RGB相机与红外相机的相对位置关系,可以定位出目标结构光图像中相应的前景目标所在的区域,进一步可以提取这一区域的像素作为目标结构光图像。Finally, based on the relative positional relationship between the RGB camera and the infrared camera, the target structured light image on the pixel corresponding to the foreground target on the infrared structured light image is extracted. After confirming the region where the foreground target is located in the RGB image, according to the relative positional relationship between the RGB camera and the infrared camera, the region of the corresponding foreground target in the target structured light image can be located, and the pixel of the region can be further extracted as a target. Structured light image.
如图11所示,在以上的各个步骤后,处理器10将获取当前应用所需要的目标结构光图像,随后利用深度算法计算出该目标结构光图像中各个像素的深度值以生成目标深度图像。在本实施例中,同样由于最终的目标结构光图像总体像素数较小,使得深度计算可以实时运算,从而达到高帧率的输出。在采集RGB图像与红外图像的步骤中,RGB图像与红外图像也可以不同步获取,可以采取与图3、图4所示实施例中类似的形式以一定的时序分开采集RGB图像与红外图像,此时可以降低对处理器10的存储及运算能力的要求。在时序获取模式下,红外相机可以基于RGB图像中识别出的前景目标区域,利用裁剪模式采集红外结构光图像,由此可以进一步降低数据量以保证高速输出。As shown in FIG. 11, after the above various steps, the processor 10 will acquire the target structured light image required by the current application, and then calculate the depth value of each pixel in the target structured light image by using a depth algorithm to generate a target depth image. . In the present embodiment, also because the final target structure light image has a small total number of pixels, the depth calculation can be performed in real time, thereby achieving high frame rate output. In the step of collecting the RGB image and the infrared image, the RGB image and the infrared image may also be acquired asynchronously, and the RGB image and the infrared image may be separately collected at a certain timing according to a form similar to the embodiment shown in FIG. 3 and FIG. At this time, the requirements for the storage and computing power of the processor 10 can be reduced. In the time series acquisition mode, the infrared camera can acquire the infrared structured light image by using the cropping mode based on the foreground target area identified in the RGB image, thereby further reducing the amount of data to ensure high-speed output.
以上内容是结合具体的优选实施方式对本发明所作的进一步详细说明,不能认定本发明的具体实施只局限于这些说明。对于本发明所属技术领域的技术人员来说,在不脱离本发明构思的前提下,还可以做出若干等同替代或明显变型,而且性能或用途相同,都应当视为属于本发明的保护范围。The above is a further detailed description of the present invention in connection with the specific preferred embodiments, and the specific embodiments of the present invention are not limited to the description. It will be apparent to those skilled in the art that <RTIgt; </ RTI> <RTIgt; </ RTI> <RTIgt; </ RTI> <RTIgt; </ RTI> <RTIgt;

Claims (10)

  1. 一种目标图像获取系统,其特征在于,包括:A target image acquisition system, comprising:
    采集相机,用于采集目标区域的图像;a collection camera for acquiring an image of a target area;
    泛光照明单元,用于对所述目标区域提供泛光照明;a floodlighting unit for providing floodlighting to the target area;
    结构光投影仪,用于向所述目标区域投射结构光图像;a structured light projector for projecting a structured light image to the target area;
    处理器,与所述采集相机、所述泛光照明单元以及所述结构光投影仪连接,用于:a processor coupled to the acquisition camera, the floodlighting unit, and the structured light projector for:
    控制所述采集相机采集在所述泛光照明单元照明下的目标泛光图像;Controlling the acquisition camera to acquire a target floodlight image under illumination of the floodlighting unit;
    识别所述目标泛光图像中的前景目标;Identifying a foreground target in the target flood image;
    控制所述采集相机采集在所述结构光投影仪投影下与所述前景目标对应的像素上的目标结构光图像。The acquisition camera is controlled to acquire a target structured light image on a pixel corresponding to the foreground object under projection of the structured light projector.
  2. 如权利要求1所述的目标图像获取系统,其特征在于,所述泛光照明单元是泛光照明器;或,与所述目标图像获取系统独立的照明光源。The target image acquisition system according to claim 1, wherein the floodlighting unit is a floodlight illuminator; or an illumination source that is independent of the target image acquisition system.
  3. 如权利要求1所述的目标图像获取系统,其特征在于,所述目标泛光图像是所述采集相机在低分辨率模式下采集的;所述目标结构光图像是所述采集相机在裁剪模式下采集的。The target image acquisition system according to claim 1, wherein said target floodlight image is acquired by said acquisition camera in a low resolution mode; said target structured light image is said acquisition camera in a cropping mode Collected under.
  4. 如权利要求1所述的目标图像获取系统,其特征在于,所述泛光照明单元和结构光投影仪交叉激活;所述采集相机在激活区间内进行曝光并采集目标泛光图像或结构光图像。The target image acquisition system according to claim 1, wherein said flood illumination unit and said structured light projector are cross-activated; said acquisition camera performs exposure in an activation interval and acquires a target flood image or a structured light image. .
  5. 如权利要求1所述的目标图像获取系统,其特征在于,所述结构光投影单元的激活时间比所述泛光照明单元的激活时间长。The target image acquisition system according to claim 1, wherein an activation time of said structured light projection unit is longer than an activation time of said flood illumination unit.
  6. 如权利要求1所述的目标图像获取系统,其特征在于,所述处理器还用于利用所述目标结构光图像计算出目标深度图像。The target image acquisition system according to claim 1, wherein said processor is further configured to calculate a target depth image using said target structured light image.
  7. 一种目标图像获取方法,其特征在于,包括:A target image acquisition method, comprising:
    S1:采集目标区域的目标泛光图像;S1: collecting a target flood image of the target area;
    S2:识别所述目标泛光图像中的前景目标;S2: identifying a foreground target in the target flood image;
    S3:采集与所述前景目标对应的像素上的目标结构光图像。S3: Acquire a target structured light image on a pixel corresponding to the foreground target.
  8. 如权利要求7所述的目标图像获取方法,其特征在于,步骤S1中,采集 相机在低分辨率模式下采集所述目标泛光图像;步骤S3中,采集相机在裁剪模式下采集所述目标结构光图像。The target image acquisition method according to claim 7, wherein in step S1, the acquisition camera acquires the target flood image in a low resolution mode; in step S3, the acquisition camera acquires the target in a crop mode Structured light image.
  9. 如权利要求7所述的目标图像获取方法,其特征在于,采集所述目标泛光图像与采集所述目标结构光图像是交叉进行的。The target image acquisition method according to claim 7, wherein the capturing of the target floodlight image is performed by intersecting the acquisition of the target structured light image.
  10. 如权利要求7所述的目标图像获取方法,其特征在于,还包括如下步骤:The target image acquisition method according to claim 7, further comprising the steps of:
    S4:利用所述目标结构光图像计算出目标深度图像。S4: Calculate the target depth image by using the target structured light image.
PCT/CN2018/099301 2018-03-31 2018-08-08 Target image acquisition system and method WO2019184183A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810278543.8 2018-03-31
CN201810278543.8A CN108540717A (en) 2018-03-31 2018-03-31 Target image obtains System and method for

Publications (1)

Publication Number Publication Date
WO2019184183A1 true WO2019184183A1 (en) 2019-10-03

Family

ID=63481962

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/099301 WO2019184183A1 (en) 2018-03-31 2018-08-08 Target image acquisition system and method

Country Status (2)

Country Link
CN (1) CN108540717A (en)
WO (1) WO2019184183A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115190288A (en) * 2022-06-23 2022-10-14 海信电子科技(深圳)有限公司 Method and device for synchronously acquiring images by multiple cameras

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020237492A1 (en) * 2019-05-28 2020-12-03 深圳市汇顶科技股份有限公司 Three-dimensional reconstruction method, device, apparatus, and storage medium
CN112068144B (en) * 2019-06-11 2022-10-21 深圳市光鉴科技有限公司 Light projection system and 3D imaging device
CN112066907B (en) * 2019-06-11 2022-12-23 深圳市光鉴科技有限公司 Depth imaging device
CN111063079B (en) * 2019-11-27 2022-04-15 深圳云天励飞技术有限公司 Binocular living body face detection method and device based on access control system
CN111263045B (en) * 2020-02-26 2022-05-24 奥比中光科技集团股份有限公司 Electronic device
CN114079709B (en) * 2020-08-07 2024-04-12 安霸国际有限合伙企业 Driver mechanism for rolling shutter sensor to acquire structured light patterns
CN117666136A (en) * 2022-09-07 2024-03-08 北京字跳网络技术有限公司 Head-mounted virtual reality device

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140267602A1 (en) * 2013-03-14 2014-09-18 CSR Technology, Inc. System and method for real time 2d to 3d conversion of a video in a digital camera
CN105049829A (en) * 2015-07-10 2015-11-11 北京唯创视界科技有限公司 Optical filter, image sensor, imaging device and three-dimensional imaging system
CN107424187A (en) * 2017-04-17 2017-12-01 深圳奥比中光科技有限公司 Depth calculation processor, data processing method and 3D rendering equipment
CN107623814A (en) * 2017-08-09 2018-01-23 广东欧珀移动通信有限公司 The sensitive information screen method and device of shooting image

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101475464B1 (en) * 2008-05-09 2014-12-22 삼성전자 주식회사 Multi-layer image sensor
CN102647552A (en) * 2011-02-22 2012-08-22 华为终端有限公司 Image-taking control method and device
CN102999152B (en) * 2011-09-09 2016-06-29 康佳集团股份有限公司 A kind of gesture motion recognition methods and system
CN103209291A (en) * 2012-01-17 2013-07-17 百度在线网络技术(北京)有限公司 Method, apparatus and device for controlling automatic image shooting
US8879827B2 (en) * 2012-06-29 2014-11-04 Intel Corporation Analyzing structured light patterns
CN104714840B (en) * 2013-12-13 2019-01-15 联想(北京)有限公司 A kind of information processing method and electronic equipment
JP6448330B2 (en) * 2014-11-28 2019-01-09 キヤノン株式会社 Imaging apparatus, control method thereof, and program

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140267602A1 (en) * 2013-03-14 2014-09-18 CSR Technology, Inc. System and method for real time 2d to 3d conversion of a video in a digital camera
CN105049829A (en) * 2015-07-10 2015-11-11 北京唯创视界科技有限公司 Optical filter, image sensor, imaging device and three-dimensional imaging system
CN107424187A (en) * 2017-04-17 2017-12-01 深圳奥比中光科技有限公司 Depth calculation processor, data processing method and 3D rendering equipment
CN107623814A (en) * 2017-08-09 2018-01-23 广东欧珀移动通信有限公司 The sensitive information screen method and device of shooting image

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115190288A (en) * 2022-06-23 2022-10-14 海信电子科技(深圳)有限公司 Method and device for synchronously acquiring images by multiple cameras
CN115190288B (en) * 2022-06-23 2023-04-25 海信电子科技(深圳)有限公司 Method and equipment for synchronously acquiring images by multiple cameras

Also Published As

Publication number Publication date
CN108540717A (en) 2018-09-14

Similar Documents

Publication Publication Date Title
WO2019184183A1 (en) Target image acquisition system and method
WO2019184184A1 (en) Target image acquisition system and method
WO2019184185A1 (en) Target image acquisition system and method
TWI668997B (en) Image device for generating panorama depth images and related image device
CN108307675B (en) Multi-baseline camera array system architecture for depth enhancement in VR/AR applications
WO2018161466A1 (en) Depth image acquisition system and method
US9846968B2 (en) Holographic bird&#39;s eye view camera
US8786679B2 (en) Imaging device, 3D modeling data creation method, and computer-readable recording medium storing programs
CN108718373B (en) Image device
US20140104394A1 (en) System and method for combining data from multiple depth cameras
CN110572630B (en) Three-dimensional image shooting system, method, device, equipment and storage medium
CN112189147B (en) Time-of-flight (TOF) camera and TOF method
US10359277B2 (en) Imaging system with synchronized dynamic control of directable beam light source and reconfigurably masked photo-sensor
CN108399596B (en) Depth image engine and depth image calculation method
WO2018028152A1 (en) Image acquisition device and virtual reality device
US11641524B2 (en) Electronic device and method for displaying image in electronic device
CN110213491B (en) Focusing method, device and storage medium
JPH11175762A (en) Light environment measuring instrument and device and method for shading virtual image using same
JPWO2020183710A1 (en) Image processing equipment and 3D measurement system
US20190302598A1 (en) Projection device, projection method, and projection control program
JPH1023465A (en) Image pickup method and its device
CN112750157B (en) Depth image generation method and device
WO2020084894A1 (en) Multi-camera system, control value calculation method and control device
Popovic et al. State-of-the-art multi-camera systems
TWI837854B (en) Depth processing system and operational method thereof

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18912510

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 18912510

Country of ref document: EP

Kind code of ref document: A1