CN108540717A - Target image obtains System and method for - Google Patents
Target image obtains System and method for Download PDFInfo
- Publication number
- CN108540717A CN108540717A CN201810278543.8A CN201810278543A CN108540717A CN 108540717 A CN108540717 A CN 108540717A CN 201810278543 A CN201810278543 A CN 201810278543A CN 108540717 A CN108540717 A CN 108540717A
- Authority
- CN
- China
- Prior art keywords
- image
- target
- acquisition
- camera
- floodlight
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/24—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures
- G01B11/25—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object
- G01B11/2513—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object with several lines being projected in more than one direction, e.g. grids, patterns
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/521—Depth or shape recovery from laser ranging, e.g. using interferometry; from the projection of structured light
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/10—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/56—Cameras or camera modules comprising electronic image sensors; Control thereof provided with illuminating means
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
- H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
- H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
- H04N25/131—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements including elements passing infrared wavelengths
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
- H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
- H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
- H04N25/133—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements including elements passing panchromatic light, e.g. filters passing white light
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
- H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
- H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
- H04N25/135—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements based on four or more different wavelength filter elements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/2224—Studio circuitry; Studio devices; Studio equipment related to virtual studio applications
- H04N5/2226—Determination of depth image, e.g. for foreground/background separation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/04—Indexing scheme for image data processing or generation, in general involving 3D image data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10048—Infrared image
Abstract
The present invention provides a kind of target image acquisition System and method for, the system comprises：Acquire camera, the image for acquiring target area；Floodlighting units, for providing floodlighting to the target area；Structured light projection instrument is used for the target area projective structure light image；Processor connect with the acquisition camera, the floodlighting units and the structured light projection instrument, is used for：Control target floodlight image of the acquisition camera acquisition under floodlighting units illumination；Identify the foreground target in the target floodlight image；Control object construction light image of the acquisition camera acquisition under structured light projection instrument projection in pixel corresponding with the foreground target.Object construction light image is become to structure light image processing before depth image acquisition, then depth calculation is carried out to object construction light image and obtains target depth image, it is ensured that the high frame per second of depth image exports.
Description
Technical field
The present invention relates to the acquisitions of target image more particularly to a kind of target image to obtain System and method for.
Background technology
The appearance of consumer level depth camera brings change, such as 3D modeling, gesture interaction, recognition of face to various fields
Deng.Different application scenarios are also different to the performance requirement of depth camera, for example 3D modeling and recognition of face etc. often need
Depth camera is wanted to export high-resolution depth image to improve modeling and face recognition algorithms precision；And gesture interaction is then right
The output frame rate of depth camera has higher requirement, the depth image of high frame per second that can reduce delay, bring better user experience.
Currently, one of consumer level depth camera problem encountered is between depth image resolution ratio and output frame rate
Contradiction, especially for the depth camera based on structured light technique, when its depth image resolution ratio is higher, due to depth calculation
The increase of amount causes its output frame rate to decline to a great extent, and high-resolution can not be realized simultaneously with high frame per second.
Invention content
The present invention provides a kind of mesh to solve the problems, such as that prior art middle high-resolution can not be realized simultaneously with high frame per second
Logo image obtains System and method for.
To solve the above-mentioned problems, the technical solution adopted by the present invention is as described below：
A kind of target image acquisition system, including：Acquire camera, the image for acquiring target area；Floodlighting list
Member, for providing floodlighting to the target area；Structured light projection instrument is used for the target area projective structure light figure
Picture；Processor connect with the acquisition camera, the floodlighting units and the structured light projection instrument, is used for：Control institute
State target floodlight image of the acquisition camera acquisition under floodlighting units illumination；It identifies in the target floodlight image
Foreground target；Control acquisition camera acquisition pixel corresponding with the foreground target under structured light projection instrument projection
On object construction light image.
The present invention also provides a kind of target images to obtain system, and the floodlighting units are washers；Or, with institute
It states target image and obtains the independent lighting source of system；The floodlighting units and structured light projection instrument cross activation；It is described
Acquisition camera is exposed in activation interval and acquires target floodlight image or structure light image；The structured light projection unit
Activationary time it is longer than the activationary time of the floodlighting units.
The present invention provides a kind of target image acquisition system again, and the target floodlight image is the acquisition camera at low point
It is acquired under resolution pattern；The object construction light image is that the acquisition camera acquires under cutting pattern；The processing
Device is additionally operable to calculate target depth image using the object construction light image.
The present invention provides a kind of target image acquisition methods, including：S1：Acquire the target floodlight image of target area；S2：
Identify the foreground target in the target floodlight image；S3：Object construction in acquisition pixel corresponding with the foreground target
Light image.In step S1, acquisition camera acquires the target floodlight image under low-resolution mode；In step S3, phase is acquired
Machine acquires the object construction light image under cutting pattern.It acquires the target floodlight image and acquires the object construction light
Image is to intersect to carry out.
The present invention also provides a kind of target image acquisition methods, further include following steps：
S4：Target depth image is calculated using the object construction light image.
Beneficial effects of the present invention are：A kind of target image acquisition system and method are provided, by first obtaining target floodlight
Then image extracts the foreground target in target floodlight image, then acquires the target in pixel corresponding with the foreground target
Structure light image becomes object construction light image before depth image acquisition to structure light image processing, right again at this time
Object construction light image carries out depth calculation and obtains target depth image, since data volume is smaller relative to full resolution,
The arithmetic speed of depth algorithm also can be very fast, so as to ensure the high frame per second output of depth image.
Description of the drawings
Fig. 1 is that target image according to an embodiment of the invention obtains system schematic.
Fig. 2 is washers, structured light projection instrument and the sequential for acquiring camera according to an embodiment of the invention
Control schematic diagram.
Fig. 3 is target image acquisition methods schematic diagram according to an embodiment of the invention.
Fig. 4 is the target image acquisition methods schematic diagram according to second embodiment of the invention
Fig. 5 is acquisition camera image acquisition principle schematic diagram according to an embodiment of the invention.
Fig. 6 is the acquisition camera image acquisition principle schematic diagram according to another embodiment of the invention.
Fig. 7 is the target image acquisition methods schematic diagram according to third embodiment of the invention.
Fig. 8 is the target image acquisition methods schematic diagram according to four embodiment of the invention.
Fig. 9 is to obtain system schematic according to the target image of another embodiment of the invention.
Figure 10 is the target image acquisition methods schematic diagram according to fifth embodiment of the invention.
Figure 11 is the target image acquisition methods schematic diagram according to sixth embodiment of the invention.
Wherein, 10- processors, 11- washers, 12- structured light projection instrument, 13- acquire camera, and 71- first is acquired
Camera, 72- structured light projection instrument, 73- second acquire camera.
Specific implementation mode
The present invention is described in detail by specific embodiment below in conjunction with the accompanying drawings, for a better understanding of this hair
It is bright, but following embodiments are not intended to limit the scope of the invention.In addition, it is necessary to illustrate, the diagram provided in following embodiments
Only illustrate the basic conception of the present invention in a schematic way, is only shown in attached drawing with related component in the present invention rather than according to reality
Component count, shape when implementation and size are drawn, when actual implementation each component shape, quantity and ratio can be it is a kind of with
The change of meaning, and its assembly layout form may also be increasingly complex.
Fig. 1 is that target image according to an embodiment of the invention obtains system schematic.Target image obtains system packet
Include processor 10 and the structured light projection instrument 12 that is attached thereto and acquisition camera 13, wherein structured light projection instrument 12 be used for
Projective structure light light beam in space can form corresponding structure light image when structure light light beam is irradiated on object, the pattern
Then collected camera 13 acquires and forms the structure light image of the object, and processor 10 is further counted based on the structure light image
Calculate depth image.
Single structure light projector 12 forms monocular structure light Depth Imaging system, processor 10 with single acquisition camera 13
Depth image will be calculated based on monocular structure light trigonometry principle.In one embodiment, processor 10 will be collected currently
Object structures light image and the reference configuration light image that pre-saves carry out matching primitives, to obtain pixel between two images
Deviation value, depth value is further calculated out according to the deviation value, reference configuration light image mentioned here is in known depth
Apart from one piece of plane of upper placement, after structured light projection instrument 12 goes out structure light light beam to the plane projection by acquisition camera 13 or its
He acquires what camera collected.
In some embodiments, it is also possible to comprising two or more acquisition cameras 13, with 12 groups of structured light projection instrument
At binocular or more mesh structure light Depth Imaging systems.The binocular formed with two acquisition cameras 13 and single structure light projector 12
It is illustrated for structured-light system, after the projective structure light light beam into space of structured light projection instrument 12, two acquisition cameras
The 13 left and right two width structure light images of acquisition, processor 10 pass through the matching to left and right structure light image based on binocular vision algorithm
Calculating can also obtain depth image；It can also be respectively to left and right structure light image and corresponding reference configuration light image
It is calculated to obtain two amplitude deepness images, this have the advantage that, it in one embodiment can be by left and right acquisition module
It is arranged to the parameter for having different, such as different resolution ratio, focal length etc., it is possible thereby to which acquisition simultaneously has such as different differentiate
The structure light image of rate, field angle etc., it is possible to further obtain the depth image of different resolution, field angle etc. simultaneously；
In one embodiment, multiple depth images of acquisition can also be fused into the depth image that a width has more information.
In some embodiments, matching primitives refer in current structure light image (or reference configuration light image) with certain
A certain size subregion, such as the subregion of 7x7,11x11 pixel size are chosen centered on pixel, then in reference configuration light
Search and the most similar subregion of subregion, two sub-regions picture in two images on image (or current structure light image)
Difference between plain coordinate is deviation value；Secondly using the correspondence between deviation value and depth value, just based on deviation value
Depth value can be calculated, the depth value of multiple pixels just constitutes depth image.For left and right two width or several structure light figures
Matching primitives principle as between is similar with above-mentioned principle.
In some embodiments, it further includes the washers 11 being connect with processor 10 that target image, which obtains system,
Washers 11 are used for providing floodlighting as floodlighting units.Processor 10 controls floodlight by modes such as buses and shines
Funerary objects 11, structured light projection instrument 12 and acquisition camera 13, can also be attached, such as logical by some data transmission interfaces
It crosses the interfaces such as MIPI, VGA to connect with acquisition camera 13, to receive by 13 the image collected of acquisition camera.In one embodiment
In, washers 11 are used to emit the light beam of phase co-wavelength, such as infrared light with structured light projection instrument 12, and acquire camera 13
By containing the pixel for acquiring the wavelength light beam.Processor 10 can be realized by the control between three in sequential
Acquisition to different images can specifically control target floodlight figure of the acquisition camera acquisition under floodlighting units illumination
Picture；Identify the foreground target in target floodlight image；Control acquisition camera acquisition structured light projection instrument projection under with foreground mesh
Mark the object construction light image in corresponding pixel.In some embodiments, floodlighting units can also be its in environment
His light source, such as ambient light can be used as floodlighting.I.e. floodlighting can be the active light that the light sources such as infrared light supply are sent out,
It can also be ambient light.Comprising carrying out in the case of washers during some are with system in following specific embodiments
Description, some are the descriptions carried out using ambient light as floodlighting units, it should it is understood that according to different feelings
Condition can select the form of specific floodlighting, but its method is general, is not distinguished specifically below.
Processor 10 can be disposed on the depth calculation processor of internal system to execute, which can special place
Device such as SOC, FPGA etc. are managed, can also be general processor.In some embodiments, it is also possible to using external computing device, such as
The equipment such as computer, mobile terminal, server, external computing device are implemented after receiving the structure light image from acquisition module 13
Depth calculation, obtained depth image can be directly used for the other application of the equipment.In one embodiment, when system is as embedding
When entering formula device and being integrated into other computing terminals, such as computer, tablet, mobile phone, TV target image acquisition device, processor institute
The function of realization can by terminal processor or application to complete, for example depth calculation function deposited in the form of software module
Storage in memory, is called by the processor in terminal to realize depth calculation.It is understood that being provided using the present invention
Target image obtain system and/or all answered using the target image acquisition device of target image acquisition methods provided by the invention
This is considered as the scope of protection of the invention.
Structure light image can be candy strip, two-dimensional pattern, speckle pattern (speckle patterns) etc., and structure optical wavelength can be with
It is visible wavelength, infrared light wavelength, ultraviolet wavelength etc..
Shown in Fig. 2 is washers, structured light projection instrument and the timing control schematic diagram for acquiring camera.When wherein
Sequence Figure 20,21 and 22 correspond to washers 11, structured light projection instrument 12 and acquisition camera 13, the portion of figure protrusions respectively
Divide and indicate that corresponding device is active, for example washers 11 are located at projection state with structured light projection instrument 12, adopt
Collection camera is located at exposure status.From shown in Fig. 2 as can be seen that the present embodiment in, processor 10 control washers 11 and
Structured light projection instrument 12 carries out cross activation, while controlling acquisition camera and being exposed and acquire corresponding in each activation interval
Image.Floodlight image A is collected under the irradiation of washers 11, structure light is collected under the projection of structured light projection instrument 12
Image B, floodlight image A are sequentially output to processor with structure light image B and are handled.In some embodiments, by will be general
The reasonable of the activation duration of light illuminator 11 and structured light projection instrument 12 is arranged to collect higher-quality image, such as will
The activationary time of structured light projection instrument 12 is arranged longer, more high-quality to collect to ensure to meet enough time for exposure
The structure light image of amount；In some embodiments, according to the actual application, washers 11 and structured light projection instrument 12
Activation order can also be arranged to other forms, for example activate activation primary structure light projector 12 after washers 11 twice
Deng.
In some applications, it is desirable that obtain the high-resolution depth graph picture of measured target, however be limited to depth calculation
The limitation of algorithm and processor computing capability realizes that the acquisition of high-resolution depth graph picture generally requires to spend higher cost.
The method that will provide the acquisition high-resolution target depth image based on system shown in Figure 1 in one embodiment of the present invention.Fig. 3
It is target image acquisition methods schematic diagram according to an embodiment of the invention, this method is executed corresponding to realize by processor 10
Function.
First, target floodlight image of control acquisition camera 13 acquisition under floodlighting units illumination；Mesh said herein
It refers to the target floodlight image containing target to mark floodlight image.In prior art, such as Microsoft kinect, Intel
Realsense even depth cameras, output depth image resolution ratio is often VGA, i.e. 640x480, or more low resolution, because
This is illustrated by taking high definition resolution ratio 1280x960 as an example in the present invention, it is to be understood that other resolution ratio are also applied for
The present invention.In this step, processor 10 applies synchronous triggering signal to washers 11 and acquisition camera 13, to use
While washers 11 provide floodlighting, acquisition camera 13 collects the floodlight image A of target area, acquires at this time
Camera 13 can be full resolution output, that is, the floodlight image A for exporting 1280x960 resolution ratio also may be used in one embodiment
The low resolutions such as binning mode (merging patterns) or skipping mode (decimation pattern) are used with control acquisition camera 13
The low-resolution image of rate pattern acquiring full filed.Before output frame rate is more demanding and output interface transmission speed is certain
It puts, if full resolution image cannot achieve the output of high frame per second, low resolution output mode as described above can be taken.
Usually, containing interested foreground target, such as face, human body, object etc. in floodlight image, while can also contain
There is the scene etc. where some target contexts, such as people.For some applications, such as recognition of face, 3D modeling, often only
Foreground target information is needed, and background then needs to remove.
Secondly, the foreground target in target floodlight image is identified.In this step, it needs the foreground in floodlight image
It is split with background, many image segmentation algorithms can be applied in this step, such as thresholding method, averaging method (mean
Shift), clustering procedure etc..When choosing image segmentation algorithm, needs to take into account computational efficiency and computational accuracy, especially calculate
Efficiency, image segmentation speed can reduce final image output frame rate (depth image output frame rate) slowly.Divided in foreground area
After cutting, foreground area is identified, identify the foreground pixel region where foreground area in other words.
Finally, target of the control acquisition camera acquisition under the projection of structured light projection instrument in pixel corresponding with foreground target
Structure light image.Due to obtaining foreground pixel region in previous step, in this step, acquisition camera will be in the pattern of cutting
Under (cropping mode) only pair pixel corresponding with foreground area sample, i.e., only export it is corresponding with foreground area before
Scape image, since structured light projection instrument is in open state at this time, the foreground image obtained is object construction light image.It needs
It should be noted that for dynamic object, such as the human body moved, the pixel between front and back two images corresponding to target also has
Difference, therefore, can be appropriate according to the movement speed of human body and the parameter of camera when choosing pixel corresponding with foreground area
Expand pixel region.In fact, in the case of frame per second larger (such as 30fps, 60fps), foreground area in consecutive frame image
It is almost identical.
After above each step, processor 10 will obtain the required object construction light image of current application, although
The structure light image only includes smaller field angle, however but has higher resolution ratio.
As shown in figure 4, in the alternative embodiments of the present invention, depth calculation is carried out based on the structure light image and obtains target
Depth image, since data volume is smaller relative to full resolution, the arithmetic speed of depth algorithm also can be very fast, so as to
Ensure the high frame per second output of depth image.
Above each step is illustrated with a more intuitive embodiment now, for example acquisition camera highest can export
The image of 1280x960@60fps, if being acquired using the acquisition camera as structure light image, due to depth calculation algorithm and
The limitations such as hardware are only capable of realizing the depth image output of 1280x960@10fps, since depth image frame per second is too low, lead to not
Meet the needs of some applications.And utilize the process described above, i.e., when floodlight image and structured light projection instrument are to intersect
Sequence is opened, and acquisition camera can then obtain the floodlight image of 1280x960@30fps, is calculated in conjunction with the image segmentation of high speed
Method, (assuming that target area is located among acquisition viewing field of camera angle and accounts for entire visual field after recognizing foreground target region
50%) the object construction light image that, thus can then obtain 640x480@30fps, according to current depth computational algorithm and phase
Hardware is closed, can meet and be handled the object construction light image of 640x480@30fps and exported 640x480@30fps in real time
Depth image.Compared with the acquisition camera for directly using 640x480@30fps, the depth image acquired in the present embodiment only wraps
Containing target, detailed information is more abundant, while the step of eliminating image segmentation.
For Fig. 2 with Fig. 3, embodiment shown in Fig. 4, washers 11 project phase co-wavelength with structured light projection instrument 12
Light beam, acquisition camera for obtaining floodlight image A and structure light image B respectively in different sequential.
Fig. 5 with it is shown in fig. 6 be acquisition camera acquisition principle schematic diagram according to some embodiments of the invention.In Fig. 5, adopt
Collection camera can synchronize acquisition to the light beam of two kinds of wavelength, possess the W pixel photosensitive to white light (light of all wavelengths)
And the IR pixel photosensitive to infrared light, when floodlighting units are ambient light, structured light projection instrument is for projecting infrared structure
Light time, acquisition camera can acquire floodlight image and structure light image simultaneously, and only floodlight image and structure light image has
Acquisition camera whole pixel will be less than by imitating pixel.In the present embodiment, valid pixel is the half of whole pixel, in other implementations
In example, floodlight image and structure light image pixel may be other ratios, such as W:IR=1：3, it is possible thereby to ensure structure
Possess more image details in light image, acquired depth image has finer information.In figure 6, camera is acquired
Coloured image (RGB) and infrared image can be acquired simultaneously, for example can acquire colored floodlight image and infrared structure light simultaneously
Image can also be colored structures light image and infrared floodlight image.
It is understood that Fig. 5 and Fig. 6 only illustrate when washers 11 and 12 institute of structured light projection instrument as example
When sending out light beam wavelength difference, it can utilize to two kinds of wavelength acquisition camera photosensitive simultaneously while acquire corresponding target
Floodlight image and structure light image are not limited to the example of Fig. 5 and Fig. 6 in actual use.Fig. 7 is based on acquisition camera while to acquire
The target floodlight image of different wave length and the target image acquisition methods schematic diagram of structure light image, this method are held by processor 10
Row is to realize corresponding function.
First, the acquisition camera acquisition is controlled under washers 11 and the structured light projection instrument 12 illumination
Target floodlight image and structure light image；Washers 11 can be constantly in open state with structured light projection instrument 12,
It can be opened according to frequency and with certain gap pulsed illumination, frequency should be consistent with the acquisition frequency of exposure of camera.It is assumed that
Acquisition camera can export the image of 1280x960@30fps, in the case where washers and structured light projection instrument illuminate simultaneously, institute
The each image of acquisition contains target floodlight image information and structural light image information, for acquisition camera shown in fig. 5
Speech, then floodlight image information respectively accounts for half with structural light image information in each image, and acquisition camera can then extract respectively respectively
Other blank pixels are filled from the corresponding pixel of image, and according to up-sampling algorithm, 1280x960@30fps may finally be obtained
Floodlight image and 1280x960@30fps structure light image, and do not regarded between floodlight image and structure light image
Difference.
Secondly, the foreground target in target floodlight image is identified.In this step, it needs the foreground in floodlight image
It is split with background, many image segmentation algorithms can be applied in this step, such as thresholding method, averaging method (mean
Shift), clustering procedure etc..When choosing image segmentation algorithm, needs to take into account computational efficiency and computational accuracy, especially calculate
Efficiency, image segmentation speed can reduce final image output frame rate (depth image output frame rate) slowly.Divided in foreground area
After cutting, foreground area is identified, identify the foreground pixel region where foreground area in other words.
Finally, extract structure light image in pixel region corresponding with foreground target to obtain object construction light image.By
In the target floodlight image for not having parallax between target floodlight image and structure light image, therefore being recognized in previous step
The foreground area also foreground area in structure light image, the structure light image pixel extracted in this region is mesh
Mark structure light image.
After above each step, processor 10 will obtain the required object construction light image of current application, although
The structure light image only includes smaller field angle, however but has higher resolution ratio.
As shown in figure 8, in a kind of alternative embodiments of the present invention, the structure light image can be based on and carry out depth calculation
Target depth image is obtained, since data volume is smaller relative to full resolution, the arithmetic speed of depth algorithm also can be very fast,
So as to ensure the high frame per second output of depth image.
Fig. 7, method shown in Fig. 8 compared with Fig. 3, method shown in Fig. 4, in Fig. 3, Fig. 4 method target floodlight image with
Gap between structure light image in existence time may result in calculation when target is moving object and very fast movement velocity
Method fails；Fig. 7, Fig. 8 method target floodlight image, due to being synchronous acquisition, are adapted to quickly move with structure light image
Object, but since the collected structure light image of acquisition camera institute only includes partial pixel, obtained depth image
Detailed information have it is lost.
Fig. 9 is to obtain system schematic according to the target image of another embodiment of the invention.Target image obtains system
Including processor 10 and the first acquisition camera 71, second being attached thereto acquire camera 73 and structured light projection instrument 72 and
Floodlighting units are not drawn into because floodlighting units are using ambient light in the present system in figure.First acquisition
Camera 71 and second acquires the image that camera 73 is respectively used to acquisition different wave length.In one embodiment, the first acquisition camera
Target floodlight image for the first wave length for acquiring target area；Second acquisition camera is used to acquire the knot of target area
Structure light image.It is understood that can also include washers, or obtained with target image in one embodiment
The independent lighting source of system.It is illustrated so that ambient light is floodlight as an example below.
In one embodiment, the first acquisition camera is RGB camera, for acquiring RGB image；Second acquisition camera is red
Outer camera, for acquiring IR images；Structured light projection instrument is for emitting infrared structure light image.Due to RGB camera and infrared phase
There are parallaxes between machine, it is therefore desirable to demarcate, can be carried out using any scaling method of prior art to two cameras
Calibration, the purpose of calibration be obtain one of camera relative to another camera mutual alignment relation (translation with place matrix,
R and T).Figure 10 is target image acquisition methods schematic diagram in accordance with another embodiment of the present invention.This method is held by processor 10
Row is to realize corresponding function.
First, it controls RGB camera and infrared camera obtains RGB image and infrared structure light image.Processor 10 controls
RGB camera and infrared camera with identical frame per second extraction RGB image and infrared structure light image, RGB image and infrared image
Resolution ratio can be the same or different, and usually, RGB camera in system needs to be used for executing the tasks such as take pictures, therefore RGB
Image possesses higher resolution ratio, but in the present embodiment, and the RGB image of acquisition is used to foreground target identification application
, therefore RGB image can be acquired under low-resolution mode, the frame per second of image acquisition on the one hand can be improved in this way, simultaneously
The difficulty of follow-up foreground target identification can be reduced.
Secondly, the foreground target in RGB image is identified；In this step, it needs the foreground and background in RGB image
It is split, many image segmentation algorithms can be applied in this step, such as thresholding method, averaging method (mean
Shift), clustering procedure etc..When choosing image segmentation algorithm, needs to take into account computational efficiency and computational accuracy, especially calculate
Efficiency, image segmentation speed can reduce final image output frame rate (depth image output frame rate) slowly.Divided in foreground area
After cutting, foreground area is identified, identify the foreground pixel region where foreground area in other words.
Finally, the relative position relation based on RGB camera and infrared camera, extract infrared structure light image on foreground mesh
Mark the object construction light image in corresponding pixel.After confirming the region where foreground target in RGB image, according to RGB
The relative position relation of camera and infrared camera can orient the area where corresponding foreground target in object construction light image
Domain may further extract the pixel in this region as object construction light image.
As shown in figure 11, after above each step, processor 10 will obtain the required object construction of current application
Light image calculates the depth value of each pixel in the object construction light image to generate target depth followed by depth algorithm
Image.In the present embodiment, also due to final object construction light image overall pixel number is smaller so that depth calculation can be with
Real-time operation, to reach the output of high frame per second.In the step of acquiring RGB image and infrared image, RGB image and infrared figure
As acquisition can not also be synchronized, the form similar with Fig. 3, embodiment illustrated in fig. 4 can be taken separately to be adopted with certain sequential
Collect RGB image and infrared image, the storage to processor 10 and the requirement of operational capability can be reduced at this time.In timing acquisition mould
Under formula, infrared camera can utilize cutting type collection infrared structure light based on the foreground target region identified in RGB image
Image, it is possible thereby to further decrease data volume to ensure speedy carding process.
The above content is a further detailed description of the present invention in conjunction with specific preferred embodiments, and it cannot be said that
The specific implementation of the present invention is confined to these explanations.For those skilled in the art to which the present invention belongs, it is not taking off
Under the premise of from present inventive concept, several equivalent substitute or obvious modifications can also be made, and performance or use is identical, all answered
When being considered as belonging to protection scope of the present invention.
Claims (10)
1. a kind of target image obtains system, which is characterized in that including：
Acquire camera, the image for acquiring target area；
Floodlighting units, for providing floodlighting to the target area；
Structured light projection instrument is used for the target area projective structure light image；
Processor connect with the acquisition camera, the floodlighting units and the structured light projection instrument, is used for：
Control target floodlight image of the acquisition camera acquisition under floodlighting units illumination；
Identify the foreground target in the target floodlight image；
The acquisition camera is controlled to acquire under structured light projection instrument projection in pixel corresponding with the foreground target
Object construction light image.
2. target image as described in claim 1 obtains system, which is characterized in that the floodlighting units are floodlighting
Device；Or, obtaining the independent lighting source of system with the target image.
3. target image as described in claim 1 obtains system, which is characterized in that the target floodlight image is the acquisition
What camera acquired under low-resolution mode；The object construction light image is that the acquisition camera acquires under cutting pattern
's.
4. target image as described in claim 1 obtains system, which is characterized in that the floodlighting units and structure light are thrown
Shadow instrument cross activation；The acquisition camera is exposed in activation interval and acquires target floodlight image or structure light image.
5. target image as described in claim 1 obtains system, which is characterized in that when the activation of the structured light projection unit
Between it is longer than the activationary time of the floodlighting units.
6. target image as described in claim 1 obtains system, which is characterized in that the processor is additionally operable to utilize the mesh
Mark structure light image calculates target depth image.
7. a kind of target image acquisition methods, which is characterized in that including：
S1：Acquire the target floodlight image of target area；
S2：Identify the foreground target in the target floodlight image；
S3：Object construction light image in acquisition pixel corresponding with the foreground target.
8. target image acquisition methods as claimed in claim 7, which is characterized in that in step S1, acquisition camera is in low resolution
The target floodlight image is acquired under rate pattern；In step S3, acquisition camera acquires the object construction light under cutting pattern
Image.
9. target image acquisition methods as claimed in claim 7, which is characterized in that acquire the target floodlight image and acquisition
The object construction light image is to intersect to carry out.
10. target image acquisition methods as claimed in claim 7, which is characterized in that further include following steps：
S4：Target depth image is calculated using the object construction light image.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810278543.8A CN108540717A (en) | 2018-03-31 | 2018-03-31 | Target image obtains System and method for |
| PCT/CN2018/099301 WO2019184183A1 (en) | 2018-03-31 | 2018-08-08 | Target image acquisition system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810278543.8A CN108540717A (en) | 2018-03-31 | 2018-03-31 | Target image obtains System and method for |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108540717A true CN108540717A (en) | 2018-09-14 |
Family
ID=63481962
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810278543.8A Pending CN108540717A (en) | 2018-03-31 | 2018-03-31 | Target image obtains System and method for |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN108540717A (en) |
| WO (1) | WO2019184183A1 (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110337674A (en) * | 2019-05-28 | 2019-10-15 | 深圳市汇顶科技股份有限公司 | Three-dimensional rebuilding method, device, equipment and storage medium |
| CN111063079A (en) * | 2019-11-27 | 2020-04-24 | 深圳云天励飞技术有限公司 | Binocular living body face detection method and device based on access control system |
| CN111263045A (en) * | 2020-02-26 | 2020-06-09 | 深圳奥比中光科技有限公司 | Electronic equipment |
| CN112068144A (en) * | 2019-06-11 | 2020-12-11 | 深圳市光鉴科技有限公司 | Light projection system and 3D imaging device |
| CN112066907A (en) * | 2019-06-11 | 2020-12-11 | 深圳市光鉴科技有限公司 | Depth imaging device |
| CN114079709A (en) * | 2020-08-07 | 2022-02-22 | 安霸国际有限合伙企业 | Driver mechanism for rolling shutter sensor to obtain structured light pattern |
| WO2024051476A1 (en) * | 2022-09-07 | 2024-03-14 | 北京字跳网络技术有限公司 | Head-mounted virtual reality device |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115190288B (en) * | 2022-06-23 | 2023-04-25 | 海信电子科技（深圳）有限公司 | Method and equipment for synchronously acquiring images by multiple cameras |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101577287A (en) * | 2008-05-09 | 2009-11-11 | 三星电子株式会社 | Multilayer image sensor |
| CN102647552A (en) * | 2011-02-22 | 2012-08-22 | 华为终端有限公司 | Image-taking control method and device |
| CN102999152A (en) * | 2011-09-09 | 2013-03-27 | 康佳集团股份有限公司 | Method and system for gesture recognition |
| CN103209291A (en) * | 2012-01-17 | 2013-07-17 | 百度在线网络技术（北京）有限公司 | Method, apparatus and device for controlling automatic image shooting |
| US20140003722A1 (en) * | 2012-06-29 | 2014-01-02 | Ziv Aviv | Analyzing structured light patterns |
| CN104714840A (en) * | 2013-12-13 | 2015-06-17 | 联想(北京)有限公司 | Information processing method and electronic equipment |
| CN105657222A (en) * | 2014-11-28 | 2016-06-08 | 佳能株式会社 | Image pickup apparatus and control method for the image pickup apparatus |
| CN107424187A (en) * | 2017-04-17 | 2017-12-01 | 深圳奥比中光科技有限公司 | Depth calculation processor, data processing method and 3D rendering equipment |
| CN107623814A (en) * | 2017-08-09 | 2018-01-23 | 广东欧珀移动通信有限公司 | The sensitive information screen method and device of shooting image |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10237528B2 (en) * | 2013-03-14 | 2019-03-19 | Qualcomm Incorporated | System and method for real time 2D to 3D conversion of a video in a digital camera |
| CN105049829B (en) * | 2015-07-10 | 2018-12-25 | 上海图漾信息科技有限公司 | Optical filter, imaging sensor, imaging device and 3-D imaging system |
-
2018
- 2018-03-31 CN CN201810278543.8A patent/CN108540717A/en active Pending
- 2018-08-08 WO PCT/CN2018/099301 patent/WO2019184183A1/en active Application Filing
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101577287A (en) * | 2008-05-09 | 2009-11-11 | 三星电子株式会社 | Multilayer image sensor |
| CN102647552A (en) * | 2011-02-22 | 2012-08-22 | 华为终端有限公司 | Image-taking control method and device |
| CN102999152A (en) * | 2011-09-09 | 2013-03-27 | 康佳集团股份有限公司 | Method and system for gesture recognition |
| CN103209291A (en) * | 2012-01-17 | 2013-07-17 | 百度在线网络技术（北京）有限公司 | Method, apparatus and device for controlling automatic image shooting |
| US20140003722A1 (en) * | 2012-06-29 | 2014-01-02 | Ziv Aviv | Analyzing structured light patterns |
| CN104714840A (en) * | 2013-12-13 | 2015-06-17 | 联想(北京)有限公司 | Information processing method and electronic equipment |
| CN105657222A (en) * | 2014-11-28 | 2016-06-08 | 佳能株式会社 | Image pickup apparatus and control method for the image pickup apparatus |
| CN107424187A (en) * | 2017-04-17 | 2017-12-01 | 深圳奥比中光科技有限公司 | Depth calculation processor, data processing method and 3D rendering equipment |
| CN107623814A (en) * | 2017-08-09 | 2018-01-23 | 广东欧珀移动通信有限公司 | The sensitive information screen method and device of shooting image |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110337674A (en) * | 2019-05-28 | 2019-10-15 | 深圳市汇顶科技股份有限公司 | Three-dimensional rebuilding method, device, equipment and storage medium |
| CN112068144A (en) * | 2019-06-11 | 2020-12-11 | 深圳市光鉴科技有限公司 | Light projection system and 3D imaging device |
| CN112066907A (en) * | 2019-06-11 | 2020-12-11 | 深圳市光鉴科技有限公司 | Depth imaging device |
| CN112066907B (en) * | 2019-06-11 | 2022-12-23 | 深圳市光鉴科技有限公司 | Depth imaging device |
| CN111063079A (en) * | 2019-11-27 | 2020-04-24 | 深圳云天励飞技术有限公司 | Binocular living body face detection method and device based on access control system |
| CN111263045A (en) * | 2020-02-26 | 2020-06-09 | 深圳奥比中光科技有限公司 | Electronic equipment |
| CN111263045B (en) * | 2020-02-26 | 2022-05-24 | 奥比中光科技集团股份有限公司 | Electronic device |
| CN114079709A (en) * | 2020-08-07 | 2022-02-22 | 安霸国际有限合伙企业 | Driver mechanism for rolling shutter sensor to obtain structured light pattern |
| CN114079709B (en) * | 2020-08-07 | 2024-04-12 | 安霸国际有限合伙企业 | Driver mechanism for rolling shutter sensor to acquire structured light patterns |
| WO2024051476A1 (en) * | 2022-09-07 | 2024-03-14 | 北京字跳网络技术有限公司 | Head-mounted virtual reality device |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2019184183A1 (en) | 2019-10-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108540717A (en) | Target image obtains System and method for | |
| CN108648225A (en) | Target image obtains System and method for | |
| CN108683902A (en) | Target image obtains System and method for | |
| US11354840B2 (en) | Three dimensional acquisition and rendering | |
| US11100664B2 (en) | Depth-aware photo editing | |
| US11003897B2 (en) | Three-dimensional real face modeling method and three-dimensional real face camera system | |
| US20140104394A1 (en) | System and method for combining data from multiple depth cameras | |
| Shen et al. | Virtual mirror rendering with stationary rgb-d cameras and stored 3-d background | |
| Matsuyama et al. | 3D video and its applications | |
| EP2824923B1 (en) | Apparatus, system and method for projecting images onto predefined portions of objects | |
| CN110378838B (en) | Variable-view-angle image generation method and device, storage medium and electronic equipment | |
| EP4156681A1 (en) | Camera system, mobile terminal, and three-dimensional image acquisition method | |
| Starck et al. | The multiple-camera 3-d production studio | |
| Naemura et al. | Virtual shadows in mixed reality environment using flashlight-like devices | |
| TW201837861A (en) | Depth processing system | |
| EP3832601A1 (en) | Image processing device and three-dimensional measuring system | |
| US20190132574A1 (en) | Depth sensing system with differential imaging camera | |
| JPH11257932A (en) | 3-d-stereo-real time sensor system | |
| Kumar et al. | Projection-mapping-based object pointing using a high-frame-rate camera-projector system | |
| Kim et al. | 3-d virtual studio for natural inter-“acting” | |
| CN109389631B (en) | Depth calculation system and method | |
| Ogi et al. | Usage of video avatar technology for immersive communication | |
| Kovačević et al. | An improved CamShift algorithm using stereo vision for object tracking | |
| Oishi et al. | An instant see-through vision system using a wide field-of-view camera and a 3d-lidar | |
| CN103729060B (en) | Multi-environment virtual projection interactive system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 11-13 / F, joint headquarters building, high tech Zone, 63 Xuefu Road, Yuehai street, Nanshan District, Shenzhen, Guangdong 518000 Applicant after: Obi Zhongguang Technology Group Co., Ltd Address before: 12 / F, joint headquarters building, high tech Zone, 63 Xuefu Road, Nanshan District, Shenzhen, Guangdong 518000 Applicant before: SHENZHEN ORBBEC Co.,Ltd. |
|
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180914 |