CN108683902B - Target image acquisition system and method - Google Patents
Target image acquisition system and method Download PDFInfo
- Publication number
- CN108683902B CN108683902B CN201810278510.3A CN201810278510A CN108683902B CN 108683902 B CN108683902 B CN 108683902B CN 201810278510 A CN201810278510 A CN 201810278510A CN 108683902 B CN108683902 B CN 108683902B
- Authority
- CN
- China
- Prior art keywords
- image
- target
- structured light
- wavelength
- flood
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/271—Image signal generators wherein the generated image signals comprise depth maps or disparity maps
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/521—Depth or shape recovery from laser ranging, e.g. using interferometry; from the projection of structured light
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
- H04N13/254—Image signal generators using stereoscopic image cameras in combination with electromagnetic radiation sources for illuminating objects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/56—Cameras or camera modules comprising electronic image sensors; Control thereof provided with illuminating means
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
- H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
- H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
- H04N25/131—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements including elements passing infrared wavelengths
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
- H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
- H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
- H04N25/133—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements including elements passing panchromatic light, e.g. filters passing white light
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N9/00—Details of colour television systems
- H04N9/12—Picture reproducers
- H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
- H04N9/3141—Constructional details thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/10—Image acquisition
- G06V10/12—Details of acquisition arrangements; Constructional details thereof
- G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
- G06V10/143—Sensing or illuminating at different wavelengths
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/10—Image acquisition
- G06V10/12—Details of acquisition arrangements; Constructional details thereof
- G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
- G06V10/145—Illumination specially adapted for pattern recognition, e.g. using gratings
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
- H04N13/239—Image signal generators using stereoscopic image cameras using two 2D image sensors having a relative position equal to or related to the interocular distance
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Spectroscopy & Molecular Physics (AREA)
- Optics & Photonics (AREA)
- Electromagnetism (AREA)
- Image Analysis (AREA)
- Studio Devices (AREA)
- Measurement Of Optical Distance (AREA)
- Image Processing (AREA)
Abstract
The invention provides a target image acquisition system and a method, wherein the system comprises: the acquisition camera comprises a first pixel and a second pixel which are respectively used for acquiring a first wavelength image and a second wavelength image of the target area; a flood lighting unit for providing illumination of the target area at a first wavelength; a structured light projector for projecting a structured light image of a second wavelength to the target region; a processor connected to the collection camera, the flood lighting unit, and the structured light projector for: controlling the collection camera to synchronously collect a target flood image illuminated by the flood lighting unit and a structured-light image illuminated by the structured-light projector; identifying a foreground target in the target flood image; and extracting a pixel area corresponding to the foreground target in the structured light image to obtain a target structured light image. Obtaining the target depth image by performing depth calculation on the target structured light image ensures high frame rate output of the depth image.
Description
Technical Field
The present invention relates to target image acquisition, and more particularly, to a target image acquisition system and method.
Background
The advent of consumer-grade depth cameras has revolutionized many areas such as 3D modeling, gesture interaction, face recognition, and the like. Different application scenes have different performance requirements on the depth camera, for example, 3D modeling, face recognition and the like often require the depth camera to output a high-resolution depth image to improve the accuracy of modeling and face recognition algorithms; gesture interaction has a high requirement on the output frame rate of the depth camera, and the depth image with the high frame rate reduces delay, so that better user experience is brought.
At present, one of the problems faced by consumer-grade depth cameras is the contradiction between the resolution of depth images and the output frame rate, and particularly for depth cameras based on structured light technology, when the resolution of depth images is higher, the output frame rate is greatly reduced due to the increase of depth calculation amount, and high resolution and high frame rate cannot be realized at the same time.
Disclosure of Invention
The invention provides a target image acquisition system and a target image acquisition method, aiming at solving the problem that high resolution and high frame rate cannot be realized simultaneously in the prior art.
In order to solve the above problems, the technical solution adopted by the present invention is as follows:
a target image acquisition system comprising: the acquisition camera comprises a first pixel and a second pixel which are respectively used for acquiring a first wavelength image and a second wavelength image of the target area; a flood lighting unit for providing illumination of the target area at a first wavelength; a structured light projector for projecting a structured light image of a second wavelength to the target region; a processor connected to the collection camera, the flood lighting unit, and the structured light projector for: controlling the collection camera to synchronously collect a target flood image illuminated by the flood lighting unit and a structured-light image illuminated by the structured-light projector; identifying a foreground target in the target flood image; and extracting a pixel area corresponding to the foreground target in the structured light image to obtain a target structured light image.
The invention also provides a target image acquisition system, wherein the floodlighting unit is a floodlighting device; or, an illumination source independent of the target image acquisition system; the first wavelength is a visible wavelength; the second wavelength is an infrared wavelength; the flood image comprises a color image or a grayscale image.
The invention also provides a target image acquisition system, wherein the floodlighting unit and the structured light projector are always in an on state or are turned on according to a frequency, and the frequency is consistent with the exposure frequency of the acquisition camera; the synchronously collected target floodlight image and the structured light image have the same or different pixels; the processor is further configured to calculate a target depth image using the target structured light image.
The invention provides a target image acquisition method, which comprises the following steps: t1: synchronously acquiring a target floodlight image with a first wavelength and a structured light image with a second wavelength of a target area by using the same acquisition camera; t2: identifying a foreground target in the target flood image; t3: extracting a pixel area corresponding to a foreground target in the structured light image to obtain a target structured light image; the first wavelength is a visible wavelength; the second wavelength is an infrared wavelength.
The invention also provides a target image acquisition method, which further comprises the following steps: t4: and calculating a target depth image by using the target structured light image.
The invention has the beneficial effects that: the method comprises the steps of simultaneously acquiring a target floodlight image with a first wavelength and a structured light image with a second wavelength, then acquiring a foreground target of the target floodlight image and extracting a pixel area corresponding to the foreground target in the structured light image to obtain a target structured light image, and then performing depth calculation on the target structured light image to obtain a target depth image.
Drawings
FIG. 1 is a schematic diagram of a target image acquisition system according to one embodiment of the present invention.
Fig. 2 is a schematic diagram of the timing control of a floodlight, a structured light projector and a capture camera according to one embodiment of the present invention.
FIG. 3 is a schematic diagram of a target image acquisition method according to one embodiment of the invention.
FIG. 4 is a schematic diagram of a target image acquisition method according to a second embodiment of the present invention
Fig. 5 is a schematic diagram of the acquisition camera image acquisition principle according to one embodiment of the present invention.
Fig. 6 is a schematic diagram of a capture camera image capture principle according to yet another embodiment of the invention.
Fig. 7 is a schematic diagram of a target image acquisition method according to a third embodiment of the present invention.
Fig. 8 is a schematic diagram of a target image acquisition method according to a fourth embodiment of the present invention.
FIG. 9 is a schematic diagram of a target image acquisition system according to yet another embodiment of the invention.
Fig. 10 is a schematic diagram of a target image acquisition method according to a fifth embodiment of the present invention.
Fig. 11 is a schematic diagram of a target image acquisition method according to a sixth embodiment of the present invention.
Among these, 10-processor, 11-flood illuminator, 12-structured light projector, 13-capture camera, 71-first capture camera, 72-structured light projector, 73-second capture camera.
Detailed Description
The present invention will be described in detail below with reference to the following embodiments in order to better understand the present invention, but the following embodiments do not limit the scope of the present invention. It should be noted that the drawings provided in the following embodiments are only for illustrating the basic concept of the present invention, and the drawings only show the components related to the present invention rather than the number, shape and size of the components in actual implementation, the shape, number and proportion of the components in actual implementation can be changed freely, and the layout of the components can be more complicated.
FIG. 1 is a schematic diagram of a target image acquisition system according to one embodiment of the present invention. The object image acquisition system comprises a processor 10 and connected thereto a structured light projector 12 and an acquisition camera 13, wherein the structured light projector 12 is adapted to project a structured light beam into the space, which when impinging on an object forms a corresponding structured light image, which pattern is subsequently acquired by the acquisition camera 13 and forms a structured light image of the object, and the processor 10 further calculates a depth image based on the structured light image.
A single structured light projector 12 and a single capture camera 13 comprise a monocular structured light depth imaging system and processor 10 will calculate a depth image based on the principle of monocular structured light triangulation. In one embodiment, the processor 10 performs a matching calculation on the currently acquired object structured-light image and a pre-stored reference structured-light image acquired by the acquisition camera 13 or other acquisition cameras after the structured-light projector 12 projects the structured-light beam onto a plane, to obtain a pixel offset between the two images, and further calculates a depth value based on the offset.
In some embodiments, two or more acquisition cameras 13 may also be included, which together with the structured light projector 12 form a binocular or multi-purpose structured light depth imaging system. Taking a binocular structured light system composed of two collecting cameras 13 and a single structured light projector 12 as an example for explanation, when the structured light projector 12 projects structured light beams into a space, the two collecting cameras 13 collect two left and right structured light images, and the processor 10 can also obtain a depth image through matching calculation of the left and right structured light images based on a binocular vision algorithm; the left and right structured light images and the corresponding reference structured light images can be calculated to obtain two depth images, respectively, which is advantageous in that in one embodiment, the left and right acquisition modules can be set to have different parameters, such as different resolutions, focal lengths, and the like, so that the structured light images having different resolutions, angles of view, and the like can be acquired at the same time, and further, the depth images having different resolutions, angles of view, and the like can be acquired at the same time; in one embodiment, the acquired depth images can be fused into a depth image with more information.
In some embodiments, the matching calculation refers to selecting a sub-region with a certain size, such as a sub-region with a size of 7x7 or 11x11, centered on a certain pixel on the current structured light image (or the reference structured light image), and then searching a sub-region most similar to the sub-region on the reference structured light image (or the current structured light image), where the difference between the pixel coordinates of the two sub-regions on the two images is the deviation value; and secondly, calculating the depth value based on the deviation value by utilizing the corresponding relation between the deviation value and the depth value, wherein the depth values of a plurality of pixels form a depth image. The principle of the calculation of the matching between the left and right two or more structured light images is similar to the above-described principle.
In some embodiments, the object image acquisition system further comprises a floodlight 11 connected to the processor 10, the floodlight 11 being used as a floodlight unit for providing floodlight. The processor 10 controls the floodlight 11, the structured light projector 12 and the capturing camera 13 via a bus or the like, and may also be connected with the capturing camera 13 via some data transmission interface, such as an interface of MIPI, VGA or the like, to receive the image captured by the capturing camera 13. In one embodiment, flood illuminator 11 is used to emit a light beam of the same wavelength as structured light projector 12, such as infrared light, and collection camera 13 is comprised of pixels for collecting the light beam of that wavelength. The processor 10 can realize the collection of different images by controlling the time sequence among the three, and particularly can control the collection camera to collect a target floodlight image under the illumination of the floodlight illumination unit; identifying a foreground target in a target flood image; and controlling the acquisition camera to acquire a target structured light image on pixels corresponding to the foreground target under the projection of the structured light projector. In some embodiments, the flood lighting unit may also be other light sources in the environment, for example, ambient light may be used as flood lighting. The floodlight can be active light emitted by light sources such as an infrared light source and the like, and can also be ambient light. In the following embodiments, some are described in the context of a system including a flood light and some are described in the context of ambient light as a flood lighting unit, it being understood that the particular form of flood lighting may be selected in accordance with different circumstances, but the method is generic and will not be distinguished in detail below.
The processor 10 may be a depth calculation processor configured inside the system, and the processor may be a special purpose processor such as SOC, FPGA, etc., or a general purpose processor. In some embodiments, an external computing device, such as a computer, a mobile terminal, a server, or the like, may also be used, and the external computing device receives the structured light image from the acquisition module 13 and then performs depth calculation, so that the obtained depth image may be directly used for other applications of the device. In one embodiment, when the system is integrated as an embedded device into other computing terminals, such as a target image capturing device like a computer, a tablet, a mobile phone, a television, etc., the functions implemented by the processor may be implemented by a processor or an application in the terminal, for example, the depth calculation function is stored in the memory in the form of a software module and called by the processor in the terminal to implement the depth calculation. It is understood that the object image acquiring system provided by the present invention and/or the object image acquiring apparatus using the object image acquiring method provided by the present invention should be considered as the protection scope of the present invention.
The structured light image may be a stripe pattern, a two-dimensional pattern, a speckle pattern (spot pattern), etc., and the structured light wavelength may be a visible light wavelength, an infrared light wavelength, an ultraviolet light wavelength, etc.
Fig. 2 is a schematic diagram illustrating the timing control of the floodlight, the structured-light projector and the capture camera. Where timing diagrams 20, 21, and 22 correspond to flood light 11, structured light projector 12, and capture camera 13, respectively, where the raised portions indicate that the respective devices are active, such as flood light 11 in a projection state with structured light projector 12 and capture camera in an exposure state. As can be seen from fig. 2, in this embodiment, processor 10 controls floodlight 11 and structured light projector 12 to be cross-activated, and controls the capture camera to expose and capture corresponding images during each activation interval. A floodlight image A is collected under the illumination of the floodlight illuminator 11, a structured light image B is collected under the projection of the structured light projector 12, and the floodlight image A and the structured light image B are sequentially output to the processor for processing. In some embodiments, by setting the duration of activation of floodlight 11 and structured light projector 12 appropriately to capture a higher quality image, such as setting the activation time of structured light projector 12 longer, to ensure that sufficient exposure time is met to capture a higher quality structured light image; in some embodiments, the order of activating floodlight 11 and structured light projector 12 may be configured in other ways, such as activating floodlight 11 twice followed by activating structured light projector 12 once, etc., depending on the application.
In some applications, it is required to acquire a high-resolution depth image of a measured target, but the acquisition of the high-resolution depth image is limited by a depth calculation algorithm and the calculation capability of a processor, and the acquisition of the high-resolution depth image is often costly. In one embodiment of the present invention, a method for acquiring a high resolution target depth image based on the system shown in fig. 1 is provided. Fig. 3 is a schematic diagram of a target image acquisition method according to an embodiment of the invention, which is executed by the processor 10 to implement the corresponding functions.
Firstly, controlling a collecting camera 13 to collect a target floodlight image under the illumination of a floodlight unit; the target flood image referred to herein is a target flood image containing a target. In the prior art, for example, the resolution of the output depth image of a depth camera such as microsoft kinect and Intelrealsense is usually VGA, i.e. 640x480, or lower, so in the present invention, high definition resolution 1280x960 is taken as an example for explanation, and it is understood that other resolutions are also applicable to the present invention. In this step, the processor 10 applies a synchronous trigger signal to the floodlight 11 and the capturing camera 13 to acquire the floodlight image a of the target area by the capturing camera 13 while the floodlight 11 provides floodlight, where the capturing camera 13 may be a full-resolution output, that is, outputs the floodlight image a with 1280 × 960 resolution, and in one embodiment, the capturing camera 13 may also be controlled to acquire a low-resolution image of a full field of view by using a low-resolution mode such as a binning mode or a cropping mode. On the premise that the requirement of the output frame rate is high and the transmission speed of the output interface is constant, if the full-resolution image cannot realize the output of the high frame rate, the low-resolution output mode can be adopted.
Generally, the floodlight image contains interested foreground objects, such as human faces, human bodies, objects, and the like, and also contains some background objects, such as scenes where people are located. For some applications, such as face recognition, 3D modeling, etc., often only foreground object information is needed, while the background needs to be removed.
Second, foreground objects in the target flood image are identified. In this step, the foreground and the background in the floodlight image need to be segmented, and various image segmentation algorithms can be applied to this step, such as a threshold segmentation method, a mean value method (mean shift), a clustering method, and the like. When an image segmentation algorithm is selected, both calculation efficiency and calculation accuracy, especially calculation efficiency, need to be considered, and a final image output frame rate (depth image output frame rate) is reduced due to a slow image segmentation speed. After the foreground region is segmented, the foreground region, or the foreground pixel region where the foreground region is located, is identified.
And finally, controlling the acquisition camera to acquire a target structured light image on pixels corresponding to the foreground target under the projection of the structured light projector. Since the foreground pixel area is obtained in the previous step, in this step, the capture camera will only sample the pixels corresponding to the foreground area in the cropping mode (cropping mode), that is, only output the foreground image corresponding to the foreground area, and since the structured light projector is in the on state at this time, the obtained foreground image is the target structured light image. It should be noted that, for a dynamic object, such as a moving human body, pixels corresponding to the object between the front and rear images may also be distinguished, and therefore, when selecting pixels corresponding to the foreground region, the pixel region may be expanded appropriately according to the moving speed of the human body and parameters of the camera. In fact, in the case of a large frame rate (e.g., 30fps, 60fps, etc.), the foreground regions in the adjacent frame images are nearly the same.
After the above steps, the processor 10 will acquire the target structured light image required by the current application, which includes only a small field angle but has a higher resolution.
As shown in fig. 4, in the alternative embodiment of the present invention, the target depth image is obtained by performing depth calculation based on the structured light image, and since the amount of data is small relative to the full resolution, the operation speed of the depth algorithm is also high, so that high frame rate output of the depth image can be ensured.
The above steps are now described with a more intuitive embodiment, for example, the capturing camera can output 1280x960@60fps images at the highest, if the capturing camera is used for structured light image capturing, due to the limitations of depth calculation algorithm and hardware, only the output of 1280x960@10fps depth images can be realized, and due to the too low frame rate of depth images, the requirements of some applications cannot be met. The method is characterized in that the floodlight image and the structured light projector are started in a cross time sequence, the collecting camera can obtain the floodlight image of 1280x960@30fps, and the target structured light image of 640x480@30fps can be obtained after the foreground target area is identified (the target area is supposed to be positioned in the middle of the field angle of the collecting camera and occupies 50% of the whole field angle) by combining a high-speed image segmentation algorithm, so that the target structured light image of 640x480@30fps can be processed in real time and the depth image of 640x480@30fps can be output according to the current depth calculation algorithm and related hardware. Compared with the acquisition camera directly adopting 640x480@30fps, the depth image acquired by the embodiment only contains the target, the detail information is richer, and meanwhile, the image segmentation step is omitted.
In the embodiment shown in fig. 2 and fig. 3 and 4, floodlight 11 and structured light projector 12 project light beams of the same wavelength, and the capturing cameras are used to acquire floodlight image a and structured light image B, respectively, at different timings.
Fig. 5 and 6 are schematic diagrams illustrating the acquisition principle of an acquisition camera according to some embodiments of the present invention. In fig. 5, the collecting camera can synchronously collect light beams with two wavelengths, and has a W pixel sensitive to white light (light with all wavelengths) and an IR pixel sensitive to infrared light, and when the floodlight lighting unit is ambient light and the structured light projector is used for projecting infrared structured light, the collecting camera can simultaneously collect a floodlight image and a structured light image, but effective pixels of the floodlight image and the structured light image are lower than the whole pixels of the collecting camera. In this embodiment, the effective pixels are half of the total pixels, and in other embodiments, the ratio of the flood image pixels to the structured-light image pixels may be other, such as W: IR ═ 1: 3, more image details in the structured light image can be ensured, and the acquired depth image has more fine information. In fig. 6, the collecting camera can simultaneously collect a color image (RGB) and an infrared image, for example, can simultaneously collect a color flood image and an infrared structured light image, and can also collect a color structured light image and an infrared flood image.
It will be appreciated that fig. 5 and 6 are merely illustrative and are not limited to the examples of fig. 5 and 6 in practice, and that when the wavelengths of the light beams emitted by floodlight 11 and structured-light projector 12 are different, the corresponding flood image of the target and structured-light image can be simultaneously captured by using the capture cameras simultaneously sensitive to the two wavelengths. Fig. 7 is a schematic diagram of a target image acquisition method based on a collection camera simultaneously collecting a target flood image and a structured light image of different wavelengths, which is executed by the processor 10 to implement the corresponding functions.
Firstly, controlling the collecting camera to collect a target floodlight image and a structural light image under the illumination of the floodlight 11 and the structural light projector 12; floodlight 11 and structured light projector 12 may be on at all times or may be pulsed on at a frequency that corresponds to the exposure frequency of the capture camera and with a certain interval. Assuming that the collecting camera can output 1280x960@30fps images, under the simultaneous illumination of the floodlight illuminator and the structured light projector, each acquired image contains target floodlight image information and structured light image information, and for the collecting camera shown in fig. 5, the floodlight image information and the structured light image information in each image account for half of each image, the collecting camera then respectively extracts the corresponding pixels of each image and fills other blank pixels according to an upsampling algorithm, so that a floodlight image of 1280x960@30fps and a structured light image of 1280x960@30fps can be finally obtained, and no parallax exists between the floodlight image and the structured light image.
Second, foreground objects in the target flood image are identified. In this step, the foreground and the background in the floodlight image need to be segmented, and various image segmentation algorithms can be applied to this step, such as a threshold segmentation method, a mean value method (mean shift), a clustering method, and the like. When an image segmentation algorithm is selected, both calculation efficiency and calculation accuracy, especially calculation efficiency, need to be considered, and a final image output frame rate (depth image output frame rate) is reduced due to a slow image segmentation speed. After the foreground region is segmented, the foreground region, or the foreground pixel region where the foreground region is located, is identified.
And finally, extracting a pixel area corresponding to the foreground target in the structured light image to obtain a target structured light image. Because there is no parallax between the target floodlight image and the structured light image, the foreground region in the target floodlight image identified in the previous step is also the foreground region in the structured light image, and the structured light image pixel extracted from the foreground region is the target structured light image.
After the above steps, the processor 10 will acquire the target structured light image required by the current application, which includes only a small field angle but has a higher resolution.
As shown in fig. 8, in an alternative embodiment of the present invention, a target depth image can be obtained by performing depth calculation based on the structured light image, and since the amount of data is small relative to the full resolution, the operation speed of the depth algorithm is also high, so that high frame rate output of the depth image can be ensured.
Compared with the methods shown in fig. 3 and 4, in the methods shown in fig. 7 and 8, the floodlight image and the structured light image of the target in the methods shown in fig. 3 and 4 have a difference in time, which may cause the algorithm to fail when the target is a moving object and the moving speed is high; in the methods of fig. 7 and 8, the target floodlight image and the structured light image are acquired synchronously, so that the method can adapt to a fast moving object, but because the structured light image acquired by the acquisition camera only contains partial pixels, the detail information of the obtained depth image is lost.
FIG. 9 is a schematic diagram of a target image acquisition system according to yet another embodiment of the invention. The object image acquisition system comprises a processor 10 and connected thereto a first acquisition camera 71, a second acquisition camera 73 and a structured light projector 72 and a flood lighting unit, which is not shown because it uses ambient light in the system. The first and second capturing cameras 71 and 73 are used to capture images of different wavelengths, respectively. In one embodiment, a first capture camera is used to capture a target flood image of a target area at a first wavelength; the second acquisition camera is used for acquiring the structured light image of the target area. It will be appreciated that in one embodiment, a flood illuminator may also be included, or an illumination source separate from the target image acquisition system. The following description will take the case of the ambient light as the flood light.
In one embodiment, the first capture camera is an RGB camera for capturing RGB images; the second acquisition camera is an infrared camera and is used for acquiring IR images; the structured light projector is configured to emit an infrared structured light image. Because of the parallax between the RGB camera and the infrared camera, the two cameras need to be calibrated, and the calibration can be performed by any calibration method in the prior art, and the calibration is to obtain the mutual position relationship (translation and placement matrix, R and T) of one camera with respect to the other camera. FIG. 10 is a schematic diagram of a target image acquisition method according to another embodiment of the invention. The method is executed by the processor 10 to implement the corresponding functionality.
Firstly, the RGB camera and the infrared camera are controlled to acquire an RGB image and an infrared structured light image. The processor 10 controls the RGB camera and the infrared camera to extract the RGB image and the infrared structured light image at the same frame rate, the RGB image and the infrared image may have the same or different resolutions, and generally, the RGB camera in the system needs to be used to perform tasks such as taking a picture, so the RGB image has a higher resolution.
Secondly, recognizing a foreground target in the RGB image; in this step, the foreground and the background in the RGB image need to be segmented, and various image segmentation algorithms can be applied to this step, such as a threshold segmentation method, a mean value method (mean shift), a clustering method, and the like. When an image segmentation algorithm is selected, both calculation efficiency and calculation accuracy, especially calculation efficiency, need to be considered, and a final image output frame rate (depth image output frame rate) is reduced due to a slow image segmentation speed. After the foreground region is segmented, the foreground region, or the foreground pixel region where the foreground region is located, is identified.
And finally, extracting a target structured light image on a pixel corresponding to the foreground target on the infrared structured light image based on the relative position relation between the RGB camera and the infrared camera. After the area where the foreground target is located is confirmed in the RGB image, the area where the corresponding foreground target is located in the target structured light image can be located according to the relative position relationship between the RGB camera and the infrared camera, and further the pixel of the area can be extracted to serve as the target structured light image.
As shown in fig. 11, after the above steps, the processor 10 will obtain the target structured light image required by the current application, and then calculate the depth values of the pixels in the target structured light image by using the depth algorithm to generate the target depth image. In this embodiment, the final target structured light image has a small total pixel number, so that the depth calculation can be performed in real time, and the output of a high frame rate is achieved. In the step of acquiring the RGB image and the infrared image, the RGB image and the infrared image may also be acquired asynchronously, and the RGB image and the infrared image may be acquired separately in a certain time sequence in a form similar to that in the embodiment shown in fig. 3 and 4, so that the requirement on the storage and operation capability of the processor 10 may be reduced. In the time sequence acquisition mode, the infrared camera can acquire the infrared structured light image by using the cutting mode based on the foreground target area identified in the RGB image, so that the data volume can be further reduced to ensure high-speed output.
The foregoing is a more detailed description of the invention in connection with specific preferred embodiments and it is not intended that the invention be limited to these specific details. For those skilled in the art to which the invention pertains, several equivalent substitutions or obvious modifications can be made without departing from the spirit of the invention, and all the properties or uses are considered to be within the scope of the invention.
Claims (10)
1. An object image acquisition system, comprising:
the acquisition camera comprises a first pixel and a second pixel which are respectively used for acquiring a first wavelength image and a second wavelength image of the target area; the first wavelength and the second wavelength are different;
a flood lighting unit for providing illumination of the target area at a first wavelength;
a structured light projector for projecting a structured light image of a second wavelength to the target region;
a processor connected to the collection camera, the flood lighting unit, and the structured light projector for:
controlling the collection camera to synchronously collect a target flood image illuminated by the flood lighting unit and a structured-light image illuminated by the structured-light projector; the target flood image is of a different resolution than the structured light image;
identifying a foreground target in the target flood image;
and extracting a pixel area corresponding to the foreground target in the structured light image to obtain a target structured light image.
2. The target image acquisition system of claim 1 wherein the flood lighting unit is a flood light.
3. The target image acquisition system of claim 1 wherein the flood lighting unit is a separate lighting source from the target image acquisition system.
4. The target image acquisition system of claim 1 wherein the first wavelength is a visible wavelength; the second wavelength is an infrared wavelength.
5. The target image acquisition system of claim 4 wherein the target flood image comprises a color image or a grayscale image.
6. The object-image acquisition system of claim 1, wherein the flood lighting unit and the structured-light projector are always on or are turned on at a frequency that coincides with an exposure frequency of the capture camera.
7. The target image acquisition system of claim 1 wherein the processor is further configured to calculate a target depth image using the target structured light image.
8. A target image acquisition method is characterized by comprising the following steps:
t1: synchronously acquiring a target floodlight image with a first wavelength and a structured light image with a second wavelength of a target area by using the same acquisition camera; the target flood image and the structured light image are of the same or different resolution;
t2: identifying a foreground target in the target flood image;
t3: and extracting a pixel area corresponding to the foreground target in the structured light image to obtain a target structured light image.
9. The target image acquisition method of claim 8, further comprising the steps of:
t4: and calculating a target depth image by using the target structured light image.
10. The target image acquisition method of claim 8, wherein the first wavelength is a visible wavelength; the second wavelength is an infrared wavelength.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810278510.3A CN108683902B (en) | 2018-03-31 | 2018-03-31 | Target image acquisition system and method |
| PCT/CN2018/099302 WO2019184184A1 (en) | 2018-03-31 | 2018-08-08 | Target image acquisition system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810278510.3A CN108683902B (en) | 2018-03-31 | 2018-03-31 | Target image acquisition system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108683902A CN108683902A (en) | 2018-10-19 |
| CN108683902B true CN108683902B (en) | 2020-06-30 |
Family
ID=63800704
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810278510.3A Active CN108683902B (en) | 2018-03-31 | 2018-03-31 | Target image acquisition system and method |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN108683902B (en) |
| WO (1) | WO2019184184A1 (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109561260A (en) * | 2018-12-28 | 2019-04-02 | 深慧视(深圳)科技有限公司 | A kind of automatic exposure control method |
| CN110012195B (en) * | 2019-03-18 | 2021-07-06 | 奥比中光科技集团股份有限公司 | Method and device for reducing interference on depth camera, terminal equipment and storage medium |
| CN110490920A (en) * | 2019-07-12 | 2019-11-22 | 深圳奥比中光科技有限公司 | Merge depth calculation processor and 3D rendering equipment |
| CN113520228B (en) * | 2020-04-22 | 2023-05-26 | 科沃斯机器人股份有限公司 | Environment information acquisition method, autonomous mobile device and storage medium |
| CN112749610A (en) * | 2020-07-27 | 2021-05-04 | 腾讯科技（深圳）有限公司 | Depth image, reference structured light image generation method and device and electronic equipment |
| CN114079709B (en) * | 2020-08-07 | 2024-04-12 | 安霸国际有限合伙企业 | Driver mechanism for rolling shutter sensor to acquire structured light patterns |
| CN114125193A (en) * | 2020-08-31 | 2022-03-01 | 安霸国际有限合伙企业 | Timing mechanism for contaminant free video streaming using RGB-IR sensors with structured light |
| CN116320357A (en) * | 2023-05-17 | 2023-06-23 | 浙江视觉智能创新中心有限公司 | 3D structured light camera system, method, electronic device and readable storage medium |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101477631B (en) * | 2009-01-20 | 2011-01-19 | 深圳先进技术研究院 | Method, equipment for extracting target from image and human-machine interaction system |
| KR102070778B1 (en) * | 2012-11-23 | 2020-03-02 | 엘지전자 주식회사 | Rgb-ir sensor with pixels array and apparatus and method for obtaining 3d image using the same |
| US10237528B2 (en) * | 2013-03-14 | 2019-03-19 | Qualcomm Incorporated | System and method for real time 2D to 3D conversion of a video in a digital camera |
| US9953428B2 (en) * | 2015-03-03 | 2018-04-24 | Microsoft Technology Licensing, Llc | Digital camera unit with simultaneous structured and unstructured illumination |
| CN105049829B (en) * | 2015-07-10 | 2018-12-25 | 上海图漾信息科技有限公司 | Optical filter, imaging sensor, imaging device and 3-D imaging system |
| CN106934394B (en) * | 2017-03-09 | 2024-01-19 | 奥比中光科技集团股份有限公司 | Dual wavelength image acquisition system and method |
| CN107105217B (en) * | 2017-04-17 | 2018-11-30 | 深圳奥比中光科技有限公司 | Multi-mode depth calculation processor and 3D rendering equipment |
| CN107341481A (en) * | 2017-07-12 | 2017-11-10 | 深圳奥比中光科技有限公司 | It is identified using structure light image |
| CN107623814A (en) * | 2017-08-09 | 2018-01-23 | 广东欧珀移动通信有限公司 | The sensitive information screen method and device of shooting image |
| CN107707831A (en) * | 2017-09-11 | 2018-02-16 | 广东欧珀移动通信有限公司 | Image processing method and device, electronic installation and computer-readable recording medium |
-
2018
- 2018-03-31 CN CN201810278510.3A patent/CN108683902B/en active Active
- 2018-08-08 WO PCT/CN2018/099302 patent/WO2019184184A1/en active Application Filing
Also Published As
| Publication number | Publication date |
|---|---|
| CN108683902A (en) | 2018-10-19 |
| WO2019184184A1 (en) | 2019-10-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108683902B (en) | Target image acquisition system and method | |
| CN108648225B (en) | Target image acquisition system and method | |
| WO2019184183A1 (en) | Target image acquisition system and method | |
| JP6587421B2 (en) | Information processing apparatus, information processing method, and program | |
| US20230377183A1 (en) | Depth-Aware Photo Editing | |
| US8786679B2 (en) | Imaging device, 3D modeling data creation method, and computer-readable recording medium storing programs | |
| JP6417702B2 (en) | Image processing apparatus, image processing method, and image processing program | |
| JP5916248B2 (en) | Image generating apparatus, image generating method, program, and computer-readable information storage medium | |
| US20200082160A1 (en) | Face recognition module with artificial intelligence models | |
| CN112189147B (en) | Time-of-flight (TOF) camera and TOF method | |
| CN111345029B (en) | Target tracking method and device, movable platform and storage medium | |
| CN109299662B (en) | Depth data calculation device and method, and face recognition device | |
| CN108399596B (en) | Depth image engine and depth image calculation method | |
| US9049369B2 (en) | Apparatus, system and method for projecting images onto predefined portions of objects | |
| US11143879B2 (en) | Semi-dense depth estimation from a dynamic vision sensor (DVS) stereo pair and a pulsed speckle pattern projector | |
| JPWO2015098288A1 (en) | Image processing apparatus and image processing method | |
| JP2020042772A (en) | Depth data processing system capable of optimizing depth data by image positioning with respect to depth map | |
| US20210250498A1 (en) | Electronic device and method for displaying image in electronic device | |
| US20230062973A1 (en) | Image processing apparatus, image processing method, and storage medium | |
| CN113870213A (en) | Image display method, image display device, storage medium, and electronic apparatus | |
| KR20160146567A (en) | Method and device for detecting variable and fast moving object | |
| KR20150101343A (en) | Video projection system | |
| JP2014011639A (en) | Imaging device | |
| US20230316640A1 (en) | Image processing apparatus, image processing method, and storage medium | |
| GB2573593A8 (en) | Augmented reality rendering method and apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |