CN108924408B - Depth imaging method and system - Google Patents
Depth imaging method and system Download PDFInfo
- Publication number
- CN108924408B CN108924408B CN201810619054.4A CN201810619054A CN108924408B CN 108924408 B CN108924408 B CN 108924408B CN 201810619054 A CN201810619054 A CN 201810619054A CN 108924408 B CN108924408 B CN 108924408B
- Authority
- CN
- China
- Prior art keywords
- image
- structured light
- depth
- target
- light field
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
- G03B17/00—Details of cameras or camera bodies; Accessories therefor
- G03B17/48—Details of cameras or camera bodies; Accessories therefor adapted for combination with other photographic or optical apparatus
- G03B17/54—Details of cameras or camera bodies; Accessories therefor adapted for combination with other photographic or optical apparatus with projector
Abstract
The invention is suitable for the field of optical measurement and manufacture, and provides a depth imaging method and a depth imaging system, wherein the depth imaging method comprises the following steps: controlling the projector to emit a structured light beam toward the target; acquiring a target structured light field image of a target object acquired by a light field camera, wherein the target structured light field image is formed by irradiating the target object with a structured light beam; calculating at least two target structure light view angle images under different view angles according to the target structure light field image; forming a first depth image with a reference structured light image from the target structured light field image or the target structured light view image; forming a second depth image according to at least two optical view angle images of the target structure; and fusing the first depth image and the second depth image to obtain a third depth image. The embodiment of the invention adopts the first depth image and the second depth image based on the different-depth imaging principle to carry out fusion, thereby realizing the depth imaging with large measurement range and high precision.
Description
Technical Field
The invention belongs to the field of optical measurement and manufacturing, and particularly relates to a depth imaging method and system.
Background
Depth imaging based on structured light technology has received widespread attention and great development in recent years. And structured light depth imaging by using a depth camera is applied to equipment such as televisions, robots and mobile terminals to realize functions such as somatosensory interaction, 3D modeling, obstacle avoidance and face recognition.
However, in current structured light depth imaging, whether it is a monocular structured light depth camera or a binocular structured light depth camera, the baseline between the projection module, the collection camera, or between multiple collection cameras is limited by the physical dimensions of the module or the camera itself. For structured light technology, the length of the baseline often determines the measurement range and measurement accuracy of the depth camera, and longer baselines are more beneficial for long-distance depth measurement, and vice versa. At present, the minimum distance of the baseline is also in the centimeter magnitude, and the nearest measurement distance of the depth camera is about 40cm, which undoubtedly limits the application range of the depth camera, for example, the depth imaging of a close object cannot be performed. Furthermore, as the measurement distance increases, the accuracy of depth imaging may severely degrade. Therefore, how to realize a large measurement range and high-precision depth imaging is a difficult problem which is difficult to overcome by a structured light depth camera.
Disclosure of Invention
In view of this, embodiments of the present invention provide a depth imaging method and system to implement depth imaging with a large measurement range and high precision.
The invention provides a depth imaging method in a first aspect, which comprises the following steps:
controlling the projector to emit a structured light beam toward the target;
acquiring a target structured light field image of a target object acquired by a light field camera, wherein the target structured light field image is formed by irradiating the target object with a structured light beam;
calculating at least two target structure light view angle images under different view angles according to the target structure light field image;
forming a first depth image with a reference structured light image from the target structured light field image or the target structured light view image; forming a second depth image according to at least two optical view angle images of the target structure;
and fusing the first depth image and the second depth image to obtain a third depth image.
A second aspect of the present invention provides a depth imaging method, comprising:
acquiring a target structured light field image of a target object acquired by a light field camera, wherein the target structured light field image is formed by irradiating the target object with natural light or a structured light beam emitted by a projector;
calculating at least two target structure light view angle images under different view angles according to the target structure light field image;
forming a first depth image according to at least two optical view angle images of the target structure;
detecting a target region of interest in the first depth image;
according to the depth information of the interested target area, carrying out digital zooming or focusing on the target structured light field image to obtain a target structured light digital zooming image or a target structured light digital focusing image;
forming a second depth image with the reference structured light image according to the target structured light digital zoom image or the target structured light digital focusing image;
and fusing the first depth image and the second depth image to obtain a third depth image.
A third aspect of the invention provides a depth imaging system comprising a projector, a light field camera and a processing device, the projector and light field camera being arranged along a baseline;
the projector is used for emitting a structured light beam to a target object;
the light field camera is used for collecting a target structured light field image of a target object, and the target structured light field image is formed by irradiating the target object with a structured light beam;
the processing equipment is used for controlling the projector to emit the structured light beam to the target object and controlling the light field camera to collect the target structured light field image of the target object; acquiring a target structured light field image of a target object acquired by a light field camera; calculating at least two target structure light view angle images under different view angles according to the target structure light field image; forming a first depth image with a reference structured light image from the target structured light field image or the target structured light view image; forming a second depth image according to at least two optical view angle images of the target structure; and fusing the first depth image and the second depth image to obtain a third depth image.
A fourth aspect of the invention provides a depth imaging system comprising a projector, a light field camera and a processing device, the projector and the light field camera being arranged along a baseline;
the projector is used for emitting a structured light beam to a target object;
the light field camera is used for collecting a target structured light field image of a target object, and the target structured light field image is formed by irradiating the target object with structured light beams or natural light;
the processing equipment is used for controlling the light field camera to collect a target structured light field image of a target object; acquiring a target structured light field image of a target object acquired by a light field camera; calculating at least two target structure light view angle images under different view angles according to the target structure light field image; forming a first depth image according to at least two optical view angle images of the target structure; detecting a target region of interest in the first depth image; according to the depth information of the interested target area, carrying out digital zooming or focusing on the target structured light field image to obtain a target structured light digital zooming image or a target structured light digital focusing image; forming a second depth image with the reference structured light image according to the target structured light digital zoom image or the target structured light digital focusing image; and fusing the first depth image and the second depth image to obtain a third depth image.
A fifth aspect of the invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, performs the steps of the method according to the first or second aspect.
The embodiment of the invention adopts two depth images based on the different depth imaging principle to carry out fusion calculation, thereby realizing the depth imaging with large measurement range and high precision.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.
FIG. 1 is a schematic diagram of a depth imaging system according to an embodiment of the present invention;
FIG. 2 is a method flow diagram of a depth imaging method provided by an embodiment of the present invention;
FIG. 3 is a method flow diagram of another depth imaging method provided by embodiments of the present invention;
FIG. 4 is a method flow diagram of another depth imaging method provided by embodiments of the present invention;
FIG. 5 is a method flow diagram of another depth imaging method provided by embodiments of the present invention;
FIG. 6 is a method flow diagram of another depth imaging method provided by embodiments of the present invention;
fig. 7 is a flowchart of another depth imaging method according to an embodiment of the present invention.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary detail.
Fig. 1 is a schematic diagram of a depth imaging system according to an embodiment of the present invention. The depth imaging system 1, which is used to enable depth imaging of a subject 30, comprises a light field camera 10, a structured light projector 20 and a processing device 40.
The light field camera 10 includes, as main constituent components, an image sensor 101, a filter (not shown in fig. 1), a Micro Lens Array (MLA) 102, and a Lens 103. The image sensor 101 may be a Charge Coupled Device (CCD) or a complementary metal-Oxide-Semiconductor (CMOS) image sensor. The filter may be a bayer filter, an infrared filter, or the like.
Light field cameras can be classified into conventional light field cameras and focused light field cameras according to the distance between the microlens array 102 and the image sensor 101 and the lens 103. Conventional light field cameras such as the light field camera product of lytro corporation; focused light field cameras such as those available from Raytrix corporation. The present invention will be described by taking a conventional light field camera as an example, and it is understood that any kind of light field camera is suitable for the present invention.
In a conventional light field camera, the microlens array 102 is located at the focal plane of the lens 103, and the image sensor 101 is located at the focal plane of the microlens array 102. The light field camera 10 is different from a general camera in that a micro lens array for recording light direction information is included therein, and on this basis, effects of multi-view imaging, digital focusing imaging, digital zoom imaging, and the like can be further achieved, and specific principles are not described herein.
For convenience of the following description, the original image directly acquired by each pixel of the image sensor 101 in the light field camera 10 is referred to as a light field image, such as a structured light field image, in the present invention; images of a plurality of view angles obtained by processing an original image are referred to as view angle images, and the view angle images include images obtained by summing pixel arrays corresponding to the microlens array 102, and the like; images on different focal lengths acquired by digitally processing an original image are called digital zoom images; images on different image planes obtained by digitally processing the original image are referred to as digital in-focus images.
It is to be understood that any form of light field camera may be used in the present invention, such as a light field camera array composed of multiple cameras, or a light field camera formed by replacing the microlens array 102 with a mask, etc.
The main components of the structured light projector 20 include a light source 201 and an optical assembly 202, wherein the optical assembly 202 is configured to modulate a light beam emitted from the light source 201 and emit the modulated light beam to the outside. The light source 201 may be a laser diode, a semiconductor laser, or the like, and may also be an edge-emitting laser, a vertical cavity surface laser transmitter, a corresponding array laser, or the like; the wavelength of the light source may be infrared, or ultraviolet, etc. The optical assembly 202 may be a refractive optical element, or a diffractive optical element, or a combination of both, for example, in one embodiment of the present invention, the optical assembly 202 includes a refractive optical element lens for converging the laser beam, and a diffractive optical element for diffractively splitting the beam converged by the lens to form the structured light. The structured light beam may be in the form of an image of speckles, spots, stripes, or a two-dimensional pattern.
It will be appreciated that when the structured light projector 20 projects light at a wavelength λ, a corresponding filter is often required in the light field camera 10 to pass the light beam at the wavelength λ, so as to improve the image quality.
The light field camera 10 and the structured light projector 20 are positioned along a baseline direction, such as the x-direction shown in fig. 1, and their optical axes may be parallel or at an angle. By arranging the optical axes of the light field camera 10 and the structured light projector 20 to be parallel as an embodiment of the present invention, the structured light depth imaging algorithm can be simplified.
The imaging system of the invention adopts the cooperation of the light field camera and the projector, reduces the volume of the whole imaging system under the condition of not increasing the cost, has compact structure and small size, and can be better integrated in other equipment, such as televisions, robots, mobile terminals and the like.
The processing device 40 is used to control the light field camera 10 and the structured light projector 20, while also being used to perform some data processing tasks. Such as receiving raw data from the light field camera 10 and performing data processing such as multi-view imaging, digital zooming, depth image calculation, etc. The processing device 40 may include one or more processors and one or more memories, at least some of which may also be disposed in the light field camera 10 and/or the structured light projector 20 in some embodiments of the present invention. The Processor may include one or a combination of a Digital Signal Processor (DSP), an Application Processor (MAP), a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), and the like, and the Memory may include one or a combination of a Random Access Memory (RAM), a Read Only Memory (ROM), a Flash Memory (Flash), and the like. The control and data processing instructions executed by the processing device may be stored in the memory in the form of software, firmware, etc. and called by the processor when necessary, or may be directly solidified into a special circuit (or a special processor) to execute the corresponding instructions, or may be implemented in the form of a combination of software and special circuit. Processing device 40 may also include input/output interfaces and/or network interfaces to support network communications. In some embodiments of the present invention, the processed data is transmitted to other devices or other units 50 in the system, such as a display unit, or an external terminal device, etc., through an interface. In other embodiments of the invention, the display unit may also be combined with one or more processors in the processing device.
Based on the depth imaging system shown in fig. 1, the present invention can realize the following three depth imaging methods.
Monocular structured light depth imaging
As shown in fig. 2, an embodiment of the present invention provides a depth imaging method for use in the case of depth imaging of a target object, which is performed by a processing device 40. The depth imaging method, as shown in fig. 2, includes steps S201 to S203.
S201, controlling the projector to emit the structured light beam to the target object.
Wherein, under control of the processing device 40, the structured light projector 20 projects a structured light beam toward the target 30 within the space, as shown in fig. 1. As an embodiment of the present invention, the structured light beam is an infrared speckle image beam.
S202, acquiring a target structured light image of a target object acquired by a light field camera, wherein the target structured light image is formed by irradiating the target object with a structured light beam.
Wherein the processing device 40 controls the light field camera 10 to acquire a structured light image of the object in the space in real time as the structured light projector 20 projects the structured light beam onto the object 30 in the space, the structured light image being reflected back by the object in the space, thereby acquiring the structured light image of the object acquired by the light field camera. It is understood that the original image directly collected by each pixel of the image sensor in the light field camera 10 actually includes the intensity and direction information of the light beam, and then the required target structured light image can be obtained by further processing the original image, that is, the target structured light image here may be a target structured light field image, a target structured light view angle image, a target structured light zoom image, a target structured light focus image, or the like.
And S203, forming a first depth image according to the target structured light image and the reference structured light image.
And calculating a depth value according to the deviation value and a structured light trigonometry method to form a first depth image.
In an embodiment of the invention, the reference structured light image is pre-acquired during a calibration phase. In one embodiment, a reference screen, such as a flat panel, is provided at a known distance from the imaging system, and the structured-light projector 20 is synchronously controlled to project the structured-light image and the light field camera 10 acquires the reference structured-light image. The reference structured light image may be a reference structured light field image, or a reference structured light view angle image, a reference structured light focusing image, or a reference structured light focusing image.
In some other embodiments of the present invention, the reference structured light image may be acquired by other cameras, for example, a 2D camera with higher resolution and larger field angle is used to acquire the reference structured light image, and the advantage of using the common 2D image as the reference structured light image is that the structured light image can be recorded more comprehensively and clearly.
According to the type of the target structured light image, step S203, the first depth image is formed according to the target structured light image and the reference structured light image, and the following cases are mainly classified, and in the following cases, any one of the reference structured light images can be selected.
In some embodiments of the present invention, the target structured light image is an original image acquired by a light field camera, i.e. a target structured light field image; although the original image has a higher resolution, the accuracy of the calculation of the deviation value is not so high when the matching calculation is performed, since it does not reflect the detailed features of the structured light image well.
In some embodiments of the present invention, the target structured light image is a 2D image obtained by further processing the original image, that is, a target structured light view angle image, for example, a 2D image obtained by summing pixels corresponding to each lens unit of the microlens array, or a 2D image at a certain viewing angle formed by pixels at the same position in the pixel array corresponding to each lens unit of the microlens array. Compared with the method of directly performing matching calculation by using the original image, the method has the advantages that the multi-view image is subjected to dimension reduction processing on the basis of the original image, so that the resolution is reduced, the calculation speed of the matching algorithm is increased, and the requirement on a memory is reduced. It should be noted that, if a 2D image at a certain viewing angle is used, when calibrating internal and external parameters of the projector and the light field camera, calibration needs to be performed on the same viewing angle image.
In some embodiments of the present invention, the target structured light image further comprises a digitally zoomed or digitally focused 2D image, i.e. a target structured light digital zoom or focus image. In this embodiment, the first depth image may be calculated using the target structured light digital zoom or focus image and the reference structured light image. Since digital focusing or zooming realizes clear imaging of the target object, the present embodiment further improves the imaging accuracy.
On the basis of the above embodiment, as shown in fig. 3, after step S203, the method further includes:
s204, detecting the interested target area in the first depth image.
S205, according to the depth information of the interested target area, carrying out digital zooming or focusing on the target structure light image, and acquiring a target structure light digital zooming image or a target structure light digital focusing image.
And S206, forming a second depth image according to the target structured light digital zoom image or the target structured light digital focusing image and the reference structured light image.
In this embodiment, any target structured light image, such as a target structured light field image, a target structured light view angle image, a target structured light focusing image, or a target structured light focusing image, is first adopted, and after a first depth image is calculated with a reference structured light image, the first depth image is subjected to image background segmentation and other processing to determine an interested target area, such as an area where a human body is located, an area where an article is located, or the like, and the target structured light field image is subjected to digital zooming or digital focusing according to depth information of the interested target area, that is, an object in the interested target area is clearly imaged. The depth information here may be a depth value of a certain point in the target region of interest, or may be an average depth value of the target region of interest, or the like. And finally, performing matching calculation by using the digital zoom or focused target structured light digital zoom or focused image and the reference structured light image to obtain a second depth image. It will be appreciated that the second depth image will have a higher imaging accuracy than the first depth image, since digital zooming or focusing enables a sharp imaging of the target object, thereby improving the matching accuracy.
In this embodiment, since the focal length (i.e., digital zoom) or the image plane position (i.e., digital focus) in the digital zoom image changes, the focal length or the image plane position also needs to be adjusted in the depth calculation algorithm corresponding to the second depth image, that is, the depth calculation algorithm needs to be adaptively adjusted when the first depth image is calculated and the second depth image is calculated, so as to implement high-precision depth image calculation.
On the basis of the foregoing embodiment as shown in fig. 3, optionally, in step S206, forming a second depth image with the reference structured light image according to the target structured light digital zoom image or the target structured light digital focusing image, includes: and forming a second depth image according to the target structured light digital zoom image or the target structured light digital focusing image and the reference structured light digital zoom or focusing image.
The method for forming the reference structured light digital zoom or focus image is similar to the method for forming the target structured light digital zoom image or the target structured light digital focus image in the embodiment shown in fig. 3, and is not repeated here.
In this embodiment, first, any target structured light image, such as a target structured light field image, a target structured light view angle image, and the like, and any reference structured light image, such as a reference structured light field image, a reference structured light view angle image, and the like, are used to perform matching calculation to obtain a first depth image; then, image background segmentation and other processing are carried out on the first depth image to determine an interested target area, such as an area where a human body is located and an area where an article is located, digital focusing or zooming is carried out on the target structured light field image according to the depth information of the interested target area, and meanwhile digital zooming or focusing is carried out on the reference structured light field image, for example, digital zooming or focusing is carried out on the reference structured light field image based on the distance of a reference screen placed in a calibration stage, or digital zooming or focusing is carried out on the reference structured light field image based on the depth information of the interested target area; and finally, performing matching calculation by using the target structured light digital zooming or focusing image and the reference structured light digital zooming or focusing image to obtain a second depth image. Here, since the digital zooming or focusing realizes clear imaging of the target object or the reference screen, or unifies the focal lengths of the target object image and the calibration image, the matching accuracy is further improved.
Based on the above embodiment as shown in fig. 3, optionally, when the first depth image is calculated, the first depth image may be further subjected to a dimensionality reduction process to obtain a rough first depth image, and at this time, since the first depth image is calculated by the dimensionality reduction process, the second depth image will also have a higher resolution than the first depth image.
It is understood that the above embodiments only schematically describe some functions of the depth imaging system of the present invention, and different depth image calculation modes can be adaptively changed according to different application requirements by using the depth imaging system of the present invention, for example, in an application embodiment with low accuracy requirement on the depth image, the depth calculation can be directly performed by using the light view angle image of the target structure, and in an application embodiment with high accuracy requirement on the depth image, the depth image is calculated by using the light view angle image/light field image of the target structure in combination with the digital zoom image.
Compared with the traditional monocular structured light depth imaging system consisting of a structured light projector and a common 2D camera, the monocular structured light depth imaging system utilizing the light field camera has obvious advantages, and on one hand, the functions are diversified, namely, the rapid and low-precision depth image acquisition can be realized, and the high-precision depth image acquisition can also be realized; on the other hand, the depth imaging system has higher precision through the detection of the interested target area and the digital zooming/focusing, can realize clear imaging even under the far field situation, and solves the problem that the precision of the traditional depth camera is sharply reduced due to the increase of the distance.
Two, multi-eye structured light depth imaging
The multi-view structured light depth imaging is an extension of the binocular structured light depth imaging, for example, the three-view structured light depth imaging can be regarded as a simple superposition of two binocular structured light depth imaging, and therefore the binocular structured light depth imaging is described as an example in the following explanation.
As shown in fig. 4, an embodiment of the present invention provides a depth imaging method for use in the case of depth imaging of a target object, which is performed by the processing device 40 shown in fig. 1. The depth imaging method, as shown in fig. 4, includes steps S401 to S403.
S401, controlling the projector to emit the structured light beam to the target object.
Wherein, under control of the processing device 40, the structured light projector 20 projects a structured light beam toward the target 30 within the space, as shown in fig. 1. As an embodiment of the present invention, the structured light beam is an infrared speckle image beam.
S402, acquiring a target structured light field image of a target object acquired by a light field camera, wherein the target structured light field image is formed by irradiating the target object with a structured light beam.
Wherein, the processing device 40 controls the light field camera 10 to acquire the structured light field image of the object acquired by the light field camera in real time while the structured light projector 20 projects the structured light beam onto the object 30 in the space, and thereby the structured light field image of the object acquired by the light field camera is acquired.
And S403, calculating at least two target structure light visual angle images under different visual angles according to the target structure light field image, and forming a first depth image according to the at least two target structure light visual angle images.
Wherein, the processing device 40 calculates at least two target structure light view angle images at different view angles according to the target structure light field image.
In the embodiment of the present invention, the processing device 40 performs matching calculation using two target structured light view images to obtain a deviation value between feature points of the images, and calculates a depth value according to the deviation value in combination with a structured light triangulation method to form a first depth image.
It should be noted that, when calculating the depth value based on the deviation value, it is necessary to obtain the relative position relationship between different viewing angles and the camera internal parameters corresponding to the viewing angle image in advance, which is similar to the binocular vision algorithm, and it is necessary to obtain the internal and external parameters between the left and right cameras in advance, where the internal and external parameters corresponding to different viewing angles may be obtained in advance by using a calibration algorithm such as the zhangyingyou calibration method, and the internal and external parameters are stored in the memory in advance, and called when the processor calculates the depth value.
It will be appreciated that multi-view structured light depth imaging does not require reference to a structured light image, and in some embodiments the depth image may be calculated even without projection by a structured light projector, as long as the target object possesses sufficient textural features and its textural image is captured by the light field camera. Therefore, the far-field imaging distance of the multi-eye structure light depth imaging is larger than that of the monocular structure light imaging.
Compared with the traditional binocular structured light depth imaging system, the depth calculation is carried out through different visual angles in the light field camera, and because the relative position deviation between the different visual angles, namely the base line, can reach millimeter level, the depth imaging can be carried out on objects which are close to each other, such as 10 centimeters or even more, but the depth imaging cannot be realized by the traditional binocular structured light depth imaging system. In addition, imaging cameras in the traditional binocular structured light depth imaging system are independent from each other and connected through a support, and can deform under the influence of heat or physical impact and the like, and finally the imaging quality of a depth image can be influenced.
On the basis of the above embodiment, as shown in fig. 5, after step S403, the method further includes:
s404, detecting the interested target area in the first depth image.
S405, according to the depth information of the interested target area, carrying out digital zooming or focusing on the target structured light field image to obtain a target structured light digital zooming image or a target structured light digital focusing image.
And S406, forming a second depth image according to the target structured light digital zoom image or the target structured light digital focusing image and the reference structured light image.
In an embodiment of the invention, the reference structured light image is pre-acquired during a calibration phase. In one embodiment, a reference screen, such as a flat panel, is provided at a known distance from the imaging system, and the structured-light projector 20 is synchronously controlled to project the structured-light image and the light field camera 10 acquires the reference structured-light image. The reference structured light image may be a reference structured light field image, or a reference structured light view angle image, a reference structured light focusing image, or a reference structured light focusing image.
In this embodiment, after the first depth image is calculated, image background segmentation and other processing are performed on the first depth image to determine an interested target area, such as an area where a human body is located or an area where an article is located, and digital zooming or digital focusing is performed on the target structured light field image according to depth information of the interested target area, that is, an object in the interested target area is clearly imaged. The depth information here may be a depth value of a certain point in the target region of interest, or may be an average depth value of the target region of interest, or the like. And finally, performing matching calculation by using the digital zoom or focused target structured light digital zoom or focused image and the reference structured light image to obtain a second depth image. It will be appreciated that the second depth image will have a higher imaging accuracy than the first depth image, since digital zooming or focusing enables a sharp imaging of the target object, thereby improving the matching accuracy.
In this embodiment, since the focal length (i.e., digital zoom) or the image plane position (i.e., digital focus) in the digital zoom image changes, the focal length or the image plane position also needs to be adjusted in the depth calculation algorithm corresponding to the second depth image, that is, the depth calculation algorithm needs to be adaptively adjusted when the first depth image is calculated and the second depth image is calculated, so as to implement high-precision depth image calculation.
On the basis of the foregoing embodiment as shown in fig. 5, optionally, in step S406, forming a second depth image with a reference structured light image according to the target structured light digital zoom image or the target structured light digital focus image, includes: and forming a second depth image according to the target structured light digital zoom image or the target structured light digital focusing image and the reference structured light digital zoom or focusing image.
In this embodiment, first, a matching calculation is performed to obtain a first depth image; secondly, performing image background segmentation and other processing on the first depth image to determine an interested target area, such as an area where a human body is located and an area where an article is located, performing digital focusing/zooming on a target structured light field image according to depth information of the interested target area, and performing digital zooming/focusing on a reference structured light field image, for example, performing digital zooming/focusing on the reference structured light field image based on the distance of a reference screen placed in a calibration stage or performing digital zooming/focusing on the reference structured light field image based on the depth information of the interested target area; and finally, performing matching calculation by using the digitally focused target structured light digital focusing image and the reference structured light digital focusing image to obtain a second depth image. Here, since the digital zoom/focus enables clear imaging of the target object or the reference screen, or unifies the focal lengths of the target object image and the calibration image, the matching accuracy is further improved.
Based on the above embodiment as shown in fig. 5, optionally, when the first depth image is calculated, the first depth image may be further subjected to a dimensionality reduction process to obtain a rough first depth image, and at this time, since the first depth image is calculated by the dimensionality reduction process, the second depth image will also have a higher resolution than the first depth image.
Three, fusion depth imaging
As shown in fig. 6, an embodiment of the present invention provides a depth imaging method for use in the case of depth imaging of a target object, which is performed by the processing device 40 shown in fig. 1. The depth imaging method, as shown in fig. 6, includes steps S601 to S605.
S601, controlling the projector to emit the structured light beam to the target object.
Wherein a structured-light beam is projected by the subject 30 within the structured-light projector 20 under the control of the processing device 40, as shown in fig. 1. As an embodiment of the present invention, the structured light beam is an infrared speckle image beam.
S602, acquiring a target structured light field image of a target object acquired by a light field camera, wherein the target structured light field image is formed by irradiating the target object with a structured light beam.
Wherein the processing device 40 controls the light field camera 10 to acquire a structured light image of the object in the space in real time while the structured light projector 20 projects the structured light beam onto the object 30 in the space, thereby acquiring the structured light image of the object acquired by the light field camera.
S603, calculating at least two target structure light view angle images under different view angles according to the target structure light field image.
And S604, forming a first depth image according to the target structured light field image or the target structured light visual angle image and a reference structured light image.
In an embodiment of the invention, the reference structured light image is pre-acquired during a calibration phase. In one embodiment, a reference screen, such as a flat panel, is provided at a known distance from the imaging system, as shown in connection with FIG. 1, and the structured light projector 20 is synchronously controlled to project the structured light image and the light field camera 10 acquires the reference structured light image. The reference structured light image may be a reference structured light field image, or a reference structured light view angle image, a reference structured light focusing image, or a reference structured light focusing image.
Forming a first depth image from the target structured light field image or the target structured light view image and a reference structured light image, comprising:
and matching and calculating the target structured light field image or the target structured light view angle image with a reference structured light image to obtain a deviation value between image characteristic points, and calculating a depth value according to the deviation value and a structured light trigonometry to form a first depth image.
And S605, forming a second depth image according to the at least two target structure optical view angle images.
And performing matching calculation by using the two target structure light visual angle images to obtain deviation values between image characteristic points, and calculating a depth value by combining a structured light triangulation method according to the deviation values to form a second depth image.
In the embodiment of the present invention, steps S604 and S605 may be performed simultaneously or sequentially, and the time sequence of the two steps is not specifically limited.
And S606, fusing the first depth image and the second depth image to obtain a third depth image.
Fusing the first depth image and the second depth image by adopting a weighting algorithm to obtain a third depth image; or fusing the first depth image and the second depth image by adopting a MAP-MRF algorithm to obtain a third depth image.
In the embodiment of the invention, because the first depth image is based on the monocular structured light depth imaging principle, the accuracy is relatively higher compared with that of the monocular structured light depth imaging, but the measurement range is limited due to the baseline reason; the light depth imaging baseline based on the multi-eye structure can reach millimeter magnitude, the distance of a measurable target object is closer, the depth imaging can be realized for a long distance, the second depth image obtained according to the principle has a larger depth imaging range as described above, but the precision is reduced due to the reduction of the baseline; in the embodiment, the first depth image and the second depth image are fused, so that the depth image with a large measurement range and high precision is realized.
As an embodiment of the present invention, a fusion algorithm for fusing the first depth image and the second depth image into the third depth image is performed by using a weighting algorithm, and if D1(u, v), D2(u, v), and D3(u, v) respectively represent the first depth image, the second depth image, and the third depth image at the pixel (u, v), and a1(u, v) and a2(u, v) respectively represent confidence weights of depth values of pixels in the first depth image and the second depth image, then the third depth image may be calculated by the following formula:
D3(u,v)＝[D1(u,v)·a1(u,v)+D2(u,v)·a2(u,v)]/[a1(u,v)+a2(u,v)]。
wherein the confidence weight may be set in a number of ways, such as for the first depth image, since its depth values at a short distance, such as <0.2m, and a long distance, such as >4m, are reliable, the weighting factor is larger for such depth values; for the second depth image, the depth value in the middle area, for example, 0.2m to 4m, is more reliable, and the weighting factor for the depth value in the middle area is larger. In addition, the weighting factor may be set by introducing some other parameters, such as setting the weighting factor of the corresponding pixel while considering the depth values of the pixels around the pixel, and calculating a smoothing factor by the depth values of the pixels around the pixel, and estimating the weighting factor by the smoothing factor.
As another embodiment of the present invention, a process of fusing the first depth image and the second depth image into the third depth image is considered as a MAP-MRF problem, in which an observed value, i.e., the first depth image and the second depth image, and an estimated value, i.e., the third depth image, are modeled using a Markov Random Field (MRF), and each pixel value of the third depth image is solved by maximizing a posterior probability (MAP).
As shown in fig. 7, an embodiment of the present invention provides another depth imaging method for use in the case of depth imaging of a target object, which is performed by the processing device 40 shown in fig. 1. The depth imaging method, as shown in fig. 7, includes steps S701 to S707.
S701, acquiring a target structured light field image of a target object acquired by a light field camera, wherein the target structured light field image is formed by irradiating the target object with natural light or a structured light beam emitted by a projector.
As an embodiment of the invention, a structured light image, such as an infrared speckle image or the like, is projected into space by projector 20 under the control of processing device 40. At the same time, the processing device 40 controls the light field camera 10 to capture a target structured light field image reflected back by a target object in space.
As another embodiment of the present invention, the acquisition of the target structured-light field image may also be a depth image acquired by a passive binocular principle without structured-light projection, that is, under natural light irradiation, the processing device 40 controls the light field camera 10 to acquire the target structured-light field image reflected back by the target object in the space.
S702, calculating at least two target structure light view angle images under different view angles according to the target structure light field image.
And S703, forming a first depth image according to at least two target structure optical view angle images.
S704, detecting the interested target area in the first depth image.
S705, according to the depth information of the interested target area, carrying out digital zooming or focusing on the target structured light field image to obtain a target structured light digital zooming image or a target structured light digital focusing image.
And S706, forming a second depth image according to the target structured light digital zoom image or the target structured light digital focusing image and the reference structured light image.
In an embodiment of the invention, the reference structured light image is pre-acquired during a calibration phase. In one embodiment, a reference screen, such as a flat panel, is provided at a known distance from the imaging system, as shown in connection with FIG. 1, and the structured light projector 20 is synchronously controlled to project the structured light image and the light field camera 10 acquires the reference structured light image. The reference structured light image may be a reference structured light field image, or a reference structured light view angle image, a reference structured light focusing image, or a reference structured light focusing image.
The forming a second depth image with a reference structured light image according to the target structured light digital zoom image or the target structured light digital focus image comprises:
and matching and calculating the target structured light digital zoom image or the target structured light digital focusing image with the reference structured light image to obtain deviation values between image characteristic points, and calculating depth values according to the deviation values and a structured light trigonometry to form a second depth image.
And S707, fusing the first depth image and the second depth image to obtain a third depth image.
The process of fusing the first depth image and the second depth image to obtain a third depth image is the same as that in the embodiment shown in fig. 6, and details are not repeated here.
In the embodiment of the invention, the first depth image is based on a multi-eye depth imaging principle, and the second depth image is based on a monocular depth imaging principle. In addition, the embodiment of the invention performs matching calculation by using the digitally focused or zoomed target structured light image and the reference structured light image to acquire the second depth image. Here, since the digital zoom/focus realizes clear imaging of the target object or unifies the focal lengths of the target object image and the calibration image, the matching accuracy is further improved.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (8)
1. A depth imaging method, comprising:
acquiring a target structured light field image of a target object acquired by a light field camera, wherein the target structured light field image is formed by irradiating the target object with natural light or a structured light beam emitted by a projector;
calculating at least two target structure light view angle images under different view angles according to the target structure light field image;
forming a first depth image according to at least two optical view angle images of the target structure;
detecting a target region of interest in the first depth image;
according to the depth information of the interested target area, carrying out digital zooming or focusing on the target structured light field image to obtain a target structured light digital zooming image or a target structured light digital focusing image;
forming a second depth image with the reference structured light image according to the target structured light digital zoom image or the target structured light digital focusing image;
and fusing the first depth image and the second depth image to obtain a third depth image.
2. The depth imaging method of claim 1, wherein the reference structured light image comprises: a reference structured light field image, a reference structured light view image, a reference structured light focus changing image, or a reference structured light focus image.
3. The depth imaging method of claim 1 or 2, wherein the forming a first depth image from the target structured light field image or the target structured light view image with a reference structured light image comprises:
and matching and calculating the target structured light field image or the target structured light view angle image with a reference structured light image to obtain a deviation value between image characteristic points, and calculating a depth value according to the deviation value and a structured light trigonometry to form a first depth image.
4. The depth imaging method of claim 1 or 2, wherein said forming a second depth image from at least two of said target structure light perspective images comprises:
and performing matching calculation by using the two target structure light visual angle images to obtain deviation values between image characteristic points, and calculating a depth value by combining a structured light triangulation method according to the deviation values to form a second depth image.
5. The depth imaging method of claim 1 or 2, wherein the fusing the first depth image and the second depth image to obtain a third depth image comprises:
fusing the first depth image and the second depth image by adopting a weighting algorithm to obtain a third depth image; or
And fusing the first depth image and the second depth image by adopting a MAP-MRF algorithm to obtain a third depth image.
6. A depth imaging system comprising a projector, a light field camera and a processing device, the projector and light field camera being disposed along a baseline;
the projector is used for emitting a structured light beam to a target object;
the light field camera is used for collecting a target structured light field image of a target object, and the target structured light field image is formed by irradiating the target object with structured light beams or natural light;
the processing equipment is used for controlling the light field camera to collect a target structured light field image of a target object; acquiring a target structured light field image of a target object acquired by a light field camera; calculating at least two target structure light view angle images under different view angles according to the target structure light field image; forming a first depth image according to at least two optical view angle images of the target structure; detecting a target region of interest in the first depth image; according to the depth information of the interested target area, carrying out digital zooming or focusing on the target structured light field image to obtain a target structured light digital zooming image or a target structured light digital focusing image; forming a second depth image with the reference structured light image according to the target structured light digital zoom image or the target structured light digital focusing image; and fusing the first depth image and the second depth image to obtain a third depth image.
7. The depth imaging system of claim 6, wherein the reference structured light image comprises: a reference structured light field image, a reference structured light view image, a reference structured light focus changing image, or a reference structured light focus image.
8. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810619054.4A CN108924408B (en) | 2018-06-15 | 2018-06-15 | Depth imaging method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810619054.4A CN108924408B (en) | 2018-06-15 | 2018-06-15 | Depth imaging method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108924408A CN108924408A (en) | 2018-11-30 |
| CN108924408B true CN108924408B (en) | 2020-11-03 |
Family
ID=64421725
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810619054.4A Active CN108924408B (en) | 2018-06-15 | 2018-06-15 | Depth imaging method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108924408B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TW202025083A (en) * | 2018-12-19 | 2020-07-01 | 財團法人工業技術研究院 | Apparatus and method for dynamically adjusting depth resolution |
| CN112672137A (en) * | 2019-10-15 | 2021-04-16 | 福州瑞芯微电子股份有限公司 | Method for obtaining depth image, structured light system and electronic device |
| CN112749610A (en) * | 2020-07-27 | 2021-05-04 | 腾讯科技（深圳）有限公司 | Depth image, reference structured light image generation method and device and electronic equipment |
| CN112752088B (en) * | 2020-07-28 | 2023-03-28 | 腾讯科技（深圳）有限公司 | Depth image generation method and device, reference image generation method and electronic equipment |
| CN115190285B (en) * | 2022-06-21 | 2023-05-05 | 中国科学院半导体研究所 | 3D image acquisition system and method |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102263979A (en) * | 2011-08-05 | 2011-11-30 | 清华大学 | Depth map generation method and device for plane video three-dimensional conversion |
| CN104903677A (en) * | 2012-12-17 | 2015-09-09 | Lsi公司 | Methods and apparatus for merging depth images generated using distinct depth imaging techniques |
| CN106210568A (en) * | 2016-07-15 | 2016-12-07 | 深圳奥比中光科技有限公司 | Image processing method and device |
| CN106454287A (en) * | 2016-10-27 | 2017-02-22 | 深圳奥比中光科技有限公司 | Combined camera shooting system, mobile terminal and image processing method |
| CN106954058A (en) * | 2017-03-09 | 2017-07-14 | 深圳奥比中光科技有限公司 | Depth image obtains system and method |
| CN107067428A (en) * | 2017-03-10 | 2017-08-18 | 深圳奥比中光科技有限公司 | Augmented reality projection arrangement and method |
| CN107113415A (en) * | 2015-01-20 | 2017-08-29 | 高通股份有限公司 | The method and apparatus for obtaining and merging for many technology depth maps |
| CN107424187A (en) * | 2017-04-17 | 2017-12-01 | 深圳奥比中光科技有限公司 | Depth calculation processor, data processing method and 3D rendering equipment |
| CN107564050A (en) * | 2017-08-09 | 2018-01-09 | 广东欧珀移动通信有限公司 | Control method, device and terminal device based on structure light |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP5197785B2 (en) * | 2011-03-30 | 2013-05-15 | キヤノン株式会社 | Image processing apparatus, imaging system, and image processing system |
| US9742993B2 (en) * | 2012-02-16 | 2017-08-22 | University Of Washington Through Its Center For Commercialization | Extended depth of focus for high-resolution optical image scanning |
-
2018
- 2018-06-15 CN CN201810619054.4A patent/CN108924408B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102263979A (en) * | 2011-08-05 | 2011-11-30 | 清华大学 | Depth map generation method and device for plane video three-dimensional conversion |
| CN104903677A (en) * | 2012-12-17 | 2015-09-09 | Lsi公司 | Methods and apparatus for merging depth images generated using distinct depth imaging techniques |
| CN107113415A (en) * | 2015-01-20 | 2017-08-29 | 高通股份有限公司 | The method and apparatus for obtaining and merging for many technology depth maps |
| CN106210568A (en) * | 2016-07-15 | 2016-12-07 | 深圳奥比中光科技有限公司 | Image processing method and device |
| CN106454287A (en) * | 2016-10-27 | 2017-02-22 | 深圳奥比中光科技有限公司 | Combined camera shooting system, mobile terminal and image processing method |
| CN106954058A (en) * | 2017-03-09 | 2017-07-14 | 深圳奥比中光科技有限公司 | Depth image obtains system and method |
| CN107067428A (en) * | 2017-03-10 | 2017-08-18 | 深圳奥比中光科技有限公司 | Augmented reality projection arrangement and method |
| CN107424187A (en) * | 2017-04-17 | 2017-12-01 | 深圳奥比中光科技有限公司 | Depth calculation processor, data processing method and 3D rendering equipment |
| CN107564050A (en) * | 2017-08-09 | 2018-01-09 | 广东欧珀移动通信有限公司 | Control method, device and terminal device based on structure light |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108924408A (en) | 2018-11-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108924408B (en) | Depth imaging method and system | |
| CN108881717B (en) | Depth imaging method and system | |
| JP7043085B2 (en) | Devices and methods for acquiring distance information from a viewpoint | |
| US8718326B2 (en) | System and method for extracting three-dimensional coordinates | |
| US20170013245A1 (en) | Single-sensor system for extracting depth information from image blur | |
| WO2019100933A1 (en) | Method, device and system for three-dimensional measurement | |
| CN105578019A (en) | Image extraction system capable of obtaining depth information and focusing method | |
| JP7057097B2 (en) | Control methods and programs for distance measuring devices, distance measuring systems, imaging devices, moving objects, and distance measuring devices | |
| EP3416370B1 (en) | Photography focusing method, device, and apparatus for terminal | |
| CN107749070B (en) | Depth information acquisition method and device and gesture recognition equipment | |
| CN103019001A (en) | Automatic focusing method and device | |
| Zhou et al. | A two-step calibration method of lenslet-based light field cameras | |
| US10904512B2 (en) | Combined stereoscopic and phase detection depth mapping in a dual aperture camera | |
| CN102997891A (en) | Device and method for measuring scene depth | |
| US11490067B2 (en) | Multi-aperture ranging devices and methods | |
| WO2021084530A1 (en) | Method and system for generating a depth map | |
| CN108924407B (en) | Depth imaging method and system | |
| EP3350772A1 (en) | An apparatus and a method for calibrating an optical acquisition system | |
| CN106023189A (en) | Light field data depth reconstruction method based on matching optimization | |
| WO2022052366A1 (en) | Fused depth measurement method and measurement device | |
| CN213091888U (en) | Depth measurement system and electronic device | |
| CN111277811B (en) | Three-dimensional space camera and photographing method thereof | |
| WO2017046393A1 (en) | An apparatus and a method for generating data representing a pixel beam | |
| CN109274954B (en) | Foveola monocular stereoscopic imaging system | |
| CN115190285B (en) | 3D image acquisition system and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |