CN113534596A

CN113534596A - RGBD stereo camera and imaging method

Info

Publication number: CN113534596A
Application number: CN202110791837.2A
Authority: CN
Inventors: 林霄; 陈泽; 周家明
Original assignee: Shengjing Intelligent Technology Jiaxing Co ltd
Current assignee: Shengjing Intelligent Technology Jiaxing Co ltd
Priority date: 2021-07-13
Filing date: 2021-07-13
Publication date: 2021-10-22
Anticipated expiration: 2041-07-13
Also published as: CN113534596B

Abstract

The invention provides an RGBD stereo camera and an imaging method, wherein the RGBD stereo camera comprises: the system comprises a lens, a first light splitting module, an RGB imaging module, a second light splitting module, a plurality of TOF imaging modules and a time sequence control module; the RGB imaging module is used for acquiring color information of the target object according to the incident second light beam; the TOF imaging module is used for acquiring depth information of the target object according to the incident fourth light beam; the time sequence control module is connected with the RGB imaging module and the TOF imaging module and used for controlling each TOF imaging module to alternately output the depth information of the target object according to a target time sequence; and the time sequence control module is also used for acquiring an RGBD image based on the color information and each depth information of the target object. The RGBD stereo camera and the imaging method provided by the invention can simultaneously acquire RGBD images with high resolution and high frame rate.

Description

RGBD stereo camera and imaging method

Technical Field

The invention relates to the technical field of optical imaging, in particular to an RGBD stereo camera and an imaging method.

Background

The RGBD stereo camera contains three-dimensional information and color information, so that more-dimensional data input can be provided for various back-end developments compared with a traditional two-dimensional color camera or a traditional depth camera, and the RGBD stereo camera has more important application value. With the rapid development of artificial intelligence and machine vision technology, the RGBD stereo camera inevitably becomes an indispensable basic information acquisition device, and has a wide application prospect in the fields of industrial manufacturing, autopilot, military navigation and the like.

When the camera has a high resolution, the amount of data per image acquired is enormous, and in the case where the data bandwidth that can be transmitted by the transmission interface is limited, the number of images that can be transmitted per unit time becomes small, that is, the frame rate decreases.

In the prior art, a depth camera measurement scheme based on grating structure light has the advantage of high precision, but the frame rate is very low, and is generally not more than 5 fps. Although the precision of the stereo camera measurement scheme based on tof (time of flight) is low, the frame rate is high, and the frame rate can reach 30fps at most. However, in contrast to conventional two-dimensional color cameras, at the same resolution, the frame rate of TOF stereo cameras is much lower than two-dimensional color cameras. Therefore, the prior art cannot realize the synchronous acquisition of the three-dimensional information and the color information with high resolution and high frame rate.

Disclosure of Invention

The invention provides an RGBD stereo camera and an imaging method, which are used for solving the defect that three-dimensional information and color information of high resolution and high frame rate cannot be synchronously acquired in the prior art and improving the frame rate of an RGBD image under the condition of unchanged frame rate.

The present invention provides an RGBD stereo camera, including: the system comprises a lens, a first light splitting module, an RGB imaging module, a second light splitting module, a plurality of TOF imaging modules and a time sequence control module;

the first light splitting module is used for splitting a first light beam incident through the lens into a second light beam and a third light beam;

the RGB imaging module is used for acquiring the color information of the target object according to the incident second light beam;

the second light splitting module is used for splitting the third light beam into a plurality of fourth light beams and projecting each fourth light beam to the corresponding TOF imaging module respectively;

the TOF imaging module is used for acquiring depth information of the target object according to the incident fourth light beam;

the time sequence control module is connected with the RGB imaging module and the TOF imaging module and is used for controlling each TOF imaging module to alternately output the depth information of the target object according to a target time sequence;

the time sequence control module is further configured to obtain an RGBD image based on the color information and each depth information of the target object;

the first light beam is a light beam reflected by the target object and incident through the lens, the second light beam is visible light in the first light beam, the third light beam is laser in the first light beam, and the number of the fourth light beams is consistent with that of the TOF imaging modules.

According to the RGBD stereo camera provided in the present invention, the first light splitting module includes a dichroic mirror.

According to the RGBD stereo camera provided by the invention, the number of the TOF imaging modules is two.

According to the RGBD stereo camera provided by the invention, the TOF imaging module comprises: TOF imaging lens and TOF sensing chip.

According to the RGBD stereo camera provided by the invention, the second beam splitting module comprises a plurality of beam splitters.

According to the RGBD stereo camera provided by the invention, the target time sequence is a time sequence with equal intervals.

The RGBD stereo camera further comprises a laser emitting module;

and the laser emission module is used for generating a fifth light beam and projecting the fifth light beam to the target object.

The invention also provides an imaging method of the RGBD stereo camera, which comprises the following steps:

the first light splitting module splits a first light beam incident through a lens into a second light beam and a third light beam so that the second light splitting module splits the third light beam into a plurality of fourth light beams, and the fourth light beams are projected to corresponding TOF imaging modules respectively;

the RGB imaging module acquires color information of a target object based on the incident second light beam, and the time sequence control module controls each TOF imaging module to output depth information of the target object based on the incident fourth light beam alternately in a target time sequence;

the time sequence control module acquires the RGBD image based on the color information and each depth information of the target object;

the first light beam is a light beam reflected by the target object and incident through the lens, the second light beam is visible light in the first light beam, the third light beam is laser in the first light beam, and the number of the fourth light beams is consistent with the number of the TOF imaging module and the depth information.

According to an imaging method of an RGBD stereo camera provided by the present invention, acquiring the RGBD image based on the color information and the depth information of the target object includes:

performing image fusion based on the depth information, the color information and a target mapping relation of the target object to obtain the RGBD image;

the target mapping relation is a pixel coordinate corresponding relation of a target object between the RGB imaging module and the TOF imaging module.

The invention also provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the imaging method as described in any one of the above when executing the program.

The RGBD stereo camera and the imaging method provided by the invention are based on the fact that a plurality of TOF imaging modules are adopted to alternately measure the same target in a target time sequence in a period time, time domain supplementary fusion is carried out on a plurality of groups of obtained depth information, the depth information with the increased frame rate is obtained, each group of depth information is sequentially fused with color information, RGBD images with the number consistent with that of the TOF imaging modules are obtained in the period time, and RGBD images with high resolution and high frame rate can be obtained simultaneously.

Drawings

In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings needed for the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.

Fig. 1 is one of schematic structural diagrams of an RGBD stereo camera provided by the present invention;

fig. 2 is a second schematic structural diagram of the RGBD stereo camera provided in the present invention;

fig. 3 is a schematic flow chart of an imaging method of an RGBD stereo camera provided by the present invention;

FIG. 4 is a schematic diagram of the relationship between the laser light received and the laser light emitted by the TOF imaging module provided by the present invention;

fig. 5 is a schematic structural diagram of an electronic device provided in the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Fig. 1 is one of schematic structural diagrams of an RGBD stereo camera provided by the present invention. As shown in fig. 1, an RGBD stereo camera provided in an embodiment of the present invention includes: the image capturing device comprises a lens 110, a first light splitting module 120, an RGB imaging module 130, a second light splitting module 140, a plurality of TOF imaging modules 150 and a timing control module 160.

It is to be understood that the dotted lines marked with arrows in fig. 1 do not represent the connection relationship between the modules, but represent only the light paths having directions, and the solid lines without arrows represent the connection relationship between the modules.

Note that, since the Frame rate is a measure for measuring the number of display Frames, the measurement unit is the number of display Frames Per Second (Frames Per Second, fps). Each frame is a still picture and the multiple frames in rapid succession create a dynamic effect of motion. With a constant resolution, a higher frame rate can result in a larger number of pictures per unit time.

For an application scenario of the RGBD stereo camera, this is not specifically limited in the embodiment of the present invention. For example, the method can be used for industrial production line online detection, intelligent traffic monitoring, military science and technology and the like.

For any application scene, the system comprises an RGBD stereo camera, an object to be shot, a visible light source and a laser light source, wherein a measurement light path of the RGBD stereo camera receives reflected light beams of the two light sources on the object to be shot under the current environment and converts the reflected light beams into corresponding digital signals.

Specifically, the RGBD stereo camera is provided with a lens 110, a first light splitting module 120, an RGB imaging module 130, a second light splitting module 140, a plurality of TOF imaging modules 150 and a timing control module 160, where the timing control module 160 may stagger measurement periods of different TOF imaging modules, so that the plurality of TOF imaging modules 150 can alternately output depth information in unit time, and are fused with color information output by the RGB imaging module 130, and a plurality of RGBD picture information can be obtained in unit time.

Preferably, the number of TOF imaging modules 150 is at least two.

The first light splitting module 120 splits the first light beam incident through the lens 110 into a second light beam and a third light beam.

The first light beam is a light beam reflected by the target object and incident through the lens, the second light beam is visible light in the first light beam, and the third light beam is laser in the first light beam.

Note that, before the first light splitting module 120 operates, the lens 110 receives a reflected light beam by a subject to be photographed and transmits the light beam to the first light splitting module 120. The light beam comprises reflected light projected to the object to be shot by natural light and reflected light projected to the object to be shot by the light source.

The first light beam, which is a reflected light beam of an object to be photographed, is used as an incident light beam of the lens 110.

The second light beam is light stripped from the first light beam and has a wavelength in the visible wavelength range.

The third light beam is light having a wavelength within a laser wavelength range, which is separated from the first light beam, wherein the wavelength division laser includes a mid-infrared laser, a near-infrared laser, a visible laser, an ultraviolet laser, and the like, and the type of the laser is not particularly limited in the embodiments of the present invention.

Preferably, the laser light is laser light intensity-modulated at different times in a sinusoidal cycle.

Specifically, the first light splitting module 120 receives the first light beam via the lens 110, splits the first light beam to obtain a second light beam and a third light beam, respectively, and projects the second light beam to the RGB imaging module 130 and the third light beam to the second light splitting module 140.

The first optical splitting module 120 is used to separate visible light from laser, and the working method of the first optical splitting module 120 is not specifically limited in the embodiment of the present invention.

For example, the first light splitting module 120 may include a common light splitting prism and a plurality of filters, and after the visible light and the laser light passing through the lens 110 are mixed, the mixture is uniformly split into two parts by the light splitting prism, and each part includes the same amount of visible light and laser light. The filter is disposed in front of the RGB image module 130, so that the RGB image module 130 only receives visible light and forms an image to shield laser interference. Another filter is set before the TOF imaging module 150 so that the TOF imaging module 150 only receives laser light and images to shield the laser light band from accidental beam interference.

The RGB imaging module 130 is configured to obtain color information of the target object according to the incident second light beam.

It should be noted that the target object refers to an object to be photographed in any application scene, and the object may be a living body (a person, an animal, or the like) or an object.

Specifically, the RGB image module 130 receives the visible light (i.e., the second light beam) split by the first light splitting module 120, and the RGB image module 130 can convert the incident light signal into an RGB digital signal, and can accordingly obtain the color information of the target object.

The RGB digital signals are pixel information in an RGB color image, and the color of each pixel is synthesized using three components of red (R), green (G), and blue (B). For a color image of arbitrary size n x m, the pixel information is a multi-dimensional data array of n x m x 3, where the elements in the array define the color information (i.e., the red, green, and blue color values of each pixel) for each pixel in the image.

And the second light splitting module 140 is configured to split the third light beam into a plurality of fourth light beams, and project each of the fourth light beams to the corresponding TOF imaging module 150.

The number of the fourth light beams is consistent with that of the TOF imaging module 150.

Specifically, the second beam splitting module 140 receives the laser beam (i.e., the third light beam) stripped from the first beam splitting module 120, and splits the second light beam into two or more fourth light beams, so that each third light beam serves as an incident light for the TOF imaging module 150.

It is understood that the second light splitting module 140 is used for splitting the third light beam, and the working method of the second light splitting module 140 is not particularly limited in the embodiment of the present invention.

For example, the second beam splitting module 140 may include a common beam splitter prism, and the third light beam passing through the first beam splitting module 120 is uniformly split into two equal parts of the fourth light beam or multiple equal parts of the fourth light beam by the beam splitter prism in the second beam splitting module 140 and is projected to the TOF imaging module 150. The light intensity of each of the fourth light beams is equal, and the number of the fourth light beams corresponds to the number of the TOF imaging modules 150.

And the TOF imaging module 150 is configured to obtain depth information of the target object according to the incident fourth light beam.

In Time of flight (TOF) 3D imaging, a light pulse is continuously transmitted to a target, and then a sensor receives light returning from the object, and the round trip Time of the light pulse is detected to obtain a target object distance.

The Depth information is information of each pixel value contained in a Depth Map (Depth Map). Where a Depth Map (Depth Map) is an image or image channel containing information about the distance of the surface of the scene object from the viewpoint, similar to a grayscale image, but different from the grayscale Map is that each pixel value of the Depth Map is the actual distance of the sensor from the object.

Specifically, the TOF imaging module 150 receives the fourth light beam split by the second light splitting module 140, and the light signal can be converted into a Depth digital signal through the TOF imaging module 150, and the Depth information of the target object can be obtained accordingly.

The Depth digital signal is pixel information in a Depth Map (Depth Map), each pixel value is similar to a gray image, namely, only one channel exists, and the gray level of each pixel value represents the brightness difference of a pixel point. However, unlike the grayscale map, each pixel value in the Depth digital signal is the actual distance of the TOF sensor from the object, i.e. the smaller the Depth value (i.e. closer to 0), the closer the TOF sensor is to the object, and vice versa.

It can be understood that, in general, the RGB image and the Depth Map (Depth Map) are registered, i.e. there is a one-to-one correspondence between pixel points.

And the time sequence control module 160 is connected with the RGB imaging module 130 and the TOF imaging module 150, and is configured to control each TOF imaging module to alternately output depth information of the target object according to a target time sequence.

It should be noted that the timing control module 160 is respectively connected to the RGB imaging module 130 and the TOF imaging module 150.

For the connection with the RGB imager 130, it is used to receive the color information generated by the RGB imager 130.

For the connection with the TOF imaging module 150, on one hand, an instruction is sent according to a target time sequence to control the TOF imaging module 150 to perform imaging, and on the other hand, depth information generated by the corresponding TOF imaging module 150 is received at the target time sequence.

The target timing is the time sequence in which each TOF imaging module 150 makes its own measurement over the same phase-delayed time window.

Specifically, the timing control module 160 may send an instruction to the corresponding TOF imaging module 150 at a target timing, so that each TOF imaging module 150 alternately works according to the target timing within the measurement period to obtain corresponding depth information.

Preferably, the phase delay is obtained according to the measurement period and the number of the TOF imaging modules 150, that is, the period is divided equally according to the number of the TOF imaging modules 150, and the value of the phase delay is not particularly limited in the embodiment of the present invention.

The timing control module 160 is further configured to obtain an RGBD image based on the color information of the target object and each set of depth information.

It should be noted that, since the RGB image and the Depth Map (Depth Map) are registered, the coordinates of the RGB image and the Depth Map have a one-to-one mapping relationship.

RGBD image, essentially two images. One of the three-channel color images is a common RGB three-channel color image used for reflecting color information of an object in a real environment. The other is a single-channel depth map for reflecting the distance information of each point of the object surface in the real environment, thereby reflecting the geometry of the object surface.

Specifically, in the target time sequence, the time sequence control module 160 fuses the color information into the depth information by using the coordinate mapping relationship between the color information acquired by the RGB imaging module 130 and the depth information acquired by the TOF imaging module 150, until the depth information acquired by the TOF imaging modules 150 in a fixed time period is completely fused with the color information acquired by the RGB imaging module 130, so as to generate a plurality of sets of color depth information, i.e., a plurality of RGBD images.

The number of RGBD images output in each period is the same as the number of TOF imaging modules 150, which is not specifically limited in the embodiment of the present invention.

The following illustrates a specific embodiment of alternately generating depth information in an RGBD stereo camera.

For example, the RGBD stereo camera may include three TOF imaging modules 150, and the phase delay between the three TOF imaging modules is 360 °/3 — 120 °, i.e. the sampling time T is four times in one period of the first TOF imaging module 150₁、T₂、T₃、T₄Respectively compared with the corresponding sampling time T of the second imaging module₁′、T₂′、T₃′、T₄' delayed by a time interval of 120 deg. phase. Similarly, the sampling time of the second TOF imaging module 150 is delayed by a time interval of 120 ° phase from the sampling time of the third TOF imaging module 150 in one period.

At a first time t₁The timing control module 160 drives the first TOF imaging module 150 to work, and fuses the acquired depth information and the color information acquired by the RGB imaging module 130 into a corresponding first RGBD image. At a second time t₂＝t₁+ Δ t, the timing control module 160 drives the second TOF imaging module 150 to work, and fuses the acquired depth information and the color information acquired by the RGB imaging module 130 into a corresponding second RGBD image, similarly at the third time t₃＝t₂+ Δ t will synthesize the corresponding third RGBD image. Where Δ t is the time interval in which the sampling period is delayed by 120 ° phase.

The measuring optical path of a general RGBD stereo camera only includes one TOF imaging module, and only one depth map can be output for RGBD image fusion after sampling within a period time under a certain resolution, so that when an image with a high resolution is obtained, a few images can be obtained within a unit time, that is, a frame rate is low. When acquiring an image with a high frame rate, in order to acquire more images per unit time, it is necessary to shorten the cycle time for each image transmission, that is, to reduce the image resolution.

The embodiment of the invention is based on that a plurality of TOF imaging modules are adopted to alternately measure the same target in a target time sequence within a period time, time domain supplementary fusion is carried out on a plurality of groups of depth information, the depth information with the increased frame rate is obtained, each group of depth information and color information are sequentially fused, RGBD images with the number being consistent with that of the TOF imaging modules are obtained within the period time, and RGBD images with high resolution and high frame rate can be simultaneously obtained.

Fig. 2 is a second schematic structural diagram of the RGBD stereo camera provided in the present invention. As shown in fig. 2, on the basis of any of the above embodiments, the first light splitting module 230 includes a dichroic mirror.

Note that the dichroic mirror transmits light of a certain wavelength almost completely, and reflects light of other wavelengths almost completely.

Specifically, the first light splitting module 230 is configured as a dichroic mirror, receives the first light beam via the lens 220, and can perform spectral splitting on the first light beam to obtain a second light beam and a third light beam.

The first light beam can penetrate through the dichroic mirror to become the second light beam with extremely low attenuation due to high transmissivity of the dichroic mirror in a visible light waveband, the dichroic mirror is high in reflectivity of a laser light waveband, and the first light beam can be reflected by the dichroic mirror with extremely low attenuation to obtain the third light beam.

The embodiment of the invention performs spectral splitting on the received light beam based on the dichroic mirror, thereby reducing the laser loss caused in the splitting process.

On the basis of any of the above embodiments, the number of the TOF imaging modules 260 is two.

Specifically, the RGBD stereo camera is provided with two TOF imaging modules 260, each TOF imaging module 260 receives the fourth light beam split by the second splitting module 250 and performs imaging, and two sets of depth information of the target object can be obtained.

The embodiment of the invention is based on two TOF imaging modules, and can output two sets of depth information. The contents of the two are approximately consistent, the measurement periods are the same, and the measurement time sequences of the two are different by a half period, so that when the two alternately output the depth information, two groups of depth information can be output in the original measurement period. The measuring light path that sets up two TOF imaging module and constitute is comparatively simple, can excessively avoid increasing the volume of camera, can also improve the frame rate.

On the basis of any of the above embodiments, the TOF imaging module 260 includes: a TOF imaging lens 261 and a TOF sensing chip 262.

Specifically, each TOF imaging module 260 in the RGBD stereo camera is composed of a TOF imaging lens 261 and a TOF sensing chip 262, and the TOF imaging lens 261 and the TOF sensing chip 262 are precisely adjusted, so that depth information of a target object can be clearly obtained on the TOF sensing chip 262.

The TOF imaging lens 261 is used for transforming (modulating) the fourth light beam corresponding to any TOF imaging module 260.

The TOF sensing chip is used for imaging the fourth light beam modulated by the TOF imaging lens 261 on the TOF sensing chip 262 and obtaining depth information of the target object.

The working principle is to convert images into digital information by utilizing a photosensitive diode (photodiode) to perform optical and electrical conversion, wherein different types of sensor chips are different in digital signal transmission modes.

For example, the TOF sensing chip 262 may be a charge coupled device (CCD sensing chip) or a Complementary Metal Oxide Semiconductor (CMOS sensing chip), and the type of the TOF sensing chip is not particularly limited in the embodiments of the present invention.

Accordingly, the RGB imaging module 240 also includes: an RGB imaging lens 241 and an RGB sensing chip 242.

Specifically, the RGB imaging module 240 in the RGBD stereo camera is composed of an RGB imaging lens 241 and an RGB sensing chip 242, and the RGB imaging lens 241 and the RGB sensing chip 242 are precisely adjusted so that the RGB sensing chip 242 can clearly obtain color information of a target object.

The RGB imaging lens 241 is used for transforming (modulating) the second light beam corresponding to the RGB imaging module 240.

The RGB sensor chip is configured to image the second light beam modulated by the RGB imaging lens 241 on the RGB sensor chip 242, and obtain color information of the target object.

The RGB sensor chip 242 may be a charge coupled device (CCD sensor chip) or a Complementary Metal Oxide Semiconductor (CMOS sensor chip), and the embodiment of the present invention does not specifically limit the type of the RGB sensor chip.

According to the embodiment of the invention, after each fourth light beam is incident to the TOF imaging module, the TOF imaging lens and the TOF sensing chip are precisely adjusted, so that clear imaging can be performed on the TOF sensing chip, and depth information used for representing three-dimensional information of the target object is obtained. And incident light is modulated and imaged by using the TOF imaging lens and the TOF sensing chip, so that the imaging resolution can be improved.

On the basis of any of the above embodiments, the second light splitting module 250 includes a plurality of light splitters.

Specifically, a plurality of beam splitters may be disposed in the second beam splitting module 250, so that the third light beam passing through the second beam splitting module 250 can be equally split into fourth light beams in accordance with the number of TOF imaging modules 260, and the number of beam splitters is not particularly limited in the embodiment of the present invention.

Preferably, a beam splitter may be disposed in the second light splitting module 250, the third light beam is split into two fourth light beams at a ratio of 1:1 by the beam splitter, and each fourth light beam is imaged on the TOF sensing chip 262 after passing through a respective TOF imaging lens 261.

The embodiment of the invention can use a plurality of spectroscopes to perform light splitting based on the split third light beam to obtain the fourth light beam with the same number as the TOF imaging modules. Furthermore, the TOF imaging modules which can be used for more times in the measuring period alternately output depth information, and the frame rate can be improved on the premise that the light path design is simple.

On the basis of any of the above embodiments, the target timing is a time sequence with equal intervals.

Specifically, the target timing sequence is obtained according to the measurement period and the number of TOF imaging modules 150, i.e. the period is divided equally according to the number of TOF imaging modules 150.

The timing control module 160 may send instructions to the corresponding TOF imaging modules 150 at equal-interval target timings, so that each TOF imaging module 150 alternately works according to the target timings within the measurement period to obtain corresponding depth information.

The embodiment of the invention enables a plurality of TOF imaging modules to work alternately based on the time sequence of equal intervals, and can improve the imaging frame rate.

On the basis of any of the above embodiments, the RGBD stereo camera further includes a laser emission module 210.

Specifically, the RGBD stereo camera may be provided with a laser light source externally, or the laser emitting module 210 may be provided inside the RGBD stereo camera, and the setting position of the laser light source is not specifically limited in the embodiment of the present invention.

Preferably, the laser emission module 210 is arranged inside the RGBD stereo camera, and can acquire the depth information of the target object according to the emitted laser light source and the TOF imaging module 260.

And a laser emitting module 210 for generating a fifth light beam and projecting the fifth light beam to the target object.

Optionally, the fifth beam is a sinusoidally modulated continuous laser beam.

Specifically, the laser emission module 210 projects the fifth light beam to the target object and reflects the fifth light beam to generate the first light beam, which is received and transmitted to the first light splitting module 230 via the lens 220.

Wherein the first beam includes a laser beam and a visible light beam reflected by the target object.

According to the embodiment of the invention, the laser emitting module is arranged in the RGBD stereo camera, and the laser emitted by the module can be approximately vertically projected on the surface of the target object, so that the waste of a light path can be avoided.

Fig. 3 is a schematic flowchart of an imaging method of an RGBD stereo camera according to an embodiment of the present invention. As shown in fig. 3, based on the contents of any one of the above embodiments, the method for imaging an RGBD stereo camera includes: step S301, the first beam splitting module splits the first light beam incident through the lens into a second light beam and a third light beam, so that the second beam splitting module splits the third light beam into a plurality of fourth light beams, and projects each of the fourth light beams to the corresponding TOF imaging module.

It should be noted that, in the optical path design scheme of the RGBD stereo camera, the modulated laser is projected to the target object and generates reflected light as a first light beam, and the first light beam is used as incident light of the lens of the RGBD stereo camera.

The first light beam comprises two parts, wherein one part is reflected light formed by visible light on the surface of the target object, and the other part is reflected light formed by modulated laser on the surface of the target object.

Specifically, the first light beam modulated by the RGBD stereo camera lens is spectrally split as a first splitting module, from which a second light beam having a visible light band and a third light beam having a modulated laser light band are stripped.

The second light beam is used as incident light of the RGB imaging module and is imaged, the third light beam is divided into a plurality of fourth light beams through the second light splitting module, and the fourth light beams are used as incident light of the TOF imaging module and are imaged.

Preferably, the sine-modulated continuous laser beam is emitted to the surface of the target object and reflected, and the reflected laser beam and the visible light are received through the lens of the RGBD stereo camera and transmitted to the dichroic mirror. The reflected visible light and the reflected laser light are spectrally dispersed using a dichroic mirror.

And for the separated visible light beams, measuring the visible light beams by using an RGB imaging module.

And for the separated laser beams, common beam splitters are used for splitting to obtain two laser beams containing three-dimensional information of the target object, and two TOF imaging modules are used for measuring the two laser beams respectively.

Step S302, the RGB imaging module acquires color information of the target object based on the incident second light beam, and the time sequence control module controls each TOF imaging module to output depth information of the target object based on the incident fourth light beam alternately in a target time sequence.

The number of the fourth light beams is consistent with the number of the TOF imaging module and the depth information.

Note that the target object is a photographic subject of the RGBD stereo camera.

The target timing is the time sequence in which each TOF imaging module makes its own measurement over the same phase-delayed time window.

Specifically, the RGB imaging module can directly image the incident second light beam, and can acquire color information of the target object. Meanwhile, the time sequence control module sends instructions to each TOF imaging module to control each TOF imaging module to image according to the fourth light beam corresponding to the target time sequence, and depth information of the target object can be acquired

Preferably, the second light beam split by the dichroic mirror is modulated by the RGB imaging module and then imaged, and color information of the target object is output. And measuring two fourth light beams by using two TOF imaging modules respectively, and outputting the depth information of the same target object. In the process of outputting the depth information, the time sequence control module drives the two TOF imaging modules to work in a time-sharing mode, and the obtained depth information is subjected to complementary fusion in a time domain.

Fig. 4 is a schematic diagram of the relationship between the laser light received by the TOF imaging module and the emitted laser light. As shown in fig. 4, a specific embodiment of temporal fusion of depth information is illustrated below.

For example, the depth information of the TOF sensor chip pixels in each TOF imaging module may be characterized by the phase delay between the corresponding emitted laser light and received laser light:

where d is depth information for each pixel, f_mIs the frequency of the sinusoidal laser beam, c is the speed of light, I₁、I₂、I₃、I₄For the pixel at T₁、T₂、T₃、T₄Four integrated electrical signals over time windows sequentially differing by a 90 phase delay.

Time sequence control module controls measurement between two TOF sensing chips in RGBD stereo cameraSequentially, the two are respectively measured on time windows with phase delay of 180 degrees, namely four sampling times T in one period of a TOF sensing chip₁、T₂、T₃、T₄Respectively than the corresponding sampling time T of another TOF sensing chip₁′、T₂′、T₃′、T₄' 180 phase delayed time interval. To this end, two sets of depth information about the same target object may be obtained by two TOF imaging modules.

It will be appreciated that before step S303, the two sets of depth information need to be calibrated and scaled so that the two sets of depth information are uniform for the same target object.

And target images obtained by the two TOF depth sensing chips are approximately consistent by precisely adjusting the two TOF imaging lenses and the corresponding TOF depth sensing chips.

And further acquiring the corresponding relation of pixel coordinates under a two-dimensional pixel coordinate system between the two chips through calibration operation, and unifying the two-dimensional pixel coordinate system.

Meanwhile, a depth calibration method is adopted to establish a retrieval relationship between depth information obtained by the measurement of the two sensing chips and the real depth, and the depth information obtained between the two sensing chips is unified according to the retrieval relationship. Finally, the depth measurement results of the two TOF depth sensing chips for the same target are approximately consistent.

The two TOF sensing chips output the same depth information, the output depth information is approximately consistent, and the measurement time sequences of the two TOF sensing chips differ by a half period, so that when the two TOF sensing chips output the depth information alternately, the measurement frame rate can be improved to be one time of that of a single TOF sensing chip, and the two TOF sensing chips use the same set of laser emission module without additionally introducing a new laser light source.

Step S303, the time sequence control module acquires an RGBD image based on the color information and each depth information of the target object.

It should be noted that any TOF sensing chip outputs corresponding depth information according to a target time sequence, and sends the depth information to the time sequence control module. Similarly, the color information output by the RGB sensor chip is also sent to the timing control module.

Specifically, after receiving each corresponding depth information output by the corresponding TOF sensor chip in the target time sequence, the time sequence control module is fused with the color information output by the RGB sensor chip, so as to obtain the RGBD image corresponding to the target time sequence.

It can be understood that, during the process of fusing the depth information and the color information, calibration and calibration are also required for the depth information and the color information, so that the information output for the same target object has uniformity.

On the basis of any one of the above embodiments, acquiring an RGBD image based on color information and depth information of a target object includes: and carrying out image fusion based on the depth information, the color information and the target mapping relation of the target object to obtain an RGBD image.

Note that, since the coordinates of the RGB image and the Depth Map (Depth Map) have a one-to-one mapping relationship, the Depth information and the color information are fused according to the mapping relationship.

Specifically, after receiving the depth information corresponding to each TOF sensor chip output in a target time sequence, the time sequence control module performs splicing and fusion on the depth information and the color information having the same pixel coordinate point according to a target mapping relationship, so as to obtain RGBD information in the corresponding pixel coordinate in the RGBD image.

It can be understood that calibration and calibration are also required in the fusion of the depth information and the color information, and this process is not specifically limited in the embodiment of the present invention.

Preferably, through calibration operation, the pixel coordinate corresponding relation of the target image between the RGB sensing chip and any TOF sensing chip is obtained, and then binocular calibration is completed. Therefore, a rotation matrix and a translation matrix between the two sensing chip pixel coordinate systems can be obtained.

By utilizing the rotation matrix and the translation matrix, the coordinate mapping of corresponding pixels of the same target pixel point between the RGB sensing chip and the TOF sensing chip can be completed, and the color information of the target pixel point on the RGB sensing chip is mapped and fused to the depth information of the corresponding pixel point on the TOF depth sensing chip.

According to the embodiment of the invention, the RGBD image fusion is carried out on the basis of completing the calibration and calibration of the image of any depth information and color information based on the target mapping relation, so that the quality of the RGBD image can be improved.

Fig. 5 illustrates a physical structure diagram of an electronic device, which may include, as shown in fig. 5: a processor (processor)510, a communication interface (communication interface)520, a memory (memory)530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other via the communication bus 540. The processor 510 may call logic instructions in the memory 530 to perform a method of imaging an RGBD stereo camera, the method including: the first light splitting module splits the first light beam incident through the lens into a second light beam and a third light beam, so that the second light splitting module splits the third light beam into a plurality of fourth light beams, and each fourth light beam is projected to the corresponding TOF imaging module respectively; the RGB imaging module acquires color information of a target object based on the incident second light beam, and the time sequence control module controls each TOF imaging module to output depth information of the target object based on the incident fourth light beam alternately in a target time sequence; the time sequence control module acquires an RGBD image based on the color information and each depth information of the target object; the first light beam is a light beam reflected by the target object and incident through the lens, the second light beam is visible light in the first light beam, the third light beam is laser in the first light beam, and the number of the fourth light beams is consistent with the number of the TOF imaging module and the depth information.

Furthermore, the logic instructions in the memory 530 may be implemented in the form of software functional units and stored in a computer readable storage medium when the software functional units are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

In another aspect, the present invention also provides a computer program product including a computer program stored on a non-transitory computer readable storage medium, the computer program including program instructions, which when executed by a computer, enable the computer to execute the method for imaging an RGBD stereo camera provided by the above methods, the method including: the first light splitting module splits the first light beam incident through the lens into a second light beam and a third light beam, so that the second light splitting module splits the third light beam into a plurality of fourth light beams, and each fourth light beam is projected to the corresponding TOF imaging module respectively; the RGB imaging module acquires color information of a target object based on the incident second light beam, and the time sequence control module controls each TOF imaging module to output depth information of the target object based on the incident fourth light beam alternately in a target time sequence; the time sequence control module acquires an RGBD image based on the color information and each depth information of the target object; the first light beam is a light beam reflected by the target object and incident through the lens, the second light beam is visible light in the first light beam, the third light beam is laser in the first light beam, and the number of the fourth light beams is consistent with the number of the TOF imaging module and the depth information.

In yet another aspect, the present invention also provides a non-transitory computer-readable storage medium having stored thereon a computer program, which when executed by a processor, is implemented to perform the method of imaging the RGBD stereo camera provided in each of the above, the method including: the first light splitting module splits the first light beam incident through the lens into a second light beam and a third light beam, so that the second light splitting module splits the third light beam into a plurality of fourth light beams, and each fourth light beam is projected to the corresponding TOF imaging module respectively; the RGB imaging module acquires color information of a target object based on the incident second light beam, and the time sequence control module controls each TOF imaging module to output depth information of the target object based on the incident fourth light beam alternately in a target time sequence; the time sequence control module acquires an RGBD image based on the color information and each depth information of the target object; the first light beam is a light beam reflected by the target object and incident through the lens, the second light beam is visible light in the first light beam, the third light beam is laser in the first light beam, and the number of the fourth light beams is consistent with the number of the TOF imaging module and the depth information.

The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.

Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.

Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims

1. An RGBD stereo camera, comprising: the system comprises a lens, a first light splitting module, an RGB imaging module, a second light splitting module, a plurality of TOF imaging modules and a time sequence control module;

2. The RGBD stereo camera of claim 1, wherein the first light splitting module comprises a dichroic mirror.

3. The RGBD stereo camera according to claim 1, wherein the number of TOF imaging modules is two, three or four.

4. The RGBD stereo camera of claim 1, wherein the TOF imaging module comprises:

TOF imaging lens and TOF sensing chip.

5. The RGBD stereo camera according to claim 1, wherein the second beam splitting module includes a plurality of beam splitters.

6. The RGBD stereo camera of claim 1, wherein the target timing is an equally spaced temporal sequence.

7. The RGBD stereo camera according to any one of claims 1 to 6, further comprising a laser emitting module;

8. The RGBD stereo camera imaging method according to any one of claims 1 to 7, comprising:

9. The method of imaging an RGBD stereo camera according to claim 8, wherein acquiring the RGBD image based on the color information and the depth information of the target object includes:

10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the imaging method as claimed in claim 8 or 9 are implemented when the processor executes the program.