CN112509023B - Multi-source camera system and RGBD registration method - Google Patents
Multi-source camera system and RGBD registration method Download PDFInfo
- Publication number
- CN112509023B CN112509023B CN202011461732.2A CN202011461732A CN112509023B CN 112509023 B CN112509023 B CN 112509023B CN 202011461732 A CN202011461732 A CN 202011461732A CN 112509023 B CN112509023 B CN 112509023B
- Authority
- CN
- China
- Prior art keywords
- depth image
- rgb
- acquisition module
- camera system
- image acquisition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/35—Determination of transform parameters for the alignment of images, i.e. image registration using statistical methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20092—Interactive image processing based on input by user
- G06T2207/20104—Interactive definition of region of interest [ROI]
Abstract
The invention relates to a multi-source camera system and an RGBD (red, green and blue) registration method, wherein the system comprises an RGB (red, green and blue) image acquisition module, a depth image acquisition module and an upper computer, wherein the depth image acquisition module is connected with the upper computer; the method comprises the steps of collecting RGB image data and depth image data of a plurality of groups of calibration plates; obtaining internal and external parameters of the RGB camera; initializing the internal and external parameters of the obtained RGB image as input; converting the depth image into a 2D plane, selecting an ROI (region of interest) to obtain a first external reference matrix; optimizing the first external parameter matrix to obtain an external parameter matrix of a second stage; and inputting an RGB image and a depth image which need to be registered, and obtaining a rendered RGBD image by using a second external reference matrix. The invention has the advantages that: the use is flexible; the depth image acquisition module has a simple structure and low cost, and can quickly acquire a depth image; the registration speed of the RGB image and the depth image is high, and the accuracy is high.
Description
Technical Field
The invention relates to the field of image acquisition, in particular to a multi-source camera system and an RGBD (red, green and blue) registration method.
Background
With the wide application of emerging technologies such as robot navigation, automobile automatic driving, virtual reality and the like, the advantage of depth information imaging is increasingly prominent. Conventional two-dimensional imaging maps a stereoscopic scene to a two-dimensional plane through perspective projection transformation, and such low-dimensional mapping causes depth information of the scene to be lost. The traditional infrared depth image can cause color information of a scene to be lost, and no matter the single traditional two-dimensional imaging image or the single infrared depth image cannot meet the requirements of the fields of three-dimensional reconstruction, augmented reality, virtual reality, human-computer interaction and the like on RGB information and depth information, so that RGBD data combining depth information and RGB image information are indispensable, and the problem of accurate calibration between a depth sensor and an RGB sensor needs to be solved firstly when the RGBD data is acquired.
Disclosure of Invention
The invention mainly solves the problems of complex structure and low registration precision of depth information and RGB information of the existing RGBD imaging device, and provides a multisource camera system and an RGBD registration method which are flexible to use, simple in structure and high in registration precision of depth information and RGB information.
The technical scheme adopted by the invention for solving the technical problem is that the multi-source camera system comprises an RGB image acquisition module, a depth image acquisition module and an upper computer, wherein the RGB image acquisition module comprises an RGB sensor unit and a first data transmission unit which are connected, the depth image acquisition module comprises a TOF sensor unit, an active illumination unit, a processor and a second data transmission unit, the processor is respectively connected with the TOF sensor unit, the active illumination unit and the second data transmission unit, the depth image acquisition module is connected with the upper computer through the first data transmission unit, and the RGB image acquisition module is connected with the upper computer through the first data transmission unit.
The RGB image acquisition module is used for acquiring RGB images; the depth image acquisition module adopts an active depth measurement method and is used for acquiring depth phase data; the upper computer is used for sending control instructions to each acquisition module and completing depth image resolving, image registration calculation and data storage.
As a preferable scheme of the above scheme, the active illumination unit includes a TOF lens, an illumination board, a sensor board and a main control board, the illumination board, the sensor board and the main control board communicate with each other, the TOF lens is disposed on the illumination board, the TOF sensor unit is disposed on the sensor board, and the processor is disposed on the main control board.
As a preferable scheme of the above scheme, the TOF lens is provided with an infrared narrow band-pass filtering optical sheet. The active illumination unit emits a beam of modulated infrared light, after diffuse reflection is carried out on the surface of a target, reflected light passes through a lens of an integrated infrared narrow band-pass filter with a fixed focal length and is finally captured by the TOF sensor unit, and the distance between the target and the depth camera is calculated by obtaining the time difference or the phase difference between the emitted light and the reflected light.
As a preferable scheme of the above scheme, the active lighting unit includes a plurality of infrared light sources, the infrared light sources are disposed on the lighting plate, and a light emitting direction of the infrared light sources is the same as a direction of the TOF lens.
Correspondingly, the invention also provides a multi-source camera RGBD registration method, which is used for the multi-source camera system and comprises the following steps:
s1: simultaneously collecting RGB image data and depth image data of a plurality of groups of calibration plates;
s2: calibrating by using a Zhang-Zhengyou to obtain internal and external parameters of the RGB camera;
s3: initializing the obtained internal and external parameters of the RGB camera as input;
s4: converting the depth image into a 2D plane, selecting an ROI (region of interest) to obtain a first external reference matrix;
s5: optimizing the first external parameter matrix to obtain an external parameter matrix of a second stage;
s6: and inputting an RGB image and a depth image which need to be registered, and obtaining a rendered RGBD image by using a second external parameter matrix.
As a preferable mode of the above, the step S4 includes the steps of:
s41: converting the depth value of the input depth image into a given set of colors to display a two-dimensional plane map with colors;
s42: selecting a depth image distance range needing to be registered, and filtering out pixel points beyond the range;
s43: selecting a region of interest by using a marking tool, and calculating a selected error value;
s44: and calculating all the calibrated interested areas to obtain an internal and external parameter matrix of the depth image, namely a first external parameter matrix.
As a preferable solution of the foregoing solution, in step S5, the first external reference matrix is optimized by using robust total least squares to obtain a second external reference matrix.
As a preferable scheme of the above scheme, the depth image is acquired by the following steps:
s01: the active illumination unit emits continuous wave modulated infrared light;
s02: the TOF sensor unit receives light beams diffusely reflected by the surface of the target through the TOF lens and the infrared narrow band pass filter;
s03: the TOF sensor unit obtains raw depth phase data of the scene by calculating the time or phase difference of the emitted and reflected light.
The invention has the advantages that: the RGB image acquisition module and the depth image acquisition module are not limited, so that a user can select RGB cameras with different resolutions and depth cameras with different detection distances according to a specific application scene, and the use is flexible; the depth image acquisition module has simple structure and low cost, and can quickly acquire a depth image; the registration speed of the RGB image and the depth image is high, and the accuracy is high.
Drawings
Fig. 1 is a block diagram of a multi-source camera system in an embodiment.
Fig. 2 is a block diagram of a depth image acquisition module in an embodiment.
Fig. 3 is a schematic flow chart of the multi-source camera RGBD registration method in the embodiment.
Fig. 4 is a schematic flowchart of a process of acquiring a depth image according to an embodiment.
FIG. 5 is a schematic flowchart illustrating a process of obtaining the first extrinsic matrix according to the embodiment.
1-RGB image acquisition module 2-depth image acquisition module 3-upper computer 11-RGB sensor unit 12-first data transmission unit 21-TOF sensor unit 22-active illumination unit 23-processor 24-second data transmission unit 601-illumination plate 602-sensor plate 603-main control plate 701-infrared VSCEL light source 606-TOF lens 607-infrared narrow bandpass filter.
Detailed Description
The technical solution of the present invention is further described below by way of examples with reference to the accompanying drawings.
The embodiment is as follows:
the multi-source camera system comprises an RGB image acquisition module 1, a depth image acquisition module 2 and an upper computer 3, wherein the RGB image acquisition module comprises an RGB sensor unit 11 and a first data transmission unit 12 which are connected, the depth image acquisition module comprises a TOF sensor unit 21, an active illumination unit 22, a processor 23 and a second data transmission unit 24, the processor is respectively connected with the TOF sensor unit, the active illumination unit and the second data transmission unit, the depth image acquisition module is connected with the upper computer through the first data transmission unit, and the RGB image acquisition module is connected with the upper computer through the first data transmission unit.
The RGB image acquisition module adopts an industrial RGB image sensor of Basler company, the model is AcA1300, the pixel size is 4.8 mu m multiplied by 4.8 mu m, the size of a photosensitive chip is 6.1mm multiplied by 4.9mm, the RGB image acquisition module is matched with a 5mm lens 605 of computer company, data transmission passes through a USB3.0 interface, and the frame rate is 200fps.
The depth image acquisition module is shown in fig. 2, and adopts a three-layer PCB plate design, which is respectively a lighting plate 601, a sensor plate 602, a main control plate 603, a TOF lens and a processor, wherein the TOF sensor unit is arranged on the sensor plate, the processor is arranged on the main control plate, the plates are transmitted through an MIPI data format, the active lighting unit comprises 4 infrared VSCEL light sources 701 arranged on the lighting plate 601 and in 850nm waveband, the light emitting direction of the infrared VSCEL light sources is the same as the direction of the TOF lens, the TOF lens 606 with fixed focal length integrates the infrared narrow bandpass filter 607 in 850nm waveband, and the TOF lens 606 completely covers the photosensitive surface of the TOF sensor unit.
Correspondingly, the embodiment also provides a multisource camera RGBD registration method, as shown in fig. 3, including the following steps:
s1: simultaneously collecting RGB image data and depth image data of a plurality of groups of calibration plates, wherein the calibration plates are standard checkerboard plates, and when collecting depth images, as shown in figure 4, a depth image collecting module executes the following steps:
s01: the active illumination unit emits continuous wave modulated infrared light;
s02: the TOF sensor unit receives the light beam diffusely reflected by the surface of the target through the TOF lens and the infrared narrow band-pass filter;
s03: the TOF sensor unit obtains the scene raw depth phase data by calculating the time or phase difference of the emitted and reflected light.
S2: calibrating by using a Zhang-Zhengyou to obtain internal and external parameters of the RGB camera;
s3: initializing the obtained internal and external parameters of the RGB camera as input;
s4: converting the depth image into a 2D plane, selecting an ROI (region of interest) to obtain a first external reference matrix; the specific steps are shown in fig. 5, and include:
s41: converting the depth value of the input depth image into a given set of colors to display a two-dimensional plane map with colors;
s42: selecting a distance range of the depth image to be registered, and filtering out pixel points beyond the range;
s43: selecting a region of interest by using a marking tool, and calculating a selected error value;
s44: and calculating all the calibrated interested areas to obtain an internal and external parameter matrix of the depth image, namely a first external parameter matrix.
S5: optimizing the first external parameter matrix to obtain a second external parameter matrix, wherein the first external parameter matrix mainly solves basic rotation and translation matrixes according to internal and external parameters of RGB and internal and external parameters of the depth image, and for each selected area in the depth image, the first external parameter matrix is optimized by utilizing a robust total least square estimation to obtain the second external parameter matrix;
s6: inputting an RGB image and a depth image which need to be registered, and obtaining a rendered RGBD image by using a second external parameter matrix, wherein the method specifically comprises the following steps:
s61: the method comprises the steps of keeping the space relation between an RGB image acquisition module and a depth image acquisition module stable, and acquiring an RGB image and a depth image of a scene;
s62: inputting the collected data into a registration code, loading a second external parameter matrix, and performing registration;
s63: the rendered depth image, i.e., the RGBD image, is output in a wrl file format.
The specific embodiments described herein are merely illustrative of the spirit of the invention. Various modifications or additions may be made to the described embodiments, or alternatives may be employed, by those skilled in the art, without departing from the spirit or ambit of the invention as defined in the appended claims.
Claims (7)
1. A multi-source camera system is characterized in that: the system comprises an RGB image acquisition module, a depth image acquisition module and an upper computer, wherein the RGB image acquisition module comprises an RGB sensor unit and a first data transmission unit which are connected, the depth image acquisition module comprises a TOF sensor unit, an active illumination unit, a processor and a second data transmission unit, the processor is respectively connected with the TOF sensor unit, the active illumination unit and the second data transmission unit, the depth image acquisition module is connected with the upper computer through the second data transmission unit, and the RGB image acquisition module is connected with the upper computer through the first data transmission unit;
the multi-source camera system can be used for executing the following multi-source camera RGBD registration method of the multi-source camera system, and comprises the following steps:
s1: simultaneously collecting RGB image data and depth image data of a plurality of groups of calibration plates;
s2: calibrating by using a Zhang-Zhengyou to obtain internal and external parameters of the RGB camera;
s3: initializing the obtained internal and external parameters of the RGB camera as input;
s4: converting the depth image into a 2D plane, selecting an ROI (region of interest) to obtain a first external reference matrix;
s5: optimizing the first external parameter matrix to obtain a second external parameter matrix;
s6: and inputting an RGB image and a depth image which need to be registered, and obtaining a rendered RGBD image by using a second external reference matrix.
2. The multi-source camera system of claim 1, wherein: the active lighting unit comprises a TOF lens, a lighting plate, a sensor plate and a main control plate, wherein the lighting plate, the sensor plate and the main control plate are communicated with each other, the TOF lens is arranged on the lighting plate, the TOF sensor unit is arranged on the sensor plate, and the processor is arranged on the main control plate.
3. A multi-source camera system as claimed in claim 2, wherein: and an infrared narrow band-pass filtering polished section is arranged on the TOF lens.
4. The multi-source camera system of claim 2, wherein: the active lighting unit comprises a plurality of infrared light sources, the infrared light sources are arranged on the lighting plate, and the light emitting directions of the infrared light sources are the same as the direction of the TOF lens.
5. The multi-source camera system of claim 1, wherein: the step S4 includes the steps of:
s41: converting the depth value of the input depth image into a given set of colors to display a two-dimensional plane map with colors;
s42: selecting a distance range of the depth image to be registered, and filtering out pixel points beyond the range;
s43: selecting a region of interest by using a marking tool, and calculating a selected error value;
s44: and calculating all the calibrated interested areas to obtain an internal and external parameter matrix of the depth image, namely a first external parameter matrix.
6. The multi-source camera system of claim 5, wherein: and in the step S5, the first external parameter matrix is optimized by using robust total least squares to obtain a second external parameter matrix.
7. A multi-source camera system as claimed in claim 1, wherein: the depth image is acquired by the following steps:
s01: the active illumination unit emits continuous wave modulated infrared light;
s02: the TOF sensor unit receives light beams diffusely reflected by the surface of a target through a TOF lens and an infrared narrow band-pass filter;
s03: the TOF sensor unit obtains raw depth phase data of the scene by calculating the time or phase difference of the emitted and reflected light.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011461732.2A CN112509023B (en) | 2020-12-11 | 2020-12-11 | Multi-source camera system and RGBD registration method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011461732.2A CN112509023B (en) | 2020-12-11 | 2020-12-11 | Multi-source camera system and RGBD registration method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112509023A CN112509023A (en) | 2021-03-16 |
| CN112509023B true CN112509023B (en) | 2022-11-22 |
Family
ID=74972543
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011461732.2A Active CN112509023B (en) | 2020-12-11 | 2020-12-11 | Multi-source camera system and RGBD registration method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112509023B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106682663A (en) * | 2015-11-11 | 2017-05-17 | 天津中科智能识别产业技术研究院有限公司 | Multi-modal biological feature identifying system and method for mobile platform |
| CN107507235A (en) * | 2017-08-31 | 2017-12-22 | 山东大学 | A kind of method for registering of coloured image and depth image based on the collection of RGB D equipment |
| CN207396939U (en) * | 2017-11-20 | 2018-05-22 | 江西服装学院 | A kind of animation filming apparatus |
| CN109360230A (en) * | 2018-11-08 | 2019-02-19 | 武汉库柏特科技有限公司 | A kind of method for registering images and system based on 2D camera Yu 3D camera |
| CN109741405A (en) * | 2019-01-21 | 2019-05-10 | 同济大学 | A kind of depth information acquisition system based on dual structure light RGB-D camera |
| CN109816731A (en) * | 2017-11-21 | 2019-05-28 | 西安交通大学 | A kind of method of RGB and depth information accuracy registration |
| CN110012280A (en) * | 2019-03-22 | 2019-07-12 | 盎锐（上海）信息科技有限公司 | TOF mould group and VSLAM calculation method for VSLAM system |
| CN110619617A (en) * | 2019-09-27 | 2019-12-27 | 中国科学院长春光学精密机械与物理研究所 | Three-dimensional imaging method, device, equipment and computer readable storage medium |
| CN111427048A (en) * | 2020-02-25 | 2020-07-17 | 深圳奥比中光科技有限公司 | ToF depth measuring device, method for controlling ToF depth measuring device and electronic equipment |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8134637B2 (en) * | 2004-01-28 | 2012-03-13 | Microsoft Corporation | Method and system to increase X-Y resolution in a depth (Z) camera using red, blue, green (RGB) sensing |
-
2020
- 2020-12-11 CN CN202011461732.2A patent/CN112509023B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106682663A (en) * | 2015-11-11 | 2017-05-17 | 天津中科智能识别产业技术研究院有限公司 | Multi-modal biological feature identifying system and method for mobile platform |
| CN107507235A (en) * | 2017-08-31 | 2017-12-22 | 山东大学 | A kind of method for registering of coloured image and depth image based on the collection of RGB D equipment |
| CN207396939U (en) * | 2017-11-20 | 2018-05-22 | 江西服装学院 | A kind of animation filming apparatus |
| CN109816731A (en) * | 2017-11-21 | 2019-05-28 | 西安交通大学 | A kind of method of RGB and depth information accuracy registration |
| CN109360230A (en) * | 2018-11-08 | 2019-02-19 | 武汉库柏特科技有限公司 | A kind of method for registering images and system based on 2D camera Yu 3D camera |
| CN109741405A (en) * | 2019-01-21 | 2019-05-10 | 同济大学 | A kind of depth information acquisition system based on dual structure light RGB-D camera |
| CN110012280A (en) * | 2019-03-22 | 2019-07-12 | 盎锐（上海）信息科技有限公司 | TOF mould group and VSLAM calculation method for VSLAM system |
| CN110619617A (en) * | 2019-09-27 | 2019-12-27 | 中国科学院长春光学精密机械与物理研究所 | Three-dimensional imaging method, device, equipment and computer readable storage medium |
| CN111427048A (en) * | 2020-02-25 | 2020-07-17 | 深圳奥比中光科技有限公司 | ToF depth measuring device, method for controlling ToF depth measuring device and electronic equipment |
Non-Patent Citations (2)
| Title |
|---|
| Robust intrinsic and extrinsic calibration of RGB-D cameras;Filippo Basso, et.al;《IEEE transactions on robotics》;IEEE;20180731;第34卷(第5期);全文 * |
| 多源数据融合技术在古建筑三维重建中的应用;徐光禹等;《测绘通报》;20191025(第10期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112509023A (en) | 2021-03-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109615652B (en) | Depth information acquisition method and device | |
| JP6552729B2 (en) | System and method for fusing the outputs of sensors having different resolutions | |
| EP3531066B1 (en) | Three-dimensional scanning method including a plurality of lasers with different wavelengths, and scanner | |
| CN1238691C (en) | Combined stereovision, color 3D digitizing and motion capture system | |
| US20100245851A1 (en) | Method and apparatus for high-speed unconstrained three-dimensional digitalization | |
| CN113538591A (en) | Calibration method and device for distance measuring device and camera fusion system | |
| CN107860337B (en) | Structured light three-dimensional reconstruction method and device based on array camera | |
| CN102438111A (en) | Three-dimensional measurement chip and system based on double-array image sensor | |
| CN111879240B (en) | 2.5D vision sensor and parameter measuring method | |
| US20210216148A1 (en) | Display device, electronic device and method for driving display device | |
| CN105004324A (en) | Monocular vision sensor with triangulation ranging function | |
| CN112254670B (en) | 3D information acquisition equipment based on optical scanning and intelligent vision integration | |
| CN113034567A (en) | Depth truth value acquisition method, device and system and depth camera | |
| US6730926B2 (en) | Sensing head and apparatus for determining the position and orientation of a target object | |
| CN102316355A (en) | Generation method of 3D machine vision signal and 3D machine vision sensor | |
| CN202406199U (en) | Three-dimensional measure chip and system based on double-array image sensor | |
| US20220180541A1 (en) | Three-dimensional coordinate scanner | |
| CN112509023B (en) | Multi-source camera system and RGBD registration method | |
| CN111654626B (en) | High-resolution camera containing depth information | |
| CN111866490A (en) | Depth image imaging system and method | |
| CN116592766A (en) | Precise three-dimensional measurement method and device based on fusion of laser and monocular vision | |
| JP7414090B2 (en) | Imaging device and method of controlling the imaging device | |
| CN112254671B (en) | Multi-time combined 3D acquisition system and method | |
| WO2022207687A2 (en) | Configuration method for the display of a building information model | |
| CN113406604A (en) | Device and method for calibrating positions of laser radar and camera |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |