CN114862923B - Image registration method and device and storage medium - Google Patents

Image registration method and device and storage medium Download PDF

Info

Publication number
CN114862923B
CN114862923B CN202210786406.1A CN202210786406A CN114862923B CN 114862923 B CN114862923 B CN 114862923B CN 202210786406 A CN202210786406 A CN 202210786406A CN 114862923 B CN114862923 B CN 114862923B
Authority
CN
China
Prior art keywords
image
mode
preset
rgb
registration
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202210786406.1A
Other languages
Chinese (zh)
Other versions
CN114862923A (en
Inventor
田照银
明幼林
吴昊
莫苏苏
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Wuhan Silicon Integrated Co Ltd
Original Assignee
Wuhan Silicon Integrated Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Wuhan Silicon Integrated Co Ltd filed Critical Wuhan Silicon Integrated Co Ltd
Priority to CN202210786406.1A priority Critical patent/CN114862923B/en
Publication of CN114862923A publication Critical patent/CN114862923A/en
Application granted granted Critical
Publication of CN114862923B publication Critical patent/CN114862923B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/32—Determination of transform parameters for the alignment of images, i.e. image registration using correlation-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)

Abstract

The embodiment of the application provides an image registration method and device and a storage medium, which are applied to an image registration device, wherein the image registration device comprises an RGB shooting module and a time of flight TOF shooting module; the method comprises the following steps: simultaneously acquiring a first RGB image and a first depth image of a target object by utilizing an RGB shooting module and a TOF shooting module; acquiring a first registration parameter obtained by performing binocular calibration on an RGB shooting module and a TOF shooting module in a full-resolution mode; and registering the first RGB image and the first depth image based on the image conversion relation between the full-resolution mode and the preset mode and the first registration parameter to obtain a registration depth RGBD image in the preset mode.

Description

Image registration method and device and storage medium
Technical Field
The present application relates to the field of computer vision, and in particular, to an image registration method and apparatus, and a storage medium.
Background
In computer vision, a depth image generated by a Three-dimensional (3D) camera usually needs to be registered with a Three primary color (Red Green Blue, RGB) image to generate a registered depth image, and actually, the purpose of the registration is to superimpose the depth image and the RGB image, that is, convert an image coordinate system of the RGB image into an image coordinate system of the depth image, and color pixels in the depth image.
The precondition of registration is that the internal parameters of an RGB camera and a depth image camera and the relative position between the two cameras need to be known, at present, in the prior art, the internal parameters of the two cameras are obtained by respectively performing monocular calibration on the two cameras in sequence, and then the relative position between the two cameras is obtained by performing binocular calibration, however, when registration is performed in different resolution modes, because the internal parameters of the cameras in different resolution modes are different, the two cameras need to be re-calibrated, so that registration images with different resolutions are output based on the re-calibrated internal parameters, and therefore, the steps during registration are complicated, and the image registration efficiency is low.
Disclosure of Invention
The embodiment of the application provides an image registration method and device and a storage medium, which can improve the efficiency of image registration.
The technical scheme of the application is realized as follows:
in a first aspect, an embodiment of the present application provides an image registration method, which is applied to an image registration apparatus, where the image registration apparatus includes an RGB shooting module and a time-of-flight TOF shooting module; the method comprises the following steps:
simultaneously acquiring a first RGB image and a first depth image of a target object by utilizing the RGB shooting module and the TOF shooting module; the first RGB image is an RGB image in a full resolution mode or a preset mode, and the first depth image is a depth image in the full resolution mode or the preset mode; the preset modes comprise a preset resolution mode, a preset mirror mode and a preset resolution and mirror mode;
acquiring a first registration parameter obtained by performing binocular calibration on the RGB shooting module and the TOF shooting module in the full-resolution mode; and registering the first RGB image and the first depth image based on the image conversion relation between the full resolution mode and the preset mode and the first registration parameter to obtain a registration depth RGBD image in the preset mode.
In the above image registration method, the first RGB image is an RGB image in the full resolution mode, the first depth image is a depth image in the full resolution mode, and the preset mode is the preset resolution mode; the registering the first RGB image and the first depth image based on the image conversion relationship between the full resolution mode and the preset mode and the first registration parameter to obtain an RGBD image in the preset mode includes:
registering the first RGB image and the first depth image based on the first registration parameter to obtain a first RGBD image in the full-resolution mode;
converting the resolution of the first RGBD image based on an image conversion relationship between the full resolution mode and the preset resolution mode to convert the first RGBD image into an RGBD image in the preset resolution mode.
In the above image registration method, the first RGB image is an RGB image in the full resolution mode, the first depth image is a depth image in the preset resolution mode, and the preset mode is the preset resolution mode; the registering the first RGB image and the first depth image based on the image conversion relationship between the full resolution mode and the preset mode and the first registration parameter to obtain the RGBD image in the preset mode includes:
determining an image mapping relationship between the full resolution mode and the preset resolution mode based on an image conversion relationship between the full resolution mode and the preset mode;
determining the coordinate position of each pixel point in the first depth image in the full-resolution mode according to the image mapping relation;
acquiring a first pixel point corresponding to the coordinate position from the first RGB image; and registering the first pixel point with a corresponding pixel point in the first depth image based on the first registration parameter to obtain an RGBD image in the preset resolution mode.
In the image registration method, the first RGB image is an RGB image in the preset resolution mode, the first depth image is a depth image in the preset resolution mode, and the preset mode is the preset resolution mode; the registering the first RGB image and the first depth image based on the image conversion relationship between the full resolution mode and the preset mode and the first registration parameter to obtain the RGBD image in the preset mode includes:
processing the first registration parameter based on the image conversion relation between the full resolution mode and the preset mode to obtain a second registration parameter corresponding to the preset resolution mode;
and registering the first RGB image and the first depth image based on the second registration parameter to obtain a registered depth image in the preset resolution mode.
In the image registration method, the first RGB image is an RGB image in the preset mirror image mode, the first depth image is a depth image in the preset resolution and mirror image mode, and the preset mode is the preset resolution and mirror image mode; before registering the first RGB image and the first depth image based on the image conversion relationship between the full resolution mode and the preset mode and the first registration parameter to obtain the RGBD image in the preset mode, the method further includes:
carrying out inverse mirror image on the first RGB image to obtain a second RGB image; carrying out inverse mirror image on the first depth image to obtain a second depth image;
correspondingly, the registering the first RGB image and the first depth image based on the image conversion relationship between the full resolution mode and the preset mode and the first registration parameter to obtain the RGBD image in the preset mode includes:
registering the second RGB image and the second depth image based on the image conversion relation between the full resolution mode and the preset mode and the first registration parameter to obtain an RGBD image in the preset mode;
correspondingly, after the registering the first RGB image and the first depth image based on the image conversion relationship between the full resolution mode and the preset mode and the first registration parameter to obtain the RGBD image in the preset mode, the method further includes:
and mirroring the RGBD image to obtain a mirrored RGBD image under the preset resolution and mirroring mode.
In the image registration method, the first RGB image is an RGB image in the preset mirror image mode, the first depth image is a depth image in the preset resolution and mirror image mode, and the preset mode is the preset resolution and mirror image mode; the registering the first RGB image and the first depth image based on the image conversion relationship between the full resolution mode and the preset mode and the first registration parameter to obtain the RGBD image in the preset mode includes:
mirroring the first registration parameter according to a mirroring rule corresponding to the preset mirroring mode to obtain a first mirrored registration parameter;
and registering the first RGB image and the first depth image based on the image conversion relation between the full resolution mode and the preset mode and the first mirror image registration parameter to obtain a mirror image RGBD image under the preset resolution and mirror image mode.
In the above image registration method, the first registration parameters include internal parameters of the RGB photographing module, internal parameters of the TOF photographing module, and relative position parameters between the RGB photographing module and the TOF photographing module.
In a second aspect, an embodiment of the present application provides an image registration apparatus, which includes an RGB capturing module and a time of flight TOF capturing module; the device comprises:
the acquisition module is used for simultaneously acquiring a first RGB image and a first depth image of a target object by utilizing the RGB shooting module and the TOF shooting module; the first RGB image is an RGB image in a full resolution mode or a preset mode, and the first depth image is a depth image in the full resolution mode or the preset mode; the preset modes comprise a preset resolution mode, a preset mirror mode and a preset resolution and mirror mode;
the acquisition module is used for acquiring a first registration parameter obtained by performing binocular calibration on the RGB shooting module and the TOF shooting module in the full-resolution mode; and registering the first RGB image and the first depth image based on the image conversion relation between the full resolution mode and the preset mode and the first registration parameter to obtain a registration depth RGBD image in the preset mode.
In a third aspect, an embodiment of the present application provides an image registration apparatus, where the apparatus includes: a processor, a memory, and a communication bus; the processor, when executing a run program stored in the memory, implements the image registration method as described in any of the above.
In a fourth aspect, the present application provides a storage medium having a computer program stored thereon, where the computer program is executed by a processor to implement the image registration method according to any one of the above-mentioned items.
The embodiment of the application provides an image registration method and device and a storage medium, which are applied to an image registration device, wherein the image registration device comprises an RGB shooting module and a time of flight TOF shooting module; the method comprises the following steps: simultaneously acquiring a first RGB image and a first depth image of a target object by utilizing an RGB shooting module and a TOF shooting module; the first RGB image is an RGB image in a full resolution mode or a preset mode, and the first depth image is a depth image in the full resolution mode or the preset mode; the preset modes comprise a preset resolution mode, a preset mirror mode and a preset resolution and mirror mode; acquiring a first registration parameter obtained by performing binocular calibration on an RGB shooting module and a TOF shooting module in a full-resolution mode; registering the first RGB image and the first depth image based on an image conversion relation and a first registration parameter between a full-resolution mode and a preset mode to obtain a registered depth RGBD image in the preset mode; by adopting the implementation scheme, the registration parameters of the two shooting modules in the full-resolution mode are obtained by performing one-time binocular calibration in the full-resolution mode, and then the RGB image and the depth image can be registered according to the image conversion relation and the registration parameters between the full-resolution mode and the preset mode, so that the registration depth image in the preset mode is obtained, the registration steps are simplified, and the image registration efficiency is improved.
Drawings
Fig. 1 is a flowchart of an image registration method provided in an embodiment of the present application;
fig. 2 is a flowchart of an exemplary image registration method provided in an embodiment of the present application;
fig. 3 is a flowchart of an exemplary image registration method provided in an embodiment of the present application;
fig. 4 is a flowchart of an exemplary image registration method provided in the embodiment of the present application;
fig. 5 is a flowchart of a fourth exemplary image registration method provided in the embodiments of the present application;
fig. 6 is a flowchart of an exemplary image registration method provided in an embodiment of the present application;
fig. 7 is a schematic structural diagram illustrating an image registration apparatus according to an embodiment of the present disclosure;
fig. 8 is a schematic structural diagram of an image registration apparatus according to an embodiment of the present application.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the present application. And are not intended to limit the present application.
The embodiment of the application provides an image registration method, which is applied to an image registration device, wherein the image registration device comprises an RGB shooting module and a time of flight TOF shooting module; fig. 1 is a flowchart of an image registration method provided in an embodiment of the present application, and as shown in fig. 1, the image registration method may include:
s101, simultaneously acquiring a first RGB image and a first depth image of a target object by utilizing an RGB shooting module and a TOF shooting module; the first RGB image is an RGB image in a full resolution mode or a preset mode, and the first depth image is a depth image in the full resolution mode or the preset mode; the preset modes include a preset resolution mode, a preset mirror mode, and a preset resolution and mirror mode.
In the embodiment of the application, an image registration device simultaneously acquires a first RGB image and a first depth image of a target object by using an RGB shooting module and a TOF shooting module; the first RGB image is an RGB image in a full resolution mode or a preset mode, and the first depth image is a depth image in the full resolution mode or the preset mode; the preset modes include a preset resolution mode, a preset mirror mode, and a preset resolution and mirror mode.
The image registration method is suitable for a scene of registering the RGB image and the depth image.
In the embodiment of the application, the image registration device acquires a first RGB image of a target object by using an RGB shooting module, and simultaneously acquires a first depth image of the target object by using a TOF shooting module; when the RGB shooting module collects the RGB image of the target object, the RGB image of the target object in any mode can be collected and output at the same time; when the TOF shooting module acquires the depth image of the target object, the TOF shooting module can acquire and output the depth image of the target object in any mode.
S102, acquiring a first registration parameter obtained by performing binocular calibration on an RGB shooting module and a TOF shooting module in a full-resolution mode; and registering the first RGB image and the first depth image based on the image conversion relation between the full-resolution mode and the preset mode and the first registration parameter to obtain a registration depth RGBD image in the preset mode.
In the embodiment of the application, after extracting a first RGB image and a first depth image, an image registration device acquires a first registration parameter obtained by performing binocular calibration on an RGB shooting module and a TOF shooting module in a full-resolution mode; and registering the first RGB image and the first depth image based on the image conversion relation between the full resolution mode and the preset mode and the first registration parameter to obtain a registered depth RGBD image in the preset mode.
It should be noted that, registration parameters of the RGB photographing module and the TOF photographing module in different modes are different, and a first registration parameter in the present application is a registration parameter corresponding to the full resolution mode.
In the embodiment of the present application, the first registration parameter includes an internal parameter of the RGB photographing module, an internal parameter of the TOF photographing module, and a relative position parameter between the RGB photographing module and the TOF photographing module.
It should be noted that, in the embodiment of the present application, the internal parameters of the RGB capturing module and the internal parameters of the TOF capturing module are both matrixes, and the relative position parameters include a rotation matrix and a translation matrix of one capturing module relative to another capturing module.
It should be noted that, when the registration function is actually used, since the TOF capturing module has multiple operating modes such as Binning, RR, ROI, and the like, it is possible to capture depth images with different resolutions by selecting different operating modes, and output registration depth images with different resolutions at the same time.
In an optional embodiment of the present application, the first RGB image is an RGB image in a full resolution mode, and the first depth image is a depth image in the full resolution mode; the preset mode is a preset resolution mode; fig. 2 is a flowchart of an exemplary image registration method provided in an embodiment of the present application, where an image registration process specifically includes:
s201, registering the first RGB image and the first depth image based on the first registration parameter to obtain a first RGBD image in a full resolution mode.
Illustratively, the specific steps of image registration are as follows:
1. constructing a three-dimensional vector
Figure 297268DEST_PATH_IMAGE001
Wherein
Figure 811426DEST_PATH_IMAGE002
Is the pixel coordinates of the point and,
Figure 629471DEST_PATH_IMAGE003
as an image
Figure 647106DEST_PATH_IMAGE004
The length of the direction is long,
Figure 203989DEST_PATH_IMAGE005
as an image
Figure 189131DEST_PATH_IMAGE006
The length of the direction is long,
Figure 325715DEST_PATH_IMAGE007
is the depth value of the pixel.
2. Internal parameter matrix for TOF camera module
Figure 994593DEST_PATH_IMAGE008
Inverse of, multiply by
Figure 936092DEST_PATH_IMAGE009
Obtaining corresponding space point coordinates
Figure 628105DEST_PATH_IMAGE010
。
3. Due to the fact that
Figure 833958DEST_PATH_IMAGE010
Is the coordinate of the point in the coordinate system of the TOF shooting module, and needs to be converted into the coordinate system of the RGB shooting module, namely, the coordinate system is multiplied by a rotation matrix
Figure 75453DEST_PATH_IMAGE011
Plus a translation vector
Figure 177401DEST_PATH_IMAGE012
To obtain
Figure 622289DEST_PATH_IMAGE013
。
4. Internal parameter matrix of module is shot with RGB
Figure 382565DEST_PATH_IMAGE014
Multiplication by
Figure 494878DEST_PATH_IMAGE013
To obtain
Figure 564465DEST_PATH_IMAGE015
，
Figure 762228DEST_PATH_IMAGE015
And
Figure 559152DEST_PATH_IMAGE009
there are 3 components in total, let
Figure 525971DEST_PATH_IMAGE016
Wherein
Figure 500880DEST_PATH_IMAGE017
The value of (d) is the distance (in mm) from the point to the RGB capture module, then
Figure 185939DEST_PATH_IMAGE018
The coordinates are the pixel coordinates of the point in the RGB image,
Figure 288019DEST_PATH_IMAGE019
as an image
Figure 312607DEST_PATH_IMAGE004
The length of the direction is longer than that of the direction,
Figure 723996DEST_PATH_IMAGE020
as an image
Figure 880040DEST_PATH_IMAGE006
And the direction length is used for extracting the color of the pixel as the color of the corresponding pixel in the depth image.
5. And performing the operation on each pixel point in the depth image to obtain a registration depth image.
S202, converting the resolution of the first RGBD image based on an image conversion relation between the full resolution mode and a preset resolution mode so as to convert the first RGBD image into an RGBD image in the preset resolution mode.
It should be noted that, in the embodiment of the present application, after obtaining the first RGBD image in the full resolution mode according to the first registration parameter corresponding to the full resolution mode, the RGB image in the full resolution mode, and the depth image, the image registration apparatus needs to convert the first RGBD image into the registration depth image in the preset resolution mode.
In an actual scene, the present application exemplarily illustrates the above embodiments, an RGB capturing module in the image registration apparatus acquires a full-resolution RGB image in a full-resolution mode, a TOF capturing module acquires a preset-resolution depth image in one of multiple working modes, such as Binning, RR, and ROI, and the image registration specifically includes the following steps:
1. and taking out the full-resolution depth image in the full-resolution mode.
2. And registering the full-resolution depth image and the full-resolution RGB image based on the first registration parameter to obtain a full-resolution registration depth image.
3. And carrying out conversion operation of one working mode of a plurality of working modes such as Binning, RR and ROI on the full-resolution registration depth image to obtain the registration depth image under the corresponding resolution.
In another optional embodiment of the present application, the first RGB image is an RGB image in a full resolution mode, and the first depth image is a depth image in a preset resolution mode; the preset mode is a preset resolution mode; fig. 3 is a flowchart of an exemplary image registration method provided in the embodiment of the present application, where an image registration process is specifically as follows:
s301, determining an image mapping relation between the full resolution mode and the preset resolution mode based on the image conversion relation between the full resolution mode and the preset mode.
It should be noted that the image conversion relationship can be understood as a positional relationship between a pixel point in an image in a preset resolution mode and a pixel point in an image in a full resolution mode.
It should be noted that the mapping relationship may be understood as to which pixel point in the image in the full resolution mode each pixel point in the image in the preset resolution mode can be mapped to.
S302, according to the image mapping relation, the coordinate position of each pixel point in the first depth image in the full-resolution mode is determined.
Exemplarily, it is assumed that a pixel point a with a coordinate as a first coordinate position exists in the first depth image, at this time, a pixel point B identical to the pixel point a is searched from the depth image in the full resolution mode, and a coordinate position of the pixel point B identical to the pixel point a in the depth image in the full resolution mode is determined as a coordinate position of the pixel point a in the depth image in the full resolution mode.
S303, acquiring a first pixel point corresponding to the coordinate position from the first RGB image; and registering the first pixel point and the corresponding pixel point in the first depth image based on the first registration parameter to obtain an RGBD image in a preset resolution mode.
It should be noted that, since the first pixel point is a pixel point in the image in the full resolution mode, the first registration parameter corresponding to the full resolution mode may be used to register the first pixel point and the second pixel point, so as to obtain the RGBD image in the preset resolution mode.
The present application exemplarily illustrates the above embodiments in an actual scene, where an RGB capturing module in an image registration apparatus acquires a full-resolution RGB image in a full-resolution mode, a TOF capturing module acquires a preset-resolution depth image in one of multiple working modes, such as Binning, RR, and ROI, and the image registration specifically includes the following steps:
1. fetching full resolution depth images in full resolution mode
Figure 769499DEST_PATH_IMAGE021
And finding a full resolution depth image
Figure 710910DEST_PATH_IMAGE021
And a preset resolution depth image
Figure 27622DEST_PATH_IMAGE022
Mapping relationship between
Figure 172426DEST_PATH_IMAGE023
。
2. By mapping relationships
Figure 865576DEST_PATH_IMAGE023
Positioning
Figure 661493DEST_PATH_IMAGE022
The middle pixel point is
Figure 414686DEST_PATH_IMAGE021
Coordinates of (5)
Figure 545322DEST_PATH_IMAGE024
。
3. Based on the first registration parameter, all the full-resolution RGB images and the preset-resolution depth images meet the mapping relation
Figure 573321DEST_PATH_IMAGE024
And carrying out registration on the pixel points.
4. And combining all the registered pixel points in the step 3, and outputting according to a preset resolution ratio to obtain a registered depth image under the corresponding resolution ratio.
In a third optional embodiment of the present application, the first RGB image is an RGB image in a preset resolution mode, and the first depth image is a depth image in the preset resolution mode; the preset mode is a preset resolution mode; fig. 4 is a flowchart of an exemplary image registration method provided in an embodiment of the present application, where an image registration process specifically includes:
s401, processing the first registration parameter based on the image conversion relation between the full resolution mode and the preset mode to obtain a second registration parameter corresponding to the preset resolution mode.
It should be noted that the registration parameters corresponding to different resolution modes are different, and the first registration parameter corresponding to the full resolution mode is processed through the image conversion relationship between the full resolution mode and the preset mode, so as to obtain different registration parameters in the preset resolution mode.
S402, registering the first RGB image and the first depth image based on the second registration parameter to obtain a registration depth image in a preset resolution mode.
In an actual scene, the present application exemplarily illustrates the above embodiments, an RGB capturing module in the image registration apparatus acquires an RGB image with a preset resolution in a preset resolution mode, a TOF capturing module acquires a depth image with a preset resolution in one of multiple working modes, such as Binning, RR, and ROI, and the image registration specifically includes the following steps:
1. obtaining a first registration parameter in full resolution mode by binocular calibration
Figure 427007DEST_PATH_IMAGE025
。
2. Calculating the corresponding relation of different resolution modes according to the mathematical relation I of different resolution modes
Figure 147838DEST_PATH_IMAGE026
The registration parameters are based on using a mathematical relationship I
Figure 516503DEST_PATH_IMAGE025
And calculating acquisition instead of binocular calibration acquisition again.
3. And directly carrying out image registration by using the registration parameters in different resolution modes to obtain registration depth images in different resolution modes.
First registration parameter
Figure 296328DEST_PATH_IMAGE025
The method comprises four parameters, namely an internal parameter INTRI1a of an RGB shooting module, an internal parameter INTRI1b of a TOF shooting module, and a rotation matrix of the TOF shooting module relative to the RGB shooting module
Figure 801259DEST_PATH_IMAGE027
The translation matrix of the TOF camera module relative to the RGB camera module is recorded as
Figure 630674DEST_PATH_IMAGE028
. Because the relative positions of the two cameras are not changed due to the change of the resolution, the internal parameters of the RGB shooting module or the TOF shooting module only need to be correspondingly transformed, the internal parameters of the RGB shooting module are transformed when the resolution of the RGB image is changed, and the internal parameters of the TOF shooting module are transformed when the resolution of the depth image is changed. Taking the depth map resolution change as an example, the conversion rule when the RGB map resolution changes is the same, specifically:
1. binning (including Analog-Binning for Analog readout and Digital-Binning for signal readout) mode of operation
The internal reference of the TOF shooting module before the working mode is not selected is INTRI1b, the internal reference of the TOF shooting module after the Binning working mode is selected is INTRI2b, Analog-Binning reading changes cy and fy, and Digital-Binning reading changes offset cx and focal length fx in the camera internal reference.
(a) Analog-Binning with Analog readout
INTRI2b (2, 3) = INTRI1b (2, 3)/2; INTRI2b (2, 2) = INTRI1b (2, 2)/2; the remaining position parameters are equal.
(b) Signal readout Digital-Binning
INTRI2b (1, 3) = INTRI1b (1, 3)/2; INTRI2b (1, 1) = INTRI1b (1, 1)/2; the remaining position parameters are equal.
2. Reduced Resolution Reduction (RR) mode of operation
Before the working mode is not selected, the internal reference of the TOF shooting module is INTRI1b, after the RR working mode is selected, the internal reference of the TOF shooting module is INTRI3b, the RR working mode changes offsets cx and cy and focal lengths fx and fy in the camera internal reference, and if the step number step of the RR working mode is n:
INTRI3b (2, 3) = INTRI1b (2, 3)/n; INTRI3b (2, 2) = INTRI1b (2, 2)/n; INTRI3b (1, 3) = INTRI1b (1, 3)/n; INTRI3b (1, 1) = INTRI1b (1, 1)/n; the remaining position parameters are equal.
3. Region Of Interest (ROI) operation mode
Before the working mode is not selected, the internal reference of the TOF shooting module is INTRI1b, after the ROI working mode is selected, the internal reference of the TOF shooting module is INTRI4b, the ROI changes the offsets cx and cy in the camera internal reference, and the upper left starting position of the ROI is (left, upper), then:
INTRI4b (2, 3) = INTRI1b (2, 3) -upside; INTRI4b (1, 3) = INTRI1b (1, 3) -left; the remaining position parameters are equal.
In the embodiment of the application, the RGB shooting module and the TOF shooting module on the image registration device both have mirror image modes, mirror image RGB images and mirror image RGB depth images can be collected through the mirror image modes, and then registration is carried out on the mirror image RGB images and the mirror image RGB depth images to obtain mirror image registration depth images.
In a fourth optional embodiment of the present application, the first RGB image is an RGB image in a preset mirror image mode, the first depth image is a depth image in a preset resolution and mirror image mode, and the preset mode is a preset resolution and mirror image mode; fig. 5 is a flowchart of an exemplary image registration method provided in an embodiment of the present application, where an image registration process specifically includes:
s501, carrying out inverse mirror image on the first RGB image to obtain a second RGB image; and carrying out inverse mirror image on the first depth image to obtain a second depth image.
It should be noted that the preset mirroring mode may include up-down mirroring, left-right mirroring, or up-down, left-right mirroring, or the like; the specific preset mirror mode may be determined according to actual conditions, and the embodiment of the present application is not specifically limited herein.
It should be noted that, in practical applications, only the mirror image mode of one of the RGB photographing module and the TOF photographing module may be used, and a specific photographing mode may be determined according to an actual situation, which is not specifically limited herein in this embodiment of the present application.
It should be noted that, when the first mirror image RGB image is inversely mirrored, a specific process of inverse mirroring needs to be determined according to a mirroring mode of the RGB photographing module; when the first mirror image depth image is subjected to inverse mirror image, a specific process of inverse mirror image needs to be determined according to a mirror image mode of the TOF shooting module.
And S502, registering the second RGB image and the second depth image based on the image conversion relation between the full resolution mode and the preset mode and the first registration parameter to obtain an RGBD image in the preset mode.
It should be noted that after the first RGB image and the first depth image are obtained, the image registration method in the foregoing embodiment is used to register the first RGB image and the first depth image to obtain an RGBD image in the preset mode, where the RGBD image is a non-mirror RGBD image, and each pixel point in the non-mirror RGBD image needs to be mirrored according to a mirror image mode to obtain a mirror RGBD image in the preset resolution and the mirror image mode.
S503, mirroring the RGBD image to obtain a mirrored RGBD image under a preset resolution and mirroring mode.
The present application exemplarily illustrates the above embodiments in an actual scene, where an RGB capturing module in an image registration apparatus acquires an RGB image with a preset resolution in a preset resolution mode, and a TOF capturing module acquires a mirror image depth image with a preset resolution in one of multiple working modes, such as Binning, RR, and ROI, and a mirror image mode, and the image registration specifically includes the following steps:
1. mirror depth image of
Figure 486635DEST_PATH_IMAGE029
In the pixel-by-pixel registration stage, firstly, mirroring the pixel point coordinates of the mirror image depth image to obtain a depth image before mirroring
Figure 308966DEST_PATH_IMAGE021
I.e. by
Figure 933983DEST_PATH_IMAGE030
。
2. Divide the preset intoResolution RGB image and pre-mirror depth image
Figure 668721DEST_PATH_IMAGE021
And after registration, mirroring the pixel point coordinates in the registration depth image to obtain a mirror registration depth image.
In a fourth optional embodiment of the present application, an RGB photographing module and a TOF photographing module on the image registration apparatus acquire a mirror image RGB image and a mirror image RGB depth image through a mirror image mode; fig. 6 is a flowchart of an exemplary image registration method fifth provided in an embodiment of the present application, where an image registration process specifically includes:
s601, mirroring the first registration parameter according to a mirroring rule corresponding to a preset mirroring mode to obtain a first mirroring registration parameter.
It should be noted that the preset mirroring mode may include up-down mirroring, left-right mirroring, or up-down, left-right mirroring, or the like; the specific preset mirror mode may be determined according to actual conditions, and the embodiment of the present application is not specifically limited herein.
S602, registering the first RGB image and the first depth image based on the image conversion relation between the full resolution mode and the preset mode and the first mirror image registration parameter to obtain a mirror image RGBD image under the preset resolution and mirror image mode.
It should be noted that the first registration parameter is a registration parameter in a non-mirror image mode, and therefore, the registration parameter needs to be subjected to mirror image processing to obtain a registration parameter in a mirror image mode, and then the first RGB image and the first depth image are registered based on the registration parameter in the mirror image mode to obtain a mirror RGBD image in a preset resolution and a mirror image mode.
In an actual scene, the foregoing embodiment is exemplarily explained, assuming that an RGB photographing module in an image registration apparatus is a left-side photographing module, and setting the RGB photographing module as a primary camera, that is, a right-side TOF photographing module is an auxiliary camera, and the specific steps of mirroring the first registration parameter in the present application are as follows:
1. original registration parameters (not mirror image)
The internal references of the RGB camera module are INTRI1a, the internal reference of the TOF camera module is INTRI1b, and the rotation matrix of the TOF camera module relative to the RGB camera module is recorded as
Figure 762710DEST_PATH_IMAGE027
The translation matrix of the TOF camera module relative to the RGB camera module is recorded as
Figure 405044DEST_PATH_IMAGE028
。
2. Registration parameters for left and right mirror mode
The internal reference of the RGB capture module is INTRI1a, the internal reference of the TOF capture module is INTRI1b, and the rotation matrix of the TOF capture module relative to the RGB capture module is denoted INTRI1a
Figure 87829DEST_PATH_IMAGE031
The translation matrix of the TOF camera module relative to the RGB camera module is recorded as
Figure 773894DEST_PATH_IMAGE032
。
The relationship between INTRI2a and INTRI1a is cx2a = wxa-cx1a, skew2a = -skew1a, where wxa is the RGB capture module image pixel width;
INTRI2b has a relationship with INTRI1b of cx2b = wxb-cx1b, skew2b = -skew1b, where wxb is the TOF capture module image pixel width;
the rotation matrix of the TOF imaging module relative to the RGB imaging module is a 3 x 3 matrix, so
Figure 338867DEST_PATH_IMAGE031
And
Figure 519313DEST_PATH_IMAGE027
is the inversion of the values at four (12, 13, 21, 31) positions of the matrix;
the translation matrix of the TOF imaging module relative to the RGB imaging module is a 3 x1 matrix, so
Figure 72916DEST_PATH_IMAGE032
And
Figure 946195DEST_PATH_IMAGE028
is the inversion of the value at one position (11) of the matrix.
3. Registration parameters for top and bottom mirror modes
The internal reference of the RGB capture module is INTRI1a, the internal reference of the TOF capture module is INTRI1b, and the rotation matrix of the TOF capture module relative to the RGB capture module is denoted INTRI1a
Figure 998464DEST_PATH_IMAGE033
The translation matrix of the TOF camera module relative to the RGB camera module is recorded as
Figure 717021DEST_PATH_IMAGE034
。
The relationship between INTRI3a and INTRI1a is cy3a = hya-cy1a, and skew3a = -skew1a, where hya is the RGB capture module image pixel height;
the relationship between INTRI3b and INTRI1b is cy3b = hyb-cy1b, and skew3b = -skew1b, where hyb is the TOF capture module image pixel height;
the rotation matrix of the TOF imaging module relative to the RGB imaging module is a 3 x 3 matrix, so
Figure 623666DEST_PATH_IMAGE033
And
Figure 667846DEST_PATH_IMAGE027
is the inversion of the values at four (12, 21, 23, 32) positions of the matrix;
the translation matrix of the TOF imaging module relative to the RGB imaging module is a 3 x1 matrix, so
Figure 207412DEST_PATH_IMAGE034
And with
Figure 480392DEST_PATH_IMAGE028
Is the inversion of the value at one position (21) of the matrix.
4. Registration parameters for left-right + up-down mirror mode
The internal reference of the RGB capture module is INTRI1a, the internal reference of the TOF capture module is INTRI1b, and the rotation matrix of the TOF capture module relative to the RGB capture module is denoted INTRI1a
Figure 992276DEST_PATH_IMAGE035
The translation matrix of the TOF camera module relative to the RGB camera module is recorded as
Figure 207357DEST_PATH_IMAGE036
。
INTRI4a has a relationship with INTRI1a of cx4a = wxa-cx1a, cy4a = hya-cy1a, where wxa is the RGB capture module image pixel width and hya is the RGB capture module image pixel height;
INTRI4b has a relationship with INTRI1b of cx4b = wxb-cx1b, cy4b = hyb-cy1b, where wxb is TOF shooting module image pixel width and hyb is TOF shooting module image pixel height;
the rotation matrix of the TOF imaging module relative to the RGB imaging module is a 3 x 3 matrix, so
Figure 217907DEST_PATH_IMAGE035
And with
Figure 747108DEST_PATH_IMAGE027
Is the inversion of the values at four (13, 23, 31, 32) positions of the matrix;
the translation matrix of the TOF imaging module relative to the RGB imaging module is a 3 x1 matrix, so
Figure 910237DEST_PATH_IMAGE036
And
Figure 775512DEST_PATH_IMAGE028
is the inversion of the value at two (11, 21) positions of the matrix.
It can be found that the parameter change of the left, right, upper and lower mirror images is the parameter change of the left, right mirror images and the parameter change of the upper and lower mirror images superposed.
The embodiment of the application provides an image registration method, which is applied to an image registration device, wherein the image registration device comprises an RGB shooting module and a time of flight TOF shooting module; the method comprises the following steps: simultaneously acquiring a first RGB image and a first depth image of a target object by utilizing an RGB shooting module and a TOF shooting module; the first RGB image is an RGB image in a full resolution mode or a preset mode, and the first depth image is a depth image in the full resolution mode or the preset mode; the preset modes comprise a preset resolution mode, a preset mirror mode and a preset resolution and mirror mode; acquiring a first registration parameter obtained by performing binocular calibration on an RGB shooting module and a TOF shooting module in a full-resolution mode; registering the first RGB image and the first depth image based on an image conversion relation and a first registration parameter between a full-resolution mode and a preset mode to obtain a registered depth RGBD image in the preset mode; by adopting the implementation scheme, the registration parameters of the two shooting modules in the full-resolution mode are obtained by performing one-time binocular calibration in the full-resolution mode, and then the RGB image and the depth image can be registered according to the image conversion relation and the registration parameters between the full-resolution mode and the preset mode, so that the registration depth image in the preset mode is obtained, the registration steps are simplified, and the image registration efficiency is improved.
Based on the above embodiments, in another embodiment of the present application, there is provided an image registration apparatus 1, including an RGB photographing module and a time of flight TOF photographing module; fig. 7 is a schematic structural diagram of an image registration apparatus provided in the present application, and as shown in fig. 7, the image registration apparatus 1 includes:
an acquisition module 10, configured to acquire a first RGB image and a first depth image of a target object simultaneously by using the RGB photographing module and the TOF photographing module; the first RGB image is an RGB image in a full resolution mode or a preset mode, and the first depth image is a depth image in the full resolution mode or the preset mode; the preset modes comprise a preset resolution mode, a preset mirror mode and a preset resolution and mirror mode;
an obtaining module 11, configured to obtain a first registration parameter obtained by performing binocular calibration on the RGB shooting module and the TOF shooting module in the full-resolution mode; and registering the first RGB image and the first depth image based on the image conversion relation between the full resolution mode and the preset mode and the first registration parameter to obtain a registration depth RGBD image in the preset mode.
Optionally, the image registration apparatus 1 further includes: a registration module and a mode conversion module;
the registration module is configured to register the first RGB image and the first depth image based on the first registration parameter, so as to obtain a first RGBD image in the full resolution mode;
the mode conversion module is configured to convert the resolution of the first RGBD image based on an image conversion relationship between the full resolution mode and the preset resolution mode to convert the first RGBD image into an RGBD image in the preset resolution mode.
Optionally, the image registration apparatus 1 further comprises: a determination module;
the determining module is configured to determine an image mapping relationship between the full resolution mode and the preset resolution mode based on an image conversion relationship between the full resolution mode and the preset mode;
the determining module is further configured to determine, according to the image mapping relationship, a coordinate position of each pixel point in the first depth image in the full-resolution mode;
the registration module is further configured to obtain a first pixel point corresponding to the coordinate position from the first RGB image; and registering the first pixel point with a corresponding pixel point in the first depth image based on the first registration parameter to obtain an RGBD image in the preset resolution mode.
Optionally, the image registration apparatus 1 further includes: a data processing module;
the data processing module is configured to process the first registration parameter based on an image conversion relationship between the full resolution mode and the preset mode to obtain a second registration parameter corresponding to the preset resolution mode;
the registration module is further configured to register the first RGB image and the first depth image based on the second registration parameter, so as to obtain a registration depth image in the preset resolution mode.
Optionally, the image registration apparatus 1 further comprises: an image processing module;
the image processing module is used for carrying out inverse mirror image on the first RGB image to obtain a second RGB image; carrying out inverse mirror image on the first depth image to obtain a second depth image;
the registration module is further configured to register the second RGB image and the second depth image based on the image conversion relationship between the full resolution mode and the preset mode and the first registration parameter, so as to obtain an RGBD image in the preset mode;
the image processing module is further configured to mirror the RGBD image to obtain a mirror RGBD image in the preset resolution and mirror mode.
Optionally, the data processing module is further configured to mirror the first registration parameter according to a mirror rule corresponding to the preset mirror mode, so as to obtain a first mirror image registration parameter;
the registration module is further configured to register the first RGB image and the first depth image based on the image conversion relationship between the full resolution mode and the preset mode and the first mirror image registration parameter, so as to obtain a mirror image RGBD image in the preset resolution and mirror image mode.
Optionally, the first registration parameter includes an internal parameter of the RGB capturing module, an internal parameter of the TOF capturing module, and a relative position parameter between the RGB capturing module and the TOF capturing module.
An embodiment of the present application provides an image registration apparatus, including: the system comprises an acquisition module and an acquisition module, wherein the acquisition module is used for simultaneously acquiring a first RGB image and a first depth image of a target object by utilizing an RGB shooting module and a TOF shooting module; the first RGB image is an RGB image in a full resolution mode or a preset mode, and the first depth image is a depth image in the full resolution mode or the preset mode; the preset modes comprise a preset resolution mode, a preset mirror mode, a preset resolution and a mirror mode; the acquisition module is used for acquiring a first registration parameter obtained by performing binocular calibration on the RGB shooting module and the TOF shooting module in a full-resolution mode; registering the first RGB image and the first depth image based on an image conversion relation and a first registration parameter between a full-resolution mode and a preset mode to obtain a registered depth RGBD image in the preset mode; by adopting the implementation scheme, the registration parameters of the two shooting modules in the full-resolution mode are obtained by carrying out primary binocular calibration in the full-resolution mode, and then the RGB image and the depth image can be registered according to the image conversion relation and the registration parameters between the full-resolution mode and the preset mode, so that the registration depth image in the preset mode is obtained, the registration steps are simplified, and the image registration efficiency is improved.
Fig. 8 is a schematic structural diagram of an image registration apparatus provided in an embodiment of the present application, and in practical application, based on the same public concept of the foregoing embodiment, as shown in fig. 8, an image registration apparatus 2 of the present embodiment includes: a processor 20, a memory 21, and a communication bus 22.
In a Specific embodiment, the acquisition module 10, the acquisition module 11, the registration module, the mode conversion module, the determination module, the data Processing module and the image Processing module may be implemented by a Processor 20 located on the image registration apparatus 2, and the Processor 20 may be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic image Processing Device (PLD), a Field Programmable Gate Array (FPGA), a CPU, a controller, a microcontroller and a microprocessor. It is understood that the electronic device for implementing the above-mentioned processor function may be other for different image registration devices, and the embodiment is not limited in particular.
In the embodiment of the present application, the communication bus 22 is used for realizing connection communication between the processor 20 and the memory 21; the processor 20 described above, when executing the operating program stored in the memory 21, implements the following image registration method:
simultaneously acquiring a first RGB image and a first depth image of a target object by utilizing the RGB shooting module and the TOF shooting module; the first RGB image is an RGB image in a full resolution mode or a preset mode, and the first depth image is a depth image in the full resolution mode or the preset mode; the preset modes comprise a preset resolution mode, a preset mirror mode and a preset resolution and mirror mode;
acquiring a first registration parameter obtained by performing binocular calibration on the RGB shooting module and the TOF shooting module in the full-resolution mode; and registering the first RGB image and the first depth image based on the image conversion relation between the full resolution mode and the preset mode and the first registration parameter to obtain a registration depth RGBD image in the preset mode.
Optionally, the processor 20 is further configured to register the first RGB image and the first depth image based on the first registration parameter, so as to obtain a first RGBD image in the full resolution mode; converting the resolution of the first RGBD image based on an image conversion relationship between the full resolution mode and the preset resolution mode to convert the first RGBD image into an RGBD image in the preset resolution mode.
Optionally, the processor 20 is further configured to determine an image mapping relationship between the full resolution mode and the preset resolution mode based on an image conversion relationship between the full resolution mode and the preset mode; determining the coordinate position of each pixel point in the first depth image in the full-resolution mode according to the image mapping relation; acquiring a first pixel point corresponding to the coordinate position from the first RGB image; and registering the first pixel point with a corresponding pixel point in the first depth image based on the first registration parameter to obtain an RGBD image in the preset resolution mode.
Optionally, the processor 20 is further configured to process the first registration parameter based on an image conversion relationship between the full resolution mode and the preset mode, so as to obtain a second registration parameter corresponding to the preset resolution mode; and registering the first RGB image and the first depth image based on the second registration parameter to obtain a registered depth image in the preset resolution mode.
Optionally, the processor 20 is further configured to perform inverse mirror image on the first RGB image to obtain a second RGB image; carrying out inverse mirror image on the first depth image to obtain a second depth image; registering the second RGB image and the second depth image based on the image conversion relation between the full resolution mode and the preset mode and the first registration parameter to obtain an RGBD image in the preset mode; and mirroring the RGBD image to obtain a mirrored RGBD image under the preset resolution and mirroring mode.
Optionally, the processor 20 is further configured to mirror the first registration parameter according to a mirror rule corresponding to the preset mirror mode, so as to obtain a first mirror image registration parameter; and registering the first RGB image and the first depth image based on the image conversion relation between the full resolution mode and the preset mode and the first mirror image registration parameter to obtain a mirror image RGBD image in the preset resolution and mirror image mode.
Optionally, the first registration parameter includes an internal parameter of the RGB capturing module, an internal parameter of the TOF capturing module, and a relative position parameter between the RGB capturing module and the TOF capturing module.
The present application provides a storage medium, on which a computer program is stored, where the computer readable storage medium stores one or more programs, and the one or more programs are executable by one or more processors and applied to an image registration apparatus, and the computer program implements the image registration method as described above.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
Through the description of the foregoing embodiments, it is clear to those skilled in the art that the method of the foregoing embodiments may be implemented by software plus a necessary general hardware platform, and certainly may also be implemented by hardware, but in many cases, the former is a better implementation. Based on such understanding, the technical solutions of the present disclosure may be embodied in the form of a software product, which is stored in a storage medium (such as a ROM/RAM, a magnetic disk, and an optical disk), and includes several instructions for enabling an image display device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device) to perform the image registration method according to the embodiments of the present disclosure.
The above description is only a preferred embodiment of the present application, and is not intended to limit the scope of the present application.

Claims (10)

1. An image registration method is applied to an image registration device, wherein the image registration device comprises an RGB shooting module and a time of flight (TOF) shooting module; the method comprises the following steps:
simultaneously acquiring a first RGB image and a first depth image of a target object by utilizing the RGB shooting module and the TOF shooting module; the first RGB image is an RGB image in a full resolution mode or a preset mode, and the first depth image is a depth image in the full resolution mode or the preset mode; the preset modes comprise a preset resolution mode, a preset mirror mode and a preset resolution and mirror mode;
acquiring a first registration parameter obtained by performing binocular calibration on the RGB shooting module and the TOF shooting module in the full-resolution mode; and registering the first RGB image and the first depth image based on the image conversion relation between the full resolution mode and the preset mode and the first registration parameter to obtain a registration depth RGBD image in the preset mode.
2. The method according to claim 1, wherein the first RGB image is an RGB image in the full resolution mode, the first depth image is a depth image in the full resolution mode, and the preset mode is the preset resolution mode; the registering the first RGB image and the first depth image based on the image conversion relationship between the full resolution mode and the preset mode and the first registration parameter to obtain the RGBD image in the preset mode includes:
registering the first RGB image and the first depth image based on the first registration parameter to obtain a first RGBD image in the full resolution mode;
converting the resolution of the first RGBD image based on an image conversion relationship between the full resolution mode and the preset resolution mode to convert the first RGBD image into an RGBD image in the preset resolution mode.
3. The method of claim 1, wherein the first RGB image is an RGB image in the full resolution mode, the first depth image is a depth image in the preset resolution mode, and the preset mode is the preset resolution mode; the registering the first RGB image and the first depth image based on the image conversion relationship between the full resolution mode and the preset mode and the first registration parameter to obtain the RGBD image in the preset mode includes:
determining an image mapping relationship between the full resolution mode and the preset resolution mode based on an image conversion relationship between the full resolution mode and the preset mode;
determining the coordinate position of each pixel point in the first depth image in the full-resolution mode according to the image mapping relation;
acquiring a first pixel point corresponding to the coordinate position from the first RGB image; and registering the first pixel point with a corresponding pixel point in the first depth image based on the first registration parameter to obtain an RGBD image in the preset resolution mode.
4. The method of claim 1, wherein the first RGB image is an RGB image in the predetermined resolution mode, the first depth image is a depth image in the predetermined resolution mode, and the predetermined mode is the predetermined resolution mode; the registering the first RGB image and the first depth image based on the image conversion relationship between the full resolution mode and the preset mode and the first registration parameter to obtain the RGBD image in the preset mode includes:
processing the first registration parameter based on the image conversion relation between the full resolution mode and the preset mode to obtain a second registration parameter corresponding to the preset resolution mode;
and registering the first RGB image and the first depth image based on the second registration parameter to obtain a registered depth image in the preset resolution mode.
5. The method according to claim 1, wherein the first RGB image is an RGB image in the predetermined mirror mode, the first depth image is a depth image in the predetermined resolution and mirror mode, and the predetermined mode is the predetermined resolution and mirror mode; before the registering the first RGB image and the first depth image based on the image conversion relationship between the full resolution mode and the preset mode and the first registration parameter to obtain the RGBD image in the preset mode, the method further includes:
carrying out inverse mirror image on the first RGB image to obtain a second RGB image; carrying out inverse mirror image on the first depth image to obtain a second depth image;
correspondingly, the registering the first RGB image and the first depth image based on the image conversion relationship between the full resolution mode and the preset mode and the first registration parameter to obtain the RGBD image in the preset mode includes:
registering the second RGB image and the second depth image based on the image conversion relation between the full resolution mode and the preset mode and the first registration parameter to obtain an RGBD image in the preset mode;
correspondingly, after the registering the first RGB image and the first depth image based on the image conversion relationship between the full resolution mode and the preset mode and the first registration parameter to obtain the RGBD image in the preset mode, the method further includes:
and mirroring the RGBD image to obtain a mirrored RGBD image under the preset resolution and mirroring mode.
6. The method according to claim 1, wherein the first RGB image is an RGB image in the predetermined mirror mode, the first depth image is a depth image in the predetermined resolution and mirror mode, and the predetermined mode is the predetermined resolution and mirror mode; the registering the first RGB image and the first depth image based on the image conversion relationship between the full resolution mode and the preset mode and the first registration parameter to obtain the RGBD image in the preset mode includes:
mirroring the first registration parameter according to a mirroring rule corresponding to the preset mirroring mode to obtain a first mirroring registration parameter;
and registering the first RGB image and the first depth image based on the image conversion relation between the full resolution mode and the preset mode and the first mirror image registration parameter to obtain a mirror image RGBD image in the preset resolution and mirror image mode.
7. The method of claim 1, wherein the first registration parameters include internal parameters of the RGB capture module, internal parameters of the TOF capture module, and relative position parameters between the RGB capture module and the TOF capture module.
8. An image registration apparatus, characterized in that it comprises an RGB capture module and a time of flight TOF capture module; the device comprises:
the acquisition module is used for simultaneously acquiring a first RGB image and a first depth image of a target object by utilizing the RGB shooting module and the TOF shooting module; the first RGB image is an RGB image in a full resolution mode or a preset mode, and the first depth image is a depth image in the full resolution mode or the preset mode; the preset modes comprise a preset resolution mode, a preset mirror mode and a preset resolution and mirror mode;
the acquisition module is used for acquiring a first registration parameter obtained by performing binocular calibration on the RGB shooting module and the TOF shooting module in the full-resolution mode; and registering the first RGB image and the first depth image based on the image conversion relation between the full resolution mode and the preset mode and the first registration parameter to obtain a registration depth RGBD image in the preset mode.
9. An image registration apparatus, characterized in that the apparatus comprises: a processor, a memory, and a communication bus; the processor, when executing the execution program stored in the memory, implements the method of any of claims 1-7.
10. A storage medium on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-7.
CN202210786406.1A 2022-07-06 2022-07-06 Image registration method and device and storage medium Active CN114862923B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210786406.1A CN114862923B (en) 2022-07-06 2022-07-06 Image registration method and device and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210786406.1A CN114862923B (en) 2022-07-06 2022-07-06 Image registration method and device and storage medium

Publications (2)

Publication Number Publication Date
CN114862923A CN114862923A (en) 2022-08-05
CN114862923B true CN114862923B (en) 2022-09-09

Family

ID=82626789

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210786406.1A Active CN114862923B (en) 2022-07-06 2022-07-06 Image registration method and device and storage medium

Country Status (1)

Country Link
CN (1) CN114862923B (en)

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102012109481A1 (en) * 2012-10-05 2014-04-10 Faro Technologies, Inc. Device for optically scanning and measuring an environment
CN106663320A (en) * 2014-07-08 2017-05-10 高通股份有限公司 Systems and methods for stereo depth estimation using global minimization and depth interpolation
CN108230437A (en) * 2017-12-15 2018-06-29 深圳市商汤科技有限公司 Scene reconstruction method and device, electronic equipment, program and medium
KR101889886B1 (en) * 2017-12-22 2018-08-21 세명대학교 산학협력단 Depth information generating method and apparatus
CN109816731A (en) * 2017-11-21 2019-05-28 西安交通大学 A kind of method of RGB and depth information accuracy registration
CN112085771A (en) * 2020-08-06 2020-12-15 深圳市优必选科技股份有限公司 Image registration method and device, terminal equipment and computer readable storage medium
CN112232362A (en) * 2020-11-04 2021-01-15 清华大学深圳国际研究生院 Cross-modal medical image registration method and computer-readable storage medium
CN112465916A (en) * 2020-11-27 2021-03-09 浙江光珀智能科技有限公司 RGBD binocular calibration method and system based on full-view-field plane calibration plate

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110211749A1 (en) * 2010-02-28 2011-09-01 Kar Han Tan System And Method For Processing Video Using Depth Sensor Information
US9497429B2 (en) * 2013-03-15 2016-11-15 Pelican Imaging Corporation Extended color processing on pelican array cameras

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102012109481A1 (en) * 2012-10-05 2014-04-10 Faro Technologies, Inc. Device for optically scanning and measuring an environment
CN106663320A (en) * 2014-07-08 2017-05-10 高通股份有限公司 Systems and methods for stereo depth estimation using global minimization and depth interpolation
CN109816731A (en) * 2017-11-21 2019-05-28 西安交通大学 A kind of method of RGB and depth information accuracy registration
CN108230437A (en) * 2017-12-15 2018-06-29 深圳市商汤科技有限公司 Scene reconstruction method and device, electronic equipment, program and medium
KR101889886B1 (en) * 2017-12-22 2018-08-21 세명대학교 산학협력단 Depth information generating method and apparatus
CN112085771A (en) * 2020-08-06 2020-12-15 深圳市优必选科技股份有限公司 Image registration method and device, terminal equipment and computer readable storage medium
CN112232362A (en) * 2020-11-04 2021-01-15 清华大学深圳国际研究生院 Cross-modal medical image registration method and computer-readable storage medium
CN112465916A (en) * 2020-11-27 2021-03-09 浙江光珀智能科技有限公司 RGBD binocular calibration method and system based on full-view-field plane calibration plate

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
Extrinsic and depth calibration of ToF-cameras;Fuchs S et al;《IEEE conference on Computer Vison & Pattern Recognition》;20081231;全文 *
基于TOF与立体匹配相融合的高分辨率深度获取;刘娇丽等;《信息技术》;20161231;第40卷(第12期);全文 *

Also Published As

Publication number Publication date
CN114862923A (en) 2022-08-05

Similar Documents

Publication Publication Date Title
JP6722323B2 (en) System and method for imaging device modeling and calibration
US9600859B2 (en) Image processing device, image processing method, and information processing device
US8482599B2 (en) 3D modeling apparatus, 3D modeling method, and computer readable medium
US10638109B2 (en) Method for the FPGA-based long range multi-view stereo with differential image rectification
CN109474780B (en) Method and device for image processing
US20100103175A1 (en) Method for generating a high-resolution virtual-focal-plane image
CN109920004B (en) Image processing method, device, calibration object combination, terminal equipment and calibration system
US11282232B2 (en) Camera calibration using depth data
JP5704975B2 (en) Image processing apparatus, image processing method, and program
JP2003254748A (en) Stereo image characteristic inspection system
CN107545586B (en) Depth obtaining method and system based on light field polar line plane image local part
US20110064298A1 (en) Apparatus for evaluating images from a multi camera system, multi camera system and process for evaluating
CN109661815A (en) There are the robust disparity estimations in the case where the significant Strength Changes of camera array
JP2000283753A (en) Device for measuring distance using stereographic picture
KR20150088583A (en) Method and apparatus for image processing
CN114862923B (en) Image registration method and device and storage medium
JP6153318B2 (en) Image processing apparatus, image processing method, image processing program, and storage medium
JP6732440B2 (en) Image processing apparatus, image processing method, and program thereof
CN108307179A (en) A kind of method of 3D three-dimensional imagings
KR101857977B1 (en) Image apparatus for combining plenoptic camera and depth camera, and image processing method
JP7369588B2 (en) Imaging equipment and imaging method
JP2018148513A (en) Image processing system, image processing method, and program
JPH11150741A (en) Three-dimensional picture displaying method and its device by stereo photographing
CN110647301B (en) Augmented reality system and image display method thereof
CN115988343B (en) Image generation method, device and readable storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant