CN115205128A - Depth camera temperature drift correction method, system, equipment and medium based on structured light - Google Patents
Depth camera temperature drift correction method, system, equipment and medium based on structured light Download PDFInfo
- Publication number
- CN115205128A CN115205128A CN202110386281.9A CN202110386281A CN115205128A CN 115205128 A CN115205128 A CN 115205128A CN 202110386281 A CN202110386281 A CN 202110386281A CN 115205128 A CN115205128 A CN 115205128A
- Authority
- CN
- China
- Prior art keywords
- image
- depth camera
- checkerboard
- temperature drift
- structured light
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000012937 correction Methods 0.000 title claims description 36
- 238000003384 imaging method Methods 0.000 claims description 13
- 238000012545 processing Methods 0.000 claims description 10
- 238000000605 extraction Methods 0.000 claims description 3
- 238000003702 image correction Methods 0.000 claims description 3
- 238000012360 testing method Methods 0.000 claims description 3
- 238000005259 measurement Methods 0.000 abstract description 12
- 230000003287 optical effect Effects 0.000 description 9
- 238000010586 diagram Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000691 measurement method Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/80—Geometric correction
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/22—Measuring arrangements characterised by the use of optical techniques for measuring depth
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/24—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures
- G01B11/25—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B21/00—Measuring arrangements or details thereof, where the measuring technique is not covered by the other groups of this subclass, unspecified or not relevant
- G01B21/02—Measuring arrangements or details thereof, where the measuring technique is not covered by the other groups of this subclass, unspecified or not relevant for measuring length, width, or thickness
- G01B21/04—Measuring arrangements or details thereof, where the measuring technique is not covered by the other groups of this subclass, unspecified or not relevant for measuring length, width, or thickness by measuring coordinates of points
- G01B21/045—Correction of measurements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20112—Image segmentation details
- G06T2207/20164—Salient point detection; Corner detection
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Theoretical Computer Science (AREA)
- Length Measuring Devices By Optical Means (AREA)
Abstract
The invention provides a method, a system, equipment and a medium for correcting the temperature drift of a depth camera based on structured light, which comprises the following steps: acquiring a plurality of checkerboard images, wherein the checkerboard images are acquired at a plurality of temperatures through a depth camera; acquiring a reference checkerboard image, and extracting angular points in the checkerboard image and the reference checkerboard image; determining the corner point corresponding relation between each checkerboard image under different temperatures and a reference checkerboard image, further determining a first position difference value of each corner point in the checkerboard image and the corresponding corner point in the checkerboard image under different temperatures, and establishing a first temperature drift compensation model according to the first position difference value; and acquiring a target speckle image, correcting the target speckle image through a first temperature drift compensation model, and collecting the target speckle image through a depth camera. The invention can correct the target speckle images of the depth camera at different temperatures, and can ensure that the depth camera keeps higher measurement precision in different working environments.
Description
Technical Field
The invention relates to a 3D imaging technology, in particular to a depth camera temperature drift correction method, a depth camera temperature drift correction system, depth camera temperature drift correction equipment and a depth camera temperature drift correction medium based on structured light.
Background
In recent years, with the development of the consumer electronics industry, depth cameras with depth sensing function are receiving increasing attention from the consumer electronics world. The current well established depth measurement method is the structured light approach.
The structured light three scheme is based on the optical triangulation measurement principle. The optical projector projects the structured light with a certain mode on the surface of the object to form a light bar three-dimensional image modulated by the surface shape of the object to be measured on the surface. The three-dimensional image is detected by a camera at another location to obtain a two-dimensional distorted image of the light bar. The degree of distortion of the light bar depends on the relative position between the optical projector and the camera and the object surface profile (height). Intuitively, the displacement (or offset) displayed along the bar is proportional to the height of the object surface, the kink indicates a change in plane, and the discontinuity indicates a physical gap in the surface. When the relative position between the optical projector and the camera is fixed, the three-dimensional profile of the object surface can be reproduced by the distorted two-dimensional light bar image coordinates.
The depth camera is widely applied to the fields of face recognition, three-dimensional modeling, gesture recognition and the like. In recent years, with the application of depth cameras in various fields, the measurement accuracy of depth cameras is receiving more and more attention. However, the projection light source, the optical device, the optical structure, the image sensor, and the like of the 3D structured light camera are easily affected by temperature, so that the speckle position is changed. Because the reference image is pre-stored, the reference image cannot be adjusted in time according to the change of the speckle position, the depth measurement precision is reduced, even the matching fails, and the depth reconstruction cannot be completed.
Disclosure of Invention
In view of the defects in the prior art, the present invention provides a method, a system, a device and a medium for correcting a depth camera temperature drift based on structured light.
The structured light-based depth camera temperature drift correction method provided by the invention comprises the following steps:
step S1: acquiring a plurality of checkerboard images, wherein the checkerboard images are acquired by a depth camera at a plurality of temperatures;
step S2: acquiring a reference checkerboard image, and extracting angular points in the checkerboard image and the reference checkerboard image;
and step S3: determining the corresponding relation of the corner points of each checkerboard image and the reference checkerboard image at different temperatures, further determining a first position difference value of each corner point in the checkerboard image and the corresponding corner point in the checkerboard image at different temperatures, and establishing a first temperature drift compensation model according to the first position difference value;
and step S4: and acquiring a target speckle image, and correcting the target speckle image through the first temperature drift compensation model, wherein the target speckle image is acquired through a depth camera.
Preferably, the method further comprises the following steps:
step S5: acquiring a preset structured light reference image and a plurality of speckle images, wherein the speckle images are acquired by a depth camera at a plurality of temperatures;
step S6: extracting characteristic points in the speckle images and the structural light reference images, and determining the corresponding relation of the characteristic points of each speckle image and the structural light reference image at different temperatures;
step S7: generating a second position difference value of each characteristic point in the structured light reference image and the corresponding characteristic point in the speckle image at different temperatures, and establishing a second temperature drift compensation model according to the second position difference value;
step S8: and acquiring the real-time temperature of a structured light projection module in the depth camera, and correcting the structured light reference image according to the second temperature drift compensation model and the real-time temperature.
Preferably, the method further comprises the following steps:
step S9: and carrying out depth reconstruction according to the corrected structured light reference image and the target speckle image to generate a depth image.
Preferably, the depth camera includes:
the structured light projection module is used for projecting a laser speckle pattern to a target person;
the image imaging module is used for receiving the laser speckle pattern reflected by the target person to generate a target speckle pattern;
the temperature sensor module is used for testing the temperatures of the structured light projection module and the image receiving module in real time;
and the processor module is used for correcting the target speckle pattern according to the temperatures of the first temperature drift compensation model and the image receiving module, correcting a preset structured light reference image according to the temperatures of the second temperature drift compensation model and the structured light projection module, and performing depth reconstruction according to the corrected target speckle image and the corrected structured light reference image to generate a depth image.
Preferably, the step S1 includes the steps of:
step S101: at the operating temperature range t of the depth camera 1 ～t n Taking n temperatures at equal intervals, t 1 、t 2 、…、t n ；
Step S102: shielding the structured light projection module of the depth camera, putting the depth camera into a high-low temperature box, and putting the checkerboard plane in front of the depth camera by a distance d r And the surface of the structured light projection module is vertical to the axial direction of the structured light projection module;
step S103: the temperature is controlled by the high-low temperature box to be stabilized at t respectively 1 、t 2 、…、t n Then, respectively collecting checkerboard images g through the image imaging module of the depth camera 1 、g 2 、…、g n And simultaneously respectively recording temperature values t 'of the temperature sensors' 1 、t′ 2 、…、t′ n 。
Preferably, the step S3 includes the steps of:
step S301: determining the corresponding relation of the corner points of each checkerboard image and the reference checkerboard image at different temperatures;
step S302: respectively calculating the checkerboard pattern g 1 、g 2 、…、g n The position difference dx of each matching corner point (x, y) in the reference checkerboard image 1 、dx 2 、…、dx n And dy 1 、dy 2 、…、dy n ；
Step S303: fitting each matching corner (x, y) with the position difference to generate the first temperature drift compensation model, wherein the first temperature drift compensation model is expressed as: dx = f x (x，y，t)、dy＝f y (x, y, t), t is the temperature value of the temperature sensor.
Preferably, the step S5 includes the steps of:
step S501: at the operating temperature range t of the depth camera 1 ～t n Taking n temperatures at equal intervals, t 1 、t 2 、…、t n ；
Step S502: placing the depth camera into a high-low temperature box, and placing the white board plane in front of the depth camera by a distance d r And having its surface perpendicular to the axis of the structured light projection module;
step S503: the temperature is controlled by the high-low temperature box to be stabilized at t respectively 1 、t 2 、…、t n Then, speckle images s are respectively collected through an image imaging module of the depth camera 1 、s 2 、…、s n And simultaneously respectively recording temperature values t ″' of the temperature sensors 1 、t″ 2 、…、t″ n 。
Preferably, the step S7 includes the steps of:
step S701: according to the first temperature drift compensation model, speckle images s are aligned 1 、s 2 、…、s n Respectively correcting to generate speckle images s' 1 、s′ 2 、…、s′ n ；
Step S702: respectively calculating speckle images s 'according to the structured light reference image' 1 、s′ 2 、…、s′ n A second position difference du from each matching feature point (u, v) in the structured light reference image 1 、du 2 、…、du n And dv 1 、dv 2 、…、dv n ；
Step S703: fitting each matching feature point (u, v) to the second position difference to generate the second temperature drift compensation model, which is expressed as: du = f u (u，v，t)、dv＝f v (u, v, t), t is the temperature value of the temperature sensor.
The depth camera temperature drift correction system based on the structured light is characterized by comprising the following modules:
the system comprises an image acquisition module, a depth camera and a processing module, wherein the image acquisition module is used for acquiring a plurality of checkerboard images, and the checkerboard images are acquired through the depth camera at a plurality of temperatures;
the angular point extraction module is used for acquiring a reference checkerboard image and extracting angular points in the checkerboard image and the reference checkerboard image;
the model establishing module is used for determining the corner point corresponding relation between each checkerboard image at different temperatures and the reference checkerboard image, determining a first position difference value between each corner point in the checkerboard image and the corresponding corner point in the checkerboard image at different temperatures, and establishing a first temperature drift compensation model according to the first position difference value;
and the image correction module is used for acquiring a target speckle image, correcting the target speckle image through the first temperature drift compensation model, and acquiring the target speckle image through a depth camera.
According to the invention, the structured light-based depth camera temperature drift correction equipment comprises:
a processor;
a memory having stored therein executable instructions of the processor;
wherein the processor is configured to perform the steps of the structured light based depth camera temperature drift correction method via execution of the executable instructions.
According to the present invention, a computer readable storage medium is provided for storing a program which, when executed, performs the steps of the structured light based depth camera temperature drift correction method.
Compared with the prior art, the invention has the following beneficial effects:
according to the invention, the first position difference between each corner point in the checkerboard image and the corresponding corner point in the checkerboard image at different temperatures is determined by comparing the corner points in the checkerboard image acquired at multiple temperatures with the preset reference checkerboard image, and then a first temperature drift compensation model is established, so that the target speckle image acquired by the depth camera can be corrected at different temperatures, and the depth camera can keep higher measurement accuracy under different working environments;
according to the invention, speckle images collected at a plurality of temperatures are compared with characteristic points in a preset structural light reference image, a second position difference value of each characteristic point in the speckle image and a corresponding characteristic point in the structural light reference image at different temperatures is determined, and a second temperature drift compensation model is further established, so that the structural light reference image can be corrected at different temperatures, and the depth camera can keep higher measurement accuracy under different working environments;
according to the invention, the first temperature drift compensation model and the second temperature drift compensation model are obtained through calibration, and the target speckle image and the structured light reference image are respectively corrected by utilizing the first temperature drift compensation model and the second temperature drift compensation model, so that the depth camera can keep higher measurement accuracy under different working environments.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts. Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments with reference to the following drawings:
FIG. 1 is a flowchart illustrating a step of temperature drift correction by a first temperature drift compensation model according to an embodiment of the present invention;
FIG. 2 is a flowchart illustrating a step of temperature drift correction by a second temperature drift compensation model according to an embodiment of the present invention;
FIG. 3 is a flowchart illustrating steps of a method for correcting a temperature drift of a depth camera based on structured light according to an embodiment of the present invention;
FIG. 4 is a block diagram of a system for correcting a temperature drift of a depth camera based on structured light according to an embodiment of the present invention;
FIG. 5 is a schematic structural diagram of a depth camera calibration apparatus according to an embodiment of the present invention; and
fig. 6 is a schematic structural diagram of a computer-readable storage medium according to an embodiment of the present invention.
Detailed Description
The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the invention, but are not intended to limit the invention in any way. It should be noted that variations and modifications can be made by persons skilled in the art without departing from the concept of the invention. All falling within the scope of the present invention.
The terms "first," "second," "third," "fourth," and the like in the description and in the claims, as well as in the drawings, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are, for example, capable of operation in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
The technical solution of the present invention will be described in detail below with specific examples. These several specific embodiments may be combined with each other below, and details of the same or similar concepts or processes may not be repeated in some embodiments.
The invention provides a depth camera temperature drift correction method based on structured light, and aims to solve the problems in the prior art.
The following describes the technical solution of the present invention and how to solve the above technical problems with specific embodiments. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present invention will be described below with reference to the accompanying drawings.
Fig. 1 is a flowchart of a step of performing temperature drift correction by using a first temperature drift compensation model in an embodiment of the present invention, and as shown in fig. 1, the method for correcting the temperature drift of a depth camera based on structured light according to the present invention includes the following steps:
step S1: acquiring a plurality of checkerboard images, wherein the checkerboard images are acquired at a plurality of temperatures at equal intervals through a depth camera;
in the embodiment of the present invention, the step S1 includes the following steps:
step S101: at the operating temperature range t of the depth camera 1 ～t n Taking n temperatures at equal intervals, t 1 、t 2 、…、t n Wherein n is a natural number;
step S102: shielding the structured light projection module of the depth camera, putting the depth camera into a high-low temperature box, and putting the checkerboard plane in front of the depth camera by a distance d r And the surface of the structured light projection module is vertical to the axial direction of the structured light projection module;
step S103: the temperature is controlled by the high-low temperature box to be stabilized at t respectively 1 、t 2 、…、t n Then, the checkerboard images g are respectively collected through the image imaging module of the depth camera 1 、g 2 、…、g n And simultaneously respectively recording temperature values t 'of the temperature sensors' 1 、t′ 2 、…、t′ n 。
Step S2: acquiring a reference checkerboard image, and extracting angular points in the checkerboard image and the reference checkerboard image;
and step S3: determining the corresponding relation of the corner points of each checkerboard image and the reference checkerboard image at different temperatures, further determining a first position difference value of each corner point in the checkerboard image and the corresponding corner point in the checkerboard image at different temperatures, and establishing a first temperature drift compensation model according to the first position difference value;
in the embodiment of the present invention, the reference checkerboard image is a checkerboard image acquired by the image imaging module in a normal temperature environment, and the normal temperature environment is 25 ℃.
In the embodiment of the present invention, the step S3 includes the following steps:
step S301: determining the corresponding relation of the corner points of each checkerboard image and the reference checkerboard image at different temperatures;
step S302: respectively calculating the checkerboard pattern g 1 、g 2 、…、g n The position difference dx of each matching corner point (x, y) in the reference checkerboard image 1 、dx 2 、…、dx n And dy 1 、dy 2 、…、dy n ；
Step S303: fitting each matching corner (x, y) with the position difference to generate the first temperature drift compensation model, wherein the first temperature drift compensation model is expressed as: dx = f x (x，y，t)、dy＝f y (x, y, t), t is the temperature value of the temperature sensor.
In the embodiment of the invention, wherein (x, y) can adopt pixel coordinates, and f can be obtained by utilizing the matched checkerboard corner point position difference data under different temperatures through a least square method x (x，y，t)、f y The temperature drift error introduced by the image imaging module can be corrected by utilizing the first temperature drift compensation modelIs positive.
And step S4: and acquiring a target speckle image, and correcting the target speckle image through the first temperature drift compensation model, wherein the target speckle image is acquired through a depth camera.
In the embodiment of the invention, the structured light reference image is a checkerboard image acquired by the image imaging module in a normal-temperature environment, wherein the normal-temperature environment is 25 ℃.
Fig. 2 is a flowchart of a step of performing temperature drift correction by using a second temperature drift compensation model in the embodiment of the present invention, and as shown in fig. 2, the method for correcting temperature drift of a depth camera based on structured light according to the present invention further includes the following steps:
step S5: acquiring a preset structured light reference image and a plurality of speckle images, wherein the speckle images are acquired by a depth camera at a plurality of temperatures;
in the embodiment of the present invention, the step S5 includes the following steps:
step S501: at the operating temperature range t of the depth camera 1 ～t n Taking n temperatures at equal intervals, t 1 、t 2 、…、t n ；
Step S502: placing the depth camera into a high-low temperature box, and placing a white board plane in front of the depth camera by a distance d r And having its surface perpendicular to the axis of the structured light projection module;
step S503: the temperature is controlled by the high-low temperature box to be stabilized at t respectively 1 、t 2 、…、t n Then, speckle images s are respectively collected through an image imaging module of the depth camera 1 、s 2 、…、s n And simultaneously respectively recording the temperature values t ″' of the temperature sensors 1 、t″ 2 、…、t″ n 。
Step S6: extracting characteristic points in the speckle images and the structural light reference images, and determining the corresponding relation of the characteristic points of each speckle image and the structural light reference image at different temperatures;
step S7: generating a second position difference value of each characteristic point in the structured light reference image and the corresponding characteristic point in the speckle image at different temperatures, and establishing a second temperature drift compensation model according to the second position difference value;
in the embodiment of the present invention, the step S7 includes the following steps:
step S701: according to the first temperature drift compensation model, speckle images s are aligned 1 、s 2 、…、s n Respectively correcting to generate speckle images s' 1 、s′ 2 、…、s′ n ；
Step S702: respectively calculating speckle images s 'according to the structured light reference image' 1 、s′ 2 、…、s′ n A second position difference du from each matching feature point (u, v) in the structured light reference image 1 、du 2 、…、du n And dv 1 、dv 2 、…、dv n ；
Step S703: fitting each of the matched feature points (u, v) to the second position difference to generate the second temperature drift compensation model, which is expressed as: du = f u (u，v，t)、dv＝f v (u, v, t), t is the temperature value of the temperature sensor.
Step S8: and acquiring the real-time temperature of a structured light projection module in the depth camera, and correcting the structured light reference image according to the second temperature drift compensation model and the real-time temperature.
In the embodiment of the invention, the temperature drift error of the structured light projection module is corrected through the second temperature drift compensation model.
Fig. 3 is a flowchart of steps of a method for correcting a temperature drift of a depth camera based on structured light according to an embodiment of the present invention, where as shown in the figure, the method for correcting a temperature drift of a depth camera based on structured light according to the present invention further includes the following steps:
step S9: and carrying out depth reconstruction according to the corrected structured light reference image and the target speckle image to generate a depth image.
In the embodiment of the invention, when the temperature drift correction is carried out on the depth camera, the temperature drift correction is carried out according to the real-time temperature t of the depth cameraThe second temperature drift compensation model corrects pixel positions (u, v) in the structural light reference image, and the position coordinates of the corrected pixel points are respectively expressed as: u' = u + f u (u，v，t)、v′＝v+f v (u, v, t). Respectively correcting the pixel positions (x, y) in the target speckle images through a first temperature drift compensation model, wherein the corrected coordinates are respectively expressed as: x' = x + f x (x，y，t)、y′＝y+f y (x，y，t)；
And then, performing depth calculation by using the pixel point position coordinates r (u ', v') in the corrected structured light reference image and the pixel point o (x ', y') in the corrected target speckle image.
In an embodiment of the present invention, the depth camera includes:
the structured light projection module is used for projecting a laser speckle pattern to a target person;
the image imaging module is used for receiving the laser speckle pattern reflected by the target person to generate a target speckle pattern;
the temperature sensor module is used for testing the temperatures of the structured light projection module and the image receiving module in real time;
and the processor module is used for correcting the target speckle pattern according to the temperatures of the first temperature drift compensation model and the image receiving module, correcting a preset structured light reference image according to the temperatures of the second temperature drift compensation model and the structured light projection module, and performing depth reconstruction according to the corrected target speckle image and the corrected structured light reference image to generate a depth image.
Fig. 4 is a schematic block diagram of a depth camera temperature drift correction system based on structured light according to an embodiment of the present invention, and as shown in fig. 4, the depth camera temperature drift correction system based on structured light according to the present invention includes the following modules:
the system comprises an image acquisition module, a depth camera and a processing module, wherein the image acquisition module is used for acquiring a plurality of checkerboard images, and the checkerboard images are acquired through the depth camera at a plurality of temperatures;
the angular point extraction module is used for acquiring a reference checkerboard image and extracting angular points in the checkerboard image and the reference checkerboard image;
the model establishing module is used for determining the corner point corresponding relation between each checkerboard image under different temperatures and the reference checkerboard image, determining a first position difference value between each corner point in the checkerboard image and the corresponding corner point in the checkerboard image under different temperatures, and establishing a first temperature drift compensation model according to the first position difference value;
and the image correction module is used for acquiring a target speckle image, correcting the target speckle image through the first temperature drift compensation model, and acquiring the target speckle image through a depth camera.
The embodiment of the invention also provides a depth camera temperature drift correction device based on the structured light, which comprises a processor. A memory having stored therein executable instructions of the processor. Wherein the processor is configured to perform the steps of the structured light based depth camera temperature drift correction method via execution of executable instructions.
As described above, in this embodiment, by comparing corner points in the checkerboard image collected at a plurality of temperatures with corner points in a preset reference checkerboard image, a first position difference between each corner point in the checkerboard image and a corresponding corner point in the checkerboard image at different temperatures is determined, and then a first temperature drift compensation model is established, so that the target speckle image collected by the depth camera can be corrected at different temperatures, and the depth camera can maintain high measurement accuracy in different working environments.
As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or program product. Thus, various aspects of the invention may be embodied in the form of: an entirely hardware embodiment, an entirely software embodiment (including firmware, microcode, etc.) or an embodiment combining hardware and software aspects that may all generally be referred to herein as a "circuit," module "or" platform.
FIG. 5 is a schematic structural diagram of a structured light-based depth camera temperature drift correction apparatus of the present invention. An electronic device 600 according to this embodiment of the invention is described below with reference to fig. 5. The electronic device 600 shown in fig. 5 is only an example and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 5, the electronic device 600 is embodied in the form of a general purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one memory unit 620, a bus 630 connecting the different platform components (including the memory unit 620 and the processing unit 610), a display unit 640, and the like.
Wherein the storage unit stores program code which is executable by the processing unit 610 such that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention as described in the above section of the structured light based depth camera temperature drift correction method of the present description. For example, processing unit 610 may perform the steps as shown in fig. 1.
The storage unit 620 may include readable media in the form of volatile storage units, such as a random access memory unit (RAM) 6201 and/or a cache storage unit 6202, and may further include a read-only memory unit (ROM) 6203.
The memory unit 620 may also include a program/utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may comprise an implementation of a network environment.
The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboard, pointing device, bluetooth device, etc.), with one or more devices that enable a user to interact with the electronic device 600, and/or with any devices (e.g., router, modem, etc.) that enable the electronic device 600 to communicate with one or more other computing devices. Such communication may occur via an input/output (I/O) interface 650. Also, the electronic device 600 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network such as the Internet) via the network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 via the bus 630. It should be appreciated that although not shown in FIG. 5, other hardware and/or software modules may be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, to name a few.
The embodiment of the invention also provides a computer readable storage medium for storing a program, and the program realizes the steps of the method for correcting the temperature drift of the depth camera based on the structured light when being executed. In some possible embodiments, the various aspects of the invention may also be implemented in the form of a program product comprising program code means for causing a terminal device to carry out the steps according to various exemplary embodiments of the invention as described in the above section of the structured light based depth camera temperature drift correction method of this specification, when the program product is run on the terminal device.
As shown above, when the program of the computer-readable storage medium of this embodiment is executed, by comparing corner points in the checkerboard images acquired at multiple temperatures with corner points in the preset reference checkerboard image, a first position difference between each corner point in the checkerboard image and a corresponding corner point in the checkerboard image at different temperatures is determined, and then a first temperature drift compensation model is established, so that the target speckle images acquired by the depth camera at different temperatures can be corrected, and the depth camera can maintain high measurement accuracy under different working environments.
Fig. 6 is a schematic structural diagram of a computer-readable storage medium of the present invention. Referring to fig. 6, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited in this regard and, in the present document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
A computer readable storage medium may include a propagated data signal with readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable storage medium may also be any readable medium that is not a readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
In the embodiment of the invention, the chessboard pattern images collected at a plurality of temperatures are compared with the angular points in the preset reference chessboard pattern image, the first position difference between each angular point in the chessboard pattern image and the corresponding angular point in the chessboard pattern images at different temperatures is determined, and then a first temperature drift compensation model is established, so that the target speckle images collected by the depth camera can be corrected at different temperatures, and the depth camera can keep higher measurement accuracy under different working environments; comparing speckle images acquired at a plurality of temperatures with characteristic points in a preset structural light reference image, determining a second position difference value of each characteristic point in the speckle images and the corresponding characteristic point in the structural light reference image at different temperatures, and further establishing a second temperature drift compensation model, so that the structural light reference image can be corrected at different temperatures, and the depth camera can keep higher measurement accuracy under different working environments; the first temperature drift compensation model and the second temperature drift compensation model are obtained through calibration, and the first temperature drift compensation model and the second temperature drift compensation model are used for respectively correcting the target speckle image and the structured light reference image, so that the depth camera can keep high measurement accuracy under different working environments.
The embodiments in the present description are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing description of specific embodiments of the present invention has been presented. It is to be understood that the present invention is not limited to the specific embodiments described above, and that various changes and modifications may be made by one skilled in the art within the scope of the appended claims without departing from the spirit of the invention.
Claims (10)
1. A depth camera temperature drift correction method based on structured light is characterized by comprising the following steps:
step S1: acquiring a plurality of checkerboard images, wherein the checkerboard images are acquired at a plurality of temperatures through a depth camera;
step S2: acquiring a reference checkerboard image, and extracting angular points in the checkerboard image and the reference checkerboard image;
and step S3: determining the corner point corresponding relation between each checkerboard image under different temperatures and the reference checkerboard image, further determining a first position difference value of each corner point in the checkerboard image and the corresponding corner point in the checkerboard image under different temperatures, and establishing a first temperature drift compensation model according to the first position difference value;
and step S4: and acquiring a target speckle image, and correcting the target speckle image through the first temperature drift compensation model, wherein the target speckle image is acquired through a depth camera.
2. The structured-light based depth camera temperature drift correction method of claim 1, further comprising the steps of:
step S5: acquiring a preset structured light reference image and a plurality of speckle images, wherein the speckle images are acquired by a depth camera at a plurality of temperatures;
step S6: extracting characteristic points in the speckle images and the structural light reference images, and determining the corresponding relation of the characteristic points of each speckle image and the structural light reference image at different temperatures;
step S7: generating a second position difference value of each characteristic point in the structured light reference image and the corresponding characteristic point in the speckle image at different temperatures, and establishing a second temperature drift compensation model according to the second position difference value;
step S8: and acquiring the real-time temperature of a structured light projection module in the depth camera, and correcting the structured light reference image according to the second temperature drift compensation model and the real-time temperature.
3. The structured-light based depth camera temperature drift correction method of claim 2, further comprising the steps of:
step S9: and carrying out depth reconstruction according to the corrected structured light reference image and the target speckle image to generate a depth image.
4. The structured-light based depth camera temperature drift correction method of claim 2, wherein the depth camera comprises:
the structured light projection module is used for projecting a laser speckle pattern to a target person;
the image imaging module is used for receiving the laser speckle pattern reflected by the target person to generate a target speckle pattern;
the temperature sensor module is used for testing the temperatures of the structured light projection module and the image receiving module in real time;
and the processor module is used for correcting the target speckle pattern according to the temperatures of the first temperature drift compensation model and the image receiving module, correcting a preset structured light reference image according to the temperatures of the second temperature drift compensation model and the structured light projection module, and performing depth reconstruction according to the corrected target speckle image and the corrected structured light reference image to generate a depth image.
5. The structured-light based depth camera temperature drift correction method of claim 1, wherein the step S1 comprises the steps of:
step S101: at the operating temperature range t of the depth camera 1 ～t n Taking n temperatures at equal intervals, t 1 、t 2 、…、t n ；
Step S102: shielding the structured light projection module of the depth camera, putting the depth camera into a high-low temperature box, and putting the checkerboard plane in front of the depth camera by a distance d r And the surface of the structured light projection module is vertical to the axial direction of the structured light projection module;
step S103: the temperature is controlled by the high-low temperature box to be stabilized at t respectively 1 、t 2 、…、t n Then, respectively collecting checkerboard images g through the image imaging module of the depth camera 1 、g 2 、…、g n And simultaneously respectively recording temperature values t 'of the temperature sensors' 1 、t′ 2 、…、t′ n 。
6. The structured-light based depth camera temperature drift correction method of claim 5, wherein the step S3 comprises the steps of:
step S301: determining the corresponding relation of the corner points of each checkerboard image and the reference checkerboard image at different temperatures;
step S302: respectively calculating the checkerboard ring image g 1 、g 2 、…、g n The position difference dx of each matching corner point (x, y) in the reference checkerboard image 1 、dx 2 、…、dx n And dy 1 、dy 2 、…、dy n ；
Step S303: fitting each matching corner (x, y) with the position difference to generate the first temperature drift compensation model, wherein the first temperature drift compensation model is expressed as: dx = f x (x，y，t)、dy＝f y (x, y, t), t is the temperature value of the temperature sensor.
7. The structured-light based depth camera temperature drift correction method of claim 2, wherein the step S5 comprises the steps of:
step S501: at the operating temperature range t of the depth camera 1 ～t n Taking n temperatures at equal intervals, t 1 、t 2 、…、t n ；
Step S502: placing the depth camera into a high-low temperature box, and placing a white board plane in front of the depth camera by a distance d r And the surface of the structured light projection module is vertical to the axial direction of the structured light projection module;
step S503: the temperature is controlled by the high-low temperature box to be stabilized at t respectively 1 、t 2 、…、t n Then, speckle images s are respectively collected through an image imaging module of the depth camera 1 、s 2 、…、s n And simultaneously respectively recording the temperature values t ″' of the temperature sensors 1 、t″ 2 、…、t″ n 。
8. A depth camera temperature drift correction system based on structured light is characterized by comprising the following modules:
the system comprises an image acquisition module, a depth camera and a processing module, wherein the image acquisition module is used for acquiring a plurality of checkerboard images, and the checkerboard images are acquired through the depth camera at a plurality of temperatures;
the angular point extraction module is used for acquiring a reference checkerboard image and extracting angular points in the checkerboard image and the reference checkerboard image;
the model establishing module is used for determining the corner point corresponding relation between each checkerboard image at different temperatures and the reference checkerboard image, determining a first position difference value between each corner point in the checkerboard image and the corresponding corner point in the checkerboard image at different temperatures, and establishing a first temperature drift compensation model according to the first position difference value;
and the image correction module is used for acquiring a target speckle image, correcting the target speckle image through the first temperature drift compensation model, and acquiring the target speckle image through a depth camera.
9. A structured light based depth camera temperature drift correction apparatus, comprising:
a processor;
a memory having stored therein executable instructions of the processor;
wherein the processor is configured to perform the steps of the structured light based depth camera temperature drift correction method of any one of claims 1 to 7 via execution of the executable instructions.
10. A computer readable storage medium storing a program, wherein the program when executed implements the steps of the structured light based depth camera temperature drift correction method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110386281.9A CN115205128A (en) | 2021-04-12 | 2021-04-12 | Depth camera temperature drift correction method, system, equipment and medium based on structured light |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110386281.9A CN115205128A (en) | 2021-04-12 | 2021-04-12 | Depth camera temperature drift correction method, system, equipment and medium based on structured light |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115205128A true CN115205128A (en) | 2022-10-18 |
Family
ID=83571423
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110386281.9A Pending CN115205128A (en) | 2021-04-12 | 2021-04-12 | Depth camera temperature drift correction method, system, equipment and medium based on structured light |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115205128A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116818129A (en) * | 2023-05-08 | 2023-09-29 | 广州图语信息科技有限公司 | Temperature estimation and thermal distortion correction method applied to structured light reconstruction |
| US11792374B1 (en) | 2022-11-15 | 2023-10-17 | Himax Technologies Limited | System and method for handling a thermal compensation |
| CN117409174A (en) * | 2023-12-14 | 2024-01-16 | 南昌虚拟现实研究院股份有限公司 | Speckle image temperature compensation method and device, readable medium and electronic equipment |
-
2021
- 2021-04-12 CN CN202110386281.9A patent/CN115205128A/en active Pending
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11792374B1 (en) | 2022-11-15 | 2023-10-17 | Himax Technologies Limited | System and method for handling a thermal compensation |
| CN116818129A (en) * | 2023-05-08 | 2023-09-29 | 广州图语信息科技有限公司 | Temperature estimation and thermal distortion correction method applied to structured light reconstruction |
| CN116818129B (en) * | 2023-05-08 | 2024-01-12 | 广州图语信息科技有限公司 | Temperature estimation and thermal distortion correction method applied to structured light reconstruction |
| CN117409174A (en) * | 2023-12-14 | 2024-01-16 | 南昌虚拟现实研究院股份有限公司 | Speckle image temperature compensation method and device, readable medium and electronic equipment |
| CN117409174B (en) * | 2023-12-14 | 2024-03-15 | 南昌虚拟现实研究院股份有限公司 | Speckle image temperature compensation method and device, readable medium and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110322500B (en) | Optimization method and device for instant positioning and map construction, medium and electronic equipment | |
| CN115205128A (en) | Depth camera temperature drift correction method, system, equipment and medium based on structured light | |
| CN111127422A (en) | Image annotation method, device, system and host | |
| US9858684B2 (en) | Image processing method and apparatus for calibrating depth of depth sensor | |
| CN109946680B (en) | External parameter calibration method and device of detection system, storage medium and calibration system | |
| US10977857B2 (en) | Apparatus and method of three-dimensional reverse modeling of building structure by using photographic images | |
| Xu et al. | A simple calibration method for structured light-based 3D profile measurement | |
| CN107784671B (en) | Method and system for visual instant positioning and drawing | |
| EP3633606B1 (en) | Information processing device, information processing method, and program | |
| JPH10124658A (en) | Method for correcting image distortion of camera by utilizing neural network | |
| WO2019029991A1 (en) | System and method for recalibrating a projector system | |
| CN112880687A (en) | Indoor positioning method, device, equipment and computer readable storage medium | |
| US20220180560A1 (en) | Camera calibration apparatus, camera calibration method, and nontransitory computer readable medium storing program | |
| CN114494388B (en) | Three-dimensional image reconstruction method, device, equipment and medium in large-view-field environment | |
| Wilm et al. | Accurate and simple calibration of DLP projector systems | |
| CN111368927A (en) | Method, device and equipment for processing labeling result and storage medium | |
| CN113587934A (en) | Robot, indoor positioning method and device and readable storage medium | |
| CN113610702B (en) | Picture construction method and device, electronic equipment and storage medium | |
| CN114463436A (en) | Calibration method, system, equipment and storage medium of galvanometer scanning device | |
| CN112927340B (en) | Three-dimensional reconstruction acceleration method, system and equipment independent of mechanical placement | |
| CN114049401A (en) | Binocular camera calibration method, device, equipment and medium | |
| CN113532266B (en) | Box volume measuring method, system, equipment and storage medium based on three-dimensional vision | |
| Bingwei et al. | Accurate geometric camera calibration technique using multi-views of a non-metric planar grid | |
| CN114945091B (en) | Temperature compensation method, device and equipment of depth camera and storage medium | |
| CN115205129A (en) | Depth camera based on structured light and method of use |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |