CN117132509B - Flying spot removing method, device, equipment and medium based on phase shift line shift fusion - Google Patents
Flying spot removing method, device, equipment and medium based on phase shift line shift fusion Download PDFInfo
- Publication number
- CN117132509B CN117132509B CN202311401878.1A CN202311401878A CN117132509B CN 117132509 B CN117132509 B CN 117132509B CN 202311401878 A CN202311401878 A CN 202311401878A CN 117132509 B CN117132509 B CN 117132509B
- Authority
- CN
- China
- Prior art keywords
- phase
- phase value
- pixel
- value
- shift
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000010363 phase shift Effects 0.000 title claims abstract description 72
- 230000004927 fusion Effects 0.000 title claims abstract description 34
- 238000000034 method Methods 0.000 title claims abstract description 31
- 238000012545 processing Methods 0.000 abstract description 15
- 238000003384 imaging method Methods 0.000 abstract description 7
- 238000005259 measurement Methods 0.000 abstract description 7
- 238000010586 diagram Methods 0.000 description 7
- 238000012986 modification Methods 0.000 description 7
- 230000004048 modification Effects 0.000 description 7
- 230000003287 optical effect Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 3
- 238000007500 overflow downdraw method Methods 0.000 description 3
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000035800 maturation Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10052—Images from lightfield camera
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/30—Assessment of water resources
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Length Measuring Devices By Optical Means (AREA)
Abstract
The invention provides a flying spot removing method, a device, equipment and a medium based on phase-shift line-shift fusion, which comprise the following steps: acquiring a phase shift structure light pattern and determining a first phase value of each pixel in the camera unit according to the phase shift structure light pattern; acquiring a line-shifting structured light pattern, and determining a second phase value corresponding to each pixel in the camera unit according to the line-shifting structured light pattern; and fusing the first phase value and the second phase value corresponding to each pixel to generate a target phase value of the pixel, and further generating a point cloud image with flying points removed. According to the invention, the phase shift structure light pattern and the line shift structure light pattern are respectively projected through the projector to generate the first phase value and the second phase value, the first phase value and the second phase value corresponding to each pixel are fused to generate the target phase value of the pixel, and then the depth reconstruction is carried out according to the target phase value to generate the point cloud image, so that flying points can be effectively removed, the imaging error of the depth camera is reduced, and the interference of flying point to point cloud processing and measurement is reduced.
Description
Technical Field
The invention relates to a depth camera, in particular to a flying spot removing method, a device, equipment and a storage medium based on phase shift line shift fusion.
Background
Common projector technologies include DLP (Digital Light Processing), linear laser coupled MEMS galvanometer or mechanical galvanometer, etc.
With the maturation of LED light sources and DLP technologies, DLP projectors are rapidly developed, and become a widely used projection mode. In 1987, the TI company invented a DMD device, which makes DLP digital light processing technology applied in the world, and further promotes the rise of DLP projectors. The DMD device is a binary pulse width modulated digital optical switch, which is currently the most complex optical switch device in the world. Thousands of tiny square mirrors are built on a hinge structure over sram to make up the DMD. Each lens may switch on and off one pixel of light. The hinge structure allows the lens to tilt between two states, +10 degrees "on" and-10 degrees "off.
In a depth map measured by a depth camera based on the DLP structured light principle, there are often a large number of erroneous depth measurement values at the edge of an object, and after a point cloud is generated, the point cloud visually appears as an invalid point flying in the air, which is called flying spot noise. Flying spot noise makes a depth camera unable to effectively acquire depth information of an object edge, causes an error of depth imaging, and causes interference to spot cloud processing and measurement.
Disclosure of Invention
Aiming at the defects in the prior art, the invention aims to provide a flying spot removing method, device, equipment and storage medium based on phase-shift line-shift fusion.
According to the flying spot removing method based on phase shift line shift fusion, which is provided by the invention, the flying spot removing method is used for a depth camera based on a structured light principle, and the depth camera comprises a projector and a camera unit, and comprises the following steps:
step S1: acquiring a phase shift structure light pattern, and determining a first phase value of each pixel in a camera unit according to the phase shift structure light pattern;
step S2: acquiring a line-shifting structured light pattern, and determining a second phase value corresponding to each pixel in the camera unit according to the line-shifting structured light pattern;
step S3: and fusing the first phase value and the second phase value corresponding to each pixel to generate a target phase value of the pixel, and further generating a point cloud image with flying points removed.
Preferably, the step S1 includes the steps of:
step S101: controlling a projector to project phase shift structured light to a target object;
step S102: controlling a camera unit to acquire a phase shift structure light pattern formed by reflecting the phase shift structure light by a target object;
step S103: a first phase value for each pixel in the camera unit is determined from the phase shifted structured light pattern.
Preferably, the step S2 includes the steps of:
step S201: controlling a projector to project linear moving structured light to a target object, wherein linear moving stripe beams in the linear moving structured light are coded by Gray codes;
step S202: controlling a camera unit to collect a line-shifting structured light pattern formed by reflecting the line-shifting structured light by a target object;
step S203: a second phase value for each pixel in the camera unit is determined from the line shifted structured light pattern.
Preferably, the step S3 includes the steps of:
step S301: judging whether the first phase value and the second phase value are invalid values or not, and determining that the target phase value corresponding to the pixel is the invalid value when any one of the first phase value and the second phase value is the invalid value;
step S302: generating a difference absolute value of the first phase value and the second phase value, and judging whether the difference absolute value is smaller than a preset difference threshold value or not;
step S303: and when the absolute value of the difference is smaller than a preset difference threshold value, determining that the target phase value corresponding to the pixel is a first phase value, otherwise, determining that the phase value corresponding to the pixel is an invalid value.
Preferably, the step S3 includes the steps of:
step S301: judging whether the first phase value and the second phase value are invalid values or not, and determining that the target phase value corresponding to the pixel is the invalid value when any one of the first phase value and the second phase value is the invalid value;
step S302: generating a difference absolute value of the first phase value and the second phase value, and judging whether the difference absolute value is smaller than a preset difference threshold value or not;
step S303: and when the absolute value of the difference is smaller than a preset difference threshold value, determining the target phase value corresponding to the pixel as a first phase value, otherwise, determining the target phase value corresponding to the pixel as one half of the sum of the first phase value and the second phase value.
Preferably, the stripe widths of the phase shift structure light pattern and the line shift structure light pattern in a single period are the same;
the stripe gray value of the phase shift structure light pattern is changed periodically in a sine way, the light stripe in the line shift structure light pattern accounts for 1/n of the stripe width, and n is a natural number larger than 2.
Preferably, the step S3 includes the steps of:
step S304: calculating to obtain a corresponding pixel coordinate value on the projector through the target phase value and the calibration information;
step S305: generating depth information of each pixel according to two pixel coordinate values matched with the camera unit and the projector;
step S306: and carrying out three-dimensional reconstruction according to the depth information of each pixel to generate the point cloud image.
The flying spot removing system based on phase-shift line-shift fusion is used for a depth camera based on a structured light principle, and the depth camera comprises a projector and a camera unit, and comprises the following modules:
a first phase generating module for acquiring a phase shift structured light pattern and determining a first phase value for each pixel in the camera unit according to the phase shift structured light pattern;
the second phase generation module is used for acquiring the line-shifting structure light pattern and determining a second phase value corresponding to each pixel in the camera unit according to the line-shifting structure light pattern;
and the point cloud generation module is used for fusing the first phase value and the second phase value corresponding to each pixel to generate a target phase value of the pixel, and further generating a point cloud image with flying spots removed.
The flying spot removing device based on phase-shift line-shift fusion provided by the invention comprises:
a processor;
a memory module having stored therein executable instructions of the processor;
wherein the processor is configured to perform the steps of the flying spot removal method based on phase-shift line-shift fusion via execution of the executable instructions.
According to the computer readable storage medium provided by the invention, the computer readable storage medium is used for storing a program, and the program is executed to realize the steps of the flying spot removing method based on phase shift line shift fusion.
Compared with the prior art, the invention has the following beneficial effects:
according to the invention, the phase shift structure light pattern and the line shift structure light pattern are respectively projected through the projector to generate the first phase value and the second phase value, the first phase value and the second phase value corresponding to each pixel are fused to generate the target phase value of the pixel, and then the depth reconstruction is carried out according to the target phase value to generate the point cloud image, so that flying spots can be effectively removed, the imaging error of the depth camera is reduced, and the interference of flying spot to point cloud processing and measurement is reduced.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art. Other features, objects and advantages of the present invention will become more apparent upon reading of the detailed description of non-limiting embodiments, given with reference to the accompanying drawings in which:
FIG. 1 is a flow chart of steps of a flying spot removal method based on phase shift line shift fusion in an embodiment of the invention;
FIG. 2 is a flowchart illustrating steps for generating a first coordinate value according to an embodiment of the present invention;
FIG. 3 is a flowchart illustrating a second coordinate value generation process according to an embodiment of the present invention;
FIG. 4 is a flowchart showing the phase fusion steps according to an embodiment of the present invention;
FIG. 5 is a flowchart showing the phase fusion steps in a modification of the present invention;
FIG. 6 is a flowchart illustrating steps for generating a point cloud image according to an embodiment of the present invention;
FIG. 7 is a schematic diagram of a phase shift structured light pattern and a line shift structured light pattern in an embodiment of the present invention;
FIG. 8 is a schematic diagram of a flying spot removal system based on phase shift line shift fusion in an embodiment of the invention;
FIG. 9 is a schematic diagram of a flying spot removal device based on phase shift line shift fusion in an embodiment of the invention; and
fig. 10 is a schematic diagram of a computer-readable storage medium according to an embodiment of the present invention.
Detailed Description
The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the present invention, but are not intended to limit the invention in any way. It should be noted that variations and modifications could be made by those skilled in the art without departing from the inventive concept. These are all within the scope of the present invention.
The terms "first," "second," "third," "fourth" and the like in the description and in the claims and in the above drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented, for example, in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
The technical scheme of the invention is described in detail below by specific examples. The following embodiments may be combined with each other, and some embodiments may not be repeated for the same or similar concepts or processes.
The following describes the technical scheme of the present invention and how the technical scheme of the present application solves the above technical problems in detail with specific embodiments. The following embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. Embodiments of the present invention will be described below with reference to the accompanying drawings.
Fig. 1 is a step flowchart of a flying spot removing method based on phase shift line shift fusion in an embodiment of the present invention, as shown in fig. 1, the flying spot removing method based on phase shift line shift fusion provided by the present invention is used for a depth camera based on a structured light principle, where the depth camera includes a projector and a camera unit, and includes the following steps:
step S1: acquiring a phase shift structure light pattern, and determining a first phase value of each pixel in a camera unit according to the phase shift structure light pattern;
in the embodiment of the invention, the projector adopts a DLP projector, the projector is used for projecting phase shift structure light, the camera unit is used for collecting a phase shift structure light pattern formed by the phase shift structure light on a target object, and the processor module is used for controlling the projector and the camera unit to synchronously switch and generating a point cloud image or a depth image of the target object according to the phase shift structure light pattern.
Fig. 2 is a flowchart of a step of generating a first coordinate value in an embodiment of the present invention, as shown in fig. 2, the step S1 includes the following steps:
step S101: controlling a projector to project phase shift structured light to a target object;
step S102: controlling a camera unit to acquire a phase shift structure light pattern formed by reflecting the phase shift structure light by a target object;
step S103: a first phase value for each pixel in the camera unit is determined from the phase shifted structured light pattern.
In an embodiment of the invention, phase-shifted structured light is projected in a direction orthogonal to the baseline, and for each camera pixel, an absolute phase value is calculated and can be mapped to the X-coordinate of the projector pixel. Based on the reconstruction method of the iterative line-plane intersection point, the 3D space point corresponding to the camera pixel can be calculated.
In the embodiment of the invention, when calculating the first coordinate value, the internal and external parameters of the projector and the internal and external parameters of the camera unit generated by calibrating the depth camera are required to be obtained in advance, and the first coordinate value is calculated and generated according to the absolute phase value and the internal and external parameters of the projector and the camera unit.
Step S2: acquiring a line-shifting structured light pattern, and determining a second phase value corresponding to each pixel in the camera unit according to the line-shifting structured light pattern;
in the embodiment of the invention, the projector adopts a DLP projector, the projector is used for projecting the linear moving structured light, the camera unit is used for collecting a linear moving structured light pattern formed by the linear moving structured light on the target object, and the processor module is used for controlling the synchronous switch of the projector and the camera unit and generating a point cloud image or a depth image of the target object according to the linear moving structured light pattern.
Fig. 3 is a flowchart of a step of generating a second coordinate value in an embodiment of the present invention, as shown in fig. 3, the step S2 includes the following steps:
step S201: controlling a projector to project linear moving structured light to a target object, wherein linear moving stripe beams in the linear moving structured light are coded by Gray codes;
step S202: controlling a camera unit to collect a line-shifting structured light pattern formed by reflecting the line-shifting structured light by a target object;
step S203: a second phase value for each pixel in the camera unit is determined from the line shifted structured light pattern.
Step S3: and fusing the first phase value and the second phase value corresponding to each pixel to generate a target phase value of the pixel, and further generating a point cloud image with flying points removed.
Fig. 4 is a flowchart of a phase fusion step in an embodiment of the present invention, as shown in fig. 4, the step S3 includes the following steps:
step S301: judging whether the first phase value and the second phase value are invalid values or not, and determining that the target phase value corresponding to the pixel is the invalid value when any one of the first phase value and the second phase value is the invalid value;
step S302: generating a difference absolute value of the first phase value and the second phase value, and judging whether the difference absolute value is smaller than a preset difference threshold value or not;
step S303: and when the absolute value of the difference is smaller than a preset difference threshold value, determining that the target phase value corresponding to the pixel is a first phase value, otherwise, determining that the phase value corresponding to the pixel is an invalid value.
Fig. 5 is a flowchart of a phase fusion step in a modification of the present invention, as shown in fig. 5, the step S3 includes the following steps:
step S301: judging whether the first phase value and the second phase value are invalid values or not, and determining that the target phase value corresponding to the pixel is the invalid value when any one of the first phase value and the second phase value is the invalid value;
step S302: generating a difference absolute value of the first phase value and the second phase value, and judging whether the difference absolute value is smaller than a preset difference threshold value or not;
step S303: and when the absolute value of the difference is smaller than a preset difference threshold value, determining the target phase value corresponding to the pixel as a first phase value, otherwise, determining the target phase value corresponding to the pixel as one half of the sum of the first phase value and the second phase value.
TABLE 1
| Test item | 8-step line shift | 8-step phase shift | Fusion method in the embodiment of the invention | Fusion method in modification of the present invention |
| Number of projected patterns | 27 | 19 | 27 | 27 |
| Effective point number 1 (precision plate) | 76.83 thousands | 77.17 ten thousand | 76.61 ten thousand | 76.88 ten thousand |
| Effective point number 2 (calibration plate) | 12.71 thousands of | 12.74 ten thousand | 11.94 ten thousand | 12.62 thousands |
| Flatness 1 (precision board) | 0.379mm | 0.363mm | 0.363mm | 0.363mm |
| Flatness 2 (calibration plate) | 0.730mm | 0.709mm | 0.600mm | 0.606mm |
| Bounding box height (calibration plate) | 13.041mm | 9.107mm | 6.427mm | 10.524mm |
As shown in table 1, the fusion method in the embodiment and the modification of the present invention can effectively improve the imaging precision of the depth camera.
FIG. 7 is a schematic diagram of a phase shift structure light pattern and a line shift structure light pattern according to an embodiment of the present invention, as shown in FIG. 7, wherein (a) is a phase shift structure light pattern and (b) is a line shift structure light pattern, and stripe widths of the phase shift structure light pattern and the line shift structure light pattern in a single period are the same;
the stripe gray value of the phase shift structure light pattern is changed periodically in a sine way, the light stripe in the line shift structure light pattern accounts for 1/n of the stripe width, and n is a natural number larger than 2.
In the embodiment of the invention, n is any natural number from 5 to 10, preferably 8.
Fig. 6 is a flowchart of a step of generating a point cloud image according to an embodiment of the present invention, as shown in fig. 6, the step S3 includes the following steps:
step S304: calculating to obtain a corresponding pixel coordinate value on the projector through the target phase value and the calibration information;
step S305: generating depth information of each pixel according to two pixel coordinate values matched with the camera unit and the projector;
step S306: and carrying out three-dimensional reconstruction according to the depth information of each pixel to generate the point cloud image.
In the embodiment of the invention, the calibration information is calculated and generated when the depth camera is calibrated. In depth reconstruction, triangulation is used.
Fig. 8 is a schematic block diagram of a flying spot removing system based on phase shift line shift fusion in an embodiment of the present invention, as shown in fig. 8, where the flying spot removing system based on phase shift line shift fusion is used in a depth camera based on a structured light principle, and the depth camera includes a projector and a camera unit, and includes the following modules:
a first phase generating module for acquiring a phase shift structured light pattern and determining a first phase value for each pixel in the camera unit according to the phase shift structured light pattern;
the second phase generation module is used for acquiring the line-shifting structure light pattern and determining a second phase value corresponding to each pixel in the camera unit according to the line-shifting structure light pattern;
and the point cloud generation module is used for fusing the first phase value and the second phase value corresponding to each pixel to generate a target phase value of the pixel, and further generating a point cloud image with flying spots removed.
The embodiment of the invention also provides flying spot removing equipment based on phase-shift line-shift fusion, which comprises a processor and a memory. A memory having stored therein executable instructions of a processor. Wherein the processor is configured to perform the flying spot removal method steps based on phase shift line shift fusion via execution of the executable instructions.
As described above, in this embodiment, the phase shift structure light pattern and the line shift structure light pattern are projected by the projector respectively to generate the first phase value and the second phase value, the first phase value and the second phase value corresponding to each pixel are fused to generate the target phase value of the pixel, and then the depth reconstruction is performed according to the target phase value to generate the point cloud image, so that flying points can be effectively removed, the imaging error of the depth camera is reduced, and the interference of flying point to point cloud processing and measurement is reduced.
Those skilled in the art will appreciate that the various aspects of the invention may be implemented as a system, method, or program product. Accordingly, aspects of the invention may be embodied in the following forms, namely: an entirely hardware embodiment, an entirely software embodiment (including firmware, micro-code, etc.) or an embodiment combining hardware and software aspects may be referred to herein as a "circuit," module "or" platform.
Fig. 9 is a schematic structural diagram of a flying spot removing device based on phase-shift line-shift fusion in an embodiment of the present invention. An electronic device 600 according to this embodiment of the invention is described below with reference to fig. 9. The electronic device 600 shown in fig. 9 is merely an example, and should not be construed as limiting the functionality and scope of use of embodiments of the present invention.
As shown in fig. 9, the electronic device 600 is in the form of a general purpose computing device. Components of electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one memory unit 620, a bus 630 connecting the different platform components (including memory unit 620 and processing unit 610), a display unit 640, etc.
Wherein the storage unit stores program code that can be executed by the processing unit 610, such that the processing unit 610 performs the steps according to various exemplary embodiments of the present invention described in the above-described flying spot removal method section based on phase shift line shift fusion of the present specification. For example, the processing unit 610 may perform the steps as shown in fig. 1.
The storage unit 620 may include readable media in the form of volatile storage units, such as Random Access Memory (RAM) 6201 and/or cache memory unit 6202, and may further include Read Only Memory (ROM) 6203.
The storage unit 620 may also include a program/utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.
Bus 630 may be a local bus representing one or more of several types of bus structures including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or using any of a variety of bus architectures.
The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboard, pointing device, bluetooth device, camera, depth camera, etc.), one or more devices that enable a user to interact with the electronic device 600, and/or any device (e.g., router, modem, etc.) that enables the electronic device 600 to communicate with one or more other computing devices. Such communication may occur through an input/output (I/O) interface 650. Also, electronic device 600 may communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet, through network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 over the bus 630. It should be appreciated that although not shown in fig. 9, other hardware and/or software modules may be used in connection with electronic device 600, including, but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, data backup storage platforms, and the like.
The embodiment of the invention also provides a computer readable storage medium for storing a program, and the method is used for removing flying spot based on phase shift line shift fusion when the program is executed. In some possible embodiments, the aspects of the invention may also be implemented in the form of a program product comprising program code for causing a terminal device to carry out the steps according to the various exemplary embodiments of the invention as described in the above-mentioned flying spot removal method section based on phase-shift line-shift fusion of the specification, when the program product is run on a terminal device.
As described above, when the program of the computer readable storage medium of this embodiment is executed, the phase shift structured light pattern and the line shift structured light pattern are projected by the projector, respectively, so as to generate the first phase value and the second phase value, the first phase value and the second phase value corresponding to each pixel are fused to generate the target phase value of the pixel, and then the depth reconstruction is performed according to the target phase value to generate the point cloud image, so that flying spots can be effectively removed, the imaging error of the depth camera is reduced, and the interference of flying spot to point cloud processing and measurement is reduced.
Fig. 10 is a schematic structural view of a computer-readable storage medium in an embodiment of the present invention. Referring to fig. 10, a program product 800 for implementing the above-described method according to an embodiment of the present invention is described, which may employ a portable compact disc read only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto, and in this document, a readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium would include the following: an electrical connection having one or more wires, a portable disk, a hard disk, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
The computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, with readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A readable storage medium may also be any readable medium that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).
In the embodiment of the invention, the phase shift structure light pattern and the line shift structure light pattern are respectively projected by the projector to generate the first phase value and the second phase value, the first phase value and the second phase value corresponding to each pixel are fused to generate the target phase value of the pixel, and then the point cloud image is generated by carrying out depth reconstruction according to the target phase value, so that flying spots can be effectively removed, the imaging error of the depth camera is reduced, and the interference of flying spot to point cloud processing and measurement is reduced.
In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other. The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing describes specific embodiments of the present invention. It is to be understood that the invention is not limited to the particular embodiments described above, and that various changes and modifications may be made by one skilled in the art within the scope of the claims without affecting the spirit of the invention.
Claims (6)
1. The flying spot removing method based on phase shift line shift fusion is used for a depth camera based on a structured light principle, and the depth camera comprises a projector and a camera unit and is characterized by comprising the following steps:
step S1: acquiring a phase shift structure light pattern, and determining a first phase value of each pixel in a camera unit according to the phase shift structure light pattern;
step S2: acquiring a line-shifting structured light pattern, and determining a second phase value corresponding to each pixel in the camera unit according to the line-shifting structured light pattern; the stripe widths of the phase shift structure light pattern and the line shift structure light pattern in a single period are the same; the stripe gray value of the phase shift structure light pattern is changed periodically in a sine way, the light stripe in the line shift structure light pattern accounts for 1/n of the stripe width, and n is a natural number larger than 2;
step S3: fusing the first phase value and the second phase value corresponding to each pixel to generate a target phase value of the pixel, and further generating a point cloud image with flying points removed; the step S3 includes the steps of:
step S301: judging whether the first phase value and the second phase value are invalid values or not, and determining that the target phase value corresponding to the pixel is the invalid value when any one of the first phase value and the second phase value is the invalid value;
step S302: generating a difference absolute value of the first phase value and the second phase value, and judging whether the difference absolute value is smaller than a preset difference threshold value or not;
step S303: when the absolute value of the difference is smaller than a preset difference threshold value, determining that a target phase value corresponding to the pixel is a first phase value, otherwise determining that the phase value corresponding to the pixel is an invalid value;
step S304: calculating to obtain a corresponding pixel coordinate value on the projector through the target phase value and the calibration information;
step S305: generating depth information of each pixel according to two pixel coordinate values matched with the camera unit and the projector;
step S306: and carrying out three-dimensional reconstruction according to the depth information of each pixel to generate the point cloud image.
2. The flying spot removing method based on phase-shift line-shift fusion according to claim 1, wherein the step S1 comprises the steps of:
step S101: controlling a projector to project phase shift structured light to a target object;
step S102: controlling a camera unit to acquire a phase shift structure light pattern formed by reflecting the phase shift structure light by a target object;
step S103: a first phase value for each pixel in the camera unit is determined from the phase shifted structured light pattern.
3. The flying spot removing method based on phase-shift line-shift fusion according to claim 1, wherein the step S2 comprises the steps of:
step S201: controlling a projector to project linear moving structured light to a target object, wherein linear moving stripe beams in the linear moving structured light are coded by Gray codes;
step S202: controlling a camera unit to collect a line-shifting structured light pattern formed by reflecting the line-shifting structured light by a target object;
step S203: a second phase value for each pixel in the camera unit is determined from the line shifted structured light pattern.
4. A flying spot removal system based on phase shift line shift fusion, which is used for a depth camera based on a structured light principle, wherein the depth camera comprises a projector and a camera unit, and comprises the following modules:
a first phase generating module for acquiring a phase shift structured light pattern and determining a first phase value for each pixel in the camera unit according to the phase shift structured light pattern;
the second phase generation module is used for acquiring the line-shifting structure light pattern and determining a second phase value corresponding to each pixel in the camera unit according to the line-shifting structure light pattern;
the point cloud generating module is configured to fuse the first phase value and the second phase value corresponding to each pixel to generate a target phase value of the pixel, and further generate a point cloud image with flying spots removed, and specifically includes: judging whether the first phase value and the second phase value are invalid values or not, and determining that the target phase value corresponding to the pixel is the invalid value when any one of the first phase value and the second phase value is the invalid value; generating a difference absolute value of the first phase value and the second phase value, and judging whether the difference absolute value is smaller than a preset difference threshold value or not; when the absolute value of the difference is smaller than a preset difference threshold value, determining that a target phase value corresponding to the pixel is a first phase value, otherwise determining that the phase value corresponding to the pixel is an invalid value; calculating to obtain a corresponding pixel coordinate value on the projector through the target phase value and the calibration information; generating depth information of each pixel according to two pixel coordinate values matched with the camera unit and the projector; and carrying out three-dimensional reconstruction according to the depth information of each pixel to generate the point cloud image.
5. Flying spot removing equipment based on phase shift line shift fusion, characterized by comprising:
a processor;
a memory module having stored therein executable instructions of the processor;
wherein the processor is configured to perform the steps of the flying spot removal method based on phase-shift line-shift fusion of any one of claims 1 to 3 via execution of the executable instructions.
6. A computer-readable storage medium storing a program, wherein the program when executed implements the steps of the flying spot removal method based on phase-shift line-shift fusion as set forth in any one of claims 1 to 3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311401878.1A CN117132509B (en) | 2023-10-27 | 2023-10-27 | Flying spot removing method, device, equipment and medium based on phase shift line shift fusion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311401878.1A CN117132509B (en) | 2023-10-27 | 2023-10-27 | Flying spot removing method, device, equipment and medium based on phase shift line shift fusion |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117132509A CN117132509A (en) | 2023-11-28 |
| CN117132509B true CN117132509B (en) | 2024-03-22 |
Family
ID=88854965
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311401878.1A Active CN117132509B (en) | 2023-10-27 | 2023-10-27 | Flying spot removing method, device, equipment and medium based on phase shift line shift fusion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117132509B (en) |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113012277A (en) * | 2021-02-03 | 2021-06-22 | 中国地质大学（武汉） | DLP (digital light processing) -surface-based structured light multi-camera reconstruction method |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9432568B2 (en) * | 2014-06-30 | 2016-08-30 | Semiconductor Components Industries, Llc | Pixel arrangements for image sensors with phase detection pixels |
-
2023
- 2023-10-27 CN CN202311401878.1A patent/CN117132509B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113012277A (en) * | 2021-02-03 | 2021-06-22 | 中国地质大学（武汉） | DLP (digital light processing) -surface-based structured light multi-camera reconstruction method |
Non-Patent Citations (2)
| Title |
|---|
| Algorithms for constructing 3D point clouds using multiple digital fringe projection patterns;Tao Peng et al;《 Computer-Aided Design and Applications 》;第2卷(第6期);第737-746页 * |
| 高效RGB格雷码与相移法结合的三维重建技术;皮金柱 等;《传感器与微系统》(第5期);第65-68页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117132509A (en) | 2023-11-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6932163B2 (en) | 3D data processing method, equipment, equipment and storage medium | |
| CN112771573B (en) | Depth estimation method and device based on speckle images and face recognition system | |
| Zheng et al. | Phase error analysis and compensation for phase shifting profilometry with projector defocusing | |
| CN107784671B (en) | Method and system for visual instant positioning and drawing | |
| JP2015135294A (en) | Three-dimensional shape measuring instrument and three-dimensional shape measurement method | |
| US20100142838A1 (en) | Gradient assisted image resampling in micro-lithographic printing | |
| CN111951376B (en) | Three-dimensional object reconstruction method fusing structural light and photometry and terminal equipment | |
| US11257232B2 (en) | Three-dimensional measurement method using feature amounts and device using the method | |
| CN112824934B (en) | TOF multipath interference removal method, system, equipment and medium based on modulated light field | |
| CN108885393A (en) | The method for reducing line waviness | |
| CN115205128A (en) | Depth camera temperature drift correction method, system, equipment and medium based on structured light | |
| CN111986154A (en) | Optical information detection method, device and equipment | |
| JP2021168127A (en) | Information generating method, information generating device, electronic device, recording medium, and computer program | |
| US20190325593A1 (en) | Image processing apparatus, system, method of manufacturing article, image processing method, and non-transitory computer-readable storage medium | |
| CN102654398A (en) | Photogrammetry measurement system | |
| JP2018194549A (en) | Three-dimensional measurement device with annotation features | |
| CN117132509B (en) | Flying spot removing method, device, equipment and medium based on phase shift line shift fusion | |
| CN114463436A (en) | Calibration method, system, equipment and storage medium of galvanometer scanning device | |
| CN112824935B (en) | Depth imaging system, method, device and medium based on modulated light field | |
| CN114945091B (en) | Temperature compensation method, device and equipment of depth camera and storage medium | |
| CN113538555B (en) | Volume measurement method, system, equipment and storage medium based on rule box | |
| CN113532266A (en) | Box volume measuring method, system, equipment and storage medium based on three-dimensional vision | |
| Ai et al. | A method for correcting non-linear geometric distortion in ultra-wide-angle imaging system | |
| Askarian Bajestani et al. | Scalable and view-independent calibration of multi-projector display for arbitrary uneven surfaces | |
| CN115294208A (en) | Temperature compensation system for depth camera |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |