US20170078649A1  Method and system for unsynchronized structured lighting  Google Patents
Method and system for unsynchronized structured lighting Download PDFInfo
 Publication number
 US20170078649A1 US20170078649A1 US15/124,176 US201515124176A US2017078649A1 US 20170078649 A1 US20170078649 A1 US 20170078649A1 US 201515124176 A US201515124176 A US 201515124176A US 2017078649 A1 US2017078649 A1 US 2017078649A1
 Authority
 US
 United States
 Prior art keywords
 image
 sequence
 pixel
 synchronized
 image frames
 Prior art date
 Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
 Abandoned
Links
 230000001360 synchronised Effects 0.000 claims abstract description 49
 239000011159 matrix material Substances 0.000 claims description 27
 238000005259 measurement Methods 0.000 claims description 12
 238000005096 rolling process Methods 0.000 claims description 10
 230000000875 corresponding Effects 0.000 claims description 6
 238000005192 partition Methods 0.000 claims description 6
 230000002194 synthesizing Effects 0.000 claims description 2
 238000000638 solvent extraction Methods 0.000 claims 1
 230000001419 dependent Effects 0.000 description 2
 230000004048 modification Effects 0.000 description 2
 238000006011 modification reaction Methods 0.000 description 2
 238000010146 3D printing Methods 0.000 description 1
 239000004927 clay Substances 0.000 description 1
 229910052570 clay Inorganic materials 0.000 description 1
 125000004122 cyclic group Chemical group 0.000 description 1
 230000003111 delayed Effects 0.000 description 1
 238000005516 engineering process Methods 0.000 description 1
 238000009776 industrial production Methods 0.000 description 1
 238000007689 inspection Methods 0.000 description 1
 230000002452 interceptive Effects 0.000 description 1
 230000004301 light adaptation Effects 0.000 description 1
 230000003287 optical Effects 0.000 description 1
Images
Classifications

 H—ELECTRICITY
 H04—ELECTRIC COMMUNICATION TECHNIQUE
 H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
 H04N13/00—Stereoscopic video systems; Multiview video systems; Details thereof
 H04N13/20—Image signal generators
 H04N13/204—Image signal generators using stereoscopic image cameras
 H04N13/254—Image signal generators using stereoscopic image cameras in combination with electromagnetic radiation sources for illuminating objects

 H04N13/0253—

 G—PHYSICS
 G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
 G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
 G03B35/00—Stereoscopic photography
 G03B35/02—Stereoscopic photography by sequential recording

 G06T7/0057—

 G—PHYSICS
 G06—COMPUTING; CALCULATING; COUNTING
 G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
 G06T7/00—Image analysis
 G06T7/50—Depth or shape recovery
 G06T7/521—Depth or shape recovery from laser ranging, e.g. using interferometry; from the projection of structured light

 H04N13/0051—

 H—ELECTRICITY
 H04—ELECTRIC COMMUNICATION TECHNIQUE
 H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
 H04N13/00—Stereoscopic video systems; Multiview video systems; Details thereof
 H04N13/10—Processing, recording or transmission of stereoscopic or multiview image signals
 H04N13/106—Processing image signals
 H04N13/167—Synchronising or controlling image signals

 G—PHYSICS
 G06—COMPUTING; CALCULATING; COUNTING
 G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
 G06T2207/00—Indexing scheme for image analysis or image enhancement
 G06T2207/10—Image acquisition modality
 G06T2207/10028—Range image; Depth image; 3D point clouds
Abstract
A system and method to capture the surface geometry a threedimensional object in a scene using unsynchronized structured lighting is disclosed. The method and system includes a pattern projector configured and arranged to project a sequence of image patterns onto the scene at a pattern frame rate, a camera configured and arranged to capture a sequence of unsynchronized image patterns of the scene at an image capture rate, and a processor configured and arranged to synthesize a sequence of synchronized image frames from the unsynchronized image patterns of the scene. Each of the synchronized image frames corresponds to one image pattern of the sequence of image patterns.
Description
 This application claims priority to earlier filed U.S. Provisional Application Ser. No. 61/949,529, filed Mar. 7, 2014, the contents of which are incorporated herein by reference.
 1. Field of the Invention
 This patent document relates generally to the field of threedimensional shape capture of the surface geometry of an object, and more particularly to structured lighting threedimensional shape capture.
 2. Background of the Related Art
 Threedimensional scanning and digitization of the surface geometry of objects is commonly used in many industries and services, and their applications are numerous. A few examples of such applications are inspection and measurement of shape conformity in industrial production systems, digitization of clay models for industrial design and styling applications, reverse engineering of existing parts with complex geometry for threedimensional printing, interactive visualization of objects in multimedia applications, threedimensional documentation of artwork, historic and archaeological artifacts, human body scanning for better orthotics adaptation, biometry or customfit clothing, and threedimensional forensic reconstruction of crime scenes.
 One technology for threedimensional shape capture is based on structured lighting. Three dimensional shape capture systems based on structure lighting are more accurate than those based on timeofflight (TOF) image sensors. In a standard structured lighting 3D shape capture system a pattern projector is used to illuminate the scene of interest with a sequence of known twodimensional patterns, and a camera is used to capture a sequence of images, synchronized with the projected patterns. The camera captures one image for each projected pattern. Each sequence of images captured by the camera is decoded by a computer processor into a dense set of projectorcamera pixel correspondences, and subsequently into a threedimensional range image, using the principles of optical triangulation.
 The main limitation of threedimensional shape capture systems is the required synchronization between projector and camera. To capture a threedimensional snapshot of a moving scene, the sequence of patterns must be projected at a fast rate, the camera must capture image frames exactly at the same frame rate, and the camera has to start capturing the first frame of the sequence exactly when the projector starts to project the first pattern.
 Therefore, there is a need for threedimensional shape measurement methods and systems based on structure lighting where the camera and the pattern projector are not synchronized.
 Further complicating matters, image sensors generally use one of two different technologies to capture an image, referred to as “rolling shutter” and “global shutter”. “Rolling shutter” is a method of image capture in which a still picture or each frame of a video is captured not by taking a snapshot of the entire scene at single instant in time but rather by scanning across the scene rapidly, either vertically or horizontally. In other words, not all parts of the image of the scene are recorded at exactly the same instant. This is in contrast with “global shutter” in which the entire frame is captured at the same instant. Even though most image sensors in consumer devices are rolling shutter sensors, many image sensors used in industrial applications are global shutter sensors.
 Therefore, there is a need for threedimensional shape measurement methods and systems based on structure lighting where the camera and the pattern projector are not synchronized, supporting both global shutter and rolling shutter image sensors.
 A system and method to capture the surface geometry a threedimensional object in a scene using unsynchronized structured lighting solves the problems of the prior art. The method and system includes a pattern projector configured and arranged to project a sequence of image patterns onto the scene at a pattern frame rate, a camera configured and arranged to capture a sequence of unsynchronized image patterns of the scene at an image capture rate; and a processor configured and arranged to synthesize a sequence of synchronized image frames from the unsynchronized image patterns of the scene, each of the synchronized image frames corresponding to one image pattern of the sequence of image patterns. Because the method enables use of an unsynchronized pattern projector and camera significant cost savings can be achieved. The method enables use of inexpensive cameras, such as smartphone cameras, webcams, pointandshoot digital cameras, camcorders as well as industrial cameras. Furthermore, the method and system enable processing the images with a variety of computing hardware, such as computers, digital signal processors, smartphone processors and the like. Consequently, threedimensional image capture using structured lighting may be used with relatively little capital investment.
 These and other features, aspects, and advantages of the method and system will become better understood with reference to the following description, appended claims, and accompanying drawings where:

FIG. 1 is an illustration of an exemplary embodiment of the method and system for unsynchronized structured lighting; 
FIG. 2 shows a flowchart of an exemplary embodiment of the method and system for unsynchronized structured lighting; 
FIG. 3 shows a flowchart of an exemplary embodiment of the method and system for unsynchronized structured lighting of synthesizing the synchronized sequence of image frames; 
FIG. 4 is a chart illustrating the timing for a method to synthesize the synchronized sequence of image frames from an unsynchronized sequence of image frames, where a global shutter image sensor is used and where the image frame rate is identical to the pattern frame rate; 
FIG. 5 is a chart illustrating the timing for a method to synthesize the synchronized sequence of image frames from an unsynchronized sequence of image frames, where a rolling shutter image sensor is used and where the image frame rate is identical to the pattern frame rate; 
FIG. 6 is a chart illustrating the timing for a method to synthesize the synchronized sequence of image frames from an unsynchronized sequence of image frames, where a global shutter image sensor is used and where the image frame rate is higher or equal than the pattern frame rate 
FIG. 7 is a chart illustrating the timing for method to synthesize the synchronized sequence of image frames from an unsynchronized sequence of image frames, where a rolling shutter image sensor is used and where the image frame rate is higher or equal than the pattern frame rate; 
FIG. 8 shows a chart illustrating a calibration pattern used to correct the time; 
FIG. 9 shows a chart illustrating image data normalized according to the method described herein; and 
FIG. 10 shows a chart illustrating a model estimating the pattern value for each pixel in a captured image.  A system and method to capture the surface geometry a threedimensional object in a scene using unsynchronized structured lighting is shown generally in
FIGS. 1 and 2 . The method and system includes a pattern projector configured and arranged to project a sequence of image patterns onto the scene at a pattern frame rate, a camera configured and arranged to capture a sequence of unsynchronized image patterns of the scene at an image capture rate; and a processor configured and arranged to synthesize a sequence of synchronized image frames from the unsynchronized image patterns of the scene, each of the synchronized image frames corresponding to one image pattern of the sequence of image patterns.  One object of the present invention is a system to synthesize a synchronized sequence of image frames from an unsynchronized sequence of image frames, illustrated in
FIG. 1 ; the unsynchronized image frames captured while a threedimensional scene was illuminated by a sequence of patterns. The system comprises a pattern projector 2020 and a camera 2030. An object 2010 is partially illuminated by the pattern projector 2020, and partially visible by the camera 2030. The pattern projector projects a sequence of patterns 2021, 2022, 2023 at a certain pattern rate. The pattern rate is measured in patterns per second. The camera 2030 captures a sequence of unsynchronized image frames 2031, 2032, 2033, 2034, at a certain frame rate. The frame rate is larger or equal than the pattern rate. The number of unsynchronized image frames captured by the camera is larger or equal than the number of patterns in the sequence of patterns. The camera starts capturing the first unsynchronized image frame not earlier than the time when the projector starts projecting the first pattern of the sequence of image patterns. The camera ends capturing the last unsynchronized image frame not earlier than the time when the projector ends projecting the last pattern of the sequence of image patterns. To capture a frame the camera opens the camera aperture, which it closes after an exposure time. The camera determines the pixel values by integrating the incoming light while the aperture is open. Since camera and projector are not synchronized, the projector may switch patterns while the camera has the aperture open. As a result, some or all of the pixels of the captured unsynchronized image frame 2032 will be partially exposed to two consecutive patterns 2021 and 2022. The resulting sequence of unsynchronized image frames, are transmitted to a computer processor 2040, which executes a method to synthesize a synchronized sequence of image frames from the unsynchronized sequence of image frames. The number of frames in the synchronized sequence of image frames will be the same as the number of patterns and represent estimates of what the camera would have captured if it were synchronized with the projector. In a preferred embodiment the camera 2030 and the computer processor 2040 are components of a single device such a digital camera, smartphone, or computer tablet.  Another object of the invention is an unsynchronized threedimensional shape capture system, comprising the system to synthesize a synchronized sequence of image frames from an unsynchronized sequence of image frames described above, and further comprising prior art methods for decoding, threedimensional triangulation, and optionally geometric processing, executed by the computer processor.
 Another object of the invention is a threedimensional snapshot camera comprising the unsynchronized threedimensional shape capture system, where the projector has the means to select the pattern rate from a plurality of supported pattern rates, the camera has the means to select the frame rate from a plurality of supported frame rates, and the camera is capable of capturing the unsynchronized image frames in burst mode at a fast frame rate. In a preferred embodiment the projector has a knob to select the pattern rate. In another preferred embodiment the pattern rate is set by a pattern rate code sent to the projector through a communications link. Furthermore, the system has means to set the pattern rate and the frame rate so that the frame rate is not slower than the pattern rate. In a more preferred embodiment the user sets the pattern rate and the frame rate.
 In a more preferred embodiment of the snapshot camera, the camera has the means to receive a camera trigger signal, and the means to set the number of burst mode frames. In an even more preferred embodiment, the camera trigger signal is generated by a camera trigger pushbutton. When the camera receives the trigger signal it starts capturing the unsynchronized image frames at the set frame rate, and it stops capturing unsynchronized image frames after capturing the set number of burst mode frames.
 In a first preferred embodiment of the snapshot camera with camera trigger signal, the projector continuously projects the sequence of patterns in a cyclic fashion. In a more preferred embodiment the system has the means of detecting when the first pattern is about to be projected, and the camera trigger signal is delayed until that moment.
 In a second preferred embodiment of the snapshot camera with camera trigger signal, the projector has the means to receive a projector trigger signal. In a more preferred embodiment the camera generates the projector trigger signal after receiving the camera trigger signal, and the camera has the means to send the projector trigger signal to the projector. In an even more preferred embodiment the camera has a flash trigger output, and it sends the projector trigger signal to the projector through the flash trigger output. When the projector receives the trigger signal it starts projecting the sequence of patterns at the set pattern rate, and it stops projecting patterns after it projects the last pattern.
 Another object of this invention is a method to synthesize a synchronized sequence of image frames from an unsynchronized sequence of image frames, generating a number of frames in the synchronized sequence of image frames equal to the number of projected patterns, and representing estimates of what the camera would have captured if it were synchronized with the projector.
 As will be described in greater detail below in the associated proofs, the
 method to synthesize the synchronized sequence of image frames from the unsynchronized sequence of image frames is shown generally in
FIG. 3 . Each of the synchronized image frames corresponds to one of a sequence of image patterns. Further, the synchronized image frames and unsynchronized image frames have the same width and height. Each image frame includes a plurality of common pixels. In a first step a) the plurality of common pixels is partitioned into a pixel partition. The pixel partition includes a plurality of pixel sets, each of which is disjoint. Further each of the plurality of common pixels is a member of one of the pixel sets. In a step b) a pixel set is selected. In a step c) a pixel set measurement matrix is built. In a step d) a pixel set projection matrix is estimated. The pixel set projection matrix projects a measurement space vector onto the column space of the pixel set measurement matrix. In a step e) a pixel set system matrix is estimated. The system matrix is parameterized by a set of system matrix parameters. In a step f) a pixel set synchronized matrix as a function of the pixel set measurement matrix and the pixel set system matrix is estimated. In a step g) steps b) to f) are repeated until all the pixels sets have been selected. In a step h) a sequence of synchronized image frames from the pixel set synchronized matrices is constructed.  In a preferred embodiment, the method to synthesize the synchronized sequence of image frames from an unsynchronized sequence of image frames, applies to a global shutter image sensor where the image frame rate is identical to the pattern frame rate.
FIG. 4 illustrates the timing for this embodiment. In this embodiment, the projector projects N patterns at a fixed frame rate, a global shutter image sensor capture N images at identical frame rate. Capturing each image takes exactly one unit of time, normalized by the projector frame rate. The start time for the first image capture t_{0 }is unknown, but the start time for the nth image capture is related to the start time for the first image capture t_{n}=t_{0}+n−1. The actual value measured by the image sensor at the (x, y) pixel of the nth image, can be modeled as 
I _{n}(x, y)=(1−t _{0})P _{n}(x, y)+t _{0} P _{n+1}(x, y)  where P_{n}(x, y) and P_{n−1}(x, y) represent the pattern values to be estimated that contribute to the image pixel (x, y) and P_{N+1}≡P_{1}. Projected patterns are known in advance, but since it is not known which projector pixel illuminates each image pixel, they have to be treated as unknown. To estimate the value of t_{0}, the following expression is minimized

$E\ue8a0\left({t}_{0}\right)=\frac{1}{2}\ue89e{\sum}_{n=1}^{N}\ue89e\sum _{\left(x,y\right)}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e{\left(\left(1{t}_{0}\right)\ue89e{P}_{n}\ue8a0\left(x,y\right)+{t}_{0}\ue89e{P}_{n+1}\ue8a0\left(x,y\right){I}_{n}\ue8a0\left(x,y\right)\right)}^{2}$  with respect to t_{0 }, where the sum is over a subset of pixels (x, y) for which the corresponding pattern pixel values P_{n}(x, y) and P_{n−1}(x, y) are known. Differentiating E(t_{0}) with respect to t_{0}, and equating the result to zero, an expression to estimate t_{0 }is obtained

${t}_{0}=\frac{{\sum}_{n=1}^{N}\ue89e{\sum}_{\left(x,y\right)}\ue89e\left({P}_{n+1}\ue8a0\left(x,y\right){P}_{n}\ue8a0\left(x,y\right)\right)\ue89e\left({I}_{n}\ue8a0\left(x,y\right){P}_{n}\ue8a0\left(x,y\right)\right)}{{\sum}_{n=1}^{N}\ue89e{\sum}_{\left(x,y\right)}\ue89e{\left({P}_{n+1}\ue8a0\left(x,y\right){P}_{n}\ue89e\left(x,y\right)\right)}^{2}}$  Once the value of t_{0 }has been estimated, the N pattern pixel values P_{1}(x, y), . . . , P_{N}(x, y) can be estimated for each pixel (x, y) by minimizing the following expression

E(P_{1}(x, y), . . . , P_{N}(x, y))=^{ 1 } _{2}Σ_{n=1} ^{N}((1−t _{0})P _{n}(x, y)+t _{0} P _{n+1}(x, y)−I _{n}(x, y))^{2}  which reduces to solving the following system of N linear equations

βP _{n−1}(x, y)+αP _{n}(x, y)+βP _{n+1}(x, y)=t_{0} I _{n−1}(x, y)+(1−t _{0})I _{n}(x, y)  for n=1 , . . ., N, where α=t^{2} _{0}+(1−t_{0})^{2 }and ⊕=t_{0}(1−−t_{0}).
 In another preferred embodiment, the method to synthesize the synchronized sequence of image frames from an unsynchronized sequence of image frames, applies to a rolling shutter image sensor where the image frame rate is identical to the pattern frame rate.
FIG. 5 illustrates the timing for this embodiment. We project N patterns at fixed framerate, a rolling shutter camera captures N images. Capture begins while pattern P_{1 }is being projected. Projector framerate is 1, pattern P_{n }is projected between time n−1 and n. A camera frame is read every t_{f }time, camera framerate is assumed equal to projector framerate but in practice may vary a little. A camera row requires t_{r }time to be readout from the sensor, thus, a sensor with Y rows needs a time Yt_{r }to read a complete frame, Yt_{f}≦t_{r}. Each camera frame is exposed t_{e }time, its readout begins immediately after exposure ends, t_{e}+t_{r}≦t_{f}.  Camera row y in image n begins being exposed at time t_{n, y}

t _{n, y} =t _{0}+(n−1)t_{f} +y t _{r} , y:0 . . . Y−1,  and exposition ends at time t_{n, y}+t_{e }
 In this model image n is exposed while pattern P_{n }and P_{n+1 }are being projected. Intensity level measured at a pixel in row y is given by

I _{n, y}=(n−t _{n, y})k _{n, y} P _{n}+(t _{n, y} +t _{e} −n)k _{n, y} P _{n+1} +C _{n, y,}  The constants k_{n, y }and C_{n, y }are scene dependent.
 Let be min {I_{n, y}} a pixel being being exposed while P(t)=0, and max {I_{n, y}} a pixel being exposed while P(t)=1, max {I_{n, y}}=t_{e}k_{n, y}+C_{n, y}. Now, we define a normalized image Ĩ_{n, y }as,

${\stackrel{~}{I}}_{n,y}=\frac{\left(n{t}_{n,y}\right)\ue89e{P}_{n}+\left({t}_{n,y}+{t}_{e}n\right)\ue89e{P}_{n+1}}{{t}_{e}}$  A normalized image is completely defined by the time variables and pattern values. In this section we want to estimate the time variables. Let's rewrite Equation 58 as

${\stackrel{~}{I}}_{n,y}=n\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e\Delta \ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e{P}_{n}\ue89e\frac{1}{{t}_{e}}+\left(n1\right)\ue89e\Delta \ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e{P}_{n}\ue89e\frac{{t}_{f}}{{t}_{e}}+y\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e\Delta \ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e{P}_{n}\ue89e\frac{{t}_{r}}{{t}_{e}}+\Delta \ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e{P}_{n}\ue89e\frac{{t}_{0}}{{t}_{e}}+{P}_{n+1}$  being t_{0 }and d unknown. Image pixel values are given by

I _{n}(x, y)=(1−t _{0} −yd)P _{n}(x, y)+(t _{0} +yd)P _{n}+1(x, y).  Same as before, P_{n}(x, y) and P_{n+1 }(x, y) represent the pattern values contributing to camera pixel (x, y), we define P_{N+}1≡P_{1}, P_{0}≡P_{N}, I_{N+1}≡I_{1}, and I_{0}≡I_{N}, and I will omit pixel (x, y) to simplify the notation. We now minimize the following energy to find the time variables t_{0 }and d

$E\ue8a0\left({t}_{0},d\right)=\frac{1}{2}\ue89e\sum _{n=1}^{N}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e\sum _{x,y}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e{\left(\left(1{t}_{0}\mathrm{yd}\right)\ue89e{P}_{n}+\left({t}_{0}+\mathrm{yd}\right)\ue89e{P}_{n+1}{I}_{n}\right)}^{2}$  The partial derivatives are given by

$\frac{\partial E\ue8a0\left({t}_{0},d\right)}{\partial {t}_{0}}=\sum _{n=1}^{N}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e\sum _{x,y}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e\left({P}_{n+1}{P}_{n}\right)\ue89e\left(\left(1{t}_{0}\mathrm{yd}\right)\ue89e{P}_{n}+\left({t}_{0}+\mathrm{yd}\right)\ue89e{P}_{n+1}{I}_{n}\right)$ $\frac{\partial E\ue8a0\left({t}_{0},d\right)}{\partial d}=\sum _{n=1}^{N}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e\sum _{x,y}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89ey\ue8a0\left({P}_{n+1}{P}_{n}\right)\ue89e\left(\left(1{t}_{0}\mathrm{yd}\right)\ue89e{P}_{n}+\left({t}_{0}+\mathrm{yd}\right)\ue89e{P}_{n+1}{I}_{n}\right)$  We set the gradient equal to the null vector and reorder as

$\left[\begin{array}{c}{t}_{0}\\ d\end{array}\right]={\left(\sum _{n,x,y}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e{\left({P}_{n+1}{P}_{n}\right)}^{2}\ue8a0\left[\begin{array}{cc}1& y\\ y& {y}^{2}\end{array}\right]\right)}^{1}\ue89e\left(\sum _{n,x,y}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e\left({P}_{n+1}{P}_{n}\right)\ue89e\left({I}_{n}\mathrm{Pn}\right)\ue8a0\left[\begin{array}{c}1\\ y\end{array}\right]\right)$  We use Equation 29 to compute t_{0 }and d when we have some known (or estimated) pattern values.
 With known t_{0 }and d we estimate pattern values minimizing

$E\ue8a0\left({P}_{1},\dots \ue89e\phantom{\rule{0.8em}{0.8ex}},{P}_{N}\right)=\frac{1}{2}\ue89e\sum _{n=1}^{N}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e{\left(\left(1{t}_{0}\mathrm{yd}\right)\ue89e{P}_{n}+\left({t}_{0}+\mathrm{yd}\right)\ue89e{P}_{n+1}{I}_{n}\right)}^{2}.$  Analogous as in Case 1 we obtain that Ap=b with A as in Equation 12 and α, β, and b defined as

α=(1t _{0} −yd)^{2}+(t _{0} +yd)^{2}, β=(1−t _{0} −yd)(t _{0} +yd) 
b=(1−t _{0} −yd)(I_{1} I _{2 } . . . , I _{N})^{T}+(t _{0} +yd)(I _{N} , I _{1} , . . . I _{N−1})^{T}  Pattern values for each pixel are given by p=A^{−1 }b.
 In another preferred embodiment, the method to synthesize the synchronized sequence of image frames from an unsynchronized sequence of image frames, applies to a global shutter image sensor where the image frame rate is higher or equal than the pattern frame rate.
FIG. 6 illustrates the timing for this embodiment. We now project M patterns at fixed framerate and we capture N images with a global shutter camera, also at a fixed framerate. We require that N≧M. Capture begins while pattern P_{1 }is being projected. We introduce a new variable d which is the camera capture delay from one row to the next. Same as in Case 1, up to two patterns may contribute to each image but here we do not know which ones are because the camera framerate is unknown. The new image equation is 
${I}_{n}={\int}_{{t}_{n1}}^{{t}_{n}}\ue89e\sum _{m=1}^{M}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e{f}_{m}\ue8a0\left(t\right)\ue89e{P}_{m}\ue89e\phantom{\rule{0.2em}{0.2ex}}\ue89e\uf74ct=\sum _{m=1}^{M}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e{P}_{m}\ue89e{\int}_{{t}_{n1}}^{{t}_{n}}\ue89e{f}_{m}\ue8a0\left(t\right)\ue89e\phantom{\rule{0.2em}{0.2ex}}\ue89e\uf74ct$ ${f}_{m}\ue8a0\left(t\right)=\{\begin{array}{cc}1& \mathrm{if}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89em1\le t\le m\\ 0& \mathrm{otherwise}\end{array}$  Let be Δt≡t_{n−1}−t_{n }the time between image frames, let be p=(P_{1}, . . . , P_{M})^{T }and Φ_{n}(t_{0}, Δt)=(Φ(n, 1, t_{0}, Δt), . . . , Φ(n, M, t_{0}, Δt))^{T }and rewrite Equation 33 as

${I}_{n}=\frac{1}{\Delta \ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89et}\ue89e{{\phi}_{n}\ue8a0\left({t}_{0},\Delta \ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89et\right)}^{T}\ue89ep$  Each function Φ(n, m, t0, Δt)=∫_{t} _{ n−1 } ^{t} ^{ n }f_{m}(t)dt can be written as

Φ(n, m, t0, Δt)=max(0, min(m, t _{n})−max(m−1, t _{n−1}))  Same as before, P_{n}(x, y) represents a pattern value contributing to camera pixel (x, y), we define P_{N+}1≡P_{1}, P_{0 }≡P_{N}, I_{N+1}≡I_{1}, and I_{0}≡I_{N}, and I will omit pixel (x, y) to simplify the notation.
 We now minimize the following energy to find the time variables t_{0 }and Δt

$E\ue8a0\left({t}_{0},\Delta \ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89et\right)=\frac{1}{2}\ue89e\sum _{n=1}^{N}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e\sum _{x,y}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e{\left(\frac{1}{\Delta \ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89et}\ue89e{{\phi}_{n}\ue8a0\left({t}_{0},\Delta \ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89et\right)}^{T}\ue89ep{I}_{n}\right)}^{2}$  We solve for t_{0 }and Δt by making ∇E(t_{0}, Δt)=0

$\u25bd\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89eE\ue8a0\left({t}_{0},\Delta \ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89et\right)=\sum _{n=1}^{N}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e\sum _{x,y}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e\frac{1}{\Delta \ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89et}\ue89eJ\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e{{\phi}_{n}\ue8a0\left({t}_{0},\Delta \ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89et\right)}^{T}\ue89ep\ue8a0\left(\frac{1}{\Delta \ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89et}\ue89e{{\phi}_{n}\ue8a0\left({t}_{0},\Delta \ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89et\right)}^{T}\ue89ep{I}_{n}\right)$  Because JΦ_{n}(t_{0}, Δt) depends on the unknown value t=(t_{0}, Δt)^{T }we solve for them iteratively

${t}^{\left(i+1\right)}={{A}_{t}\ue8a0\left({t}^{\left(i\right)}\right)}^{1}\ue89e{b}_{t}\ue8a0\left({t}^{\left(i\right)}\right)$ ${A}_{t}\ue8a0\left(t\right)=\sum _{n=1}^{N}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e\sum _{x,y}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e\frac{1}{\Delta \ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e{t}^{2}}\ue89eJ\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e{{\phi}_{n}\ue8a0\left(t\right)}^{T}\ue89e{\mathrm{pp}}^{T}\ue89e{V}_{A}\ue8a0\left(n,t\right)$ ${b}_{t}\ue8a0\left(t\right)=\sum _{n=1}^{N}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e\sum _{x,y}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e\frac{1}{\Delta \ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89et}\ue89eJ\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e{{\phi}_{n}\ue8a0\left(t\right)}^{T}\ue89ep\ue8a0\left({I}_{n}\frac{1}{\Delta \ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89et}\ue89e{{V}_{b}\ue8a0\left(n,t\right)}^{T}\ue89ep\right)$  Matrix V_{A}(n, t) and vector V_{b}(n, t) are defined such as

${\phi}_{n}\ue8a0\left(t\right)={V}_{A}\ue8a0\left(n,t\right)\ue8a0\left[\begin{array}{c}{t}_{0}\\ \Delta \ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89et\end{array}\right]+{V}_{b}\ue8a0\left(n,t\right)$  For completeness we include the following definitions:

$\phantom{\rule{1.1em}{1.1ex}}\ue89e{V}_{A}\ue8a0\left(n,t\right)={\left[{v}_{A}\ue8a0\left(n,1,t\right),\phantom{\rule{0.8em}{0.8ex}}\ue89e\dots \ue89e\phantom{\rule{0.8em}{0.8ex}},{v}_{A}\ue8a0\left(n,M,t\right)\right]}^{T}$ $\phantom{\rule{1.1em}{1.1ex}}\ue89e{V}_{b}\ue8a0\left(n,t\right)={\left[{v}_{b}\ue8a0\left(n,1,t\right),\phantom{\rule{0.8em}{0.8ex}}\ue89e\dots \ue89e\phantom{\rule{0.8em}{0.8ex}},{v}_{b}\ue8a0\left(n,M,t\right)\right]}^{T}$ $\phantom{\rule{1.1em}{1.1ex}}\ue89e{t}_{\mathrm{diff}}\equiv \mathrm{min}\ue8a0\left(m,{t}_{n}\right)\mathrm{max}\ue8a0\left(m1,{t}_{n1}\right)$ ${v}_{A}\ue8a0\left(n,m,t\right)=\{\begin{array}{cc}{\left[0,1\right]}^{T}& \mathrm{if}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89em1\le {t}_{n1}\le {t}_{n}\le m\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e\mathrm{and}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e{t}_{\mathrm{diff}}>0\\ {\left[1,1n\right]}^{T}& \mathrm{if}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89em1\le {t}_{n1}\le m\le {t}_{n}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e\mathrm{and}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e{t}_{\mathrm{diff}}>0\\ {\left[1,n\right]}^{T}& \mathrm{if}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e{t}_{n1}\le m1\le {t}_{n}\le m\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e\mathrm{and}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e{t}_{\mathrm{diff}}>0\\ {\left[0,0\right]}^{T}& \mathrm{if}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e{t}_{n1}\le m1\le m\le {t}_{n}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e\mathrm{and}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e{t}_{\mathrm{diff}}>0\\ {\left[0,0\right]}^{T}& \mathrm{otherwise}\end{array}\ue89e\text{}\ue89e{v}_{b}\ue8a0\left(n,m,t\right)=\{\begin{array}{cc}0& \mathrm{if}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89em1\le {t}_{n1}\le {t}_{n}\le m\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e\mathrm{and}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e{t}_{\mathrm{diff}}>0\\ m& \mathrm{if}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89em1\le {t}_{n1}\le m\le {t}_{n}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e\mathrm{and}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e{t}_{\mathrm{diff}}>0\\ 1m& \mathrm{if}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e{t}_{n1}\le m1\le {t}_{n}\le m\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e\mathrm{and}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e{t}_{\mathrm{diff}}>0\\ 1& \mathrm{if}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e{t}_{n1}\le m1\le m\le {t}_{n}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e\mathrm{and}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e{t}_{\mathrm{diff}}>0\\ 0& \mathrm{otherwise}\end{array}\ue89e\text{}\ue89e\phantom{\rule{1.1em}{1.1ex}}\ue89eJ\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e{\phi}_{n}\ue8a0\left(t\right)={V}_{A}\ue8a0\left(n,t\right)$  With known t_{0 }and 66 t we estimate pattern values minimizing

$E\ue8a0\left(p\right)=\frac{1}{2}\ue89e\sum _{n=1}^{N}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e{\left({{\phi}_{n}\ue8a0\left(t\right)}^{T}\ue89ep{I}_{n}\right)}^{2}$  Analogous as in Case 1 we obtain that Ap=b with

$A={\Phi}^{T}\ue89e\Phi ,\text{}\ue89eb={\Phi}^{T}\ue89eI$ $\Phi =\left[\begin{array}{c}{{\phi}_{1}\ue8a0\left(t\right)}^{T}\\ \vdots \\ {{\phi}_{N}\ue8a0\left(t\right)}^{T}\end{array}\right]$  Pattern values for each pixel are given by p=A^{−1}b.
 In another preferred embodiment, the method to synthesize the synchronized sequence of image frames from an unsynchronized sequence of image frames, applies to a rolling shutter image sensor where the image frame rate is higher or equal than the pattern frame rate.
FIG. 7 illustrates the timing for this embodiment. Projector framerate is 1, pattern P_{m }is projected between time m−1 and m. A camera frame is read every t_{f }time. A camera row requires t_{r }time to be readout from the sensor, thus, a sensor with Y rows needs a time Yt_{r }to read a complete frame, t_{f}≧Yt_{r}. Each camera frame is exposed t_{e }time, its readout begins immediately after exposure ends, t_{e}+t_{r}≦t_{f}.  Camera row y in image n begins being exposed at time t_{n, y}

t _{n, y} =t _{0}+(n−1)t _{f} +y t _{r} , y:0 . . . Y−1  and exposition ends at time t_{n, y}+t_{e}.
 In this model a pixel intensity in image n at row y is given by

${I}_{n,y}={\int}_{{t}_{n,y}}^{{t}_{n,y}+{t}_{e}}\ue89e{k}_{n,y}\ue89eP\ue8a0\left(t\right)\ue89e\phantom{\rule{0.2em}{0.2ex}}\ue89e\uf74ct+{C}_{n,y}$ ${I}_{n,y}=\sum _{m=1}^{M}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e\mathrm{max}\ue8a0\left(0,{\int}_{\mathrm{max}\ue8a0\left({t}_{n,y},m1\right)}^{\mathrm{min}\ue8a0\left({t}_{n,y}+{t}_{e},m\right)}\ue89e{k}_{n,y}\ue89e{P}_{m}\ue89e\phantom{\rule{0.2em}{0.2ex}}\ue89e\uf74ct\right)+{C}_{n,y}$  The constants k_{n, y }and C_{n, y }are scene dependent, P_{m }is either 0 or 1.
 Let be min{I_{n, y}} a pixel being exposed while P(t)=0, and max{I_{n, y}} a pixel being exposed while P(t)=1,

min{I_{n, y}}=C_{n, y} 
max{I _{n, y} }=t _{e} k _{n, y} +C _{n, y}  Now, we define a normalized image Ĩ_{n, y}as,

${\stackrel{~}{I}}_{n,y}=\frac{{I}_{n,y}\mathrm{min}\ue89e\left\{{I}_{n,y}\right\}}{\mathrm{max}\ue89e\left\{{I}_{n,y}\right\}\mathrm{min}\ue89e\left\{{I}_{n,y}\right\}}$ ${\stackrel{~}{I}}_{n,y}=\frac{1}{{t}_{e}}\ue89e\sum _{m=1}^{M}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e\mathrm{max}\ue8a0\left(0,{\int}_{\mathrm{max}\ue8a0\left({t}_{n,y},m1\right)}^{\mathrm{min}\ue8a0\left({t}_{n,y}+{t}_{e},m\right)}\ue89e{P}_{m}\ue89e\phantom{\rule{0.2em}{0.2ex}}\ue89e\uf74ct\right)$  A normalized image is completely defined by the time variables and pattern values. In this section we want to estimate the time variables. Let's rewrite the previous equation as,

$\phantom{\rule{1.1em}{1.1ex}}\ue89e{\stackrel{~}{I}}_{n,y}=\sum _{m=1}^{M}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e\phi \ue8a0\left(n,m,y\right)\ue89e{P}_{m}$ $\phantom{\rule{1.1em}{1.1ex}}\ue89e\phi \ue8a0\left(n,m,y\right)=\frac{1}{{t}_{e}}\ue89e\mathrm{max}\ue8a0\left(0,\mathrm{min}\ue8a0\left({t}_{n,y}+{t}_{e},m\right)\mathrm{max}\ue8a0\left({t}_{n,y},m1\right)\right)$ $\mathrm{Let}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e\mathrm{be}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89eh=\frac{1}{{t}_{e}}\ue89e{\left(1,{t}_{f},{t}_{r},{t}_{0}\right)}^{T},\mathrm{now}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e\mathrm{we}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e\mathrm{write}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e\phi \ue8a0\left(n,m,y\right)={v}_{\mathrm{nmy}}^{T}\ue89eh\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e\mathrm{with}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e{v}_{\mathrm{nmy}}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e\mathrm{defined}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e\mathrm{as}\ue89e\phantom{\rule{0.8em}{0.8ex}}$ ${v}_{\mathrm{nmy}}^{T}=\{\begin{array}{cc}\left[{t}_{e},0,0,0\right]& \mathrm{if}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e{t}_{n\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2}<{t}_{m\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2}\ue374{t}_{n\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1}>{t}_{m\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1}\ue374{t}_{n\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2}>{t}_{n\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1}\\ \left[{t}_{e}m+1,n1,y,1\right]& \mathrm{if}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e{t}_{n\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2}<{t}_{m\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2}\ue374{t}_{n\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1}\le {t}_{m\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1}\ue374{t}_{n\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2}>{t}_{m\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1}\\ \left[m,1n,y,1\right]& \mathrm{if}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e{t}_{n\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2}\ge {t}_{m\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2}\ue374{t}_{n\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1}>{t}_{m\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1}\ue374{t}_{m\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2}>{t}_{n\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1}\\ \left[1,0,0,0\right]& \mathrm{if}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e{t}_{n\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2}\ge {t}_{m\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2}\ue374{t}_{n\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1}\le {t}_{m\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1}\ue374{t}_{m\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2}>{t}_{m\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1}\\ \left[0,0,0,0\right]& \mathrm{otherwise}\end{array}\ue89e\text{}\ue89e\phantom{\rule{1.1em}{1.1ex}}\ue89e{t}_{n\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1}={t}_{n,y},{t}_{n\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2}={t}_{n,y}+{t}_{e},{t}_{m\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1}=m1,{t}_{m\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e2}=m\ue89e\text{}\ue89e\phantom{\rule{1.1em}{1.1ex}}\ue89e\mathrm{Let}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e\mathrm{be}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89ep={\left({P}_{1},\phantom{\rule{0.8em}{0.8ex}}\ue89e\dots \ue89e\phantom{\rule{0.8em}{0.8ex}},{P}_{M}\right)}^{T}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e\mathrm{and}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e\mathrm{let}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e{V}_{\mathrm{ny}}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e\mathrm{be}\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e\text{}\ue89e\phantom{\rule{1.1em}{1.1ex}}\ue89e{V}_{\mathrm{ny}}=\left[\begin{array}{ccc}\u2013& {v}_{n\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e1\ue89ey}^{T}& \u2013\\ \phantom{\rule{0.3em}{0.3ex}}& \vdots & \phantom{\rule{0.3em}{0.3ex}}\\ \u2013& {v}_{\mathrm{nMy}}^{T}& \u2013\end{array}\right].$  We now minimize the following energy to find the unknown h

$E\ue8a0\left(h\right)=\frac{1}{2}\ue89e\sum _{n=1}^{N}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e\sum _{x,y}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e{\left({p\ue8a0\left(x,y\right)}^{T}\ue89e{V}_{\mathrm{ny}}\ue89eh{\stackrel{~}{I}}_{n}\ue8a0\left(x,y\right)\right)}^{2}$  with the following constraints

$h>\left[\begin{array}{c}0\\ 1\\ 0\\ 0\end{array}\right]\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89e\mathrm{and}\ue89e\phantom{\rule{0.8em}{0.8ex}}\left[\begin{array}{cccc}0& 1& Y& 0\\ 0& 1& 1& 0\end{array}\right]\ue89e\phantom{\rule{0.8em}{0.8ex}}\ue89eh\le \left[\begin{array}{c}0\\ 1\end{array}\right]$  or equivalently

$\frac{{t}_{r}}{e}\frac{{t}_{f}}{{t}_{e}}\le 1\Rightarrow {t}_{r}+{t}_{e}\le {t}_{f}$  Equation E(h) cannot be minimized in closed form because the values matrix V_{n, y }depends on the unknown values. Using an iterative approach the current value h^{(i) }is used to compute V_{ny} ^{(i) }and the next value h^{(i+1)pl . }
 Up to this point we have assumed that the only unknown is h, meaning that pattern values are known for all image pixels. The difficulty lies is knowing which pattern pixel is being observed by each camera pixel. We simplify this issue by making calibration patterns all ‘black or all ‘white’, best seen in
FIG. 8 . For example, a sequence of four patterns ‘{black, black, white, white}’ will produce images with completely black and completely white pixels, as well as pixels in transition from black to white and vice versa. The all black or white pixels are required to produce normalized images, as shown inFIG. 9 , and the pixels in transition constrain the solution of the parameter h in Equation E(h).  Decoding is done in two steps: 1) the time offset t_{0 }need to be estimated for this particular sequence; 2) the pattern values are estimated for each camera pixel, as shown in
FIG. 10 . Value t_{0 }is estimated using Equation E(h) where the known components of h are fixed, but some pattern values are required to be known, specially we need to know for some pixels whether they are transitioning from ‘black’ to ‘white’ or the opposite. Nontransitioning pixels provided no information in this step. Until now, we have projected a couple of black's and white's at the beginning of the sequence to ensure we can normalized all pixels correctly and to simplify t_{0 }estimation. We will revisit this point in the future for other pattern sequences.  Similarly as for the time variables, pattern values are estimated by
 minimizing the following energy

$E\ue8a0\left(p\right)=\frac{1}{2}\ue89e\sum _{n=1}^{N}\ue89e\phantom{\rule{0.3em}{0.3ex}}\ue89e{\left({h}^{T}\ue89e{V}_{\mathrm{ny}}^{T}\ue89ep\ue8a0\left(x,y\right){\stackrel{~}{I}}_{n}\ue8a0\left(x,y\right)\right)}^{2},s.t.\phantom{\rule{0.8em}{0.8ex}}\ue89e\uf603{p}_{m}\uf604\le 1$  The matrix h^{T}V_{ny} ^{T }is bidiagonal for N=M and it is fixed if h is known.
 Therefore, it can be seen that the exemplary embodiments of the method and system provides a unique solution to the problem of using structure lighting for threedimensional image capture where the camera and projector are unsychronized.
 It would be appreciated by those skilled in the art that various changes and modifications can be made to the illustrated embodiments without departing from the spirit of the present invention. All such modifications and changes are intended to be within the scope of the present invention except as limited by the scope of the appended claims.
 What is claimed is:
Claims (22)
1. A system to capture the surface geometry a threedimensional object in a scene, comprising:
a pattern projector configured and arranged to project a sequence of image patterns onto the scene at a pattern frame rate;
a camera configured and arranged to capture a sequence of unsynchronized image patterns of the scene at an image capture rate; and
a processor configured and arranged to synthesize a sequence of synchronized image frames from the unsynchronized image patterns of the scene, each of the synchronized image frames corresponding to one image pattern of the sequence of image patterns.
2. The system of claim 1 , wherein the sequence of image patterns comprises binary patterns.
3. The system of claim 1 , wherein the number of image patterns in the sequence of image patterns is less than or equal to the number of unsynchronized image frames in the sequence of unsynchronized image frames.
4. The system of claim 1 , wherein the camera has a rolling shutter operation.
5. The system of claim 1 , wherein the camera has a global shutter operation.
6. The system of claim 1 , wherein the image capture rate of the camera is equal to the pattern frame rate of the projector.
7. The system of claim 1 , wherein the image capture rate of the camera is greater than the pattern frame rate of the projector.
8. A method of capturing the surface geometry of a threedimensional object in a scene, comprising:
projecting a sequence of image pattern into the scene at a pattern frame rate;
capturing a sequence of unsynchronized image patterns of the scene at an image capture rate; and
synthesizing a sequence of synchronized image frames from the from the unsynchronized image patterns of the scene, each of the synchronized image frames corresponding to one image pattern of the sequence of image patterns.
9. The method of claim 8 , wherein the step of projecting a sequence of image patterns comprises projecting a sequence of binary patterns.
10. The method of claim 8 , wherein the number of image patterns projected in the sequence of image patterns is less than or equal to the number of unsynchronized image frames in the sequence of unsynchronized image frames.
11. The method of claim 8 , further comprising selecting a pattern frame rate.
12. The method of claim 8 , further comprising selecting an image capture rate.
13. The method of claim 8 , wherein the image capture rate is equal to the pattern frame rate.
14. The method of claim 8 , wherein the image capture rate is greater than the pattern frame rate.
15. The method of claim 8 , further comprising decoding the sequence of synchronized image frames.
16. The method of claim 15 , further comprising applying threedimensional triangulation to the sequence of synchronized image frames.
17. The method of claim 16 , further comprising applying geometric processing to the sequence of synchronized image frames.
18. A method to synthesize a sequence of synchronized image frames synchronized from a sequence of unsynchronized image frames; each of the synchronized image frames corresponding to one of a sequence of image patterns; the synchronized image frames and unsynchronized image frames being image frames of the same width and height; the image frames comprising a plurality of common pixels; the method comprising the steps of:
a) partitioning the plurality of common pixels into a pixel partition; the pixel partition comprising a plurality of pixel sets; the pixel sets being disjoint; each of the plurality of common pixels being a member of one of the pixel sets;
b) selecting a pixel set;
c) building a pixel set measurement matrix;
d) estimating a pixel set projection matrix; the pixel set projection matrix projecting a measurement space vector onto the column space of the pixel set measurement matrix;
e) estimating a pixel set system matrix; the system matrix being parameterized by a set of system matrix parameters;
f) estimating a pixel set synchronized matrix as a function of the pixel set measurement matrix and the pixel set system matrix;
g) repeating steps b) to f) until all the pixels sets have been selected; and
h) constructing a sequence of synchronized image frames from the pixel set synchronized matrices.
19. A method as in claim 18 , where the pixel partition comprises a single pixel set, and the single pixel set contains all the pixels of the image frames.
20. A method as in claim 18 , where the number of pixel sets in the pixel partition is equal to the height of the image frames, and each row of the image frames is a pixel set.
21. A method as in claim 18 , where the pixel set measurement matrix is modeled as the product of the system matrix times the pixel set synchronized matrix, and the step of estimating the pixel set synchronized matrices reduces to the solution of a linear leastsquares problem.
22. A method as in claim 18 , where the system matrix is parameterized by an image frame period parameter, an integration time parameter, and a first pattern delay parameter.
Priority Applications (3)
Application Number  Priority Date  Filing Date  Title 

US201461949529P true  20140307  20140307  
PCT/US2015/019357 WO2015134961A1 (en)  20140307  20150309  Method and system for unsynchronized structured lighting 
US15/124,176 US20170078649A1 (en)  20140307  20150309  Method and system for unsynchronized structured lighting 
Applications Claiming Priority (1)
Application Number  Priority Date  Filing Date  Title 

US15/124,176 US20170078649A1 (en)  20140307  20150309  Method and system for unsynchronized structured lighting 
Publications (1)
Publication Number  Publication Date 

US20170078649A1 true US20170078649A1 (en)  20170316 
Family
ID=54055938
Family Applications (2)
Application Number  Title  Priority Date  Filing Date 

US15/124,176 Abandoned US20170078649A1 (en)  20140307  20150309  Method and system for unsynchronized structured lighting 
US16/434,846 Pending US20190364264A1 (en)  20140307  20190607  Method and system for unsynchronized structured lighting 
Family Applications After (1)
Application Number  Title  Priority Date  Filing Date 

US16/434,846 Pending US20190364264A1 (en)  20140307  20190607  Method and system for unsynchronized structured lighting 
Country Status (2)
Country  Link 

US (2)  US20170078649A1 (en) 
WO (1)  WO2015134961A1 (en) 
Cited By (2)
Publication number  Priority date  Publication date  Assignee  Title 

US10154248B2 (en) *  20150925  20181211  Fujitsu Limited  Encoder apparatus, encoder system, encoding method, and medium for separating frames captured in time series by imaging directions 
US20200084384A1 (en) *  20180911  20200312  Cognex Corporation  Methods and apparatus for optimizing image acquisition of objects subject to illumination patterns 
Families Citing this family (3)
Publication number  Priority date  Publication date  Assignee  Title 

WO2018144930A1 (en)  20170203  20180809  MODit3D, INC.  Threedimensional scanning device and methods 
WO2018145113A1 (en)  20170206  20180809  MODit3D, INC.  System and method for 3d scanning 
JP2019020157A (en) *  20170712  20190207  パナソニックＩｐマネジメント株式会社  Projection imaging system, measurement device, and projection imaging method 
Citations (2)
Publication number  Priority date  Publication date  Assignee  Title 

US20140035480A1 (en) *  20120802  20140206  Tsvi Blumin  Method for the control of luminance of gas discharge lamps 
US9256943B2 (en) *  20130328  20160209  Texas Instruments Incorporated  Projectorcamera misalignment correction for structured light systems 
Family Cites Families (7)
Publication number  Priority date  Publication date  Assignee  Title 

US6388654B1 (en) *  19971003  20020514  Tegrity, Inc.  Method and apparatus for processing, displaying and communicating images 
WO2007061632A2 (en) *  20051109  20070531  Geometric Informatics, Inc.  Method and apparatus for absolutecoordinate threedimensional surface imaging 
US8016434B2 (en) *  20080605  20110913  Disney Enterprises, Inc.  Method and system for projecting an animated object and concurrently moving the object's projection area through an animation pattern 
US8013904B2 (en) *  20081209  20110906  Seiko Epson Corporation  View projection matrix based high performance low latency display pipeline 
US9066087B2 (en) *  20101119  20150623  Apple Inc.  Depth mapping using timecoded illumination 
CN102760234B (en) *  20110414  20140820  财团法人工业技术研究院  Depth image acquiring device, system and method 
KR102009292B1 (en) *  20120511  20190812  한국전자통신연구원  Apparatus and method for reconstructing three dimensional face based on multiple cameras 

2015
 20150309 WO PCT/US2015/019357 patent/WO2015134961A1/en active Application Filing
 20150309 US US15/124,176 patent/US20170078649A1/en not_active Abandoned

2019
 20190607 US US16/434,846 patent/US20190364264A1/en active Pending
Patent Citations (2)
Publication number  Priority date  Publication date  Assignee  Title 

US20140035480A1 (en) *  20120802  20140206  Tsvi Blumin  Method for the control of luminance of gas discharge lamps 
US9256943B2 (en) *  20130328  20160209  Texas Instruments Incorporated  Projectorcamera misalignment correction for structured light systems 
Cited By (3)
Publication number  Priority date  Publication date  Assignee  Title 

US10154248B2 (en) *  20150925  20181211  Fujitsu Limited  Encoder apparatus, encoder system, encoding method, and medium for separating frames captured in time series by imaging directions 
US20200084384A1 (en) *  20180911  20200312  Cognex Corporation  Methods and apparatus for optimizing image acquisition of objects subject to illumination patterns 
US10791277B2 (en) *  20180911  20200929  Cognex Corporation  Methods and apparatus for optimizing image acquisition of objects subject to illumination patterns 
Also Published As
Publication number  Publication date 

US20190364264A1 (en)  20191128 
WO2015134961A1 (en)  20150911 
Similar Documents
Publication  Publication Date  Title 

US10728474B2 (en)  Image signal processor for local motion estimation and video codec  
US9600889B2 (en)  Method and apparatus for performing depth estimation  
Liu et al.  Video stabilization with a depth camera  
JP5997645B2 (en)  Image processing apparatus and method, and imaging apparatus  
US7605817B2 (en)  Determining camera motion  
US7616885B2 (en)  Single lens auto focus system for stereo image generation and method thereof  
KR101429371B1 (en)  Algorithms for estimating precise and relative object distances in a scene  
US7428019B2 (en)  System and method for increasing space or time resolution in video  
JP3734829B2 (en)  Electronic image stabilization system and method  
CN102272796B (en)  Motion vector generation apparatus and motion vector generation method  
JP5057974B2 (en)  Processing video data to correct unintentional camera movement between acquired image frames  
US20150271469A1 (en)  Image synchronization method for cameras and electronic apparatus with cameras  
KR101420684B1 (en)  Apparatus and method for matching color image and depth image  
US8422766B2 (en)  System and method for depth extraction of images with motion compensation  
Forssén et al.  Rectifying rolling shutter video from handheld devices  
US20160173842A1 (en)  CameraAssisted Two Dimensional Keystone Correction  
US9286680B1 (en)  Computational multicamera adjustment for smooth view switching and zooming  
KR101854188B1 (en)  3D image acquisition apparatus and method of acqiring depth information in the 3D image acquisition apparatus  
TWI359387B (en)  Robust camera pan vector estimation using iterativ  
US9210405B2 (en)  System and method for real time 2D to 3D conversion of video in a digital camera  
JP6510213B2 (en)  Projection system, semiconductor integrated circuit, and image correction method  
KR101665567B1 (en)  Temporal interpolation of three dimension depth image method and apparatus  
DE102015000386A1 (en)  Apparatus and method for measuring a threedimensional shape and nontransitory computerreadable storage medium  
JP3934151B2 (en)  Image generating apparatus and image generating method  
EP2291825B1 (en)  System and method for depth extraction of images with forward and backward depth prediction 
Legal Events
Date  Code  Title  Description 

AS  Assignment 
Owner name: BROWN UNIVERSITY, RHODE ISLAND Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:TAUBIN, GABRIEL;MORENO, DANIEL;CALAKLI, FATIH;SIGNING DATES FROM 20171024 TO 20171025;REEL/FRAME:047010/0246 

STCV  Information on status: appeal procedure 
Free format text: NOTICE OF APPEAL FILED 

STCB  Information on status: application discontinuation 
Free format text: ABANDONED  FAILURE TO RESPOND TO AN OFFICE ACTION 