US6535650B1 - Creating high resolution images - Google Patents

Creating high resolution images Download PDF

Info

Publication number
US6535650B1
US6535650B1 US09120257 US12025798A US6535650B1 US 6535650 B1 US6535650 B1 US 6535650B1 US 09120257 US09120257 US 09120257 US 12025798 A US12025798 A US 12025798A US 6535650 B1 US6535650 B1 US 6535650B1
Authority
US
Grant status
Grant
Patent type
Prior art keywords
images
image
program
imaging system
digital imaging
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Lifetime
Application number
US09120257
Inventor
Richard J. Poulo
Vu X. Nguyen
Vareck S. Bostrom
Walter L. Gorman
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Intel Corp
Original Assignee
Intel Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Grant date

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00Details of television systems
    • H04N5/222Studio circuitry; Studio devices; Studio equipment ; Cameras comprising an electronic image sensor, e.g. digital cameras, video cameras, TV cameras, video cameras, camcorders, webcams, camera modules for embedding in other devices, e.g. mobile phones, computers or vehicles
    • H04N5/225Television cameras ; Cameras comprising an electronic image sensor, e.g. digital cameras, video cameras, camcorders, webcams, camera modules specially adapted for being embedded in other devices, e.g. mobile phones, computers or vehicles
    • H04N5/232Devices for controlling television cameras, e.g. remote control ; Control of cameras comprising an electronic image sensor
    • H04N5/23238Control of image capture or reproduction to achieve a very large field of view, e.g. panorama
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00Geometric image transformation in the plane of the image, e.g. from bit-mapped to bit-mapped creating a different image
    • G06T3/40Scaling the whole image or part thereof
    • G06T3/4038Scaling the whole image or part thereof for image mosaicing, i.e. plane images composed of plane sub-images
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/30Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/33Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10004Still image; Photographic image
    • G06T2207/10008Still image; Photographic image from scanner, fax or copier

Abstract

A digital imaging system includes a camera to capture a plurality of images representing a scene. A program is executable on the processor to align the images with respect to a reference image and to combine the images into a composite image that represents the scene. The camera may include a low resolution camera, and the composite image may be a high resolution, digitized image.

Description

BACKGROUND

The invention relates to creating high resolution images.

Digital image processing involves capturing, manipulating and analyzing digital information. A digital imaging device can capture optical images and convert them to digital format for further processing. In a typical digital imaging system, optical components may be used to focus an optical image onto an array of pixel sensors that electrically capture the image and output analog signals, which are then converted by analog-to-digital converters into digital signals for further processing.

High resolution digital image acquisition is typically available in many types of image scanning devices, including copy machines, facsimile machines, document scanners, electronic white boards, and cameras that use either film or high resolution digital sensors. In such devices, to accurately capture fine detail in text, a photograph, or a drawing, high resolution image acquisition equipment is needed.

An optical image is a continuously varying array of shades and colors. In a digitization process, an image may be sampled into a rectangular array of pixels, with each pixel having an (x, y) coordinate that corresponds to its location within the image. The number of pixels in a digital image depends on how finely the image is sampled.

A digital camera's cost is a function of its resolution. The higher its resolution the more expensive the digital camera. For example, to capture an 8½×11 document using a 300-dots-per-inch (DPI) scan requires an image resolution of 2550×3300 pixels. Inexpensive digital cameras (including still-shot cameras and video cameras) have resolutions far below this and thus cannot capture images with the desired detail.

Thus, a need arises for a technique and mechanism to generate high resolution images using low resolution digital imaging devices.

SUMMARY

In general, in one embodiment, the invention features a digital imaging system that includes a camera to capture a plurality of images representing a scene, a processor, and a program executable on the processor to align the images with respect to a reference image and to combine the images into a composite image that represents the scene.

Other features of the invention will become apparent from the following description and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a digital imaging system that runs an application program according to an embodiment of the invention.

FIG. 2 is a flow diagram illustrating the application program of FIG. 1 that performs an image stitching process.

FIG. 3 illustrates a correlation path used by the image stitching process.

FIGS. 4 and 5 are charts of correlation function values calculated by comparing two images in the image stitching process.

FIG. 6 is a flow diagram of an image stitching process according to another embodiment that includes identifying partial mosaics.

FIG. 7 is a flow diagram of a process for grouping captured images into partial mosaics.

FIG. 8 is a flow diagram of a process for detecting and distributing residual errors.

FIG. 9 is a block diagram of a computer system that may be used in the digital imaging system of FIG. 1 and in which stitching programs according to embodiments of the invention may be run.

FIG. 10 is a block diagram of a digital camera that may be used in the digital imaging system of FIG. 1.

DETAILED DESCRIPTION

Referring to FIG. 1, in a digital imaging system 10 that includes a program 16 according to an embodiment of the invention, a digital camera or other suitable imaging device 12 (which may be a video camera or still-shot camera) captures multiple optical images 11. The camera 12 transmits streams of data representing the images (stored as frames of pixel data) to a computer 14 over a communications link 15 (which may be a serial link, for example). The digital camera 12 may be a low resolution camera. The computer 14 may then execute the application program 16 to process the captured images. According to an embodiment of the invention, under control of the application program 16, the computer 14 can combine (“stitch”) multiple frames of low resolution, digitized images together to form a high resolution composite image.

Alternatively, the application program 16 can be loaded and run in the digital camera 12 or on a device connected over a network (including a local area network, wide area network, or the Internet). Further, a portion of the application program 16 may be run in the digital camera 12 while another portion may be run in the computer 14.

To create a high resolution image using images captured by the low resolution camera 12, the camera 12 is scanned back and forth across a target scene (which may be, for example, a natural scene, a page of text, or another target). The camera 12 is focused on small parts of the overall scene while the camera is scanned to capture the entire scene. The captured low resolution images are then processed and stitched together by the application program 16 (“stitching program”) according to embodiments of the invention to create a high resolution composite image that represents the entire target scene. Key parts of the stitching process include aligning the captured frames and matching overlapping regions of the different frames of the scene. To accurately match the regions in the multiple images, the stitching program 16 compensates for misalignment or distortion resulting from motion of the camera caused by unsteady movement of the camera support, which may be, for example, a user's hands.

There are six degrees of freedom of motion of a camera, these being the three independent axes of motion (x,y,z) and the three independent rotations about those axes. Thus, given any two overlapping images about which nothing is known regarding the relative positions of the camera when capturing the two images, the six-dimensional space must be searched to match the overlapping pixels of the two images.

The stitching program 16 in one embodiment receives successive video frames that have been taken by the camera 12 at a predetermined rate as the camera 12 is swept over the scene to be captured. Using a capture rate of as little as about two frames per second, for example, may be sufficient to reduce the region to be searched in six dimensional space down to a feasible size. Using the example sampling rate, a frame is taken every one-half second. During that short time period, the camera 12 may be moved and rotated by only a relatively small amount. As a result, this can reduce all rotations and movements to a value small enough such that a linear offset of only a small number of pixels is observed from one edge of one image to the edge of the subsequent image. Given such small movement between images, the application program 16 can accurately calculate how images are distorted or misaligned with respect to each other and adjust the images to correct for such distortions or misalignments.

Because of the relatively rapid rate of image capture as the camera 12 is scanned, successive images will have large amounts of image overlap. To properly combine or stitch the low resolution frames of images, all the captured frames may be aligned with respect to a reference frame to compensate for movement and rotation of the camera 12 as a target scene is being captured. This correction process (“alignment process”) is accomplished using perspective transforms (or projective mapping) to adjust the pixel positions in each frame. In one embodiment, the perspective transforms map all captured frames back into the first captured frame. In an alternate embodiment, another frame can be chosen as the reference frame.

To perform the perspective transforms, mapping points in each frame are first identified. In one embodiment, four mapping points are selected in each frame. To accurately correlate these points in the frames, the mapping points are uniquely identifiable for each frame; that is, a mapping point should not be redundant. Another desired attribute of these mapping points is that they be found in “feature-rich” areas; that is, areas of high detail. Once the mapping points are identified, then the perspective transforms of all frames can be accurately performed using standard techniques known in the art.

After the alignment process, error correction may be performed. Because the perspective transforms are performed on a large number of frames, accumulated error may be significant. Such accumulated error is removed in the error correction phase.

Referring to FIG. 2, the stitching program 16 according to an embodiment of the invention first receives a frame from a video source, such as the digital camera 12 (step 100 ). Next, the stitching program 16 determines if the frame is a valid video frame (step 102 ). This step essentially determines if the user has stopped capturing data, in which case incoming data has stopped and the image capture and stitching process is complete. At that time, the composite image is displayed on the video display of the computer system (step 104 ). However, if a valid video frame is received, the stitching program 16 may next perform pre-processing steps, including lens distortion correction (step 106 ) and image simplification and enhancement (step 108 ). An image may be distorted through lens operations and other non-linear effects. Camera lenses often have geometric distortions, including “pin cushion” distortion and “barrel” distortion. Many standard techniques exist to compensate for such distortions.

After distortion of the image has been corrected, image simplification and enhancement may be performed (step 108 ). For example, processing may be performed to remove image noise. Further, luminance distortion may be corrected to improve the image correlation process. Luminance distortion is the effect of a camera lens in changing the relative brightness of different parts of an image, for example, making the center of an image brighter than it should be relative to the edges. To correct luminance distortion, a small number of images of blank white paper may be captured and the average taken on a pixel-by-pixel basis over the multiple images. The average image may be converted to a gray scale image {pij}, from which a gray scale histogram may be calculated. From the histogram, a threshold white value wT may be calculated such that a predetermined percentage (e.g., 92%) of pixels in the image {pij} have white values less than wT. From wT and {pij}, an array {cij} is defined according to Eq. 1. { c ij } = w T { p ij } (Eq.  1)

Figure US06535650-20030318-M00001

To correct for luminance distortion, all captured frames that are to be stitched together may be multiplied by the array {cij} on a pixel-by-pixel basis. Steps 106 and 108 are included in the pre-processing phase of the stitching process.

After the pre-processing phase, the alignment phase of the stitching process is performed to align the captured frames to compensate for the unknown relative positions of the camera 12 as it is used to capture the multiple images. The stitching program 16 identifies “feature-rich” image areas (step 110). In the image stitching process, “good” mapping points in an image are selected to properly map one image to another. Mapping points selected in one image are correlated to corresponding mapping points in another image. Poor choices in mapping points may result in missed correlation, resulting in incorrectly stitched images. As noted above, these mapping points are found in feature-rich areas. Thus, to select these mapping points, feature-rich areas are first identified. To determine what is a feature-rich area, the number of features in an image is first determined, followed by applying the measurement to find the feature-rich area.

To measure feature detail, a simplified edge detection scheme may be used in one embodiment. The measurement is defined as the summation of the absolute difference of a group of closely related pixels in both the x and y directions for a given correlation area, which is a small area (such as area A in FIG. 3) over which the feature-rich determination is made. For example, the correlation area A may be selected to be a region of 30×30 pixels. In one embodiment, a correlation area A is initially selected to be in a comer of the frame. If the initial correlation area is determined not to be feature-rich, then the correlation area A is moved according to a preselected path (e.g., path R1 shown in FIG. 3) and the correlation process performed until a feature-rich area is identified.

The captured image is represented by a frame P and the parameters x and y represent the coordinates of pixels in the frame P. Thus, P(x, y) represents the pixel value at a certain (x, y) coordinate. To determine if the selected correlation area is a high-feature (or feature-rich) area, two variables H(x, y) and V(x, y) are defined according to Eqs. 2 and 3, in which H(x, y) represents the absolute horizontal pixel difference and V(x, y) represents the absolute vertical pixel difference.

H(x,y)=|P(x−1,y)−P(x+1,y)|,  (Eq. 2)

V(x,y)=|P(x,y−1)−P(x,y+1)|,  (Eq. 3)

From the H and V values, an edge detail value D of the correlation area may be calculated using Eq. 4, D = i , j H ( i , j ) + V ( i , j ) , (Eq.  4)

Figure US06535650-20030318-M00002

in which variables i,j are bounded by the selected correlation area.

By calculating the edge detail D, the number of features in a region may be measured. After the edge detail value D is calculated for the selected correlation area the stitching program 16 then determines if the edge detail value D exceeds the minimum feature threshold value (step 112). A feature-rich area is a region where the measured detail level is greater than the minimum feature (or detail threshold) for correlation to be successful. This minimum threshold value can be preset and depends on the sensitivity of the correlation routine to the type of features found in an image. Once this threshold value is determined, a simple search over the image can be done to find an area that meets the threshold requirement.

In one embodiment, the detail threshold may be dynamically set to adapt to previous correlation passes. Feedback from prior correlations (failures or successes) indicate if the selected detail threshold level is adequate for correlation. The detail threshold level is decreased when correlation fails and increased when successful. Other ways of implementing this dynamic thresholding are also possible.

In an alternative embodiment, the calculation of the detail value D may be improved by eliminating image noise from the calculation of D. This may allow for consistency of values across images of different quality. Noise reduction may be performed by eliminating pixel differences less than a certain threshold value. The noise threshold may be based upon the type of application and the type of images a camera can capture. For example, if the captured image includes text, the noise threshold may be set high to eliminate areas with luminance irregularity problems and poor image quality (low signal-to-noise ratios). To eliminate image noise, Eqs. 5 and 6 may be used.

H(x,y)=0, if H(x,y)≦NThreshold,  (Eq. 5)

V(x,y)=0, if V(x,y)≦NThreshold,  (Eq. 6)

where Nthreshold may be the minimum pixel difference between any two pixels, which also is the noise threshold.

If the calculated detail value D is less than the minimum detail threshold (as determined in step 112), then a feature-rich area has not been found, and a new correlation area is selected along path R1 or some variation of the path shown in FIG. 3 (step 113). The feature-rich area determination of steps 110 and 112 are then performed again. If the area is determined to be feature-rich, then a mapping point may be selected in the feature-rich area (step 114).

Most natural scenes have a high level of detail, so selecting a set of mapping points at fixed positions in each image may be sufficient for correct correlation. However, in the case of a text image or other types of low detail images, the detail within the captured image may be sparse. As a result, fixed point correlation may be unsuccessful. In some embodiments, fixed point correlation may be used, but other techniques for selecting mapping points may also be used such as that described below.

A mapping point is an area of an image that is uniquely identifiable from one image to another. According to one embodiment, the criteria for what constitutes a good set of mapping points and how to find those points are as follows: (1) points that have non-redundant feature-rich detail; and (2) points that are as far apart as possible in the image. According to one embodiment, the zigzag or spiral-shaped search pattern R1 shown in FIG. 3 may be used.

In one embodiment, search patterns identical to the search pattern R1 may start from each of the other corners of the frame P image to identify four mapping points. The pattern R1 starts from corner C1 of the frame P, while the other patterns R2, R3, and R4 would start from corners C2, C3, and C4, respectively.

To select the mapping point in the selected correlation area, the current frame Pk(x, y) is searched and compared to the same correlation area in a previously captured frame, Pm(x, y). A correlation function C(x, y) may be calculated according to Eq. 7, which is the sum of the squares of the pixel differences between Pk(x, y) and Pm(x, y). C ( x , y ) = i , j [ P k ( i , j ) - P m ( i + x , j + y ) ] 2 (Eq.  7)

Figure US06535650-20030318-M00003

where i,j are bounded by the selected correlation area. To find the correlation point, the correlation process identifies an (x, y) coordinate in the correlation area of the current frame Pk(x,y) at which the correlation function C(x, y) is a minimum. In effect, according to Eq. 7, an (x,y) coordinate is selected in the current frame Pk and frame Pk is moved with respect to the frame Pm in the correlation area A to find a coordinate (x,y) that produces a minimum for C(x,y).

Although in some embodiments the correlation function C(x,y) of Eq. 7 may be used, other correlation functions are also contemplated from which a mapping point can be selected by calculating a minimum or maximum.

The technique described may provide several benefits. First, the search from the corners that follows the zigzag path of FIG. 3 in one embodiment minimizes the search time for finding adequate mapping points and yielding maximal transform area. Further, by utilizing feature-rich mapping points, white space or ambiguous regions of the image are automatically eliminated.

As discussed earlier, there are six degrees of freedom in the motion between any two successive video frames. However, because of the relatively fast capture of frames, the movement between any two successive frames is small and thus the maximum (or minimum depending on the correlation function) over the selected small correlation region A accurately defines the inter-frame motion of translation, rotation, perspective and/or scaling.

Once a mapping point is found, a correlation error and redundancy error check may be performed (step 115). Redundancy error is a result of selecting a correlation point on one image that could be mapped to several points on another image, thereby not producing a unique point. This may result from repetitive patterns in the images, which may include camera characteristics such as interlacing. To determine if redundant detail exists, correlation data is analyzed over the correlation area A. If this analysis indicates multiple minima of the same magnitude in correlation value, then redundancy has been detected. If the selected mapping point is a redundant mapping point, then it may be discarded as a mapping point.

The chart shown in FIG. 4 is a simplified chart of correlation illustrating the existence of redundant detail. Although the redundancy analysis involves a two-dimensional calculation, the one-dimensional correlation illustrated in the chart of FIG. 4 provides a simple illustration of how the redundancy error detection is performed. FIG. 4 shows the result of a horizontal search (in the x direction) in which redundant detail causes two minima when two images Pk and Pm are compared by calculating the correlation function C(x, y) according to Eq. 7. The technique includes finding the minimum of the correlation function and setting a threshold that is slightly larger than the minimum value. The same concept and technique is extended to the two-dimensional case, in which a search is performed both in the x and y directions.

A method according to one embodiment for determining a threshold and number of minima includes selecting a predetermined threshold that is a set value (e.g. 33%) greater than the minimum value of C(x, y) found in the correlation area A. If the threshold value used results in too many minima, then the threshold can be reduced until the number is manageable. Once a good set of minimum values is determined, the stitching program 16 may confirm if each is a minimum by searching all correlation points around it to make sure it is the lowest point in the area. If it is, then a true minimum has been found. If the number of true minima is greater than one per direction (x, y directions), then redundancy has been detected and the selected mapping point cannot be used.

The stitching program 16 in step 115 may also check to determine if the correlation has failed. Correlation failure detection is similar to redundancy failure detection (x or y). If the correlation areas of two images are mismatched, there would be no discernable minimum, which may result in multiple, unequal minima. Thus, determining the number of minima found in the selected correlation area provides a way to validate if the correlation was successful or not. The chart shown in FIG. 5 illustrates what the correlation data could look like if the correlation was unsuccessful.

If the point correlation is unsuccessful due either to a detected redundancy error or a correlation error (step 116), then a new correlation area is selected (step 113) to find a new mapping point (steps 110, 112, 114, 115). However, if the point correlation is indicated as being successful (step 116), then the stitching program 16 determines if enough mapping points (e.g., 4) are selected to perform the perspective transform or projective mapping (step 118). If not, the process starts from another corner of the image to find another feature-rich area to find the next mapping point and to correlate the point (steps 110, 112, 113, 114, 115, and 116).

Given a correlation point at (xM, yM) that has been calculated by moving one image (e.g., Pk) over a second image (e.g., Pm), the stitching program 16 may optionally search for a sub-pixel position (a position located between two pixels) that provides even better correlation. The pixel located at (xM, yM), which has a better correlation than its neighbors, may be referred to as the maximal pixel. Aligning the images at this pixel provides a better match then at any other pixel, but in general, alignment of the images Pk and Pm at a fractional pixel offset may provide better alignment.

To find this sub-pixel position, a 3×3 array of pixels defined by pixel positions (XL, xM, xR) and (yT, yM, yB) is identified. In the x direction (xL, xM, xR) represents the x-positions of the maximal pixel and its two neighbors. Similarly, (yT, yM, yB) represents the y-positions of the maximal pixel and its top and bottom neighbors. In the middle column of the 3×3 array, correlation values CT, CM, and CB are defined at positions (xM, yT), (xM, yM) and (xM, yB), respectively. Similarly, in the middle row of the 3×3 array, correlation values CL, CM, and CR are defined at positions (xL, yM), (xL, yM), and (xR, yM), respectively. Since CM is a minimum in the entire correlation area, it will have a value less than CL, CR, CT, and CB. Thus, to calculate a sub-pixel position that may provide better alignment than the (xM, yM) position, the mapping point (x,y) may be modified according to Eqs. 8 and 9. x = x M + C L - C R 2 C L - 4 C M + 2 C R , (Eq.  8) y = y M + C B - C T 2 C B - 4 C M + 2 C T , (Eq.  9)

Figure US06535650-20030318-M00004

where x is computed from positions (xL, yM), (xM, yM) and (xR, yM) by using the correlation values (CL, CM and CR), while y is computed from position (xM, yB), (xM, yM), (xM, yT) by using the correlation values (CB, CM and CT). After the mapping points are selected, the perspective transform is performed (step 120). The current frame Pk is mapped using the perspective transform back to a reference frame, which in some embodiments may be the first frame Pl.

In a perspective transform, the forward mapping function may be defined by Eqs. 10 and 11. x = a 11 u + a 21 v + a 31 a 13 u + a 23 v + a 33 , and (Eq.  10) y = a 12 u + a 22 v + a 32 a 13 u + a 23 v + a 33 , (Eq.  11)

Figure US06535650-20030318-M00005

where (x,y) is the coordinate of the current frame Pk; (u,v) is the coordinate of the reference frame Pl; and the nine coefficients aij (i=1-3, j=1-3) define a 3×3 matrix T that represents the transform. T = [ a 11 a 12 a 13 a 21 a 22 a 23 a 31 a 32 a 33 ] (Eq.  12)

Figure US06535650-20030318-M00006

Without loss of generality, the matrix T can be normalized so that the coefficient a33 is set to unity (a33=1). This leaves eight degrees of freedom for the projective mapping. The eight coefficients can be determined by establishing correspondence between four points in the reference frame Pl and the current frame Pk. Let (un, vn) and (xn, yn), for n=0, 1, 2, 3 be these four points in the Pl and Pk frames, respectively, and assuming a33=1, Eqs. 10 and 11 will yield

x=a 11 u+a 21 v+a 31 −a 13 ux−a 23 vx,  (Eq. 13)

y=a 12 u+a 22 v+a 32 −a 13 uy−a 23 vy.  (Eq. 14)

Applying these new x, y equations (13 and 14) to the four pairs of corresponding points in frames Pl and Pk yields an 8×8 system, as shown in Eq. 15. X = [ u 0 v 0 1 0 0 0 - u 0 x 0 - v 0 x 0 u 1 v 1 1 0 0 0 - u 1 x 1 - v 1 x 1 u 2 v 2 1 0 0 0 - u 2 x 2 - v 2 x 2 u 3 v 3 1 0 0 0 - u 3 x 3 - v 3 x 3 0 0 0 u 0 v 0 1 - u 0 y 0 - v 0 y 0 0 0 0 u 1 v 1 1 - u 1 y 1 - v 1 y 1 0 0 0 u 2 v 2 1 - u 2 y 2 - v 2 y 2 0 0 0 u 3 v 3 1 - u 3 y 3 - v 3 y 3 ] B , (Eq.  15)

Figure US06535650-20030318-M00007

where A=[a11 a21 a31 a12 a22 a32 a13 a23] are the unknown coefficients, and X=[x0 x1 x2 x3 y0 y1 y2 y3] are the known coordinates of the selected mapping points in the current frame Pk. Once the coefficients aij (i=1-3, j=1-3) are determined, the matrix T is known and can be used to perform the mapping of the frame Pk to the reference frame Pl. The mapping performed using Eqs. 13 and 14 is done on a pixel-by-pixel basis.

At the kth frame, a set of k−1 transforms Tj (j=1, . . . , k−1) has been created to map the frame Pk back to the coordinates of Pl. The transforms Tj (j=1, . . . , k−1) are multiplied together to form a resultant transform that is applied to Pk. The aligned frame Pk is then added to the composite image using known techniques.

Next, for the given frame Pk, the accumulated errors of the perspective transforms may be calculated and error correction performed in the error correction phase (step 122). The next frame Pk+1 is then retrieved and the frame processed (step 100).

Referring to FIG. 6, to further improve the accuracy of the stitching process, the stitching program 16 according to some embodiments may first group the captured frames (step 402) into multiple strips according to the direction in which the camera 12 is scanned over the target scene. From these strips, partial mosaics each including a subset of all the captured frames may be constructed.

The perspective transforms described above may then be applied to these partial mosaics to stitch the partial mosaics together to form a final mosaic. After the partial mosaics are identified, the correlation process of FIG. 2 is performed to stitch together frames in each partial mosaic (step 404). After the frames in each partial mosaic have been aligned and combined, each partial mosaic is then treated as any ordinary frame, and the partial mosaics are stitched together using the process described above (step 406). A new set of perspective transforms are derived to align the partial mosaics for placement in the final mosaic.

Accumulated correlation error is proportional to the number of images processed. By breaking the total number of images into multiple sets (the partial mosaics), the accumulated correlation error may be reduced.

Referring to FIG. 7, to group the scanned frames into partial mosaics (step 402), the stitching program 16 according to some embodiments first obtains the position of the next frame (step 202), which may be the position of a corner of the frame or at the center of the frame. Next, the program 16 determines if the frame obtained is the last frame (step 204), and if so, the frame is stored in the current partial mosaic and the process is completed.

However, if the frame is not the last frame, then the program 16 determines if the direction of the current frame Pk is consistent with the previous frame Pk−1 (step 206). If not, then the partial mosaic has ended. The frame Pk is stored in a new partial mosaic (step 208) and the position of the next frame Pk+1 is retrieved (step 202). However, if the direction of the current frame Pk is consistent with the previous frame Pk−1, the frame Pkis stored in the current partial mosaic and the next frame is retrieved.

Referring to FIG. 8, a flow for detecting and distributing residual errors is illustrated. To scan in a scene, the user may be instructed to end the image scan where he or she began. In this way, the first and last frames Pl and PN should overlap. Applying all intermediate projective transforms to the last frame PN should thus in theory make PN align perfectly with the first frame Pl. However, slight errors (e.g., on the order of a fraction of a pixel) may occur in the calculation of each transform Tj. If the number N of captured images is large, slight errors will accumulate to significant values as the transforms Tl, . . . , TN are multiplied.

This alignment error (referred to as the “residual alignment error”) is calculated and stored in a residual transform R. The matrix R represents the residual transformation and is defined by Eq. 16. R = [ 1 + δ 11 δ 12 0 δ 21 1 + δ 22 0 δ 31 δ 32 1 ] (Eq.  16)

Figure US06535650-20030318-M00008

If N frames are captured, then a matrix F is defined as the Nth root of R as approximated by Eq. 17 F = [ 1 + δ 11 N δ 12 N 0 δ 21 N 1 + δ 22 N 0 δ 31 N δ 32 N 1 ] (Eq.  17)

Figure US06535650-20030318-M00009

where δij (i=1-3, j=1-2) represent the amount of the accumulated errors of the coefficients aij (i=1-3, j=1-3), respectively. Because the accumulated errors are small, R is close to identity and the product of any two δijs is negligible. As a result, the derived matrix F is a close approximation of the Nth root of R.

To perform the error correction, the stitching program 16 starts at the first frame and retrieves the correlation error (Fi) for the next frame Pi (step 302). Next, the program 16 checks if the frame is the last frame to correct (step 304), and if so, the error correction process is completed. If not, the program 16 analyzes to determine if the current frame Pi is sufficiently overlapped with any of previous frames (Pi to Pi). Whether a frame P is sufficiently overlapped with a previously captured frame may be predetermined or dynamically set based on prior correlations. If there is not sufficient overlap, then error correction is not needed on the frame Pi and the next frame correlation error is retrieved (step 302).

However, if sufficient overlap exists between Pi and one of Pl−Pi−l (referred to as Px), then the correlation error Fi is applied to the transform Tx,i which represents the transform from frame Px to Pi, according to Eq. 18.

T x,i ←F i T x,i.  (Eq. 18)

In this manner, the residual error R is distributed uniformly over all intermediate projective transformations by modifying each transform T. The error correction is forward propagated by passing the corrected transform Tx,i for further corrections (step 310). After this modification of the transforms T, the first and last frames Pland PN will become perfectly aligned.

Referring to FIG. 9 in some embodiments, the computer 14 may include a microprocessor 80 that runs the stitching program 16 according to embodiments of the invention. The system memory 88, the microprocessor 80, and the bridge/system controller circuitry 84 are all coupled to a host bus 82. The bridge circuitry 84 provides an interface from the host bus 82 to a downstream bus 99 that is coupled to an I/O controller 90 and a network interface card at 92, as examples. The computer 14 may also have, as examples, a CD-ROM or DVD-ROM drive 100, a floppy disk drive 94, and/or a hard disk drive 96.

According to some embodiments, the stitching program 16 may be stored on a suitable mass storage medium, such as the CD-ROM or DVD-ROM drive 100, the floppy disk drive 94, or the hard disk drive 96. During execution, the program 16 or portions of the program 16 may be loaded into the system memory 88 for execution by the microprocessor 80. In addition, the captured frames may be stored in one of the listed storage media or in the camera 12, or alternatively, in an external location.

Referring to FIG. 10, the camera 12, in some embodiments, includes optical components 60 that focus an optical image to be captured onto an array of pixel sensors 69 (e.g., a CMOS pixel sensor array) which electrically captures the image. Under control of a controller 62, an analog-to-digital (A/D) converter 64 receives analog signals from the sensor array 69 and furnishes the signals to other components in the camera 12 for further processing. These other components may include a scaling unit 66 that scales the resolution of the captured frame, if desired, and a compression unit 66 to compress the size of the frame that is transmitted over the communication link 15 through a bus interface 70.

In some embodiments, the stitching program 16 or a portion of the stitching program may be executed in the digital camera 12 by the controller 62, with the camera 12 in effect becoming the computer that is capable of running the stitching program. In such cases, the computer system 14 may not be required.

While the invention has been disclosed with respect to a limited number of embodiments, those skilled in the art will appreciate numerous modifications and variations therefrom. It is intended that the appended claims cover all such modifications and variations as fall within the spirit and scope of the invention.

Claims (25)

What is claimed is:
1. A digital imaging system comprising:
an imaging device to capture a plurality of images representing a scene;
a processor; and
a program executable on the processor to:
identify high detail regions in each image;
identify mapping points in the high detail regions;
perform projective mappings of the images using the identified mapping points to align the images; and
combine the images into a composite image that represents the scene.
2. The digital imaging system of claim 1, wherein the program is further executable to identify a high detail region by measuring the amount of detail in a selected region in each image.
3. The digital imaging system of claim 2, wherein the program is further executable to move the selected region along a predetermined path through the image until a region having a minimum amount of detail is found.
4. The digital imaging system of claim 3, wherein the predetermined path starts in a comer of the image and follows a spiral pattern.
5. The digital imaging system of claim 1, wherein the program is further executable to correct accumulated error resulting from performing the projective mappings of the captured images.
6. The digital imaging system of claim 5, wherein the accumulated error is distributed over all intermediate projective mappings.
7. The digital imaging system of claim 1, wherein the program is executable to select mapping points in each image based on a comparison of high-detail regions between two images.
8. The digital imaging system of claim 7, wherein a correlation function is calculated in the comparison, and the mapping point is selected if a minimum is found.
9. The digital imaging system of claim 8, wherein the program is further executed to indicate a correlation error if a minimum of the correlation function cannot be found.
10. The digital imaging system of claim 8, wherein the program is further executable to indicate a redundancy error if more than one minima of the correlation function is found.
11. The digital imaging system of claim 7, wherein the program is further executable to select four mapping points in each image.
12. The digital imaging system of claim 1, wherein the program is executable to align the images with respect to a reference image.
13. The digital imaging system of claim 1, wherein the program is further executable to automatically identify the mapping points in the high detail regions.
14. A computer-readable storage medium in a digital imaging system having a camera, the storage medium containing a program comprising instructions for causing the system to:
receive a plurality of digitized images each representing a portion of a scene;
identify mapping points in feature-rich regions of each image;
align the images based on the mapping points in each image to compensate for misalignment caused by movement of the camera; and
combine the images to form a composite image representing the scene.
15. The computer-readable storage medium of claim 14, wherein the program further comprises instructions to cause the system to align the images by performing perspective transforms of the images using the identified mapping points.
16. The computer-readable storage medium of claim 14, wherein the program further comprises instructions to cause the system to identify the feature-rich regions in each image.
17. The computer-readable storage medium of claim 16, wherein the program further comprises instructions to cause the system to compare a feature-rich region in a first image with a corresponding region in a second image, and to select a mapping point based on the comparison.
18. The digital imaging system of claim 14, wherein the program is further executable to automatically identify mapping points in feature-rich regions of each image.
19. In a digital imaging system, a method of generating a high resolution image representing a target scene, the method comprising:
capturing, using an imaging device, multiple images each representing a portion of the target scene;
a program running in the digital imaging system aligning the images to remove misalignment due to movement of the imaging device; and
the program combining the images to form a composite image representing the scene,
wherein aligning the images comprise identifying a feature-rich region by selecting a predetermined region and moving the predetermined region along a predetermined path in the image until a predetermined amount of detail is found.
20. The method of claim 19, wherein the multiple images are captured in rapid succession.
21. The method of claim 19, further comprising scanning the camera across the target scene and capturing the images at a high rate to reduce the amount of movement of the camera between any two successive images.
22. The method of claim 19, wherein aligning the images further comprises identifying a mapping point in the feature-rich region in each image.
23. A digital imaging system comprising:
an imaging device to capture a plurality of images representing a scene;
a processor; and
a program executable on the processor to:
group the captured images into partial mosaics;
automatically identify mapping points in high details regions in each image of each partial mosaic;
align images based on mapping points in each image of each partial mosaic;
combine the images in each partial mosaic; and
combine into partial mosaic into a composite image that represents the scene.
24. The digital imaging system of claim 23, wherein the program is further executable to align the partial mosaics with respect to each other.
25. The digital imaging system of claim 23, wherein the program is further executable to stitch the partial mosaics into the composite image that represents the scene.
US09120257 1998-07-21 1998-07-21 Creating high resolution images Expired - Lifetime US6535650B1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US09120257 US6535650B1 (en) 1998-07-21 1998-07-21 Creating high resolution images

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US09120257 US6535650B1 (en) 1998-07-21 1998-07-21 Creating high resolution images

Publications (1)

Publication Number Publication Date
US6535650B1 true US6535650B1 (en) 2003-03-18

Family

ID=22389163

Family Applications (1)

Application Number Title Priority Date Filing Date
US09120257 Expired - Lifetime US6535650B1 (en) 1998-07-21 1998-07-21 Creating high resolution images

Country Status (1)

Country Link
US (1) US6535650B1 (en)

Cited By (64)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020018589A1 (en) * 2000-06-02 2002-02-14 Beuker Rob Anne Method and apparatus for merging images into a composit image
US20020159654A1 (en) * 2001-04-25 2002-10-31 Nikon Corporation Method for processing an image of a concrete construction
US20030076361A1 (en) * 2001-09-12 2003-04-24 Haruo Hatanaka Image synthesizer, image synthesis method and computer readable recording medium having image synthesis processing program recorded thereon
US20030091214A1 (en) * 2001-11-14 2003-05-15 Gould Dwayne Martin Computerized pattern texturing
US20030122938A1 (en) * 2001-12-27 2003-07-03 Takuji Yoshida Resolution correction apparatus, resolution correction program, and computer-readable recording medium having same program recorded thereon
US20030133009A1 (en) * 1999-04-09 2003-07-17 Carl S Brown System and method for detecting with high resolution a large, high content field
US20030223057A1 (en) * 2002-02-06 2003-12-04 Ramsey Craig C. Wireless substrate-like sensor
US20040008903A1 (en) * 2002-07-15 2004-01-15 Samsung Electronics Co., Ltd. Image quality enhancement circuit and method using inter-frame correlativity
US20040086193A1 (en) * 2002-08-28 2004-05-06 Fuji Photo Film Co., Ltd. Video image synthesis method, video image synthesizer, image processing method, image processor, and programs for executing the synthesis method and processing method
US20040175028A1 (en) * 2003-03-03 2004-09-09 Kla-Tencor Technologies Corporation Method and apparatus for identifying defects in a substrate surface by using dithering to reconstruct under-sampled images
US20040189674A1 (en) * 2003-03-31 2004-09-30 Zhengyou Zhang System and method for whiteboard scanning to obtain a high resolution image
US20040223051A1 (en) * 1999-09-16 2004-11-11 Shmuel Peleg System and method for capturing and viewing stereoscopic panoramic images
US20050041736A1 (en) * 2003-05-07 2005-02-24 Bernie Butler-Smith Stereoscopic television signal processing method, transmission system and viewer enhancements
US20050200706A1 (en) * 2003-10-14 2005-09-15 Makoto Ouchi Generation of static image data from multiple image data
US20050224899A1 (en) * 2002-02-06 2005-10-13 Ramsey Craig C Wireless substrate-like sensor
US20050224902A1 (en) * 2002-02-06 2005-10-13 Ramsey Craig C Wireless substrate-like sensor
US20050233770A1 (en) * 2002-02-06 2005-10-20 Ramsey Craig C Wireless substrate-like sensor
US20060023276A1 (en) * 2004-07-27 2006-02-02 Samsung Electronics Co., Ltd. Digital imaging apparatus capable of creating panorama image and a creating method thereof
US7006707B2 (en) 2001-05-03 2006-02-28 Adobe Systems Incorporated Projecting images onto a surface
WO2006032184A1 (en) * 2004-09-20 2006-03-30 Fuchen Wang A scanning method for image photographing
US20060104541A1 (en) * 2004-11-15 2006-05-18 Baker Henry H Methods and systems for producing seamless composite images without requiring overlap of source images
US7095905B1 (en) * 2000-09-08 2006-08-22 Adobe Systems Incorporated Merging images to form a panoramic image
US7103236B2 (en) 2001-08-28 2006-09-05 Adobe Systems Incorporated Methods and apparatus for shifting perspective in a composite image
US20060256397A1 (en) * 2005-05-12 2006-11-16 Lexmark International, Inc. Method and system for combining images
US20070008312A1 (en) * 2005-07-08 2007-01-11 Hui Zhou Method for determining camera position from two-dimensional images that form a panorama
US20070097266A1 (en) * 2005-11-02 2007-05-03 Apple Computer, Inc. Spatial and temporal alignment of video sequences
US20070222462A1 (en) * 2006-02-21 2007-09-27 Gardner Delrae H Capacitive distance sensing in semiconductor processing tools
US20080106608A1 (en) * 2006-11-08 2008-05-08 Airell Richard Clark Systems, devices and methods for digital camera image stabilization
US20080228430A1 (en) * 2007-03-12 2008-09-18 Cyberoptics Semiconductor, Inc. Wireless sensor for semiconductor processing systems
US7428019B2 (en) * 2001-12-26 2008-09-23 Yeda Research And Development Co. Ltd. System and method for increasing space or time resolution in video
US20080231291A1 (en) * 2006-02-21 2008-09-25 Ramsey Craig C Capacitive Distance Sensing In Semiconductor Processing Tools
US20080239314A1 (en) * 2006-09-29 2008-10-02 Bonciolini Dennis J Substrate-like particle sensor
US20080246493A1 (en) * 2007-04-05 2008-10-09 Gardner Delrae H Semiconductor Processing System With Integrated Showerhead Distance Measuring Device
US20080263879A1 (en) * 2005-06-21 2008-10-30 East Precision Measuring Tool Co., Ltd. Level with Magnetic Device
US20090015268A1 (en) * 2007-07-13 2009-01-15 Gardner Delrae H Device and method for compensating a capacitive sensor measurement for variations caused by environmental conditions in a semiconductor processing environment
US20090028462A1 (en) * 2007-07-26 2009-01-29 Kensuke Habuka Apparatus and program for producing a panoramic image
US20090060348A1 (en) * 2007-08-28 2009-03-05 Donald Martin Monro Determination of Image Similarity
US20090226111A1 (en) * 2008-03-06 2009-09-10 Kabushiki Kaisha Toshiba Image processing apparatus and method thereof
US20090309853A1 (en) * 2008-06-13 2009-12-17 Polyvision Corporation Electronic whiteboard system and assembly with optical detection elements
US7660464B1 (en) 2005-12-22 2010-02-09 Adobe Systems Incorporated User interface for high dynamic range merge image selection
US20110102915A1 (en) * 2009-10-30 2011-05-05 Michael Pham Device to create high definition photos
US20110122223A1 (en) * 2009-11-24 2011-05-26 Michael Gruber Multi-resolution digital large format camera with multiple detector arrays
US20110122300A1 (en) * 2009-11-24 2011-05-26 Microsoft Corporation Large format digital camera with multiple optical systems and detector arrays
US7978925B1 (en) 2005-04-16 2011-07-12 Apple Inc. Smoothing and/or locking operations in video editing
WO2012000650A1 (en) * 2010-06-28 2012-01-05 Precitec Kg A method for classifying a multitude of images recorded by a camera observing a processing area and laser material processing head using the same
US8306274B2 (en) 2010-05-25 2012-11-06 The Aerospace Corporation Methods for estimating peak location on a sampled surface with improved accuracy and applications to image correlation and registration
US8368774B2 (en) 2010-11-22 2013-02-05 The Aerospace Corporation Imaging geometries for scanning optical detectors with overlapping fields of regard and methods for providing and utilizing same
US8385662B1 (en) 2009-04-30 2013-02-26 Google Inc. Principal component analysis based seed generation for clustering analysis
US8391634B1 (en) 2009-04-28 2013-03-05 Google Inc. Illumination estimation for images
US8396325B1 (en) * 2009-04-27 2013-03-12 Google Inc. Image enhancement through discrete patch optimization
WO2013112295A1 (en) * 2012-01-25 2013-08-01 Audience, Inc. Image enhancement based on combining images from multiple cameras
US8558899B2 (en) 2009-11-16 2013-10-15 The Aerospace Corporation System and method for super-resolution digital time delay and integrate (TDI) image processing
US8583656B1 (en) 2009-04-23 2013-11-12 Google Inc. Fast covariance matrix generation
US8611695B1 (en) 2009-04-27 2013-12-17 Google Inc. Large scale patch search
US8619148B1 (en) 2012-01-04 2013-12-31 Audience, Inc. Image correction after combining images from multiple cameras
US8645870B2 (en) 2005-03-31 2014-02-04 Adobe Systems Incorporated Preview cursor for image editing
US8699821B2 (en) 2010-07-05 2014-04-15 Apple Inc. Aligning images
US8798393B2 (en) 2010-12-01 2014-08-05 Google Inc. Removing illumination variation from images
US8938119B1 (en) 2012-05-01 2015-01-20 Google Inc. Facade illumination removal
US20160029887A1 (en) * 2012-03-17 2016-02-04 Wei Su Eye imaging apparatus with sequential illumination
US20170228904A1 (en) * 2012-07-12 2017-08-10 The Government Of The United States, As Represented By The Secretary Of The Army Stitched image
US9848773B2 (en) 2015-01-26 2017-12-26 Visunex Medical Systems Co. Ltd. Disposable cap for an eye imaging apparatus and related methods
US9986908B2 (en) 2014-06-23 2018-06-05 Visunex Medical Systems Co. Ltd. Mechanical features of an eye imaging apparatus
US10016178B2 (en) 2012-02-02 2018-07-10 Visunex Medical Systems Co. Ltd. Eye imaging apparatus and systems

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5550937A (en) * 1992-11-23 1996-08-27 Harris Corporation Mechanism for registering digital images obtained from multiple sensors having diverse image collection geometries
US5590261A (en) * 1993-05-07 1996-12-31 Massachusetts Institute Of Technology Finite-element method for image alignment and morphing
US5696848A (en) * 1995-03-09 1997-12-09 Eastman Kodak Company System for creating a high resolution image from a sequence of lower resolution motion images
US5706416A (en) * 1995-11-13 1998-01-06 Massachusetts Institute Of Technology Method and apparatus for relating and combining multiple images of the same scene or object(s)
US5798761A (en) * 1996-01-26 1998-08-25 Silicon Graphics, Inc. Robust mapping of 2D cursor motion onto 3D lines and planes
US5907626A (en) * 1996-08-02 1999-05-25 Eastman Kodak Company Method for object tracking and mosaicing in an image sequence using a two-dimensional mesh
US5973700A (en) * 1992-09-16 1999-10-26 Eastman Kodak Company Method and apparatus for optimizing the resolution of images which have an apparent depth
US5999662A (en) * 1994-11-14 1999-12-07 Sarnoff Corporation System for automatically aligning images to form a mosaic image
US6018349A (en) * 1997-08-01 2000-01-25 Microsoft Corporation Patch-based alignment method and apparatus for construction of image mosaics
US6075905A (en) * 1996-07-17 2000-06-13 Sarnoff Corporation Method and apparatus for mosaic image construction

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5973700A (en) * 1992-09-16 1999-10-26 Eastman Kodak Company Method and apparatus for optimizing the resolution of images which have an apparent depth
US5550937A (en) * 1992-11-23 1996-08-27 Harris Corporation Mechanism for registering digital images obtained from multiple sensors having diverse image collection geometries
US5590261A (en) * 1993-05-07 1996-12-31 Massachusetts Institute Of Technology Finite-element method for image alignment and morphing
US5999662A (en) * 1994-11-14 1999-12-07 Sarnoff Corporation System for automatically aligning images to form a mosaic image
US5696848A (en) * 1995-03-09 1997-12-09 Eastman Kodak Company System for creating a high resolution image from a sequence of lower resolution motion images
US5706416A (en) * 1995-11-13 1998-01-06 Massachusetts Institute Of Technology Method and apparatus for relating and combining multiple images of the same scene or object(s)
US5798761A (en) * 1996-01-26 1998-08-25 Silicon Graphics, Inc. Robust mapping of 2D cursor motion onto 3D lines and planes
US6075905A (en) * 1996-07-17 2000-06-13 Sarnoff Corporation Method and apparatus for mosaic image construction
US5907626A (en) * 1996-08-02 1999-05-25 Eastman Kodak Company Method for object tracking and mosaicing in an image sequence using a two-dimensional mesh
US6018349A (en) * 1997-08-01 2000-01-25 Microsoft Corporation Patch-based alignment method and apparatus for construction of image mosaics

Cited By (122)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030133009A1 (en) * 1999-04-09 2003-07-17 Carl S Brown System and method for detecting with high resolution a large, high content field
US20040223051A1 (en) * 1999-09-16 2004-11-11 Shmuel Peleg System and method for capturing and viewing stereoscopic panoramic images
US7477284B2 (en) * 1999-09-16 2009-01-13 Yissum Research Development Company Of The Hebrew University Of Jerusalem System and method for capturing and viewing stereoscopic panoramic images
US20050157166A9 (en) * 1999-09-16 2005-07-21 Shmuel Peleg Digitally enhanced depth imaging
US20020018589A1 (en) * 2000-06-02 2002-02-14 Beuker Rob Anne Method and apparatus for merging images into a composit image
US7123779B2 (en) * 2000-06-02 2006-10-17 Koninklijke Philips Electronics, N.V. Method and apparatus for merging images into a composite image
US7646932B1 (en) 2000-09-08 2010-01-12 Adobe Systems Incorporated Merging images to form a panoramic image
US20060291747A1 (en) * 2000-09-08 2006-12-28 Adobe Systems Incorporated, A Delaware Corporation Merging images to form a panoramic image
US7095905B1 (en) * 2000-09-08 2006-08-22 Adobe Systems Incorporated Merging images to form a panoramic image
US7386188B2 (en) 2000-09-08 2008-06-10 Adobe Systems Incorporated Merging images to form a panoramic image
US20020159654A1 (en) * 2001-04-25 2002-10-31 Nikon Corporation Method for processing an image of a concrete construction
US7006707B2 (en) 2001-05-03 2006-02-28 Adobe Systems Incorporated Projecting images onto a surface
US7103236B2 (en) 2001-08-28 2006-09-05 Adobe Systems Incorporated Methods and apparatus for shifting perspective in a composite image
US7471849B2 (en) 2001-08-28 2008-12-30 Adobe Systems Incorporated Methods and apparatus for shifting perspective in a composite image
US20060274972A1 (en) * 2001-08-28 2006-12-07 Adobe Systems Incorporated Methods and apparatus for shifting perspective in a composition image
US20030076361A1 (en) * 2001-09-12 2003-04-24 Haruo Hatanaka Image synthesizer, image synthesis method and computer readable recording medium having image synthesis processing program recorded thereon
US7256799B2 (en) * 2001-09-12 2007-08-14 Sanyo Electric Co., Ltd. Image synthesizer, image synthesis method and computer readable recording medium having image synthesis processing program recorded thereon
US7356453B2 (en) 2001-11-14 2008-04-08 Columbia Insurance Company Computerized pattern texturing
US20030091214A1 (en) * 2001-11-14 2003-05-15 Gould Dwayne Martin Computerized pattern texturing
US7428019B2 (en) * 2001-12-26 2008-09-23 Yeda Research And Development Co. Ltd. System and method for increasing space or time resolution in video
US20030122938A1 (en) * 2001-12-27 2003-07-03 Takuji Yoshida Resolution correction apparatus, resolution correction program, and computer-readable recording medium having same program recorded thereon
US7228007B2 (en) * 2001-12-27 2007-06-05 Sharp Kabushiki Kaisha Resolution correction apparatus, resolution correction program, and computer-readable recording medium having same program recorded thereon
US20030223057A1 (en) * 2002-02-06 2003-12-04 Ramsey Craig C. Wireless substrate-like sensor
US7283255B2 (en) 2002-02-06 2007-10-16 Cyberoptics Semiconductor, Inc. Wireless substrate-like sensor
US7289230B2 (en) * 2002-02-06 2007-10-30 Cyberoptics Semiconductors, Inc. Wireless substrate-like sensor
US20060151606A1 (en) * 2002-02-06 2006-07-13 Cyberoptics Semiconductor, Inc. Wireless substrate-like sensor
US20050233770A1 (en) * 2002-02-06 2005-10-20 Ramsey Craig C Wireless substrate-like sensor
US20050224902A1 (en) * 2002-02-06 2005-10-13 Ramsey Craig C Wireless substrate-like sensor
US20050224899A1 (en) * 2002-02-06 2005-10-13 Ramsey Craig C Wireless substrate-like sensor
US7456977B2 (en) 2002-02-06 2008-11-25 Cyberoptics Semiconductor, Inc. Wireless substrate-like sensor
US7286716B2 (en) * 2002-07-15 2007-10-23 Samsung Electronics Co., Ltd. Image quality enhancement circuit and method using inter-frame correlativity
US20040008903A1 (en) * 2002-07-15 2004-01-15 Samsung Electronics Co., Ltd. Image quality enhancement circuit and method using inter-frame correlativity
US20100195927A1 (en) * 2002-08-28 2010-08-05 Fujifilm Corporation Method and device for video image processing, calculating the similarity between video frames, and acquiring a synthesized frame by synthesizing a plurality of contiguous sampled frames
US8805121B2 (en) * 2002-08-28 2014-08-12 Fujifilm Corporation Method and device for video image processing, calculating the similarity between video frames, and acquiring a synthesized frame by synthesizing a plurality of contiguous sampled frames
US8275219B2 (en) * 2002-08-28 2012-09-25 Fujifilm Corporation Method and device for video image processing, calculating the similarity between video frames, and acquiring a synthesized frame by synthesizing a plurality of contiguous sampled frames
US8078010B2 (en) 2002-08-28 2011-12-13 Fujifilm Corporation Method and device for video image processing, calculating the similarity between video frames, and acquiring a synthesized frame by synthesizing a plurality of contiguous sampled frames
US20110255610A1 (en) * 2002-08-28 2011-10-20 Fujifilm Corporation Method and device for video image processing, calculating the similarity between video frames, and acquiring a synthesized frame by synthesizing a plurality of contiguous sampled frames
US7729563B2 (en) * 2002-08-28 2010-06-01 Fujifilm Corporation Method and device for video image processing, calculating the similarity between video frames, and acquiring a synthesized frame by synthesizing a plurality of contiguous sampled frames
US20040086193A1 (en) * 2002-08-28 2004-05-06 Fuji Photo Film Co., Ltd. Video image synthesis method, video image synthesizer, image processing method, image processor, and programs for executing the synthesis method and processing method
US20120321220A1 (en) * 2002-08-28 2012-12-20 Fujifilm Corporation Method and device for video image processing, calculating the similarity between video frames, and acquiring a synthesized frame by synthesizing a plurality of contiguous sampled frames
US7227984B2 (en) * 2003-03-03 2007-06-05 Kla-Tencor Technologies Corporation Method and apparatus for identifying defects in a substrate surface by using dithering to reconstruct under-sampled images
US20040175028A1 (en) * 2003-03-03 2004-09-09 Kla-Tencor Technologies Corporation Method and apparatus for identifying defects in a substrate surface by using dithering to reconstruct under-sampled images
US20040189674A1 (en) * 2003-03-31 2004-09-30 Zhengyou Zhang System and method for whiteboard scanning to obtain a high resolution image
US7301548B2 (en) * 2003-03-31 2007-11-27 Microsoft Corp. System and method for whiteboard scanning to obtain a high resolution image
US20050104902A1 (en) * 2003-03-31 2005-05-19 Microsoft Corporation System and method for whiteboard scanning to obtain a high resolution image
US7119816B2 (en) * 2003-03-31 2006-10-10 Microsoft Corp. System and method for whiteboard scanning to obtain a high resolution image
US20050041736A1 (en) * 2003-05-07 2005-02-24 Bernie Butler-Smith Stereoscopic television signal processing method, transmission system and viewer enhancements
US20050200706A1 (en) * 2003-10-14 2005-09-15 Makoto Ouchi Generation of static image data from multiple image data
US7535497B2 (en) * 2003-10-14 2009-05-19 Seiko Epson Corporation Generation of static image data from multiple image data
WO2005112448A2 (en) * 2004-05-07 2005-11-24 Cobalt Entertainment, Llc Stereoscopic television signal processing method, transmission system and viewer enhancements
WO2005112448A3 (en) * 2004-05-07 2006-07-13 Cobalt Entertainment Llc Stereoscopic television signal processing method, transmission system and viewer enhancements
US20060023276A1 (en) * 2004-07-27 2006-02-02 Samsung Electronics Co., Ltd. Digital imaging apparatus capable of creating panorama image and a creating method thereof
US7593635B2 (en) * 2004-07-27 2009-09-22 Samsung Electronics Co., Ltd. Digital imaging apparatus capable of creating panorama image and a creating method thereof
WO2006032184A1 (en) * 2004-09-20 2006-03-30 Fuchen Wang A scanning method for image photographing
US20060104541A1 (en) * 2004-11-15 2006-05-18 Baker Henry H Methods and systems for producing seamless composite images without requiring overlap of source images
US7619658B2 (en) * 2004-11-15 2009-11-17 Hewlett-Packard Development Company, L.P. Methods and systems for producing seamless composite images without requiring overlap of source images
US10114520B2 (en) 2005-03-31 2018-10-30 Adobe Systems Incorporated Preview cursor for image editing
US8645870B2 (en) 2005-03-31 2014-02-04 Adobe Systems Incorporated Preview cursor for image editing
US8750636B2 (en) 2005-04-16 2014-06-10 Apple Inc. Smoothing and/or locking operations in video editing
US7978925B1 (en) 2005-04-16 2011-07-12 Apple Inc. Smoothing and/or locking operations in video editing
US20060256397A1 (en) * 2005-05-12 2006-11-16 Lexmark International, Inc. Method and system for combining images
US20080263879A1 (en) * 2005-06-21 2008-10-30 East Precision Measuring Tool Co., Ltd. Level with Magnetic Device
US7565029B2 (en) 2005-07-08 2009-07-21 Seiko Epson Corporation Method for determining camera position from two-dimensional images that form a panorama
US20070008312A1 (en) * 2005-07-08 2007-01-11 Hui Zhou Method for determining camera position from two-dimensional images that form a panorama
US20070097266A1 (en) * 2005-11-02 2007-05-03 Apple Computer, Inc. Spatial and temporal alignment of video sequences
US7912337B2 (en) 2005-11-02 2011-03-22 Apple Inc. Spatial and temporal alignment of video sequences
US9530220B2 (en) 2005-11-02 2016-12-27 Apple Inc. Spatial and temporal alignment of video sequences
US9998685B2 (en) 2005-11-02 2018-06-12 Apple Inc. Spatial and temporal alignment of video sequences
US20110116767A1 (en) * 2005-11-02 2011-05-19 Christophe Souchard Spatial and temporal alignment of video sequences
US7660464B1 (en) 2005-12-22 2010-02-09 Adobe Systems Incorporated User interface for high dynamic range merge image selection
US8027538B1 (en) * 2005-12-22 2011-09-27 Adobe Systems Incorporated User interface for high dynamic range merge image selection
US20080231291A1 (en) * 2006-02-21 2008-09-25 Ramsey Craig C Capacitive Distance Sensing In Semiconductor Processing Tools
US7893697B2 (en) 2006-02-21 2011-02-22 Cyberoptics Semiconductor, Inc. Capacitive distance sensing in semiconductor processing tools
US20070222462A1 (en) * 2006-02-21 2007-09-27 Gardner Delrae H Capacitive distance sensing in semiconductor processing tools
US7804306B2 (en) 2006-02-21 2010-09-28 CyterOptics Semiconductor, Inc. Capacitive distance sensing in semiconductor processing tools
US20080239314A1 (en) * 2006-09-29 2008-10-02 Bonciolini Dennis J Substrate-like particle sensor
US8823933B2 (en) 2006-09-29 2014-09-02 Cyberoptics Corporation Substrate-like particle sensor
US20080106608A1 (en) * 2006-11-08 2008-05-08 Airell Richard Clark Systems, devices and methods for digital camera image stabilization
US7714892B2 (en) 2006-11-08 2010-05-11 Avago Technologies Ecbu Ip (Singapore) Pte. Ltd. Systems, devices and methods for digital camera image stabilization
US7778793B2 (en) 2007-03-12 2010-08-17 Cyberoptics Semiconductor, Inc. Wireless sensor for semiconductor processing systems
US20080228430A1 (en) * 2007-03-12 2008-09-18 Cyberoptics Semiconductor, Inc. Wireless sensor for semiconductor processing systems
US20080246493A1 (en) * 2007-04-05 2008-10-09 Gardner Delrae H Semiconductor Processing System With Integrated Showerhead Distance Measuring Device
US20090015268A1 (en) * 2007-07-13 2009-01-15 Gardner Delrae H Device and method for compensating a capacitive sensor measurement for variations caused by environmental conditions in a semiconductor processing environment
US8588546B2 (en) * 2007-07-26 2013-11-19 Morpho, Inc. Apparatus and program for producing a panoramic image
US20090028462A1 (en) * 2007-07-26 2009-01-29 Kensuke Habuka Apparatus and program for producing a panoramic image
US20090060348A1 (en) * 2007-08-28 2009-03-05 Donald Martin Monro Determination of Image Similarity
US8155448B2 (en) * 2008-03-06 2012-04-10 Kabushiki Kaisha Toshiba Image processing apparatus and method thereof
US20090226111A1 (en) * 2008-03-06 2009-09-10 Kabushiki Kaisha Toshiba Image processing apparatus and method thereof
US20090309853A1 (en) * 2008-06-13 2009-12-17 Polyvision Corporation Electronic whiteboard system and assembly with optical detection elements
US8583656B1 (en) 2009-04-23 2013-11-12 Google Inc. Fast covariance matrix generation
US8571349B1 (en) * 2009-04-27 2013-10-29 Google Inc Image enhancement through discrete patch optimization
US8396325B1 (en) * 2009-04-27 2013-03-12 Google Inc. Image enhancement through discrete patch optimization
US8611695B1 (en) 2009-04-27 2013-12-17 Google Inc. Large scale patch search
US8391634B1 (en) 2009-04-28 2013-03-05 Google Inc. Illumination estimation for images
US8385662B1 (en) 2009-04-30 2013-02-26 Google Inc. Principal component analysis based seed generation for clustering analysis
US20110102915A1 (en) * 2009-10-30 2011-05-05 Michael Pham Device to create high definition photos
US8558899B2 (en) 2009-11-16 2013-10-15 The Aerospace Corporation System and method for super-resolution digital time delay and integrate (TDI) image processing
US20110122223A1 (en) * 2009-11-24 2011-05-26 Michael Gruber Multi-resolution digital large format camera with multiple detector arrays
US20110122300A1 (en) * 2009-11-24 2011-05-26 Microsoft Corporation Large format digital camera with multiple optical systems and detector arrays
US8665316B2 (en) 2009-11-24 2014-03-04 Microsoft Corporation Multi-resolution digital large format camera with multiple detector arrays
US8542286B2 (en) 2009-11-24 2013-09-24 Microsoft Corporation Large format digital camera with multiple optical systems and detector arrays
US8306274B2 (en) 2010-05-25 2012-11-06 The Aerospace Corporation Methods for estimating peak location on a sampled surface with improved accuracy and applications to image correlation and registration
US9105077B2 (en) 2010-06-28 2015-08-11 Precitec Kg Method for classifying a multitude of images recorded by a camera observing a processing area and laser material processing head using the same
WO2012000650A1 (en) * 2010-06-28 2012-01-05 Precitec Kg A method for classifying a multitude of images recorded by a camera observing a processing area and laser material processing head using the same
US8760537B2 (en) 2010-07-05 2014-06-24 Apple Inc. Capturing and rendering high dynamic range images
US10038855B2 (en) 2010-07-05 2018-07-31 Apple Inc. Operating a device to capture high dynamic range images
US8699821B2 (en) 2010-07-05 2014-04-15 Apple Inc. Aligning images
US9420198B2 (en) 2010-07-05 2016-08-16 Apple Inc. Operating a device to capture high dynamic range images
US8885978B2 (en) 2010-07-05 2014-11-11 Apple Inc. Operating a device to capture high dynamic range images
US8368774B2 (en) 2010-11-22 2013-02-05 The Aerospace Corporation Imaging geometries for scanning optical detectors with overlapping fields of regard and methods for providing and utilizing same
US8798393B2 (en) 2010-12-01 2014-08-05 Google Inc. Removing illumination variation from images
US8619148B1 (en) 2012-01-04 2013-12-31 Audience, Inc. Image correction after combining images from multiple cameras
WO2013112295A1 (en) * 2012-01-25 2013-08-01 Audience, Inc. Image enhancement based on combining images from multiple cameras
US10016178B2 (en) 2012-02-02 2018-07-10 Visunex Medical Systems Co. Ltd. Eye imaging apparatus and systems
US9907467B2 (en) 2012-03-17 2018-03-06 Visunex Medical Systems Co. Ltd. Eye imaging apparatus with a wide field of view and related methods
US20160029887A1 (en) * 2012-03-17 2016-02-04 Wei Su Eye imaging apparatus with sequential illumination
US9907468B2 (en) * 2012-03-17 2018-03-06 Visunex Medical Systems Co. Ltd. Eye imaging apparatus with sequential illumination
US8938119B1 (en) 2012-05-01 2015-01-20 Google Inc. Facade illumination removal
US20170228903A1 (en) * 2012-07-12 2017-08-10 The Government Of The United States, As Represented By The Secretary Of The Army Stitched image
US20170228904A1 (en) * 2012-07-12 2017-08-10 The Government Of The United States, As Represented By The Secretary Of The Army Stitched image
US9986908B2 (en) 2014-06-23 2018-06-05 Visunex Medical Systems Co. Ltd. Mechanical features of an eye imaging apparatus
US9848773B2 (en) 2015-01-26 2017-12-26 Visunex Medical Systems Co. Ltd. Disposable cap for an eye imaging apparatus and related methods

Similar Documents

Publication Publication Date Title
Gruen et al. Geometrically constrained multiphoto matching
US6738154B1 (en) Locating the position and orientation of multiple objects with a smart platen
US6594403B1 (en) Systems and methods for registering scanned documents
US7006709B2 (en) System and method deghosting mosaics using multiperspective plane sweep
US7305146B2 (en) Tilt correction of electronic images
US6993184B2 (en) Object extraction method, and image sensing apparatus using the method
US6385334B1 (en) System and method for adjusting stereo camera
US7268803B1 (en) Image processing method and apparatus, digital camera, image processing system and computer readable medium
US5497236A (en) Method and apparatus for distortion correction of scanned images
US6493031B1 (en) Visual information processing method and apparatus for extracting feature quantities from a two-dimensional image signal
US6173087B1 (en) Multi-view image registration with application to mosaicing and lens distortion correction
US6353678B1 (en) Method and apparatus for detecting independent motion in three-dimensional scenes
US20040165786A1 (en) System and method for converting whiteboard content into an electronic document
US7046401B2 (en) Camera-based document scanning system using multiple-pass mosaicking
US20070030396A1 (en) Method and apparatus for generating a panorama from a sequence of video frames
US6570612B1 (en) System and method for color normalization of board images
US5835241A (en) Method for determining the profile of a bound document with structured light
US6754369B1 (en) License plate reading apparatus and method
US20130010073A1 (en) System and method for generating a depth map and fusing images from a camera array
US20040189674A1 (en) System and method for whiteboard scanning to obtain a high resolution image
US6434280B1 (en) System and method for generating super-resolution-enhanced mosaic images
US20080199044A1 (en) Image Processing Apparatus, Image Processing Method, and Program
US20080019611A1 (en) Imaging System Performance Measurement
US20040085256A1 (en) Methods and measurement engine for aligning multi-projector display systems
US6400848B1 (en) Method for modifying the perspective of a digital image

Legal Events

Date Code Title Description
AS Assignment

Owner name: INTEL CORPORATION, CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:POULO, RICHARD J.;NGUYEN, VU X.;BOSTROM, VARECK S.;AND OTHERS;REEL/FRAME:009337/0532;SIGNING DATES FROM 19980714 TO 19980720

FPAY Fee payment

Year of fee payment: 4

FPAY Fee payment

Year of fee payment: 8

FPAY Fee payment

Year of fee payment: 12