WO2020168094A1 - Simultaneous depth profile and spectral measurement - Google Patents

Abstract

Techniques are disclosed for 3D spectral scanning for simultaneous depth profile and spectral measurement and for using 3D spectral scanning for material processing. A structured light source and a spectral imaging device are configured to acquire images of an object illuminated by the structured light source at different frequencies. Spectral segmentation and 3D reconstruction are performed using the images to identify material differences and a depth profile of the object. A computer representation of the object showing spatial dimensions and material differences in response to spectral segmentation and 3D reconstruction is generated as an output. The output informs detection and diagnosis of issues to automate material processing steps in industrial manufacturing and other applications.

Description

SIMULTANEOUS DEPTH PROFILE AND SPECTRAL MEASUREMENT

CROSS REFERENCE TO OTHER APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No.

62/806,384 entitled METHOD TO AUTOMATICALLY DISTINGUISH COMPONENTS IN ASSEMBLIES USING 3D SPECTRAL SCANNING filed February 15, 2019, which is incorporated herein by reference for all purposes, and U.S. Provisional Patent Application No. 62/806,418 entitled AUTOMATED IMAGE SEGMENTATION BY MERGING 3D SENSING AND SPECTRAL IMAGING filed February 15, 2019, which is incorporated herein by reference for all purposes, and U.S. Provisional Patent Application No. 62/893,537 entitled

SIMULTANEOUS DEPTH PROFILE AND SPECTRAL MEASUREMENT filed August 29, 2019, which is incorporated herein by reference for all purposes.

BACKGROUND OF THE INVENTION

[0002] In the field of industrial metrology, there are increasing industry needs in machine vision, automated inspection, and robot perception. An under-served area in this field is the integrated capture of 3D shape data of parts and assemblies coupled together with material identification. Current systems lack an ability to represent an object showing a 3D shape of its constituent parts based on each part’s respective material composition. There is a need for an integrated inspection instrument that can do both 3D shape and material identification simultaneously such that spatial and spectral data are automatically co-registered and interpreted as an aggregate data set in order to render the 3D shape of an object’s constituent parts based on each part’s material composition.

BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Various embodiments of the invention are disclosed in the following detailed description and the accompanying drawings.

[0004] FIG. 1 is a block diagram of an embodiment of a 3D spectral scanning system for simultaneous depth profile and spectral measurement. [0005] FIG. 2 is a flowchart of an embodiment of a method for 3D spectral scanning system for simultaneous depth profile and spectral measurement that can be performed by the exemplary system depicted in FIG. 1.

[0006] FIG. 3 shows a sequence of multi-shot binary coded structured patterns projected on an object or target of interest.

[0007] FIG. 4 shows a sequence of multi-shot gray code structured patterns projected on an object or target of interest.

[0008] FIG. 5 is a flowchart of an embodiment of a method of performing spectral segmentation using the images to identify material differences in the object.

[0009] FIG. 6 is a flowchart of another embodiment of a method of performing spectral segmentation using the images to identify material differences in the object.

[0010] FIG. 7 depicts a processing chain for 3D spectral scanning data acquisition and processing flow as described herein.

[0011] FIG. 8 shows the representation of four induced endmembers extracted by PPI.

[0012] FIG. 9 is a flowchart of an embodiment of a method of performing spectral segmentation that includes generating a lull scan data cube.

[0013] FIG. 10A is a flowchart of an embodiment of a method of performing spectral segmentation that includes matching spectrally distinct signatures or endmembers to a particular reference signature associated with a particular material.

[0014] FIG. 10B is a flowchart of an embodiment of a method of performing spectral segmentation for generating a 3D reconstruction or 3D point cloud based on or in response to a target spectral signature associated with a material of interest.

[0015] FIG. IOC is a flowchart of an embodiment of a method of performing spectral segmentation using a segmentation algorithm.

[0016] FIG. 11 is a diagram illustrating an embodiment of a measurement device.

[0017] FIG. 12 is a diagram illustrating an embodiment of a measurement device.

[0018] FIG. 13 is a diagram illustrating an embodiment of a measurement device. [0019] FIG. 14 is a diagram illustrating an embodiment of a measurement device.

[0020] FIG. 15 is a diagram illustrating an embodiment of a measurement device.

[0021] FIG. 16 is a diagram illustrating an embodiment of a measurement system.

[0022] FIG. 17 is a flow diagram illustrating an embodiment of a process for making a simultaneous depth profile and spectral measurement.

[0023] FIG. 18 is a flow diagram illustrating an embodiment of a process for determining a depth calibration.

[0024] FIG. 19 is a flow diagram illustrating an embodiment of a process for determining a depth measurement.

[0025] FIG. 20 is a flow diagram illustrating an embodiment of a process for determining a spectral calibration.

[0026] FIG. 21 is a flow diagram illustrating an embodiment of a process for determining a spectral measurement.

[0027] FIG. 22 is a diagram illustrating an embodiment of data from a device measuring height and a spectrum of a target simultaneously.

[0028] FIG. 23 is a diagram illustrating an embodiment of data from a device measuring height and a spectrum of a target simultaneously.

[0029] FIG. 24 is a diagram illustrating an embodiment of a twyman green structured light system using a broadband source.

[0030] FIG. 25 is a diagram illustrating an embodiment of a twyman green structured light system using a series of laser diode sources.

[0031] FIG. 26 depicts an example of finding the mean background irradiance by calculating the mean of the two images.

[0032] FIG. 27 depicts a synthetic wavelength example.

[0033] FIG. 28 depicts equivalent synthetic wavelengths. [0034] FIG. 29 depicts a diagram of a structured light system using laser sources and a polarization grating.

[0035] FIG. 30 depicts a diagram of a structured light system using fiber coupled LEDs and polarization grating.

[0036] FIG. 31 shows a table listing examples of various industrial applications for 3D spectral scanning as disclosed herein.

[0037] FIG. 32 is a block diagram of an embodiment of a material processing system or platform 3200 that uses 3D spectral scanning to execute material processing steps.

[0038] FIG. 33 is a flowchart of an embodiment of a method for material processing using

3D spectral scanning that can be performed by the exemplary system depicted in FIG. 32.

[0039] FIG. 34 illustrates an application of 3D spectral scanning to provide 3D segmentation by material for automated optical inspection.

[0040] FIG. 35 shows the difference between traditional 3D scanning and 3D spectral scanning as disclosed herein.

DETAILED DESCRIPTION

[0041] The invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored on and or provided by a memory coupled to the processor. In this specification, these implementations, or any other form that the invention may take, may be referred to as techniques.

In general, the order of the steps of disclosed processes may be altered within the scope of the invention. Unless stated otherwise, a component such as a processor or a memory described as being configured to perform a task may be implemented as a general component that is temporarily configured to perform the task at a given time or a specific component that is manufactured to perform the task. As used herein, the term‘processor’ refers to one or more devices, circuits, and or processing cores configured to process data, such as computer program instructions.

[0042] A detailed description of one or more embodiments of the invention is provided below along with accompanying figures that illustrate the principles of the invention. The invention is described in connection with such embodiments, but the invention is not limited to any embodiment. The scope of the invention is limited only by the claims and the invention encompasses numerous alternatives, modifications and equivalents. Numerous specific details are set forth in the following description in order to provide a thorough understanding of the invention. These details are provided for the purpose of example and the invention may be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical material that is known in the technical fields related to the invention has not been described in detail so that the invention is not unnecessarily obscured.

[0043] The machine vision market is evolving rapidly with increasing demands for speed, complex identification, inspection, and sorting. The drivers come from a need for automation of industrial processes. The performance of existing machine vision systems can be enhanced by extracting more information available in images taken of objects in industrial processes. For example, spectral data and 3D shape information can be leveraged through advanced data analysis (such as support vector machine analysis, K-means clustering, and neural networks) to enable decisions. In some embodiments, chemometric algorithms can be applied to perform true unknown analysis to identify materials from large stock libraries.

[0044] Accelerating changes in industries such as semiconductors, food processing, pharmaceuticals, and consumer electronics, have created demand for machine vision tools to enhance automation and productivity. In the field of industrial metrology, there are increasing industry needs in machine vision, automated inspection, and robot perception. An under-served area in this field is the integrated capture of 3D shape data of parts and assemblies coupled together with material identification. Currently, instruments for 3D shape and material identification are only available as separate systems— a 3D scanner can provide shape information about an object, while a spectral imaging system can provide information about different materials in an object typically rendered in a two-dimensional xy-pixel area. Flowever, what these systems lack is an ability to perform integrated 3D scanning for simultaneous depth profile and spectral measurement that enables generating a representation of an object showing the 3D shape of its constituent parts based on each part’s respective material composition.

[0045] Amidst this changing industrial landscape, a need exists for an integrated 3D inspection or measuring system that can perform both 3D shape and material identification simultaneously. Such a system enables spatial and spectral data to be automatically co-registered and interpreted as an aggregate data set and provides an ability to render the 3D shape of an object’s constituent parts based on each part’s material composition. As discussed in further detail herein, the addition of 3D shape information coupled with spectral data results in a system that can perform object identification and classification based on both the object’s material composition and its 3D shape profile.

[0046] Additionally, once an object has been identified, detected, or classified in an inspection, the rendering of its components as a 3D point cloud segmented by material can be used to inform detection and diagnosis of issues to automate material processing steps or actions in industrial manufacturing or other applications (e.g., additive manufacturing applications, removal processing).

[0047] In some embodiments, techniques are described for reverse engineering components in an assembly by providing an integrated 3D spectral scanning system configured to provide 3D surface data and to distinguish sub-component parts in an assembly by their material spectral signatures. In some cases, an integrated inspection technique is disclosed that can perform both 3D shape and material identification simultaneously such that spatial and spectral data are automatically co-registered and interpreted as an aggregate data set.

[0048] Current systems for inspection or reverse engineering of parts and assemblies collect geometric information by 3D imaging and scanning methods. The resulting CAD file has the geometric 3D point cloud of the part. However the systems cannot distinguish between different materials or sub components in an assembly. Multi-component assemblies are labelled as one component. The user has to input the different sub-component assignments manually.

[0049] In some embodiments, an automated system for parts inspection and reverse engineering is provided configured to distinguish and label components made of different materials within an assembly. The disclosed system comprises a 3D imaging system, a spectral imaging camera, and engineering software configured to represent an object in a computer (e.g., using CAD). In some cases, the 3D imaging system comprises a structured light (SL) camera for extracting a detailed 3D geometric mesh of a part under inspection. The spectral imaging camera is configured to provide images of the object at different wavelengths, thereby capturing spectral signatures that enable detection of different materials in the part under inspection. The software is configured to merge the 3D surface and material differences to output a computer representation of the part showing spatial dimensions and different components based on a material distinction.

[0050] Existing commercial software allows users to inspect parts in 3D and to manipulate those parts in a computer. The disclosed system can provide an improvement to such products by providing an ability to identify whether a part is made of different materials and, if so, to tag associated surfaces, and assign them as different sub-components automatically (e.g., in a CAD file).

[0051] In some embodiments, a 3D spectral scanning or imaging technique is disclosed that can discriminate material composition and 3D object shape in industrial machine vision applications. In particular, a 3D spectral scanning computer system for simultaneous depth profile and spectral measurement comprises a processor and a memory coupled to the processor configured to provide the processor with instructions that when executed cause the processor to: receive multispectral or hyperspectral images of an object from which 3D data can be extracted, perform spectral segmentation using the images to identify material differences in the object; perform 3D reconstruction using the images to identify a depth profile based on the object; and output a computer representation of the object showing spatial dimensions and material differences in response to the spectral segmentation and the 3D reconstruction.

[0052] While exemplary methods are described herein for performing spectral segmentation

3D reconstruction, the disclosed techniques for 3D spectral scanning using structured illumination are not limited to any particular method of performing either spectral segmentation or 3D reconstruction. In some embodiments, an important element is that the images of the object or target of interest used as a basis for spectral segmentation and for 3D reconstruction are acquired or captured in a manner so as to include both spectral data and 3D data or data from which a depth profile can be extracted. In some embodiments, using the disclosed technique, spectral segmentation and 3D reconstruction are performed using the images to provide a 3D point cloud segmented by material. In some cases, 3D reconstruction is performed based on or in response to spectral segmentation. In other cases, spectral segmentation is performed based on or in response to 3D reconstruction.

[0053] In some embodiments, the images are acquired or captured such that a 3D point cloud or 3D geometric mesh of the object can be extracted from the images. Performing 3D reconstruction comprises extracting a 3D point cloud or 3D geometric mesh of the object from the images. Extracting a 3D point cloud or 3D geometric mesh of the object from the images is in some cases based on the spectral segmentation and identified material differences.

[0054] In some embodiments, the images of the object comprise a sequence of multispectral or hyperspectral image frames generated by projecting a sequence of structured light patterns from the structured light source on a scene comprising the object, wherein the sequence of structured light patterns is configured to provide 3D data of the scene. [0055] In some embodiments, a 3D spectral scanning system for simultaneous depth profile and spectral measurement comprises a structured light source and a spectral imaging device configured to acquire or capture images of an object illuminated by the structured light source at different frequencies. The system fiirther comprises a processor and a memory coupled to the processor configured to provide the processor with instructions that when executed cause the processor to: perform spectral segmentation using the images to identify material differences in the object; perform 3D reconstruction using the images to identify a depth profile based on the object; and output a computer representation of the object showing spatial dimensions and material differences in response to the spectral segmentation and the 3D reconstruction.

[0056] One area for the application of 3D spectral scanning is for material processing applications such as additive manufacturing and removal processing. For example, material processing steps may be controlled or initiated based on or in response to the 3D shape and material composition data output by a 3D scanning system.

[0057] In some embodiments, a material processing system comprises obtaining 3D shape and material composition data output by 3D scanning an object using the 3D spectral scanning techniques as described herein.

[0058] In some embodiments, a method for material processing using 3D spectral scanning comprises performing 3D spectral scanning on an object. The 3D spectral scanning may be performed according to any of the disclosed techniques as described herein. The method also includes performing detection and diagnosis to identify issues associated with the object based on the 3D spectral scanning; generating a decision in response to the detection and diagnosis;

outputting a control action in response to the decision; and actuating or controlling a device in response to the control action to address an issue associated with the object. These steps may be repeated for a subsequent 3D spectral scanning of the same object or for 3D spectral scanning of a subsequent object.

[0059] In some cases, a material processing system or platform is configured to obtain both the 3D shape and the material of an object (such as a part being manufactured) using the 3D spectral scanning techniques disclosed herein, detect and diagnose issues associated with the object (e.g., recognize a contaminated, deteriorated, degraded, or otherwise substandard or abnormal portion of the object, provide a recommended action to address the issues, and execute the action (e.g., clean up contaminated portions or overlay or replace deteriorated or degraded portions while leaving normal portions intact). [0060] FIG. 1 is a block diagram of an embodiment of a 3D spectral scanning system for simultaneous depth profile and spectral measurement. The system 100 can be configured to provide 3D data and to distinguish parts in an object by their material spectral signatures. In some embodiments and as shown in FIG. 1, the system 100 comprises a structured light source (e.g., Structured Light Source 110) and a spectral imaging device (e.g., Spectral Imaging Device 120) configured to acquire or capture images of an object (e.g., Object 101).

[0061] In some cases, the structured light source includes a projector. The structured light source can also include other structured light systems and sources, some of which are discussed in more detail below with respect to various embodiments. As an example, the structured light source includes a broadband source, multiple individual sources, laser-line sources, a series of laser diode sources that can be coupled using a grating or a combination of fiber-coupled LEDs. In some embodiments, the structured light source comprises a multi-spectral fringe projection system. The fringes can be generated interferometrically, creating different fringe frequencies per illumination wavelength. In some embodiments, the fringes are generated using laser sources and a polarization grating. In some cases, the fringes are generated by interfering the +1 and -1 orders of a polarization grating which are orthogonally circularly polarized.

[0062] Imaging is performed using a spectral imaging device. In this case, an image is obtained or acquired such that the image captures spectral and 3D data of an object, target of interest, or a scene comprising an object or target of interest. The spectral imaging device can be a multispectral or a hyperspectral camera. In some embodiments, the spectral imaging device comprises a detector. A detector can take many forms (e.g., 2-dimensional imaging device, multi wavelength detector, snapshot mosaic sensor, spectrum detector, multi-wavelength imaging spectrometer, pushbroom spectrometer, etc.). The spectral imaging device can also include an imaging spectrometer such as an IRIS imaging spectrometer, a multiple-image Fourier transform spectrometer, or a compact birefringent Fourier transform spectrometer. In some embodiments, imaging is performed with a multi-wavelength imaging spectrometer through multiple phase shifts achieved by the rotation of a polarizer.

[0063] In some embodiments, a structured light camera (e.g, shown at 102 of FIG. 1) comprises a structured light source and a spectral imaging device. In other embodiments, a structured light camera comprises or provides a structured light source and the 3D spectral scanning system comprises the structured light camera and a spectral imaging device. As will be discussed in fiirther detail below, in some embodiments, the structured light source is configured to project a sequence of structured light patterns on a scene comprising an object. The sequence of structured light patterns projected by the structured light source is configured to provide 3D data of the scene used to perform 3D reconstruction from images acquired or captured by the spectral imaging device. The object is illuminated by the structured light source at different frequencies such that a 3D point cloud or 3D geometric mesh of the object can be extracted from the images. The spectral imaging device can be a multispectral or hyperspectral camera.

[0064] 3D spectral scanning system 100 includes a computer system 130 made up of various subsystems including least one microprocessor subsystem also referred to as a central processing unit, or CPU (not shown). The CPU can be implemented by a single-chip processor or by multiple processors. In some embodiments the CPU is a general purpose digital processor that controls the operation of the computer system 130. Using instructions retrieved from a memory (not shown) coupled to the processor, the CPU controls the reception and manipulation of input data (e.g., spectral images received from Spectral Imaging Camera 120), and the output and display of data on output devices.

[0065] In some embodiments, the memory is configured to provide the processor with instructions that when executed cause the processor to: perform spectral segmentation using the images to identify material differences in the object; perform 3D reconstruction using the images to identify a depth profile based on the object; and output a computer representation of the object showing spatial dimensions and material differences in response to the spectral segmentation and the 3D reconstruction. In some embodiments, performing 3D reconstruction comprises extracting a 3D point cloud or 3D geometric mesh of the object from the images and a computer representation of the object showing spatial dimensions and material differences based on the spectral segmentation and the 3D point cloud or 3D geometric mesh is output by the system. In some cases, extracting a 3D point cloud or 3D geometric mesh of the object from the images is based on the spectral segmentation and identified material differences.

[0066] Spectral Segmentation Module 140 is configured to perform spectral segmentation using the images to identify material differences in the object. For example, spectral segmentation can be performed based on spectral imaging data of an object (e.g., spectral images received from Spectral Imaging Device 120). Spectral imaging data includes multispectral or hyperspectral images received from a multispectral camera or hyperspectral camera respectively.

[0067] 3D Reconstruction Module 150 is configured to perform 3D reconstruction based on or using the images acquired or captured by the spectral imaging device to identify a depth profile based on the object. In some embodiments, 3D Reconstruction Module 150 is configured to perform 3D reconstruction including by extracting a 3D point cloud or 3D geometric mesh of the object from the images. In some cases, extracting a 3D point cloud or 3D geometric mesh of the object from the images is based on the spectral segmentation and identified material differences.

[0068] Finally, computer system 130 outputs a computer representation of the object (e.g.,

Output 160) showing spatial dimensions and material differences in response to the spectral segmentation and the 3D reconstruction.

[0069] In some embodiments, performing spectral segmentation includes obtaining spectral signatures from multispectral or hyperspectral images to identify material differences in the object. In cases where performing 3D reconstruction includes extracting a 3D point cloud or 3D geometric mesh of the object from the images, the system is configured to output a computer representation of the object (e.g., Output 160) showing spatial dimensions and material differences based on the spectral signatures and the 3D point cloud or 3D geometric mesh.

[0070] FIG. 2 is a flowchart of an embodiment of a method 200 for 3D spectral scanning for simultaneous depth profile and spectral measurement that can be performed by the exemplary system 100 depicted in FIG. 1.

[0071] As shown in FIG. 2, the method 200 includes illuminating an object with a structured light source at 210 and capturing images of the object illuminated by the structured light source at different frequencies at 220. As an example, a structured light source (e.g., Structured Light Source 110 of FIG. 1) illuminates the object at different frequencies. In some cases, the object is illuminated such that a 3D point cloud or 3D geometric mesh of the object can be extracted from images taken of the object. The images are acquired or captured using a spectral imaging device (e.g., Spectral Imaging Device 120 of FIG. 1). The spectral imaging device can be a multispectral or hyperspectral camera.

[0072] At 230, spectral segmentation is performed using the images to identify material differences in the object. For example, Spectral Segmentation Module 140 can perform spectral segmentation using the images to identify material differences in the object. In particular, and as described in more detail below, spectral segmentation can be performed by obtaining spectral signatures from the images based on spectral imaging data of an object (e.g., spectral images received from Spectral Imaging Device 120). Spectral imaging data includes multispectral or hyperspectral images received from a multispectral camera or hyperspectral camera respectively. [0073] At 240, 3D reconstruction is performed using the images to identify a depth profile based on the object. In some cases, the images are acquired or captured such that a 3D point cloud or 3D geometric mesh of the object can be extracted from the images. In such cases, 3D reconstruction comprises extracting a 3D point cloud or 3D geometric mesh of the object from the images. In some embodiments and as discussed in more detail below, extracting a 3D point cloud or 3D geometric mesh of the object from the images is based on the spectral segmentation and identified material differences.

[0074] At 250, a computer representation of the object showing spatial dimensions and material differences in response to the spectral segmentation and the 3D reconstruction is output by the system. In some cases, outputting a computer representation of the object showing spatial dimensions and material differences is based on spectral segmentation and the 3D point cloud or 3D geometric mesh.

[0075] STRUCTURED LIGHT IMAGING SYSTEM CALIBRATION

[0076] An essential part of structured light imaging, as in any 3D imaging technology, is the calibration of both the spectral imaging device (e.g., a multispectral or hyperspectral camera) and the structured light source (e.g., a projector) to ensure the two elements are properly configured to acquire or capture images of an object or target of interest. The camera and projector calibration procedures establish the relationship between a pixel on a 2D image (in camera coordinates) and a straight line in 3D space (world coordinates) along with the object location by taking the lens distortion into consideration. Calibration procedures are performed before the acquisition of data (e.g., capturing images of an object or target of interest).

[0077] Camera calibration a key aspect in 3D imaging technology as the 3D image reconstruction depends on the camera calibration approach. The aim of the calibration of both the camera and projector is to obtain their intrinsic and extrinsic parameters including parameters of lens distortion and parameters of rigid transformations from camera and projector coordinates to world coordinates. The intrinsic parameters include the focal length, principle point, pixel skew factor or skew coefficient, distortion coefficients, and pixel size. The extrinsic parameters include the rotation and translation from a world coordinate system to a camera or projector coordinate system.

[0078] In some embodiments and prior to image acquisition, a camera calibration procedure is performed using a set of images taken at different angles and distances of a known calibration object such as a planar checkboard (calibration image) that has distinctive comers that are easy to detect. The comers of each square are detected and extracted in each perspective image scene of the checkboard. All the intrinsic and extrinsic parameters of the camera are determined and used in the 3D reconstruction of scanned objects. The next step is to calibrate the projector and refine the camera calibration against any imperfect planarity of the checkerboard that is used in this calibration.

[0079] The calibration of a projector can be performed in a similar manner as for the camera by projecting a pre-calibrated checkboard pattern and extracting the square comers. The projected points are evaluated to match those of the camera calibration images. To find the intrinsic and extrinsic parameters of the projector, as the spatial relationship between the camera and the planar checkboard is already known, only the correspondence between projector and camera needs to be established.

[0080] In some cases, a Single-Shot Spatial Structured Illumination (SI) Light method is performed to calibrate the projector. This method requires one shot per calibration pose and adopts spatial multiplexed patterns, in which the feature point correspondences are uniquely encoded in a single structured light (SL) pattern. An advantage of this method is that it refines imperfectly calibrated camera and projector parameters due to imperfect planar calibration board using a bundle adjustment method. To proceed with this calibration, after completing the selection of four extreme comers in each image, all the comers of all squares in the projected image are selected and extracted. The intersection is computed between the rays from the projector toward the camera and used to establish the matching between the camera and projector.

[0081] The calibration process described herein for the camera and structured light source or structured light imaging system is an example of how these elements may be calibrated. Other calibration methods and systems including various algorithms and techniques for determination of the intrinsic and extrinsic parameters of structured light imaging system can be applied without limiting the scope of the disclosed 3D spectral scanning techniques.

[0082] After the calibration process, the system can be used for data acquisition to acquire or capture images of the scene comprising the object or target of interest. In particular, once the camera and structured light source have been calibrated, structured light patterns can be projected on the object or target to enable 3D geometric shape reconstruction from the images.

[0083] STRUCTURED LIGHT FOR 3D SHAPE MEASUREMENT [0084] The measurement of 3D shapes using structured light projection (SLP) is a non- contact, high-resolution optical method to extract the 3D profiles of objects. The method is based on camera imaging of an object while illuminated by a projected light pattern, such as a grid of bright and dark lines. When observed at an angle, the projected patterns are deformed by the object’s surface shape. Surface height information is extracted directly from the analysis of the deformed grid pattern recorded in the image.

[0085] In some embodiments, the structured light source comprises a structured light illumination system. As an example, a structured illumination system for projecting a sequence of structured light (SL) patterns from a structured light source on a scene comprising an object or target of interest includes a 2D array detector, a light projector, and a processing unit or processor for projecting and acquiring SL patterned scenes. The projector can use preloaded SL patterns or user defined patterns that can be generated and loaded into the projector. A camera is used to acquire or capture a sequence of 2D images of the object under the structured light illumination. In most cases, the projected SL patterns are distorted by the object’s surface features and therefore the 3D information of scanned objects can inferred by correcting the distortion.

[0086] Depth extraction can be performed in a number of different ways depending on the nature of the projected patterns. A single shot (one projected pattern and one image) can provide enough information to extract the 3D profile. Alternatively, a multi-shot approach can be used in which a sequence of multiple, distinct patterns can be projected and the associated images collected. Multi-shot methods can improve the information content and resolution of the 3D profile produced.

[0087] The projected pattern can be modulated in numerous ways. For example, simple binary patterns of alternating bright and dark stripes or even a projected laser line can be used. The latter is an example of one-dimensional pattern. Alternatively, patterns can be modulated in two dimensions, as is the case with a binary checkerboard pattern. More complex two-dimensional patterns can be formed such as arrays of segmented stripes, dot arrays, pseudo-random 2D arrays, and random speckle patterns. Another variation in SLP methodology is the use of grayscale indexing or the introduction of color coding to the projected pattern. The modulation depth of projected patterns can be binary (light or dark) or it may have a gradual grading throughout the pattern.

[0088] In some embodiments, the images of the object comprise a sequence of multispectral or hyperspectral image frames generated by projecting a sequence of structured light patterns from the structured light source (e.g., Structured Light Source 110 of FIG. 1) on a scene comprising the object. The sequence of structured light patterns is configured to provide 3D data of the scene. Multiple frames of data are acquired in order to complete the full depth/spectrum measurement of the scene. As described in more detail below, multi-shot sequential binary coded or gray coded patterns can be used in the 3D image reconstruction of an object or scanned target.

[0089] In some embodiments, in order to enable performing 3D reconstruction using the sequence of multispectral or hyperspectral image frames, a multi-shot sequential binary coded or gray code structured light illumination 3D imaging technique is used. These imaging techniques are based on the use of structured light with special designed 2D intensity patterns that allow the estimation of depth maps of objects by measuring the deformation in the projected SL patterns compared to SL reference patterns.

[0090] FIG. 3 shows an example of a sequence 300 of structured light patterns configured to provide 3D data of an object in a scene. In particular, FIG. 3 shows a sequence of multi-shot binary coded structured patterns 300 projected on an object or target of interest. As shown in FIG. 3, the sequence 300 consists of black and white stripes that form a sequence of projection patterns such that each point on the surface of the object or target possesses a unique binary code that differs from any other code for a different point on the surface. These binary SL codifications encode a scene by using just two intensity values: all black (intensity 0) and all white (intensity 1). The 3D coordinates (x, y, z) of the scanned object or target can be computed along each horizontal and vertical line, thus reconstructing the 3D geometric shape of the scanned object or target. For a calibrated camera-projector system, these pixel entities can be converted into a world coordinate system and then 3D coordinates (x, y, z) of the scanned objects can be recovered via triangulation.

[0091] FIG. 4 shows another example of a sequence 400 of structured light patterns configured to provide 3D data of an object in a scene. In particular, FIG. 4 shows a sequence of multi-shot gray code structured patterns 400 projected on an object or target of interest. The multi shot sequential gray code structure illumination (SI) light can be advantageous in that it tends to be more robust with respect to noise and varying surface characteristics as compared to binary encoding. As in the case of binary coded structured patterns described above, the gray code designed 2D intensity patterns allow the estimation of the depth maps of objects by measuring the deformation in the projected SI light patterns compared to the SI light reference patterns. As shown in FIG. 4, the sequence 400 consists of black and white stripes that form a sequence of projection patterns such that each point on the surface of the object possesses a unique gray code that differs from any other codes of different points. The 3D coordinates (x, y, z) of the scanned target can be computed along each horizontal and vertical line via triangulation, thus reconstructing the 3D geometric shape of the scanned target.

[0092] SPECTRAL SEGMENTATION

[0093] In some embodiments, once the images have been acquired or captured (e.g., as shown at step 220 of FIG. 2), the system performs spectral segmentation (e.g., as shown at step 230 of FIG. 2) using the images to identify material differences in the object. Exemplary methods for performing spectral segmentation are discussed below with respect to FIGS. 5-10.

[0094] FIG. 5 is a flowchart of an embodiment of a method 500 of performing spectral segmentation (e.g., as shown at step 220 of FIG. 2) using the images to identify material differences in the object. In the example shown, the method 500 of performing spectral segmentation comprises storing each of the frames in the sequence of multispectral or hyperspectral image frames as a hypercube that captures spectral data and xy-pixel data at 510 and spectrally segmenting each hypercube to generate one or more abundance maps for each of the frames in the sequence of multispectral or hyperspectral image frames, wherein each abundance map is generated in response to a spectrally distinct signature or endmember at 520.

[0095] The acquired or captured images comprise both the structured illumination (SI) and spectral data (e.g., multispectral or hyperspectral data). In some embodiments, the acquired or captured images are stored as hypercubes. Each hypercube is a three dimensional matrix that contains spectral information in the third dimension organized by x, y pixel position in the first and second dimensions respectively. As an example, an RGB image can be thought of as a data cube with R, G and B values for each x, y pixel position, while a spectral image can be thought of as a hypercube with raw spectral data (for different wavelengths) for each x, y pixel position. In some embodiments, the different wavelengths comprise more than RGB wavelengths (e.g., multispectral and hyperspectral wavelengths).

[0096] FIG. 6 is a flowchart of an embodiment of a method 600 of performing spectral segmentation (e.g., as shown at step 220 of FIG. 2) using the images to identify material differences in the object. In the example shown, the method 600 of performing spectral segmentation comprises storing each of the frames in the sequence of multispectral or hyperspectral image frames as a hypercube that captures spectral data and xy-pixel data at 610. At 620, the method includes extracting one or more spectrally distinct signatures or endmembers based on the spectral data from each of the hypercubes. At 630, the method includes spectrally segmenting each hypercube to generate one or more abundance maps for each of the frames in the sequence of multispectral or hyperspectral image frames, wherein each abundance map is generated in response to an extracted spectrally distinct signature or endmember.

[0097] In some embodiments, the spectral data (e.g., multispectral or hyperspectral data) is processed by (e.g., by Spectral Segmentation Module 140 of FIG. 1) by extracting spectral signatures of materials and their associated abundance maps. A processing pipeline illustrating the data acquisition and processing flow is shown in the FIG. 7.

[0098] FIG. 7 depicts a processing chain 700 for 3D spectral scanning data acquisition and processing flow as described herein. As shown in FIG. 7, the processing chain 700 comprises different components and blocks that include a data acquisition block (e.g., shown at 701), a spectral segmentation block (e.g., shown at 702), and a 3D reconstruction block (e.g., shown at 703).

[0099] The data acquisition block 701 depicts the processing flow for acquiring or capturing images of an object illuminated by the structured light source at different frequencies. In the example shown, structured light patterns necessary for 3D reconstruction are preloaded (e.g., into a DLP projector) for projection to illuminate an object or target of interest. Each pattern is projected for a period of time configured to synchronize with the camera burst exposure. In some embodiments, the start of the scanning process (start scan acquisition) can be initiated by manually starting the DLP sequence and camera burst using their respective software applications.

Synchronization between the DLP projector and the camera can be performed using open loop control (e.g., manually by a human operator) or it can be automated through the use of an external “master” device such as an industrial programmable logic controller (PLC) capable of sub millisecond timing. Using a PLC results in reduced scanning time by eliminating wait states in open loop control as well as facilitating the use of a pulsed light source.

[0100] The spectral segmentation block 702 depicts the flow for performing spectral segmentation by processing the acquired images or frames an input source of raw data (e.g., spectral imaging camera), including by performing data filtration, masking, pre-processing, endmember extraction (EE), and abundance map reconstructions (AMR) to identify or detect different materials.

[0101] The processing chain includes acquiring the images or frames from the spectral imaging camera at 721 to generate a sequence of frames or complete scan at 722, wherein a frame (e.g., an image of an individual pattern) comprises a hypercube corresponding to the acquisition of one structured illumination pattern from the sequence of patterns necessary to produce a scan.

Here, a scan is an acquisition of the full set of structured illumination patterns necessary to reconstruct the 3D point cloud of one pose of the target object, wherein a pose is one specific location and field of view of a target object. For example, an object on a turn table will have one pose per rotational step or height change. It is often necessary to scan multiple poses of an object to build a 3D model. In other words, a scan can be thought of as the images of a complete sequence, wherein a sequence is a series of structured illumination patterns necessary to reconstruct a 3D point cloud. As described herein, a sequence contains multiple patterns, wherein a pattern comprises a ID series of black and white stripes (binary coded or gray code) or grayscale sinusoids (phase) programmed into a structured light source (e.g., a DLP projector for projecting structured illumination patterns). Each pattern is one of the sequence necessary to reconstruct 3D positions.

[0102] At 723, each frame of the sequence of frames (i.e., each hypercube in the sequence of hypercubes) is processed to extract one or more spectrally distinct signatures or endmembers based on the spectral data from each of the hypercubes. An endmember can be thought of as a material that is spectrally unique in the wavelength bands. An endmember is associated to the pixel spectra or spectral signature of materials after the pixel un-mixing and extraction or spectral segmentation process to generate abundance maps that display the fractional amount of material present at each pixel.

[0103] In some embodiments, principal component analysis (PCA) is used to select the best bands for classification, reduce the hypercube dimensionalities, and analyze their contents. PCA is based on the fact that neighboring bands of multispectral or hyperspectral images are highly correlated and often convey almost the same information about the object. This analysis is used to transform the original data so to remove the correlation among the bands.

[0104] ENDMEMBER EXTRACTION AND ABUNDANCE MAP GENERATION

[0105] In some embodiments, identification and extraction of endmembers from hypercubes and reconstruction of abundance maps is based on a linear spectral mixture analysis (LSMA) model. In some embodiments, the Pixel Purity Index (PPI) algorithm, an unsupervised spectral algorithm implemented in commercially available hyperspectral machine vision software, enables the determination and extraction of distinct spectral signatures existing in hypercubes. The PPI algorithm determines the spectral signatures associated to each pixel vector by calculating the spectral response of multiple components and by decomposing them into a collection of spectrally pure constituent spectra.

[0106] FIG. 8 shows the representation of four induced endmembers extracted by PPI. In particular, FIG. 8 is a plot of the reflectance of four extracted endmembers (e.g., shown at 801, 802, 803, and 804 respectively) from a hyperspectral data set using the PPI algorithms as a function of wavelength, where each line is one distinct spectrum in the image.

[0107] Returning to the spectral segmentation block 702 of FIG. 7, in some embodiments, each abundance map (Data Cube) is generated in response to an extracted spectrally distinct signature or endmember and the set of abundance maps or data cubes are combined into a single file at 725 to generate a full scan data cube (e.g., Full Scan Data Cube shown at 730). The Full Scan Data Cube (one data cube per frame) at 730 is a larger Data Cube representing each frame in a scan as an individual data cube. In the example shown, the Full Scan Data Cube is generated based on the abundance maps and captures spectrally distinct signature or endmember data and xy-pixel data for each of the frames in the sequence of multispectral or hyperspectral image frames.

[0108] FIG. 9 is a flowchart of an embodiment of a method 900 of performing spectral segmentation (e.g., as shown at step 220 of FIG. 2) that includes generating a full scan data cube (e.g., Full Scan Data Cube shown 730) as shown in the spectral segmentation block 702 of FIG. 7. In this case, the method 900 of performing spectral segmentation comprises storing each of the frames in the sequence of multispectral or hyperspectral image frames as a hypercube that captures spectral data and xy-pixel data at 910. At 920, one or more spectrally distinct signatures or endmembers are extracted based on the spectral data from each of the hypercubes. At 930, each hypercube is spectrally segmented to generate one or more abundance maps for each of the frames in the sequence of multispectral or hyperspectral image frames, wherein each abundance map is generated in response to an extracted spectrally distinct signature or endmember. As described with respect to FIG. 7 above, at 940, a full scan data cube is generated based on the abundance maps. The full scan data cube captures spectrally distinct signature or endmember data and xy- pixel data for each of the frames in the sequence of multispectral or hyperspectral image frames.

[0109] FIG. 10A is a flowchart of an embodiment of a method 1000 of performing spectral segmentation (e.g., as shown at step 220 of FIG. 2) that includes matching spectrally distinct signatures or endmembers to a particular reference signature associated with a particular material. The method 1000 of performing spectral segmentation comprises storing each of the frames in the sequence of multispectral or hyperspectral image frames as a hypercube that captures spectral data and xy-pixel data at 1010. At 1020, the method includes extracting one or more spectrally distinct signatures or endmembers based on the spectral data from each of the hypercubes. At 1030, the method includes matching one or more of the spectrally distinct signatures or endmembers to a particular reference signature, wherein the particular reference signature identifies or is associated with a particular material. At 1040, the method includes spectrally segmenting each hypercube to generate one or more abundance maps for each of the frames in the sequence of multispectral or hyperspectral image frames, wherein each abundance map is generated in response to an extracted spectrally distinct signature or endmember. At 1050, the method includes generating a lull scan data cube based on the abundance maps that captures spectrally distinct signature or endmember data and xy-pixel data for each of the frames in the sequence of multispectral or hyperspectral image frames.

[0110] In some embodiments, each abundance map is generated in response to a spectrally distinct signature or endmember associated with a material identified based on matching the one or more spectrally distinct signatures or endmembers to a particular reference signature identifying or associated with the material. In some cases, each of the reference signatures is associated with a different material and the reference signatures associated with different materials are stored in a spectra library (e.g., Library of spectra for materials shown at 724 of FIG. 7). In such cases, the system is configured to perform 3D spectral scans, extract spectral signatures or endmembers and match these to a library entry or reconstructed endmember corresponding to a specific spectral signature associated with a material. As discussed in fiirther detail below, materials common to a selected use case are spectrally distinct and can be distinguished by their spectral fingerprints using even simple algorithms.

[0111] In some embodiments and as described with respect to FIG. 10B, given a particular library entry or target spectral signature, the system is configured to extract spectral data from the hypercube based on or in response to the particular library entry or target spectral signature to generate an abundance map. In particular, the system is configured to match a target spectral signature to spectral data in the hypercube and spectrally segment the hypercube to generate an abundance map based on xy-pixel data associated with the target spectral signature. In some cases, the target spectral signature is associated with a material of interest.

[0112] FIG. 10B is a flowchart of an embodiment of a method 1001 of performing spectral segmentation (e.g., as shown at step 220 of FIG. 2) for generating a 3D reconstruction or 3D point cloud based on or in response to a target spectral signature associated with a material of interest. The method 1001 of performing spectral segmentation comprises storing each of the frames in the sequence of multispectral or hyperspectral image frames as a hypercube that captures spectral data and xy-pixel data at 1011. At 1021, the method includes matching a target spectral signature to the spectral data in the hypercube. In some cases, the target spectral signature is associated with a material of interest. At 1031, the method includes spectrally segmenting each hypercube to generate one or more abundance maps for each of the frames in the sequence of multispectral or hyperspectral image frames, wherein each abundance map is generated in response to the identified xy-pixel data based on matching the target spectral signature to the spectral data in the hypercube. At 1041, the method includes generating a full scan data cube based on the abundance maps.

[0113] FIG. IOC is a flowchart of an embodiment of a method 1002 of performing spectral segmentation (e.g., as shown at step 220 of FIG. 2) using a segmentation algorithm. In particular, the method 1002 includes storing each of the frames in the sequence of multispectral or hyperspectral image frames as a hypercube that captures spectral data and xy-pixel data at 1012 and spectrally segmenting each hypercube using a segmentation algorithm to generate one or more abundance maps for each of the frames in the sequence of multispectral or hyperspectral image frames, wherein each abundance map is generated in response to a spectrally distinct signature or endmember at 1022. In some embodiments, the segmentation algorithm includes unsupervised learning algorithms such as K-means clustering and neural networks.

[0114] DEPTH PROFILING AND 3D RECONSTRUCTION

[0115] Returning to FIG. 7, the 3D reconstruction block (e.g., shown at 703) depicts the processing flow for generating 3D point cloud data from the abundance maps. As shown in FIG. 7, the full scan data cube (e.g., Full Scan Data Cube at 730) is generated in the spectral segmentation block (e.g., shown at 702) based on the abundance maps (e.g., Data Cubes segmented by material). The full scan data cube captures spectrally distinct signature or endmember data and xy-pixel data for each of the frames in the sequence of multispectral or hyperspectral image frames.

[0116] At 731 , based on the full scan data cube, one or more abundance maps for each of the frames in the sequence of multispectral or hyperspectral image frames is identified or associated with a particular material.

[0117] At 732, a frame sequence or scan of one or more abundance maps associated with a particular material is generated for each identified material.

[0118] At 733, a 3D point cloud or 3D geometric mesh of the object is extracted or generated from the images based on the spectral segmentation and identified material differences. In this case, a 3D point cloud or 3D geometric mesh of the object is extracted or generated using a sequence of the one or more abundance maps associated with the particular material (e.g., from each frame sequence shown at 732). In this manner, different abundance maps associated to different extracted endmembers are reconstructed and using the sequences of reconstructed abundance maps, 3D spectral point clouds associated to each extracted endmember are reconstructed.

[0119] At 734, the 3D point cloud or 3D geometric mesh for each material (or each extracted endmember) are included as segments to form or generate a final point cloud (e.g., a segmented point cloud showing segmentation of the object by material). The segmented point cloud is displayed at 735.

[0120] COMPONENT SELECTION CONSIDERATIONS

[0121] The design of instruments to perform the two measurements simultaneously involves significant multiplexing of the data. While an advantage for the purposes of merging depth and spectral information, this poses unique design challenges to acquire all required information from the test object rapidly. Accordingly, the selection of particular components for the system often involves a tradeoff between spatial sampling and spectral sampling. This selection is informed by the considerations and requirements of the particular industrial application and objectives of performing 3D spectral scanning, including type of materials to be distinguished by spectral segmentation, and characteristics of the object or target and the scene for depth profiling and 3D reconstruction.

[0122] Due to computational considerations there is generally a tradeoff between spectral resolution and 3D resolution. For the selection of a spectral imaging component (e.g., multispectral versus hyperspectral camera), smaller datasets associated with multispectral versus hyperspectral reduce computational complexity with some implications to the selection of processing algorithms used for performing spectral segmentation. As an example, to distinguish a variety of materials commonly used in automotive assemblies, a broadband illumination source and multispectral imaging camera or system having an NIR (600-1000nm) wavelength range and a fairly limited wavelength resolution (e.g., 25 bands) is typically sufficient. This configuration provides a lower- risk optical layout as compared to hyperspectral imaging. However, in some situations, the spectral resolution of a lull hyperspectral system (100+ bands) may be required to differentiate certain materials. Additionally, the use of a broadband illumination source may not be able to achieve the same level of spatial accuracy as laser-line sources used in laser-based 3D analysis products. [0123] With respect to choosing a structured light source, the low light intensity of continuous halogen light sources in the NIR region is generally insufficient to enable 3D reconstruction (e.g., the generation of a 3D spectral point cloud) due to exposure time limitations.

A more powerful light source providing a higher light intensity (e.g., a xenon flashlamp) may be needed in certain applications where reducing 3D spectral scanning time is a concern. In some cases and in particular to reduce camera exposure time, a high intensity light source is preferable to permit multispectral or hyperspectral image collection. Projection brightness can also be improved by optimizing projector field of view with an appropriate selection of projector lens focal length (e.g., through a projector lens retrofit or supplementary optical system).

[0124] Signal intensity and dynamic range are both critical parameters in design of a 3D spectral scanning system. In some embodiments, the 3D point cloud reconstruction step is performed after the spectral segmentation step. In some cases, the 3D point cloud reconstruction algorithms use gray code (or phase) to calculate distance and thus, dynamic range or depth of signal modulation is critical for accurate depth calculation.

[0125] BROADBAND LINE SOURCE AND LASER LINE TRIANGULATION

[0126] As noted above, determining simultaneous spectral and depth measurement of a part comprised of one or more different materials is challenging because it is difficult to make a simultaneous measurement of profile height and spectral measurement of a target part. In some embodiments and as described more fully below, the structured illumination source comprises a line source and a physical dimension measurement is performed by measuring a lateral offset of a line projected on a target surface at an angle.

[0127] In particular, system for simultaneous depth profile and spectral measurement is disclosed. The system includes one or more illumination sources, one or more detectors, and a processor. The one or more detectors is/are configured to capture one or more frames of a target illuminated using the one or more illumination sources. The processor is configured to determine a depth measurement based at least in part on the one or more frames; determine a spectral measurement based at least in part on the one or more frames; and determine a corrected map of depth and spectral information by correcting the spectral measurement using the depth measurement.

[0128] The system enables inspection of a target both for physical dimensions (e.g., height profiles over a 2-dimensional plane) and material properties (e.g., a spectral measurement over 2- dimensional plane). The system enables the two types of inspections to be performed simultaneously. The physical dimension measurement is performed by measuring a lateral offset of a line projected on a target surface at an angle. The spectral measurement is performed by measuring the back reflected spectrum or forward reflected/scattered spectrum of the target surface illuminated using a broadband source.

[0129] FIG. 11 is a diagram illustrating an embodiment of a measurement device. In the example shown, illumination source 1100 (e.g., a broadband illumination source, a white laser source, etc.) exits a fiber and is collimated using optic 1102 along path 1124. Cylindrical lens 1104 transforms the beam to be focused in one dimension (e.g., a line). The one-dimensional beam passes through beam splitter 1106, optic 1112, and dichroic splitter 1114 to be projected onto the surface of target 1118 at an angle off of normal to the surface of target 1118. The surface of target 1118 with the line projected (e.g., line in and out of the page of FIG. 11 is translated along direction 1132 (e.g., left and right in the page of FIG. 11) so that a measurement can be taken over the entire surface of target 1118. A backscattered beam path 1126 passes through dichroic splitter 1114 and optic 1112 and is reflected using beam splitter 1106 to be measured by detector 1110 (e.g., spectrum detector, pushbroom spectrometer, etc.) after passing through optic 1108.

[0130] Illumination source 1116 (e.g., a monochromatic source, a narrowband laser source, etc.) generates a line that is projected on the surface of target 1118 (e.g., line in and out of the page of FIG. 11) along path 1128 after being reflected by dichroic splitter 1114 (e.g., dichroic beam splitter). The surface of target 1118 is imaged normal to the surface of target 1118 using detector 1122 (e.g., a 2-dimensional imaging device) using optic 1120 following beam path 130.

[0131] FIG. 12 is a diagram illustrating an embodiment of a measurement device. In the example shown, illumination source 1200 (e.g., a broadband illumination source, a white laser source, etc.) exits a fiber and is collimated using optic 1202 along path 1224. Cylindrical lens 1204 transforms the beam to be focused in one dimension (e.g., a line). The one-dimensional beam passes through beamsplitter 1206 and optic 1212 to be projected onto the surface of target 1218 at an angle off of normal to the surface of target 1218. The surface of target 1218 with the line projected (e.g., line in and out of the page of FIG. 12) is translated along direction 1232 (e.g., left and right in the page of FIG. 12) so that a measurement can be taken over the entire surface of target 1218. A backscattered beam path 1226 passes through optic 1212 and is reflected using beamsplitter 1206 to be measured by detector 1210 (e.g., spectrum detector, pushbroom spectrometer, etc.) after passing through optic 1208. [0132] The surface of target 1218 with the line projected (e.g., line in and out of the page of

FIG. 12) is translated along direction 1232 (e.g., left and right in the page of FIG. 12) so that a measurement can be taken over the entire surface of target 1218. The surface of target 1218 is imaged normal to the surface of target 1218 using detector 1222 (e.g., a 2-dimensional imaging device) using optic 1220 following beam path 1230.

[0133] FIG. 13 is a diagram illustrating an embodiment of a measurement device. In the example shown, illumination source 1300 (e.g., a broadband illumination source, a white laser source, etc.) exits a fiber and is collimated using optic 1302 along path 1324. Cylindrical lens 1304 transforms the beam to be focused in one dimension (e.g., a line). The one-dimensional beam passes through optic 1312 to be projected onto the surface of target 1318 at an angle off of normal to the surface of target 1318. The surface of target 1318 with the line projected (e.g., line in and out of the page of FIG. 13) is translated along direction 1332 (e.g., left and right in the page of FIG.

13) so that a measurement can be taken over the entire surface of target 1318. Backscattered beam path 1326 is measured by detector 1310 (e.g., spectrum detector, pushbroom spectrometer, etc.) after passing through optic 1308.

[0134] The surface of target 1318 with the line projected (e.g., line in and out of the page of

FIG. 13) is translated along direction 1332 (e.g., left and right in the page of FIG. 13) so that a measurement can be taken over the entire surface of target 1318. The surface of target 1318 is imaged normal to the surface of target 1318 using detector 1322 (e.g., a 2-dimensional imaging device) using optic 1320 following beam path 1330.

[0135] Illumination beam path 1336 (e.g., also path 1324) is incident to target 1318 at angle

1334 (e.g., 20 degrees) from normal. Incidence imaging direction 1332 (e.g., also beam path 1330) is normal to the surface of target 1318. Backscatter path 1340 (e.g., also beam path 1326) is at angle 1338 (e.g., 1 degree) off of illumination beam path 1336.

[0136] FIG. 14 is a diagram illustrating an embodiment of a measurement device. In the example shown, illumination source 1400 (e.g., a broadband illumination source, a white laser source, etc.) exits a fiber and is collimated using optic 1402 along path 1424. Cylindrical lens 1404 transforms the beam to be focused in one dimension (e.g., a line). The one-dimensional beam passes through optic 1412 to be projected onto the surface of target 1418 at an angle off of normal to the surface of target 1418. The surface of target 1418 with the line projected (e.g., line in and out of the page of FIG. 14) is translated along direction 1432 (e.g., left and right in the page of FIG.

14) so that a measurement can be taken over the entire surface of target 1418. Forward reflected/scattered beam path 1426 is measured by detector 1410 (e.g., spectrum detector, pushbroom spectrometer, etc.) after passing through optic 1408.

[0137] The surface of target 1418 with the line projected (e.g., line in and out of the page of

FIG. 14) is translated along direction 1432 (e.g., left and right in the page of FIG. 14) so that a measurement can be taken over the entire surface of target 1418. The surface of target 1418 is imaged normal to the surface of target 1418 using detector 1422 (e.g., a 2-dimensional imaging device) using optic 1420 following beam path 1430.

[0138] Illumination beam path 1436 (e.g., also path 1424) is incident to target 1418 at angle

1434 (e.g., thetal) from normal. Incidence imaging direction 1432 (e.g., also beam path 1430) is normal to the surface of target 1418. Backscatter path 1440 (e.g., also beam path 1426) is at angle 1438 (e.g., theta2) off of illumination beam path 1436.

[0139] FIG. 15 is a diagram illustrating an embodiment of a measurement device. In the example shown, illumination source 1500 (e.g., a broadband illumination source, a white laser source, etc.) exits a fiber and is collimated using optic 1502 along path 1524. Cylindrical lens 1504 transforms the beam to be focused in one dimension (e.g., a line). The one-dimensional beam passes through optic 1512 to be projected onto the surface of target 1518 at an angle off of normal to the surface of target 1518. The surface of target 1518 with the line projected (e.g., line in and out of the page of FIG. 15) is translated along direction 1532 (e.g., left and right in the page of FIG. 15) so that a measurement can be taken over the entire surface of target 1518. Forward reflected/scattered beam path 1526 is split off beam path 1530 using diffraction grating 1521 and measured by detector 1510 (e.g., spectrum detector, pushbroom spectrometer, etc.) after passing through optic 1508.

[0140] The surface of target 1518 with the line projected (e.g., line in and out of the page of

FIG. 15) is translated along direction 1532 (e.g., left and right in the page of FIG. 15) so that a measurement can be taken over the entire surface of target 1518. The surface of target 1518 is imaged off normal to the surface of target 1518 using detector 1522 (e.g., a 2-dimensional imaging device) using optic 1520 following beam path 1530 and beam path 1531 after passing through diffraction grating 1521.

[0141] FIG. 16 is a diagram illustrating an embodiment of a measurement system. In some embodiments, measurement device 1614 in the measurement system of FIG. 16 is the measurement device of FIG. 11, FIG. 12, FIG. 13, FIG. 14, or FIG. 15. In the example shown, measurement device 1614 is used to measure a target. Controller 1602 controls measurement device 1614 using signals 1610 (e.g., laser control signals, detector control signals, etc.) for operating measurement device 1614. Detector data signals 1612 are transmitted from measurement device 1614 and controller 1602. Controller 1602 includes interface 1604 and processor 1606 for controlling measurement device 1614 to take a measurement of a target and provide data for processing. User device 1600 interacts with controller 1602 to request and provide measurement of a target.

[0142] FIG. 17 is a flow diagram illustrating an embodiment of a process for making a simultaneous depth profile and spectral measurement. In some embodiments, the process of FIG.

17 uses a measurement device (e.g., measurement device 1614 of FIG. 16) and is executed using a processor (e.g., processor 1606 of FIG. 16). In the example shown, in 1700 a relative position (in X) is set between a target and a sensor. For example, a position is set for a stage that translates the target being measured or the device measuring the target. In 1702, a measurement is taken by capturing frames. For example, one or more detectors (e.g., an imaging detector for a height measurement and/or a spectrometer for a spectral measurement) take(s) a measurement of light reflected or scattered from target. In 1704, a depth measurement (for Y) is determined. For example, a height map is determined by processing the captured frames and using a calibration that corresponds a lateral offset of a portion of a line as imaged in the captured frames with a height. In 1706, a spectral measurement (for Y) is determined. For example, a spectrum is determined by processing the captured frames and using a calibration that corresponds a backscattered light spectrum for the line as imaged in the captured frames with calibrated wavelengths. In 1708, it is determined whether there are more positions. For example, it is determined whether all the desired X positioned have been imaged for the target sample. In response to there being more positions, control passes to 1700. In response to there not being more positions, control passes to 1710. In 710, a corrected map (for XY) of spectral and depth information is determined using depth measurement and spectral measurement. For example, the spectral measurement is corrected using information from the depth measurement. Specifically, the spectral measurement is corrected for the shift caused by the depth measurement (e.g., using a triangulation measurement).

[0143] FIG. 18 is a flow diagram illustrating an embodiment of a process for determining a depth calibration. In some embodiments, the depth calibration of FIG. 18 is used to determine a depth measurement from a frame as in 1704 of FIG. 17. In the example shown, in 1800 a height calibration target is positioned. For example, a measurement is taken of a calibration standard at a position on the height calibration target. In some embodiments, multiple measurements are taken at multiple positions to determine a height calibration. In 1802, a measurement is taken by capturing frames. For example, the image of a height standard is imaged using a 2-dimensional imaging device (e.g., a camera). In 1804, a depth measurement (for Y) is determined. In 1806, depth calibration is determined using the depth measurement and calibration height. For example, the frames are analyzed to determine a height or depth and the measurement is corresponded to the standard’s calibrated height.

[0144] FIG. 19 is a flow diagram illustrating an embodiment of a process for determining a depth measurement. In some embodiments, the process of FIG. 19 is used to implement 1704 of FIG. 17. In the example shown, in 1900 frames are received. In 1902, frames are corrected using the depth calibration. In 1904, the depth measurement, the corrected frames, and correction are provided. For example, the frames are analyzed and determined heights are corrected using the depth calibration. The correction and corrected frames are provided to later correct a spectral measurement of the target.

[0145] FIG. 20 is a flow diagram illustrating an embodiment of a process for determining a spectral calibration. In some embodiments, the spectral calibration of FIG. 20 is used to determine a spectral measurement from a frame as in 1706 of FIG. 17. In the example shown, in 2000 a spectral calibration target is positioned. For example, a measurement is taken of a spectral standard at a position on the spectral calibration target. In some embodiments, multiple measurements are taken at multiple positions to determine a spectral calibration. In 2002, a measurement is taken by capturing frames. For example, the image of a spectral standard is imaged using a spectral measurement device (e.g., a pushbroom spectrometer). In 2004, a spectral measurement (for Y) is determined. In 2006, a spectral calibration is determined using the spectral measurement and spectral calibration. For example, the frames are analyzed to determine a spectrum and the measurement is corresponded to the spectral standard’s calibrated spectrum.

[0146] FIG. 21 is a flow diagram illustrating an embodiment of a process for determining a spectral measurement. In some embodiments, the process of FIG. 21 is used to implement 1706 of FIG. 17. In the example shown, in 2100 frames are received. In 2102, frames are corrected using the spectral calibration. In 2104, the spectral measurement and the corrected frames are provided. For example, the frames are analyzed and spectra are corrected using the spectral calibration.

[0147] FIG. 22 is a diagram illustrating an embodiment of data from a device measuring height and a spectrum of a target simultaneously. In some embodiments, the target frame data of FIG. 22 comprises target 1118, target 1218, target 1318, target 1418, or target 1518. In the example shown, a line from an illumination source is projected onto a target shown in 2200 with a profile as shown. The illumination source is projected at an angle so that a surface that is at a different height is shifted. In 2201, an illumination line incident at an angle 9 to a feature of height z appears shifted to the right. The shift amount of the beam (x) with respect to the height of the part (z) is given by: x = z * tan(d).

[0148] The frame image is then converted to heights by using the shift to determine a height map in 2202. A spectral response of the line is shown in 2204 also with the lateral shifts for each of the spectral responses (e.g., shown as the lines in different colors). The corrected spectral response of the lines is shown in 2206 as shifted using the height information from 2202 and 2200 so that a height mapping and spectral mapping can be generated and overlaid.

[0149] FIG. 23 is a diagram illustrating an embodiment of data from a device measuring height and a spectrum of a target simultaneously. In some embodiments, the target frame data of FIG. 23 comprises data related to target 1118, target 1218, target 1318, target 1418, or target 1518. In the example shown, image frame 2301 shows an o-ring in a depression. The line cut is indicated by 2303 through image frame 2301 is shown in height profile 2300. Height profile 2300 is a one dimensional cut along the X-dimension and shows a step and the surface cut of the o-ring. 2302 indicates the depression and the corresponding spectrum line as well as the center of the step 2304 and its corresponding spectrum line in the spectrum versus position graph 2306. The spectrum related to 2302 is shown in a line graph of 2312 as indicated by 2308, and the spectrum related to 2304 is shown in a line graph of 2312 as indicated by 2310. Note that the spectra are different implying that the material of the o-ring is different from the material at the center of the step.

[0150] INTERFEROMETRICALLY CONSTUCTED STRUCTURED LIGHT

[0151] In some embodiments, the structured light source comprises a multi-spectral fringe projection system. The fringes are generated interferometrically, creating different fringe frequencies per illumination wavelength. Imaging is done through multiple phase shifts with a multi-wavelength imaging spectrometer.

[0152] In some embodiments, fringes are generated using an interferometer with a tilted mirror. Fringes are imaged onto an object, target of interest, or part under test. At an angle, Q, a multi-wavelength detector/imaging spectrometer views the object. By combining phase-shifted images, the system is capable of measuring both spectral content of the object as well as the depth map. [0153] FIG. 24 is a diagram illustrating an embodiment of a twyman green structured light system using a broadband source. In the example shown, the system 2400 comprises a broadband source 2410 (e.g. a xenon lamp), an interferometer 2420 comprising a mirror on PZT 2422 and a tilted mirror 2424, a detector 2430, and lenses (e.g., shown at 2426 and 2436 respectively). As shown in FIG. 24, broadband source 2410 is incident on a beamsplitter 2415. The signal is divided into two beams using an interferometer 2420, such as a twyman green as is shown in FIG. 24.

Other interferometer configurations, such as a Mach-Zender, can also be implemented. Flat mirrors are in both arms of the interferometer 2420, but in one arm, one of the mirrors is tilted (e.g., tilted mirror 2424) to create tilt fringes. The other mirror (e.g., mirror on PZT 2422) is attached to a PZT for phase-shifting. Particularly for non-laser sources, it is important that the paths are matched, a.k.a. the distance from the beamsplitter face to the mirror in each arm of the interferometer (e.g., shown at 2401 and 2402) must be equal for highest contract interference fringes. In this case, the maximum tilt angle of the mirror where high contrast fringes will occur will be limited by the coherence properties of the source.

[0154] Since the wavelengths sampled at the detector will likely be few and narrow band, using a broadband source as shown in FIG. 24 may be somewhat inefficient and could result in poor height resolution due to limitations in the generated fringe frequencies.

[0155] Accordingly, in some embodiments, in order to be more light efficient, instead of a broadband source, a series of laser diode sources coupled using a grating (as shown in FIG. 25) or a combination of fiber-coupled LEDs (not shown) are used.

[0156] FIG. 25 is a diagram illustrating an embodiment of a twyman green structured light system 2500 having components as shown in FIG. 24 but using a series of laser diode sources 2510 in place of the broadband source 2410 as a structured light source. In the example shown, the system 2500 comprises a series of laser diode sources 2510, an interferometer 2520 comprising a mirror on PZT 2522 and a tilted mirror 2524, a detector 2530 and lenses (e.g., shown at 2526 and 2536 respectively). As shown in FIG. 25, the series of laser diode sources 2510 are incident on a beamsplitter 2515. The signal is divided into two beams using an interferometer 2520, such as a twyman green as is shown in FIG. 25. Other interferometer configurations, such as a Mach- Zender, can also be implemented. Flat mirrors are in both arms of the interferometer 2520, but in one arm, one of the mirrors is tilted (e.g., tilted mirror 2524) to create tilt fringes. The other mirror (e.g., mirror on PZT 2522) is attached to a PZT for phase-shifting. Particularly for non-laser sources, it is important that the paths are matched, a.k.a. the distance from the beamsplitter face to the mirror in each arm of the interferometer (e.g., shown at 2501 and 2502) must be equal for highest contract interference fringes. Also important to note that the maximum tilt angle of the mirror where high contrast fringes will occur will be limited by the coherence properties of the source.

[0157] The generated interference pattern is imaged using a lens (e.g., shown at 2426 of

FIG. 24 and at 2526 of FIG. 25) onto an object or target of interest, which in this case is a sample under test (e.g., shown at 2440 of FIG. 24 and at 2540 of FIG. 25). Any height deviation on the sample will cause a local shift in the fringe pattern that can be detected by the imaging system. The imaging system consists of a lens (e.g., shown at 2436 of FIG. 24 and at 2536 of FIG. 25) that images the part under test onto a multi-wavelength 2D detector (e.g., shown at 2430 of FIG. 24 and at 2530 of FIG. 25). This detector could take many forms (e.g., a snapshot mosaic sensor). In this case, the part can be directly imaged onto the detector as shown in FIGS. 24 and 25. Another alternative is to use an imaging spectrometer such as an IRIS imaging spectrometer, a multiple- image Fourier transform spectrometer, or a compact birefringent Fourier transform spectrometer. For all these types of imaging spectrometers, the result is a 2D image for a given sampled wavelength band (also referred to as a channel). A brief discussion of how to recover the spectral map and depth map are separately discussed below.

[0158] SPECTRUM RECOVERY

[0159] To recover the spectral information for a given scene, a measurement must be acquired that (A) has no interference pattern or (B) the cumulative effect of multiple measurements yields an equivalent measurement with no interference pattern. Eliminating interference fringes (A) could be achieved by blocking one arm of the interferometer, however that would eliminate half the illumination light. An alternative is (B) where the cumulative effect of multiple measurements nets an image with no fringes. The most straightforward way to accomplish this is to acquire two images where the phase difference between the two measurements is out of phase. The mean background irradiance can be found by calculating the mean of the two images. An example of this is shown in FIG. 26. Note that no interference fringes are visible in the mean image. An out of phase measurement must be taken for each wavelength band if this approach is employed.

[0160] FIG. 26 shows a first projected pattern 2610, a second projected pattern 2620 that is out of phase with respect to first projected pattern 2610, and the resultant image 2630 if the two images (e.g., shown at 2610 and 2620 respectively) are averaged. No fringes are visible in the result. [0161] In some embodiments, many images are acquired with small phase shifts such that the net sum of all images appears to have no residual interference pattern.

[0162] In some embodiments, a LSPSI (least squares phase shifting interferometry) approach is used that naturally calculates the mean intensity during the phase estimation process. The result of the mean image is a map of the reflectance as a function of wavelength for the part under test. If the part is comprised of multiple materials, for example a metal part with rubber o- rings, the spectrum can be used to identify the different materials present in the measurement. As discussed herein, a spectra library may be used to store reference signatures is associated with different materials to be matched against a spectrum generated by the above-disclosed techniques.

[0163] DEPTH MAP RECOVERY

[0164] Because the source for this system covers a potentially wide range of wavelengths, the projected fringe pattern for each wavelength band has a different frequency. Like all interferometric systems, to recover the measured phase, at least three phase shifted measurements are required. The number of phase shifts and phase shift distance must be sufficient to recover the phase for all wavelengths in the system.

[0165] MULTI-WAVELENGTH INTERFEROMETRY ADVANTAGE

[0166] Because this interferometrically constructed structured light system generates a separate wrapped phase map for each wavelength in the system, the result phase maps can be combined to generate synthetic wavelengths. This is commonly done in multi-wavelength interferometry in order to measure surfaces where the phase difference due to a large feature exceeds l/2. In this case where fringes are projected onto the object, the phase difference cannot

[0167] The general concept is that the wrapped phases generated by each individual wavelength can be combined to form a synthetic wavelength with a corresponding synthetic wrapped phase, calculated by simply substracting the wrapped phases of the two original wavelengths, Df_ea = Af_z— Df₁. The equivalent wavelength for such a system is X_ep

It can be shown in simulation that this works very well.

[0168] FIG. 27 depicts a synthetic wavelength example using 0.8 and 0.9um fringes to generate a source of effective 7.2um fringes. In the example shown, a simulation was performed using 5 wavelengths (0.5-0.9um in O.lum increments) measuring a small bump of height 0.0796um. An input OPD map is shown at 2710. The simulated fringe pattern for wavelengths of 0.8um and 0.9um are shown at 2720 with their corresponding wrapped phases. For comparison, the fringe pattern and wrapped phase for a 7.2um source is also calculated and compared to the synthetic result shown at 2730. This example also highlights that the wrapped phase of the 7.2um result requires no phase unwrapping, only tilt removal to detect the height of the object. By using the large synthetic wavelengths to guide the phase unwrapping of the small wavelengths, a system of this kind can retain good depth resolution while also having large dynamic range (be able to measure objects » l_RGo]«Lkΐ/2).

[0169] To fiirther illustrate the utility, FIG. 28 depicts equivalent synthetic wavelengths for all combinations of wavelength from 0.5-0.9um.

[0170] Typically with structured illumination techniques performed using projectors, the fringe patterns are phase shifted in many different frequencies. This is to reduce the complexity of the unwrapping problem and be able to measure greater height differences. The synthetic wavelength approach used here achieves the same end goal.

[0171] A challenge in multi-l interferometry is that the accuracy of the measurement is highly dependent on the wavelength stability of the source. Most systems of this nature require expensive and highly stable laser sources. In this method, there is an opportunity to filter the wavelength at the detector and restrict the measured bandwidth, thus enforcing the wavelength stability of the system.

[0172] In particular, because many different fringe patterns are generated at different fringe frequencies, a synthetic wavelength can be generated by combining wavelengths of different frequencies. This large synthetic wavelength can be used to guide the phase unwrapping of the small wavelengths, and a system of this kind can retain excellent depth resolution while also having large dynamic range, a.k.a. be able to measure objects » 7pro_|ccied/2.

[0173] As described herein, the disclosed techniques allow for simultaneous measurement of height/depth measurement and spectral response. Additionally, a design of this type allows for the ability to confirm a part was built correctly, with both the proper mechanical dimensions as well as confirm or identify that the proper materials were utilized during the manufacturing process.

[0174] This design is advantageous because there is no scanning of the part under test required (as opposed to laser line triangulation systems that require the laser line to scan the part to build up a depth map over time). [0175] In this technique, the fringes that are generated are caused by interference. This results in a larger depth of focus than systems that project the image of fringes onto the sample. Additionally, this technique can use a broadband source, or multiple individual sources for versatility.

[0176] STRUCTURED LIGHT USING GRATINGS

[0177] In some embodiments, the structured light source comprises a multi-spectral fringe projection system, where the fringes are generated by interfering the +1 and -1 orders of a polarization grating which are orthogonally circularly polarized, and imaging is performed with a multi-wavelength imaging spectrometer through multiple phase shifts achieved by the rotation of a polarizer.

[0178] FIG. 29 depicts a diagram of a structured light system 2900 using laser sources and a polarization grating. In the example shown, laser sources 2910 are coupled using a grating 2920 (e.g. to coalign different wavelengths shown at 2901, 2902 and 2903) and are used and after the illumination light passes through a polarizer 2922 to illuminate a polarization grating 2924, it is imaged onto an object, target of interest, or part under test (e.g., shown at 2930). At an angle, 0 (e.g., shown at 2925), a multi-wavelength camera 2940 views the object 2930 through a polarizer 2942 that can be rotated to achieve different phase shifts of the fringe pattern.

[0179] FIG. 30 depicts a diagram of a structured light system 3000 using fiber coupled

LEDs and polarization grating. In the example shown, a multi-l fiber-coupled LED 3010 is used as the structured light source and the illumination light goes through a polarizer 3022 to become linearly polarized. The illumination light then goes through a polarization grating 3024 which is imaged onto an object, target of interest, or part under test (e.g., shown at 3030). At an angle, 0 (e.g., shown at 3025), a multi-wavelength camera 3040 views the object 3030 through a polarizer 3042 that can be rotated to achieve different phase shifts of the fringe pattern.

[0180] The source of the system can be for example lasers coupled with a grating (FIG. 29) or fiber-coupled LEDs (FIG. 30). In some embodiments, interference fringes are generated using a polarization grating which sends orthogonally circularly polarized light into the +1 and -1 orders. Virtual fringes are imaged onto part under test (they are virtual because the two beams are orthogonally polarized). At an angle, 0, a multi-wavelength detector/imaging spectrometer views the part under test. A polarizer sits in front of the camera and is rotated to achieve phase shifts of the fringe pattern. By combining phase shifted images, the system is capable of measuring both spectral content of the part under test as well as the depth map.

[0181] The multi-wavelength 2D detector can take many forms. One example is a snapshot mosaic sensor. In that case, the part could be directly imaged onto the detector as shown in FIG. 29 and FIG 30. In some embodiments, an imaging spectrometer such as an IRIS imaging spectrometer, a multiple-image Fourier transform spectrometer, or a compact birefringent Fourier transform spectrometer can be used.

[0182] DEPTH MAP RECOVERY:

[0183] The projected fringe frequency for all wavelengths in this system will be the same since a grating is used to generate the interference pattern. To recover the measured phase, at least three phase-shifted images must be acquired with varying the rotation angle of the imaging polarizer. A common choice is acquiring 4 images, rotating the polarizer by an angle of 45° between measurements such that the phase shift corresponds to a shift of l/4, a quarter of a period. In that case, the phase can be recovered from the equation: tan(</>) = Ii is a single

camera frame at a single wavelength band. The maximum step that can be measured using a system of this type will be limited to /prqjcctcd/2.

[0184] The disclosed techniques allow for simultaneous measurement of height/depth measurement and spectral response. Additionally, a design of this type allows for the ability to confirm a part was built correctly, with both the proper mechanical dimensions as well as confirm or identify that the proper materials were utilized during the manufacturing process.

[0185] This design is advantageous because there is no scanning of the part under test required (as opposed to laser line triangulation systems that require the laser line to scan the part to build up a depth map over time). In this case, the fringes that are generated are caused by interference. This results in a larger depth of focus than systems that project the image of fringes onto the sample. This design can use a broadband source, or multiple individual sources.

[0186] INDUSTRIAL APPLICATIONS

[0187] FIG. 31 shows a table 3100 listing examples of various industrial applications for applying 3D spectral scanning as disclosed herein. As shown in FIG. 31 , there are many examples of 3D spectral scanning applications that span various fields and major industries including automotive design and manufacturing, material processing, and plant inspection applications. In particular, because of high unit volumes, the automotive industry has fully adopted automation and robotics in their manufacturing lines. Thus, automated optical inspection is a crucial application for automotive and other industries such as pharma and food. The added value of spectral imaging is the availability of material composition information. The need is driven by reducing the cost of correcting errors in manufacturing. Finding errors early saves money.

[0188] One area for the application of 3D spectral scanning is for material processing applications such as additive manufacturing and removal processing. For example, material processing steps may be controlled or initiated based on or in response to the 3D shape and material composition data output by a 3D spectral scanning system as described herein. The 3D spectral scanning system can also be a component of a material processing platform designed or configured to achieve various objectives (e.g., material inspection, detection and diagnosis, automated removal processing based on or in response to 3D point cloud representations segmented by material). In some cases, inspection using the 3D spectral scanning techniques as described herein can be part of a feedback loop to inform a material processing system in making decisions for next steps and actions.

[0189] In some embodiments, a material processing method comprises receiving 3D shape and material composition data output by scanning an object using the 3D spectral scanning techniques as described herein and taking material processing steps in response to the received output. In some cases, the material processing steps include

[0190] FIG. 32 is a block diagram of an embodiment of a material processing system or platform 3200 that uses 3D spectral scanning to execute material processing steps. As shown in FIG. 32, the material processing platform 3200 comprises a 3D spectral scanning system (e.g., 3D Spectral Scanner 100) for simultaneous depth profile and spectral measurement as shown in FIG. 1 and a Material Processing Unit 3201. The various components of the 3D spectral scanning system are described herein with respect to FIG. 1.

[0191] Material Processing Unit 3201 is configured to execute various material processing steps in response to the output (e.g. Output 160) received from 3D Spectral Scanner 100. In the example shown, Material Processing Unit 3201 comprises various modules including Diagnostic Module 3210, Decision Module 3220, and Control Module 3210.

[0192] In some embodiments, Diagnostic Module 3210 performs detection and diagnosis to identify issues associated with the object based on the 3D point cloud segmented by material. For example, based on a 3D point cloud representation of the object segmented by material, Diagnostic Module 3210 detects abnormal or unexpected areas on an object (e.g., contamination, deterioration, leakage, deformities in shape etc.) based on the 3D depth profile and material composition provided by the output of 3D Spectral Scanner 100. In some cases, a database or library (e.g., Data 3215) is used to store and retrieve reference data for detection and diagnostic purposes. For example, the database can be used to store data associated with common contaminants, or data associated with the shape of common manufacturing defects.

[0193] Decision Module 3220 is configured to generate a decision in response to the detection and diagnosis performed by Diagnostic Module 3210. In some embodiments, Decision Module 3220 generates a decision using machine learning techniques. Control Module 3230 outputs a control action in response to the decision generated by Decision Module 3220. In some embodiments, Control Module 3230 actuates or controls one or more devices (e.g., Devices 3240) in response to the control action to address the issue related to the object. In some cases, the one or more devices (e.g., Devices 3240) perform the control action received from Control Module 3230 to address the issue related to the object.

[0194] For example, in the case where the Diagnostic Module 3210 detects and diagnoses contamination on the object, Decision Module 3220 outputs a decision to apply a cleaning process to the contaminated area based on the 3D point cloud showing the shape of the contaminated area and providing the composition of the contaminated material. Control Module 2330 addresses the issue by instructing a device to spray the contaminated area with a cleaning element suitable to address the contamination.

[0195] In another example, the Diagnostic Module 3210 detects and diagnosis an incorrect part on an object or a manufacturing defect on the object. Decision Module 3220 outputs a decision to remove and replace the incorrect part, or to reject the object due to the manufacturing defect. Control Module addresses the issue by instructing a device (e.g., a robotic arm with end effector) to remove and replace the incorrect part, or by instructing a device to dispose of the object having the manufacturing defect.

[0196] In both cases, the added value is early detection and diagnosis of issues to enable fixes or preventative measures to address the issues associated with the object that have been detected and diagnosed. [0197] FIG. 33 is a flowchart of an embodiment of a method for material processing using

3D spectral scanning that can be performed by the exemplary system depicted in FIG. 32. As shown in FIG. 33, at 3310 the method includes performing 3D spectral scanning on an object. The 3D spectral scanning may be performed according to any of the disclosed techniques as described herein.

[0198] At 3320, the method includes performing detection and diagnosis to identify issues associated with the object based on the 3D spectral scanning. For example, detection and diagnosis is based on the 3D point cloud segmented by material provided by the 3D spectral scanning system.

[0199] At 3330, the method includes generating a decision in response to the detection and diagnosis. For example, the decision can include a recommended course of action to address an issue associated with the object.

[0200] At 3340, the method includes outputting a control action in response to the decision.

As an example, based on a recommended course of action, one or more instructions are generated in order to produce an action.

[0201] At 3350, the method includes actuating or controlling a device in response to the control action to address an issue associated with the object. For example, the one or more instructions can be used to generate an action taken by a device or system configured to address a particular issue.

[0202] At 3360, the method includes repeating the process starting at the 3D spectral scanning step. For example, after an issue associated with an object has been detected, diagnosed, and a recommended action has been executed, the system performs 3D spectral scanning on the same or a subsequent object and repeats the process to identify and address other issues.

[0203] Thus, the material processing system or platform is configured to obtain both the 3D shape and the material of an object (such as a part being manufactured) using the 3D spectral scanning techniques disclosed herein, detect and diagnose issues associated with the object (e.g., recognize a contaminated, deteriorated, degraded, or otherwise substandard or abnormal portion of the object, provide a recommended action to address the issues, and execute the action (e.g., clean up contaminated portions or overlay or replace deteriorated or degraded portions while leaving normal portions intact). [0204] Another area for the application of 3D spectral scanning is for automotive inspection applications. For example, automated inspection of module assemblies in a production line verifies an assembly for the correct installation of components and confirms the correct materials. Such an inspection is difficult to do automatically with only 3D scanning without spectral segmentation to identify and distinguish different components by their material composition. In current systems, material confirmation must be done manually by a human. FIG. 34 illustrates an application of 3D spectral scanning to provide 3D segmentation by material for automated optical inspection.

[0205] Another example is automated inspection of auto interior upholstering and fitting.

Materials of similar color are often used in auto interiors. Inspection for errors in a particular configuration are checked by a human inspector. This inspection could be done automatically with a 3D spectral scanning system.

[0206] Another area the application of 3D spectral scanning is for food quality analysis and control. For natural as well as processed foods, quality and safety analysis are required by federal institutions in most countries. 3D spectral scanning can provide non-destructive analysis and automation to identify many quality items such as: ripeness and quality (bruising and texture) of fruits and vegetables, contaminants, bacterial content, and process control in food processing plants.

[0207] 3D spectral scanning can also be of use in the pharmaceutical industry.

Pharmaceutical drugs and vitamins are commonly configured in the form of a solid pill. The pills have standard shapes. The shape and composition of pills can be readily verified with 3D spectral scanning. The applications in pharma are in manufacturing: pill conveyor belt sorting, quality control, defects, contamination, and dose. Additional applications are possible when drugs are dispensed to patients. Detection of the correct medication and security against counterfeit or contaminated sources can be done at the point of distribution or dispensing.

[0208] In the medical industry, there are also possible applications in which 3D spectral scanning can prove advantageous. For instance, in certain surgeries, too often a surgical instrument such as a clamp, hemostat or sponge is inadvertently left in in body after the incision is closed. In trauma surgeries, a goal is often the removal of foreign objects such as dirt, glass shards, shrapnel or bullets, and it can be difficult to ensure that 100% of the foreign objects have been located and removed. In some surgeries, to remove therapeutic devices that were previously intentionally installed (screws, wires, stents, etc.), it can be difficult to locate 100% of the devices. These problems can lead to poor patient outcomes such as infection or the need for additional surgeries. [0209] In these surgical scenarios, the 3D spectral scanning techniques as disclosed herein can be used to acquire or capture images inside a surgical site. Such images can be acquired or taken once before closing the incision, during or throughout the course of a special scanning procedure wherein a surgeon or a robot methodically probes tissue to expose more area to the imager, or periodically or continuously during the surgery (e.g., attached to a surgical robot, an endoscope, a dedicated scanning robot arm, a surgery light fixture or simply as a handheld imager).

[0210] Based on or using the images acquired from a surgical site, the 3D spectral scanning techniques as disclosed herein can be used to outline and classify all objects or targets of interest in the images by material and depth profile or 3D shape. For instance, a 3D spectral scanning system can be used to identify all shapes that are made of foreign substances not naturally present in the body (e.g., using spectral segmentation and material identification), and can fiirther identify foreign shapes that are not naturally present in the body. For surgical instruments, a library of CAD shapes of various different instruments (e.g., clamps, hemostats, surgical screws, etc.) is used to look for partial matches, such as the visible end of a clamp nestled in tissue (which is a subset of the stored CAD rendering of the clamp). For traumatic foreign objects, AI or similar techniques can be used to classify 3D shapes for example as“naturally present in the body” or“not naturally present.” Glass fragments, bullets and fragments and dirt all have shape characteristics that are readily differentiated from the shapes of components of the human body. A display system displays and highlights all foreign objects and suspected foreign objects, color coded and/or labeled as to type and shown against the color-coded surrounding tissue to make them easy to be located. After attempted retrieval of the foreign object by the surgeon or surgical robot, the entire sequence can be repeated until a clean scan is obtained.

[0211] In some embodiments, a 3D spectral scanning system as disclosed herein can be used by a surgical robot to methodically remove all foreign objects meeting certain criteria during a surgical procedure by performing a sequence of steps including (1) probe and image objects in a scene (e.g., the surgical site); (2) classify or identify objects by depth profile and material composition; (3) remove identified objects; and (4) repeat steps (l)-(3) to probe, image, classify, identify and remove additional objects throughout the course of the surgical procedure. Examples of applications of this method are automated cleaning of a wound site or removal of shrapnel or glass fragments. In some cases, the 3D spectral scanning system can be operated in a video mode, with the surgeon (or robot) using the video feed (which may be displayed on goggles or a screen for a surgeon) to“navigate” in real time, probing a surgical site and removing each tagged object as it becomes visible. [0212] In some embodiments, the 3D spectral scanning system can be used to distinguish normal from abnormal tissues (such as certain tumors, growths, gall stones, kidney stones, etc.)· The system uses a combination of shape (morphological) and material information to classify the tissue. In some embodiments, the material differences between tissues are enhanced by the introduction into the body of dyes or other materials that are differentially absorbed or retained by normal vs. abnormal tissue.

[0213] FIG. 35 shows the difference between traditional 3D scanning and 3D spectral scanning as disclosed herein. By adding material capability to 3D metrology and scanning, the disclosed method can automatically differentiate components and add material identity. This improves the information content and value of the system by saving time, enhancing productivity, and improving quality.

[0214] As discussed herein, existing systems lack the ability to represent an object segmented into its constituent parts, wherein the representation shows the 3D shape of each constituent part based on its material composition. By providing simultaneous depth profile and spectral measurement capabilities, the disclosed integrated 3D spectral scanning techniques provide a superior and enhanced tool for identifying and classifying objects. Such a platform can be migrated across business sectors such as industrial metrology, quality control, borders and security, agriculture, pharmaceuticals, and the food industry. In this manner, an integrated 3D spectral scanning instrument platform that can discriminate material composition and 3D object shape in industrial machine vision applications would offer higher value and greater functionality over existing systems.

[0215] Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and not restrictive.

Claims

1. A 3D spectral scanning system configured to provide simultaneous depth profile and spectral measurement, comprising:

a structured light source;

a spectral imaging device configured to acquire or capture images of an object illuminated by the structured light source at different frequencies such that a 3D point cloud or 3D geometric mesh of the object can be extracted from the images; and

a processor and a memory coupled to the processor configured to provide the processor with instructions that when executed cause the processor to:

perform spectral segmentation using the images to identify material differences in the object;

perform 3D reconstruction using the images to identify a depth profile based on the object; and

output a computer representation of the object showing spatial dimensions and material differences in response to the spectral segmentation and the 3D reconstruction.

2. The system of claim 1 , wherein performing 3D reconstruction comprises extracting a 3D point cloud or 3D geometric mesh of the object from the images.

3. The system of claim 1, wherein performing 3D reconstruction comprises extracting a 3D point cloud or 3D geometric mesh of the object from the images based on the spectral segmentation and identified material differences.

4. The system of claim 1, wherein the images of the object comprise a sequence of multispectral or hyperspectral image frames generated by projecting a sequence of structured light patterns from the structured light source on a scene comprising the object, wherein the sequence of structured light patterns is configured to provide 3D data of the scene.

5. The system of claim 1, wherein the images of the object comprise a sequence of multispectral or hyperspectral image frames generated by projecting a sequence of structured light patterns from the structured light source on a scene comprising the object, wherein the sequence of structured light patterns is configured to provide 3D data of the scene, and wherein performing spectral segmentation using the images to identify material differences in the object comprises: storing each of the frames in the sequence of multispectral or hyperspectral image frames as a hypercube that captures spectral data and xy-pixel data; and spectrally segmenting each hypercube to generate one or more abundance maps for each of the frames in the sequence of multispectral or hyperspectral image frames, wherein each abundance map is generated in response to a spectrally distinct signature or endmember.

6. The system of claim 1, wherein the images of the object comprise a sequence of multispectral or hyperspectral image frames generated by projecting a sequence of structured light patterns from the structured light source on a scene comprising the object, wherein the sequence of structured light patterns is configured to provide 3D data of the scene, and wherein performing spectral segmentation using the images to identify material differences in the object comprises: storing each of the frames in the sequence of multispectral or hyperspectral image frames as a hypercube that captures spectral data and xy-pixel data;

extracting one or more spectrally distinct signatures or endmembers based on the spectral data from each of the hypercubes; and

spectrally segmenting each hypercube to generate one or more abundance maps for each of the frames in the sequence of multispectral or hyperspectral image frames, wherein each abundance map is generated in response to an extracted spectrally distinct signature or endmember.

7. The system of claim 1, wherein the images of the object comprise a sequence of multispectral or hyperspectral image frames generated by projecting a sequence of structured light patterns from the structured light source on a scene comprising the object, wherein the sequence of structured light patterns is configured to provide 3D data of the scene, and wherein performing spectral segmentation using the images to identify material differences in the object comprises: storing each of the frames in the sequence of multispectral or hyperspectral image frames as a hypercube that captures spectral data and xy-pixel data;

extracting one or more spectrally distinct signatures or endmembers based on the spectral data from each of the hypercubes;

spectrally segmenting each hypercube to generate one or more abundance maps for each of the frames in the sequence of multispectral or hyperspectral image frames, wherein each abundance map is generated in response to a spectrally distinct signature or endmember; and

generating a full scan data cube based on the abundance maps that captures spectrally distinct signature or endmember data and xy-pixel data for each of the frames in the sequence of multispectral or hyperspectral image frames.

8. The system of claim 1, wherein the images of the object comprise a sequence of multispectral or hyperspectral image frames generated by projecting a sequence of structured light patterns from the structured light source on a scene comprising the object, wherein the sequence of structured light patterns is configured to provide 3D data of the scene, and wherein performing spectral segmentation using the images to identify material differences in the object comprises: storing each of the frames in the sequence of multispectral or hyperspectral image frames as a hypercube that captures spectral data and xy-pixel data;

extracting one or more spectrally distinct signatures or endmembers based on the spectral data from each of the hypercubes;

matching one or more of the spectrally distinct signatures or endmembers to a particular reference signature, wherein the particular reference signature identifies or is associated with a particular material;

spectrally segmenting each hypercube to generate one or more abundance maps for each of the frames in the sequence of multispectral or hyperspectral image frames, wherein each abundance map is generated in response to a spectrally distinct signature or endmember; and

generating a full scan data cube based on the abundance maps that captures spectrally distinct signature or endmember data and xy-pixel data for each of the frames in the sequence of multispectral or hyperspectral image frames.

9. The system of claim 1, wherein the images of the object comprise a sequence of multispectral or hyperspectral image frames generated by projecting a sequence of structured light patterns from the structured light source on a scene comprising the object, wherein the sequence of structured light patterns is configured to provide 3D data of the scene, and wherein performing spectral segmentation using the images to identify material differences in the object comprises: storing each of the frames in the sequence of multispectral or hyperspectral image frames as a hypercube that captures spectral data and xy-pixel data;

extracting one or more spectrally distinct signatures or endmembers based on the spectral data from each of the hypercubes;

matching one or more of the spectrally distinct signatures or endmembers to a particular reference signature, wherein the particular reference signature identifies or is associated with a particular material; and

spectrally segmenting each hypercube to generate one or more abundance maps for each of the frames in the sequence of multispectral or hyperspectral image frames, wherein each abundance map is generated in response to a spectrally distinct signature or endmember associated with a material identified based on matching the one or more spectrally distinct signatures or endmembers to a particular reference signature identifying or associated with the material; and generating a full scan data cube based on the abundance maps that captures spectrally distinct signature or endmember data and xy-pixel data for each of the frames in the sequence of multispectral or hyperspectral image frames.

10. The system of claim 9, wherein each of the reference signatures is associated with a different material and wherein the reference signatures associated with different materials are stored in a spectra library.

11. The system of claim 9, wherein performing 3D reconstruction comprises extracting a 3D point cloud or 3D geometric mesh of the object using a sequence of abundance maps associated with an identified material based on at least one of the sequences of abundance maps generated using the sequence of multispectral or hyperspectral image frames.

12. The system of claim 1, wherein the images of the object comprise a sequence of multispectral or hyperspectral image frames generated by projecting a sequence of structured light patterns from the structured light source on a scene comprising the object, wherein the sequence of structured light patterns is configured to provide 3D data of the scene, and wherein performing spectral segmentation using the images to identify material differences in the object comprises: storing each of the frames in the sequence of multispectral or hyperspectral image frames as a hypercube that captures spectral data and xy-pixel data;

matching a target spectral signature to the spectral data in the hypercube; and spectrally segmenting each hypercube to generate one or more abundance maps for each of the frames in the sequence of multispectral or hyperspectral image frames, wherein each abundance map is generated in response to identified xy-pixel data based on matching the target spectral signature to the spectral data in the hypercube; and

generating a full scan data cube based on the abundance maps that captures spectrally distinct signature or endmember data and xy-pixel data for each of the frames in the sequence of multispectral or hyperspectral image frames.

13. The system of claim 1, wherein the images of the object comprise a sequence of multispectral or hyperspectral image frames generated by projecting a sequence of structured light patterns from the structured light source on a scene comprising the object, wherein the sequence of structured light patterns is configured to provide 3D data of the scene, and wherein performing spectral segmentation using the images to identify material differences in the object comprises: storing each of the frames in the sequence of multispectral or hyperspectral image frames as a hypercube that captures spectral data and xy-pixel data; spectrally segmenting each hypercube using a segmentation algorithm to generate one or more abundance maps for each of the frames in the sequence of multispectral or hyperspectral image frames, wherein each abundance map is generated in response to a spectrally distinct signature or endmember.

14. The system of claim 13, wherein one or more abundance maps for each of the frames in the sequence of multispectral or hyperspectral image frames is identified or associated with a particular material.

15. The system of claim 13, wherein one or more abundance maps for each of the frames in the sequence of multispectral or hyperspectral image frames is identified or associated with a particular material and wherein performing 3D reconstruction comprises extracting a 3D point cloud or 3D geometric mesh of the object using a sequence of the one or more abundance maps associated with the particular material.

16. A method for integrated 3D spectral scanning comprising:

illuminating an object with a structured light source;

capturing images of the object illuminated by the structured light source at different frequencies such that a 3D point cloud or 3D geometric mesh of the object can be extracted from the images;

performing spectral segmentation using the images to identify material differences in the object;

performing 3D reconstruction using the images to identify a depth profile based on the object; and

outputting a computer representation of the object showing spatial dimensions and material differences in response to the spectral segmentation and the 3D reconstruction.

17. The method of claim 16, wherein performing 3D reconstruction comprises extracting a 3D point cloud or 3D geometric mesh of the object from the images.

18. The method of claim 16, wherein performing 3D reconstruction comprises extracting a 3D point cloud or 3D geometric mesh of the object from the images based on the spectral segmentation and identified material differences.

19. The method of claim 16, wherein the images of the object comprise a sequence of multispectral or hyperspectral image frames generated by projecting a sequence of structured light patterns from the structured light source on a scene comprising the object, wherein the sequence of structured light patterns is configured to provide 3D data of the scene, wherein performing spectral segmentation using the images to identify material differences in the object comprises storing each of the frames in the sequence of multispectral or hyperspectral image frames as a hypercube that captures spectral data and xy-pixel data and spectrally segmenting each hypercube to generate one or more abundance maps for each of the frames in the sequence of multispectral or hyperspectral image frames, wherein each abundance map is generated in response to a spectrally distinct signature or endmember, and wherein performing 3D reconstruction comprises extracting a 3D point cloud or 3D geometric mesh of the object from the images based on the spectral segmentation and identified material differences.

20. A computer program product for integrated 3D spectral scanning, the computer program product being embodied in a tangible non-transitory computer readable storage medium and comprising computer instructions for:

receiving multispectral or hyperspectral images of an object illuminated by a structured light source at different frequencies such that a 3D point cloud or 3D geometric mesh of the object can be extracted from the images;

performing spectral segmentation using the images to identify material differences in the object;

performing 3D reconstruction using the images to identify a depth profile based on the object; and

outputting a computer representation of the object showing spatial dimensions and material differences in response to the spectral segmentation and the 3D reconstruction.

21. A system for measuring a target, comprising:

one or more illumination sources;

one or more detectors configured to capture one or more frames of a target illuminated using the one or more illumination sources; and

a processor configured to:

determine a depth measurement based at least in part on the one or more frames; determine a spectral measurement based at least in part on the one or more frames; and

determine a corrected map of depth and spectral information by correcting the spectral measurement using the depth measurement.

22. The system of claim 21, wherein one of the one or more illumination sources is used to make a triangulation depth measurement.

23. The system of claim 22, wherein the one of the one or more illumination sources comprises a monochromatic source.

24. The system of claim 23, wherein the monochromatic source comprises a single line laser source.

25. The system of claim 21, wherein one of the one or more illumination sources is used to make a spectral measurement.

26. The system of claim 25, wherein the one of the one or more illumination sources comprises a broadband source.

27. The system of claim 26, wherein the broadband source comprises a white light laser source.

28. The system of claim 21, wherein the depth measurement includes determining a depth from the one or more frames using a depth calibration.

29. The system of claim 28, wherein the depth calibration corresponds a lateral displacement in a frame of the one or more frames with a height.

30. The system of claim 21, wherein the spectral measurement includes determining a spectrum from the one or more frames using a spectral calibration.

31. The system of claim 30, wherein the spectral calibration corresponds spectral lines in a frame of the one or more frames with wavelengths.

32. The system of claim 21, wherein correcting the spectral measurement using the depth measurement includes shifting the spectral measurement by a lateral displacement based at least in part on the depth measurement.

33. The system of claim 21, wherein one of the one or more detectors comprises a pushbroom spectrometer.

34. The system of claim 21, wherein one of the one or more detectors comprises a 2- dimensional imaging device.

35. The system of claim 21, wherein the one or more illumination sources comprises only one broadband illumination source.

36. The system of claim 21, wherein the one or more illumination source comprises only two illumination sources including a narrow band illumination source and a broadband illumination source.

37. The system of claim 21, wherein the one or more detectors comprises only one 2- dimensional imaging device.

38. The system of claim 21, wherein the one or more detectors comprises only two detectors including one depth measurement detector and one spectral measurement detector.

39. The system of claim 21, wherein the one of the one or more illumination sources comprises a multi-spectral fringe projection system.

40. The system of claim 21, wherein the one of the one or more illumination sources comprises a multi-spectral fringe projection system using laser sources configured to generate fringes using a polarization grating.

41. The system of claim 21, wherein the one of the one or more illumination sources comprises one or more fiber coupled LEDs configured to generate fringes using a polarization grating.

42. The system of claim 21, wherein the one of the one or more illumination sources comprises a multi-spectral fringe projection system configured to generate fringes interferometrically.

43. The system of claim 21, wherein one of the one or more detectors comprises a multi wavelength imaging spectrometer.

44. A method for a target, comprising:

capturing, using one or more detectors, one or more frames of a target illuminated using one or more illumination sources;

determining a depth measurement based at least in part on the one or more frames;

determining a spectral measurement based at least in part on the one or more frames; determining a corrected map of depth and spectral information by correcting the spectral measurement using the depth measurement.

45. A system for measuring a target, comprising:

an illumination source configured to illuminate the target with light;

a detector configured to detect light from the target; and

a processor configured to:

determine a depth measurement and a spectral measurement based at least in part on the light detected by the detector; and

determine a map of depth and spectral information by correcting the spectral measurement using the depth measurement.

46. A method for a target, comprising:

detecting, using a detector, light from the target illuminated using an illumination source; determining a depth measurement and a spectral measurement based at least in part on the light detected by the detector; and determining a map of depth and spectral information by correcting the spectral measurement using the depth measurement.

47. A method for material processing using 3D spectral scanning, comprising:

performing 3D spectral scanning on an object;

performing detection and diagnosis to identify issues associated with the object based on the 3D spectral scanning;

generating a decision in response to the detection and diagnosis;

outputting a control action in response to the decision; and

actuating or controlling a device in response to the control action to address an issue associated with the object.