EP3915511A1 - Compensation de flou de mouvement - Google Patents

Compensation de flou de mouvement Download PDF

Info

Publication number
EP3915511A1
EP3915511A1 EP21175420.5A EP21175420A EP3915511A1 EP 3915511 A1 EP3915511 A1 EP 3915511A1 EP 21175420 A EP21175420 A EP 21175420A EP 3915511 A1 EP3915511 A1 EP 3915511A1
Authority
EP
European Patent Office
Prior art keywords
motion
scanner
camera
focus
focus plane
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP21175420.5A
Other languages
German (de)
English (en)
Inventor
Rune Fisker
Mike Van Der Poel
Henrik ÖJELUND
Kristian Evers HANSEN
Anders GAARDE
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
3Shape AS
Original Assignee
3Shape AS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 3Shape AS filed Critical 3Shape AS
Publication of EP3915511A1 publication Critical patent/EP3915511A1/fr
Withdrawn legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/73—Deblurring; Sharpening
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G06T7/571—Depth or shape recovery from multiple images from focus
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H04N23/681—Motion detection
    • H04N23/6812—Motion detection based on additional sensors, e.g. acceleration sensors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10141—Special mode during image acquisition
    • G06T2207/10148—Varying focus
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20172—Image enhancement details
    • G06T2207/20201—Motion blur correction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30036—Dental; Teeth

Definitions

  • This invention generally relates to a system and a method for compensating for motion blur. More particularly, the invention relates to motion blur compensation in 3D scanning of at least a part of an object by means of a 3D scanner.
  • Motion blur compensation is generally known from 2D scanning or imaging of objects.
  • the 3D scanning comprises:
  • the 3D scanning comprises:
  • the in-focus position of a sensor element corresponds to the position of the focus plane in which the focus plane intersects a portion of the object surface and where the light rays returning from this portion of the object surface are imaged onto this particular sensor element.
  • the in-focus position for a given sensor element may correspond to the position of the focus plane where the pattern is in focus on the object being scanned for this sensor element.
  • the camera may record a high value of the correlation measure for that sensor element at the in-focus position, when the position of the focus plane is varied over a range of values.
  • the 3D surface is generated from the sequence of focus plane images by motion compensating each focus plane image prior to the generation of the 3D surface.
  • the 3D surface is generated from the sequence of focus plane images and motion compensation is subsequently performed to determine the relative motion during the focus plane acquisition.
  • the position of the focus plane is varied by translating the focusing optics relative to other optical components of the 3D scanner and/or by moving the focusing optics relative to the casing of the scanner.
  • focus plane images There may be for example more than 200 focus plane images, such as 225 focus plane images, in a sequence of focus plane images used in generating a 3D surface.
  • the focus plane images are 2D images.
  • the optical system used for imaging on the camera transmitted light returned from the object to the camera is also used for transmitting the probe light towards the object and/or for transmitting light returned from the object to the camera,
  • the array of sensor elements may comprise image sensor(s), photo sensor(s) and the like which is suitable for acquiring images in a scanner.
  • scanning is generally meant optical scanning or imaging using laser light, white light etc.
  • a sequence of focus plane images captured along the direction of the optical axis are combined to provide a are depth image.
  • At least a part of the object is in focus in at least one of the focus plane images in a sequence of focus plane images.
  • the time period between acquisition of each focus plane image in a sequence is fixed/predetermined/known.
  • Each focus plane image may be acquired a certain time period after the previous focus plane image was acquired.
  • the focusing optics may move between the acquisition of each image, and thus each focus plane image may be acquired in a different distance from or at a different position relative to the object than the previous focus plane images.
  • One cycle of focus plane image capture may be from when the focusing optics is in position P until the focusing optics is again in position P. This cycle may be denoted a sweep. There may such as 15 sweeps per second.
  • a number of 3D surfaces or sub-scans may then be combined to create a full scan of the object for generating a 3D model of the object.
  • a 3D model of the object can be generated by combining a number of said 3D surfaces or sub-scans acquired during a scanning of at least part of the object.
  • a sub-scan for a given sequence of focus plane images is generated by analyzing the individual focus plane images and identifying the portions (if any) of each focus plane wherein a section of the object is imaged in focus into the camera. The in-focus portions are then combined to provide the sub-scan.
  • the sub-scan may thus comprise 3D points corresponding to the identified in-focus portions.
  • a 3D surface for a given sub-scan can be generated by a triangulation of the 3D points of the sub-scan using computer implemented algorithms.
  • a sequence of focus plane images is obtained every time the focus plane position is varied from one end of the range of positions covered by the cycle to the opposite end of this range, such that two sequences are acquired for each cycle.
  • determining the relative motion of the scanner during the acquisition of the sequence of focus plane images is performed by analysis of the sequence in itself.
  • the relative motion of the scanner and the object during the acquisition of the sequence of focus plane images can be determined by analyzing the focus plane images of the sequence.
  • the relative motion can e.g. be determined based on structures or surface sections identified in different focus plane images.
  • determining the relative motion of the scanner during the acquisition of the sequence of focus plane images is performed by sensors in and/or on the scanner and/or by sensors on the object and/or by sensors in the room where the scanner and the object are located.
  • the motion sensors may be small sensor such as microelectromechanical systems (MEMS) motion sensors.
  • MEMS microelectromechanical systems
  • the motion sensors may measure all motion in 3D, i.e., both translations and rotations for the three principal coordinate axes.
  • the benefits are:
  • a 3-axis accelerometer can provide the direction of gravity relative to the scanning device.
  • a magnetometer can provide directional information relative to the scanning device, in this case from the earth's magnetic field. Therefore, such devices can help with stitching/registration.
  • the motion is determined by determining the position and orientation of one or more of the sensors.
  • the motion is determined by means of one or more physical components arranged in the handheld scanner.
  • the motion is determined by means of 3D position sensors.
  • the motion is determined by means of optical tracking.
  • the optical tracking may comprise LED(s) and camera(s), where the LED(s) may flash and the flashing can be detected by the camera(s).
  • the motion is determined by means of one or more gyroscopes.
  • a gyroscope is a device for measuring or maintaining orientation, based on the principles of conservation of angular momentum.
  • a mechanical gyroscope is essentially a spinning wheel or disk whose axle is free to take any orientation.
  • the gyroscopes used to determine the orientation of the sensor may be mechanical gyroscopes, electronic, microchip-packaged MEMS gyroscope devices, solid state ring lasers, fibre optic gyroscopes, quantum gyroscope and/or the like.
  • the motion is determined by means of one or more accelerometers.
  • the motion is determined by means of one or more magnetometers.
  • the motion is determined by means of one or more electromagnetic coils.
  • the motion is determined by means of a computerized measurement arm.
  • the measurement arm may for instance be from FARO Technologies. There may be goniometers in the measurement arm for measuring the movements of the arm.
  • the motion is determined by means of one or more axes on which the sensor is configured to move.
  • An example of an axes based system is a coordinate measuring machine (CMM), which is a device for measuring the physical geometrical characteristics of an object. This machine may be computer controlled.
  • CMM coordinate measuring machine
  • a typical CMM is composed of three axes, X, Y and Z, and these axes are orthogonal to each other in a typical three dimensional coordinate system.
  • Each axis has a scale system that indicates the location of that axis. Measurements may be defined by a probe attached to the third moving axis of this machine, and the machine will read the input from the touch probe. Probes may be mechanical, optical, laser, or white light, among others.
  • the axes on which the sensor is configured to move are translational and / or rotational axes.
  • the scanner is a rigid body which can perform motion in a three dimensional space, where the motion can be translation in three perpendicular axes, x, y, z, which is movement forward/backward, up/down, left/right, and this is combined with rotation about the three perpendicular axes.
  • the motion has six degrees of freedom as the movement along each of the three axes is independent of each other and independent of the rotation about any of these axes.
  • the motion is determined by means of a texture image sensor having a depth of focus which is larger than the depth of focus of the focusing optics.
  • the optical system of the 3D scanner is configured for imaging with a first depth of focus on the camera at least part of the transmitted light returned from the object to the array of sensor elements of the camera.
  • 3D scanner comprising a second camera
  • the 3D scanner comprises a second camera configured for capturing at least some of the selected light rays with a second depth of field which is larger than the first depth of field.
  • the second depth of field image has such a large depth of field that all scanned parts of the object are in focus.
  • the second depth of field image may also have the same perspective as the first depth of field images.
  • the 3D scanner comprises a second optical system comprising means for selecting a portion of light rays returned from the object and imaging these light rays onto the second camera.
  • the second camera is a miniature camera with a diameter of a few millimeters.
  • the miniature camera may be sufficiently small such that it can be placed in beam path of the optical system of the 3D scanner without changing the field of view significantly.
  • the second depth of field image can have a large frame rate, such as e.g. 30 fps or 60 fps.
  • the high frame rate may be used for aiding a tracking of the relative motion of the scanner and the object being scanned by analyzing the motion of feature points in the large depth of field images.
  • the second depth of field image can be in colors. This may allow for overlay of color texture onto the 3D scanned surface.
  • the second depth of field image in color can be used for distinguishing different materials or surfaces on the object being scanned. Such a distinction can be useful in further treatment of the 3D data of the object.
  • the second depth of field image is substantially unaffected by varying the position of the focus plane on the object.
  • determining the relative motion of the scanner during the acquisition of the sequence of focus plane images is performed by analysis of the sequence and one or more other sequences obtained before and/or after said sequence.
  • each sequence involved in the determination of the motion covers a portion of the surface of the object having at least a partial overlap with a portion of the surface covered by at least one other sequence.
  • At least a part of the object is a rigid body.
  • the motion determination and 3D surface generation comprises:
  • the motion determination and 3D surface generation comprises:
  • the motion determination and 3D surface generation comprises:
  • first and the second sequences and the first and second 3D surfaces are not necessarily the first and the second sequence or the first and second 3D surface which is acquired during a scanning of an object but may be any two sequences and 3D surfaces acquired during this scanning.
  • the relative scanner position for a generate 3D surface may correspond to an average position of the scanner relative to the object during the acquisition of the sequence of focus plane images from which this 3D surface is generated. This may in particular be the case if there is a relative movement during the acquisition of the sequence of focus plane images.
  • the relative scanner motion during the first sequence may be obtained from the obtained relative scanner positions in the common coordinate system based on an assumption that the movement is smooth from the scanner position relative to the first generated 3D surface to the scanner position relative to the second generated 3D surface.
  • a time stamp may be needed on each generated 3D surface and/or on the focus plane images for determining the relative scanner motion.
  • the relative orientation and position of the 3D surfaces can be determined.
  • the generated 3D surfaces may be projected into a common coordinate system.
  • the estimated relative scanner positions may be given relative to the generated 3D surface and these positions can also be projected into the common coordinate system, such that the 3D surfaces and the relative scanner positions can be expressed in the common coordinate system.
  • the motion determination and 3D surface generation comprises performing, in an iterative fashion, the following steps one or more times:
  • the motion determination and 3D surface generation comprises:
  • At least some of the steps a: to d: are performed in an iterative fashion, such that at least some of these steps are performed one or more times.
  • the motion compensation of each of the 3D surfaces performed in step b comprises a motion compensation of the individual focus plane images based on the determined relative scanner motion.
  • the further motion compensation of each of the 3D surfaces performed in step e comprises a motion compensation of the individual focus plane images based on the more precise measure of the relative scanner motion.
  • the scanner is hand held.
  • the acquisition of the focus plane images is adapted to be performed during relative motion of the scanner and the object.
  • the acquisition of focus plane images can be performed while the sensor is moving relative to the object.
  • Some of the focus plane image may however still be acquired while the sensor and the object is not moving relative to each other. But it is understood that the focus plane images may be acquired while the sensor is moving relative to the object.
  • Prior art does not provide for motion compensation in relation to a 3D scanner where focus plane images can be acquired while the handheld scanner, and thereby, sensor is moving, i.e. in prior art scanners focus plane images acquired during motion cannot be processed and used to obtain an assembled scan of an object.
  • the motion of the scanner is free and not restricted to any predetermined/certain/specific motion pattern.
  • the scanner is a desktop scanner.
  • the scanner and/or the object is constrained to move along one or more axes.
  • said axes are translational and/or rotational axes.
  • the motion is determined along one spatial or rotational coordinate.
  • the motion is determined along any one or more of two - six spatial and/or rotational coordinates.
  • the determined motion is registered/recorded/represented as a velocity vector for at least two focus plane images of the sequence of focus plane images.
  • the determined motion is registered/recorded/represented as a constant velocity and / or rotation vector for the whole sequence of focus plane images in a sequence.
  • the determined motion is registered/recorded/represented as a curve/trajectory fitted between the focus plane images.
  • the fitted curve/trajectory is along one or more translational and / or rotational dimensions.
  • the motion compensation comprises virtually shifting one or more of the focus plane images in a sequence into a position and/or orientation corresponding to the position and/or orientation of another one of the focus plane images in the sequence, such as the first one of the focus plane images in the sequence or a focus plane near the middle of said sequence.
  • the another one of the focus plane images in the sequence which the one or more focus plane images are shifted to may be a focus plane image selected based on the quality of the focus in that image.
  • the motion compensation comprises substituting a first 3D point on the generated 3D surface with a second 3D point according to the corresponding virtually shifted focus plane image.
  • the motion compensation comprises shifting one or more pixels, such as shifting data acquired by the sensor elements.
  • a pixel group of 6x6 pixels, 8x8 pixels or the like may be shifted.
  • a virtual 3D model of the object is created by means of one or more of the generated 3D surfaces.
  • the method disclosed may also provide for motion unwarping, optimization of LED power to the relevant material, e.g. teeth, based on multiple criteria, hole closing by means of LED adjustment, virtual preparation, combination of old and news scans, separate scans using the focus scanning, auto segmentation from a 2D image, detection of other features from the 2D image, features for facilitating use of multiple cords when performing preparations, use of the focus scanning in knee diagnosis, in brain applications, in surgery tools, alignment of x-ray scans with the focus scanning, real-time preparation guides etc.
  • the motion between at least two subsequent 3D surfaces are determined by aligning/registering the at least two subsequent 3D surfaces.
  • ICP iterative closest point
  • ICP Iterative Closest Point
  • ICP can be used for aligning, and it is employed to minimize the difference between two clouds of points.
  • ICP can be used to reconstruct 2D or 3D surfaces from different scan.
  • ICP iteratively revises the transformation, i.e. translation or rotation, needed to minimize the distance between the points of two raw scans.
  • the input for ICP may be points from two raw scans, initial estimation of the transformation, and criteria for stopping the iteration.
  • the output will thus be a refined transformation.
  • the method of least squares fit can be used in alignment.
  • aligning/registering is performed by selecting corresponding points on the at least two 3D surfaces, and minimizing the distance between the at least two 3D surfaces.
  • Corresponding points may the closest points on two surfaces, or point determined by a normal vector from a point on the other surface etc
  • the distance may be minimized with regards to translation and rotation.
  • aligning/registration is continued in an iterative process to obtain an improved motion estimation.
  • the sensor position of each sequence is determined based on the alignment.
  • aligning comprises aligning the coordinate systems of at least two 3D surfaces.
  • aligning comprises aligning by means of matching / comparing one or more specific features, such as one or more specific features common to the at least two 3D surfaces, such as the margin line of a prepared tooth.
  • aligning comprises aligning by means of matching / comparing one or more peripheral features of the at least two 3D surfaces.
  • aligning comprises registration of the at least two 3D surfaces.
  • aligning comprises applying a predefined criterion for maximum allowed error in the registration.
  • the motion compensation comprises reconstructing a self-consistent surface model and motion and/or rotation of the scanner relative to the object from two or more scans of the object where two successive scans overlap at least partially.
  • the method and apparatus described in this invention is for providing a 3D surface registration of objects using light as a non-contact probing agent.
  • the probe light is provided in the form of an illumination pattern to provide a light oscillation on the object.
  • the variation / oscillation in the pattern may be spatial, e.g. a static checkerboard pattern, and/or it may be time varying, for example by moving a pattern across the object being scanned.
  • the invention provides for a variation of the focus plane of the pattern over a range of focus plane positions while maintaining a fixed spatial relation of the scanner and the object.
  • the signals from the array of sensor elements are light intensity.
  • One embodiment of the invention comprises a first optical system, such as an arrangement of lenses, for transmitting the probe light towards the object and a second optical system for imaging light returned from the object to the camera.
  • a first optical system such as an arrangement of lenses
  • only one optical system images the pattern onto the object and images the object, or at least a part of the object, onto the camera, such as along the same optical axis, however along opposite optical paths.
  • an optical system provides an imaging of the pattern onto the object being probed and from the object being probed to the camera.
  • the focus plane may be adjusted in such a way that the image of the pattern on the probed object is shifted along the optical axis, such as in equal steps from one end of the scanning region to the other.
  • a probe light incorporating the pattern provides a pattern of light and darkness on the object. Specifically, when the pattern is varied in time for a fixed focus plane then the in-focus regions on the object may display an oscillating pattern of light and darkness. The out-of-focus regions will display smaller or no contrast in the light oscillations.
  • the description of the invention is based on the case where the light incident on the object is reflected diffusively and/or specularly from the object's surface. But it is understood that the scanning apparatus and method are not limited to this situation. They are also applicable to e.g. the situation where the incident light penetrates the surface and is reflected and/or scattered and/or gives rise to fluorescence and/or phosphorescence in the object. Inner surfaces in a sufficiently translucent object may also be illuminated by the illumination pattern and be imaged onto the camera. In this case a volumetric scanning is possible. Some planktic organisms are examples of such objects.
  • a single sub-scan can be obtained by collecting a number of 2D images at different positions of the focus plane and at different instances of the pattern.
  • the pattern will be projected onto the surface point in-focus and with high contrast, thereby giving rise to a large variation, or amplitude, of the pixel value over time.
  • For each pixel it is thus possible to identify individual settings of the focusing plane for which each pixel will be in focus.
  • the focus position is calculated by determining the light oscillation amplitude for each of a plurality of sensor elements for a range of focus planes.
  • a single sub-scan can be obtained by collecting a number of 2D images at different positions of the focus plane.
  • the pattern will be projected onto the surface point in-focus and with high contrast.
  • the high contrast gives rise to a large spatial variation of the static pattern on the surface of the object, thereby providing a large variation, or amplitude, of the pixel values over a group of adjacent pixels.
  • For each group of pixels it is thus possible to identify individual settings of the focusing plane for which each group of pixels will be in focus.
  • the focus position is calculated by determining the light oscillation amplitude for each of a plurality of groups of the sensor elements for a range of focus planes.
  • the 2D to 3D conversion of the image data can be performed in a number of ways known in the art.
  • the 3D surface structure of the probed object can be determined by finding the plane corresponding to the maximum light oscillation amplitude for each sensor element, or for each group of sensor elements, in the camera's sensor array when recording the light amplitude for a range of different focus planes.
  • the focus plane is adjusted in equal steps from one end of the scanning region to the other.
  • the focus plane can be moved in a range large enough to at least coincide with the surface of the object being scanned.
  • the scanner comprises at least one beam splitter located in the optical path.
  • an image of the object may be formed in the camera by means of a beam splitter. Exemplary uses of beam splitters are illustrated in the figures.
  • light is transmitted in an optical system comprising a lens system.
  • This lens system may transmit the pattern towards the object and images light reflected from the object to the camera.
  • the optical system is substantially telecentric in the space of the probed object.
  • the optical system may also be telecentric in the space of the pattern and camera.
  • the variation, i.e. scanning, of the focus plane is possible without moving the scanner in relation to the object being scanned.
  • the focus plane may be varied, such as continuously varied in a periodic fashion, even when the pattern generation means, the camera, and the object being scanned are fixed in relation to each other. Further, the 3D surface acquisition time should be small enough to reduce the impact of relative movement between probe and teeth, e.g. reduce effect of shaking.
  • the focus plane is varied by means of at least one focusing element, such as moved by moving the focusing optics.
  • the focus plane is periodically varied with a predefined frequency. Said frequency may be at least 1 Hz, such as at least 2 Hz, 3, 4, 5, 6, 7, 8, 9 or at least 10 Hz, such as at least 20, 40, 60, 80 or at least 100 Hz.
  • the focusing element is part of the optical system.
  • the focusing element may be a lens in a lens system.
  • Some embodiments of the scanner comprise means, such as a translation stage, for adjusting and controlling the position of the focusing element. In that way the focus plane may be varied, for example by translating the focusing element back and forth along the optical axis.
  • Some embodiments of the scanner thus comprises means for reducing and/or eliminating the vibration and/or shaking from the focusing element adjustment system, thereby increasing the stability of the scanner.
  • This may at least partly be provided by means for fixing and/or maintaining the centre of mass of the focusing element adjustment system, such as a counter-weight to substantially counter-balance movement of the focusing element; for example, by translating a counter-weight opposite to the movement of the focusing element. Ease of operation may be achieved if the counter-weight and the focusing element are connected and driven by the same translation means. This may however, only substantially reduce the vibration to the first order.
  • a further embodiment of the invention thus comprises means for reducing and/or eliminating the first order, second order, third order and/or higher order vibration and/or shaking from the focusing element adjustment system, thereby increasing the stability of the scanner.
  • more than one optical element is moved to shift the position of the focus plane. In that embodiment it is desirable that these elements are moved together and that the elements are physically adjacent.
  • the optical system is telecentric, or near telecentric, for all focus plane positions. Thus, even though one or more lenses in the optical system may be shifted back and forth to change the focus plane position, the telecentricity of the optical system is maintained.
  • Focus gearing is the correlation between movement of the lens and movement of the focus plane position.
  • a focus gearing of 2 means that a translation of the focusing element of 1 mm corresponds to a translation of the focus plane position of 2 mm.
  • Focus gearing can be provided by a suitable design of the optical system. The advantage of focus gearing is that a small movement of the focusing element may correspond to a large variation of the focus plane position.
  • the focus gearing is between 0.1 and 100, such as between 0.1 and 1, such as between 1 and 10, such as between 2 and 8, such as between 3 and 6, such as least 10, such as at least 20.
  • the focusing element is a liquid lens.
  • a liquid lens can control the focus plane without use of any moving parts.
  • the camera may be a standard digital camera accommodating a standard CCD or CMOS chip with one A/D converter per line of sensor elements (pixels).
  • the scanner according to the invention may comprise a high-speed camera accommodating multiple A/D converters per line of pixels, e.g. at least 2, 4, 8 or 16 A/D converters per line of pixels.
  • the 3D scanner is configured for providing a probe light with an embedded pattern that is projected on to the object being scanned.
  • the pattern may be static or time varying.
  • the time varying pattern may provide a variation of light and darkness on and/or in the object. Specifically, when the pattern is varied in time for a fixed focus plane then the in-focus regions on the object will display an oscillating pattern of light and darkness. The out-of-focus regions will display smaller or no contrast in the light oscillations.
  • the static pattern may provide a spatial variation of light and darkness on and/or in the object. Specifically, the in-focus regions will display an oscillating pattern of light and darkness in space. The out-of-focus regions will display smaller or no contrast in the spatial light oscillations.
  • Light may be provided from an external light source, however preferably the scanner comprises at least one light source and pattern generation means to produce the pattern. It is advantageous in terms of signal-to-noise ratio to design a light source such that the intensity in the non-masked parts of the pattern is as close to uniform in space as possible.
  • the light source and the pattern generation means is integrated in a single component, such as a segmented LED.
  • a segmented LED may provide a static pattern and/or it may provide a time varying pattern in itself by turning on and off the different segments in sequence.
  • the time varying pattern is periodically varying in time.
  • the static pattern is periodically varying in space.
  • the pattern generation means comprises at least one translucent and/or transparent pattern element.
  • the mask comprises a plurality of radial spokes, preferably arranged in a symmetrical order.
  • the scanner may also comprise means for rotating and/or translating the pattern element.
  • a glass plate with an opaque mask can be used.
  • the mask comprises a line pattern or checkerboard pattern. In general said mask preferably possesses rotational and/or translational periodicity.
  • the pattern element is located in the optical path.
  • light from the light source may be transmitted through the pattern element, e.g. transmitted transversely through the pattern element.
  • the time varying pattern can then be generated by rotating and/or translating the pattern element.
  • a pattern element generating a static pattern does not need to be moved during a scan.
  • One object of the invention is to provide short scan time and real time processing, e.g. to provide live feedback to a scanner operator to make a fast scan of an entire tooth arch.
  • real time high resolution 3D scanning creates an enormous amount of data. Therefore some data processing may preferably be provided in the scanner housing, i.e. close to the optical components, to reduce data transfer rate to e.g. a cart, workstation or display.
  • various correlation techniques may be embedded / implemented. This may for example be implemented in the camera electronics to discriminate out-of-focus information.
  • the pattern may be applied to provide illumination with an embedded spatial structure on the object being scanned.
  • Determining in-focus information may then relate to calculating a correlation measure of this spatially structured light signal (which we term input signal) with the variation of the pattern itself (which we term reference signal).
  • the magnitude of such a correlation measure is high if the input signal coincides with the reference signal. If the input signal displays little or no variation then the magnitude of the correlation measure is low. If the input signal displays a large spatial variation but this variation is different than the variation in the reference signal then the magnitude of the correlation measure is also low.
  • the scanner and/or the scanner head may be wireless, thereby simplifying handling and operation of the scanner and increasing accessibility under difficult scanning situations, e.g. intra-oral or in the ear scanning. However, wireless operation may further increase the need for local data processing to avoid wireless transmission of raw 3D data.
  • the reference signal is provided by the pattern generating means and may be periodic.
  • the variation in the input signal may be periodic and it may be confined to one or a few periods.
  • the reference signal may be determined independently of the input signal.
  • the phase between the oscillating input and reference signal may be known independently of the input signal.
  • the correlation is typically related to the amplitude of the variation. If the phase between the oscillating input and reference signals is not known it may be necessary to determine both cosine and sinusoidal part of the input signal before the input signal's amplitude of variation can be determined. This is not necessary when the phase is known.
  • the object To obtain a sharp image of an object by means of a camera the object must be in focus and the optics of the camera and the object must be in a fixed spatial relationship during the exposure time of the image sensor of the camera.
  • the exposure time of the sensor array should be as high as the sensor frame rate permits.
  • images are recorded (sampled) in the camera while the pattern is continuously varying (e.g. by continuously rotating a pattern wheel) and the focus plane is continuously moved.
  • images are recorded (sampled) in the camera while the pattern is fixed and the focus plane is continuously moved, i.e. no movement of the pattern.
  • the light source is a segmented light source, such as a segment LED that flashes in an appropriate fashion.
  • the knowledge of the pattern is obtained by a combination of prior knowledge of the geometry of the individual segments on the segmented LED give rise to a variation across light sensing elements and the applied current to different segments of the LED at each recording.
  • images are recorded (sampled) in the camera while the pattern is continuously varying and the focus plane is fixed.
  • images are recorded (sampled) in the camera while the pattern and the focus plane are fixed.
  • the above mentioned correlation principles require the pattern to be varying in time. If the optical system and camera provides a lateral resolution which is at least two times what is needed for the scan of the object then it is possible to scan with a static pattern, i.e. a pattern which is not changing in time. This principle is termed "spatial correlation".
  • the correlation measure is thus at least computed with sensor signals recorded at different sensor sites.
  • the lateral resolution of an optical system is to be understood as the ability of optical elements in the optical system, e.g. a lens system, to image spatial frequencies on the object being scanned up to a certain point.
  • Modulation transfer curves of the optical system are typically used to describe imaging of spatial frequencies in an optical system.
  • the resolution of the camera is a combined effect of the spacing of the individual camera sensor elements and the resolution of the optical system.
  • the correlation measure refers to a correlation between input signal and reference signal occurring in space rather than in time.
  • the resolution of the measured 3D geometry is equal to the resolution of the camera.
  • the resolution of the measured 3D geometry is lower than the resolution of the camera, such as at least 2 times lower, such as at least 3 times lower, such as at least 4 times lower, such as least 5 times lower, such as at least 10 times lower.
  • the sensor element array is divided into groups of sensor elements, preferably rectangular groups, such as square groups of sensor elements, preferably adjacent sensor elements. The resolution of the scan, i.e. the measured 3D geometry, will then be determined by the size of these groups of sensor elements.
  • the oscillation in the light signal is provided within these groups of sensor elements, and the amplitude of the light oscillation may then be obtained by analyzing the groups of sensor elements.
  • the division of the sensor element array into groups may be provided in the data processing stage, i.e. the division is not a physical division thereby possibly requiring a specially adapted sensor array. Thus, the division into groups is "virtual" even though the single pixel in a group is an actual physical pixel.
  • the pattern possesses translational periodicity along at least one spatial coordinate.
  • the spatially periodic pattern is aligned with the rows and/or the columns of the array of sensor elements.
  • the rows or columns of the pixels in the camera may be parallel with the lines of the pattern.
  • the row and columns of the checkerboard may be aligned with the rows and columns, respectively, of the pixels in the camera.
  • aligning is meant that the image of the pattern onto the camera is aligned with the "pattern" of the sensor element in the sensor array of the camera.
  • At least one spatial period of the pattern corresponds to a group of sensor elements.
  • all groups of sensor elements contain the same number of elements and have the same shape.
  • the period of a checkerboard pattern corresponds to a square group of e.g. 2x2, 3x3, 4x4, 5x5, 6x6, 7x7, 8x8, 9x9, 10x10 or more pixels on the camera.
  • one or more edges of the pattern is aligned with and/or coincide with one or more edges of the array of sensor elements.
  • a checkerboard pattern may be aligned with the camera pixels in such a way that the edges of the image of the checkerboard pattern onto the camera coincide with the edges of the pixels.
  • spatial correlation has the advantage that no moving pattern is required. This implies that knowledge of the pattern configuration may be obtained prior to the 3D scanning. Conversely, the advantage of temporal correlation is its higher resolution, as no pixel grouping is required.
  • Relating identified focus position(s) for camera sensor or camera sensor groups to 3D world coordinates may be done by ray tracing through the optical system. Before such ray tracing can be performed the parameters of the optical system need to be known.
  • One embodiment of the invention comprises a calibration step to obtain such knowledge.
  • a further embodiment of the invention comprises a calibration step in which images of an object of known geometry are recorded for a plurality of focus positions. Such an object may be a planar checkerboard pattern. Then, the scanner can be calibrated by generating simulated ray traced images of the calibration object and then adjusting optical system parameters as to minimize the difference between the simulated and recorded images.
  • the calibration step requires recording of images for a plurality of focus positions for several different calibration objects and/or several different orientations and/or positions of one calibration object.
  • a single sub-scan can be obtained by collecting a number of 2D images at different positions of the focus plane and at different instances of the pattern.
  • the pattern will be projected onto the surface point in-focus and with high contrast, thereby giving rise to a large variation, or amplitude, of the pixel value over time.
  • For each pixel it is thus possible to identify individual settings of the focusing plane for which each pixel will be in focus.
  • the focus position is calculated by determining the light oscillation amplitude for each of a plurality of sensor elements for a range of focus planes.
  • a single sub-scan can be obtained by collecting a number of 2D images at different positions of the focus plane.
  • the pattern will be projected onto the surface point in-focus and with high contrast.
  • the high contrast gives rise to a large spatial variation of the static pattern on the surface of the object, thereby providing a large variation, or amplitude, of the pixel values over a group of adjacent pixels.
  • For each group of pixels it is thus possible to identify individual settings of the focusing plane for which each group of pixels will be in focus.
  • the focus position is calculated by determining the light oscillation amplitude for each of a plurality of groups of the sensor elements for a range of focus planes.
  • the 2D to 3D conversion of the image data can be performed in a number of ways known in the art.
  • the 3D surface structure of the probed object can be determined by finding the plane corresponding to the maximum light oscillation amplitude for each sensor element, or for each group of sensor elements, in the camera's sensor array when recording the light amplitude for a range of different focus planes.
  • the focus plane is adjusted in equal steps from one end of the scanning region to the other. It may be preferred that the focus plane can be moved in a range large enough to at least coincide with the surface of the object being scanned.
  • the present invention relates to different aspects including the method described above and in the following, and corresponding methods, devices, uses and/or product means, each yielding one or more of the benefits and advantages described in connection with the first mentioned aspect, and each having one or more embodiments corresponding to the embodiments described in connection with the first mentioned aspect and/or disclosed in the appended claims.
  • the means for determining whether there is a relative motion between the scanner and the object during the acquisition of the sequence of focus plane images, the means for performing a motion compensation based on the determined motion, and the means for generating a 3D surface from the sequence of focus plane images comprises a data processing device, such as a computer or a microprocessor.
  • the 3D scanner comprises a non-transitory computer readable medium having one or more computer instructions stored thereon, where said computer instructions comprises instructions for carrying out the method for compensating for motions blur according to the present invention.
  • a scanner system comprising
  • the invention relates to a computer program product comprising program code means for causing a data processing system to perform the method according to any of the embodiments, when said program code means are executed on the data processing system, and a computer program product, comprising a computer-readable medium having stored there on the program code means.
  • Pattern A light signal comprising an embedded spatial structure in the lateral plane. May also be termed "illumination pattern”.
  • Time varying pattern A pattern that varies in time, i.e. the embedded spatial structure varies in time. May also be termed “time varying illumination pattern”. In the following also termed “fringes”.
  • Static pattern A pattern that does not vary in time, e.g. a static checkerboard pattern or a static line pattern.
  • Pattern configuration The state of the pattern. Knowledge of the pattern configuration at a certain time amounts to knowing the spatial structure of the illumination at that time. For a periodic pattern the pattern configuration will include information of the pattern phase. If a surface element of the object being scanned is imaged onto the camera then knowledge of the pattern configuration amounts to knowledge of what part of the pattern is illuminating the surface element.
  • Focus plane A surface where light rays emitted from the pattern converge to form an image on the object being scanned.
  • the focus plane does not need to be flat. It may be a curved surface.
  • Optical system An arrangement of optical components, e.g. lenses, that transmit, collimate and/or images light, e.g. transmitting probe light towards the object, imaging the pattern on and/or in the object, and imaging the object, or at least a part of the object, on the camera.
  • optical components e.g. lenses
  • that transmit, collimate and/or images light e.g. transmitting probe light towards the object, imaging the pattern on and/or in the object, and imaging the object, or at least a part of the object, on the camera.
  • Optical axis An axis defined by the propagation of a light beam.
  • An optical axis is often a straight line.
  • the optical axis is defined by the configuration of a plurality of optical components, e.g. the configuration of lenses in the optical system.
  • the optical axis can be defined by the propagation of the light in the optical system transmitting the pattern onto the object and imaging the object onto the camera. The optical axis will often coincide with the longitudinal axis of the scanner.
  • Optical path The path defined by the propagation of the light from the light source to the camera. Thus, a part of the optical path often coincides with the optical axis. Whereas the optical axis may be a straight line, the optical path may be a non-straight line, for example when the light is reflected, scattered, bent, divided and/or the like provided e.g. by means of beam splitters, mirrors, optical fibers and the like.
  • Telecentric system An optical system that provides imaging in such a way that the chief rays are parallel to the optical axis of said optical system.
  • out-of-focus points have substantially same magnification as in-focus points. This may provide an advantage in the data processing.
  • a perfectly telecentric optical system is difficult to achieve, however an optical system which is substantially telecentric or near telecentric may be provided by careful optical design. Thus, when referring to a telecentric optical system it is to be understood that it may be only near telecentric.
  • Scan length A lateral dimension of the field of view. If the probe tip (i.e. scan head) comprises folding optics to direct the probe light in a direction different such as perpendicular to the optical axis then the scan length is the lateral dimension parallel to the optical axis.
  • Scan object The object to be scanned and on which surface the scanner provides information. "The scan object” may just be termed “the object”.
  • Imaging sensor comprising a plurality of sensors that respond to light input onto the imaging sensor.
  • the sensors can be ordered in a 2D array in rows and columns.
  • Input signal Light input signal or sensor input signal from the sensors in the camera. This can be integrated intensity of light incident on the sensor during the exposure time or integration of the sensor. In general, it translates to a pixel value within an image. May also be termed "sensor signal”.
  • Reference signal A signal derived from the pattern.
  • a reference signal may also be denoted a weight function or weight vector or reference vector.
  • Correlation measure A measure of the degree of correlation between a reference and input signal.
  • the correlation measure can be defined such that if the reference and input signal are linearly related to each other then the correlation measure obtains a larger magnitude than if they are not. In some cases the correlation measure is a light oscillation amplitude.
  • Image An image can be viewed as a 2D array of values (when obtained with a digital camera) or in optics, an image indicates that there exists a relation between an imaged surface and an image surface where light rays emerging from one point on said imaged surface substantially converge on one point on said image surface.
  • intensity is a measure of light power per unit area.
  • intensity may be used to term the recorded light signal on the individual sensing elements.
  • intensity reflects a time integration of light power per unit area on the sensing element over the exposure time involved in the image recording.
  • Aligning is used in two different contexts in this document. Aligning may mean aligning two 3D surfaces, such as registering the two 3D surfaces to each other in order to create a virtual model of the scanned object. Aligning may also mean that the pattern for example is aligned with the rows and/or the columns of the array of sensor elements.
  • Fig. 1 shows a flowchart of a method for compensating for motion blur when performing a 3D scanning of at least a part of an object by means of a 3D scanner.
  • the motion blur occurs because the scanner and the object are moved relative to each other while the scanning is performed.
  • the motion blur compensation comprises that:
  • a first 3D surface has been generated prior to steps 101-103 and the motion compensation in step 102 and/or the generation of the 3D surface in step 103 may be based on this first 3D surface.
  • the motion compensation may also involve determining the relative scanner motion based on the sequence of focus plane images itself.
  • Fig. 2 shows an example of a handheld scanner.
  • the handheld scanner 201 comprises a light source 202 for emitting light, a spatial pattern 203, a beam splitter 204, movable focusing optics 205, such as lenses, and an image sensor 206.
  • the light from the light source 202 travels back and forth through the optical system.
  • the optical system images the pattern 203 onto the object 208 being scanned and further images the object 208 being scanned onto the image sensor 206.
  • the movable focusing optics 205 comprises a focusing element, such as a lens, and can be moved such that the position of the focus image plane of the pattern 203 on the probed object 208 can be shifted.
  • One way to embody the focusing optics is to physically move a single lens element back and forth along the optical axis.
  • the device may include polarization optics and/or folding optics which directs the light out of the device in a direction different to the optical axis of the lens system, e.g.
  • the optical system provides an imaging of the pattern onto the object being probed and from the object being probed to the image sensor, e.g. camera.
  • the image sensor e.g. camera.
  • One application of the device could be for determining the 3D structure of teeth in the oral cavity.
  • Another application could be for determining the 3D shape of the ear canal and the external part of the ear.
  • the optical axis in fig. 2 is the axis defined by a straight line through the light source, optics and the lenses in the optical system. This also corresponds to the longitudinal axis of the scanner illustrated in fig. 2 .
  • the optical path is the path of the light from the light source to the object and back to the camera. The optical path may change direction, e.g. by means of beam splitter and folding optic.
  • the position of the focusing optics or the construction of the focusing element is adjusted in such a way that the image of the pattern on the scanned object is shifted along the optical axis, for example in equal steps from one end of the scanning region to the other.
  • the pattern is varied in time in a periodic fashion for a fixed focus position then the in-focus regions on the object will display a spatially varying pattern.
  • the out-of-focus regions will display smaller or no contrast in the light variation.
  • the 3D surface structure of the probed object may be determined by finding the plane corresponding to an extremum in the correlation measure for each sensor in the image sensor array or each group of sensor in the image sensor array when recording the correlation measure for a range of different focus positions. In some cases the focus position is moved in equal steps from one end of the scanning region to the other.
  • the scanning region may also be referred to as a scan volume.
  • An example of an in-focus region is seen in the insert 209, and an example of an out-of-focus region is seen in the insert 210.
  • Fig. 3 shows an example of a handheld scanner.
  • the handheld scanner 301 comprises a light source 302 for emitting light, a spatial pattern 303, a beam splitter 304, movable focus optic 305, such as lenses, an image sensor 306, and a tip or probe 307 which can be entered into a cavity for scanning an object 308.
  • the object 308 is teeth in an intra-oral cavity.
  • Fig. 4 shows an example of a section of teeth in the mouth which can be covered in a scan.
  • the scan volume 411 for one sequence of focus plane images is indicated by the transparent box.
  • the scan volume may be such as 17x15x20 mm, where the 15 mm may the "height" of the scan volume corresponding to the distance the focusing optics can move the focus plane.
  • Fig. 5 shows an example of how different sub-scans used for generating 3D surfaces are distributed across a set of teeth.
  • Each sub-scan provides a 3D surface of the scanned teeth.
  • the 3D surfaces are partly overlapping, whereby a motion of the scanner performed during the acquisition of the focus plane image can be determined by comparing the overlapping parts of two or more 3D surfaces.
  • Fig. 6 shows an example of two in-focus images of a tooth.
  • Fig. 6a shows a focus plane image 613 where a region in the upper left part of the image is in focus.
  • Fig. 6b shows another focus plane image 614 of the same area of the tooth as in fig. 6a ) where another region, which is in the lower middle-right part of the image is in focus.
  • Fig. 6c shows that the focus plane image 613 in fig. 6a ) is a focus plane image taken at the top of the tooth, and that the focus plane 614 in fig. 6b ) is the focus plane image taken at the centrally located depression in the tooth.
  • the arrow line 615 indicates the distance which a whole sequence of focus plane images covers.
  • Fig. 6d shows that the two focus positions are determined by determining the extremum point of amplitude, e.g. contrast, light intensity etc. for a sensor element or a group of sensor elements.
  • the focus plane image 613 has an extremum point at approximately 1400
  • the focus plane image 614 has an extremum point at approximately 4100.
  • Fig. 7 shows an example of focus plane images.
  • Fig. 7a shows a number of focus plane images, seen from the front, taken at different times, t0, t1 ...tn. No features are shown on the images for simplicity but normally the focus plane images could be like the images 613 and 614 shown in fig. 6 . This is also indicated by the reference 713 on the focus plane image taken at time t0, and the reference 714 on the focus plane image taken at time t4. Note that in real life the two focus plane images in fig. 6 would probably not lie so close in time that suggested in fig. 7
  • Fig. 7b shows a number of focus plane images, seen from above, taken at different times t0, t1, t2, t3, t4, t5.
  • an area or feature, the circular spot, on the focus plane images is seen to be located in different positions on the different focus plane images.
  • the reason for this can be that the handheld scanner has moved during the acquisition of the focus plane images. Since the area or feature moves to the right in the images, this corresponds to that the scanner has moved towards left while acquiring the images. This is indicated by the direction of the vector V in the figure.
  • the circular spots should really be more or less sharp in their contour in the different focus images because the area or feature will be in focus and out of focus in one or more of the images.
  • Fig. 7c shows that from the sequence of focus plane images in fig. 7a) and 7b ) a 3D surface 716 can be generated.
  • Fig. 8 shows an example of compensating for motion in the focus plane images.
  • Fig. 8a shows a number of focus plane images taken at different times t0, t1...t5.
  • the area indicated by the circular spot is not in the same position in the different focus plane images which indicates that the object and the scanner has moved relative to each other during the acquisition of the focus plane images.
  • Fig. 8b shows that the motion has been compensated for by correcting the focus plane images translationally towards left in this example.
  • Fig. 8c shows that the motion can also be compensated for by correcting the focus plane images rotationally, if this was needed.
  • the focus plane images are only drawn in 2D in these figures, it may not be seen that the focus plane images also needs rotational correction for compensating for the motion.
  • Fig. 9 shows an example of aligning 3D surfaces and compensating for motion in a 3D surface.
  • Fig. 9a shows a 3D surface 916, which for example may be generated from the focus plane images seen in fig. 7
  • Fig. 9b shows another 3D surface 917, which may have been generated in a subsequent sequence of focus plane images relative to the images in fig. 7 .
  • Fig. 9c shows the two 3D surface 916, 917 are attempted to be aligned/registered. Since the two 3D surfaces 916, 917 have 3D points which correspond to the same area of a tooth, it is possible to perform the alignment by comparing the corresponding points in the two 3D surfaces.
  • Fig. 9d shows the resulting 3D surface 918 when the two 3D surfaces 916, 917 have been merged together.
  • Fig 9e shows that based on the resulting 3D surface 918 the relative motion performed by the scanner during the acquisition of the focus plane images generating 3D surface 916 and 917 can be determined, and based on this determined motion the resulting 3D surface 918 can be corrected to a final "correct" 3D surface 919.
  • Fig. 10 shows an example of motion compensation for a 3D surface of a sub-scan.
  • Fig. 10a shows three 3D surfaces 1020, 1021, 1022 generated from focus plane images acquired in three consecutive sub-scans of a scanning of the object.
  • the sub-scans are referred to as sub-scan A (1020), sub-scan B (1021), and sub-scan C (1022).
  • Motion compensation for sub-scan B can be based on sub-scan A and sub-scan
  • Fig. 10b shows the three 3D surfaces 1020, 1021, 1022 aligned without compensating for a relative motion of the 3D scanner and the object.
  • Fig. 10c shows a schematic of the motion compensated 3D surface 10211 corresponding to sub-scan B which has been determined e.g. from a Motion Compensation Algorithm, such as a Motion Compensation Algorithm taken into account sub-scans A and C.
  • a Motion Compensation Algorithm such as a Motion Compensation Algorithm taken into account sub-scans A and C.
  • the Motion Compensation Algorithm can be used for every sub-scan acquired in a scanning of the object and the motion compensated 3D surfaces for all these sub-scans can then be combined to generate a 3D model 1023 of the object as seen in Fig. 10d ).
  • the generated 3D model 1023 any blur caused by a relative motion of the scanner and the object during the scanning has been mitigated or potentially reduced to an insignificant amount.
  • a sub-scan can consists of a point cloud with a list of 3D points (x,y,z). Each 3D point furthermore has a timestamp that indicates when the 3D point was recorded, i.e., from which image in the stack of focus images the point originates.
  • the motion of the scanner relative to the scanned object must be sufficiently smooth.
  • the two transformations are compared.
  • the transformations are time-normalized to have duration of 1 second for both transformations, by raising it to a non-integer power.
  • the scan volume is approximately a cube and for each of the 8 corners of this cube, the effects of the two time-normalized transformations are evaluated.
  • This procedure is carried out for all 8 corner points, and if all the found distances are less than a certain threshold, e.g. 2 mm, then it is deemed than the three sub-scans are recorded during a fluid motion and that the motion compensation algorithm hence is applicable.
  • a certain threshold e.g. 2 mm
  • This transformation corresponds to a time of 2 seconds (since it is combined from two transformations of 1 second each). For each point in sub-scan B, we can now find its motion-compensated position.
  • the transformation depends on whether the timestamp of the point is smaller than or larger than t B .
  • the procedure utilizing such a motion compensating procedure may be iterative such that the motion compensated sub-scans are used as input in a further motion compensation.
  • an ICP algorithm can be applied to determine new transformations T A , T B , and T C and from these transformations the further motion compensation can be based on the illustrated motion compensation algorithm and provide new motion compensated sub-scans from which 3D surfaces may be generated by e.g. triangulation. This is continued until final sub-scans and corresponding 3D surfaces are obtained and the 3D model of the object can then be generated by combining these final sub-scans or 3D surfaces.
  • the features of the method described above and in the following may be implemented in software and carried out on a data processing system or other processing means caused by the execution of computer-executable instructions.
  • the instructions may be program code means loaded in a memory, such as a RAM, from a storage medium or from another computer via a computer network.
  • the described features may be implemented by hardwired circuitry instead of software or in combination with software.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Length Measuring Devices By Optical Means (AREA)
EP21175420.5A 2010-12-21 2011-12-21 Compensation de flou de mouvement Withdrawn EP3915511A1 (fr)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
DKPA201001172 2010-12-21
US201061426126P 2010-12-22 2010-12-22
PCT/DK2011/050508 WO2012083968A1 (fr) 2010-12-21 2011-12-21 Compensation de flou de mouvement
EP11850309.3A EP2654606B1 (fr) 2010-12-21 2011-12-21 Compensation de flou de mouvement

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
EP11850309.3A Division EP2654606B1 (fr) 2010-12-21 2011-12-21 Compensation de flou de mouvement

Publications (1)

Publication Number Publication Date
EP3915511A1 true EP3915511A1 (fr) 2021-12-01

Family

ID=44263265

Family Applications (2)

Application Number Title Priority Date Filing Date
EP11850309.3A Active EP2654606B1 (fr) 2010-12-21 2011-12-21 Compensation de flou de mouvement
EP21175420.5A Withdrawn EP3915511A1 (fr) 2010-12-21 2011-12-21 Compensation de flou de mouvement

Family Applications Before (1)

Application Number Title Priority Date Filing Date
EP11850309.3A Active EP2654606B1 (fr) 2010-12-21 2011-12-21 Compensation de flou de mouvement

Country Status (5)

Country Link
US (3) US10218955B2 (fr)
EP (2) EP2654606B1 (fr)
CN (1) CN103491897B (fr)
ES (1) ES2884700T3 (fr)
WO (1) WO2012083968A1 (fr)

Families Citing this family (70)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
BR112012000189B1 (pt) 2009-06-17 2020-01-21 3Shape As aparelho de varredura com foco.
USRE48221E1 (en) 2010-12-06 2020-09-22 3Shape A/S System with 3D user interface integration
EP2654606B1 (fr) 2010-12-21 2021-05-26 3Shape A/S Compensation de flou de mouvement
US9403238B2 (en) 2011-09-21 2016-08-02 Align Technology, Inc. Laser cutting
US9489753B1 (en) * 2016-07-19 2016-11-08 Eyedeal Scanning, Llc Reconstruction of three dimensional model of an object from surface or slice scans compensating for motion blur
US9375300B2 (en) 2012-02-02 2016-06-28 Align Technology, Inc. Identifying forces on a tooth
US9220580B2 (en) 2012-03-01 2015-12-29 Align Technology, Inc. Determining a dental treatment difficulty
US9414897B2 (en) 2012-05-22 2016-08-16 Align Technology, Inc. Adjustment of tooth position in a virtual dental model
US8948482B2 (en) 2012-11-01 2015-02-03 Align Technology, Inc. Motion compensation in a three dimensional scan
US9717402B2 (en) 2012-11-21 2017-08-01 Align Technology, Inc. Confocal imaging using astigmatism
US11363938B2 (en) * 2013-03-14 2022-06-21 Ormco Corporation Feedback control mechanism for adjustment of imaging parameters in a dental imaging system
WO2015118120A1 (fr) 2014-02-07 2015-08-13 3Shape A/S Détection d'une nuance de dent
WO2015151098A2 (fr) * 2014-04-02 2015-10-08 M.S.T. Medical Surgery Technologies Ltd. Laparoscope articulé dont la structure utilise la lumière
US11419583B2 (en) * 2014-05-16 2022-08-23 Koninklijke Philips N.V. Reconstruction-free automatic multi-modality ultrasound registration
US10772506B2 (en) 2014-07-07 2020-09-15 Align Technology, Inc. Apparatus for dental confocal imaging
US9675430B2 (en) 2014-08-15 2017-06-13 Align Technology, Inc. Confocal imaging apparatus with curved focal surface
JP6483832B2 (ja) * 2014-08-29 2019-03-13 トヨタ モーター ヨーロッパ Ｒｇｂ−ｄセンサを使用して物体を走査する方法とシステム
US9610141B2 (en) 2014-09-19 2017-04-04 Align Technology, Inc. Arch expanding appliance
US10449016B2 (en) 2014-09-19 2019-10-22 Align Technology, Inc. Arch adjustment appliance
US9744001B2 (en) 2014-11-13 2017-08-29 Align Technology, Inc. Dental appliance with cavity for an unerupted or erupting tooth
US10504386B2 (en) 2015-01-27 2019-12-10 Align Technology, Inc. Training method and system for oral-cavity-imaging-and-modeling equipment
US9684970B2 (en) * 2015-02-27 2017-06-20 Qualcomm Incorporated Fast adaptive estimation of motion blur for coherent rendering
KR102372087B1 (ko) * 2015-10-28 2022-03-08 삼성전자주식회사 깊이 영상 촬영장치 및 방법
US11931222B2 (en) 2015-11-12 2024-03-19 Align Technology, Inc. Dental attachment formation structures
US11554000B2 (en) 2015-11-12 2023-01-17 Align Technology, Inc. Dental attachment formation structure
US11596502B2 (en) 2015-12-09 2023-03-07 Align Technology, Inc. Dental attachment placement structure
US11103330B2 (en) 2015-12-09 2021-08-31 Align Technology, Inc. Dental attachment placement structure
CN105471884B (zh) * 2015-12-21 2019-05-31 联想(北京)有限公司 一种认证方法、服务器
CA2958766C (fr) * 2016-02-25 2018-01-02 Synaptive Medical (Barbados) Inc. Systeme et methode d'acquisition de carte en profondeur fondee sur la portee
EP3988048B1 (fr) 2016-06-17 2024-01-17 Align Technology, Inc. Dispositif de surveillance de performances d'un appareil orthodontique
EP3471599A4 (fr) 2016-06-17 2020-01-08 Align Technology, Inc. Appareils intraoraux avec détection
KR102595753B1 (ko) 2016-07-27 2023-10-30 얼라인 테크널러지, 인크. 치아 진단 기능이 있는 구강 내 스캐너
US10390913B2 (en) 2018-01-26 2019-08-27 Align Technology, Inc. Diagnostic intraoral scanning
US10507087B2 (en) 2016-07-27 2019-12-17 Align Technology, Inc. Methods and apparatuses for forming a three-dimensional volumetric model of a subject's teeth
CN113648088B (zh) 2016-11-04 2023-08-22 阿莱恩技术有限公司 用于牙齿图像的方法和装置
WO2018102702A1 (fr) 2016-12-02 2018-06-07 Align Technology, Inc. Caractéristiques d'un appareil dentaire permettant l'amélioration de la parole
US11376101B2 (en) 2016-12-02 2022-07-05 Align Technology, Inc. Force control, stop mechanism, regulating structure of removable arch adjustment appliance
ES2845198T3 (es) 2016-12-02 2021-07-26 Align Technology Inc Expansor palatino
CN113440273A (zh) 2016-12-02 2021-09-28 阿莱恩技术有限公司 一系列腭扩张器及其形成方法和形成设备
US10548700B2 (en) 2016-12-16 2020-02-04 Align Technology, Inc. Dental appliance etch template
US10456043B2 (en) * 2017-01-12 2019-10-29 Align Technology, Inc. Compact confocal dental scanning apparatus
US10779718B2 (en) 2017-02-13 2020-09-22 Align Technology, Inc. Cheek retractor and mobile device holder
US10613515B2 (en) 2017-03-31 2020-04-07 Align Technology, Inc. Orthodontic appliances including at least partially un-erupted teeth and method of forming them
US11045283B2 (en) 2017-06-09 2021-06-29 Align Technology, Inc. Palatal expander with skeletal anchorage devices
CN110769777B (zh) 2017-06-16 2023-08-11 阿莱恩技术有限公司 牙齿类型和萌出状态的自动检测
WO2019005808A1 (fr) 2017-06-26 2019-01-03 Align Technology, Inc. Indicateur de performance de biocapteur pour appareils intrabuccaux
US10885521B2 (en) 2017-07-17 2021-01-05 Align Technology, Inc. Method and apparatuses for interactive ordering of dental aligners
WO2019018784A1 (fr) 2017-07-21 2019-01-24 Align Technology, Inc. Ancrage de contour palatin
EP3658067B1 (fr) 2017-07-27 2023-10-25 Align Technology, Inc. Système et procédés de traitement d'un aligneur orthodontique au moyen d'une tomographie par cohérence optique
EP3658070A1 (fr) 2017-07-27 2020-06-03 Align Technology, Inc. Teinte, transparence et émaillage dentaire
US11116605B2 (en) 2017-08-15 2021-09-14 Align Technology, Inc. Buccal corridor assessment and computation
US11123156B2 (en) 2017-08-17 2021-09-21 Align Technology, Inc. Dental appliance compliance monitoring
TWI630525B (zh) * 2017-09-05 2018-07-21 友達光電股份有限公司 非嵌入式顯示觸控裝置及其觸控偵測方法
US10813720B2 (en) 2017-10-05 2020-10-27 Align Technology, Inc. Interproximal reduction templates
CN114939001A (zh) 2017-10-27 2022-08-26 阿莱恩技术有限公司 替代咬合调整结构
CN116602778A (zh) 2017-10-31 2023-08-18 阿莱恩技术有限公司 具有选择性牙合负荷和受控牙尖交错的牙科器具
WO2019089989A2 (fr) 2017-11-01 2019-05-09 Align Technology, Inc. Planification automatique de traitement
US11534974B2 (en) 2017-11-17 2022-12-27 Align Technology, Inc. Customized fabrication of orthodontic retainers based on patient anatomy
EP3716885B1 (fr) 2017-11-30 2023-08-30 Align Technology, Inc. Appareils buccaux orthodontiques comprenant des capteurs
WO2019118876A1 (fr) 2017-12-15 2019-06-20 Align Technology, Inc. Méthodes et appareils de traitement orthodontique adaptatif à boucle fermée
US10980613B2 (en) 2017-12-29 2021-04-20 Align Technology, Inc. Augmented reality enhancements for dental practitioners
US11937991B2 (en) 2018-03-27 2024-03-26 Align Technology, Inc. Dental attachment placement structure
EP3773320B1 (fr) 2018-04-11 2024-05-15 Align Technology, Inc. Appareils d'expansion palatine libérables
US20220079683A1 (en) * 2018-06-30 2022-03-17 Koninklijke Philips N.V. Registering optical shape sensing device with three-dimensional representation of region of interest
DE102018213142A1 (de) * 2018-08-06 2020-02-06 Carl Zeiss Industrielle Messtechnik Gmbh Anordnung und Verfahren zum Erfassen einer an einem Objekt angeordneten Markeranordnung
DK3628271T3 (da) * 2018-09-27 2021-09-06 Sirona Dental Systems Gmbh Apparat til ændring af et fokuspunkt for et optisk system i en dental 3d-scanner og dental 3d-scanner
EP4062865A2 (fr) * 2019-12-26 2022-09-28 Medit Corp. Procédé d'alignement par données groupées
CN111383332B (zh) * 2020-03-26 2023-10-13 深圳市菲森科技有限公司 一种三维扫描和重建系统、计算机设备和可读存储介质
DE102021130529B3 (de) * 2021-11-22 2023-03-16 Karl Storz Se & Co. Kg Bildgebendes Verfahren zum Abbilden einer Szene und System
CN114529652B (zh) * 2022-04-24 2022-07-19 深圳思谋信息科技有限公司 点云补偿方法、装置、设备和存储介质

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100303341A1 (en) * 2009-06-01 2010-12-02 Haeusler Gerd Method and device for three-dimensional surface detection with a dynamic reference frame

Family Cites Families (33)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2523369B2 (ja) * 1989-03-14 1996-08-07 国際電信電話株式会社 動画像の動き検出方法及びその装置
JP3869876B2 (ja) * 1995-12-19 2007-01-17 キヤノン株式会社 画像計測方法及び画像計測装置
IL125659A (en) * 1998-08-05 2002-09-12 Cadent Ltd Method and device for three-dimensional simulation of a structure
US7068825B2 (en) * 1999-03-08 2006-06-27 Orametrix, Inc. Scanning system and calibration method for capturing precise three-dimensional information of objects
US6648640B2 (en) 1999-11-30 2003-11-18 Ora Metrix, Inc. Interactive orthodontic care system based on intra-oral scanning of teeth
US7013191B2 (en) 1999-11-30 2006-03-14 Orametrix, Inc. Interactive orthodontic care system based on intra-oral scanning of teeth
JP4389371B2 (ja) * 2000-09-28 2009-12-24 株式会社ニコン 画像修復装置および画像修復方法
US7110490B2 (en) * 2002-12-10 2006-09-19 General Electric Company Full field digital tomosynthesis method and apparatus
US7004754B2 (en) * 2003-07-23 2006-02-28 Orametrix, Inc. Automatic crown and gingiva detection from three-dimensional virtual model of teeth
US7711179B2 (en) * 2004-04-21 2010-05-04 Nextengine, Inc. Hand held portable three dimensional scanner
ATE383817T1 (de) * 2004-06-17 2008-02-15 Cadent Ltd Verfahren zum bereitstellen von daten im zusammenhang mit der mundhöhle
US8837026B2 (en) * 2004-07-23 2014-09-16 3Shape A/S Adaptive 3D scanning
US7657081B2 (en) * 2004-09-03 2010-02-02 National Research Council Of Canada Recursive 3D model optimization
FI20045445A0 (fi) * 2004-11-18 2004-11-18 Nokia Corp Menetelmä, laitteisto, ohjelmisto ja järjestely kuvadatan muokkaamiseksi
US7494338B2 (en) * 2005-01-11 2009-02-24 Duane Durbin 3D dental scanner
US20060241371A1 (en) * 2005-02-08 2006-10-26 Canesta, Inc. Method and system to correct motion blur in time-of-flight sensor systems
EP1869403B1 (fr) * 2005-03-03 2017-06-14 Align Technology, Inc. Systeme et procede pour l'exploration d'une cavite intrabuccale
US7336763B2 (en) * 2005-05-02 2008-02-26 Oy Ajat Ltd Dental extra-oral x-ray imaging system and method
US7676022B2 (en) 2005-05-02 2010-03-09 Oy Ajat Ltd. Extra-oral digital panoramic dental x-ray imaging system
US7609875B2 (en) * 2005-05-27 2009-10-27 Orametrix, Inc. Scanner system and method for mapping surface of three-dimensional object
US7417213B2 (en) * 2005-06-22 2008-08-26 Tripath Imaging, Inc. Apparatus and method for rapid microscopic image focusing having a movable objective
US8035637B2 (en) * 2006-01-20 2011-10-11 3M Innovative Properties Company Three-dimensional scan recovery
JP2010533008A (ja) * 2007-06-29 2010-10-21 スリーエム イノベイティブ プロパティズ カンパニー ビデオデータ及び三次元モデルデータの同期ビュー
US7982776B2 (en) * 2007-07-13 2011-07-19 Ethicon Endo-Surgery, Inc. SBI motion artifact removal apparatus and method
WO2009089127A1 (fr) * 2008-01-04 2009-07-16 3M Innovative Properties Company Signatures d'image pour une utilisation dans une reconstruction tridimensionnelle à base de mouvement
US8199988B2 (en) * 2008-05-16 2012-06-12 Geodigm Corporation Method and apparatus for combining 3D dental scans with other 3D data sets
EP2521095B1 (fr) * 2008-07-14 2015-03-04 Cefla S.C. Correction d'erreur dynamique en imagerie radiographique
WO2010048960A1 (fr) 2008-10-28 2010-05-06 3Shape A/S Analyseur optique à balayage avec commande à rétroaction
EP2430826A1 (fr) * 2009-05-12 2012-03-21 Koninklijke Philips Electronics N.V. Appareil photo, système comprenant un appareil photo, procédé de fonctionnement d'un appareil photo et procédé de déconvolution d'une image enregistrée
US8134719B2 (en) * 2010-03-19 2012-03-13 Carestream Health, Inc. 3-D imaging using telecentric defocus
US8045046B1 (en) * 2010-04-13 2011-10-25 Sony Corporation Four-dimensional polynomial model for depth estimation based on two-picture matching
US8358833B2 (en) * 2010-04-27 2013-01-22 Indian Institute Of Technology Madras Recovering 3D structure using blur and parallax
EP2654606B1 (fr) 2010-12-21 2021-05-26 3Shape A/S Compensation de flou de mouvement

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100303341A1 (en) * 2009-06-01 2010-12-02 Haeusler Gerd Method and device for three-dimensional surface detection with a dynamic reference frame

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
BANNO A ET AL: "Motion Estimation of a Moving Range Sensor by Image Sequences and Distorted Range Data", 2005 IEEE/RSJ INTERNATIONAL CONFERENCE ON INTELLIGENT ROBOTS AND SYSTEMS; EDMONTON, AB, CANADA; 2 - 6 AUGUST 2005, IEEE, PISCATAWAY, NJ, USA, 2 August 2005 (2005-08-02), pages 4043 - 4048, XP010857694, ISBN: 978-0-7803-8912-0, DOI: 10.1109/IROS.2005.1545600 *
SREENIVASA K MADA ET AL: "Overview of passive and active vision techniques for hand-held 3D data acquistion", OPTOMECHATRONIC MICRO/NANO DEVICES AND COMPONENTS III : 8 - 10 OCTOBER 2007, LAUSANNE, SWITZER; [PROCEEDINGS OF SPIE , ISSN 0277-786X], SPIE, BELLINGHAM, WASH, vol. 4877, 1 January 2003 (2003-01-01), pages 16 - 27, XP009150756, ISBN: 978-1-62841-730-2, DOI: 10.1117/12.463773 *
YUNSU BOK ET AL: "Accurate Motion Estimation and High-Precision 3D Reconstruction by Sensor Fusion", 2007 IEEE INTERNATIONAL CONFERENCE ON ROBOTICS AND AUTOMATION - 10-14 APRIL 2007 - ROMA, ITALY, IEEE, PISCATAWAY, NJ, USA, 10 April 2007 (2007-04-10), pages 4721 - 4726, XP031389531, ISBN: 978-1-4244-0601-2 *

Also Published As

Publication number Publication date
EP2654606A4 (fr) 2017-09-27
US10218955B2 (en) 2019-02-26
US10735706B2 (en) 2020-08-04
US20190191141A1 (en) 2019-06-20
CN103491897A (zh) 2014-01-01
US20140022352A1 (en) 2014-01-23
EP2654606A1 (fr) 2013-10-30
US20210160471A1 (en) 2021-05-27
WO2012083968A1 (fr) 2012-06-28
EP2654606B1 (fr) 2021-05-26
US11825062B2 (en) 2023-11-21
CN103491897B (zh) 2016-01-20
ES2884700T3 (es) 2021-12-10

Similar Documents

Publication Publication Date Title
US11825062B2 (en) Motion blur compensation
US10064553B2 (en) Detection of a movable object when 3D scanning a rigid object
US11321817B2 (en) Motion compensation in a three dimensional scan
US8134719B2 (en) 3-D imaging using telecentric defocus
AU2010262191B2 (en) Focus scanning apparatus
KR101605386B1 (ko) 측정 물체 표면 위에서 3d 좌표들을 결정하기 위한 광학 측정 방법 및 측정 시스템
US20170307363A1 (en) 3d scanner using merged partial images
TW488145B (en) Three-dimensional profile scanning system
JP2004309318A (ja) 位置検出方法、その装置及びそのプログラム、並びに、較正情報生成方法
TW202400965A (zh) 表面形狀測定裝置及表面形狀測定方法

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN PUBLISHED

AC Divisional application: reference to earlier application

Ref document number: 2654606

Country of ref document: EP

Kind code of ref document: P

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

B565 Issuance of search results under rule 164(2) epc

Effective date: 20211028

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20220601

RBV Designated contracting states (corrected)

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20230720

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN

18D Application deemed to be withdrawn

Effective date: 20231201