WO2013185936A1 - Apparatus and method for estimating a property of a surface using speckle imaging - Google Patents

Abstract

An apparatus for estimating a property of a surface of an object comprises a coherent light source (105) which generates at least one light spot on the surface. A camera (111) captures the out-of-focus image of at least one part of the surface which comprises the light spot. The out-of-focus image comprises a light spot image object for the light spot with this image object having a speckle pattern. An analyzer (113) is coupled to the camera and estimates the property of the surface in response to the speckle pattern. The camera uses a rolling shutter for capturing the out-of-focus image. The analyzer (113) may determine a temporal property of the surface in response to a spatial parameter variation of the speckle pattern in the out-of-focus image. Specifically, a relative movement can be determined from a spatial analysis of a single out-of-focus image.

Description

Apparatus and method for estimating a property of a surface using speckle imaging

FIELD OF THE INVENTION

The invention relates to estimating a property of a surface of an object using speckle imaging, and in particular, but not exclusively to detection of a relative movement of the surface based on variations in a speckle pattern.

BACKGROUND OF THE INVENTION

When illuminating a rough surface with coherent light, minute path length differences in the reflected field result in interference/speckle patterns that can be observed by a defocused camera. The coherent light is typically generated by a laser light source. A speckle pattern may be thought of as a random intensity pattern produced by the mutual interference of a set of wave-fronts. Analysis of these patterns and their dynamic behavior allows for high-precision detection of, for example, target translation and rotation, flow parameters and material characterization. Over the years, speckle imaging has found diverse applications in industrial metrology, medical applications, material characterization, vital signs analysis, blood flow measurements, measurements of small displacements and many more.

Laser-speckle imaging enables distant, contactless measurement of very small surface motion, such as induced by sound or by vital signs (heart-beat, respiration), or of very distant motion such as a handheld remote interaction device (game controller, pointing device).

One type of speckle imaging is full field speckle contrast imaging where the temporal variance of the laser speckle induced by flows is of interest. In this technique, a laser beam is expanded or defocused while the imaging camera is arranged to keep the reflecting surface in focus. This approach may for example allow imaging of blood flow and perfusion near a surface.

Another approach to speckle imaging is to use a laser which is focused on a surface to generate a small spot on the surface. An image of the spot is captured using an imaging objective which is defocused. Defocusing of the camera results in a„circle of confusion" or„blur circle". Due to the coherent nature of the light from the laser, this circle is not uniform in intensity, but rather contains a speckle pattern caused by interference between different wave-fronts. The speckle pattern is dependent on the surface which reflects the laser light. In particular, the roughness and small variations in the surface texture result in varying phase dependencies of reflected wave-fronts which result in the interference speckle pattern. Furthermore, small movements of the object surface will be visible in the speckle pattern as translations. A particular advantage of speckle imaging is that the object motion is highly amplified in the translation of the speckle pattern thereby making it practical to detect even very small movements. In practice, even a small change in the position or orientation of a laser-illuminated surface gives rise to large displacements of the associated speckle field. In addition, if the motion contains temporally high- frequent variations, the associated speckle field will exhibit the same temporal frequency characteristics.

These characteristics have for example been used by to measure heart beats and speech at a large distance (several meters or more) by use of a collimated laser and a defocused camera as disclosed in Zeev Zalevsky, Yevgeny Beiderman, Israel Margalit, Shimshon Gingold, Mina Teicher, Vicente Mico, and Javier Garcia, "Simultaneous remote extraction of multiple speech sources and heart beats from secondary speckles pattern", Optics Express, Vol. 17, Issue 24, pp. 21566-21580, 2009.

However, although speckle imaging provides a number of advantages and options, it also has some associated disadvantages. Indeed, it is necessary to capture a sufficiently accurate image of the speckle pattern which means that the resolution of the camera must be relatively high. Also, in order to capture the high frequency variations, the temporal resolution must be sufficiently high. Thus, there is an inherent trade-off between the accuracy and temporal resolution of the speckle pattern analysis and the resolution and speed of the camera. In many practical applications this results in a need for very expensive high speed cameras, and often simultaneously having high resolution. Also, the speckle pattern analysis may often be relatively complex and resource demanding and may not be as accurate or reliable as desired.

Hence, an improved speckle pattern system would be advantageous and in particular a system allowing for increased flexibility, reduced resource demand, reduced cost, facilitated implementation, reduced complexity, relaxed camera requirements and/or improved performance would be advantageous. SUMMARY OF THE INVENTION

Accordingly, the invention seeks to preferably mitigate, alleviate or eliminate one or more of the above mentioned disadvantages singly or in any combination. The invention is defined by the independent claims. Advantageous embodiments are defined in the dependent claims.

According to an aspect of the invention there is provided an apparatus for estimating a property of a surface of an object, the apparatus comprising: a coherent light source for generating at least one light spot on the surface; a camera for capturing an out-of- focus image of at least a part of the surface comprising the light spot, the out-of- focus image comprising a light spot image object for the light spot, and the light spot image object having a speckle pattern; an analyzer for estimating the property of the surface in response to the speckle pattern; wherein the camera comprises a rolling shutter for capturing the out-of- focus image.

The invention may in particular allow improved, facilitated, and/or reduced complexity speckle based sensing of a surface. In particular, the invention may in many embodiments allow improved trade-off between temporal resolution and camera

requirements. Specifically, in many embodiments the need for high-speed cameras can be mitigated or obviated while still achieving the high temporal resolution associated with such cameras. The approach may allow determination of high frequency variations in the property of the surface without requiring high frame rate cameras. The approach may reduce implementation cost very substantially as the cost reduction associated with e.g. normal frame rate cameras relative to high frame rate cameras is very substantial. Furthermore, in many embodiments simplified or improved processing can be achieved allowing an improved performance versus complexity and resource usage trade-off. In many embodiments, the approach may allow temporal characteristics to be determined from a single out-of- focus image.

The property may specifically be a movement property of the surface. In many embodiments, the property will be a relative property, such as a change in position rather than an absolute position. The coherent light source may in many embodiments be a laser light source arranged to generate a laser light spot on the surface. The coherent light source may be focused on the surface to provide a sufficiently small light spot, typically with an area of less than 2 mm², and often preferably less than 1 mm² or even 1 mm². The apparatus may in particular be arranged to generate a relatively coarse speckle pattern, and in particular the coherent light source may generate the light spot to be sufficiently small to result in an average speckle grain size of at least 10 square pixels.

The camera may be a lens-less camera.

In some embodiments the apparatus may include functionality for capturing a plurality of out-of-focus images comprising a light spot image object for the light spot where the rolling shutter propagation direction is different for the different images. Specifically, the apparatus may comprise a plurality of cameras arranged to have different rolling shutter propagation directions. This may facilitate determination of e.g. motion of the surface in two directions.

In accordance with an optional feature of the invention, the analyzer is arranged to determine a temporal property of the surface in response to a spatial variation of a property of the speckle pattern in the out-of-focus image. This may provide improved performance, more reliable and/or facilitated analysis. Furthermore, it may reduce the requirements for the camera, and may in particular avoid the need for expensive high speed cameras. The rolling shutter effect may be used to convert temporal characteristics of the surface into spatial characteristics of the resulting speckle pattern and this may further be analyzed to provide an estimate for a property of the surface. In particular, a temporal resolution which exceeds the temporal resolution of the camera used to capture the out-of- focus image can be achieved.

In accordance with an optional feature of the invention, the analyzer is arranged to determine the temporal property in response to a spatial pattern variation of the speckle pattern in a direction corresponding to the rolling shutter propagation direction. This may in particular allow improved determination of temporal characteristics and may allow for an efficient and/or improved performance analysis.

In accordance with an optional feature of the invention, the analyzer is arranged to determine a temporal property of the surface in response to a spatial frequency analysis of the speckle pattern. This may in many embodiments allow for a facilitated and/or improved analysis. In particular, a spatial frequency transform may be applied to the speckle pattern and the property may be determined in response to frequency domain property.

Specifically, a two-dimensional movement may be determined in response to a two- dimensional intensity distribution in the two dimensional spatial frequency domain.

The spatial frequency analysis may in particular provide an efficient approach for detecting or analyzing effects of an asymmetry in the spectral pattern caused by the temporal effect introduced in the speckle pattern by the rolling shutter. This may in particular facilitate detection or analysis of movement of the surface.

In accordance with an optional feature of the invention, the rolling shutter is arranged to capture the out-of- focus image line sequentially; and the analyzer is arranged to determine a speckle pattern property for each group of a plurality of groups which each comprise at least part of a number of adjacent lines of the out-of-focus image, and to determine the property in response to a comparison of the speckle pattern property for different groups. This may in many embodiments allow for a facilitated and/or improved analysis. In particular, it may allow for low complexity analysis to provide accurate estimation of a property of the surface, such as specifically a movement property.

Each group may specifically comprise one line of the out-of-focus image. The line sequential operation of the rolling shutter may comprise a time sequential capturing of lines Thus, the capturing of the image may be achieved in a plurality of sequential time intervals wherein only a subset of lines are captured in each time interval (often a single line). The rolling shutter will thus have a propagation direction perpendicularly to the line direction. Depending on the orientation of the rolling shutter, the lines may typically be considered to correspond to rows or columns of the image/ image sensor.

In accordance with an optional feature of the invention, the rolling shutter is arranged to capture the out-of-focus image line sequentially; and the analyzer is arranged to determine the property in response to a comparison of speckle patterns for sequential lines. This may in many embodiments allow for a facilitated and/or improved analysis. In particular, it may allow for low complexity analysis to provide accurate estimation of a property of the surface, such as specifically a movement property.

Each group may specifically comprise one line of the out-of-focus image. The line sequential operation of the rolling shutter may comprise a time sequential capturing of lines. Thus, the capturing of the image may be achieved in a plurality of sequential time intervals wherein only a subset of lines are captured in each time interval (often a single line). The rolling shutter will thus have a propagation direction perpendicularly to the line direction. Depending on the orientation of the rolling shutter, the lines may typically be considered to correspond to rows or columns of the image/image sensor.

In accordance with an optional feature of the invention, the analyzer is arranged to determine the property in response to a comparison between the speckle pattern and a reference speckle pattern. This may in many embodiments allow for a facilitated and/or improved analysis. In particular, it may allow for a low complexity analysis to provide accurate estimation of a property of the surface, such as specifically a movement property. In particular, it may allow a finer speckle pattern with no or less correlation between different lines to be used.

The reference image may specifically be another image of the speckle pattern of the surface, such as e.g. a time averaged image or an image taken when specific conditions are known to exist, such as when the surface is known to not be moving.

In accordance with an optional feature of the invention, the analyzer is arranged to determine a two dimensional movement of the surface in response to a detection of a speckle pattern transformation, the transformation being asymmetric between a first direction corresponding to the rolling shutter propagation direction and a second direction not corresponding to the rolling shutter propagation direction. This may in many embodiments allow for a facilitated and/or improved analysis. In particular, it may allow for low

complexity analysis to provide accurate estimation of a two-dimensional movement of the surface. The pattern transformation may specifically be a pattern deformation and the deformation may specifically include a scaling which is asymmetric between the first and second direction. The transformation/ deformation may be evaluated by comparing a speckle pattern of a previous or reference out-of- focus image to the speckle pattern of the (current) out-of-focus image. The analyzer may specifically determine a scaling between speckle patterns for the first and second direction, and determine a two-dimensional movement from the scaling factors.

The deformation may in particular be determined by a speckle pattern matching between two out-of-focus images, the matching determining parameters for an asymmetric transformation of one speckle pattern such that the transformed pattern matches the second speckle pattern. The asymmetric transformation reflects the asymmetry between the effect of motion in the rolling shutter propagation direction and in the other direction. The two-dimensional movement may be determined from the determined transformation parameters.

In accordance with an optional feature of the invention, the apparatus is arranged to generate the speckle pattern as an anisotropic speckle pattern. This may facilitate and/or improve the analysis. In particular, the use of an anisotropic speckle pattern may provide an improved speckle pattern for a rolling shutter based analysis.

In some embodiments, the coherent light source may be arranged to generate the first light spot as an anisotropic light spot. This may result in an anisotropic speckle pattern. In some embodiments, the coherent light source may be arranged to generate the first light spot as a substantially isotropic light spot, and the camera may be arranged to generate the speckle pattern for the isotropic light spot as an anisotropic speckle pattern. This may be achieved by using anisotropic imaging optics and/or sensors.

In some embodiments, the apparatus is arranged to generate the light spot image object to have an anisotropic speckle pattern. An average speckle dimension may be different in a first direction corresponding to the rolling shutter propagation direction relative to the average speckle dimension in a second direction perpendicular to the first direction. The difference may be no less than a factor of two, or even four, in many embodiments.

In accordance with an optional feature of the invention, an average extension of speckle grains in a first direction corresponding to the rolling shutter propagation direction is at least twice that of an average extension of speckle grains in a second direction perpendicular to the first direction.

This may provide an improved speckle pattern for a rolling shutter based analysis. For example, it may allow high resolution within each line captured by the rolling shutter, while providing a high correlation between different lines thereby facilitating or improving e.g. the speckle contrast for each line while allowing time dependent differences to be more easily detected as differences between lines.

In accordance with an optional feature of the invention, the coherent light source is arranged to generate a plurality of light spots on the surface, and the camera is arranged to capture the plurality of light spots in the out-of- focus image; and further comprising: a selector arranged to select a subset of light spots for analysis by the analyzer. This may facilitate and improve operation for many applications and may in many embodiments provide an improved determination of the surface property. Specifically, the approach may allow one or more light spots in particularly suitable positions on the surface to be detected and used for the analysis. The approach may in particular allow for the speckle pattern analysis to be used more flexibly and in applications with higher levels of variability or uncertainty, not least in terms of the positioning of the object with the surface to be analyzed. For example, if used for detecting movement of the surface of a patient, the approach may allow a suitable surface area to be determined without requiring the patient to be positioned with extreme accuracy. Improved analysis may often be possible as the light spot resulting in the speckle pattern with the best characteristics (e.g. speckle pattern contrast) can be used. The subset of light spots comprises the first light spot, and indeed in some embodiments the subset may consist of the first light spot. In some embodiments, the selector is arranged to select the first light spot from the plurality of light spots.

The plurality of light spots may form a regular or irregular grid of light spots. The light spots may preferably be arranged to be non-overlapping in the out-of- focus image. The coherent light source may comprise a plurality of light sources, such as a laser for each light spot.

In accordance with an optional feature of the invention, the selector is arranged to select the subset of light spots using a lower processing resolution than used by the analyzer when determining the property. This may reduce complexity and/or resource usage yet may provide a reliable and high performance detection of suitable light spots for the analysis.

In accordance with an optional feature of the invention, the selector is arranged to select the subset in response to at least one of: an intensity for light spots of the plurality of the light spots; a speckle contrast for light spots of the plurality of the light spots; a speckle pattern variation for light spots of the plurality of the light spots; a correlation between variations for different light spots of the plurality of the light spots; and a change in a light spot pattern of the plurality of the light spots. This may provide particularly advantageous selection of a subset of light spots for use in the speckle pattern analysis. In particular, it may in many embodiments result in an improved determination of the surface property as it may allow light spots with particularly suitable speckle patterns to be used.

In accordance with an optional feature of the invention, the selector is arranged to select the subset in response to a non-speckle pattern image. This may provide improved selection of the subset in many embodiments. For example, the selection may be based on a normal in focus image of the surface or area in which the object having the surface is placed. The (normal) in focus image may allow the selector to select light spots which are placed on the surface.

According to an aspect of the invention there is provided a method of estimating a property of a surface of an object, the method comprising: generating at least one light spot on the surface using a coherent light source; capturing an out-of-focus image of at least a part of the surface comprising the light spot, the out-of-focus image comprising a light spot image object for the light spot, and the light spot image object having a speckle pattern; estimating the property of the surface in response to the speckle pattern; and capturing the out-of-focus image using a rolling shutter. These and other aspects, features and advantages of the invention will be apparent from and elucidated with reference to the embodiment(s) described hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention will be described, by way of example only, with reference to the drawings, in which

FIG. 1 illustrates an example of a speckle imaging apparatus in accordance with some embodiments of the invention;

FIGs. 2- 6 illustrate examples of speckle patterns;

FIG. 7 illustrates an example of an edge enhanced speckle pattern;

FIGs. 8 and 9 illustrate an example of an estimated motion for a surface; FIG. 10 illustrates an example of a speckle pattern;

FIG. 11 illustrates an example of spatial frequency transform of the speckle pattern of FIG. 10;

FIG. 12 illustrates an example of a thresholded spatial frequency transform of the speckle pattern of FIG. 10;

FIG. 13 illustrates examples of pattern transformations for rolling shutter images;

FIG. 14 illustrates examples of speckle patterns for isotropic and anisotropic speckle patterns;

FIG. 15 illustrates an example of a speckle imaging apparatus in accordance with some embodiments of the invention; and

FIGs. 16 to 18 illustrate examples of speckle pattern imaging using a grid of light spots.

DETAILED DESCRIPTION OF SOME EMBODIMENTS OF THE INVENTION

FIG. 1 illustrates an example of a setup for estimating a property of a surface. The setup comprises a speckle imaging apparatus 101 which is arranged to estimate a characteristic of a surface 103 of an object, such as specifically a relative movement of the surface 103.

The speckle imaging apparatus 101 comprises a coherent light source 105 which is arranged to generate at least one light spot on the surface 103. In the specific example, the coherent light source is a laser light source, and it comprises a laser 107 and a lens 109 which is capable of focusing the laser light source on the surface 103 such that a sufficiently small light spot is generated on the surface 103.

The speckle imaging apparatus 101 further comprises a camera 111 which is arranged to capture an image of (part of) the surface 103 including the light spot generated by the coherent light source 105. The camera 111 is arranged such that it captures an out-of- focus image of the surface 103, i.e. the camera is arranged to capture the image with a focal plane that differs from the surface 130. Thus, the focus distance for the camera differs from the distance from the surface 103 to the camera 111 when the apparatus is in use. In some embodiments, the camera may be a camera without any focusing lens. Indeed, a lens-less camera corresponding to a bare sensor may be used in some embodiments. Indeed, such a camera can be considered a special case of unfocused imaging with the focus distance being on the sensor itself.

The camera 111 is accordingly arranged to have a focus distance that is different from the distance from the camera 111 to an operating distance range in which the object may be positioned. It will be appreciated that the specific distances involved, the positioning of the surface etc. may depend on the individual application etc. Furthermore, it will be appreciated that the actual design and implementation of the speckle imaging apparatus 101 does not rely on the surface 103 being present or at a specific position. Rather, the speckle imaging apparatus 101 may be designed for the surface 103 to be positioned within a given operating volume/ distance range. The coherent light source 105 and the camera 111 may then be arranged to provide acceptable performance when an object is positioned with the surface to be monitored within this operating volume/ distance range.

Typically, the focus distance of the camera 111 will be at least twice, and often at least five times, the maximum distance of the operating distance interval. The operating distance interval is the interval for which the speckle imaging apparatus 101 has been designed, i.e. it is the range in which the surface 103 should be placed when the apparatus is in use.

It will be appreciated that in some embodiments, the speckle imaging apparatus 101 may be arranged to manually or automatically adapt to a specific positioning of the surface 103. For example, the focusing of the light from the coherent light source 105 can be manually adjusted by changing the distance between the laser 107 and the lens 109. As another example, the adjustment may be automatic and may be based on a feedback loop which minimizes the spot size of the light spot on the surface 103. Similarly, the focusing of the camera 111 may be manually adjustable or may be automatically adjustable (for example based on a feedback system which maximizes the size of the image object corresponding to the light spot, or which maximizes the speckle pattern contrast). In other embodiments, the focus may be constant. For example, the camera 111 may be set to have an infinite focus distance.

In typical embodiments, the coherent light source 105 will be arranged to provide light spots with a total area of no more than 1 mm², and often advantageously significantly smaller such as no more than 0.5 mm², or even no more than 0.1 mm². Thus, when the surface 103 is within the operating interval, the coherent light source 105 can provide such small light spots (either fixedly or using manual and/or automatic adaptation).

It will be appreciated that the observed speckle size is not only dependent on the light spot size but also depends on other parameters such as observation distance, imaging optics and physical sensor resolution. Typically, it is however more practical to control the light spot size.

In the setup of FIG. 1, the image object of the light spot generated by the coherent light source 105 will comprise a speckle pattern. In particular, the defocusing of the camera creates a 'circle of confusion' or 'blur circle' which contains the interference pattern, known as a speckle pattern. This interference results from the phase variations between different waveform reflections caused by slight variations in the surface. Thus, whereas the incident light is coherent, the surface variations results in the reflected waves having differing phases, and by capturing a defocused image these variations result in an interference pattern. The speckle pattern is dependent on the surface properties (such as the roughness) of the surface from which the laser is reflecting. Furthermore, small movements of the object surface will be visible in the speckle pattern as translations. An important characteristic is that such object motion (in particular translation and out-of-plane rotation) is highly amplified in the translation of the speckle pattern thereby allowing for even small movements to be detected. In the speckle imaging apparatus 101 of FIG. 1, the camera 111 is coupled to an analysis processor 113 which proceeds to determine a property of the surface (such as a movement characteristic) based on the speckle pattern captured by the camera 111, and specifically based on variations in the speckle pattern.

Indeed, although the light spot generated by the coherent light source 105 would appear as a small dot in a focused image of the surface 103, the corresponding image object becomes a relatively large (typically circular area) when the focus is changed from a sharp focus. Although defocused objects acquire the familiar blurred appearance, the coherently illuminated patch causes the corresponding image object to have a distinct and sharp speckle pattern. The size of the speckle pattern is determined by the object distance in relation to the (de)focus distance, which can be infinity. The larger the difference between object distance and focus distance, the larger the area that is filled with the speckle pattern.

The spatial frequency bandwidth of the speckle pattern, which determines the granularity of its appearance, is determined by the size of the illuminated patch. The smaller the illuminated patch, the smaller the spatial frequency bandwidth, and the coarser the speckle grains.

An example of a speckle pattern is illustrated in FIG. 2.

Speckle pattern imaging is very useful for detecting very small movements of the surface. However, in order to measure fast movements, conventional approaches require high speed cameras to be used, and in particular require the frame rate of cameras to be at least as high as the desired temporal resolution for the measurement of the movement. Thus, for fast movements, fast cameras are needed. This typically increases cost very substantially. In addition, the need to analyze speckle patterns for many images tends to result in complex and resource demanding applications.

However, in the speckle imaging apparatus 101 of FIG. 1 the capturing of the out-of-focus image is based on a rolling shutter mechanism. Rather than using a traditional camera wherein the image is created by sensing light in the same time interval for all parts of the image, the camera 1 11 of the speckle imaging apparatus 101 of FIG. 1 uses a time offset sampling of different areas of the image. Thus, the image may be divided into a plurality of areas having capture instants that are offset relative to each other. Accordingly, the sampling instants are not constant for all pixels of the image but rather vary across the image.

As a specific example, the rolling shutter may capture the image in a line sequential manner. Specifically, it may generate the image one line at a time with the sampling/capture instant being offset for each line. Thus, the actual capture instant will increase for each line. In many embodiments, the image may be generated by the camera sampling the outputs of an imaging sensor (such as a charge coupled device CCD sensor). The rolling shutter may specifically result in a line by line capture and may be implemented by each line of the imaging sensor being sampled substantially simultaneously but with there being a time offset between the lines. Thus, the lines may be sampled sequentially, one line at a time (or possibly N lines at the time where N is an integer).

The resulting image will accordingly reflect the surface at slightly different times since each line will correspond to a different time instant. As a consequence, the captured speckle pattern does not just represent characteristics of the surface at one single time instant but also contains temporal information, i.e. it may also reflect how the surface properties vary over time.

The following description will focus on an embodiment wherein a line sequential rolling shutter is implemented. Thus, in the example, the propagation direction of the rolling shutter will be in the perpendicular direction to the line direction. For example, when the rolling shutter reads one row at a time (i.e. a line of the line sequential operation corresponds to a row of pixels of the image sensor), the propagation direction is in the column direction. Similarly, if the rolling shutter reads one column at a time (i.e. a line of the line sequential operation corresponds to a column of pixels of the image sensor), the propagation direction is in the row direction. The following descriptions will focus on examples wherein the rolling shutter reads one horizontal row at a time, and thus where the propagation direction for the rolling shutter is in the vertical direction.

It will be appreciated that in other embodiments, the rolling shutter may read more than one line at a time, or that it may be arranged in other directions. For example, in some embodiments the rolling shutter may have a diagonal propagation direction, and it may thus sample the image sensor in lines that are perpendicular to this diagonal (i.e. parallel to the opposite diagonal for a square sensor. It will be appreciated that the propagation direction corresponds to the direction from the area (e.g. center point) being sampled at a given sample instant to the area (e.g. center point) being sampled at the next sample instant.

The analysis performed by the analysis processor 113 is arranged to determine a property of the surface (and specifically a relative movement) based on an analysis which takes into account the relationship between spatial and temporal characteristics of the captured image. In particular, the camera exploits the Inventors' realization that a rolling shutter introduces a temporal effect to the spatial image properties and that by analyzing the spatial image properties in even a single image, information of the temporal characteristics can be obtained.

Accordingly, the analysis processor 113 may determine a temporal characteristic of the surface based on spatial characteristics for the rolling shutter captured speckle pattern.

The approach may in particular mitigate or obviate the need for high speed cameras. Indeed, a temporal resolution of the determined property which is substantially higher than the image frame rate can be achieved. Indeed, in many applications, a temporal resolution no less than ten times higher than the frame rate of the image can be achieved. Indeed, in many embodiments standard frame rate cameras with a frame rate of less than 50 frames per second can be used to detect movements with frequencies of up to 20 kHz.

Also, the system may reduce complexity and resource demand of the required processing in many embodiments. Indeed, the transformation of temporal characteristics into spatial characteristics of a spatial pattern in a single image may not only reduce the resource demand due to the need to analyze fewer pictures but may in addition allow many low complexity algorithms to be used. In particular, a number of spatial analysis algorithms may be less resource demanding than algorithms based on temporal analyses between different images.

An example of a speckle pattern for a moving surface captured by a camera using a rolling shutter approach is illustrated in FIG.3. As can be seen, the speckle pattern exhibits a spatial pattern variation which reflects how the pattern translates between the different sampling instant. In the specific example, a vibrating motion is introduced to the surface and as can be seen this results in a spatial pattern with translations in the horizontal direction as a function of the vertical position. In the example, the rolling shutter is row sequential and accordingly the vertical direction of the pattern also reflects a temporal dimension. Specifically, the pattern of FIG. 3 exhibits vertical waves corresponding to the sinusoidal vibrations of the surface. The horizontal translations as a function of vertical positions thus provide information of the temporal variation of the surface, and specifically of the movement of the surface.

Other examples of speckle patterns having spatial characteristics reflecting the movement of a surface are illustrated in FIGs. 4 and 5. FIG. 4 illustrates an example wherein the surface has a movement corresponding to a sinus wave with a frequency of 246Hz, and FIG. 5 illustrates an example wherein the surface is subject to a short pulse with associated vibrations.

The analysis processor 113 is arranged to exploit the spatial representation of the temporal variations of the surface to determine motion characteristics. Thus, the analysis processor 113 performs a spatial analysis on the speckle pattern and based on this analysis a temporal variation characteristic is determined for the surface. Furthermore, the analysis processor 113 is arranged to determine the temporal characteristics by analyzing how the spatial speckle pattern varies in the spatial direction reflecting the time variation, i.e. in the direction of propagation of the rolling shutter. Thus, in the specific example, the variation between speckle patterns at different vertical positions is analyzed. It will be appreciated that the analysis processor 113 may use different algorithms for determining the surface property. A number of approaches will be described in the following. However, it should be appreciated that the analysis processor 113 is not limited to these examples but that other approaches may be used in other embodiments dependent on the specific preferences and requirements of the individual embodiment.

In some embodiments, the analysis processor 113 may be arranged to perform a line based analysis to determine the property of the surface. Specifically, the camera may be arranged to sequentially sample a group of adjacent lines at a time followed by the next group of lines etc. Typically, the camera captures one line at a time, but in some

embodiments it may capture N lines at a time where N is any integer. The total image is thus made up from a plurality of groups of adjacent lines captured at different times. The analysis processor 113 may in such embodiments proceed to compare the speckle patterns of different groups, and may specifically proceed to determine an estimated translation in the direction perpendicular to the line direction between the speckle patterns of the different groups. It will be appreciated that in some embodiments or scenarios, each group may only comprise part of each line. For example, the analysis processor 113 may only analyze the part of each line which corresponds to the image object having the speckle pattern.

As a specific example, each group may comprise a single horizontal row. In this case the analysis processor 113 can proceed to compare row speckle patterns of different rows to determine e.g. a relative movement of the surface. Specifically, the analysis processor 113 may correlate adjacent lines to determine an estimated pattern translation. The relative movement of the surface at the mid-time between the two time instants of the two rows may then be calculated from this translation. It will be appreciated that averaging over multiple lines, filtering of generated movement estimations etc. may be applied.

In this approach, the imaged speckle pattern is preferably generated to have relatively coarse speckle grains, and in particular to have speckle grains for which at least 80% of the speckles have an extension in the rolling shutter propagation direction (i.e.

vertically in the specific example) which exceeds two lines (or 2N lines if the shutter captures N lines at a time).

As a specific example, FIG. 6 illustrates part of a speckle pattern for a surface of a piezoelectric sound transducer that is driven by an electric signal from a function generator. The speckle pattern clearly exhibits a wave-shaped distortion which becomes more visible after enhancement of vertical edges, such as e.g. a spatial high pass filtering of pixel intensities. FIG. 7 illustrates the speckle pattern after such an edge enhancement operation. It will be appreciated that the edge enhancement is optional and that the skilled person will be aware of many suitable edge enhancement algorithms. The enhanced image is then used as the input image for a row-by-row cross correlation operation which estimates the

displacement between consecutive rows with sub-pixel accuracy.

Specifically, the displacement, dx, can be estimated using a line cross correlation according to:

dx(y) = arg max_d∑_Vx i_y(x) · i_y+1 (x + d) where y is the line/row number, x in column number, and i indicates the pixel value.

The resulting displacements from applying this approach to the image of FIG.

7 are illustrated in FIG. 8. As can be seen, the estimated displacement directly corresponds to the sine wave motion of the surface.

In many embodiments and applications the specific absolute motion is not required but rather the frequency of the movement, the time of the movement or a direction of a movement may be sufficient. Such values can typically be generated directly from the speckle pattern.

In case specific values are required for the movement, these can be calculated from the time difference between sampling of adjacent lines, the distance to the surface etc. as will be known to the skilled person.

Specifically, the actual displacement/motion may be determined based on a calibration of the system. If the values are to be explicitly calculated this can be achieved by taking into account observation distance, angle of incidence of the laser, angle of the camera, lens and sensor magnification, lens focal distance.

Examples of the calculation of actual object movement values from speckle patterns may be found in the article "Simultaneous remote extraction of multiple speech sources and heart beats from secondary speckles pattern" by Zeev Zalevsky, Yevgeny Beiderman, Israel Margalit, Shimshon Gingold, Mina Teicher, Vicente Mico, and Javier Garcia, Optics Express, Vol. 17, Issue 24, pp. 21566-21580, 2009.

In practice there will often be many unknown and interacting parameters such as surface properties, exact initial orientation etc. Therefore, it will typically be more accurate to use a calibration procedure to learn and model the relation between the measured pixel shift and the physical rotation and/or displacement of the object of study. In a calibration procedure, a motor controlled microstage or another precise actuator could be used for which the actual rotation and displacement are known. Given the input rotation and or displacement, these can be related to the resulting pixel shift, and a linear model (or non-linear) could be applied to model the mapping between physical values and measured values.

As the actuator could be frequency limited, the movements can perhaps only be measured by global frame shifts instead of line shifts as they take more time. Hence in the calibration process, a temporal shift estimation based on frames can be applied, e.g.

measuring the shift of the complete speckle pattern between the first and second state of the actuator.

Typically displacement and rotation are entangled in the measurements. To single out or emphasize one or the other the system setup can be optimized for that property. Specifically, focussing on infinity will emphasize rotation, and conversely focussing closer and minimizing the target camera distance will emphasize the displacement as the effect of rotation increases with distance.

Finally it should be noted that if the temporal behaviour of the object under study is of interest, there might not be a need for calibration as temporal behaviour can be measured accurately, i.e. it can be deduced from the line rate of the camera. This is for example the case in measuring the onset of a given pulse.

In some embodiments, the analysis processor 113 may be arranged to determine the property in response to a comparison between the speckle pattern and a reference speckle pattern. For example, if the speckle pattern is so fine that the line to line correlation becomes too low, the previous exemplary approach will tend to provide unsatisfactory results.

However, in such cases the captured speckle pattern may be compared to a reference speckle pattern. This reference speckle pattern may specifically be a pattern captured for the surface when this is known to be static, or may e.g. be generated by an averaging of a number of speckle patterns. Thus, in this embodiment, the displacement for each line may be estimated by performing the correlation operation on corresponding lines of the captured speckle pattern and of the reference pattern.

FIG. 9 illustrates an example of the result of applying such an analysis to a surface of a piezoelectric sound transducer fed with a sinusoidal drive signal.

In some embodiments, a plurality of images may be captured using rolling shutters with different propagation directions. Such an approach may be implemented in a single camera but may often be implemented by two cameras with different orientations, or at least by two sensors with different orientations. In such embodiments, the motion may be determined independently for the two cameras and then combined. Specifically, a line based correlation approach may be used independently on each image to determine motion in the rolling shutter propagation direction for that image. The two motion vectors can then be combined into a single two dimensional motion vector.

Specifically, the line based motion estimation described with reference to FIGs. 6-9 determines only motion in a direction corresponding to the horizontal motion of the speckle pattern. In order to track motion in two directions, two cameras could be used. By placing a second camera with a 90 degree offset orientation with respect to the first camera, any motion can be tracked as the local motion can be decomposed from the two concurrent motion vectors from each camera signal. It should be noted that there is no restriction with regard to the location of the second camera with respect to the first. E.g. there is no need for a beam splitter in order to share a single optical system. The cameras can be placed at any arbitrary spot, provided that the row orientation of each system differs sufficiently. Again a mutual angle of 90 degrees provides maximal orthogonality of the motion components and may often be preferred.

As another example of an analysis approach, the analysis processor 113 may apply a spatial frequency analysis to (at least part of) the speckle pattern and a temporal characteristic of the surface may be determined based on this analysis.

Specifically the analysis processor 113 may apply a spatial frequency transform to the speckle pattern and may analyze the resulting spatial frequency distribution to determine e.g. a relative movement of the surface.

FIG. 10 illustrates an example of a part of a speckle pattern and FIG. 11 illustrates the result of a two dimensional spatial FFT transformation of this speckle pattern. The analysis processor 113 may proceed to apply a threshold operation to the spatial FFT transformation, e.g. resulting in the distribution illustrated in FIG. 12. Based on the spatial dimensions of the resulting object/ distribution (i.e. of spatial frequencies that are above the threshold), the movement parameters can be estimated.

The approach may specifically take into account that, as the Inventors have realized, when there is no motion, the characteristics of a speckle pattern captured using a rolling shutter is symmetric and thus the speckle pattern will result in a uniform noise pattern with no dominant directionality. However, when there is motion, this will introduce pattern distortions and these distortions will be asymmetric. E.g. in the specific example of a row sequential rolling shutter, different deformations will occur of the pattern in the row and column directions. This asymmetry will be reflected in the frequency transform

representation which accordingly will reflect not only the magnitude of the motion but also the direction. Indeed, any anisotropic distortion will reveal both the direction as well as the magnitude of the observed motion.

In fact the motion induced distortions of the rolling shutter image can be modeled by an affine transformation in the spatial domain. From the coefficients obtained from an estimated affine transformation between the rest state of the system, the motion independent horizontal a vertical motion components can be calculated (as described in more detail later). An affine transformation in the spatial domain introduces a different but still affine transformation in the 2D frequency spectrum of the image. In fact an affine transformation with matrix A in the spatial domain is equal to an affine transformation in the frequency domain with the matrix A^"T and an amplitude correction with the determinant of A as is for example described in the article "Estimating Affine Transformations In The

Frequency Domain" by Lucchese, Proceedings of the International Conference on Image Processing, 2001, vol. 2. Thus from the coefficients obtained from the estimated affine transformation between the rest state and the motion state the motion components can be calculated in a similar way as in the spatial domain.

Another way to obtain the motion from the spectrum distortion measured by its rotation and scaling is by model fitting combined with a calibration phase. By going through a prescribed set of calibration motions and measuring the resulting change in spectrum orientation (determined by its major axis), the length of the major and minor axis, and possibly its bounding box parameters after thresholding, their relations to the object motion can be determined. This can be achieved by fitting a multi parameter model, possibly a linear model, to map the shape parameters of the spectrum to the calibration motions. With the obtained model the motion can then be calculated from the spectrum of the speckle pattern.

An advantage of such an approach is that it may provide a two-dimensional motion estimate whereas the pure line based comparison will provide information of the relative movement in one direction.

In addition, since the speckle granularity is determined by the apparent size of the illuminated patch, the granularity increases as the patch appears smaller observed from a larger camera distance. Consequently the area occupied by of the associated two dimensional spatial- frequency spectrum becomes smaller. As a consequence, the absolute distance can be estimated from the overall size of the spectral footprint of the thresholded object.

In some embodiments, the analysis processor 113 may be arranged to determine (at least) a two dimensional movement of the surface in response to a detection of a speckle pattern deformation. The deformation of the speckle pattern is asymmetric between the direction of the rolling shutter propagation direction and other directions. Specifically, the temporal characteristics of the rolling shutter results in a markedly asymmetry between the propagation direction and (e.g.) a direction perpendicular thereto. The analysis processor 113 may accordingly estimate a deformation of the speckle pattern taking into account the asymmetry, and may from this calculate the two-dimensional motion.

The deformation may be estimated by comparing the speckle pattern of the current image with the speckle pattern of a reference image. The reference speckle pattern may specifically be of a previous image/frame resulting in the estimated motion of the surface being a relative motion with respect to the position of the surface at this instant. As another example, the reference speckle pattern may be one captured when the surface is known to be at rest in a specific position.

Thus, whereas the line based examples are based on a horizontal motion of the speckle pattern resulting in a distinct transformation (translation) of the speckle pattern in the shape of the wave form of the motion, the analysis processor 113 may in some embodiments analyze the pattern deformation/transformation (including translation) in all planar directions, both horizontally and vertically.

Indeed, a horizontal motion in the plane of the image sensor causes the pattern to undergo a shearing operation:

where p = (Px Vy fc)^T aKnd p' = (Vx Vy Y [pixel position, x for columns, y for rows] respectively indicate an original and a displaced position in the image plane and v =

(v_x ^VyY [pixel/rune] is the (local) pattern velocity in the image plane. The symbol T_\ms is the time difference between the sampling/capturing of consecutive rows. In general, T _ne will be equal to the readout time of one single row of the image sensor, which is generally in the order of 100 μβ.

A vertical motion in the plane of the image sensor causes the pattern to undergo a vertical aling operation,

(Note that a velocity with a vertical component of exactly v_y =— 1, should be avoided as it scales the pattern down to a single line. As such, the collection of velocities for which, v = (y_x —iy [pixel/ Jline], can be regarded as singular velocities.) So, provided a velocity does not adopt such a singular value, the local deformation of the pattern follows from a combination of both transformations, such that the displaced position ' is found by

This is illustrated in FIG. 13 which shows velocity-dependent shape transformations for some exemplary velocities for a vertical rolling shutter capture starting with the top row of the output image. The four grids represent a fraction of the output image of a pattern under different velocities of the pattern in the plane of the sensor. A fragment of the pattern is represented by the boxed letter "A". The grid lines symbolize the pixel spacing on the sensor, such that the indicated velocities are in [pixels/ T _ne .

It should be noted that FIG. 13 illustrates only the local deformation of the pattern. In practice, the position of the pattern will also change as a result of the accumulation of velocities prior to the capturing the rows containing this particular pattern fragment. The translation of the pattern fragment itself thus follows from an integral over the

transformations prior to the current row instance.

The analysis processor 113 may use the knowledge of the transformation caused by motion (and the asymmetry between transformation in the direction of the propagation of the rolling shutter and other directions (and specifically the perpendicular direction) to perform a pattern matching. For example, speckle patterns for two consecutive frames may be evaluated. The analysis processor 113 may then apply the transformation of

(pA _ v_{x x}x

to the first speckle pattern for various values of v_x and v_y. For each value, the pattern is compared to the speckle pattern of the subsequent frame (e.g. performing a correlation). The values which provide the highest similarity correspond to the estimated movement components.

It will be appreciated that many other methods for determining parameter values for a transformation exist and may be used without detracting from the invention.

In the previous examples, the light spots generated by the coherent light source 105 have been considered to be circular spots, and the capture of the circular spots has resulted in circular image objects. Such an approach is typically used for speckle imaging. However, in some embodiments the speckle imaging apparatus 101 of FIG. 1 is arranged such that the speckle pattern captured by the image sensor of the camera 111 is anisotropic. Such a non-circular speckle imaging may result in an improved speckle pattern for a rolling shutter based analysis, and may in particular facilitate the analysis of the speckle pattern.

The anisotropic imaging may specifically be achieved by the coherent light source being arranged to generate the light spot on the surface as an anisotropic light spot. Thus, rather than generating a circular light spot, the coherent light source 105 generates a light spot which may e.g. be elliptical.

Indeed, the size of the speckles in the speckle pattern is inversely related to the laser spot size, such that the smaller the spot size, the coarser the speckles. The inventors have realized that this consideration can also be applied to each dimension separately to obtain an anisotropic speckle pattern and that this is particularly beneficial when using a rolling shutter capture. In particular, an anisotropic light spot can be used to increase the vertical correlation of the speckle pattern (i.e. in the rolling shutter propagation direction) while maintaining a fine horizontal displacement resolution (i.e. in the perpendicular direction).

The speckle pattern can for example be controlled by the lens 109 of the speckle imaging apparatus 101 of FIG. 1 being a cylindrical lens. This will result in an elliptical light spot. If the major axis of the light spot ellipse is oriented horizontally (i.e. aligned with a direction perpendicular to the rolling shutter propagation direction, and in the specific example aligned with the image sensor column direction), this will result in speckles which are elongated in the vertical direction (i.e. aligned with the rolling shutter propagation direction, and in the specific example aligned with the image sensor row direction). This approach may provide a higher correlation between lines while maintaining the high resolution and speckle variation of each line. This may result in improved motion estimation.

The effect is illustrated in FIG. 14 which shows a comparison between corresponding speckle patterns for an isotropic and non- isotropic light spot for respectively no motion and a surface sinusoidal motion.

FIG. 14 specifically shows a speckle pattern of an anisotropic light spot 1401 captured with a rolling shutter camera when there is no motion (pattern 1403) and when there is a high-frequency surface motion (pattern 1405). It further shows a speckle pattern of an isotropic light spot 1407 captured with a rolling shutter camera when there is no motion (pattern 1409) and when there is a high-frequency surface motion (pattern 1411). As can be seen, the isotropic light spot results in an isotropic speckle pattern wherein the motion information (corresponding to the horizontal line shifts) is hard to discern. However, for the anisotropic light spot, the speckles are elongated resulting in the motion information being much easier to detect. In the example, the anisotropic light spot 1407 has been extended in the horizontal direction relative to the isotropic light spot 1401. This clearly results in an increased vertical dimension of the speckles and thus provides a higher line to line correlation. At the same time, the horizontal resolution is maintained.

It will be appreciated that the exact anisotropicity of the light spot will depend on the specifics of the individual embodiment. However, in many embodiments the average extension of speckle grains in a direction corresponding to the rolling shutter propagation direction is at least twice that of the average extension of the speckle grains in a direction perpendicular thereto. Thus, in many embodiments, the anisotropic light spot image (or indeed the light spot itself) may have a longest dimension which is at least twice that of the shortest dimension.

The anisotropic speckle pattern need not be generated by an anisotropic light spot on the surface. Rather, in some embodiments, the coherent light source 105 may be arranged to generate the light spot as an isotropic light spot with the camera being arranged to generate the corresponding speckle pattern as an anisotropic light spot image object.

Thus, in some embodiments the imaging optics may be modified instead of the projection optics. This may for example be achieved by the use of an anisotropic optical aperture, astigmatic optics, anamorphic optics, or prisms. Indeed, by changing the imaging optics, it is possible to stretch the image more in one direction than in another. This can affect both the outline of the blur circle and the shape of the (observed) speckles.

Specifically, this may be done similarly to the use of anamorphic lenses in film recording and cinema projection in order to obtain ultra wide screen images by change of the aspect ratio in the capture and/or projection elements. The optics might include cylindrical lenses or curved mirrors.

In the example, the anamorphic optics would be oriented such that the speckles are relatively stretched in the propagation direction of the rolling shutter.

The previous embodiments have focused on examples wherein the coherent light source 105 generates a single light spot which then creates a speckle pattern image object which is analyzed. However, in some embodiments, the coherent light source 105 can be arranged to generate a plurality of light spots on the surface. Such an exemplary embodiment is illustrated in FIG. 15. The example corresponds to the example of FIG. 1 but with the coherent light source 105 generating a plurality of light spots. This may for example be achieved by the use of multiple laser light sources, a single source with a diffraction grating, or the use of beam splitters and mirrors.

The camera is arranged such that it captures the plurality of light spots. The speckle imaging apparatus 101 furthermore includes a selector 1501 which is arranged to select a subset of the plurality of light spots. The selector 1501 is furthermore coupled to the analysis processor 113 which proceeds to analyze the speckle patterns of the subset of light spots.

The approach may be particularly suitable for automatic or semi-automatic adaptation to the specific positioning of the surface, and may in particular provide an increase flexibility and freedom in positioning the object to be monitored.

For example, the plurality of light spots may form a regular or non-regular grid. An object to be measured may then be placed within a relatively coarse test area, and the system may evaluate the plurality of light spots to find one or more of the light spots which are located at a suitable position on the surface. The analysis may then be based on the selected light spots. Furthermore, by performing a selection of a subset of light spots prior to the detailed analysis, a more efficient and less resource demanding system can be achieved.

Thus, the approach may use the projection of multiple light spots, e.g. in a regular pattern, to increase the likelihood of illuminating an interesting part of the object or subject to be measured. Furthermore, from the set of light spots a subset can be selected by selecting a region of interest on the camera sensor. The spatial resolution and frame rate can then be increased e.g. by only capturing and processing the selected region of interest.

In many applications, the likelihood of illuminating a preferential or even suitable spot on the subject or object under study without performing a manual adaption or requiring a very specific placement of the object is very small. Indeed, in most applications it is required that the analyzed light spot is positioned on an area of the surface wherein e.g. suitable vibrations are experienced. Typically, this is a relatively small area and the light spot must be positioned carefully. However, by using a plurality of light spots, only a coarse and flexible positioning of the object to be monitored relative to the coherent light source 105

(and camera) is required, and the speckle imaging apparatus 101 can then automatically adapt and select light spots positioned at suitable areas of the surface.

A disadvantage of observing multiple dots, however, is that the amount of information increases which may increase the resource demand and processing required. However, by having a separate selection of suitable spots for analysis, this may be mitigated and the resulting increase computational demands may be kept very low. For example, the spatial or temporal resolution may be decreased. For example, the spatial sensor resolution may be reduced by binning or sub sampling. As another example, the temporal resolution may be decreased by using a lower frame rate, e.g. by skipping frames when performing the selection. Furthermore, once a subset of light spots has been identified, these may be analyzed with full pixel resolution (determined as the physical or optical resolution rather than just the amount of pixels) and frame rate thereby ensuring that there is no degradation in the estimation of the surface motion.

As an example, a two megapixel sensor with a frame rate of 12 frames per second at full resolution may be used for the selection. However, when performing the full analysis only the small area corresponding to e.g. one selected light spot image object may be selected. This may allow a much faster frame rate, such as e.g. 200 frames per second.

As a specific example, the speckle imaging apparatus 101 may be used to monitor vital characteristics for a patient. The coherent light source 105 may generate a grid of light spots as illustrated in FIG. 16. The setup can for example be positioned above a patient bed where the dot pattern overlaps with the chest area of a resting patient. The pattern need not to be visible, preferably it will be based on invisible infra-red illumination and sensing. A lens or set of lenses is not necessary for the projection but can improve the signal quality as the speckle coarseness and hence effective contrast is related to the size of the laser spots.

FIG. 17 illustrates the image that may be captured by the out-of- focus camera 11. As can be seen a speckle pattern is generated for each light spot. FIG. 18 illustrate corresponding images captured for an experimental setup wherein the surface is a surface of a piezoelectric transducer,

The example of FIG. 16 and 17 may specifically be used for remote measurement of vital signs of the patient, such as specifically heart beat and respirations. Heart beats are manifested on the body of a patient by small surface vibrations. These vibrations appear and can be measured, among others places, in the neck, on the chest, or on the wrist. Respiration can be measured by measurement of movement of the chest, but might also be measurable on other places on the body, such as the head, the back, the arms, etc.

In the example, the analysis of the speckle patterns may thus provide information of the movement of the chest etc. of the patient, and this may provide a measurement of vital signs. In particular, if the surface from which the light spots reflect is changing in time the speckle pattern will also be changing in time. As such, many of the speckle patterns will have an apparent motion which in FIG. 17 is indicated by arrows. These movements can be detected using e.g. the analysis approaches previously described. It can be seen that not all motions are correlated and not all dots are related to the subject. Therefore, it is important to select light spots that provide relevant and accurate indications.

Therefore, in order to increase the accuracy of measuring and tracking vital signs, the system is arranged to select a small subset of the light spots. In the specific example, four light spots may be selected but it will be appreciated that in other embodiments other numbers may be selected (and specifically only one light spot may be selected in many embodiments. Following the selection of the subset (which corresponds to a smaller area of the image sensor), the spatial resolution (of the sensor) and frame rate/temporal resolution can be increased for more accurate measurements. It should be borne in mind that the spatial resolution does not just indicate the total number of pixels but the number of pixels per unit length or resolving power.

The exact algorithm and criteria used for selecting the subset will depend on the preferences and requirements of the individual embodiment.

In some embodiments, the selector 1501 may be arranged to select the subset in response to intensity of light spots of the light spots in the out-of- focus image. As can be seen from FIG. 18 the intensity of the light spots in the out-of- focus image varies for different light spots. In particular, the average brightness for a light spot image object in the out-of-focus image may depend on the reflection properties of the reflecting surface and this may be used to ensure that the selected light spots are indeed from the appropriate surface. Therefore, the subset may be selected to comprise the light spots that have a high intensity thereby resulting in improved analysis.

In some embodiments, the selector 1501 may be arranged to select the subset in response to a speckle contrast of light spots of the plurality of the light spots. As illustrated in FIG. 18, the speckle contrast may depend on the exact properties associated with the specific position of the light spot. For example, if the distance to the light spot deviates from the exact focus distance, the speckle pattern may become coarser and may result in a reduced contrast. By selecting the subset of light spots to have a high speckle contrast, an improved analysis can be performed. E.g. a correlation estimation may become more accurate and reliable.

In some embodiments, the selector 1501 may be arranged to select the subset in response to a speckle pattern variation of light spots of the plurality of the light spots. In particular, the subset may be selected based on correlations in the temporal behavior of speckle patterns of different light spots. Thus, in some embodiments, the selector 1501 may be arranged to select the subset in response to a correlation between variations of different light spots of the plurality of the light spots.

In some embodiments, the subset may be selected in response to the speckle pattern variations having a temporal behavior which meets a similarity criterion. For example, the subset may be selected to only include light spots for which the speckle patterns change at substantially corresponding time instants, or which e.g. have repeating variations with the same frequency. For example, when monitoring a heart rate, this can ensure that only light spots positioned on surfaces which move with the pulse of the patient are considered.

In some embodiments, the variation of the speckle pattern may be measured as a motion estimate for the speckle pattern. Specifically low complexity motion estimation may be performed and used to select light spots that have corresponding motions.

In some embodiments, the selector 1501 may be arranged to select the subset in response to a change in a light spot pattern of the light spots. Specifically, the coherent light source 105 may be arranged to generate a regular grid of light spots. However, when the grid covers an area with significant depth variations (e.g. both the patient's chest and part of the hospital bed), the depth distances result in a relative displacement of the light spots in the captured image if the laser and the camera are not aligned, i.e. a parallax effect or occurs. Thus, a non-regular grid may be recorded in the captured image, and this deviation may be used to identify light spots that do not have the expected depth. This approach may for example be used to detect which light spots hit the patient's chest.

In some embodiments, the selector 1501 may be arranged to select the subset in response to a change in the size of the image objects for the light spots. Specifically, the size of the image objects may depend on the distance to the focal plane, and thus the size may be indicative of a depth of the associated light spot. This may be used to select light spots at a suitable distance.

In some embodiments, the selector 1501 may be arranged to select the subset in response to a non-speckle pattern image. For example, the light spot positions relative to the positioning of the patient may be evaluated using another image. This image may for example be an in- focus image which may show the light spots as small spots together with the patient and part of the bed. The system may then evaluate which spots are overlaying the patient's chest. As other examples, the relation of the spot positions to the patient position might be derived from an additional camera image, the background image, or prior information.

It will be appreciated that the subset selection need not be performed for every frame of a video based imaging system. For example, the subset selection may be repeated at given time intervals. For example, reselecting the light spots for the subset every couple of seconds would allow the system to track patient movements.

As mentioned previously, by the use of a coherent light source that generates very small light spots, the speckles of the speckle patterns can be made very coarse and the sensitivity of the system can be very high. However, at the same time the response becomes more random and difficult to track. This can result in an effect where the speckle circles start to flicker at frequencies related to the motion. In some embodiments, such phenomena may be analyzed to provide further motion information.

In some embodiments, the system may comprise feedback functionality which may for example control the coherent light source to switch some of the light spots on and off. Also, whereas the regular grid of substantially identical light spots may often be used, the system can also be used with a non-uniform grid. Indeed, not only may the grid spacing vary but so may the light spot sizes. This may be used to optimize the monitoring for the specific characteristics of the application.

It will also be appreciated that whereas the use of multiple light spots may be particularly suitable for rolling shutter speckle imaging, it may also be suitable for many other speckle imaging types and applications.

It will be appreciated that the described approach can be used for many different applications including for example user interface applications, patient monitoring etc.

It will be appreciated that the above description for clarity has described embodiments of the invention with reference to different functional circuits, units and processors. However, it will be apparent that any suitable distribution of functionality between different functional circuits, units or processors may be used without detracting from the invention. For example, functionality illustrated to be performed by separate processors or controllers may be performed by the same processor or controllers. Hence, references to specific functional units or circuits are only to be seen as references to suitable means for providing the described functionality rather than indicative of a strict logical or physical structure or organization. The invention can be implemented in any suitable form including hardware, software, firmware or any combination of these. The invention may optionally be

implemented at least partly as computer software running on one or more data processors and/or digital signal processors. The elements and components of an embodiment of the invention may be physically, functionally and logically implemented in any suitable way. Indeed the functionality may be implemented in a single unit, in a plurality of units or as part of other functional units. As such, the invention may be implemented in a single unit or may be physically and functionally distributed between different units, circuits and processors.

Although the present invention has been described in connection with some embodiments, it is not intended to be limited to the specific form set forth herein. Rather, the scope of the present invention is limited only by the accompanying claims. Additionally, although a feature may appear to be described in connection with particular embodiments, one skilled in the art would recognize that various features of the described embodiments may be combined in accordance with the invention. In the claims, the term "comprising" does not exclude the presence of other elements or steps.

Furthermore, although individually listed, a plurality of means, elements, circuits or method steps may be implemented by e.g. a single circuit, unit or processor.

Additionally, although individual features may be included in different claims, these may possibly be advantageously combined, and the inclusion in different claims does not imply that a combination of features is not feasible and/or advantageous. Also the inclusion of a feature in one category of claims does not imply a limitation to this category but rather indicates that the feature is equally applicable to other claim categories as appropriate.

Furthermore, the order of features in the claims do not imply any specific order in which the features must be worked and in particular the order of individual steps in a method claim does not imply that the steps must be performed in this order. Rather, the steps may be performed in any suitable order. In addition, singular references do not exclude a plurality. Thus references to "a", "an", "first", "second" etc. do not preclude a plurality. Reference signs in the claims are provided merely as a clarifying example, and shall not be construed as limiting the scope of the claims in any way.

Claims

CLAIMS:

1. An apparatus for estimating a property of a surface of an object, the apparatus comprising:

a coherent light source (105) for generating at least one light spot on the surface;

a camera (111) for capturing an out-of-focus image of at least a part of the surface comprising the light spot, the out-of-focus image comprising a light spot image object for the light spot, and the light spot image object having a speckle pattern;

an analyzer (113) for estimating the property of the surface in response to the speckle pattern;

wherein the camera (111) comprises a rolling shutter for capturing the out-of- focus image.

2. The apparatus of claim 1, wherein the analyzer (113) is arranged to determine a temporal property of the surface in response to a spatial variation of a property of the speckle pattern in the out-of-focus image.

3. The apparatus of claim 1 or 2, wherein the analyzer (113) is arranged to determine the temporal property in response to a spatial pattern variation of the speckle pattern in a direction corresponding to the rolling shutter propagation direction.

4. The apparatus of claim 1, wherein the analyzer (113) is arranged to determine a temporal property of the surface in response to a spatial frequency analysis of the speckle pattern.

5. The apparatus of claim 1 or 2, wherein the rolling shutter is arranged to capture the out-of-focus image line sequentially; and the analyzer (113) is arranged to determine a speckle pattern property for each group of a plurality of groups which each comprise at least part of a number of adjacent lines of the out-of-focus image, and to determine the property in response to a comparison of the speckle pattern property for different groups.

6. The apparatus of claim 1 or 2, wherein the rolling shutter is arranged to capture the out-of- focus image line sequentially; and the analyzer is arranged to determine the property in response to a comparison of speckle patterns for sequential lines.

7. The apparatus of claim 1 or 2, wherein the analyzer (113) is arranged to determine the property in response to a comparison between the speckle pattern and a reference speckle pattern.

8. The apparatus of claim 1, wherein the analyzer (113) is arranged to determine a two-dimensional movement of the surface in response to a detection of a speckle pattern transformation, the transformation being asymmetric between a first direction corresponding to the rolling shutter propagation direction, and a second direction not corresponding to the rolling shutter propagation direction.

9. The apparatus of claim 1 or 2, wherein the apparatus is arranged to generate the speckle pattern as an anisotropic speckle pattern.

10. The apparatus of claim 9, wherein an average extension of speckle grains in a first direction corresponding to the rolling shutter propagation direction is at least twice that of an average extension of speckle grains in a second direction perpendicular to the first direction.

11. The apparatus of claim 1 or 2, wherein the coherent light source (105) is arranged to generate a plurality of light spots on the surface, and the camera (111) is arranged to capture the plurality of light spots in the out-of- focus image; the apparatus further comprising:

a selector (1501) arranged to select a subset of light spots for analysis by the analyzer(l 13).

12. The apparatus of claim 11, wherein the selector (1501) is arranged to select the subset of light spots using a lower processing resolution than used by the analyzer (113) when determining the property.

13. The apparatus of claim 11 or 12, wherein the selector (1501) is arranged to select the subset in response to at least one of:

an intensity for light spots of the plurality of the light spots;

a speckle contrast for light spots of the plurality of the light spots; a speckle pattern variation for light spots of the plurality of the light spots; a correlation between variations for different light spots of the plurality of the light spots; and

a change in a light spot pattern of the plurality of the light spots.

14. The apparatus of claim 12, wherein the selector (1501) is arranged to select the subset in response to a non-speckle pattern image.

15. A method of estimating a property of a surface of an object, the method comprising the steps of:

generating at least one light spot on the surface using a coherent light source (105);

capturing an out-of- focus image of at least a part of the surface comprising the light spot, the out-of-focus image comprising a light spot image object for the light spot, and the light spot image object having a speckle pattern;

estimating the property of the surface in response to the speckle pattern; and capturing the out-of-focus image using a rolling shutter.