EP4069072A1 - Verfahren und vorrichtung zur kontaktfreien bestimmung von zeitlichen farb- und intensitätsveränderungen bei objekten - Google Patents
Verfahren und vorrichtung zur kontaktfreien bestimmung von zeitlichen farb- und intensitätsveränderungen bei objektenInfo
- Publication number
- EP4069072A1 EP4069072A1 EP20815781.8A EP20815781A EP4069072A1 EP 4069072 A1 EP4069072 A1 EP 4069072A1 EP 20815781 A EP20815781 A EP 20815781A EP 4069072 A1 EP4069072 A1 EP 4069072A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- measurement
- overview
- color
- scene
- spectral range
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/02—Detecting, measuring or recording for evaluating the cardiovascular system, e.g. pulse, heart rate, blood pressure or blood flow
- A61B5/024—Measuring pulse rate or heart rate
- A61B5/02416—Measuring pulse rate or heart rate using photoplethysmograph signals, e.g. generated by infrared radiation
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/02—Detecting, measuring or recording for evaluating the cardiovascular system, e.g. pulse, heart rate, blood pressure or blood flow
- A61B5/024—Measuring pulse rate or heart rate
- A61B5/02416—Measuring pulse rate or heart rate using photoplethysmograph signals, e.g. generated by infrared radiation
- A61B5/02427—Details of sensor
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/0059—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence
- A61B5/0073—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence by tomography, i.e. reconstruction of 3D images from 2D projections
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/0059—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence
- A61B5/0077—Devices for viewing the surface of the body, e.g. camera, magnifying lens
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/145—Measuring characteristics of blood in vivo, e.g. gas concentration or pH-value ; Measuring characteristics of body fluids or tissues, e.g. interstitial fluid or cerebral tissue
- A61B5/1455—Measuring characteristics of blood in vivo, e.g. gas concentration or pH-value ; Measuring characteristics of body fluids or tissues, e.g. interstitial fluid or cerebral tissue using optical sensors, e.g. spectral photometrical oximeters
- A61B5/14551—Measuring characteristics of blood in vivo, e.g. gas concentration or pH-value ; Measuring characteristics of body fluids or tissues, e.g. interstitial fluid or cerebral tissue using optical sensors, e.g. spectral photometrical oximeters for measuring blood gases
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C11/00—Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying
- G01C11/04—Interpretation of pictures
- G01C11/06—Interpretation of pictures by comparison of two or more pictures of the same area
- G01C11/12—Interpretation of pictures by comparison of two or more pictures of the same area the pictures being supported in the same relative position as when they were taken
- G01C11/14—Interpretation of pictures by comparison of two or more pictures of the same area the pictures being supported in the same relative position as when they were taken with optical projection
- G01C11/16—Interpretation of pictures by comparison of two or more pictures of the same area the pictures being supported in the same relative position as when they were taken with optical projection in a common plane
- G01C11/18—Interpretation of pictures by comparison of two or more pictures of the same area the pictures being supported in the same relative position as when they were taken with optical projection in a common plane involving scanning means
Definitions
- the invention relates to a method and a device for the contact-free determination of changes in color and intensity over time in objects in a scene, for example in sports centers or hospitals, but also in plants.
- a number of methods are known in the prior art for monitoring the state of health or for performance diagnostics in athletes, but these are predominantly dependent on contact with the people.
- the changes in color and intensity relate to light of certain wavelengths which either the object itself actively emits, for example an LED, or which is partially absorbed or weakened by the object when it is illuminated and is therefore not completely reflected.
- the term "color changes" usually refers to changes in the wavelength, i.e., for example, the shift of a reflection maximum or transmission maximum within narrow wavelength ranges.
- intensity changes usually refers to changes in the maximum reflection or transmission at a constant wavelength.
- color changes are also understood to mean changes in intensity, because these seem to cause a change in color subjectively and only apparently through a change in brightness, which is actually not the case for a single color or wavelength, but only for the one Overall impression caused by the mixture of all colors or wavelengths: If the intensity of a focal color changes, a different overall impression of color arises for an observer in the color mixture of the individual wavelengths.
- jaundice in people, especially newborns The higher concentration of bilirubin in the blood leads to an increased reflection in the wavelength range between 420 nm and 450 nm and thus to the clearly more yellow color perception of the skin than in healthy people.
- EKG electrocardiogram
- PPG photoplethysmography
- Both methods are contact-based, but due to their high reliability, objectivity and validity, they are considered to be the gold standard method.
- Their major disadvantage is, on the one hand, that they are contact-based methods, that is, only one person can be observed at a time, and that specially trained specialists are absolutely necessary for the exact application and application of the electrodes required for the measurement.
- the skin must be prepared by disinfection and, if necessary, by shaving; Anatomical knowledge is required to place the electrodes.
- Contact-based systems such as the EKG and the PPG are also used to monitor the state of health of patients in a hospital. Especially in hospitals, where it would be desirable to monitor several people at the same time, this contact-based method has a disadvantage, since several systems would have to be kept and operated in parallel to record several people at the same time. The acquisition costs are correspondingly high.
- non-contact measurement methods are also known in the prior art, but they are not used to record the performance or health status of people.
- soil and plant populations are investigated when observing the earth from the air or by satellite, for which purpose conventional spectrometers are used, which reproduce an image broken down into many wavelengths.
- the evaluation of such data is particularly costly when it comes to special spectral properties that serve, for example, as indicators for the condition or pollution of forests or arable soils.
- a breakdown according to depth information which could be seen, for example, whether the forest floor or the treetops are measured, is not provided for with this approach.
- nano-optical filters are used, which can filter light of certain wavelengths or wavelength ranges due to special interaction mechanisms.
- Such nano-optical elements are described, for example, in an article by S. Young et al. “Multispectral CMOS sensors with on-chip nanostructures for wavelength monitoring of LED devices”, published in Proceedings of SPIE (8641), page 86411 B (2013).
- the nano-optical filters are placed in front of the camera in order to allow easy adaptation to other wavelengths through interchangeability, for example if other LEDs are to be measured. These are purely static, two-dimensional measurements in the smallest of spaces.
- the object of the invention is therefore to develop a method and a device with which the contact-free determination of changes in color and intensity over time can take place in several objects at the same time with subsequent analysis.
- the “scene” can be, for example, an enclosed space in a sports center or a hospital, but also a larger area outdoors, for example a soccer ball or baseball field.
- object includes everything in which the measurement of changes in color and intensity over time can provide information about the state of the object, for example light-emitting diodes or plants, but in particular people who can also move freely in the scene.
- the objects are distributed in the scene and the aim is to find the interesting color and intensity changes in certain areas of all objects measure at the same time and draw conclusions on specific parameters that describe the state of the objects based on the changes in color and intensity.
- a stereoscopic overview map of the scene is first generated by detecting a first monoscopic overview image of the scene from a predetermined first viewing direction and a second monoscopic overview image from a predetermined second viewing direction in a first spectral range.
- the first and second viewing directions are different from one another and their position in relation to one another is known.
- the two monoscopic overview images are offset to form the stereoscopic overview map, from which the spatial positions of objects to be measured are then finally determined.
- a first and a second overview camera are provided in the device according to the invention. The two cameras each detect the same wavelength range or the same wavelength and are aligned with the scene under a fixed, predetermined first or second viewing direction.
- the viewing directions preferably intersect at a point in or behind the scene, so that viewing angles can be determined on the basis of this point and a reference axis.
- the information that is detected with the overview cameras is independent of the temporal changes in color and intensity that are to be detected, but serve to identify the position and possibly also the movement and direction of movement of objects in the scene. The faster an object moves, the more overview images have to be recorded and overview maps calculated, unless an approximate determination of the movement is possible.
- a two-dimensional detail image from a predetermined third viewing direction is detected in a second spectral range.
- the third viewing direction can be different from the first and second viewing directions, which may facilitate the offsetting. But it can also correspond to one of the two viewing directions.
- a camera can be used both for an overview image and for a detail image.
- the two-dimensional detail image is preferably recorded simultaneously with the recording of the overview images, so that the image content depicts an identical moment, or immediately before or after it is recorded, for example if the object is not moving.
- a detail camera is used for recording, which is aimed at the scene from the third viewing direction.
- the two-dimensional detail image is then projected onto the stereoscopic overview map.
- Measurement areas are selected as areas of interest in the scene on the basis of measurement criteria, which are specified as a function of parameters on which conclusions are to be drawn from the changes in color and intensity. This takes place - as well as the calculation of the two monoscopic overview images to form the stereoscopic overview map - in a computing unit designed for this purpose.
- the position of all three viewing directions with respect to one another is known; they preferably intersect at a point in the rear area of the scene or behind the scene. Since the position of the cameras in relation to one another is known, a geometric calibration of the position of the detail camera in relation to the position of the overview cameras can be carried out.
- This calibration can then be used to determine transformation matrices for a one-to-one transformation of the spatial image data of the stereoscopic image to the two-dimensional image data of the detailed image and vice versa.
- the position of measurement areas which are also referred to as regions of interest (ROIs)
- ROIs regions of interest
- optimal measurement areas can be defined, for example areas of the skin of people on which a particularly strong color change can be measured due to good blood circulation, such as the forehead, the neck or the wrists.
- a detailed camera that has a very high resolution, in particular a resolution that is higher than that of the overview cameras, so that the measurement areas can be identified as precisely as possible.
- a somewhat lower resolution is usually sufficient for recording the overview image, since the three-dimensional structures, which can be represented by points and vectors, for example as grids, usually have dimensions that extend over a larger area of pixels.
- overview cameras with a resolution of 832 x 632 pixels and detail cameras with a resolution of 1280 x 1024 pixels can be used; With these resolutions, good results can be achieved, especially in closed rooms.
- overview cameras with a higher resolution can be used in order to be able to determine further parameters if necessary. If all cameras are identical with the same resolution, the evaluation can possibly be simplified.
- the measuring camera or the area detector for registering the image data of this camera advantageously has the same resolution as the detail camera - for example 1280 x 1024 pixels, since this makes it easier to assign the measuring areas pixel by pixel.
- the detection takes place in a spatially resolved manner, ie by means of an area detector, for example a CMOS sensor. The detection also takes place - if at least two measurement spectral ranges are specified wavelength resolved, which means that the spectral ranges are detected independently of one another on the same area detector.
- the frame rate i.e. the number of images that are recorded per second (frame rate) depends on the type of objects to be measured and how much the objects move in the scene. If, for example, static objects such as plants are observed from a static position over a longer period of time, then color changes tend to take place over a longer period of time. In this case it is sufficient, for example, to take pictures at intervals of several minutes, hours or days, in which case an overview map and a detailed picture are usually generated at the same time in order to be able to compensate for any changes in position.
- a frame rate of at least 15 or 20 frames per second will be selected in order to be able to correctly record a high pulse rate, i.e. the number of heartbeats per minute.
- a high pulse rate i.e. the number of heartbeats per minute.
- a similar procedure is followed for athletes, where as a rule new overview maps and detailed images have to be generated again and again between two series of measurement images, since the athletes generally move faster.
- the number of overview maps and detailed images can be reduced if the objects are analyzed with regard to movement between the recording of two such data records at different times and a displacement of the measuring surfaces can also be forecast in this way.
- the reliability of the measurements can be increased, since the spatial position of the measurement objects in the scene can be taken into account and corresponding corrections can be made in the image.
- By projecting the two-dimensional detail image recorded measuring surfaces on the objects can be corrected against perspective distortions. Falsified measurement data or those with low reliability can also be excluded in this way. Overall, this improves the reliability of the measurement.
- the projection can also be used to determine the exact position of measured signals, ie the reflected colors, which is advantageous when observing a group of moving objects.
- the measurement is only carried out for wavelengths in at least one measurement spectral range that is predetermined as a function of the measurement criteria - ultimately depending on the parameters on which conclusions are to be drawn.
- a corresponding filter which only allows light in the specified measurement spectral range, can be connected upstream of the measurement camera in the beam path.
- the specified measurement spectral range can - depending on the parameters on which conclusions are to be drawn - be selected very narrowly and only allow one wavelength to pass if, for example, only intensity fluctuations are examined, or a wider wavelength range if color fluctuations are also to be measured.
- the measurement spectral range can be set to 420 nm to 450 nm when monitoring patients with jaundice, since bilirubin, which is then present in higher concentrations, reflects in this range.
- exemplary measurement spectral ranges are between 630 nm and 750 nm for the range of the visible Hb band (Hb - hemoglobin) and between 800 nm and 940 nm for the range of the Hb band in the near infrared.
- the data detected on the measurement surfaces are analyzed in the measurement images for temporal variation in color and / or intensity, and the results of the analysis are then output.
- the computing unit which is designed to analyze the temporal variation of the intensity and / or the color of the light in the at least one measurement spectral range in the series of measurement images and to transmit the results to a corresponding output unit.
- This can be, for example, a monitor, a storage unit, or - if the values are critical - a signal transmitter that alerts the on-call staff.
- the pulse rate of an athlete or a newborn can be derived from temporal color fluctuations in the near infrared wavelength range at a wavelength of around 830 nm in a range of +/- 50 nm around this wavelength, whereby only the values of the measuring surfaces that are on free Areas of the skin where the veins run in shallow depths are arranged to be analyzed.
- the device and the method can also be used to observe other parameters of interest, for example the oxygen content in the blood, for which measurements at a wavelength of about 660 nm are necessary Measure the oxygen saturation of the hemoglobin bound in the blood.
- the measurement image is detected in at least two predetermined measurement spectral ranges with spatial and wavelength resolution.
- a filter element is advantageously used that is designed as a nano-optical filter matrix. This is applied to a planar measuring sensor that detects the light emitted by the measuring surfaces in the measuring camera.
- the nano-optical filter matrix has two types of filter structures for the purpose of detecting two measurement spectral ranges.
- First filter structures are used to filter a first measurement spectral range and second filter structures are used to filter a second measurement spectral range.
- First and second filter structures are preferably arranged in a checkerboard manner in the nano-optical filter matrix.
- Each of the filter structures covers a larger area of pixels.
- the effect of the nano-optical filter elements is based on the phenomenon of extraordinary optical transmission, which occurs when electromagnetic radiation interacts with periodically arranged holes of nanoscopic size.
- the spectral selectivity can be determined by means of the material - for example gold or silicon -, the arrangement period (pitch) of the holes and the diameter of the holes. If further measurement spectral ranges are to be analyzed, a correspondingly expanded checkerboard structure with three or more types of filter structures can be used.
- Wavelengths each spatially resolved and each resolved according to wavelengths or spectral ranges i.e. the measurement spectral ranges are detected unaffected by one another, since the pixels arranged under the filter structures detect either only one or the other wavelength or one or the other measurement spectral range, i.e. actually, due to the filter arrangement only measure the intensity at one wavelength or in a wavelength range unaffected by the other.
- the measurement signal is prepared before the final evaluation by the processing unit, i.e. the hardware used.
- no further calculations are necessary here and only the color / wavelength of interest is actually measured. This reduces the time required to process the data by software, which reduces the latency of the measurement and increases the real-time capability.
- the derivation of parameters from the data becomes more precise if no algorithmic-numerical errors occur at all.
- nano-optical filter masks on an image sensor also facilitates mass production: the application or structuring of the nano-optical filter matrices can take place in the same manufacturing step as the insertion of the camera sensor.
- the integration of the nano-optical filter elements also increases the mechanical robustness against shocks or vibrations of the measuring system and means compression compared to conventional systems in which the filters are placed in front of the camera, as is also possible when using only one wavelength.
- the nano-optical filter matrices can be applied to the camera sensor, for example, by means of so-called bonding, ie using an adhesive that cures under UV radiation and - in the cured state - a refractive index similar to that of glasses.
- the filter matrix can be produced separately on wafers, sawn out of them and glued onto an existing chip. Another possibility is to integrate the filter matrices directly into the manufacturing process of CMOS chips, for example by means of electron beam lithography.
- the projection of a pattern onto the scene can be used, especially in closed rooms.
- a pattern is projected onto the scene, which is detected in the monoscopic overview images.
- a pattern projector is used to project the pattern, which projects a pattern onto the scene in the first spectral range.
- the method of fringe projection can be used with particular preference, i.e. the pattern is preferably strip-shaped, without being limited to this embodiment; in this way, other patterns that produce significant contrast differences, such as checkerboard patterns, can also be used.
- the first spectral range preferably includes the near infrared, so that a projected pattern is not visible and is not disruptive to humans and other living beings.
- NIR-GOBO projector NIR - near infrared, GOBO - GOes Before Optics
- the first spectral range and the second spectral range advantageously do not overlap, so that the recording of the overview images and the detail image can take place simultaneously without any mutual influencing taking place here.
- the detailed image is usually recorded in the wavelength range visible to the human eye, whereas the overview images are recorded in the - preferably near - infrared range in order to avoid irritation.
- the scene is illuminated with an additional light source which emits light exclusively in the first spectral range, for example in the near infrared. This increases the accuracy of the overview measurement and thus of the stereoscopic overview map.
- the directions of view of all cameras are known to one another or the cameras are geometrically calibrated with respect to one another.
- the first and the second overview camera, the detail camera and the measuring camera can be arranged on an arc of a circle, with the first and second overview cameras in FIG End points of the circular arc section are arranged in order to realize the largest possible angular distance, which is advantageous for the accuracy of the 3D measurement. All viewing directions then intersect at a point in the scene or behind it.
- a much more compact arrangement is obtained if the first and the second overview cameras are arranged on a first circular arc section and the detail camera and the measuring camera are arranged on a second circular arc section.
- the two circular arc sections then intersect at an angle other than zero, preferably at an angle of 90 °.
- the optional sample projector can be placed in the middle, where the two circular arc sections intersect.
- Such an arrangement can then be integrated as a structural unit in a housing, with the geometric calibration taking place during assembly. This unit can then be used flexibly and can be transported without tools.
- the method and device can be used in particular to determine performance in sports.
- Medical applications consist in monitoring the biosignals of patients in an outpatient or inpatient environment.
- vital parameters can be measured with less complications, especially in patients whose vital parameters are unstable, since electrodes do not have to be applied. This is particularly advantageous when monitoring patients who are particularly keen to move, such as newborns.
- Another possible application is the detection of stress at the workplace within a company health management system.
- psychological diagnostics the emotional and physical stress values of patients with emotional dysfunctions, such as those that occur after trauma or with congenital mental illnesses, can be measured and assessed.
- a frequent problem here is the inadequate self-assessment of such patients about the level of their stress level and a lack of communication through gestures and facial expressions with a therapist or doctor.
- diseases such as autism
- an additional complication is that contact-based vital sensors cannot be attached to patients or can only be attached to patients under great stress.
- a non-contact system makes it possible to support the therapist in his work as well as to provide feedback and training for the patients.
- the pulse rate and oxygen content can only be indicative of a stress level and cannot be used solely for evaluation; other individual and subjective parameters should also be taken into account.
- the method according to the invention and the device are also suitable for monitoring locations with large crowds, such as train stations or airports.
- Accelerated help in medical emergencies can be guaranteed, in the sense of an early warning system.
- the data recorded by the measuring camera can be used to determine not only the pulse rate, but also the temporal pulse sequence, which reflects the heart rhythm. If irregularities occur here in certain, known patterns, these can, for example, indicate a heart attack, which is recognized by the system.
- people with increased stress levels can be monitored separately as a potential threat to safety in order to initiate safety measures if necessary.
- a high pulse rate could serve as a preselection for people who are then subjected to a more detailed behavioral analysis.
- FIGS. 2a-b show possible implementations of devices for carrying out the method
- FIGS. 3a-b show the basic structure of a nano-optical filter
- FIG. 4 shows a further embodiment of a nano-optical filter element
- FIG. 5 shows a nano-optical filter element for two wavelengths
- FIG. 6 shows the arrangement of a nano-optical filter element in a camera
- FIG. 7 shows the transmission properties of such a nano-optical element for two exemplary wavelength ranges.
- Fig. 1 the basic sequence of a method for the contact-free determination of temporal color changes in several objects in a scene is shown.
- a scene 1 is initially shown symbolically. This can be, for example, a room in a fitness center or sports center, an area of a stadium, for example in the area of the running track, or wards or rooms in a hospital, to name a few examples. Forest areas or fields, plantations or parts thereof, etc. can also be such scenes.
- At least one object whose color changes are to be measured This can be, for example, a plant or several plants, technical devices that emit light, or people whose vital parameters are to be monitored, for example.
- the scene is recorded with three different camera systems, each comprising one or more cameras, in order to enable the corresponding parameters to be monitored and evaluated.
- the first camera system comprises a first overview camera 2, which is directed at the scene 1 from a predefined first viewing direction, and a second overview camera, which is directed at the scene 1 from a predefined second viewing direction. Both directions of view are different from one another, but are known, so that a geometric calibration of the two cameras is possible. Exemplary structures are shown in FIGS. 2a and 2b.
- Each of the two overview cameras 2, 3 records a monoscopic overview image of the scene 1 in a first spectral range.
- the two monoscopic overview images are calculated into a stereoscopic overview map, which also contains the depth information of the objects, i.e. their distance to the cameras or another reference point, so that basically the spatial position of all objects to be measured in scene 1, i.e. in all three spatial coordinates of a Cartesian system, is known.
- a separate light source 4 which emits light in the first spectral range.
- a range in the near infrared is preferably selected as the first spectral range, which comprises the wavelength range between approximately 780 nm and approximately 3000 nm.
- Such a light source 4 for the separate illumination of the scene is shown in FIG. 2a, in which all cameras are arranged on an arc of a circle.
- the accuracy can also or additionally be increased if, in order to generate the stereoscopic overview map, a pattern is projected onto the scene 1 which is detected in the monoscopic overview images.
- the arrangement comprises a device shown in FIG.
- the pattern projector 5 which in the first spectral range a pattern on the scene 1 projected, preferably a stripe-shaped or checkerboard-shaped pattern.
- a wavelength range that is imperceptible to the human eye, such as near infrared, is then particularly well suited as the first spectral range for the projection of a pattern.
- the first spectral range is of course not exclusively limited to the near infrared; wavelengths greater than 3000 nm can also be used, and wavelengths in the visible range or smaller are also conceivable if, for example, it is not living objects that are being measured.
- the overview cameras 2 and 3 can have a high resolution, but do not have to, since in principle only the position of the objects in space is to be determined here, which are then broken down into individual areas, the extent of which, however, is usually greater than the extent of a pixel of a sensor used in a camera.
- a two-dimensional detail image is detected in a second spectral range from a predetermined third viewing direction, the position of which is known in relation to the first and second viewing directions.
- the third viewing direction can be different from the first two or it can coincide with one of these.
- a detail camera 6 is used, which has the highest possible resolution.
- the recording of the detail image takes place in a second spectral range, which is preferably different from the first spectral range.
- the recording is preferably made in the spectral range visible to the human eye, i.e. in the spectral range between about 380 nm and about 780 nm.
- the two-dimensional detail image is projected onto the stereoscopic overview map, which is done with the aid of a computing unit.
- Measurement areas are then selected as areas of interest in the scene on the basis of measurement criteria, which are dependent on parameters on which conclusions are to be drawn from the changes in color and intensity. For athletes training on specially designed devices or for newborns, these are, for example, areas of exposed skin such as the neck, forehead or wrists that can be used to determine the vital parameters on the basis of color changes. Due to the projection, disruptive influences that could falsify the measurement of the color and intensity changes in the identified areas can also be taken into account and corrected. Such disruptive influences are, for example, the position of the measuring objects relative to the camera, shadows or perspective distortions. By defining measuring areas that correspond to defined areas on a sensor of a camera, the effort for the subsequent evaluation can be significantly reduced.
- a fourth viewing direction whose position in relation to the other three viewing directions is also known and can thus be geometrically calibrated, becomes light that is emitted by the measurement areas in at least one Measurement spectral range, which is specified on the basis of the measurement criteria, detected in a spatially resolved manner in a continuously recorded series of measurement images.
- a measuring camera 7 is used, which is directed onto the scene from the fourth viewing direction.
- the color and intensity values measured at the pixels of an area detector in the at least one predetermined measurement spectral range, which are assigned to the respective measurement areas, are analyzed for temporal variation in the intensity and / or color of this light in the at least one measurement spectral range,
- a computing unit 8 which also processes the information from the other images.
- the results of the analysis are then saved and / or output; Should critical changes occur - for example in the vital parameters of an athlete - corresponding warning signals can also be issued or responsible persons can be notified. In the case of monitoring the vital parameters of patients, this can be the doctor on duty, for example.
- FIG. 2a and 2b show two possible arrangements for the elements used in a device for the contact-free determination of temporal color and intensity fluctuations in objects of a scene.
- the first overview camera 2, the second overview camera 3, the detail camera 6 and the measuring camera 7 are arranged on an arc of a circle; the directions of view intersect at a point in the scene 1.
- a pattern projector 5 with a light source 4 which emits light in the first spectral range is optionally provided.
- the sample projector can also be identical to or integrated into one of the two overview cameras 1, 2; A stereoscopic overview image can also be generated in this way.
- the first and the second overview cameras 2, 3 are arranged on a first segment of a circular arc.
- the detail camera 6 and the measuring camera 7 are arranged on a second circular arc section, and both circular arc sections intersect at an angle of 90 °.
- the optional sample projector 5 is located at the intersection of the circular arc sections. All of them are here too Cameras aligned with their viewing directions so that they intersect at a point in scene 1 or behind it in order to be able to geometrically calibrate the cameras.
- the structure of the device shown in FIG. 2b is compact and can therefore also be transported without great effort, for example between different rooms of a sports facility or a hospital; the structure can be designed as a separate, compact structural unit, which optionally also includes a computing unit 8.
- the number of measurement images that are recorded per second for fixed measurement surfaces depends on the type of color and intensity changes to be measured and the associated parameters. In the case of determining vital parameters in athletes or patients, the number of images that are recorded per second should not be less than 15 in order to obtain meaningful measurements that allow a reliable determination of the pulse rate.
- the chlorophyll content of plants does not have to be monitored at such a high frame rate; here, much longer intervals of several hours to several days are sufficient.
- the rate at which overview and detailed images are recorded, on the basis of which the measurement areas are determined also depends on the characteristics of the scene. When monitoring the chlorophyll content of plants, for example, before each recording of a measurement image, the measurement areas can be recalibrated by recording overview images and a detailed image.
- the rate at which overview and detailed images are recorded can be in the range of a tenth of a second or less.
- the number of specified measurement spectral ranges in which the color and / or intensity changes are to be measured and analyzed depends on the parameters on which conclusions are to be drawn. For the monitoring of light-emitting diodes or plants, it may be sufficient to carry out the measurement in just a single measurement spectral range. This can usually be achieved with filters that are placed in front of a lens of the measuring camera, which allows easy adaptation to different situations in which measurements only have to be made in one measuring spectral range.
- the spatially and wavelength-resolved determination of the color and intensity changes is problematic if these changes are to be measured and analyzed in two or more measurement spectral ranges per measurement area.
- An example of this are the Vital parameters of athletes or newborns, where temporal color fluctuations - as mentioned above, are meant here intensity fluctuations of one or more colors that are reflected in a change in the overall color impression - at a wavelength of around 750 nm, 830 nm or 940 nm, the Determine the pulse rate of the person concerned.
- the oxygen saturation of the hemoglobin bound in the blood can be determined, for which the measured intensities of both wavelengths must be put in relation, i.e. at the wavelength of 830 nm not only color fluctuations, but also temporal intensity fluctuations must be determined. These values must therefore be determined for both wavelengths at the same time in order to be able to measure the relative oxygen saturation of the hemoglobin.
- the two wavelengths are about 150 nm apart, but the two measurement spectral ranges that should be measured in order to also take into account absolute color fluctuations in the wavelength overlap in the range in between.
- a nano-optical filter matrix is used, which is arranged in such a way that a spatially and wavelength-resolved detection of the light emitted by the measurement surfaces is possible.
- the measuring camera 7 generally has an area detector as a sensor, for example a CMOS sensor.
- the nano-optical filter matrix is applied directly to the measuring sensor.
- the nano-optical filter matrix which can be used in the device for the contact-free determination of temporal intensity and color fluctuations in objects of a scene 1, will be explained in more detail below with reference to FIGS. 3-5.
- FIG. 3 a shows the basic structure of a filter element configured as a nano-optical filter matrix 9 with first filter structures 10, which is used to filter a first measurement spectral range.
- Holes 12 with an essentially square shape are arranged in a carrier material 11, for example gold or silicon, wherein the holes also have a round shape or can be designed in the form of slits.
- the holes 12 form an array or a matrix, the 3x3 structure shown here is also referred to as a so-called nanohole array (NHA).
- the holes have a diameter d that corresponds to the length of the edge, and the same edges are spaced apart from one another by a distance P.
- Each of the holes 12 has a few 100 nm in diameter, the size of the structure shown in Fig. 3a is approximately between 10 gm and 15 gm.
- 3x3 structures other sizes are also conceivable for the individual matrix elements, for example 4x4 or 5x5, depending on the measurement situation.
- 3x3 structures are the optimum because, on the one hand, a very large area is made available in each matrix element for measuring one of the spectral ranges, and, on the other hand, the wavelength resolution takes place with a high degree of accuracy in the spatial resolution.
- the wavelength selectivity is essentially influenced by three parameters: the diameter or the edge length d of the holes, the distance - the pitch - P of two holes 12 to one another and the dielectric constant or the electrical conductivity of the carrier material 11 used.
- 3b shows the arrangement of the nano-optical filter matrix 9 on a planar measuring sensor 13, for example a CMOS chip.
- the nano-optical filter matrix 9 is applied directly to the planar measuring sensor 13, for example by bonding. While light of many wavelengths hits the nano-optical filter element, the filter is structured in such a way that it ideally only lets through one wavelength li, with a maximum bandwidth of 100 nm to 150 nm, so that on the one hand the color fluctuations in the wavelength can be registered, on the other hand, however, the transmission ranges of two filter structures overlap as little as possible.
- the nano-optical filter matrix 9 for a specific wavelength is indeed maximally permeable for this wavelength, but the remaining spectral range is also transmitted in the range of a maximum of +/- 75 nm around this wavelength, but is only greatly attenuated.
- the intensity fluctuations are only of interest in a small area around the central peak wavelength at which the filter element has the highest sensitivity or highest transmission.
- FIG. 4 now shows the implementation of a nano-optical filter matrix 9 which is transmissive for two peak wavelengths.
- the area on the left corresponds to the configuration shown in FIG. 3b; this first filter structure 10 is transmissive for a first measurement spectral range.
- a second filter structure 14 is shown which has smaller holes than the first filter structure 10, the distance P being identical in the case of the first filter structure 10 and the second filter structure 14.
- the second filter structure 14 is permeable to a second measurement spectral range around a central wavelength or peak wavelength ⁇ 2. While only the wavelength li is allowed to pass on the left side, the right side is only allowed to pass through for the wavelength ⁇ 2 or for the respective measurement spectral ranges defined around these central wavelengths.
- first and second filter structures 10, 14, as shown in FIG. 5, are arranged in a checkerboard manner - each of the filter structures corresponds to a nano-optical filter matrix 9 - and this checkerboard structure is arranged on the planar measuring sensor 13 and connected to it, is a Detection with spatial and wavelength resolution over the entire measuring sensor 13 is possible.
- Each of the first filter structures 10 and the second filter structures 14 covers an area of one or a small number of several pixels, so that it is ensured that several of the first filter structures 10 and the second filter structures 14 are assigned to each measurement area.
- the installation in the camera is shown by way of example in FIG. 6, here incident light hits a lens system 15 of a measuring camera 7.
- the wavelengths of interest are now spatially and wavelength-resolved via the nano-optical filter matrix 9 consisting of the first filter structures 10 and the second filter structures 14 detected from the direction of the measuring surfaces. Since the corresponding wavelengths are already separated here by means of hardware, less time is required to process the data using software, which leads to increased real-time capability and reduced latency of the measurement.
- the manufacturing process is also simplified with regard to mass production, since the application or structuring of the nano-optics can take place in the same manufacturing step as the insertion of the camera sensor.
- the integration of the nano-optical filter elements on the sensor also increases the robustness with regard to shocks or vibrations.
- the structure can also be made somewhat more compact, since a filter in front of the measuring camera is no longer necessary.
- the use of a camera system is particularly suitable for monitoring people in closed rooms.
- premature babies or newborns can be monitored in a corresponding neonatal ward, or athletes can be monitored in a training room.
- the camera system can be installed as a fixed measuring instrument, for example it can be aligned so that it is pointed from the ceiling or the wall at the person to be monitored, for example on the bed or the changing table in the case of newborns.
- the camera system first records the person's body with the two overview cameras 2, 3. With the simultaneously recorded detail image of the detail camera 6, which is projected onto the overview map generated from the overview images, areas of exposed skin can be marked which correspond to the measurement areas.
- the later one Analysis only the measuring surfaces are considered.
- the measuring camera 7 and the nano-optical filter structures are designed to allow wavelengths around a central wavelength of 830 nm and 660 nm to pass, the first filter structures 10 being transparent to one wavelength and the second filter structures to the other wavelength.
- the number can be derived directly from the temporal color fluctuations - in medical application in this special case it is actually intensity fluctuations at one wavelength - in the near infrared at the wavelength of 830 nm, which can be derived from the consecutive recordings of the measuring surfaces of heartbeats per minute, determine the heart rate.
- the spectral range around a wavelength of about 660 nm is detected by the measuring surfaces, from the ratio of the measured intensities of the two wavelengths - here the maximum values of the intensity distributions are compared - the relative oxygen saturation of the hemoglobin bound in the blood can be determined determine.
- look-up tables these values can be referenced and converted into quantitative absolute values.
- FIG. 7 An example of a measurement of the two wavelengths is shown in FIG. 7 for any point in time when the series of measurement images was recorded.
- the solid line corresponds to the spectral range that is transmitted by the first filter structures 10 with a transmission maximum here at approximately 850 nm.
- the dashed line corresponds to the spectrum that is transmitted by the second filter structures 14 with a maximum at approximately 660 nm.
- the breathing frequency and the breathing volume can also be determined; these values can be derived from the three-dimensional contraction and expansion of the chest.
- Overview images must then be recorded in quicker succession, with a frequency that is at least twice the breathing rate.
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Heart & Thoracic Surgery (AREA)
- Medical Informatics (AREA)
- Molecular Biology (AREA)
- Surgery (AREA)
- Animal Behavior & Ethology (AREA)
- Biophysics (AREA)
- Public Health (AREA)
- Veterinary Medicine (AREA)
- Pathology (AREA)
- Cardiology (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Radiology & Medical Imaging (AREA)
- Physiology (AREA)
- Multimedia (AREA)
- General Physics & Mathematics (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Spectroscopy & Molecular Physics (AREA)
- Optics & Photonics (AREA)
- Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
Abstract
Description
Claims
Applications Claiming Priority (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| DE102019132708 | 2019-12-02 | ||
| DE102020108064.6A DE102020108064A1 (de) | 2019-12-02 | 2020-03-24 | Verfahren und Vorrichtung zur kontaktfreien Bestimmung von zeitlichen Farb- und Intensitätsveränderungen bei Objekten |
| PCT/EP2020/083497 WO2021110531A1 (de) | 2019-12-02 | 2020-11-26 | Verfahren und vorrichtung zur kontaktfreien bestimmung von zeitlichen farb- und intensitätsveränderungen bei objekten |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| EP4069072A1 true EP4069072A1 (de) | 2022-10-12 |
Family
ID=75896819
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| EP20815781.8A Withdrawn EP4069072A1 (de) | 2019-12-02 | 2020-11-26 | Verfahren und vorrichtung zur kontaktfreien bestimmung von zeitlichen farb- und intensitätsveränderungen bei objekten |
Country Status (4)
| Country | Link |
|---|---|
| US (1) | US12053267B2 (de) |
| EP (1) | EP4069072A1 (de) |
| DE (1) | DE102020108064A1 (de) |
| WO (1) | WO2021110531A1 (de) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| BE1027625B1 (fr) * | 2021-02-25 | 2022-05-04 | Iot D | Dispositif multifonctionnel et système multifonctionnel pour la surveillance ergonomique et à distance d'une affection cutanée médicale ou cosmétique |
| US20240388774A1 (en) * | 2023-05-15 | 2024-11-21 | William Chase | Imaging alteration device and method of altering an image |
Family Cites Families (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150379362A1 (en) * | 2013-02-21 | 2015-12-31 | Iee International Electronics & Engineering S.A. | Imaging device based occupant monitoring system supporting multiple functions |
| US20150190077A1 (en) * | 2014-01-07 | 2015-07-09 | Samsung Electronics Co., Ltd. | Electronic device and photoplethysmography method |
| US10915618B2 (en) * | 2014-08-28 | 2021-02-09 | Facetec, Inc. | Method to add remotely collected biometric images / templates to a database record of personal information |
| US10242278B2 (en) | 2014-12-01 | 2019-03-26 | Koninklijke Philips N.V. | Device and method for skin detection |
| US9750420B1 (en) * | 2014-12-10 | 2017-09-05 | Amazon Technologies, Inc. | Facial feature selection for heart rate detection |
| WO2016154256A1 (en) * | 2015-03-25 | 2016-09-29 | Quanttus, Inc. | Contact-less blood pressure measurement |
| JP6665696B2 (ja) * | 2016-06-09 | 2020-03-13 | 株式会社デンソー | 車載装置 |
| KR102398291B1 (ko) * | 2017-03-27 | 2022-05-16 | 삼성전자 주식회사 | 생체 정보를 측정하기 위한 전자 장치 및 방법 |
| EP4183328A1 (de) * | 2017-04-04 | 2023-05-24 | Aranz Healthcare Limited | Anatomische oberflächenbeurteilungsverfahren, vorrichtungen und systeme |
| WO2019157190A1 (en) * | 2018-02-07 | 2019-08-15 | Arthur Wallace | Methods and systems for patient monitoring |
| US20190246920A1 (en) * | 2018-02-14 | 2019-08-15 | Han-Wei Zhang | Intelligent examination device |
| US12560941B2 (en) * | 2019-01-03 | 2026-02-24 | Lucomm Technologies, Inc. | Robotic post |
| EP3838128A1 (de) * | 2019-12-16 | 2021-06-23 | Koninklijke Philips N.V. | Vorrichtung und verfahren zur bestimmung eines vitalzeichens eines probanden |
| US20230027982A1 (en) * | 2021-07-26 | 2023-01-26 | Welch Allyn, Inc. | Regularized multiple-input pain assessment and trend |
-
2020
- 2020-03-24 DE DE102020108064.6A patent/DE102020108064A1/de active Pending
- 2020-11-26 EP EP20815781.8A patent/EP4069072A1/de not_active Withdrawn
- 2020-11-26 US US17/779,972 patent/US12053267B2/en active Active
- 2020-11-26 WO PCT/EP2020/083497 patent/WO2021110531A1/de not_active Ceased
Also Published As
| Publication number | Publication date |
|---|---|
| US12053267B2 (en) | 2024-08-06 |
| DE102020108064A1 (de) | 2021-06-02 |
| US20220409078A1 (en) | 2022-12-29 |
| WO2021110531A1 (de) | 2021-06-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3768157A1 (de) | Vorrichtung und verfahren zur aufnahme und analyse von bildern der haut | |
| DE69227463T2 (de) | Verfahren und vorrichtung zur diagnose von lebenden organismen | |
| EP2851662A2 (de) | Vorrichtung und Verfahren zur Aufnahme eines Hyperspektralbildes | |
| EP1906811B1 (de) | Einrichtung und verfahren zur beobachtung, dokumentation und/oder diagnose des augenhintergrundes | |
| DE60024836T2 (de) | Bilderzeugungsvorrichtung zur darstellung von konzentrationsverhältnissen | |
| DE19850350C1 (de) | Verfahren und Vorrichtung zur Erzeugung von Daten für die Diagnose des Schädigungsgrades von Hautgewebe eines Patienten | |
| DE102009040017A1 (de) | System und Verfahren zur Detektion einer Temperatur auf einer Oberfläche eines Körpers | |
| DE102012216013A1 (de) | Minimal invasive bildbasierte Bestimmung der Konzentration von Kohlendioxid (CO2) in ausgeatmetem Atem | |
| DE4215946A1 (de) | Vorrichtung zum ueberwachen physiologischer daten zur erfassung einer drogenbeeintraechtigung | |
| EP2584957A1 (de) | Vorrichtung und verfahren zur kombinierten optischen und nuklearen bilderfassung | |
| EP1241979A1 (de) | Verfahren und vorrichtung zur bestimmung der örtlichen verteilung einer messgrösse | |
| EP3399914B1 (de) | Vorrichtung und verfahren zur kontinuierlichen und nicht invasiven bestimmung von physiologischen parametern eines probanden | |
| DE112019000207T5 (de) | Optische Erfassung von Reaktionen von Haut und Gewebe mittels Spektroskopie | |
| DE102011084444A1 (de) | Verfahren zur Ermittlung von Begrenzungsinformationen eines Patienten und/oder eines Objekts | |
| EP2218395A2 (de) | Vorrichtung und Verfahren zum Erfassen zumindest eines Vitalparameters einer Person; Vitalparametererfasssungssystem | |
| WO2021110531A1 (de) | Verfahren und vorrichtung zur kontaktfreien bestimmung von zeitlichen farb- und intensitätsveränderungen bei objekten | |
| DE102016218060A1 (de) | An einem handgelenk getragenes gerät zum erfassen eines biosignals | |
| DE102012002086A1 (de) | Verfahren zum Untersuchen von biologischem Gewebe und Vorrichtungen zum Untersuchen und Behandeln des Gewebes | |
| DE112016005019B4 (de) | Endoskopsystem | |
| DE112017005214T5 (de) | Endoskopiesystem | |
| DE102017215158B4 (de) | Detektionssystem | |
| DE10225077B4 (de) | Objektverfolgeanordnung für medizinische Operationen | |
| EP3741290A1 (de) | Vorrichtung zur abbildungserzeugung von hautläsionen | |
| EP0390896B1 (de) | Vorrichtung zur gesichtsfeldprüfung | |
| EP2777491A1 (de) | Verfahren und Vorrichtung zur Überwachung von Vitalfunktionen |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: UNKNOWN |
|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE |
|
| PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
| 17P | Request for examination filed |
Effective date: 20220603 |
|
| AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
| DAV | Request for validation of the european patent (deleted) | ||
| DAX | Request for extension of the european patent (deleted) | ||
| GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
| INTG | Intention to grant announced |
Effective date: 20230725 |
|
| GRAJ | Information related to disapproval of communication of intention to grant by the applicant or resumption of examination proceedings by the epo deleted |
Free format text: ORIGINAL CODE: EPIDOSDIGR1 |
|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
| INTC | Intention to grant announced (deleted) | ||
| GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
| INTG | Intention to grant announced |
Effective date: 20240103 |
|
| STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN |
|
| 18D | Application deemed to be withdrawn |
Effective date: 20240504 |