EP3449463A1

EP3449463A1 - Motion analysis system and motion tracking system comprising same of moved or moving objects that are thermally distinct from their surroundings

Info

Publication number: EP3449463A1
Application number: EP17736558.2A
Authority: EP
Inventors: Andreas Russ; Philipp Russ
Original assignee: Simi Reality Motion Systems GmbH
Current assignee: Simi Reality Motion Systems GmbH
Priority date: 2016-04-25
Filing date: 2017-04-24
Publication date: 2019-03-06
Also published as: US20220237808A1; DE102016107667A1; CA3061269A1; DE112017002158A5; WO2017186225A1

Abstract

The invention relates to a motion analysis system and motion tracking system comprising same of moved or moving objects (90) that are thermally distinct from their surroundings. Said system is characterized by a camera group (11, 12,...; 21, 22,... ) having at least one thermographic camera (11), a calibration unit (51), a synchronization unit (52), a segmentation unit (61), a reconstruction unit (62), a projection unit (63) and an identification unit (64). Such a system advantageously allows a thermally aided segmentation both of the 2D thermogragraphic images and of any 2D video images, specifically irrespective of the ambient conditions of an object (90) to be analyzed and/or to be tracked and without requiring marker elements attached to the object (90).

Description

Motion analysis system and a comprehensive motion tracking system of moving or moving, thermally contrasting from their environment objects

FIELD OF THE INVENTION The present invention relates to a motion analysis system and a motion tracking system comprising moving or moving objects that are thermally different from their surroundings.

Background of the invention

The need for motion analysis and / or tracking systems of moving or moving objects is widespread and in a variety of fields. In particular, movement analyzes are carried out on living objects such as humans in order to improve biomechanics in medicine or sport and to uncover weak points in a movement sequence. A comparable goal is pursued in industrial objects with the analysis of the movements of robot arms or similar grippers. The basis of any movement analysis is the reliable detection, in particular of distance and angle / orientation data, if possible in real time.

Marker-based systems

In many applications, the object to be analyzed is provided with several marker elements. The data acquisition takes place with the aid of video cameras which record the changes in position of the marker elements attached to the musculoskeletal system of the object by means of continuous digital storage of 2D video images by means of at least one video image recorder and make these recordings available to a data processing system for evaluation. One difficulty with these applications is tracking the movement of each individual marker element in the 2D video image in real time and automatically assigning it a unique identity. A system suitable for this purpose was known commercially under the name Aktisys® and described in detail in WO 2011/141531 AI.

Markerless systems In addition to marker-based systems, there is also a high demand in the field of motion analysis and tracking for flexibly deployable systems that can do without attaching additional markers to the target object.

To solve this, motion analysis and / or tracking systems have been developed that aim to locate markerless objects in digital image sequences. In this regard, reference is made by way of example to US Pat. No. 7,257,237 B 1, WO 2008/109567 A2 and WO 2012/156141 Al. Although there are also a variety of different publications on the subject, known markerless motion analysis and / or tracking systems differ very little in the basic approach: all known systems require some form of segmentation of the video images, which are 2D pixel regions generates predefined homogeneity criteria and assigns them to the objects to be acquired.

However, the known, previously existing and employed markerless motion analysis and / or tracking systems function reasonably well only under laboratory conditions with stable and controlled properties of the environment and / or target objects.

However, out of a controlled laboratory environment, an automatic, robust and highly accurate state-of-the-art motion analysis and / or tracking is not available. A stable segmentation in any environment is prevented in particular by

moving objects in the background of video images such as spectators at a sporting event or trees moving in the wind, etc .; and or rapidly and inhomogeneously changing illumination intensities such as moving light sources (headlights), moving shadow sources (clouds) or reflections from moving surfaces (water) etc .; and or

insufficient illumination intensities such as in particular in caves, chambers, or in twilight / night shots etc.

Especially for applications in the outdoor area, all of these points represent major challenges. In the interest of possible applications

However, without a solution to the segmentation problems described above, out- door applications remain closed to the use of motion analysis and / or tracking systems. This includes, for example, the analysis of outdoor sports competitions as well as the behavioral analysis of animals in their natural environment. However, applications in closed spaces (sports competition analysis, security technology, animal research) that are of interest also have to cope with a subset of the obstacles described, as soon as the environment of an object to be analyzed and / or tracked is not tuned to the use of the known systems, as in a laboratory can.

The problem underlying the invention

On this basis, it is an object of the present invention to provide an improved motion analysis system and a motion tracking system comprising moving or moving objects which thermally contrast with their surroundings,

which overcomes the problems known from the prior art when used outside a controlled laboratory environment,

Preferably, without requiring marker elements to be attached to the object. Inventive solution

The object on which the present invention is based is achieved by a motion analysis system and a movement tracking system comprising moving or moving objects that are thermally lifted from their surroundings with the features of the independent patent claims 1 and 12, respectively.

Advantageous embodiments of the invention, which can be used individually or in combination with one another, are specified in the dependent claims.

A motion analysis system according to the invention is characterized by a camera group with at least one thermal imaging camera, a calibration unit, a synchronization unit, a segmentation unit, a reconstruction unit, a projection unit and an identification unit.

A motion tracking system according to the invention comprises such a motion analysis system and is characterized by a motion tracking unit which realigns a model of the object (s) from associated correspondences.

Both systems advantageously allow thermally assisted segmentation regardless of the environmental conditions of an object to be analyzed and / or tracked and without requiring marker elements to be attached to the object. New uses

The present invention opens up hitherto closed application possibilities for motion analysis and / or tracking systems of interest here, in particular in the areas of sports competition analysis, security technology and animal research:

Among other things, sport science has the problem that it analyzes movements primarily in laboratories, but not where the sports movements actually take place: under competitive conditions and in the open field. With markerless acquisition and thermal based segmentation as with the present invention. teaches, it is now possible for the first time to analyze the exact biomechanics of, for example, a football player at the moment of his cruciate ligament injury and to follow its movement. This and similar information is of high relevance to sport, in particular with regard to explanation and illustration of performance and injury issues.

In addition, the data recorded by the thermal imaging camera or cameras can also be advantageously used for thermographic analysis. Thermographic measurement methods provide the opportunity to directly determine the average skin temperature during a sport activity, as well as to map the muscle groups involved in a movement. As a result, physiological processes of thermoregulation of the body can not only be tracked directly, but also sports-specific questions can be considered in detail during the course of physical activity. In general practice, thermographic analysis is primarily used to detect local sources of inflammation. Since the heat generation and radiation of a healthy body is relatively symmetrical, deviations from this symmetry may indicate injuries and possibly diseases. Thus, for example, diseased blood vessels, the formation of certain cancer cells, dysfunction of the thyroid gland, but also fractures or, in a comparatively reduced heat radiation, circulatory disorders can be detected in thermographic images.

Recent scientific work shows that people can also be identified by their unique gait and movement pattern. Where methods such as fingerprints and facial recognition reach their limits, movement characteristics are less easily circumvented. With a system of markerless motion analysis and / or tracking and thermal based segmentation as taught by the present invention, it is also possible for the first time to employ gait and movement patterns as features in the identification of persons. In the field of research on animal habitat or animal behavior patterns (animal science) as well as preclinical research, the analysis and / or tracing of the movement of animals is an essential research element. New medical procedures are being investigated, particularly in the areas of Parkinson's, paresis and Alzheimer's disease. The attachment of markers to animals is problematic, as these are usually do not accept markers. The markerless registration is particularly in demand here. However, very small coat endings and hides often complicate clean segmentation and motion analysis and / or tracing. Furthermore, specially installed light sources affect the behavior of the animals and thus distort the data obtained. Finally, with the aid of thermally assisted segmentation as taught by the present invention, it has been possible for the first time to circumvent these problems and to offer powerful markerless motion analysis and / or tracking in animals. Brief description of the drawings

Additional details and further advantages of the invention will be described below with reference to preferred embodiments, to which, however, the present invention is not limited, and in conjunction with the accompanying drawings.

1 shows schematically by way of example a flow analysis diagram of a movement analysis and / or tracking system according to the invention; for example, the arrangement of a first group of cameras, which comprises at least one thermal imaging camera and at least two video image cameras;

FIG. 3 shows, by way of example, the arrangement of a second group of cameras, which comprises at least two thermal imaging cameras and optionally video image cameras;

4 shows by way of example the arrangement of a third group of cameras, which exclusively comprises two or more thermal imaging cameras; and FIG. 5 shows, by way of example, the arrangement of a fourth group of cameras, which comprises a multiplicity of heat and video image cameras arranged arbitrarily with one another. Detailed description of the figures

In the following description of preferred embodiments of the present invention, like reference characters designate like or similar components.

FIG. 1 shows (distributed over three sheets: FIGS. 1 a, 1 b, 1 c) a movement analysis and / or tracking system 1 according to the invention by way of example with reference to a flowchart.

In the context of the present invention, "video image cameras" 21, 22,... Refer to devices which record electromagnetic radiation in the visible light range (wavelength range from 400 to 780 nm) by means of special detectors (video image pickup 20) and from the received electrical signals Generating signals (two-dimensional) 2D video images VB These 2D video images VB are then in the form of pixel graphics.

A pixel graphic is a computer readable form of describing an image in which the pixels are arranged in a raster pattern and each pixel is assigned a pixel value. In the case of 2D video images VB in the context of this invention, the pixels are typically assigned a particular color (a particular wavelength of visible light) as the pixel value.

The term "voxel" (volumetric pixel) is accordingly understood to mean a data element ("image" point) in a three-dimensional grid. In the context of the invention, 3D voxel models 93 of the object or objects 90 are made up of segmented 2D pixel regions 91; 92 is reconstructed from recordings of at least two cameras by means of a reconstruction unit 62 according to the invention, i. designed. The design (reconstruction) of a 3D voxel model 93 of segmented 2D pixel regions is sometimes referred to as "space carving".

In the context of the present invention, "thermal imaging cameras" 1 1, 12,... Designate devices which record electromagnetic radiation in the infrared range ("heat radiation", wavelength range: 780 nm to 1 mm), which is particularly of living objects (humans, animals ) is emitted (emitted). The pixel values of the resulting 2D Thermal images WB represent temperature values, which can advantageously also be used for the plausibility of depth information in the 2D heat WB and / or 2D video image VB.

In this regard, the invention makes use of the fact that the body temperature of living objects (humans, animals) normally differs from the temperature of inanimate objects 90 in the environment, so that silhouettes become more vivid from thermal images via so-called thresholding Extracting objects well from the inanimate environment (as a background), environmental factors such as the exposure conditions, the color similarity between the object and the background as well as shading, as they are disturbing for video cameras, are irrelevant in thermal imaging cameras.

Even in strong sunlight and the associated heating of the environment compared to the body temperature of living objects, silhouettes can be advantageously distinguished from the background, in particular by calibrating the temperature range of the image recording to a narrow range around the respective body temperature.

The movement analysis system according to the invention is initially characterized by a group of cameras 11, 12, ..., arranged arbitrarily to one another. 21, 22, ... that the field of view 111, 121, ...; 211, 221, ... of each camera 11, 12, ...; 21, 22, ... with the field of view 111, 121, ...; 211, 221, ... at least one other camera 11, 12, ...; 21, 22, ... of the group intersects so that the set of all fields of view 111, 121,: 211, 211, ... is at least indirectly connected, and

the camera group comprises at least one first and second camera whose objectives 112, 122, ...; 212, 222, ... arranged at a distance x of at least two meters and / or their optical axes 113, 123, ...; 213, 223, ... are aligned with each other at an angle α of at least 45 °,

wherein the first camera is a thermal imaging camera 11 for recording thermal radiation by continuous digital storage of 2D Thermal images WB by means of at least one thermal image Aufhehmers 10, and

being the second camera

another thermal imaging camera 12 is

or

A video image camera 21 is for recording light radiation by continuous digital storage of 2D video images VB by means of at least one video image pickup 20. By means of a calibration unit 51 is - for example, according to known prior art - a simultaneous 3D spatial calibration of all heat- 1 1, 12, ... and possibly video cameras 21, 22, ... with overlapping fields of view 1 1 1, 121, 21 1, 221, ... ensured. In addition, a synchronization unit 52 ensures that the recording, ie the continuous digital storage of 2D heat WB and any 2D video images VB takes place at the same time and / or the recording times of the 2D heat WB and any 2D video images VB are known , In the case of a mixed camera group comprising thermal heat 1 1, 12,... And video image cameras 21, 22, the frequency of the video cameras 21, 22,... Has proven to be an integer multiple of the frequency of the thermal cameras 11 , 12, ... to put. The recording times can be controlled for example by an external trigger signal.

Furthermore, according to the invention, a segmentation unit 61 is provided, which in the 2D heat WB and any 2D video images VB associated 2D pixel regions 91; 92 of the object or objects 90 according to predefined homogeneity criteria segmented, ie determined. The term "segmentation" refers to the generation of contiguous regions by combining adjacent pixels or voxels according to predefined homogeneity criteria. <br/><br/> For segmentation, particularly image processing methods 80 such as background subtraction, edge detection, threshold value methods, region-based methods, and orientation to model silhouettes can be used In this case, 2D methods of thermal imaging WB can optionally use methods other than 2D video images VB - for example Bayesian classifiers. In the context of the invention, "homogeneity criteria" are in particular the pixel and / or voxel values "color" and / or "temperature".

By means of a reconstruction unit 62 according to the invention, segmented 2D pixel regions 91; 92 reconstruct a 3D voxel model 93 of the object or objects 90, i. design. The design (reconstruction) of a 3D voxel model 93 of segmented 2D pixel regions is sometimes referred to as "space carving." In the context of the present invention, the selection of the cameras 11, 12, actually used for "space carving" can be selected. ...; 21, 22,... Are advantageously application-related, that is to say flexible - for example, all 11, 12, 21, 22,... Or even only the thermal cameras 11, 12,... Can always be used. The consideration of the fields of view 111, 121, ...; 211, 221, ... of a plurality of spatially offset cameras 11, 12, ...; 21, 22, whose lenses 112, 122, ...; 212, 222, ... arranged at a distance x of at least two meters and / or their optical axes 113, 123, ...; 213, 223, ... are aligned with each other at an angle α of at least 45 °, advantageously allows the reconstruction of a 3D voxel model 93, in which each 3D voxel the information of several, in different fields of view 111, 121, .. .; 21 1, 221, ... taken from synchronously present 2D heat WB and / or any 2D video images VB united. Synchronization 52 and calibration 51 are necessary prerequisites for the reconstruction unit 62.

The present invention is further distinguished by a projection unit 63, by means of which the 3D voxel model 93, which combines pixels from a plurality of synchronously present 2D heat WB and / or any 2D video images VB, as a reference for a search space SR is backprojected into the 2D heat WB and any 2D video images VB. In this case, the back-projected pixels of the 3D voxel model 93 correspond to the fields of view 111, 121, ...; 211, 221, ... of the respective 2D heat WB and / or any 2D video images VB, in particular segmentations of individual poorly segmentable 2D heat WB and / or any 2D video images VB of good segmentation results of synchronously present 2D -Warm- WB and / or any 2D video images VB from the fields of view 1 11, 121, ...; 211, 221, ... other cameras 11, 12, ...; 21, 22, ... benefit. This has the advantage that the identification of silhouettes 94 of the object or objects 90 in the individual 2D heat WB and / or any 2D video images VB can be limited to the search space SR thus created. Finally, the present invention finally features an identification unit 64, by means of which silhouettes 94 of the object or objects 90 are identified, ie recognized, in synchronously present 2D heat WB and possible 2D video images VB on the basis of the search space SR defined by the backprojection , to let.

The motion analysis system according to the invention advantageously allows thermally assisted segmentation of both the 2D thermal images and any 2D video images, regardless of the environmental conditions of an object 90 to be analyzed and / or tracked, and without requiring marker elements to be attached to the object 90. The present invention thus opens up hitherto closed application possibilities for motion analysis and / or tracking systems 1 of interest here, in particular in the areas of sports competition analysis, security technology and animal research.

As far as the frame rate, i. the number of images per unit of time, which is higher than the frame rate of the captured by means of a video camera 21, 22, ... 2D video images VB 2 by means of a thermal imaging camera 1 1, 12, ... recorded 2D thermal images WB is in a preferred embodiment of Invention proposed a 2D supplementary unit 53, which complements missing 2D thermal images WB so that for each 2D video image VB always a synchronous 2D thermal image WB is present. For this purpose, in practice, for example, a (not shown) Key Frame Inteφolation device for the data of the 2D thermal image WB has been proven.

Insofar as the model frequency, ie the number of generated models per unit of time, the 3D voxel models 93 generated by the reconstruction unit 62 is less than the frame rate of the 2D thermal images WB recorded by means of a thermal imaging camera 11, 12, a video camera 21, 22, ... recorded 2D video images VB, in a preferred embodiment of the invention, an SD supplementation unit 54 is proposed, which complements missing 3D voxel models 93 so that each 2D heat WB and possibly 2D video image VB always a synchronous 3D voxel model 93 is present. Again, this has in practice, for example, a (not shown) Key Frame Inteφolation device for the 3D data of the voxel model 93 proven.

2 shows, by way of example, the arrangement of a first group of cameras which comprises at least one thermal imaging camera 11, 12,... And at least two video image cameras 21, 22,. In Fig. 2 it can be seen how

a thermal imaging camera 11 is provided as first camera and a video image camera 21 as second camera, whose objectives 112; 212 arranged at a distance x of at least two meters and / or their optical axes 113; 213 are aligned with each other at an angle α of at least 45 °,

and as a third camera, a video image camera 22 is provided, the lens 222 is disposed immediately adjacent to the lens 112 of the thermal imaging camera 11 so that the optical axes 113; 223 of both cameras 11, 22 are aligned substantially parallel to each other.

The arrangement of at least one thermal imaging camera 11 in a group of cameras comprising at least two video cameras 21, 22, advantageously permits at least a first plausibility check of insufficiently segmentable 2D video images VB by the segmentation unit 61 and thus the advantageous reconstruction of less faulty 3D cameras. Voxel models 93 are found in the prior art.

FIG. 1 optionally shows an iterative sequence (process) in which the segmentation unit 61 and the reconstruction unit 62 are run through repeatedly. In another preferred embodiment, the movement analysis and / or tracking system 1 comprises a further segmentation unit 61 which additionally takes into account restrictions of the search space SR based on the results of the 3D voxel model of the preceding iteration step and adapts homogeneity criteria to the current iteration step. This advantageously increases the robustness of the system 1, in particular the segmentations of individual poorly segmentable 2D heat WBs and / or any 2D video images VB benefit from good segmentation results in other synchronously present 2D heat WBs and / or any 2D video images VB Fields of view 111, 121, ...; 211, 221, ... other cameras 11, 12, ...; 21, 22, .... In an alternative or cumulative embodiment, the robustness of the system 1 can be further increased by a reconstruction unit 62, which additionally uses in an iterative sequence the segmented 2D pixels regions 91, 92 used for the reconstruction of the 3D voxel model 93 as a function of the current iteration step , the type of camera 11; 21; 31 and / or the quality criteria of the 2D pixel regions. In particular, a reconstruction unit 62 which reconstructs the 3D voxel model 93 on the basis of the depth image TB of a depth image camera 31 has proven itself. As a result, an iterative sequence of search space restrictions is advantageously available, consisting of segmentation unit 61, reconstruction unit 62 and projection unit 63.

3 shows, by way of example, the arrangement of a second group of cameras, which encompasses at least two thermal imaging cameras 1 1, 12,... And possibly video image cameras 21, 22,. In Fig. 3 it can be seen, as for example

as a first camera, a thermal imaging camera 11 and as a second camera, a thermal imaging camera 12 are provided, the lenses 112; 122 arranged at a distance x of at least two meters and / or their optical axes 113; 123 are aligned with each other at an angle α of at least 45 °,

and as a third camera, a video image camera 21 is provided, the lens 212 is disposed immediately adjacent to the lens 122 of the second thermal imaging camera 12 so that the optical axes 123; 213 of both camera 12, 21 are aligned substantially parallel to each other.

FIG. 4 shows, by way of example, the arrangement of a third group of cameras which comprises a plurality of video cameras 21, 22,..., Arranged in any desired manner, in particular two to three, heat 11, 12,..., And in particular 5 to 6 , With fewer cameras, one would conveniently occupy the positions of the cameras in the order of the camera numbers 11, 12, 21, 22, ... or adapt them to the specific needs of the application-specific motion analysis and / or tracking.

The arrangement of a group of cameras encompassing at least two thermal imaging cameras 11, 12,... - as suggested by way of example in FIG. 3 or FIG. 4 - advantageously permits a particularly reliable segmentation of 2D thermal images WB by means of the navigation system. mentation unit 61. Thus, two thermal cameras 1 1, 12 whose optical axes 1, 13, 123, ... are arranged at a suitable angle α already enable the reconstruction of a 3D voxel model 93 purely from 2D thermal images WB. According to the invention, therefore, a reconstruction unit 62 which first reconstructs a 3D voxel model 93 of the object or objects 90 from segmented 2D WB pixel regions 91 alone is preferred.

The 3D WB voxel model 93 thus obtained purely from data of the 2D thermal image WB is particularly reliable insofar as there is a particularly robust restriction of a search space SR in the 2D video images VB of the video cameras 21, 22,. .. supplies. According to the invention, therefore, a projection unit 63 is preferred, which first projects back an SD thermal image WB voxel model 93 as a reference for a search space SR into the synchronously present 2D heat WB and any 2D video images VB. In a further preferred embodiment, the motion analysis and / or tracking system 1 further comprises an allocation unit 65 which assigns points of the identified silhouettes 94 points of known silhouettes 95 of a model MO of the object or objects 90 as correspondence and / or points of the known silhouettes 95 of a Model MO of the object (s) 90 assigns points of the identified silhouettes 94 as correspondence. The model MO advantageously represents a virtual image of the object or objects 90. As a rule, it will be designed as a kinematic chain with associated point grid and possibly further references to sensors. It is thus possible to project the model MO in its current orientation by means of the calibration unit 51 into the 2D heat WB and possibly 2D video images VB and to display the outline, i. the Silhouette 95, to determine.

In this case, in particular, an allocation unit 65 has proven to be useful which, in addition to the correspondences obtained from data of the silhouettes 95, optionally uses data, in particular further sensors 40, any image processing units 80 and / or a depth camera 31, 32, to generate correspondence. These are related to state variables of a model MO, ie properties related to the current orientation of a model MO. As a result, an allocation unit is available, which advantageously additional correspondences by assignment of further state variables of a model MO of the object or objects 90 to data, in particular of further sensors 40, any image processing units 80 and / or a depth image camera 31, 32, ... created. In particular, orientation sensors (gyroscopes), acceleration sensors or active heat markers have proven to be suitable for further sensors 40. At image processing units 80, known face recognizers, pattern recognizers or so-called pattern matchers are preferred. As a depth image camera 31, 32, ... all cameras can be used, which allow the pictorial representation of distances. In this case, each pixel does not receive the color of the object 90 to be seen, as in a video camera 21, 22,... Or the temperature of the object as in a thermal imaging camera 11, 12,..., But the distance of the point of the object 90, is visible in the corresponding pixel. Depth image cameras 31, 32, ... are available in different versions such as:

Stereo cameras;

Structured Light, in which a light pattern, produced by light of the visible or infrared wavelength range, is projected onto the scene to be recorded, recorded with a camera and the depth information is calculated from the distortion of the pattern with respect to the undistorted pattern;

TOF cameras (time of flight), which close the distance from transit time measurements of the light; or

Light field cameras, which also determines the angle in addition to position and intensity of the incident light and thus allows the calculation of depth information.

Likewise, a weighting unit 66 has proven itself, which weighs correspondences created according to fixed predefined and / or variable parameters. In particular, weighting criteria can be implemented, which a user can possibly adjust by means of parameters.

In a further preferred refinement, the movement analysis and / or tracking system 1 further comprises a movement tracking unit 71, which realigns a model MO of the object or objects 90 from associated correspondences.

In this case, a movement tracking unit 71 has proved particularly useful, which in an iterative procedure after each iteration with already existing correspondences and / or re-aligning a model MO of the object (s) 90 with correspondence created again based on an updated model MO until the orientation of the model MO meets a predefined criterion. Such a criterion is met, for example, if the orientation of the model MO changes less than a predefined threshold or if a certain number of iterations has been reached. This advantageously provides new silhouettes 95.

In a further preferred embodiment, the movement analysis and / or tracking system 1 further comprises a movement analysis unit 72, which from a final alignment of a model MO of the object or objects 90 positions, in particular existing knee or other joint angle, and / or movements of or of the objects 90 analyzed.

In a further advantageous embodiment, the motion analysis and / or tracking system 1 can be supplemented by a visualization unit 73. In this case, a visualization unit 73 has proved successful with which either the temperature data of the segmented 2D WB pixel regions 91 can be displayed on the 3D voxel model 93 or on a, in particular final, aligned model MO by means of texture mapping. Such a visualization of the temperature data can advantageously enable thermographic analyzes, in particular during the course of movement of an object 90 to be examined.

Finally, FIG. 5 shows by way of example the arrangement of a fourth group of cameras which exclusively comprises two or more thermal imaging cameras 11, 12, 13,. FIG. 5 shows how the objectives 1 12, 122, 132 of the exemplary three thermal imaging cameras 11, 12, 13 are arranged at a distance x of at least two meters and / or their optical axes 113, 123, 133 to each other below a Angle α of at least 45 ° are aligned. The arrangement of a group of cameras formed exclusively of thermal imaging cameras 11, 12, 13,... Advantageously allows the movement analysis and / or tracking of objects 90 even in complete darkness, which is particularly useful for the investigation of nocturnal animals or in the fight against crime is of interest. The motion analysis and / or tracking system 1 of the present invention advantageously allows for thermally-assisted segmentation of both the 2D thermal images and any 2D video images, regardless of the environmental conditions of an object 90 to be analyzed and / or tracked, and without Object 90 marker elements to be attached requires. The present invention thus opens up hitherto closed application possibilities for motion analysis and / or tracking systems 1 of interest here, in particular in the areas of sports competition analysis, security technology and animal research.

LIST OF REFERENCE NUMBERS

1 motion analysis and / or tracking system

10 thermal imagers

I I, 12, ... thermal imaging cameras

I I I, 121, ... Field of view of the thermal imaging camera (11, 12, ...)

112, 122, ... Lens of the thermal imaging camera (11, 12, ...)

113, 123, ... optical axis (viewing direction) of the thermal imaging camera (11, 12, ...)

20 video image pickup

21, 22, ... video cameras

211, 221, ... Field of view of the video camera (21, 22, ...)

212, 222, ... Lens of the video camera (21, 22, ...)

213, 223, ... optical axis (viewing direction) of the video image camera (21, 22, ...)

30 depth image recorders

31, 32, ... depth image cameras

311, 321, ... Field of view of the depth image camera (31, 32, ...)

312, 322, ... Lens of the depth image camera (31, 32, ...)

313, 333, ... optical axis (viewing direction) of the depth image camera (31, 32, ...) 40 other sensors

51 Calibration unit

52 Synchronization unit

53 2D supplementary unit, esp. Key frame integration unit 54 3D supplementary unit, in particular key frame integration unit

61 segmentation unit

62 reconstruction unit

63 projection unit 64 identification unit

65 allocation unit

66 weighting unit

71 motion tracking unit

72 motion analysis unit

73 visualization unit

80 different image processing units

90 object

91 2D WB pixel region (s) of the object or objects (90)

92 2D VB pixel region (s) of the object or objects (90)

93 3D voxel model of the object or objects (90)

94 identified silhouettes of the object or objects (90)

95 (pre-) known silhouettes of the model (MO) of the object or objects (90)

MO model of the object (90)

WB 2D thermal image

VB 2D video image

TB depth image

SR search space x distance between the lens (112) of the first camera (11) and the

Lens (122) or (212) of the second camera (12) or (21) α angle between the optical axis (113) of the first camera (11) and the optical axis (123) or (213) of the second camera (12) or (21)

Claims

claims

1. Motion analysis system of moving or moving objects thermally lifted from their surroundings (90),

at least comprising:

a group of such arbitrarily arranged cameras (11, 12, ...; 21, 22, ...),

the field of view (111, 121, ...; 211, 221, ...) of each camera (11, 12, ...; 21, 22, ...) coincides with the field of view (111, 121,. ..; 211, 221, ...) of at least one other camera (11, 12, ...; 21, 22, ...) of the group intersects so that the set of all fields of view (111, 121, 21 1 , 21 1, ...) is at least indirectly connected, and

the camera group comprises at least one first and second camera whose objectives (112, 122, ...; 212, 222, ...) are arranged at a distance x of at least two meters from one another and / or their optical axes (113, 123 , ...; 213, 223, ...) are aligned with each other at an angle α of at least 45 °,

wherein the first camera is a thermal imaging camera (11) for receiving thermal radiation by continuous digital storage of 2D thermal images (WB) by means of at least one thermal imager (10), and wherein the second camera

another thermal imaging camera (12) is or

a video image camera (21) is for receiving light radiation by continuous digital storage of 2D video images (VB) by means of at least one video image recorder (20); a calibration unit (51), which performs a 3D spatial calibration of all cameras (11, 12, ...; 21, 22, ...) with overlapping fields of view (111, 121, ...; 211, 221, ... ) ensures;

a synchronization unit (52) which ensures that the recording of 2D thermal (WB) and any 2D video images (VB) takes place at the same time and / or the recording times of the images are known; a segmentation unit (61) which, in synchronously present thermal (WB) and any video images (VB), segments associated 2D pixel regions (91; 92) of the one or more objects (90) according to predefined homogeneity criteria;

a reconstructing unit (62) reconstructing a 3D voxel model (93) of the object or objects (90) from segmented 2D pixel regions (91; 92);

marked by

a projecting unit (63) which backprojects the 3D voxel model (93) as reference for a search space (SR) into the synchronously present 2D heat (WB) and any 2D video images (VB); and an identification unit (64) that identifies silhouettes (94) of the object (s) (90) in synchronous present 2D heat (WB) and any 2D video images (VB) based on the search space (SR) defined by the backprojection.

Motion analysis system according to claim 1, wherein the image frequency by means of a video camera (21, 22, ...) recorded 2D video images (VB) is higher than the image frequency by means of a thermal imaging camera (11, 12, ...) recorded 2D thermal images (WB ), further comprising:

a 2D augmentation unit (53) which adds missing 2D thermal images (WB) such that a synchronous 2D thermal image (WB) is always present for each 2D video image (VB).

A motion analysis system according to claim 1 or 2, wherein the model frequency of the 3D voxel models (93) generated by the reconstruction unit (62) is less than the frame rate recorded by means of a thermal imager (11, 12, ...). 2D thermal images (WB) and / or any 2D video images (VB) taken with a video image camera (21, 12), further comprising:

a 3D augmentation unit (54) which supplements missing 3D voxel models (93) such that a synchronous 3D voxel model is always provided for each 2D heat (WB) and / or any 2D video image (VB) 93) is present.

Motion analysis system according to one of claims 1 to 3, wherein the camera group comprises at least one thermal imaging camera (11, 12, ...) and at least two video image cameras (21, 22, ...),

wherein the first camera is a thermal imaging camera (11) and the second camera is a video camera (21) whose objectives (112; 212) are arranged at a distance x of at least two meters from one another and / or their optical axes (113; are oriented at an angle α of at least 45 °,

and wherein the third camera is a video image camera (22) whose lens (222) is disposed immediately adjacent to the lens (112) of the thermal imager (11) such that the optical axes (113; 223) of both cameras (11, 22) are mutually aligned are aligned substantially parallel.

Motion analysis system according to one of the preceding claims; further comprising:

a segmentation unit (61), which is traversed in an iterative sequence and additionally takes into account search space (SR) restrictions from the preceding iteration step and adapts homogeneity criteria to the current iteration step.

a reconstruction unit (62) which in an iterative sequence additionally uses the segmented 2D pixels regions (91, 92) used for the reconstruction of the 3D voxel model (93) as a function of the current iteration step, the type of camera (11; 31) and / or the quality criteria of the 2D pixel regions. Motion analysis system according to one of the preceding claims, wherein the camera group comprises at least two thermal imaging cameras (11, 12); further comprising:

a reconstruction unit (62) which first reconstructs a 3D WB voxel model (93) of the object (s) (90) solely from segmented 2D WB pixel regions (91).

The motion analysis system of claim 7, further comprising:

a projection unit (63), which first backprojects a 3D WB voxel model (93) as reference for a search space (SR) into the synchronously present 2D heat (WB) and any 2D video images (VB).

Motion analysis system according to one of the preceding claims, further comprising:

an allocation unit (65) which assigns points of the identified silhouettes (94) points of previously known silhouettes (95) of a model (MO) of the object (s) (90) as correspondence and / or points of the known silhouettes (95) of a model (MO) the object (s) (90) assigns points of the identified silhouettes (94) as correspondence.

an allocation unit (65), which optionally in addition to the obtained from data of the silhouettes (95) correspondence data, in particular further sensors (40), any image processing units (80) and / or a depth camera (31, 32, ...) for the creation of Correspondences used.

A motion analysis system according to claim 9 or 10, further comprising: a weighting unit (66), which weighs created correspondences according to fixed predefined and / or variable parameters.

Motion analysis and tracking system

marked by

a motion analysis system according to any one of the preceding claims, and a motion tracking unit (71) which realigns a model (MO) of the object (s) (90) from associated correspondences.

Motion analysis and tracking system according to claim 12, characterized by:

a movement tracking unit (71) which realizes a model (MO) of the object (s) (90) in an iterative procedure after each iteration with already existing correspondences and / or with newly created correspondences until the alignment of the model (MO) predefined criterion met.

Motion analysis system according to one of the preceding claims 1 to 11 and / or motion tracking system according to claim 12 or 13, characterized by:

a motion analysis unit (72) that analyzes positions and / or motions of the one or more objects (90) from a final alignment of a model (MO) of the object (s) (90).

The motion analysis and / or tracking system (1) of claim 14, further comprising:

a visualization unit (73), which for the visualization of the temperature data of the segmented 2D WB pixel regions (91) on the 3D voxel model (93) and / or a, in particular final, aligned model (MO), a texture mapping uses.

3.7

4.7

Fig. 2

11 5.7

Fig. 3

6.7

Fig. 4

11 7.7