EP4264183A1

EP4264183A1 - Method for data merging in a heterogeneous network of interconnected devices, heterogeneous network of interconnected devices, computer program product and computer-readable data carrier

Info

Publication number: EP4264183A1
Application number: EP20835733.5A
Authority: EP
Inventors: Mark DEN HARTOG
Original assignee: Robert Bosch GmbH
Current assignee: Robert Bosch GmbH
Priority date: 2020-12-16
Filing date: 2020-12-16
Publication date: 2023-10-25
Also published as: WO2022128075A1

Abstract

The invention relates to a method (200) for data merging in a heterogeneous network of interconnected devices comprising the following steps: °Mapping (201) first measuring data (2011) of a first interconnected device from a first local coordinate system to a common coordinate system using a first map which is based on an algebraic model, °Mapping (202) second measuring data (2012) of a second interconnected device from a second local coordinate system to the common coordinate system using a second map which is based on an algebraic model, °Merging (203) the mapped first measuring data (2013) and the mapped second measuring (2014) data into merged measuring data (2010) in the common coordinate system.

Description

Title

Method for data merging in a heterogeneous network of interconnected devices, Heterogeneous network of interconnected devices, computer program product and computer-readable data carrier

State of the art

“Robust Fusion of LiDAR and Wide-Angle Camera Data for Autonomous Mobile Robots” (de Silva et al., Sensors 2018, 18(8), 2730; https://doi.org/10.3390/s18082730) addresses two fundamental issues surrounding sensor data fusion, namely the resolution difference in heterogeneous sensors and making sense of heterogeneous sensor data streams while accounting for varying uncertainties in the data sources. A geometric model for alignment of the two sensor types are presented, followed by Gaussian process-based matching of resolutions of the two sensors.

Core and advantages of the invention

In a heterogeneous network of interconnected devices, it is often difficult to combine measurements acguired by different interconnected devices, taken at different points in time, to derive new information.

A typical example is the triangulation problem, where the 3D location of an object is inferred by triangulating, using the object’s projected 2D location in two or more views from non-collinear cameras. To solve these kinds of inferences, usually hugely simplified model are used, so it can be redefined as a simple minimization problem (for examples, the DLT algorithm in computer vision). Common simplifications include linearization, using Gaussian noise, assuming all combined measurements are acquired simultaneously and assuming all inferences use a common estimator.

In a network of heterogeneous devices, these assumptions generally do not hold, resulting in loss of information. This is most apparent when dealing with error propagation or with the introduction to new sensor types in an existing network, making it difficult to estimate errors and/or make inferences on quantities that include derivatives (e.g. velocities).

When a physical quantity is measured by sensors (e.g. cameras), there will always be some uncertainty about its true value. This can either be due to intrinsic limitations (e.g. the Heisenberg Uncertainty, shot noise), limited precision of the measurement procedure (e.g. discretization, hysteresis) or the statistical nature of the measured value (e.g. estimating population averages with limited sample size). In science, engineering and (of course) statistics, is therefore imperative to not only provide the numerical value of the measurement, but also give an indication of the accuracy and precision of the measurement. The simplest and most common way is to provide only a limited number of parameters with the measuring data, like the validity range, confidence interval or a standard deviation. For simple, single, physical measurements, like temperature, light intensity or position, the sensor manufacturers will typically provide these numbers in the sensor’s data sheet. Of course, having a limited number of parameters describing the measure of uncertainty implies the underlying distribution is simple, usual Gaussian, Uniform or Poisson.

When inferring information based on multiple, transformed random variables, such a simple statistical model becomes far too limited. For starters, random variables will usually be highly correlated and the transformations non-linear. The resulting probability distributions can generally not be described as a simple 1 D parametrized distribution, so one needs a more complex, multidimensional statistical description. This is usually done by modeling the full system and continuously update the probabilities, when new information is available (like in state estimation). The question rises how one could combine information from more or less independent devices, when one misses the explicit model of relationship between them, like commonly encountered in real-life loT networks? In addition, how can one add a new sensor type, where the acquisition model is unknown at present, and still use these measurements? One could continuously augment the full system model every time significant changes happen, to be able to describe the full joint probability distribution, but this becomes untenable for arbitrary large systems that are designed/implemented/operated autonomously.

One possibility to solve this problem is to combine the measured values heuristically, disregarding any complex stochastic behavior. After the model is created, inference errors normally can only be estimated by synthetically corrupt calibrated inputs and they can evaluate the effects of system-induced perturbations. Simple adjustments by including standard deviations, covariant matrices and/or covariant ellipsoids, could be used, but these can obviously not represent any cross-correlation between devices and all implicitly assuming simple statistical models.

When making inferences of stochastic variables that involve derivatives, like velocities, matter becomes even more complicated. This is due to the fact that tensorial objects like vectors are generally only well defined at one, single location. For example, when two cameras, each tracking the same, moving object, each will measure an angular velocity of the object in their own device specific coordinate system (device coordinate system). However, when inferring the velocity in ‘real world coordinates’ one needs first triangulate the location of the object, before each angular velocity can be uniquely defined. First triangulating the position and then triangulate the velocity, assumes they are both independent. Since this is obviously not the case, one would prefer a (general) approach for solving both at the same time.

The following disclosure provides a framework to augment each measurement vector of interconnected devices, with a full algebraic model of all geometric and probabilistic properties associated with the measurements, instead of exchanging only scalar data (e.g. transformation and covariance matrices). The present invention relates to a method for data merging in a heterogeneous network of interconnected devices, a heterogeneous network of interconnected devices, a computer program product and a computer-readable data carrier.

An advantage of the invention comprising the features of the independent claims is that sending the parameters (measuring data) together with the full algebraic model/ and/or that sending the full algebraic description of the geometric/probabilistic model with each measuring data, allows for transformation, combination and merging of measuring data, with a minimal a- priory assumptions.

This is achieved by a method for data merging in a heterogeneous network of interconnected devices comprising the following steps:

• Mapping a first measuring data of a first interconnected device from a first local coordinate system to a common coordinate system using a first map which is based on an algebraic model,

• Mapping a second measuring data of a second interconnected device from a second local coordinate system to the common coordinate system using a second map which is based on an algebraic model,

• Merging the mapped first measuring data and the mapped second measuring data into merged measuring data in the common coordinate system.

A heterogeneous network is a network which comprises at least two interconnected devices. These interconnected devices could be of the same type, e.g. two heterogenous cameras or different types of sensors, e.g. a LIDAR system and a wide-angle camera, etc. The interconnected devices of a heterogeneous network differ e.g. in their hardware components such as lenses, detectors, etc., the accuracy and precision of the measurement, the software, etc. In general, the interconnected devices of a heterogeneous system differ in at least one property from each other complicating data merging of the measuring data captured by the different interconnected devices.

Measuring data comprise data representing a physical variable, such as an object location, a velocity, etc. in the respective local coordinate system and internal parameters of the respective interconnected device, such as lens aparature or motor positions. The physical variable can be a stochastic variable.

Additionally or alternatively the measuring data comprise probability density functions as an algebraic description and/or algebraic models of the maps. Additionally or alternatively, the measuring data comprise all estimated parameter values of the map that transforms coordinates of the common coordinate system into the device coordinate system (e.g. the location, pose and linear/angular velocities of the device, at the time of measurement) and their joint probability density function expressed in algebraic form.

For example, a moving camera as one of the interconnected devices that tracks an object that itself is moving independently, could send the following measuring data to an analysis unit configured to merge data from different interconnected devices:

• The 2D pixel location of the object and its probability density function expressed in algebraic form

• The 2D pixel ‘velocity’ of the object and its probability density function expressed in algebraic form

• The time of acquisition and the probability density function expressed in algebraic form

• The algebraic representation of the map that transforms coordinates of a global reference coordinate system (=common coordinate system) into the local coordinate system (i.e. the algebraic model that projects a point in ‘world space’(=common coordinate system) to a pixel value on the image (local coordinate system))

• All estimated parameter values of the previously mentioned map (e.g. the location, pose and linear/angular velocities of the device, at the time of measurement) and their joint probability density function expressed in algebraic form.

Sending the parameters of the measuring data together with their semantic model, allows for transformation and combination of different measuring data, with a minimal a-priory assumptions. Under realistic assumptions, the derived inferences can themselves be fully described by daisy chaining algebraic description of the constituent measurements.

The algebraic descriptions and computations could be executed using existing methods, such as opaque computation/data graph format used by existing computational algebra frameworks like in “Naiad: a timely dataflow system”, Deric G. Murray et al, Proceedings of the Twenty Fourth ACM Symposium on Operating system principles, 2013, “TensorFlow: Large-scale machine learning on heterogeneous systems”, Martin Abadi et al. 2015 (www.tensorflow.org) or PyTorch and exchanged in a common Machine learning format like ONNX (Open Neural Network Exchange format). Alternatively, they could be directly implemented as code/scripts, using algebraic software frameworks like Sympy or Mathematica©.

Sometimes a full algebraic description can only be approximated. In those cases, the geometric/probabilistic descriptions could be represented using (e.g. piecewise) defined/interpolated splines, kernels/partition functions, Multidimensional look-up tables, etc., and/or a combination of all mentioned representations.

It is advantageous to send the full algebraic models for the geometric/probabilistic model, next to the actual measuring data and their associated statistical descriptions (e.g. probability density function, standard deviation, covariance matrix, etc.) because sending the full algebraic description of the geometric/probabilistic model with each measurement, would allow each measurement in the network to be used, interpreted and/or combined with other measurements, with minimal a-priory assumptions.

The maps correspond to mathematical mappings. They can be expressed as mathematical functions or transformations. Preferably, they are used in their algebraic representation or are approximated by algebraic descriptions. In other word, the maps are based on algebraic models.

The common coordinate system can be defined as an Euclidean differential manifold. The common coordinate system can be used as a world coordinate system. The mapped first measuring data and the mapped second measuring correspond to the first measuring data in the common coordinate system and the second measuring data in the common coordinate system, respectively.

The effective alignment (either spatially, geometrically or temporally) of multiple measuring data of heterogeneous devices, and utilization of the diversity offered by multimodal sensing is referred to as data merging.

According to an embodiment, the first local coordinate system could be an image coordinate system. In other words, the first local coordinate system is the coordinate system of an object measured by the first interconnected device (also called ‘first screen coordinate system’).

According to an embodiment, the second local coordinate system could be an image coordinate system. In other words, the second local coordinate system is the coordinate system of an object measured by the second interconnected device (also called ‘second screen coordinate system’)

The first measuring data and the second measuring data are mapped to the common coordinate system using the first map and the second map, respectively. Solving the resulting mapping equations with respect to the physical variable of interest gives us the physical variable of interest in the common coordinate system (world coordinates).

F. ex. an object location can be determined by merging the first measuring data and the second measuring data using triangulation. In this example, the object location would correspond to the merged measuring data.

According to an embodiment the first map is an inverse of a third map whereby the third map comprises an algebraic model that projects a point in the common coordinate system to a point in the first local coordinate system or that the first map comprises a first pull-back map comprising an algebraic description.

According to an embodiment, the second map is an inverse of a fourth map whereby the fourth map comprises an algebraic model that projects a point in the common coordinate system to a point in the second local coordinate system or the second map is a second pull-back map comprising an algebraic description.

It is possible to represent maps from the ‘world coordinate system’ to the local coordinate system, possibly augmented with additional (internal) degrees of freedom, but sometimes the inverse (and thus the full Jabobian matrix) can be undefined or at least singular for certain values.

To circumvent this, the measuring data is augmented with an additional algebraic form of the Jacobian (a so-called ‘pull-back’ map) with optionally placeholders for any missing degrees of freedom with respect to the work coordinate system. Advantageously this would allow all interconnected devices to perform the tensorial operations with the maximal knowledge available.

Merging the first and the second measuring data comprising dependent stochastic variables requires integrating over multiple, but constraint variables, when evaluating the estimator operator. If the original algebraic models are available in algebraic form, one can still approximate the estimator, using approximation techniques like Monte Carlo.

According to an embodiment, the first pull-back map comprises at least one placeholder for any missing degree of freedom of the first local coordinate system compared with the common coordinate system. It is an advantage of this embodiment that it allows all devices to perform the tensorial operations with the maximal knowledge available (e.g. estimating the projected form of gradients in the common coordinate systems or triangulating accelerations).

According to an embodiment, the second pull-back map comprises at least one placeholder for any missing degree of freedom of the second local coordinate system compared with the common coordinate system. It is an advantage of this embodiment that it allows all devices to perform the tensorial operations with the maximal knowledge available (e.g. estimating the projected form of gradients in the common coordinate systems or triangulating accelerations).

According to an embodiment the third map comprises • a first algebraic transformation description, mapping the first measuring data of the first interconnected device from the common coordinate system to a first device coordinate system and

• a second algebraic transformation description, mapping the first measuring data from the first device coordinate system to the first local coordinate system and/or the fourth map comprises

• a third algebraic transformation description, mapping the second measuring data of the second interconnected device from the common coordinate system to a second device coordinate system and

• a fourth algebraic transformation description, mapping the second measuring data from the second device coordinate system to the second local coordinate system.

F. ex. in a heterogeneous network of interconnected cameras an image/object is observed by at least two heterogeneous cameras. The image/object position is mapped to the camera coordinate system (device coordinate system) and then it is mapped to the world coordinate system (common coordinate system). These steps are performed for each camera. By solving the resulting mapping equations, the image/object position in world coordinates can be determined. The image/object position in world coordinates corresponds to the merged measuring data in the common coordinate system.

According to an embodiment the first measuring data comprise at least one internal parameter of the first interconnected device, a location, a physical variable, a physical variable involving a derivative and/or a velocity of an object in the first local coordinate system and/or its probability density function as an algebraic description and the second measuring data comprise at least one internal parameter of the second interconnected device, a physical variable, a location, a physical variable involving a derivative and/or a velocity of an object in the second local coordinate system and/or its probability density function as an algebraic description. Attaching the full algebraic description of the geometric/probabilistic model with each measurement, would allow each measurement in the network to be used, interpreted and/or combined with other measurements, with minimal a-priory assumptions. According to an embodiment, the step merging the mapped first measuring data and the mapped second measuring data into merged measuring data in the common coordinate system comprises determining a physical variable, especially the object location, in the common coordinate system by triangulation. It is an advantage that using the measuring data comprising algebraic models and descriptions increases the accuracy of the triangulation result.

According to an embodiment, the step merging the mapped first measuring data and the mapped second measuring data into merged measuring data in the common coordinate system comprises the following steps:

• Mapping first base vectors of the first local coordinate system to base vectors of the common coordinate system,

• Projecting a physical variable involving a derivative, especially the object velocity, to the mapped first base vectors in the common coordinate system, whereby the projections are equal to the physical variable involving a derivative in the first local coordinate system,

• Mapping second base vectors of the second local coordinate system to base vectors of the common coordinate system,

• Projecting the physical variable involving a derivative, especially the object velocity, to the mapped second base vectors in the common coordinate system, whereby the projections are equal to the physical variable involving a derivative in the second local coordinate system,

• determining the physical variable involving a derivative, especially the object velocity, in the common coordinate system by triangulation using the results of the projection equations of the first local coordinate system and the second local coordinate system.

F. ex. a velocity vector in the local coordinate systems is measured in terms of projections to the local base vectors. However, the base vectors of the local coordinate systems are only defined locally and there representation in world space coordinates varies from location to location. However, since we are missing a degree of freedom (depth) we cannot map these vectors back to world coordinate space. However, by using the aforementioned method it is possible to triangulate the velocity vector in a very accurate way. Using this method is possible, because the algebraic models/descriptions are part of the measuring data and they can advantageously be used for algebraically solving the aforementioned velocity triangulation problem.

The method can be used for any arbitrary tensorial object, not just velocities.

According to an embodiment, the first measuring data comprise the algebraic model of the third map and/or the algebraic description of the probability density function. Alternatively or additionally the second measuring data comprise the algebraic model of the fourth map and/or the algebraic description of the probability density function. Sending the measuring data together with their semantic model, makes transformation and combination of measure data possible, with minimal a-priory assumptions. Under realistic assumptions, the derived inferences can themselves be fully described by daisy chaining algebraic description of the constituent measurements.

The algebraic models/ descriptions and computations could be executed using existing methods, such as opaque computation/data graph format used by existing computational algebra frameworks like in “Naiad: a timely dataflow system”, Deric G. Murray et al, Proceedings of the Twenty Fourth ACM Symposium on Operating system principles, 2013, “TensorFlow: Large-scale machine learning on heterogeneous systems”, Martin Abadi et al. 2015 (www.tensorflow.org) or PyTorch and exchanged in a common Machine learning format like ONNX (Open Neural Network Exchange format). Alternatively, they could be directly implemented as code/scripts, using algebraic software frameworks like Sympy or Mathematica©.

Sometimes a full algebraic description/model can only be approximated. In those cases, the geometric/probabilistic descriptions could be represented using (e.g. piecewise) defined/interpolated splines, kernels/partition functions, Multidimensional look-up tables, etc., and/or a combination of all mentioned representations. According to an embodiment, the first measuring data and/or the second measuring data comprise a reference to a database entry with the algebraic descriptions of the probability density functions and/or the algebraic models of at least one of the maps.

It is obviously not necessary that the full geometric/probabilistic description be physically attached to every measurement. This would lead to unnecessary large messages and many unnecessary computations. In a real implementation, one could provide the algebraic models/descriptions once, when needed (e.g., when the device starts) and/or refer to a database entry with common algebraic descriptions/models. Then, the measuring data would only have to contain the measurement and the model parameters at the time of measurement. This advantageously allows to both reducing the message size and reducing the computations needed at run-time.

A heterogeneous network of interconnected devices comprise

• a first interconnected device configured to capture a first measuring data,

• a second interconnected device configured to capture a second measuring data,

• an analysis unit which is configured to receive the first measuring data from the first interconnected device and to receive the second measuring data from the second interconnected device and which is configured to carry out the steps of the method of one of the preceding claims.

The heterogeneous network could comprise at least two interconnected devices of different types and/or different properties.

The heterogeneous network is especially designed to implement the methods as described before.

A further subject matter of the invention is a computer program product implementing the process as described above, preferably in a heterogeneous network as described above. The computer program product comprises instructions, which cause the heterogeneous network of interconnected devices to carry out the steps of the method as described before.

A further subject matter of the invention is a computer-readable data carrier having stored thereon the aforementioned computer program product.

Further features, advantages and effects of the invention will become apparent by the description of preferred embodiments of the invention and the figures as attached. The figures show:

Figure 1 a flow chart of a method for data merging in a heterogeneous network of interconnected devices,

Figure 2 a schematic drawing of a heterogeneous network of interconnected devices,

Figure 3 a schematic drawing of maps connecting different coordinate systems, Figure 4 a schematic drawing of a common coordinate system and a device coordinate system,

Figure 5 a schematic drawing of a device coordinate system and a local coordinate system and

Figure 6 a schematic drawing of the behavior of base vectors in polar coordinates.

Embodiments of the invention

In order to solve a simple data merging problem concerning, one single stochastic variable X, with it probability density function f_x (x). Using the probability density function, we can infer all information of X, like calculating the probability that X falls in the interval [xi, X2]: f^x2

P(.x_± < X < x₂) ⁼ I fx(x)dx x¹

With the expectation operator E[g(x)] = f_Dg(x)f_x(x)dx (where we integrate over the full domain 2) of x), we can calculated the expectation value as:

And the variance: var

If one would transform the stochastic variable X to a new stochastic variable Y, using the map O: x = 0(y), the associated probability density function f_Y(y) would become assuming the inverse map O T x = 0 ¹(y) exists (i.e. the map O is bijective).

So sharing the full algebraic form of _x(x) and the algebraic form (model) of the map O, one can recover the full probabilistic description f_r(y).

Y could be a common ‘reference’ stochastic variable representation in a common coordinate system, e.g. world coordinate system.

If a receiver of the measurement wants to represent the measurement in its own, locally meaningful, representation using stochastic variable Z, one only needs a second invertible map O': z = <p'(y) to define a new map O" = O' ° o^-1 as:

O": z = < >'(< > ^-1(x))

Since both maps are provided in algebraic forms, the derivative — — can now be calculated in closed form.

This recipe can be extended to multiple variables, where X and Z would be reinterpreted as vectors describing the representation of measuring data in arbitrary local coordinate systems and Y would represent a representation of the measuring data in the common coordinate system.

Assuming the number of independent parameters of X, Y and Z would be equal and all maps remain invertible (including the ‘daisy chained’ maps), the one-dimensional illustration could be simply replaced by its multivariable version: is the Jacobian determinant.

Unfortunately, the dimensionality of the ‘coordinate spaces’ might not be the same, nor would the maps always be invertible. For instance, in a camera device, the 3D ‘world coordinates’ are projected on a 2D image plane (local coordinate system, making the distance from the object to the camera unrecoverable. The same could happen when a map becomes ‘singular’, such as in the case of a ‘gimbal lock’, when one degree of freedom is lost for a certain configuration. Another case could arise where some of the coordinates are correlated.

In all mentioned cases we can still represent maps from the ‘world coordinate system’ to the local coordinate system, possible augmented with additional (internal) degrees of freedom, but the inverse (and thus the full Jabobian matrix) is undefined or at least singular for certain values.

To circumvent this, it is possible to optionally augment the measured data with an additional algebraic form of the Jacobian (‘pull-back’ map) optionally comprising placeholders for any missing degrees of freedom with respect to the work coordinate system. This allows all devices to perform the tensorial operations with the maximal knowledge available.

A similar problem occurs when we want to combine multiple, but dependent stochastic variables. This would require integrating over multiple, but constraint variables, when evaluating the estimator operator. If the original algebraic models are available in algebraic form, one could still approximate the estimator, using approximation techniques like Monte Carl.

In Figure 1 a method 200 for data merging in a heterogeneous network of interconnected devices is illustrated by a flow chart. A first measuring data 2011 of a first interconnected device is mapped 201 from a first local coordinate system to a common coordinate system using a first map, whereby the first map is based on an algebraic model. Besides a second measuring data 2012 of a second interconnected device is mapped 202 from a second local coordinate system to the common coordinate system using a second map, whereby the second map is based on an algebraic model. The algebraic models comprise algebraic descriptions of the maps. The measuring data could comprise these algebraic models or they comprise a reference to a database entry where the algebraic models are stored. The mapped first measuring data 2013 and the mapped second measuring data 2014 are merged 203 into merged measuring data 2010 in the common coordinate system. F. ex. the merged measuring data 2010 could be an object location in common coordinates (world coordinates) determined by two different, heterogeneous cameras observing the object. Another example is a velocity of an object.

Figure 2 is a schematic drawing of a heterogeneous network 300 comprising a first interconnected device 301, a second interconnected device 302 whereby each of them is configured to capture measuring data 2011, 2012. Besides the heterogeneous network 300 comprises an analysis unit 303 which is configured to receive the first measuring data 2011 from the first interconnected device 301 and to receive the second measuring data 2012 from the second interconnected device 302 and which is configured to carry out the steps of the aforementioned method 200. The three dots indicate that the heterogeneous network 300 could comprise even more than two heterogeneous interconnected devices 301 , 302. Optionally the heterogeneous network 300 comprises a database 304. The database comprises database entries, which can be addressed by reference. The measuring data 2011 , 2012 can comprise such references to database entries. The database entries comprise many different algebraic models/descriptions, which will be used for the aforementioned method.

Figure 3 is a schematic overview of the maps between the different coordinate systems. In this embodiment, the third map 103 comprises a first algebraic transformation description 111 , mapping the first measuring data 2011 of the first interconnected device 301 from the common coordinate system 100 to a first device coordinate system 1001 and a second algebraic transformation description 112, mapping the first measuring data from the first device coordinate system 1001 to the first local coordinate system 101. The fourth map 104 comprises a third algebraic transformation description 113, mapping the second measuring data 2012 of the second interconnected device 302 from the common coordinate system 100 to a second device coordinate system 1002 and a fourth algebraic transformation 114 description, mapping the second measuring data from the second device coordinate system 1002 to the second local coordinate system 102. The first map 1030 maps the first measuring data 2011 from the first local coordinate system 101 to the common coordinate system 100 and the second map 1040 maps the second measuring data 2012 from the second local coordinate system 102 to the common coordinate system 100. The first map 1030 could be the inverse of the third map 103 and/or the second map 1040 could be the inverse of the fourth map 104. Additionally or alternatively, the first map 1030 comprises a first pull-back map comprising an algebraic description and/or the second map 1040 comprises a second pull-back map comprising an algebraic description.

If a receiver of the measurement wants to represent the measurement in his own, locally meaningful user coordinate system 105 he could use a user map 106 to represent the data in his own coordinate system 105 (this corresponds to the example using the stochastic variable Z, defining a second invertible map 106 O': z = < >'(y) to define a new map O" = O' ° o^-1 see above)

Figure 4 shows a schematic drawing of the common coordinate system 100 (world coordinate system 100) and the first device coordinate system 1001 of the first interconnected device 301. In this embodiment, the first interconnected device 301 is represented by a pictogram of a camera. Besides, the first algebraic transformation description 111 , mapping the first measuring data 2011 of the first interconnected device 301 from the common coordinate system 100 to the first device coordinate system 1001 and the inverse transformation 1110, referred to as in the following, which corresponds to the pull-back map (that means that differential forms in the first device coordinate system 1001 can be pulled back to the world coordinate system) are sketched. The physical location of the device is modeled as an Euclidean differential manifold (aka ‘the world coordinates’) using the usual 3+t dimensional coordinates 4 c ]R³ ® ]R: where the label W indicates that the (coordinate) coefficients should be interpreted in the world coordinate system 100, referred to as O^w in the following. The first interconnected device 301 in the heterogeneous network, here a camera, is marked with an <A. One can define a map < > corresponding to the first algebraic transformation description 111 , which takes a coordinate from the common coordinate system 100, referred to as O^w, to a local coordinate, defined in the first device coordinate system 1001 referred to as O'^A in the following. The map < > depends on the position of the first interconnected device 301 (here a camera), referred to as (where A is the label for the camera position, not the coordinate space!) and its orientation/pose matrix (t)), usually represented as the ‘extrinsic camera matrix’:

Next an additional map corresponding to the second algebraic transformation description 112 is introduced in order to map a coordinate vector from the camera coordinates O^A to a position on an image defining an image coordinate system (aka first local coordinate system 101 , also referred to as ‘screen coordinate system’ O^SA ).

Figure 5 shows a schematic drawing of the maps 1120, 112 connecting the first device coordinate system 1001 and the first local coordinate system 101 referred to as ‘screen coordinate system’ O^SA . The inverse transformation 1120 of the second algebraic transformation description 112, referred to as in the following, corresponds here to a pull-back map pulling differential form in screen space 0^{s A} back to camera space O^M . The map corresponding to the second algebraic transformation description 112 is traditionally called the intrinsic camera map. is non-linear (due to projections, lens properties distortions, etc.) and non-invertible (due to the projection form 3D space (dimension of the first device coordinate system 1001) onto a 2D plane (dimension of the screen coordinate system 101). The full map <5 from the world coordinate system 100 to the screen coordinate system 101 (=first local coordinate system) is then defined by two coordinate functions:

Where x^§A and y^§A are the components of the location in screen coordinates and

..., «« (t) represent all internal parameters of the first interconnected device 301 (pan/tilt/roll angle, focal length, distortion coefficients, etc.) and where <p_x ^A and <p_y ^A are the components of the map

Although the depth information is lost, it is also possible to keep the z^§A coordinate as a placeholder

This would be an example of how to use the .placeholder' parameter. Analogous transformations described for camera Jl and sketched in figures 4 and 5 could also be done for the second interconnected device 302, which could be f. ex. a further camera referred to as camera B, observing the same image. The first interconnected device 301 and the second interconnected device 302, notably camera c/Z and camera B, are heterogeneous devices. In particular, they can differ regarding the internal parameters.

One can define a map corresponding to the third algebraic transformation description 113, which takes a coordinate from the common coordinate system 100, referred to as O^w, to a local coordinate, defined in the second device coordinate system 1002 referred to as O^B in the following. The map depends on the position of the second interconnected device 301 (here a camera), referred to as (where B is the label for the camera position, not the coordinate space!) and its orientation/pose matrix.

An additional map < _b ^sb corresponding to the fourth algebraic transformation description 114 is introduced in order to map a coordinate vector from the camera coordinates O^B to a position on an image defining an image coordinate system (aka second local coordinate system 102, also referred to as ‘screen coordinate system’ O^SB ). The full map from the world coordinate system 100 to the screen coordinate system 102 (=second local coordinate system) is then defined by two coordinate functions:

Where x^§B and y^§B are the components of the location in screen coordinates and a^B (t), ... , a^B(t) represent all internal parameters of the first interconnected device 301 (pan/tilt/roll angle, focal length, distortion coefficients, etc.) and where cp_x ^B and cpy^B are the components of the map

In this embodiment, the first measuring data 2011 comprise at least X^SA, y^SA, and the algebraic representation of O (or a reference on this to a database entry). The second measuring data 2012 comprise at least x^§B , y^§B, ao (t), and the algebraic representation (or a reference on this to a database entry), where x^§B and y^§B are the components of the location in screen coordinates and a® (t), represent all internal parameters of the second interconnected device 302 (pan/tilt/roll angle, focal length, distortion coefficients, etc.).

Using the results of each interconnected device, one can find the vector corresponding to the object location , given the two pixel location vectors q₀ ^§A= (x^SA,y^SA and q₀ ^§B = (x^SB,y^§By , by triangulation, by simultaneously solving the four following equations:

Where </>^Sa = (yp_x ^A , <py^A)^T and (/ ^Sb = (yp_x ^sB , <t>y^By . (For simplicity assuming that, the measurements of the first measuring data 2011 and the second measuring data 2012 are acquired at the same time and all internal state parameters a (t) and a (t) are already known by other means).

However, what will happen if we want to solve for velocities? By definition, a velocity vector in O^§A is measured in terms of projections to the local base vectors x^^A and y ^A of O^§A. Exemplarily the first local base vector 101’ and the second local base vector 101” of 0^{s A} are sketched in Fig. 5. These vectors 10T, 101” are only defined locally and there representation in world space coordinates varies from location to location. However, since we are missing a degree of freedom (depth, because the world coordinate system 100 is 3D and the first local coordinate system 101 is 2D ) we cannot map these vectors back from the first local coordinate system 101 to world coordinate space 100. This is illustrated in figure 6: Figure 6 shows the behavior of base vectors in polar coordinates. The base vector § 407 and the base vector r 408, used to describe velocities, are different at every point (r^s , e^s)^T where r^s corresponds to the first radial coordinate 402 and 6^s corresponds to the first angular coordinate 404. If we would look at the base vectors at a different location, say (r'^s , 9^,S^^T described by a second radial component 401 r'^s and a second angular component 403 6'^s, the base vectors 405 and the base vector r ' 406, defined at that location are generally pointing in the same direction. Comparing to vectors, defined in two different location directly, is only possible in Euclidean space. Formally, for a general differential manifold, we would need to ‘parallel transport’ one vector from point 1 to point 2, using ‘connections’ between individual tangent spaces.

However, we can express each base vector of the screen coordinate system x'_A and y _A in terms of our Euclidean world coordinate system O^w by:

Where x^w is a first base vector 100’ of the world coordinate system 100, y^w is a second base vector 100” of the world coordinate system 100 and z^w is a third base vector 100’” of the world coordinate system 100 as depicted in figure 4. In figure 4 the base vectors 100’, 100”, 100’” form an orthonormal basis. q^w = (x^w,y^w,z^w)^T is an arbitrary location in world coordinates.

We know that the ‘x^§A ‘ component of the object velocity vector cff in world coordinates, is given by in addition, we know that the ‘y^§A ‘ component of the object velocity vector cff in world coordinates, is given by using < , . > the inner product of O^w.

In other words, the object velocity vector q is projected on the base vectors 101’, 101” of the first local coordinate system 101. By solving these equations we will get the object velocity vector qf

Therefore, as long as we can evaluate the Jacobean matrix elements of type at every point in the world coordinate system 100, we can ‘pull-back’ the ‘vector projection functions’ to our common coordinate system, using the pull-back map associated with the differential map This way we can use the Euclidean properties of the world coordinate system to ‘triangulate’ velocities, defined at different locations. Note this recipe also works for arbitrary tensorial object, not just velocities.

Note that we assumed the background to be Euclidean come equipped with an inner product. This is done purely for illustration purposes, since we could replace the inner project < x cff > = x₀ ^sA with the coordinate free representation of the same projection using differential forms, i.e. dx^§A q₀ = x₀ ^sA.

Claims

- 23 - Claims

1. Method (200) for data merging in a heterogeneous network (300) of interconnected devices (301 , 302) comprising the following steps:

• Mapping (201) first measuring data (2011) of a first interconnected device (301) from a first local coordinate system (101) to a common coordinate system (100) using a first map (1030) which is based on an algebraic model,

• Mapping (202) second measuring data (2012) of a second interconnected device (302) from a second local coordinate system (102) to the common coordinate system (100) using a second map (1040) which is based on an algebraic model,

• Merging (203) the mapped first measuring data (2013) and the mapped second measuring data (2014) into merged measuring data (2010) in the common coordinate system (100).

2. Method (200) according to one of preceding claims,

• characterized in that the first map (1030) comprises an inverse of a third map (103) whereby the third map (103) comprises an algebraic model that projects a point in the common coordinate system (100) to a point in the first local coordinate system (101) or

• characterized in that the first map (1030) comprises a first pull-back map comprising an algebraic description.

3. Method (200) according to one of preceding claims,

• characterized in that the second map (1040) comprises an inverse of a fourth map (104) whereby the fourth map (104) comprises an algebraic model that projects a point in the common coordinate system (100) to a point in the second local coordinate system (102) or

• characterized in that the second map (1040) comprises a second pull-back map comprising an algebraic description.

4. Method (200) according to claim 2, characterized in that the first pullback map comprises at least one placeholder for any missing degree of freedom of the first local coordinate system (101) compared with the common coordinate system (100).

5. Method (200) according to claim 3, characterized in that the second pullback map comprises at least one placeholder for any missing degree of freedom of the second local coordinate system (102) compared with the common coordinate system (100).

6. Method (200) according to claim 2 or 3, characterized in that

• the third map (103) comprises

- a first algebraic transformation description (111), mapping the first measuring data (2011) of the first interconnected device (301) from the common coordinate system (100) to a first device coordinate system (1001) and

- a second algebraic transformation description (112), mapping the first measuring data from the first device coordinate system (1001) to the first local coordinate system (101) and/or

• the fourth map (104) comprises

- a third algebraic transformation description (113), mapping the second measuring data (2012) of the second interconnected device (302) from the common coordinate system (100) to a second device coordinate system (1002) and

- a fourth algebraic transformation (114) description, mapping the second measuring data from the second device coordinate system (1002) to the second local coordinate system (102).

7. Method (200) according to one of preceding claims, characterized in that the interconnected devices (301 , 302) comprise at least two heterogeneous cameras.

8. Method (200) according to one of preceding claims, • characterized in that the first measuring data (2011) comprise at least one internal parameter of the first interconnected device (301), a location, a physical variable, a physical variable involving a derivative and/or a velocity of an object in the first local coordinate system (101) and/or its probability density function as an algebraic description and

• characterized in that the second measuring data (2012) comprise at least one internal parameter of the second interconnected device (302), a physical variable, a location, a physical variable involving a derivative and/or a velocity of an object in the second local coordinate system (302) and/or its probability density function as an algebraic description.

9. Method (200) according to claim 8, characterized in that the step merging the mapped first measuring data and the mapped second measuring data into merged measuring data in the common coordinate system comprises determining a physical variable, especially the object location, in the common coordinate system by triangulation.

10. Method according to one of preceding claims, characterized in that the step merging (203) the mapped first measuring data (2013) and the mapped second measuring data (2014) into merged measuring data (2010) in the common coordinate system (100) comprises the following steps:

• Mapping first base vectors (101’, 101”) of the first local coordinate system (101) to base vectors (100’, 100”, 100’”) of the common coordinate system (100),

• Projecting a physical variable involving a derivative, especially the object velocity, to the mapped first base vectors in the common coordinate system, whereby the projections are equal to the physical variable involving a derivative in the first local coordinate system (101),

• Mapping second base vectors of the second local coordinate system (102) to base vectors (100’, 100”, 100’”) of the common coordinate system (101),

• Projecting the physical variable involving a derivative, especially the object velocity, to the mapped second base vectors in the common coordinate system (100), whereby the projections are equal to the physical variable involving a derivative in the second local coordinate system, - 26 -

• determining the physical variable involving a derivative, especially the object velocity, in the common coordinate system by triangulation using the results of the projection equations of the first local coordinate system (101) and the second local coordinate system (102).

11. Method (200) according to one of preceding claims, characterized in that the first measuring data (2011) comprise the algebraic model of the third map (103) and/or the algebraic description of the probability density function and/or characterized in that the second measuring data (2012) comprise the algebraic model of the fourth map (104) and/or the algebraic description of the probability density function.

12. Method (200) according to one of preceding claims, characterized in that the first measuring data (2011) and/or the second measuring data (2012) comprise a reference to a database entry with the algebraic descriptions of the probability density functions and/or the algebraic models of the first map (1030) and the second map (1040).

13. Heterogeneous network (300) of interconnected devices (301, 302) comprising

• a first interconnected device (301) configured to capture a first measuring data (2011),

• a second interconnected device (302) configured to capture a second measuring data (2012),

• an analysis unit (303) which is configured to receive the first measuring data (2011) from the first interconnected device (301) and to receive the second measuring data (2012) from the second interconnected device (302) and which is configured to carry out the steps of the method (200) of one of the preceding claims.

14. A computer program product comprising instructions, which cause the heterogeneous network (300) of interconnected devices (301, 302) of claim 13 to carry out the steps of the method (200) of one of the claims 1 to 12. - 27 -

15. A computer-readable data carrier having stored thereon the computer program product of claim 14.