LU507794B1 - Pre-processing Point-Clouds to Optimize Data-Processing in Networks - Google Patents

Abstract

A computer (500) identifies property data (PROPERTY) that belong to a physical object that has a relation to technical equipment (400). A pre-processor module (510) of the computer, receives a point-cloud data-set (200) from sensors that are associated with the equipment (400). The point-cloud data-set (200) has N data-points. The pre- processor module (510) applies a transformation to the data-set that includes mapping of calculated functional values to a derived data-set pre-defined mapping rule. A previously trained machine-learning tool processes the derived data-set to identify the property data (PROPERTY).

Description

Pre-processing Point-Clouds to Optimize Data-Processing in Networks

Technical field

[0001] In general, the disclosure relates to data processing by networks, and more in particular, the disclosure relates to computer systems, methods and computer- program products for processing sensor data with the goal to identify properties of objects, and also for controlling technical equipment accordingly.

Background of the Invention

[0002] Generally, computers can represent objects of the physical world by data.

Object data can arrive from sensors, and the computer can arrange the data according to a standardized data structure. For example, the object may reflect light and the computer may receive object data from image sensors (such as from cameras). The data would be structured as images with pixels. Light images could be 2D-images with (x, y) location coordinates, or could be 3D-images with (x, y, z) location coordinates (the pixels would turn into "voxels").

[0003] Other types of sensors may provide further or alternative information. For example, radar or lidar sensors may provide object data that indicate if the object (or a part of the object) is physically moving away from the sensor or approaching the sensor. Simplified, a radar image (in 2D or 3D) would map data-points to coordinates in a 2D or 3D grid structure.

[0004] The skilled person is familiar with other sensor types, such as temperature sensors (to measure, for example heat radiation), light intensity sensors, vibration sensors, acoustics sensors (for example to measure sound distribution by directed microphones).

[0005] The skilled person is also familiar with using sensor data (from one sensor, or from multiple sensors of the same type or of different types) to derive conclusions.

For example, an object might perform a certain movement pattern (such as a human or animal making some gesture for that the computer should identify the semantic).

[0006] Technically, the computer receives the data in an order or sequence that is dictated by the data structure, for example, "row by row" for 2D images, "2D-image by 2D-image" for 3D images, or the like.

[0007] Feature extraction techniques allow computers to process the object data to identify properties of the object. These techniques can be based on machine- learning, and the skilled person is familiar with neural networks (or the like) that are implemented by the computer.

[0008] The network techniques are adapted to the type of object data: extracting features from light images or from radar images can be different in the way the neural network is configured.

[0009] The computer can use the identified property for particular and pre-defined purposes, among them controlling technical equipment.

[0010] For example, the computer may be associated with the equipment, and the object may be a living organism that might interfere with the equipment (or vice versa). The computer would monitor the object to identify properties that are related - for example - to object movement.

[0011] Depending on the identified object property, the computer could act on the equipment. In the example, the interference might not be desired, because the object might collide with the equipment: for example, an animal may crash into a vehicle, a human operator may reach the operation zone of an industrial machine.

The computer would instruct the equipment to stop or to move away from the object.

[0012] However, processing data to act on the equipment faces two conflicting challenges (in terms of data and time). Much simplified, the amount of data that the computer receives may be relatively large so that the computer can not act in due time.

[0013] Data that is structured as an image may allow the computer to identify an object property, but the computer would have to process the image completely. As the computer would process the data completely, it would process all pixels (or voxels) of the image (i.e., of the light image, or the radar image or both). Many pixels would be "empty" but would nevertheless have to be processed.

[0014] Consequently, data processing might take relatively long (and even too long). For example, an object-equipment collision might be unavoidable because the computer would fail to identify the property in due time. Such a late prediction of a collision would not allow the computer to trigger appropriate prevention measures.

[0015] The skilled person may address the challenges by upgrading the computer to use further memory, to use further processing cores, and so on. Having more hardware resources might allow data processing in parallel thus speeding up processing. However, for certain equipment such hardware updates may not be possible.

[0016] New data structures have become available in recent years, among them so-called point-clouds or the like.

[0017] In a particular scenery (or space), individual elements of the object (i.e., reality points) are associated with data, but other elements are not associated.

Metaphorically, the plurality of data-points forms the point-cloud. The order by that the computer receives data can be different.

[0018] Computer architectures for processing data from point-clouds, in order to classify an object, to segment (an object) etc. are known in the art, for example from

Qi, Charles R., et al.: "Pointnet: Deep learning on point sets for 3d classification and segmentation." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.

Summary of the Invention

[0019] A computer-implemented method enables a computer to identify property data that belong to a physical object that has a relation to technical equipment.

[0020] By a pre-processor module, the computer is receiving a point-cloud data- set from one or more sensors that are associated with the technical equipment. The point-cloud data-set represents elements of the physical object. The point-cloud data-set has N data-points, wherein each data-point comprises a multi-value tuple with a number D of values. The number D corresponds to the number of dimensions.

[0021] By the pre-processor module, the computer is applying a transformation to the point-cloud data-set by applying dimension-specific numerical values to each data-point separately. This results in a transformed data-set that is a matrix S with a number of R intermediate N-tuples.

[0022] By the pre-processor module, the computer - according to a pre-defined mapping rule - is mapping the intermediate N-tuples - separately - to derived Q- tuples of a derived data-set.

[0023] By a previously trained machine-learning tool, the computer is processing the derived data-set to identify the property data.

[0024] Optionally, receiving the point-cloud data-set comprises to receive the point- cloud data-set as input matrix X as a D x N matrix, wherein applying the transformation comprises to multiply a transformation matrix P as a R x D matrix with the input matrix X, resulting in the matrix S as a R x N matrix.

[0025] Optionally, receiving the point-cloud data-set from the one or more sensors comprises to receive the point-cloud data-set from a sensor arrangement with multiple channels, wherein the dimensions correspond to the multiple channels.

[0026] Optionally, mapping the R intermediate N-tuples to the derived N-tuples according to a pre-defined mapping rule comprises mapping the calculated functional values that are the intermediate N-tuples with tuple-elements to characteristic expressions such that intermediate N-tuples with tuple-elements that are identical in number and type are mapped to the same characteristic expression.

[0027] Optionally, mapping comprises to sort the elements of each row in the matrix

S selectively in ascending or in descending order.

[0028] Optionally, mapping comprises to calculate an offset and to normalize the elements of the intermediate N-tuples accordingly.

[0029] Optionally, processing the derived data-set to identify the property data is performed by classifying.

[0030] Optionally, wherein the computer further provides a control command to the technical equipment, based on the identified property data.

[0031] Optionally, the derived data-set is processed by the previously trained machine-learning tool in the form of a concatenation.

[0032] Optionally, the steps applying the transformation to the point-cloud data-set and mapping are performed for dimension groups separately after grouping data- channels from the one or more sensors.

[0033] Optionally, the pre-processor module is receiving the point-cloud data-set with different data-sub-sets, with the data-sub-sets corresponding to multiple portions of the object that are different in time or different in space, and applying the transformation to the different sub-data-sets separately.

[0034] Optionally, the method identifies a property of the physical object in relation to the technical equipment, wherein the relation and the pre-processing are selected from the following: the physical object is moving in relation to the technical equipment; the technical equipment is moving in relation to the physical object; and both technical equipment and the physical object are Moving.

[0035] The use of the computer-implemented method by a computer to determine a property of a physical object in relation to technical equipment is disclosed. The computer is associated with the technical equipment to control the technical equipment based on the determined property of the physical object.

[0036] A computer system is associated with technical equipment and is adapted to execute the computer-implemented method, to identify property data that belong to a physical object that has a relation to technical equipment, and to use the property data to control the technical equipment based on the determined property of the physical object.

[0037] A computer program product that - when loaded into a memory of a computer and being executed by at least one processor of the computer causes the computer to perform the steps of the method.

Brief Description of the Drawings

[0038] Further details and advantages of the present invention will be apparent from the following detailed description of not limiting embodiments with reference to the attached drawing, wherein:

FIG. 1 illustrates a physical object in its relation to technical equipment, and symbolizes data that represent elements of the physical object;

FIG. 2 illustrates the technical equipment as it sends data to a computer with a machine-learning (ML) tool to provide object property data;

FIG. 3illustrates a flow-chart of a computer-implemented method to identify property data that belong to the physical object;

FIG. 4 illustrates an example for a point-cloud data-set in matrix notation X, a transformation matrix P, as well as a transformed point-cloud data-set in notation of an intermediate matrix S;

FIG. 5 illustrates the mapping step of the method by showing an example in that the intermediate matrix can be sorted;

FIG. 6 illustrates an object as well as illustrates a point-cloud data-set and a derived data-set;

FIG. 7 illustrates the grouping of data-channels and processing data-channels separately; and

FIG. 8 illustrates an object that splits into multiple objects.

Description of Preferred Embodiments

Writing Convention

[0039] The description uses uppercase letters (e.g., D, N, R) for numbers, such as for the number of elements in a set (i.e., the cardinality of the set), and uses corresponding lowercase letters (e.g., d, n, r) as indices. The term "Q-tuple" with Q =D, N, Q, R etc. stands for a set of numerical values with cardinality Q (i.e. a finite sequence in that every element appears only once). Tuples can also be considered as vectors.

[0040] The description also uses uppercase letters for matrices (e.g., X, P, S), and the number of rows and columns is symbolized by writing "rows x columns". The matrices could be transposed if needed, for calculations or for simplicity of understanding (e.g., PT in FIG. 4).

[0041] Occasionally, matrices are written as X={[]1,[12, … [J.n, … [IN }, wherein the [ ] stand for columns (n = 1 to N). Or, matrices can be written as, for example, P € RR*D.

[0042] As the mathematical elements of matrices can be identified by double indices, the description separates out index n by a small dot to the left (".n").

[0043] FIG. 1 illustrates physical object 100 in its relation to technical equipment 400. At least one sensor 410 (i.e., a single sensor or an arrangement of multiple sensors) is associated with technical equipment 400. Sensor 410 provides point- cloud data-set 200 with data that represent elements of physical object 100 by multi- dimensional data-points [x].n. In the simplified example, FIG. 1 illustrates physical object 100 symbolically by a circle, and illustrates elements E1, E2 and E3 by black dots.

[0044] The number of 3 elements in object 100 is just a simplification. As the data- points [x].n correspond to particular elements, data-point [x].1 corresponds to E1, data-point [x].2 corresponds to E2, and data-point [x].3 corresponds to E3. In other words, real-world points or elements (at least some, but not necessarily all) are represented by data-points.

[0045] Sensor 410 provides data in repetitions. As used herein, sensor 410 provides a first point-cloud data-set 200|t1 at a time-point t1, a second point-cloud data-set 200|t2 at a time-point t2, and - more in general - point-cloud data-set 200|tm at time-point tm. The time-points can occur at pre-defined intervals delta_t = t(m+1) - tm (i.e., equidistant intervals), but the time-points can show any other time intervals. Sensor 410 may even provide data only for some occasions (i.e. not at pre-defined intervals).

[0046] As the attribute "point-cloud" already suggests, data-set 200 may lack inherent structure, mostly in the sense that sensor 410 can provide the data-points in different orders.

[0047] As illustrated by square brackets [ ], each data-point [x].n is structured as a

D-tuple with tuple-elements (such as coordinates and/or measurement data) for that the structure is pre-defined and remains unchanged.

[0048] In contrast, as illustrated by curly brackets { }, data from multiple data-points may arrive (at a single time-point tm) as point-cloud data-set 200 from sensor 410 in any order.

[0049] The orders can be a quasi-standard arrival order { [x].1, [x].2, [x].3 } or an alternative arrival order { [x].2, [x].1, [x].3 }, or { [X].2, [X].3, [X].1 }, etc. That arrival order may not be deterministic, but could be a random order.

[0050] When sensor 410 continues to operate in repetitions, the arrival order may change at random, for example data-set 200|t1 in a first order (e.g., quasi-standard with the indices corresponding to the order), the next data-set 200|t2 in a second, different order.

[0051] Point-cloud data-set 200 is being forwarded to a computer for processing, with the general goal to extract one or more features from the data. In other words, there is a goal to identify one or more properties of object 100 as it might interact with technical equipment 400. But the random arrival order does influence the processing as a technical constraint.

[0052] For simplicity of illustration, FIG. 1 symbolizes data forwarding by an arrow to the right, but omits the computer.

[0053] FIG. 2 illustrates technical equipment 400 that sends point-cloud data-set 200 to computer 500. Computer 500 comprises a machine-learning (ML) tool 520 that provides object property data (or PROPERTY in short). PROPERTY can be data that represents the status of equipment 400 (as a technical system in interaction between object and equipment), and can be data to control equipment 400 (in the function of feedback or the like, arrow to the left).

[0054] The skilled person is able to select the type of tool according to PROPERTY that are of interest. In implementations, the machine-learning (ML) tool can be an artificial neural network, a tree-based model, a support vector machine (SVM) or another tool that is able to provide the property data. ML-tool 520 applies machine- learning techniques. ML-tool 520 is a previously trained tool (i.e., training has been completed in advance, also "pre-trained"). For simplicity of explanation, the description explains implementations in that ML-tool 520 is a neural network, such as a convolutional neural network (CNN).

[0055] In that case, ML-tool 520 can be a network with, for example from the input to the output: - two layers to perform 3D convolution, - a flatten layer, - a first dense layer with a relatively high number of nodes, - a second dense layer with a relatively high number of nodes, and - a softmax layer for a prediction (that can be a classification).

[0056] The skilled person can use other configurations. As such neural networks with multiple layers are known, a figure is not presented.

[0057] Simplified, it takes computer 500 a certain time interval AT from receiving point-cloud data-set 200 to providing PROPERTY. That interval AT is a delay that can be relevant for controlling equipment 400. (The skilled person knows that AT is also related to delta_t). AT can have an analogy to the reaction time that takes a human in reacting to a sudden event.

[0058] Details for computer 500 (that executes a computer-implemented method to identify a feature of the physical object) will be explained with FIGS. 3-5.

Computer 500 uses pre-processor module 510 (in FIG. 2, highlighted in bold) to pre- process data-set 200 before processing data by ML-tool 520). Pre-processing leads to derived data-set 250 that is equivalent in the sense to preserve the data (from data-set 200) at least in a quality that allows to obtain PROPERTY.

[0059] Although AT is the sum of AT1 (pre-processing) and AT2 (processing to identify PROPERTY), pre-processing allows to keep AT2 relatively short so that AT is relatively short as well.

[0060] At the first glance, adding a pre-pre-processing step (with AT1) appears as a paradox. But pre-processing compensates the mentioned lack of structure in the point-cloud (i.e., the changes in the arrival order) by transforming the point-cloud to an order-agnostic data structure (i.e., derived data-set 250) that can be processed in a relatively short time interval AT2. In other words, high-speed processing is possible despite sparse data structures. Further, pre-processing keeps the amount of data (i.e., point-cloud data-set 200, derived data-set 250) relatively constant. ML- tool 520 does not have to deal with computation-intensive voxel data or the like.

Scenarios

[0061] For illustration only, the description gives an overview to some scenarios.

[0062] In a first scenario, technical equipment 400 could be an industrial machine (such as a robot) or could be a vehicle. The motion of object 100 (such as a person, or a group of persons) in its relation to the machine or vehicle should be classified.

For example, ML-tool 520 would provide PROPERTY as a binary classification: - "a person in the vicinity of the machine/vehicle is absent", and - "a person in the vicinity of the machine/vehicle is present".

[0063] Based on pre-defined rules, the PROPERTY would eventually allow the equipment to continue operating, or would stop its operation (e.g., to prevent an expected crash with the person).

[0064] For example, the vehicle should have sensors and some computer- supported control mechanism. If there would be a physical thing or person (i.e., object”) in the expected travel direction of the vehicle, the vehicle should slow down or even stop moving. For the industrial machine, the same principle applies for its moving parts.

[0065] The first scenario may have further aspects, such as gesture recognition.

PROPERTY would be a classification in that classes correspond to gesture semantics. While humans make gestures with some intention, the detection algorithm for human-related but non-intentional events would be similar (e.g., if a person falls down or gets into a panic).

[0066] In a second scenario, technical equipment 400 and computer 500 could belong to a surveillance system. It may be installed at an entrance (or exit) of a building to count visitors, at a road to count vehicles, and so on.

[0067] In a third scenario, technical equipment 400 can be an air vehicle. That scenario will also serve as a fictitious example, further below.

[0068] The scenarios have different equipment and different objects, and different relations between the objects and the equipment. But there is one common aspect:

The precision by that the computer identifies PROPERTY is related to the use of

PROPERTY. For example, gesture recognition needs more classes than a binary absence/presence distinction so that precision level would be different.

[0069] The first scenario is related to industrial and traffic safety, but is does not matter if the person in motion is - for example - a man or a woman. The precision to detect the presence of a person (i.e., the object) in reach of the machine or a vehicle is sufficient.

[0070] The second scenario is generally related to security, but again it does not matter what visitors are arriving, or if the vehicles are driven by combustion engines or otherwise.

[0071] In other words, the processing precision may be less relevant than processing speed (relatively short AT).

[0072] The skilled person may reduce the processing speed by providing computer hardware to operate, for example, in parallel. But such an approach may not be desired or may not be possible to take. In general, additional computation efforts should be avoided.

Details for the point-cloud

[0073] The description now turns to details, again in view of FIG. 1.

[0074] As already mentioned, point-cloud data-set 200 comprises N data-points [X].n. Index n identifies a particular data-point (i.e., n = 1 to N). In the example, there are N = 3 data-points [x].1, [x].2 and [x].3 (as mentioned, corresponding to the object elements E1, E2 and E3).

[0075] Each of the data-points [x].n can be written as a D-tuple [x1, x2, ..., xD].n, with each element in the tuple representing a dimension. Hence there are D dimensions.

[0076] The term "dimension" has a technical equivalent in the term "data-channel" and the sensor would be a multi-channel sensor (i.e., a sensor arrangement with multiple channels). In the example, there are D = 3 dimensions (i.e., channels) that are spatial coordinates x1, x2 and x3. In implementations, there could be D > 3 dimensions. Further dimensions (i.e., data-channels) could be, for example, the following: - a value that indicates so-called Doppler-data (if the object element is approaching the sensor or moving away from the sensor), - one or more values that indicate color data (such as color-coding by RGB- values leads to three further dimensions RGB for color), - one or more values for a property that is already identified by sensor 410 (in the example of FIG. 1, the elements that identified by the point-cloud (matrix

X) may have certain properties such as color = black, size = standard size, etc. - one or more values for a signal strength (or intensity), for example, from radar sensors.

[0077] The tuple-elements in [ ] are order-invariant. For example, a sensor would always provide coordinates in a certain order, and would provide Doppler-data at position d = 4. In other words, the order of index d within [ ] does not change because sensor 410 provides data-points without variations. For example, of the sensor provides location data with x1, x2 and x3 being Cartesian coordinates (cf. FIG. 1), and such coordinates are applied to each data-point.

[0078] Point-cloud data-set 200 can also be written matrix X:

X = { [x1, x2, …, xD].1, [x1, x2, ..., xD].2, …, [x1, x2, ....xD].n, …, [x1, x2, ..., xD].N }

[0079] A shorter notation of that set can be

X = { [x].1, [x].2, … [x].n, … [x] N }

[0080] In a likewise notation, matrix X with D rows (from d = 1 to D) and N columns (from n= 1 to N, i.e., columns [ ]) can be written in full. This D x N convention is convenient for explanation herein, but a different convention can be applied as well.

Formula (1) x1.1 x12 xin … x1.N x2.1 x2.2 .. x2n … x2.N xd.1 xd.2 … xdn … xd.N xD. 1 xD2 … xDn … xD.N

[0081] As already mentioned, index d = 1 to D identifies the dimension, and index n = 1 to N identifies a particular data-point [x].n.

[0082] But as explained above, for data-set 200, the picture is different, so the index n does not identify the order by that sensor 410 provides that data-point [x].n.

[0083] Point-cloud matrix X' can comprise the same data-points as point-cloud matrix X and hence stands for the same point-cloud (and the same object), but in a different arrangement or order of the data (e.g. in the columns). The dash here symbolizes a permutation.

[0084] For example, data-points [x].1 and [x].2 may just be swapped. The arrival order becomes different, the computer (such as the neural network or the ML-tool) would have to process data from column 2 before data from column 1. Or, the network would receive data in parallel, the data would be located at different memory addresses or the like.

[0085] There can be many reasons for that, sensor 410 may simply provide [x].2 earlier than [x].1. On other occasions, sensor 410 may return to the original order [x].1, [x].2 etc. It is noted that some sensors provide data when they detect something, otherwise they remain silent. As already mentioned, the order might be a random order.

[0086] It should be noted that the operation of a sensor may be related to signal propagation times (such as a radar signal), and to sensor-internal processing delays. Phase shifts between radio signals or the like may become relevant as well.

Signal reflections (for example of radio waves or sound waves, by the equipment or by other structures such as buildings) may influence the arrival of data as well. In a further example, sound waves have different propagation speeds in different media (e.g., air and water) so that the arrival order may be influenced for that reason as well.

[0087] If an ML-tool (such as a neural network) would receive matrix X, the output would be PROPERTY, and if it would receive matrix X', the output would be

PROPERTY". Since both X and X' represent the same object, the ML-tool should provide PROPERTY = PROPERTY", but it may provide PROPERTY # PROPERTY".

Consequently, the reaction (i.e., CONTROL, FEEDBACK etc.) may be different.

[0088] Or - depending on the training - the ML-tool may provide PROPERTY =

PROPERTY" correctly, but would need more time (when it processes the data, also during training) to accommodate the permutations.

Excurse to highlight an aspect of the problem

[0089] The description here takes a very short excurse to written communication between humans. There is a certain analogy between data-points and letters, and between point-cloud data and words. Individual letters would keep their internal structure [ ], for example, the letter "R" would not turn into "A", and letter-sequences { } within the words should not be altered.

[0090] For example, the reader of this text would easily interpret the sequence { [M] [A] [T] [R] [1] [X] } in the meaning as a mathematical convention. The reader might interpret a permutation such as { [M] [I] [T] [R] [A] [X] } likewise, but that interpretation would be based on experience (with an analogy to training neural networks to be error agnostic) and certainly the reader would need some extra time to understand the meaning. Other readers might not understand the permutation at all, and might even misinterpret it.

Pre-processor

[0091] In terms of point-cloud data, the skilled person would apply voxelization techniques to avoid that, but with additional computation efforts.

[0092] Pre-processing by module 510 helps here, module 510 provides input to

ML-tool 520 and that input is derived data-set 250 that is - (I) non-specific to the column order (i.e. to permutations or the like in the arrival order), but - (ii) still relatively easy to process by ML-tool 520 (no voxelization required).

[0093] To explain the operation of pre-processor module 510, the description refers to the point-cloud matrix X (D x N matrix of formula (1)). Data for a particular point [X].n is available in all its D dimensions in the column with index n.

[0094] Point-cloud data-set 200 - if processed according to the computer- implemented method (600 in FIG. 3) - conveys one or more features of object 100 in relation to technical equipment 400.

[0095] In a simplified example, object 100 could be a flock of birds that is flying within industrial premises such as in a manufacturing hall (i.e., equipment 400).

[0096] The example has some aspects for the above-mentioned scenarios: safety to avoid collisions and potentially some surveillance. Much simplified, the existence of a data-point [x].n in point-cloud data-set 200 would indicate the existence of a bird. (Other dimensions such as Doppler etc. could be considered, but the description omits them for simplicity.) This is - of course - not yet the PROPERTY.

[0097] For processing point-cloud data-set 200, the computer can use - in principle - multiple approaches to represent the data.

[0098] In a first approach - the voxel model - the D dimensions provide a coordinate system, here with a scale accuracy of 5 coordinate values (e.g., 0, 1, 2, 3, 4) per dimension: 5 * 5 * 5 = 125 (i.e., 125 coordinate cells). The computer would process 125 values, 3 of them with logical YES, and 122 of them with logical NO. The numbers in this simplified example might appear very low, but more practical implementations have better accuracy and potentially further dimensions. For example, if sensor 410 would measure only N = 200 points (within a certain timeframe or at time-point tm), creating a reasonably fine 3D grid (i.e., voxel model)

could require as many as 1000 * 1000 * 1000 cells, but most of which would remain empty.

[0099] In a second approach, the same information could be given by vectors in space (or arrows). For example, starting from point [x].1 at coordinates (2, 2, 1), there would be a first vector (+2,-2 +1) to reach data-point [x].2, and there would be a second vector (-3,+2, +2) to reach data-point [x].3. The computer would have to process these vectors in a particular order. The data-points and the vectors could be regarded as a graph with nodes (points) and edges (vectors). In a variation of the second approach, the (inverse) distances between points can as serve as weights of the edges.

[0100] In the third approach, and this is the more relevant approach here, the computer might just receive the point-cloud with data-points (here in the example with D = 3 spatial coordinates). - [2, 2, 11.1, - [4, 0, 21.2 and - [1, 2, 4].3 in that first order [ ].1, [ ].2 and [ ].3. The number on the right side of [ ] is index n.

[0101] However, sensor 410 does not have to provide the data in that first order [ 1.11 1.2[ 1.3 (quasi-standard), but sensor 410 could provide the data in a second order that is a different order (e.g., [1.1 [1.3 [ ].2, or[].2[].3[].1, etc). The second order can be regarded as a permutation to the first order, and the third order can be regarded as a permutation of the second order, and so on. Any order would also be representable in matrix X of formula (1).

[0102] The first and second approaches are generally more computationally demanding (by ML-tools, such as neural networks) than direct processing of the point-cloud in the third approach.

[0103] However, tools such as neural networks are sensitive to permutations in the input data. Assuming that the neural network would receive point-cloud data-set 200 (cf. the matrix) directly, permutation may have an undesired effect (e.g., different

PROPERTY, further training required etc.).

[0104] But pre-processing by module 510 compensates for that, and pre- processing makes the identification of PROPERTY tolerant to order changes.

Method to identify property data

[0105] FIG. 3 illustrates a flow-chart of computer-implemented method 600 to identify property data (PROPERTY) that belong to physical object 100 in relation to technical equipment 400. Method 600 is performed by computer 500 with pre- processor module 510 (steps 610, 620 and 630) and by ML-tool 520 (step 640).

[0106] Method 600 comprises receiving 610 point-cloud data-set 200 from one or more sensors 410 that are associated with equipment 400 (by module 510). The description has already explained that point-cloud data-set 200 represents elements of physical object 100, the data-set (200) and has already explained several notations, among them the notation as point-cloud matrix X.

[0107] Method 600 further comprises a step sequence 620/630 to pre-process point-cloud data-set 200 to derived data-set 250 (or pre-processed data-set), also performed by module 510. The pre-processing sequence comprises, - applying 620 a transformation to obtain intermediate N-tuples (in a transformed data-set 220, or matrix S), and - mapping 630 the functional values to (obtain) derived data-set 250 (with Q- tuples) according to one or more pre-defined rules (for example, in rule- implementations by sorting).

[0108] FIG. 3 symbolizes pre-processing by showing data-set 200 in the variants "abc", "bca”, and "cab" (i.e., some permutations) and showing derived data-set 250 as single symbol (1). It does not matter how derived data-set 250 looks like, but it contains the information from data-set 200 to derive PROPERTY, no matter if data- set 200 shows a permutation or not.

[0109] Finally, ML-tool 520 is processing (step 640) the derived data-set to identify the property data (PROPERTY). As it will be explained, point-cloud data-set 200 can be matrix X or can be matrix X', but PROPERTY = PROPERTY".

[0110] It is noted that the skilled person would be able to implement a method with receiving a point-cloud data-set and with using an ML-tool (e.g. a neural network) to identify property data, such as outlined by Qi et al.

[0111] However, applying pre-processing (within transforming and mapping) solves problems that are related to data structure, AT, permutations etc. As pre- processing involves some computations, they are explained next.

Pre-processing

[0112] FIG. 3 also illustrates pre-processing by pre-processor module 510 in steps 620 and 630. FIG. 4 gives an example for step 620 (transforming) and FIG. 5 gives an example for step 630 (mapping, such as by sorting).

[0113] Pre-processing can comprise activities to remove data-outliers, data inconsistencies and the like. The skilled person would clean up the data as early as possible, such as upon receiving point-cloud data-set X (in step 610). It is however also possible to perform such clean-up activities later, to clean up the N-tuples or the Q-tuples. By way of example, such activities can comprise clipping (e.g., value above a threshold are set to the threshold) or normalizing.

Splitting data in an optional approach

[0114] On its right side, FIG. 3 also illustrates an optional approach. Computer- implemented method 600-5A to identify property data comprises the steps of method 600, but introduces splitting the point-cloud data-set X, according to criteria such as dimension of progress of time and others. Splitting can also be considered as grouping or clustering.

[0115] Steps 620 and 630 can be performed separately, steps 620-5 and 630-0 for data in the group à to obtain a first derived data-set and steps 620-A and 630- À for data in the group A to obtain a second derived data-set (i.e., first and second characteristic expressions). Step 640 would be performed after associating the derived data-sets (step 641).

[0116] Details for such data sub-set specific processing will be explained below, for example, for a split or grouping in the dimensions (FIG. 7), and for a split in time (section "dynamically adapting feature functions”.

Computer program

[0117] FIG. 3 also illustrates a computer program or a computer program product.

The computer program product - when loaded into a memory of a computer and being executed by at least one processor of the computer - causes the computer to perform the steps of computer-implemented method 600. In other words, the blocks in FIG. 3 illustrate that the method can be implemented by the computer under the control of the program, and illustrates that the computer has a module to receive the point-cloud data-set (cf. step 610), a module to apply the transformation (cf. step 620, belonging to pre-processor 510, cf. FIG. 2), a module to perform the mapping (step 630, belonging to pre-processor 510 as well, cf. FIG. 2), and a module to process the data (i.e., ML-tool 520 in FIG. 2).

[0118] FIG. 4 illustrates an example for - point-cloud data-set 200 noted as point-cloud matrix X, - transformation matrix 210 (also noted as transformation matrix P), as well as - transformed data-set 220 (also noted transformed matrix S being an intermediate matrix, with N-tuples 221-r).

[0119] Regarding step receiving 610, the details have already been explained. In equivalent notation, point-cloud data-set 200 can be expressed as X € RP*N, wherein D is the number of dimensions of each point [x].n and N is the number of points in the point-cloud (as explained with FIG. 1). (As already explained D = 3 can be implementations with spatial coordinates alone, D =4 can be for spatial coordinates plus a further dimension like Doppler, etc.

[0120] In the example of FIG. 4, data-set 200 is that of FIG. 1 with D = 3.

[0121] Regarding the transformation to use functional values (fr) in step 620 to obtain intermediate N-tuples, module 510 performs the following:

[0122] Feature functions fj, ..., fr can be defined, with R being the number of such feature functions. The individual feature functions fr each lead to scalar values (e.g., implemented by the computer as an integer, as a real number or the like). As the functions have scalar values, there values can be considered to be in 1D (one dimension). The resulting intermediate N-tuples (with these scalar values) are illustrated with references 221-1, 221-2, 221-3, 221-4, 221-5 for r=1 to R=5, or 221- rin general.

[0123] This is - however - not a dimension reduction, but a transformation. By computing several (i.e., R) features fr, there is substantially no loss of information (that would prevent to obtain PROPERTY). In embodiments, the transformation is a linear transformation (by linear transformation matrix P).

[0124] With index r identifying a particular function (within f,, ..., fr) the following notation applies: f.: RP > R . In other words, a data-point [x] with (a number of) D tuple-elements (i.e., dimensions) is being processed for each tuple-element by a pre-defined function and the result is an integer (i.e., a 1D-feature). The calculation is performed for each of the N data-points separately.

[0125] Module 510 generates (a number of) R 1D-features. The pre-defined function.

Formula (2) x > PX can be expressed by a transformation matrix P.

[0126] Transformation matrix 210 or P € RR*P is an R x D matrix, and x € RP is the data-point [x] (the point-cloud 200). (Index n could be added because the pre- defined function in transformation matrix P is applied to each data-point [x].n separately.)

[0127] As already mentioned, that data-point [x].n is represented by a column of point-cloud matrix X. In this notation, the transformation of the point-cloud matrix X (i.e., data-set 200 as a whole) writes as

Formula (3)

S= PX

[0128] Transformed matrix S (S € RR*N ) denotes the output of the transformation, i.e., transformed matrix 220 or S = PXfor a linear transformation. Non-linear transformations are also contemplated. The output of the transformation would be an transformed matrix S, and the matrix elements would be s(r, n).

[0129] For example, for point-cloud data-set 200 with D = 3 dimensions (i.e., just the spatial coordinates x1, x2 and x3, as in FIG. 1), a first feature function f; could be the sum of the values: fi = x1 + x2 + x3 (i.e., the column sum). In terms of transformation matrix P in formula (3), x1, x2 and x3 would have to be multiplied by factors (that are p11 = 1, p12 = 1 and p13 = 1) and summed up.

[0130] More in detail, point-cloud data-set 200 of FIG. 1 would be transformed to intermediate data-set S for r = 1 with s(r, n)

- s(1,1)=2+2+1=5, - s(1,2)=4+0+2=6 - s(1,3)=1+2+4=7

[0131] In case that point-cloud data-set 200 arrives in a permutation (e.g., ( )1, ()3, ( 2), X') the following applies, with s'(r, n) - s'(1,1)=2+2+1=5 - s'(1,3)=1+2+4=7 - s'(1,22)=4+0+2=6

[0132] A second feature function could be defined, for example, as fa = 2*x1 - x2 + x3, or in the example: - s(2,1)= 2*2-2 +1 =3, - s(2,2)= 2*4-0+2=10 - $(23)=2"1-2+4=4

[0133] In case of the permutation, with matrix S', the feature functions would have to be calculated: - s'(2,1) = 2*2 -2 +1 =3, - §'(23)=2"-2+4=4 - s'(2,2)= 2*4 -0 +2 = 10

[0134] The description is simplified here, S for fi and S for f> would be different. It is noted that a number of R functions can be defined (cf. FIG. 4 with R = 5).

[0135] FIG. 4 shows also an example for matrix S with the s(r, n) values just presented, with R rows and N columns. As the terms "row" and "column" are interchangeable, the description uses the term "tuple", the matrix S comprises a number of R N-tuples 221-r. The N-tuples are intermediate N-tuples because they will be processed in the next method step.

Mapping in general and Sorting as an example

[0136] FIG. 5 illustrates the method step mapping 630 in arrow notation / / — and also illustrates two matrices S and S'.

[0137] The calculated values for s (r, n) in intermediate data-set 220 (matrix S) and s' (r, n) in intermediate data-set 220' (matrix S') would be N-tuples (because both matrices are R x N matrices). They could be written as / /, as / / and for further permutations even as / /" and so on.

[0138] In the example, - for r = 1 the N-tuples are /5, 6, 7/ (cf. 221-1) and /5, 7, 6/ (as in the first rows of Sand S') - for r = 2 the N-tuples are /3, 10, 4/ (cf. 221-2) and /3, 4, 10/ (as in the second rows of S and S').

[0139] The elements of these N-tuples are scalar values. The computer would represent them as integer values. (Depending on the functions fr some values could be zero or could be negative.

[0140] The tuple-elements in / / and in / / are identical in type and quantity. For example in N-tuples 221-1 and 221-1', the tuple-elements are: one element of the type "5", one element of the type "6", one element of the type "7" for / /, and are one element of the type "3", one element of the type "4" and one element of the type "10" for //.

[0141] In other words, the tuples / / and // are comparable to a set of playing cards.

In a set with 32 cards, there are for example, 4 "colors", and 8 values for each "color". In that sense, the tuples are shuffled.

[0142] The computer applies pre-defined mapping rules (i.e., a mathematical function in the broader sense, and/or a mapping algorithm) that maps - individual type-and-quantity tuples (/ /, / / at the input, N-tuples) to - characteristic data or characteristic expressions 231-r, such as Q-tuples | | that are characteristic for type-and-quantity (at the output), cf. 231-1, 231-2, etc. in

FIG. 5.

[0143] Symbolically, the / / symbols are just harmonized in that they are given in upright position | |. In other notation, mapping can be expressed as

Ea

[0144] Mapping could also be regarded as coding (or even as providing a signature”).

[0145] While the (intermediate) N-tuples each have N-elements, the characteristic expressions can be Q-tuples, with Q > N (mapping to a larger target), Q < N (mapping to a smaller target) and Q = N (mapping to an equally sized target).

[0146] Intermediate N-tuples 221-r with tuple-elements that are identical in number (as the term already suggests, that would be N) and type (e.g., N-tuple 221-1 with

N=3 elements, one of type "5", one of type "6" and one of type "7", and N-tuple 221- 1' in the first row of S') can be mapped to the same characteristic expression (e.g., item 231-1 with |5, 6, 7].

[0147] A mathematical function for Q = N can be sorting, such as into ascending order | 3, 4, 10] and | 5, 6, 7 |. For sorting, the number of elements in the tuples remains constant: Q = N.

[0148] Other algorithms are possible to apply as well. For example, the tuples / 2, 3, 6, 1, 5, 4/and/ 2, 6, 3, 1, 4, 5 / (not illustrated) could be sorted to - | 1, 2, 3, 4, 5, 6] (ascending), - | 6, 5, 4, 3, 2, 1| (descending) - | 1, 3, 5, 6, 4, 2| (odds ascending, even descending), - | 6, 4, 2, 1, 3, 5, | (even descending to the left, odds ascending to the right)

[0149] For example, the tuples / 4, 0, 6, 0, 4, 4 / and / 4, 0, 6, 4, 4, O / could be mapped to |2, 0, 0, 0, 3, 0, 1| (two O, zero 1, zero 2, zero 3, three, zero 5, one 6, by indicating the numbers of different values).

[0150] There are further possibilities. For example, /3, 4, 10/ could be normalized to a rounded mean value (here: the average of 3, 4 and 10 is about 5), and expressed as the difference to that mean value |-2, -1, 5|. Normalizing to a rounded mean value can be considered as mapping with compensating an offset (e.g., an offset of 5).

[0151] For multiple N-tuples, such as / 3, 4, 10/, / 4, 5, 11/, / 2, 3, 9/, a common offset could be calculated so that the N-tuples could be mapped by normalizing to 19, 1, 7].

[0152] While the intermediate N-tuples are defined by numerical values (because they result from calculations), the mapped Q-tuples (i.e., 231-r) can be characters, such as the characters of the Latin alphabet (Q = 1, A to Z), or even non-Latin characters (cf., (1) in FIGS. 2 and 7). The skilled person is familiar with unicode character encoding (i.e., Q < N, Q = 2 or two bytes for one character). There is no need to relate the N-tuples to a human-understandable semantic, they only need to be distinguishable by the ML-tool (cf. 520).

[0153] In other words, for the ML-tool (i.e. the neural network), the form of the characteristic expression does not matter, provided that the mapping function remains the same (for training, and for the other phases, such as when the method is to be performed).

[0154] In that sense, mapping the general approach but sorting the approach that is more convenient for illustration. The acronym "sort" is just an example for: the mapping has been performed already.

[0155] FIG. 5 illustrates two matrices S and S' (that has been calculated by transforming, items 220 and 220', before mapping) and derived matrices S_sort and

S'_sort (items 230 and 230', after mapping).

[0156] In the subsequent step, mapping 630 the functional values to obtain derived data-set 250, module 510 maps the entries in each row of matrix S to equivalent entries. As in the figure, mapping can be, for example, sorting in descendent or ascendent order), resulting in S_sort € RF*N. Sorted matrix S_sort has the same R x N numbers as transformed matrix S. The same applies for S'_sort.

[0157] FIG. 5 illustrates mapping 630 by directed arrows from matrix S' to matrix

S'_sort. Of course, mapping is applicable for all matrices S, no matter if permutations apply or not.

[0158] The description uses the acronym "sort" as pars per toto for the mapping / / — | | and for the | | result of the mapping. In the example of FIG. 5, the mapping algorithm is sorting from the smallest to the largest elements, row-wise (ascending order).

[0159] It is important to note that mapping (such as by sorting) is performed mutually independent for every row of S, which implies that the column wise correspondence of the columns to a single point in the original point-cloud is lost. In that sense, the mapping is a many-to-one relation (here for S and for S', two matrices) to one sorted matrix S_sort = S'_sort. The mapping does not have to be reversible or invertible.

[0160] In the example of FIG. 5, the rows in matrices S and S' are sorted in descending order. It does not make any difference if that sorting order is "descending" or "ascending".

[0161] However, the crucial effect achieved by this processing is that S_sort is independent of the ordering of the points in point-cloud matrix X. That is, if the columns of point-cloud matrix X are permuted to give another point-cloud matrix X', representing the same point-cloud, the resulting feature matrix S'_sort will be the same. In other words, S_sort = S'_sort.

[0162] In the examples, the sorted order would be 7, 6, 5 (for the first feature) and 10, 4, 3 (for the second feature) no matter of the data-points have arrived in whatever permutation.

[0163] That data (in a concatenation) can be regarded as derived data-set 250 that is processed by ML-tool 520, as described already. For example, derived data-set 250 would be a vector with first row in matrix S_sort (or S'_sort), the second row, the third row and so on.

Implementation aspects for the transformation matrix

[0164] Inthe example of FIG. 4, transformation matrix P is presented in an example with arbitrary integer numbers. The description now discusses some implementation options: - As transformation matrix P is a R x D matrix, D can correspond to the number of dimensions D. - Alternatively, a R x D- matrix could be used with D- < D, in that not all dimensions are used. In other words, D~ indicates that method 600 may be applied to the data only partially (or differentially according to the dimensions).

An example is given further below with two matrices with D1 < D and D2 < D. - The number R of functions is selected according the accuracy by that

PROPERTY has to be provided, but limited by computation (to calculate matrix

S). - Convenient magnitudes of R are between R= 3 and R = N/100. For example, for point-clouds with N = 1.000 data-points to be expected, R would be approximately 10. - The numerical values within transformation matrix P can be positive or negative. - The numerical values can be integers (to keep computation resources low). - It is possible to set a value range from a minimal value to a maximal value, such as, for example, a range [-100, +100]. The example of FIG. 4 uses a range [-10, +10]. - It is possible to select the values at random. - A random selection might lead to situation with "all zero", "all 1", to diagonal matrices, to sparce matrices (zeros in majority), etc. But it is possible to de- limit such situations by setting up rules the check random matrices and to eventually disregard some of them. - To avoid such random effects, the distribution of values can be pre-defined, for example, to equal distribution (with given tolerances) of the matrix elements. - Formula (3) presents a matrix multiplication in that transformation matrix T defines a linear transformation. This does not exclude the possibility to introduce non-linear calculations for some of the data-points.

Dynamic system

[0165] Equipment 400 and object 100 can be regarded as a system. For method 600 to identify property data (PROPERTY, that belongs to physical object 100 that has a relation to technical equipment 400), a change in that system can be expected but it a change should not be a constraint to the method. The system can be a dynamic system because, for example, of the following circumstances: - Equipment 400 and object 100 may change the relative location (i.e., the distance if the equipment and/or the object moves). - Equipment 400 and/or object 100 may change the orientation and or change the way how they move. - Equipment 400 may be vehicle and/or an industrial robot that is approaching object 100 (that would be obstacle), or that is moving away from the object.

[0166] The PROPERTY of the object is expected to change over time as well. For example, some or all of its elements (cf. E1 etc. in FIG. 1) would move, change the color, and so on.

[0167] The description now introduces some optional concepts (or variants) to address such and other constraints: - by using relative data-sets at the input of the pre-processor, - by separating feature functions into groups within the pre-processor (cf. item 510 in FIG. 2). - by processing data for different dimensions (d) separately.

[0168] The description explains details for these concepts separately, but it is noted that the concepts can be combined.

[0169] The skilled person is able to use off-the-shelf tools to provide certain auxiliary data. Some of the tools are widely used even in consumer goods (such as in digital cameras) and/or being developed for autonomous driving. Some examples are given: - Object trackers can identify the physical location of an objects and can even predict a change of the location for the future. The tracker can provide location data or position data. - Object separators can separate data that relates to the objects, so that for example, location data can be available for different parts of the object. For example, if image data of the object is available (not necessarily the point- cloud data-set) individual parts of the objects can be identified. The separator can provide location data as well. In other word, location data can be available for part of the object.

Absolute and relative data-sets

[0170] Changes can be detected by repeating method 600 (cf. FIG. 3) and for each repetition, the result of method execution (property data PROPERTY) would be different. The skilled person can simply monitor how PROPERTY develops.

[0171] It is however possible to consider changes already at the input of pre- processor 510.

[0172] For example, [x].1 = [2, 2, 1], [x].2 = [4, O, 2], and [x].3 = [1, 2, 4] at a first time-point t1 could change to [x].1 = [2, 3, 1], [x].2 = [4, 1, 2], and [x].3 = [1, 3, 4] at a second time-point t2.

[0173] Provided that the arrival order remains (no permutations), by simply calculating differences, an alternative data-set can be calculated as a differential point-cloud data-set 200: [x].1 = [O, 1, 0], [x].2 = [0, 1, 0], and [x].3 = [0, 1, 0]. The point-cloud data-set 200 turns from a set with absolute values to a set with relative values, but for pre-processor module 510 the method steps remain the same.

[0174] Such differential point-cloud data-set 200 could be the input to pre- processor 510 as well.

[0175] It would also be possible to combine both approaches: to use point-cloud data-sets 200 that are based an absolute value and to use point-cloud data-sets 200 that are based on relative values. It is noted that images sensors may provide consecutive images as video streams that use absolute and relative values.

Multiple objects and different feature functions

[0176] FIG. 6 illustrates an object as well as illustrates a point-cloud data-set and a derived data-set.

[0177] It can be assumed that a single sensor (such as sensor 410 in FIG. 1, multiple channels) provides a single point-cloud data-set 200. The elements of object 100 are shown in the "scenery under sensation” that the sensor can monitor.

In the figure, N is about 30.

[0178] As illustrated on the left side of the figure, the data-points [x] would belong to a single object and derived data-set 250 would belong to that single object as well. For example, the single object could be a number of persons that are located in proximity to the equipment (cf. FIG. 1). Or, in view of the bird flock example, derived data-set 250 would be derived from that flock.

[0179] As illustrated on the right side of the figure, the space can be divided into sub-spaces. This is not necessarily performed by the pre-processor (cf. FIG. 2), but can be implemented a computer-vision tool or by similar techniques that the skilled person can apply.

[0180] Sub-spaces are occupied with different objects 100-1, 100-2, and 100-3.

The dashed lines merely symbolize that differentiation. The different objects are physical items that can not be co-located at the same space. They are disjunct in space.

[0181] The feature functions can be divided in groups such as in a first group with

R1 functions ((f,,…, fr1), a Second group with R2 function ((f,,…, fra) and so on.

Each group can be assigned to a certain spatial region in the scenery under sensation. That is, the first group is only applied to points lying in a certain region (i.e., the points representing elements in the region), while the second group is only applied to points lying in another region.

[0182] Consequently, the derived data-set 250 would be a collection of data-sets 250-1, 250-2 and 250-3 (or 250-K, with K in general as the number of groups).

[0183] This concept could be useful in use-case scenarios where points in different spatial regions can be expected to correspond to different persons. Different persons may have different properties. For example, a first person (i.e., object 100- 1) would wear protective clothing (in yellow or the like), the second and third persons would not.

[0184] In relation to the equipment, the object properties may be expected to be different in different spaces. For example, in the vicinity of the equipment, persons with protective clothing may be allowed. The feature group for that space would be specialized to detect the type of clothing (or at least to distinguish protective clothing from normal clothing). For other spaces, other detection might be more important.

[0185] In view of the bird flock example, some of the birds (e.g., that of object 100- 3) would be in proximity of the equipment. Doppler-data can be processed as well, this leading to a transformation matrix P that take the "doppler dimension" into account (for object 100-3) and to a more simple transformation matrix that does not (for objects 100-1 and 100-2 that are in non-critical distance).

Dynamically adapting feature functions

[0186] Specializing feature functions to certain spatial regions (as in FIG. 4) is not the only way to specialize functions.

[0187] The feature functions fj, ..., fr could be defined to depend on the time step in a dynamic point cloud scenario. That is, the feature functions can be written as fıltm ... fr|tm for tm = t1 to tM. The functions can be different for every tm (for example, for M time-slots from t1 to t = tM. This concept could be useful in a scenario where certain objects in the scenario are tracked so that the functions are dynamically adapted to the predicted positions.

[0188] This concept can also be relevant if the system itself undergoes some dynamic changes. As for instance, the system could be mounted on a vehicle or robot and the functions could then be adapted to different locations, orientation, or velocities of the vehicle, resp. robot.

[0189] Certain dimensions - such instance Doppler-data or intensity data - are commonly found in radar point-clouds. Data in these dimensions convey characteristics that are different (compared with spatial information).

[0190] Thus, in some cases it can be more adequate to process these dimensions separately.

[0191] FIG. 7 illustrates the grouping of data-channels and processing data- channels separately. The explanation of FIG. 7 is further supported by the right side of FIG. 3 (method 600-5A)

[0192] Inthe left side, FIG. 7 repeats matrix X (from FIG. 1 with spatial dimension) but expands it by a further dimension, for example, Doppler data (e.g., "5" for n=1, "4" for n =2 and "4" for n=3). In other words, the data is grouped into a first group and a second group, into matrices X1 and Xz (i.e., sub-data-sets) respectively.

[0193] These dimensions can initially be separated, giving X, € RP!*N, X, € RP2xN with D1 + D2 = D so that the concatenation of Xs, Xo, along the second dimension coincides with X. Then X4 is processed as described above to give S_sort_1. The correspondence of the features in the single rows in S_sort_1 to the original points in X resp. Xo is stored. Then subsequently, the remaining dimensions can be accordingly concatenated to the rows of S_sort_1 to give S_sort.

[0194] As in the figure, it can be assumed that there is a first set of dimensions à (e.g., spatial coordinates, cf. dimensions x1, x2 and x3 in FIG. 1), and a second set of dimensions A (e.g., Doppler-data, intensity of radar reflections etc.).

[0195] The overall number of dimensions is D = à + A. Matrix X (cf. formula (1) can simply be split into an upper part (X1, 5 dimensions) and a lower part (Xo, A dimensions):

Formula (4) x1.1 x12 xin … x1.N x2.1 x2.2 .. x2n … x2.N x0.1 x0.2 .. xo.n … x0.N

Formula (5) x1.1 x1.2 xin … x1.N x21 x2.2 .. x2n … x2.N xA1 xA2 … xAn … xA.N

[0196] There are separate transformation matrices (not illustrated), with R x à and with R x À (R could be different as well. The matrix for the upper part leads to S_sort (5) and the matrix for the lower part leads to S_sort (A), cf. steps 620 and 630 that are performed separately. Variations of that approach are possible, for example, to

S_ sort (8) and subsequently to concatenate the values of S_sort (A) to S_sort (5), cf. step 641 associating in FIG. 3.

[0197] Processing data for different data dimensions separately gives further options. For example, if the number of dimensions in each matrix part would be the same (i.e., 5 = A), the sorted intermediate results S_sort could be added, for example element-wise (the first element in S_sort (5) plus the first element in S_sort (A) and so on).

[0198] Introducing pre-defined weights is a further option, for example, to give

S_sort (5) more (or less) weight than S_sort (A) in processing by ML-tool 520.

[0199] But returning to FIG. 7, the upper part (e.g., the à = 3 dimensions with the spatial data) can be processed as described already (method 600, the operations as in FIGS. 4-5 by pre-processor 510-0), and the lower part (e.g., A=1 dimension from the Doppler channel) can be processed by otherwise, here symbolized by pre- processor 510-A.

[0200] At the output of both pre-processors, the derived Q-tuples (i.e., the characteristic expressions) would be concatenated and forwarded both to ML-tool 520 (cf. more in general, step associating 641 in FIG. 3).

[0201] In the example of FIG. 7, examples of derived Q-tuples would be "(1)" (from the spatial data) and "4" (from the Doppler). In this example, pre-processing uses a transformation (in step 620, or rather step 620-A) that is much simplified to obtain the most occurring data (that is 4). For example, the Doppler data "5" and "4" indicate that one element (cf. the elements E1, E2 and E3 in FIG. 1) has a measured speed of 5 and that two elements have the measured speed of 4. In the example,

ML-tool 520 would receive the concatenation "(1)4".

[0202] For any arrival order, the derived value "4" and is an indication that most of the elements in object 100 are related to a particular speed.

[0203] Applying further rules is possible as well: Doppler data may simply be incorrect so that outliers would be disregarded).

[0204] It is also possible to merely bypass the dimension A data (no matrix processing required, no mapping required, method from step 610 to step 640). FIG. 3 illustrates bypassing by a dashed arrow from step 610 to step 641.

Continuation of the description

[0205] Pre-processing point-cloud data-set 200 to derived data-set 250, has been described for data-sets that are static and that have only relatively few dimensions (e.g., D = 3). However, as object 100 changes over time (in its relation to equipment 400), point-cloud data-set 200 would change over time as well.

[0206] Consequently, the complexity rises. The description continues with a fictious example, to explain how computer 500 can accommodate further complexity.

Fictitious example

[0207] Technical equipment 400 could be an unmanned arial vehicle (UAV) that is flying over an agricultural field. The UAV is not flying alone, but it is accompanied by birds that occasionally land on the ground to enjoy the food that the field offers them.

[0208] In a use-case scenario "bird watching", the UAV carries a camera to take light-images of the field and of the birds. A light-image processing computer (that is located at the UAV or that is located remotely) processes the image to identify the kind of birds. That knowledge allows the farmer to take some measures, such as to create some bird-specific sound that invite them to eat somewhere else. Such on- board image processing should be available to the skilled person.

[0209] But the UAV is adapted to a further scenario "collision avoiding". The UAV should have a radar/lidar sensor (cf. sensor 410 in FIG. 1) that captures radar/lidar data from its surroundings. The UAV has been programmed to stay within the borders of the fields so that it should not touch trees or walls that are located outside.

However, the UAV faces a collision danger. Flying birds would be the "elements of the object". The on-board computer executes computer-implemented method 600 (details in FIG. 3, as described) to identify a property of these objects. In this simplified example, property data PROPERTY indicates: (a) that a bird flies away from the UAV, or (b) that a bird approaches the UAV.

[0210] The on-board ML-tool (cf. 520 in FIG. 2) would classify the data (derived data-set 250) to differentiate elements in bird-size (that potentially cause an unwanted collision) from elements in mosquito-size (that are allowed to collide). In other words, this is a simply binary classification.

[0211] The ML-tool (or an auxiliary tool as mentioned) also provides approximate location data of the objects. Consequently, computer 500 can detect the presence of a birds and can instruct the UAV to modify its movement (control signal derived from PROPERTY, to avoid collision).

[0212] In other words, depending on the scenario, the computers process data differently. As there is no collision danger, there is plenty of time for the computers to do bird watching, and even to select some bird-specific measures. However, if a bird is approaching the UAV, the computer can immediately take actions.

[0213] FIG. 8 illustrates object 100 that splits into multiple objects 100-A and 100-

B. To stay with the birds and the UAV, object 100 can be the flock at a first time- point. But the flock is separating, and at a second time-point, there should be objects 100-A and 100-B (i.e., two flocks).

[0214] The above-explained dynamics come into play: the on-board computer would detect a location change with the consequence that object 100-A is more proximate to the UAV than object 100-B. There are two point-clouds to be processed: one for object 100-A and one for object 100-B. As calculation time is a precious resource, the computer can apply a pre-defined rule to consider Doppler- data for the proximate object 100-A, and to ignore Doppler-data for object 100-B. In other words, the explained à-A-split can be used. That saves computation.

[0215] Assuming that the evaluation of the Doppler-data for object 100-A would indicate object movement with a collision danger, the UAV can be directed to stay away from object 100-A.

[0216] Further rules can be applied, such as for example, to check Doppler-data for object 100-B as well, but subsequently (because that object is further away).

Computers

[0217] Embodiments of the invention can be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them.

The invention can be implemented as a computer program product, for example, a computer program tangibly embodied in an information carrier, for example, in a machine-readable storage device, for execution by, or to control the operation of, data processing apparatus, for example, a programmable processor, a computer, or multiple computers. À computer program as claimed can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program can be deployed to be executed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network. The described methods can all be executed by corresponding computer products on the respective devices, for example, the first and second computers, the trusted computers and the communication means.

[0218] Method steps of the invention can be performed by one or more programmable processors executing a computer program to perform functions of the invention by operating on input data and generating output. Method steps can also be performed by, and apparatus of the invention can be implemented as, special purpose logic circuitry, for example, a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

[0219] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computing device. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are at least one processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, for example, magnetic, magneto-optical disks, optical disks or solid state disks. Such storage means may also provisioned on demand and be accessible through the Internet (e.g., Cloud Computing). Information carriers suitable for embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, for example, EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and

DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in special purpose logic circuitry.

[0220] To provide for interaction with a user, the invention can be implemented on a computer having a display device, for example, a cathode ray tube (CRT) or liquid crystal display (LCD) monitor, for displaying information to the user and an input device such as a keyboard, touchscreen or touchpad, a pointing device, for example, a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, for example, visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0221] The invention can be implemented in a computing system that includes a back-end component, for example, as a data server, or that includes a middleware component, for example, an application server, or that includes a front-end component, for example, a client computer having a graphical user interface or a

Web browser through which a user can interact with an implementation of the invention, or any combination of such back-end, middleware, or front-end components. Client computers can also be mobile devices, such as smartphones, tablet PCs or any other handheld or wearable computing device. The components of the system can be interconnected by any form or medium of digital data communication, for example, a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), for example, the Internet or wireless LAN or telecommunication networks.

[0222] The computing system can include clients and servers. A client and a server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

List of Reference Symbols /1— |] mapping [x1, x2, ...,xD].n data-point d dimension index

D dimension number, the number of dimensions and thereby tuple- elements of each data-point

D-, D1, D2 partial use of dimensions, also à for D1 and A for D2 n data-point index to identify a particular data-point

N number of data-points in the data-set

P transformation matrix

R number of feature functions

S matrix t1, t2, tm time-points [X].n data-point

X data-set written in set notation { } as input matrix 100 object 100-A, 100-B objects that belong to object 100 100-1, 100-2 different objects 200 point-cloud data-set 210 transformation matrix 220 transformed data-set, transformed matrix S 221-1, 221-r … intermediate N-tuples 231-r derived Q-tuples, characteristic expressions 230, 230' derived matrices 250 derived data-set 400 technical equipment

410 sensor 500 computer 510 pre-processor module 520 machine learning (ML) tool, such as a neural network 600, 6xx method, method steps

Claims (16)

Claims

1. Computer-implemented method (600) to identify property data (PROPERTY) that belong to a physical object (100) that has a relation to technical equipment (400), the method (600) comprising: by a pre-processor module (510) of a computer (500), receiving (610) a point- cloud data-set (200) from one or more sensors (410) that are associated with the technical equipment (400), the point-cloud data-set (200) representing elements (E1, E2, E3) of the physical object (100), the point-cloud data-set (200) X = {[x1, x2, …, xD].1, [x1, x2, ..., xD].2, …, [x1, x2, ...,.xD].n, ..., [x1, x2, ...,

xD].N} having N data-points ([x].n), wherein each data-point ([x].n) comprises a multi-value tuple ([x1, x2, ...,xd, … xD].n) with a number D of values, wherein the number D corresponds to the number of dimensions; by the pre-processor module (510) of the computer (500), applying (620) a transformation to the point-cloud data-set (200) by applying dimension-specific numerical values to each data-point ([x].n) separately, resulting in a transformed data-set (220) that is a matrix S with a number of R intermediate N-tuples (221- 1, 221-2, 221-3, 221-4, 221-5); by the pre-processor module (510), according to a pre-defined mapping rule, mapping (630) the intermediate N-tuples (221-1, 221-2, 221-3, 221-4, 221-5) - separately - to derived Q-tuples of a derived data-set (250); by a previously trained machine-learning tool (520), processing (640) the derived data-set (250) to identify the property data (PROPERTY).

2. Method (600) according to claim 1, wherein receiving (610) the point-cloud data- set (200) comprises to receive the point-cloud data-set (200) as input matrix X as a D x N matrix, wherein applying (620) the transformation comprises to multiply a transformation matrix P (210) as a R x D matrix with the input matrix X, resulting in the matrix S (220) as a R x N matrix.

3. Method (600) according to any of claims 1 and 2, wherein receiving (610) the point-cloud data-set (200) from the one or more sensors (410) comprises the receive the point-cloud data-set (200) from a sensor arrangement with multiple channels, wherein the dimensions correspond to the multiple channels.

4. Method according to claim 2, wherein mapping (630) the R intermediate N- tuples to the derived N-tuples according to a pre-defined mapping rule comprises mapping calculated functional values that are the intermediate N- tuples (221-r, / /) with tuple-elements to characteristic expressions (231-r, | |) such that intermediate N-tuples with tuple-elements that are identical in number and type are mapped to the same characteristic expression.

5. Method according to claim 4, wherein mapping (630) comprises to sort the elements of each row in the matrix S (220) selectively in ascending or in descending order.

6. Method according to any of claims 4 to 5, wherein mapping (630) comprises to calculate an offset and to normalize the elements of the intermediate N-tuples (221-r) accordingly.

7. Method (600) according to claim 1, wherein processing (640) the derived data- set (250) to identify the property data is performed by classifying.

8. The method according to any of claims 1 to 7, wherein the computer (500) further provides a control command to the technical equipment (400), based on the identified property data.

9 Method according to any of claims 1 to 8, wherein the derived data-set (250) is processed by the previously trained machine-learning tool (520) in the form of a concatenation.

10. Method according to any of claims 1 to 9, wherein the steps applying (620) the transformation to the point-cloud data-set (200) and mapping (630) are performed for dimension groups separately after grouping data-channels (D1, D2) from the one or more sensors (410).

11. Method according to any of claims 1 to 10, wherein the pre-processor module (510) is receiving (610) the point-cloud data-set (200) with different data-sub- sets, with the data-sub-sets corresponding to multiple portions (100-A, 100-B) of the object (100) that are different in time or different in space, and applying (620) the transformation to the different sub-data-sets separately.

12. Method according to any of claims 1 to 11, wherein the method identifies a property of the physical object (100) in relation to the technical equipment (400), wherein the relation and the pre-processing are selected from the following: the physical object (100) is Moving in relation to the technical equipment (400): the technical equipment (400) is moving in relation to the physical object (100); and both technical equipment (400) and the physical object (100) are moving.

13. Method according to any of claims 1 to 12, wherein processing (640) the derived data-set (250) to identify the property data (PROPERTY) is performed by a machine-learning tool (520) that is selected from the following: artificial neural network, tree-based model, and support vector machine.

14. Use of a computer-implemented method (600) according to any of claims 1 to 13 by a computer (500) to determine a property of a physical object (100) in relation to technical equipment (400), wherein the computer (500) is associated with the equipment (400) to control the technical equipment (400) based on the determined property of the physical object (100).

15. Computer system (500) that is associated with technical equipment (400) and that adapted to execute a computer-implemented method (600) according to any of claims 1 to 13, to identify property data (PROPERTY) that belong to a physical object (100) that has a relation to technical equipment (400), and to use the property data to control the technical equipment (400) based on the determined property of the physical object (100).

16. Computer program product that - when loaded into a memory of a computer and being executed by at least one processor of the computer causes the computer to perform the steps of the method according to any of claims 1 to 13.