BR112018072597B1 - ESTIMATION OF ELECTROMECHANICAL QUANTITIES USING DIGITAL IMAGES AND MODEL-BASED FILTRATION TECHNIQUES - Google Patents

Abstract

This is a method for estimating one or more of the following quantities from a machine and/or electromechanical component, and the method comprises the creation of a photorealistic numerical model of the electromechanical machine or parts thereof, a measurement step to combine outputs from physical sensors of which at least one is an imaging device to visualize the outer surface of the physical electromechanical machine into at least a two-dimensional image, an estimation step that combines the photorealistic numerical model and the measurement step to provide an estimation of desired electromechanical quantities, with the estimation step being based, at least, on the use of a similarity metric between the (at least one) two-dimensional image of the electromechanical machine or parts thereof and the images generated by the photorealistic numerical model.

Description

FIELD OF INVENTION

[001] The present invention generally refers to the field of model-based estimation of electromechanical quantities. More specifically, the present invention relates to the estimation of electromechanical quantities using digital images and model-based filtering techniques due to the aid of a photorealistic electromechanical numerical model.

BASICS OF THE INVENTION

[002] In recent years, several methods for combining experimental measurements with numerical representations ("models") of machines and/or electromechanical or mechatronic components have been proposed in order to infer (estimate) information that is difficult or cannot be measured directly.

[003] Different methods have been presented and tuned to increase complexity levels in the range from a single indeformable body (e.g. for pose estimation) to lumped parameter system models to nonlinear and linear (reduced) finite element models. and flexible multibody. These techniques have been successfully applied in both academic and industrial settings. All of these share a fundamental core functionality: they exploit the knowledge embedded in the model in order to estimate a set of model variables (model states, inputs, parameters or any combination thereof) while minimizing, in some sense ( weighted), the difference between the measured quantities (through a physical sensor in a physical object/system) and those predicted (obtained from the evaluation of a numerical model).

[004] The connection between the model and measurements is achieved through inverse or estimation algorithms. General optimization algorithms can be used to adjust a certain model variable so that the model better matches the measurements. Specific techniques such as Kalman filtering and Moving Horizon Estimation can be used to obtain an optimal agreement between the numerical model and measurement accuracy for the resulting estimate.

[005] Different measurements are used for different applications. Often, discrete sensors (accelerometers, strain gauges, microphones, etc.) are used that can be directly linked to one or more model variables. Growing interest has recently pointed toward the use of field measurements (such as digital video frames, videos, and camera images) due to their unique ability to collect high-density spatial data from a landscape of interest.

[006] Current prior art methods hardly exploit the full range of information captured from full-field sensors (e.g., camera sensors). The methods focused on extracting information for rigid pose estimation only, which requires much less accurate model and processing information. The aforementioned is, for example, described in international patent application WO2014/055355, US patent application no 2009/0297020 and GB patent application no 2504822. Their use as visual measurement tools has usually been complemented by some type of middleware to in order to obtain a relatively smaller number of data points that can be used as discrete measurement points (e.g. point cloud generation for pose estimation, feature tracking for location measurement as e.g. described in the application for international patent WO2012/0407009 and US application no. 9412040); or recover indirect measured quantities (a specific example of the same is Digital Image matching in which image information is converted into an estimated strain measurement in a structure). As a consequence, camera field measurement has never been fully and directly explored within an estimation framework.

[007] This leads to several limitations of the existing approaches described above:

[008] • When relying on discrete sensors:

[009] different types of sensors are required to estimate different quantities;

[010] an impractically large number of sensors may be required in order to provide sufficient accuracy;

[011] discrete sensors tend to be intrusive (the presence of sensors disturbs the quantity under investigation);

[012] • When relying on field sensors:

[013] information is often manipulated to generate discrete measurement points. Therefore, most of the available data is lost as the available model information is not ideally matched with the available sensor information.

[014] o Information is often processed in order to extract full-field quantities through other methodologies (e.g. DIC), where valuable knowledge about the physical system/object cannot be exploited in order to improve accuracy .

[015] o Pure field measurement approaches may require some preparation of the structure under consideration.

[016] o In the case of 3D machines and/or electromechanical components, available methods require at least one stereo equipment for each side of the machine and/or electromechanical component under investigation, which quickly leads to expensive requirements for realistic structures. These methods also only provide information about the visible parts of the structure, but do not provide any understanding about invisible quantities (e.g. internal deformations).

[017] o Methods employing more expensive imaging techniques can be used (CTI Computed Tomography Imaging, MRI Magnetic Resonance Imaging, SPECT Single Photon Emission Computed Tomography, PET Positron Emission Tomography, etc.), but need of a controlled environment and are not directly applicable to machines and electromechanical components during, for example, their regular operation. Furthermore, the mentioned techniques allow 3D images of the system including internal non-visible parts, which is not necessary in electromechanical applications and dramatically increases the cost of the method. Furthermore, most of these methods can only be used for quasi-static image acquisitions, and their use for electromechanical applications is difficult, if not impossible, due to the disturbance of the magnetic fields introduced by the measuring device.

[018] In international patent application WO2014/127321, images from expensive and intrusive medical devices such as CTI, MRI, SPECT or PET are used to evaluate the position of organs in a human body during surgery and assist the surgeon to understand the exact location of delicate parts that are difficult, if not impossible, to see with less precise intraoperative means. It exploits such 3D medical images in combination with a numerical model to solve an optimization problem that finally allows creating a 3D model of human body organs deformed during operations and superimposing it on less precise intraoperative images. This allows the doctor to see a more detailed, numerically created model of the patient's organs. Although this approach is quite interesting, it does not have the ability to achieve the level of precision and dynamic range in estimated quantities combined with the computational efficiency that is necessary in electromechanical applications. Furthermore, it is not applicable as such to machines and/or electromechanical components operating in their operational environment.

[019] Therefore, there is a need for improved model-based detection methods.

SUMMARY OF THE INVENTION

[020] It is an objective of embodiments of the present invention to provide the estimation of electromechanical quantities through digital images and model-based filtering techniques. The above objective is achieved by a method and a device according to the present invention.

[021] It is an advantage of the modalities of the present invention that they can make use of low-cost images, that is, from low-cost imaging techniques, such as video and/or photographic camera images. It is an advantage of embodiments of the present invention that 2D images of visible parts can be used. It is thus an advantage that the modalities do not require internal images or images that are large or expensive, for example, as obtained from 3D devices such as CTI, MRI, SPECT or PET. It is an advantage of embodiments of the present invention that use is made of model-based filtration in order to infer surface but also internal quantities of electromechanical systems, such as, for example, machines.

[022] It is an advantage of the embodiments according to the present invention that the time history of the estimated quantities and/or information is explored. The former allows you to cover temporary dynamic movement, which is fundamental in electromagnetic applications. It is an advantage of embodiments of the present invention that a vision-based approach is coupled with dynamic estimators, such as, for example, Kalman-based filters and/or Moving Horizon estimators.

[023] It is an advantage of the embodiments according to the present invention that high fidelity models are used allowing the required level of precision to be obtained. Large computational burdens are avoided by adopting advanced model order reduction schemes when required.

[024] In a first aspect, the present invention provides a method for detecting a physical object, the method comprising providing a photorealistic virtual object of the physical object, performing a measuring step, the measuring step comprising recording the physical object and acquiring physical field measurements of the physical object comprising at least one two-dimensional image and performing an estimation step, the estimation step comprising applying external excitations to the photorealistic virtual object to create photorealistic virtual field measurements and comparing the field measurements photorealistic virtual field with physical field measurements and therefore detect the physical object.

[025] The method may comprise using a time history of at least one two-dimensional image.

[026] Detecting a physical object may comprise estimating one or more of the following quantities of the physical object, for example, one or more of the following electromechanical quantities of a machine or electromechanical component:

[027] • States: including, but not limited to, positions, speeds, accelerations, deformations, deformation rates, currents;

[028] • Input: including, but not limited to, mechanical forces, mechanical torques, mechanical pressures, tensions;

[029] • Parameters: including, but not limited to, density, Young's moduli, Poisson's ratios, material parameters, physical dimensions, resistance, capacitance.

[030] The method may comprise providing a time history of the estimation of said quantities of the physical object.

[031] The physical object can be a machine and/or electromechanical component.

[032] Providing a photorealistic virtual object may comprise creating a photorealistic numerical model of the physical object.

[033] Performing a measuring step may comprise combining outputs from physical sensors of which at least one is an imaging device to visualize the external surface of the physical object in at least one two-dimensional image.

[034] Performing an estimation step may comprise combining the photorealistic virtual field measurements with the physical field measurements to provide an estimate of the desired quantity or quantities and wherein the estimation step is based at least on the use of a metric of similarity between at least one two-dimensional image of the physical object or parts thereof, and the images generated by the photorealistic virtual object.

[035] Performing an estimation step may comprise combining the photorealistic numerical model and the measurement step to provide an estimate of desired electromechanical quantities and wherein the estimation step is based, at least, on the use of a similarity metric between the at least a two-dimensional image of the electromechanical machine, or parts thereof, and the images generated by the photorealistic numerical model.

[036] The physical behavior of the photorealistic virtual object of the physical object can be described by a static or dynamic model obtained as a discretized approximation of a system.

[037] It is an advantage of embodiments of the present invention that images or sensors that generally detect 2D images of a machine and/or electromechanical component are combined with numerical models that are capable of representing localized and distributed fields. A more effective way to combine camera images and videos into an estimation framework is provided, exploiting the full potential of measured information.

[038] In preferred embodiments, the physical behavior of the photorealistic electromechanical numerical model is described by a static or dynamic model obtained as a discretized approximation of a system described by ordinary differential, partial differential or differential algebraic equations, such as, but without limitation: lumped parameter system models, finite element models, computational fluid dynamics models, flexible multibody models, or combinations thereof.

[039] In preferred embodiments, the photorealistic electromechanical numerical model is made computationally efficient by using one or more linear and nonlinear model order reduction techniques. High accuracy can be guaranteed by a high dimensional description of the electromechanical model or at least the parts thereof that are compared to camera images and/or videos. High precision is necessary to obtain quantitative results while model order reduction is necessary to preserve the computational accuracy of such computationally expensive models. As the model is solved in the time domain, it is required that it reaches acceptable computational loads. The one or more linear or nonlinear model order reduction techniques may be one or more of component mode synthesis, Krylov-based methods, correct orthogonal decomposition, dynamic mode decomposition, balanced truncation, discrete empirical interpolation method, sampling and energy conservation weighting.

[040] These techniques intrinsically allow to treat the dynamic effects and time history of electromechanical estimated quantities providing an ideal trade-off (in some defined sense) between image accuracy and model accuracy. In preferred embodiments, the estimate is obtained from dynamic filtering techniques combining both digital images and the photorealistic electromechanical numerical model. These techniques include, but are not limited to: Kalman filter-based techniques, Moving Horizon Estimation techniques, Luenberger observer, etc.

[041] In preferred embodiments, the estimator allows the evaluation of stochastic error limits of the estimated electromechanical quantities.

[042] The model can be solved in the time domain;

[043] The estimate can be obtained from dynamic filtering techniques that combine videos, images and the photorealistic numerical model.

[044] The estimation step can allow the evaluation of stochastic error limits of the estimated magnitudes of the physical object.

[045] The method may comprise measuring and using a subset of the magnitudes of the physical object as a known input to the photorealistic virtual object.

[046] The method may comprise measuring and using additional electromechanical quantities, in addition to videos and camera images during the estimation step.

[047] In preferred embodiments, a subset of the electromechanical quantities is known. These known quantities are used to improve the accuracy of the remaining estimated quantities within the estimation algorithm.

[048] In preferred embodiments, a subset of the electromechanical quantities is measured. These measured quantities are used to improve the accuracy of the estimated quantities within the estimation algorithm. Such measurements include, but are not limited to: strain gauges, accelerometers, fiber Bragg grating strain sensors, gyroscopes, voltage sensors, current sensors, temperature sensors, microphones, etc.

[049] In preferred embodiments, providing a photorealistic electromechanical numerical model comprises creating a numerical model of the machine and/or physical electromechanical component and texturing the geometry of the model with at least one image.

[050] In preferred embodiments, the physical field measurements of the electromechanical machine are deformed physical fields.

[051] In a second aspect, the present invention provides devices comprising a processing system, the processing system being adapted to carry out a method in accordance with embodiments of the present invention. In a third aspect, the present invention relates to a system for characterizing a physical object, the system comprising at least one means for measuring magnitudes of the physical object and a processor adapted to receive the output of the at least one means for measuring fields. of the physical object and adapted to perform the method as described above.

[052] In one aspect, the present invention also provides a system for estimating electromechanical quantities, the system comprising:

[053] • at least one means for acquiring digital images of the physical electromechanical machine;

[054] • a processor adapted to receive the output of at least one means for obtaining images and videos of the physical electromechanical system and adapted to carry out the method presented in the preferred embodiments.

[055] It is an advantage of the embodiments that a photorealistic electromechanical numerical model is created so that it can be viewed from one or more different points of view and configurations and that it visually resembles the machine and/or physical electromechanical component: In short, be photoconsistent with the machine and/or physical electromechanical component.

[056] The electromechanical numerical model can be rendered on any screen or device that allows it to be viewed.

[057] A method according to the embodiment of the present invention comprises several steps involving different technologies. It is an advantage of the embodiments of the present invention that an ideal method is endowed with high precision and great computational efficiency.

[058] Computational efficiency can be enabled as follows: using numerical model efficiency as further detailed in the description of the invention, 2D or 3D models can be used to create the photorealistic electromechanical numerical model. Dedicated techniques for obtaining such numerical models are (but without limitation): linear and nonlinear finite elements, finite volumes, finite differences, multibody and flexible multibody models, isogeometric models or modeling methods in some way allowing to discretize systems of algebraic equations ordinary differentials, partial differentials and differentials. The techniques described lead to a precise numerical description of the physical electromechanical machine, but often result in a very large number of degrees of freedom (up to several million) and equations to be solved to obtain a representation of 2D and 3D distributed fields (positions, accelerations, deformations, etc.). The computational burden required to solve this set of equations (e.g., equations of motion, estimating equations, etc.) can become prohibitive. In recent decades, several techniques that allow for a faster but more accurate solution have been developed under the name of linear and nonlinear Model Order Reduction (MOR). The application of these techniques is a field of study in itself and represents a non-trivial step that can be taken during model preparation. MOR techniques can be used in the present invention to achieve a considerable reduction in computational burden that leads to practicable computational time during the estimation step of the invention. MOR techniques are not necessary for the successful implementation of the present invention, but represent a way to achieve a scalable method in which accuracy and computational speed represent a trade-off to be chosen by the modeler. In particular, techniques such as (but without limitation): modal truncation, component mode synthesis, Krylov space reduction, balanced truncation, correct orthogonal decomposition, energy conservation weighting and sampling, discrete and empirical empirical interpolation method, global modal parameterization, generalized component mode synthesis, piecewise linear trajectory model reduction, ManiMor, Generalized Correct Decomposition, parametric model order reduction techniques, hyperreduction techniques, or any other method that is suitable for reducing the computational load imposed by numerical models by reducing the original number of degrees of freedom and/or equations. These techniques not only have the advantage of reducing the number of degrees of freedom, but also show some other valuable benefits such as reducing the numerical rigidity of the problem manually with the consequent advantages from a solver's point of view (larger time interval allowed , improved stability, etc.). If a MOR technique is used, it is of crucial importance to select the appropriate method according to a specific problem; MOR is a technique in itself and strong engineering judgment is still required to obtain optimal computational performances without drastically compromising accuracy.

[059] Effective rendering could also be explored in the present invention, but it is not a mandatory step. It can be enabled by efficient rendering techniques - such as the process of obtaining images from photorealistic electromechanical numerical models - by the use of modern GPU architectures, with their programmable pipeline. The combination with reduced order models benefits the invention by allowing rapid generation of digital images of the photorealistic electromechanical numerical model. More generally, the use of massively parallel accelerators to efficiently compute simulated field quantities, within a MOR scheme, is not a trivial technical step, which increases the potential fields of industrial application of the presented invention as opposed to others. methods such as, for example, in international patent application WO2014/127321, which depend on a controlled (medical) environment for their success.

[060] The estimation of electromechanical quantities can be enabled by one or more estimation techniques. It is an advantage of this invention to allow the use of time domain estimators, such as (but without limitation) Kalman filtering, moving horizon estimation and/or Luenberger observer, etc. Estimators can enable the use of dynamic temporary numerical models of the machines and/or electromechanical component that incorporate information about the full time history of the estimated quantities including videos and images measured with a camera and/or created with the photorealistic electromechanical numerical model.

[061] In some embodiments, the estimators may also allow the recovery of a stochastic error measurement from the discrepancy between the physical electromechanical quantities and the estimated electromechanical quantities.

[062] The quality of the videos or camera images acquired together with the estimation approach and similarity measurements can allow achieving an accuracy that is one or three orders of magnitude smaller than a pixel size.

[063] In one aspect, the present invention also provides a method for estimating one or more of the following electromechanical quantities of a machine or electromechanical component:

[064] • States: including, but not limited to, positions, speeds, accelerations, deformations, deformation rates, currents;

[065] • Input: including, but not limited to, mechanical forces, mechanical torques, mechanical pressures, tensions;

[066] • Parameters: including, but not limited to, density, Young's moduli, Poisson's ratios, material parameters, physical dimensions, resistance, capacitance

[067] and the method comprises:

[068] • creating a photorealistic numerical model of the electromechanical machine or part thereof;

[069] • a measurement step that allows combining physical sensors, at least one of which is an imaging device that allows the capture of the appearance of the external surface of the physical electromechanical machine or parts thereof in one or more 2D images,

[070] • an estimation step that combines the photorealistic numerical model and the measurement step to provide an estimate of the desired electromechanical quantities,

[071] • The estimation step is mainly, but not only, based on the use of a similarity metric between at least said one or more 2D images of the electromechanical machine or parts thereof and the images generated due to the photorealistic numerical model.

[072] In one aspect, the present invention relates to a method for detecting a physical object, the method comprising providing a photorealistic virtual object of a physical object and performing an operational step, the operational step comprising a measuring step , the measurement step comprising recording the physical object and acquiring physical field measurements of the physical object, and an estimation step, the estimation step comprising applying external excitations to the photorealistic virtual object to create photorealistic virtual field measurements and compare the photorealistic virtual field measurements to the physical field measurements and therefore detect the physical object.

[073] Comparing photorealistic virtual field measurements to physical field measurements may comprise matching photorealistic virtual field measurements to physical field measurements and recovering the resulting excitation in said correspondence.

[074] Comparing photorealistic virtual field measurements to physical field measurements may comprise minimizing a cost function representing a similarity measurement of the photorealistic virtual field measurements to physical field measurements.

[075] External excitations can be known or estimated.

[076] Providing a photorealistic virtual object of a physical object may comprise creating a model or virtual object of the physical object and texturing the model with at least one image of the physical object.

[077] Creating a model of the physical object may comprise a model creation step and a model registration step.

[078] The model registration step may comprise positioning and orienting the physical object and the model. The model can be a numerical model.

[079] The model can be a two-dimensional or three-dimensional model.

[080] Texturing the model with at least one image may comprise using an image-based rendering method so that the model or virtual object of the physical object is consistent with the at least one image of the physical object.

[081] At least one image can be obtained during registration in the measurement step.

[082] Wherein the at least one image may be obtained by a photo or video imaging means, magnetically based imaging means, radiographic imaging means, scanning light detection or solid state imaging means and/or means of thermographic imaging.

[083] Physical field measurements of the physical object can be obtained by measuring the physical object in static or dynamic excitation.

[084] Physical field measurements of the physical object may be distributed and/or deformed physical fields.

[085] The present invention also relates to a device comprising a processing system, the processing system being adapted to carry out the method as described above.

[086] The present invention further relates to a system for detecting a physical object, the system comprising

[087] - at least one means for measuring fields of the physical object;

[088] - a processing adapted to receive the output of at least one means for measuring fields of the physical object and adapted to perform the method as described above.

[089] The particular and preferred aspects of the invention are presented in the attached independent and dependent claims. Features of dependent claims may be combined with features of independent claims and with features of other dependent claims as appropriate and not merely as explicitly set forth in the claims. Similarly, optional and/or standard features of the different aspects may be combined as appropriate and not only as explicitly set forth in the claims. These and other aspects of the invention will be apparent from and elucidated with reference to the embodiment (or modalities) described below.

BRIEF DESCRIPTION OF THE FIGURES

[090] Figure 1 illustrates a method according to embodiments of the present invention.

[091] Figure 2 illustrates a method according to embodiments of the present invention.

[092] Figures 3A to 3C illustrate the creation of a model according to embodiments of the present invention.

[093] Figures 4A to 4C illustrate a model registration step in accordance with embodiments of the present invention.

[094] Figures 5A to 5C illustrate the creation of a photorealistic numerical (electro)mechanical model in accordance with embodiments of the present invention.

[095] Figure 6 illustrates a measurement of images of the machine and/or physical (electro)mechanical component according to embodiments of the present invention.

[096] Figures 7A to 7D illustrate an estimation step or phase in accordance with embodiments of the present invention. Figure 8 illustrates a workflow of a method in accordance with specific embodiments of the present invention.

[097] Figure 9 illustrates possible hardware-software systems according to embodiments of the present invention.

[098] Figure 10 illustrates a summary of a method according to embodiments of the present invention.

[099] The figures are schematic only and are not limiting. In the Figures, the size of some of the elements may be exaggerated and not drawn to scale for illustrative purposes. Any numerical references in the claims should not be construed as limiting the scope.

[0100] In the different Figures, the same numerical references refer to the same or similar elements.

DETAILED DESCRIPTION OF THE ILLUSTRATIVE EMBODIMENTS

[0101] The present invention will be described in relation to particular embodiments and with reference to certain Figures, but the invention is not limited thereto, but only by the claims. The Figures described are schematic only and are not limiting. In the Figures, the size of some of the elements may be exaggerated and not drawn to scale for illustrative purposes. The dimensions and relative dimensions do not correspond to the actual reductions for practicing the invention.

[0102] Furthermore, the terms first, second and similar in the description and claims are used to distinguish between simulative elements and not necessarily to describe a sequence, whether temporal, spatial, in classification or in any other way. It should be understood that the terms so used are interchangeable under appropriate circumstances and that the embodiments of the invention described herein are capable of operation in sequences other than those described or illustrated herein. Furthermore, the terms upper, lower and similar terms in the description and claims are used for descriptive purposes and not necessarily to describe relative positions. It should be understood that the terms so used are interchangeable under appropriate circumstances and that the embodiments of the invention described herein are capable of operation in orientations other than those described or illustrated herein.

[0103] It should be noted that the term “comprising”, used in the claims, should not be interpreted as being restricted to those listed below; it does not exclude other elements or steps. The same shall therefore be interpreted as specifying the presence of the features, integers, steps or components determined as named, but does not exclude the presence or addition of one or more other features, integers, steps or component or groups thereof . Thus, the scope of the expression “a device comprising means A and B” should not be limited to devices consisting only of components A and B. This means that, in relation to the present invention, the only relevant components of the device are A and B. Reference throughout this specification to "a (numeral) embodiment" or "an (indefinite article) embodiment" means that a particular feature, structure or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Therefore, the appearance of the phrases "in a (numeral) modality" or "in an (indefinite article) modality" in various places throughout this specification does not necessarily refer to the same modality, but may. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner, as will be appreciated by a person of ordinary skill in the art of this disclosure, in one or more embodiments.

[0104] Similarly, it can be seen that, in the description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together into a single embodiment, figure or description thereof for the purpose of simplifying the disclosure and assisting in the understanding of a or more of the various inventive aspects. This method of disclosure, however, should not be interpreted as reflecting an intention that the claimed invention requires more resources than expressly stated in each claim. Instead, as the following claims reflect, the inventive aspects rest on less than all the features of a single prior disclosed embodiment. Therefore, the claims following the detailed description are expressly incorporated herein into this detailed description, wherein each claim stands as a separate embodiment of this invention.

[0105] Furthermore, although some embodiments described herein include some, but not other, features included in other embodiments, combinations of features from different embodiments should not be within the scope of the invention and form different embodiments, as will be understood by those versed in the technique. For example, in the following claims, any of the claimed embodiments can be used in combination. In the description provided in this document, numerous specific details are presented. However, it is understood that embodiments of the invention may be practiced without these specific details. In other instances, well-known methods, structures and techniques have not been shown in detail so as not to obscure an understanding of this description.

[0106] In a first aspect, as evidenced in Figure 1, the present invention provides methods for detecting a machine and/or physical electromechanical component, the method comprising providing a photorealistic electromechanical numerical model of the machine and/or physical electromechanical component and performing an estimation step. The estimation step comprises a measurement and estimation step, wherein the measurement step comprises recording the machine and/or physical electromechanical component and acquiring one or more images or videos of the machine and/or physical electromechanical component, and wherein the step estimation comprises performing a time simulation of the photorealistic electromechanical numerical model to create simulated photorealistic images of the electromechanical numerical model and comparing the simulated images and/or videos to the images and/or videos of the physical electromechanical machine and therefore "detecting" the machine and/or physical electromechanical component within an estimation algorithm. In additional embodiments, as evidenced in Figure 2, providing a photorealistic electromechanical numerical model of the machine and/or physical electromechanical component can be enabled in a calibration step by creating a numerical model of the machine and/or physical electromechanical component and texturing the model numeric with at least one image of the machine and/or physical electromechanical component. As a result, embodiments of the present invention describe a workflow and methodology for augmenting numerical models of systems and/or physical objects. More specifically, to augment numerical models with field measurements, e.g. images and/or videos of the machine and/or physical electromechanical component. This increase can be obtained by texturing the numerical model using images and/or videos of the machine and/or physical electromechanical component.

[0107] Images or videos of the machine and/or physical electromechanical component can be obtained from any acquisition hardware dedicated to capturing images, e.g., digital photo/video cameras.

[0108] According to embodiments of the present invention, the images or videos are two-dimensional images or videos.

[0109] In embodiments of the present invention, the photorealistic electromechanical numerical model is created such that (parts of) it closely resembles the appearance of the machine and/or electromechanical component that is imaged due to the hardware (e.g., video cameras). photo and video) and acquisition software. This similarity is called "photoconsistency". The created photorealistic electromechanical numerical model allows photorealistic visualization under different conditions (e.g., rest, motion, deformed state, temperature) to be subsequently used to obtain images and/or videos of such photorealistic electromechanical numerical model.

[0110] The photorealistic electromechanical numerical model can be augmented with a visualization (parts) of the estimated electromechanical quantities that are of interest to the user of embodiments of the present invention.

[0111] In current prior art estimation approaches, data obtained from field sensors are transformed into variables that can be found directly in the virtual object (e.g., camera images are transformed into the movement of a limited number of points that can then be compared to the movement of the same points in a mechanical model). However, in embodiments of the present invention, the photorealistic electromechanical numerical model is created, such as to visually behave like the machine and/or physical electromechanical component. As such, the images of the photorealistic electromechanical numerical model can be directly compared to images of the machine and/or electromechanical component, for example, in your working environment.

[0112] The comparison may be based on methods belonging to the field of digital image matching (DIC), comparing pixel intensities, through motion estimation, due to feature extraction or any other suitable means. Estimation techniques can be subsequently applied through, for example, Kalman filter, Moving Horizon Estimation (MHE) or Luenberger observers. This step is preferred to achieve estimates of unknown electromechanical quantities, such as "distributed fields" (e.g. displacements, velocities, stresses, deformations, accelerations, temperatures, etc.), "localized measurements" (e.g. displacements, velocities, mechanical and thermal stresses/deformations, accelerations, temperatures, etc.), "system states", unknown input or internal loads or any type of "input" (e.g. forces, torques, thermal sources, voltage, etc.) and/or "parameters" (geometric dimensions, stiffness, mass, density, material properties, etc.) - possibly varying over time. Distributed fields, unknown excitations, localized measurement system states and parameters are additionally and previously called estimated electromechanical quantities.

[0113] Achieving accurate estimates of variables as defined above is linked to one or more of several steps.

[0114] Embodiments of the present invention provide a numerical/experimental process comprising one or more of the following steps or aspects: a preparation step and an estimation step. Whereby, the preparation step may comprise a model definition and creation step and a photorealistic "augmentation" of the electromechanical numerical model. The estimation step may comprise a measurement phase and an estimation phase and, more specifically, an estimation based on a photorealistic model.

[0115] As indicated, in general, a method according to embodiments of the present invention can be divided into two main phases. A preparation phase may comprise the following: steps in which the electromechanical numerical model is prepared, recorded, and converted into a photorealistic electromechanical numerical model in accordance with embodiments of the present invention. However, other alternative techniques known in the art, which lead to the same result as the preparation phase, such as a photorealistic electromechanical numerical model, can also be used directly as an input to the estimation phase when available by other means. In accordance with some embodiments of the present invention, the computational burden of simulating the photorealistic electromechanical numerical model is reduced through model order reduction techniques.

[0116] The estimation phase according to the embodiments comprises a measurement phase in which digital images or videos of the machine and/or electromechanical component are acquired together with the time series of other optional sensors and a correct estimation phase in which the values of the desired variables are evaluated. The estimation phase advantageously uses a photorealistic electromechanical numerical model in the estimation context. Images or videos according to embodiments of the present invention comprise two-dimensional images or videos.

[0117] In embodiments of the present invention, a preparation phase or step may be provided. Said calibration phase or step may comprise (1) a model definition step, (2) a model registration step and (3) a photorealistic model creation.

[0118] A model creation step in accordance with embodiments of the present invention may comprise a creation step, whereby a numerical method is used to create a numerical representation of a machine and/or physical electromechanical component, wherein the variables must be estimated. In particular, methods that allow a 2D or 3D representation of the machine and/or physical electromechanical component (e.g., a realistic visualization or photoconsistency) are used. Figures 3A to 3C illustrate the definition and creation of a model in accordance with embodiments of the present invention. As a first step, the model geometry is created, for example by point cloud, CAD, etc. Then, discretization of the model is performed, such as a finite element model (FEM), a flexible multibody model, etc.

[0119] As an illustration, Figures 3A and 3B illustrate structural FEMs of an (electro)mechanical component, for example, simple beam structures. More specifically, Figure 3A illustrates a top view of a FEM model of a beam, while Figure 3B illustrates a FEM model of an L-shaped beam. Figure 3C illustrates a detailed flexible multibody model of a vehicle suspension. . In embodiments, the complete or partial shape reconstruction of the machine and/or electromechanical component is reflected in the model. It is preferred, but without limiting the invention, that the chosen method has the ability to represent distributed variables, such as positions, accelerations, speeds, mechanical and thermal deformations, stresses, temperatures, etc. Numerical models and modeling methods may be, but are not limited to: (a) finite elements (linear and non-linear), finite volumes, finite differences, multibody and flexible multibody models, isogeometric models, etc.; (2) a possible choice is modeling methods that allow to discretize systems of algebraic, ordinary differential, partial differential and differential algebraic equations; and (3) a further choice of modeling methods that allow a continuous representation of the machine and/or electromechanical component is also possible.

[0120] Typical examples are thermal and structural finite element models of mechanical components and flexible multibody systems of mechanisms as found in all machinery, vehicles, power production, electromechanical models, etc. These 3D models can be based on CAD design figures or 3D scans of the machine and/or electromechanical component. Generally speaking, 2D and 3D numerical models that are capable of representing distributed fields are computationally expensive. In this case, techniques such as linear and non-linear Model Order Reduction ("MOR") are the preferred choice to gain computational speed with lower accuracy losses.

[0121] Various commercial and research tools are available to perform model definition and creation. Ideally, the electromechanical numerical model should be carefully updated to adjust the behavior of the machine and/or physical electromechanical component with respect to, for example, static and dynamic characteristics or any other characteristic or behavior of interest.

[0122] After creating the photorealistic electromechanical numerical model, in a next step according to the embodiments of the present invention, a model registration can be carried out. Model registration may comprise at least one of the following steps: a shape reconstruction step, updating the model geometry, and positioning and changing the orientation of the machine and/or virtual and electromechanical component according to one of several methods known in the art. .

[0123] Different methods can be used in order to increase the accuracy of the comparison between the images of the machine and/or physical electromechanical component and the images of the photorealistic electromechanical numerical model by adding more "granularity" or "contrast" to the machine and/or or physical electromechanical component. Methods may be (but are not limited to): applying high contrast markers or patterns via, for example, spray paint, stickers or any means that can be used to precisely trace their geometry. This last step is not necessary, and the natural (surface) features of the machine and/or physical electromechanical component can be used when precision allows.

[0124] In additional embodiments of the present invention, the model registration step may comprise modifying the geometry, location, orientation in space of the virtual object in order to closely match the same (geometric) characteristics of the machine and/or electromechanical component in relation to to a common chosen frame of reference. This step can be achieved by optimization routines that allow precise alignment and positioning in space. In particular, two or more points in a reference configuration can be used to create a direct correspondence between numerical model geometry and physical system/component geometry in space. These points are called control points and are used to orient the numerical model as precisely as possible and align it with the physical system/component. The last approach is just one possibility that can be used and the general framework is not restricted to it. Any method that allows correct alignment and geometric correspondence between the numerical model geometry and the physical system/component can be used.

[0125] Figures 4A to 4C illustrate model registration according to embodiments of the present invention, wherein Figure 4A illustrates an image of a machine and/or (electro)mechanical component that is fixed and, more specifically, a intensified cantilever beam with a grain pattern. Figure 4B illustrates model reconstruction and, more specifically, machine and/or (electro)mechanical component point cloud reconstructions. The result of a 3D shape reconstruction is provided in which a point cloud is obtained. Finally, Figure 4C illustrates an updated finite element (FE) model of the cantilever beam, in which the numerical model is updated and its orientation changed so that it is an FE model oriented and positioned similarly to the machine and/or component. physical (electro)mechanical. More specifically, the surface of the numerical FE model corresponds to the point cloud and is oriented like the object given in Figure 4A. To enable the above, motion tracking image acquisition software can be used: e.g. image and video acquisition, blob tracking, feature detection, etc.

[0126] At least one field measurement device can be used, whereby these devices could be, but without limitation (in this invention, the use of photo and video cameras in preferred but different media can still be used in the Within the framework of this invention, these other means may increase or decrease the price of the installation, restrict or extend its industrial applicability and increase or reduce its precision): (a) photo or video cameras, in which the quality of the images is, preferably, relatively high and noise-free, additionally, still images can be used, multi-image mediation can improve the use of low-cost cameras, and, depending on the desired accuracy, cameras can range from low-fidelity media such as webcams , high-fidelity, high-speed megapixel cameras. According to embodiments of the present invention, at least 2D images or videos are used. Images and videos can be acquired in grayscale and/or color (e.g., 8-bit and above), (b) a magnetic imaging medium, such as an MRI scanner, on which these scanners perform imaging based on the magnetic response of the object under research. These scanners are often combined with tomography methods in order to construct complete 3D information. This last step is not, however, necessary for the estimation purposes discussed in this paper; (c) a means of radiographic imaging such as, for example, X-ray imaging, whereby these devices perform field imaging of an object based on radiographic principles and is particularly suitable for obtaining information within an object; (d) solid-state or scanning light sensing and distance measuring devices, such as LiDar, which provide a field of distance information from a large number of points relative to the sensor, and (e) imaging means thermographics or cameras, which, by capturing light in the infrared range (rather than in the visible range like a still camera), can be used to detect temperatures and can also be used in the absence of light sources.

[0127] In additional embodiments, data transfer from the camera to the target PC can be performed with any type of protocol depending on the speed and amount of data to be transferred, e.g., CameraLink, USB3, GiGe, CoaXPress, etc. . This data transfer can happen in real time, online or offline depending on the application. In additional embodiments, frame grabbers may be used: given the potentially large amount of data acquired, a frame grabber may be necessary to allow a fast enough connection between the cameras and a storage medium, such as a RAM memory, a hard drive or an SSD or any type of memory that can be used for this purpose;

[0128] In additional embodiments of the present invention, the preparation step may further comprise creating and visualizing a photorealistic electromechanical numerical model. The procedure allows you to create a rendered image of the numerical model such that the virtual measuring field closely matches/resembles the (part of) the field measured in the machine and/or electromechanical component - or, in other words, the numerical model becomes photoconsistent. The resulting created model can be called a "photorealistic electromechanical numerical model". This aspect can be accomplished by any image-based rendering procedure (e.g., view-dependent texture mapping) or any procedure that allows achieving the same result.

[0129] For example, images of the machine and/or physical electromechanical component stored in any of the previous steps can be used, and each part of the reconstructed surface of the photorealistic electromechanical numerical model (e.g., composed of triangles and quadrilaterals) is assigned a specific texture by combining all or part of the images that visually cover this specific surface patch. The combination can be obtained, for example, by applying interpolation techniques. Interpolation can be performed, for example, by selecting weights proportional to a metric of the distance between the selected surface and the camera. Other methods for combining textures or interpolating textures can be applied. Interpolation can respect the partition of unity rule. Finally, a photorealistic electromechanical numerical model is created so that it can be viewed from one or more different points of view and configurations and that it visually resembles the machine and/or electromechanical component: in a word, it is photoconsistent with the machine. and/or physical electromechanical component. Photoconsistent images of the electromechanical numerical model or virtual object can be obtained with techniques such as rendering to texture or any other technique that allows the storage of images reproduced on screens or devices in general. The step of creating a photorealistic electromechanical numerical model can be performed by any technique available in the prior art, in the field of image-based rendering and modeling.

[0130] Figures 5A to 5C illustrate the step of creating a photorealistic electromechanical numerical model or virtual object according to embodiments of the present invention. By image-based rendering and modeling (e.g., by view-dependent texture mapping), a photorealistic electromechanical numerical model or virtual object is created. Figure 5A illustrates an image of a machine and/or physical electromechanical component, e.g., a clamped beam, in which the machine and/or physical electromechanical component is intensified with a grainy pattern. This image is then used to create a textured virtual object. Figure 5B illustrates an approximate portion of the FE model surface (comprising triangles) where the beam is discretized and overlays the grainy patterns provided by the image (the grainy beam in the image of Figure 5A). In Figure 5B, texture is assigned to the FE mesh triangles. Finally, Figure 5C illustrates a photorealistic electromechanical numerical model or virtual object in accordance with embodiments of the present invention, more specifically, this figure illustrates a rendered FE model of the beam that closely resembles or is photoconsistent with the machine and/or component. physical electromechanical represented in Figure 5A. Embodiments of the present invention comprise an operational step, wherein the operational step comprises a measuring and estimating step. In preferred embodiments, the measurement phase or step comprises recording the machine and/or physical electromechanical component through a single or a series of videos, snapshots or images. They may be, but are not limited to, (a) the machine and/or physical electromechanical component at rest in its physical environment; (b) the machine and/or physical electromechanical component subjected to any type of internal or external excitation whether static, steady state, temporary, etc. The machine and/or physical electromechanical component may be positioned in a test rig and may be subjected to internal or external excitation (including zero excitation) that leads to "deformed distributed fields". They may be different from the distributed fields (deformed or not deformed) captured at these points (a) and (b). In preferred embodiments, the machine and/or physical electromechanical component is subjected to any type of internal or external excitation being static, steady state, temporary, etc. The machine and/or physical electromechanical component may be positioned in its working environment and may be subjected to internal or external excitation (including zero excitation) that leads to "deformed distributed fields". They may be different from the distributed fields (deformed or not deformed) captured at these points (a) and (b).

[0131] In preferred embodiments, any situation in which deformed or distributed distributed fields are created and are suitable for recording and/or imaging can be taken (e.g., the machine and/or physical electromechanical component is visible).

[0132] Following one of the aforementioned scenarios, a series of video records or images (one or more) are captured.

[0133] Potential hardware and/or software that can be used to enable the measurement step can be as follows:

[0134] • Image acquisition and motion tracking software: e.g., image and video acquisition, blob tracking, feature detection, etc. They can be commercial and research codes.

[0135] • Still or video cameras: Depending on the desired precision, cameras can range from low-fidelity media, such as webcams, to high-fidelity, high-speed megapixel cameras. Images and videos can be acquired in grayscale or color.

[0136] • Sensor acquisition system: in the measurement phase, measurements other than visual measurements can be acquired to complement visual measurements, for example, accelerations, speeds, positions, deformations, temperatures, etc. Dedicated data acquisition systems (DAQs) can be used for this purpose following standard measurement procedures.

[0137] • Data transfer from the camera to the target PC can be performed with any type of protocol depending on the speed and amount of data to be transferred, for example, Cameralink, USB3, GiGe, CoaXPress, etc.

[0138] • Frame grabbers: Given the potentially large amount of data acquired, a frame grabber may be necessary to allow a fast enough connection between the cameras and a storage medium, such as a RAM memory, a hard drive or an SSD or any type of memory that can be used for this purpose;

[0139] The images collectively provided as Figure 6 illustrate several examples of physical field measurements (images) of the cantilever beam that are acquired under different loaded conditions (e.g., external (static) excitation) and views.

[0140] In embodiments of the present invention, the method comprises an estimation step, more specifically, an estimation based on a photorealistic electromechanical numerical model. In additional embodiments, a photorealistic electromechanical numerical model is used to create photoconsistent images or videos or snapshots resulting in "photorealistic field measurements" or images of the photoconsistent electromechanical numerical model. In practice, deformed or distributed fields will cause a variation (locally and/or globally) of distributed field measurements (or photorealistic virtual measurements - e.g. photoconsistent electromechanical numerical model images), for example (but without limitation), intensities of pixels, which leads to differences between the images of the machine and/or physical electromechanical component and the images of the photorealistic electromechanical numerical model. The images of the photorealistic electromechanical numerical model can be compared by different means with the images of the machine and/or physical electromechanical component. If these images present a mismatch, the photorealistic electromechanical numerical model can be updated or modified by one of the techniques described below (or any other technique that achieves the same objective) until the mismatch is minimized in some sense. Updating can be achieved by varying or perturbing the excitation, parameters, or system states of the photorealistic electromechanical numerical model. In simpler terms, when the mismatch between the images of the photorealistic electromechanical numerical model and the images of the machine and/or physical electromechanical component is completely removed or minimized, then the excitation, parameters or states of the virtual object will approach the excitation , parameters or states of the machine and/or physical electromechanical component and will provide an estimate of the foregoing. Preferably, but not necessarily, the matching approach uses an optimal trade-off between the virtual field measurement uncertainty in the numerical model and the field measurement uncertainty in the machine and/or physical electromechanical components, this can be achieved by techniques related to , for example, Kalman-based filter and/or Moving Horizon estimators.

[0141] In particular, during updating, distributed fields or distributed deformed fields are created by exciting the photorealistic electromechanical numerical model resulting in photoconsistent videos or images in a manner similar to the machine and/or physical electromechanical component. The images obtained from the photorealistic electromechanical numerical model are explored to infer information about the machine and/or physical electromechanical component. Embodiments of the present invention advantageously surpass existing procedures known in the art by enabling this step in a time-efficient and accurate manner using deformed and distributed field images including primarily distributed field measurement variations, for example (but without limitation). , pixel intensities. The use of field measurement devices (e.g., photo and video cameras) is advantageously less invasive than mounting popular discrete sensors such as force cells, accelerometers, strain gauges, etc. Furthermore, the embodiment of the present invention advantageously allows the use of (2D) images captured by a potentially single field sensor to infer information on the full 3D fields of interest. This is a clear differentiator in relation to any other available technique and solves problems related to cost, calibration, ease of use and portability allowing for much greater productivity.

[0142] In embodiments of the present invention, estimation (during the estimation step) may occur in an online or offline manner. Online means here that estimation is performed as a parallel task, but not necessarily synchronized with image acquisition as a continuous recursive process as long as the machine and/or physical electromechanical component is under analysis. Offline here refers to the fact that a limited set of images can be stored and can potentially be ported to another physical location at later stages in time for further estimation processing. In this case, the estimation procedure becomes a post-processing step.

[0143] The following are typically anticipated (but not limiting) applications of the method in accordance with embodiments of the present invention:

[0144] • Model Update: In this application, some specific tests are performed on the machine and/or physical electromechanical component and images are captured from one or multiple views and/or sensors. During testing, a known (potentially unknown) excitation causes the machine and/or physical electromechanical component to vary its physical state relative to the reference state used, for example, for model calibration and creates distributed fields or distributed strain fields which in turn will result in variations of the captured physical field measurements (or images/videos). As an example, one can think of mechanical components that undergo movement and/or deformation. These movements and deformations will cause a variation (potentially zero) in the pixel intensities of the captured images in relation to any other reference configuration. The photorealistic electromechanical numerical model can be excited with the same known excitation that was applied to the machine and/or physical electromechanical component in order to create approximations of the deformed or distributed distributed fields passed through the machine and/or physical electromechanical component. Photorealistic field measurements are simulated in the numerical model (e.g. images from the photorealistic electromechanical numerical model). If the numerical model's photorealistic field measurements and the physical field measurements do not match a desired level of accuracy, the photorealistic numerical model's parameters can then be updated until the match is achieved. Matching can be achieved due to (but not only) a combination of optimization techniques, DIC techniques, feature extraction, etc. In one embodiment, pixel intensities are used to create a correlation or similarity metric that is used as an objective function to be minimized through any suitable optimization strategy. As a matter of example, a machine and/or physical electromechanical component that undergoes movement and/or deformation is considered. The photorealistic electromechanical numerical model is deformed, and the applied texture that resembles the appearance of the machine and/or physical electromechanical component deforms accordingly. The photorealistic field measurements in the numerical model can then be simulated due to known techniques such as rendering to texture. Calculations can be performed, for example, on a CPU or a GPU.

[0145] The parameters of the photorealistic electromechanical numerical model can be updated until the correspondence between the photorealistic field measurements in the numerical model and the physical field measurements is satisfactory (e.g., they present a pixel intensity distribution as similar as possible to each other , meaning that the images of the photorealistic electromechanical numerical model are similar to the images of the machine and/or physical electromechanical component). A satisfactory match can be achieved, for example, by updating parameters. These may be, but are not limited to, material parameters, connection stiffness, damping, etc. DIC techniques make it possible to track pixel intensity variations with subpixel-level precision so that movement or deformation that is not usually visible to the human eye can potentially be detected. Possible application cases are (but without limitation): (a) upgrading the kinematic characteristics of the (industrial) mechanism (e.g. robotic manipulators, industrial machines involved in series production and automation - e.g. looms, laser, automotive production chains, etc.), suspension systems, etc.; (b) update material parameters of components in their operating environment (e.g. cranes, buildings, vehicles, vehicle components, robotic manipulators, industrial machines involved in series production and automation, wind turbines, etc.)

[0146] • Estimation of inputs-states-parameters: A particularly active field of research deals with the estimation of inputs, states and parameters of mechatronic, mechanical and, in general, multiphysics systems. In particular, the field of Kalman-based filtering and Moving Horizon Estimation (MHE) are often used as frameworks to combine localized measurements taken, for example, of position, velocities, accelerations, deformations, temperatures, etc. If an up-to-date photorealistic electromechanical numerical model is available, various field measurements of the machine and/or physical electromechanical component can be captured during real operating conditions or under test, for example, on a dedicated test apparatus. Excitations can be applied to the photorealistic electromechanical numerical model until the virtual field measurements (e.g., images and/or videos) in the numerical model closely match the images and/or videos on the machine and/or physical electromechanical component. In particular, the photorealistic electromechanical numerical model is deformed and the applied texture will deform with it, the images can then be simulated due to known techniques, such as rendering to texture, performed, for example, on a CPU or a GPU.

[0147] In one embodiment, the applied static excitations are known and there is interest in estimating states of the machine and/or physical electromechanical component and/or in estimating the time evolution of machine parameters and/or physical electromechanical component. This can be achieved with the aid of any available local or global optimization strategy suitable for the purpose (e.g. non-linear least squares optimizers, Leuvenberq Marquardt, internal point, genetic or evolutionary strategy, etc.). As a secondary but relevant by-product of the estimation, the photorealistic electromechanical numerical model allows to recover accurate estimated 2D and/or 3D deformed distributed fields of the object. In a second modality, the applied dynamic excitations are known and there is interest in estimating the states of the machine and/or physical electromechanical component and/or in estimating the time evolution of the parameters of the machine and/or physical electromechanical component. This can be achieved with the aid of one (but without limitation) of the following techniques: (1) using a Kalman-based approach (linear or non-linear) in which images are included as measurements and photorealistic field measurements represent the equations of measurements. The photorealistic field measurements in the numerical model can potentially be combined with one or more alternative sensor types, such as, for example, accelerometers, position sensors, strain gauges, etc. Kalman-based techniques include (but are not limited to) linear Kalman filter/smoothers/predictors, non-linear point sigma or extended Kalman filter, minimum variance filters, etc.; (2) Using a Kalman-based approach (linear or non-linear) in which one or more sensors, such as, for example, accelerometers, position sensors, strain gauges, etc. are used as in a traditional environment. After the estimated states/parameters are updated, the visual measurements can be used to obtain an improved correspondence of the photorealistic field measurements with the physical field measurements. This two-step approach can be performed iteratively until Kalman prediction and matching of visual measurements (e.g., images and/or videos) are concomitantly achieved (see point 1 for a non-exhaustive list of Kalman-based techniques); (3) techniques based on Moving Horizon Estimation (MHE) in which photorealistic field measurements are used as measurement equations; and/or (4) Any other method (e.g., optimization-based or filter-based) that allows you to combine measurements and numerical modeling to obtain improved estimates of states and/or parameters (e.g., gradient-based filters, non-gradient, particle filters and stochastic methods).

[0148] As a secondary but relevant by-product of the estimation, the photorealistic electromechanical numerical model allows to recover accurate estimated deformed distributed fields, such as, for example, position and deformation fields.

[0149] In a third modality, the applied excitations are not known and there is interest in estimating the excitations themselves and potentially the states and parameters of the object under analysis. This can be achieved by one (but not limited) of the following techniques: (1) Using a Kalman-based approach (linear or non-linear) in which images are included as measurements and the photorealistic field measurements in the numerical model represent the equations of measurements. Visual field measurements can potentially be combined with one or more alternative sensor types, such as, for example, accelerometers, position sensors, strain gauges, etc. Kalman-based techniques include (but are not limited to) linear Kalman filter/smoothers/predictors, non-linear point sigma or extended Kalman filter, minimum variance filters, etc.; (2) Using a Kalman-based approach (linear or non-linear) in which one or more sensors, such as, for example, accelerometers, position sensors, strain gauges, etc. are used as in a traditional environment. After the estimated states/parameters are updated, the visual measurements can be used to obtain an improved correspondence of the photorealistic field measurements in the numerical model with the visual field measurements in the physical object. This two-step approach can be performed iteratively until Kalman prediction and visual measurement matching are concomitantly achieved (see previous point for a non-exhaustive list of Kalman-based techniques); (3) techniques based on Moving Horizon Estimation (MHE) in which photorealistic field measurements in the numerical model are used as measurement equations; (4) Any other method (e.g., optimization-based or filter-based) that allows you to combine measurements and numerical modeling to obtain improved estimates of excitations and/or states and/or parameters (e.g., gradient-based filters, in non-gradient, particle filters and stochastic methods). As a secondary but relevant by-product of the estimation, the photorealistic electromechanical numerical model allows to recover accurate estimated deformed distributed fields.

[0150] Potential hardware and software that can be used in the estimation phase can be the following:

[0151] • Modeling environment: Research or commercial software can be used to perform dynamic and static simulations (e.g., finite element, finite volume, finite difference software, multibody (flexible) software)

[0152] • Image/field matching software: Image matching can be performed with dedicated ad-hoc software (research or commercial if available) that allows you to perform any or some form of image correlation and matching (e.g. a software that is used to perform image matching between the photorealistic model and the physical system/component using pixel intensities on single pixels or subsets of pixels, DIC software, etc.).

[0153] • Optimization and estimation software: Parameter and state input estimation can be performed due to (adaptation of) Kalman filter-based libraries, MHE-based libraries, Luenberger observer libraries and optimization algorithms, such such as, but not limited to, interior point methods, Levenberg-Marquardt, linear and non-linear least squares, etc. They can be commercial and research codes.

[0154] Figures 7A to 7D summarize the estimation step according to embodiments of the present invention. Figure 7A illustrates a photorealistic image made by applying external excitations to the photorealistic electromechanical numerical model (photorealistic beam). More specifically, it provides evidence of a deformed photorealistic field measurement in the numerical model used in embodiments of the present invention. Figure 7B provides three images that represent the subtraction between the physical field measurement (physical beam image) and the photorealistic field measurement in the numerical model (image created using the photorealistic beam model). Each figure represents a different excitation applied to the photorealistic electromechanical numerical model. It is clear that the rightmost figure shows a higher degree of similarity (being mostly black, which shows that the difference between the two images is small). The correspondence of physical field measurement and photorealistic field measurement is illustrated here, in that the right-hand figure shows better similarity compared to the first two, which is an indication that the excitation is captured well. Figure 7C shows a graph in which the horizontal geometric axis represents the load applied to the photorealistic electromechanical numerical model and a cost function that represents a measurement of the similarity between the physical field measurements (physical image) and the photorealistic field measurement (obtained from the photorealistic electromechanical numerical model). The cost function (image matching similarity measurement) versus a single external excitation is thus provided. A minimum is clearly found around the location of the exact load that was applied to the physical cantilever beam. Figure 7D shows a graph in which the horizontal plane represents the amplitude of the two external loads applied to the photorealistic electromechanical numerical model and a cost function representing a measurement of the similarity between the physical field measurements (physical image) and the field measurement photorealistic in numerical (obtained from the photorealistic electromechanical numerical model). The cost function (image matching similarity measurement) versus multiple external excitation is thus provided. A minimum is clearly found around the location of the exact load that was applied to the physical cantilever beam. One direction is clearly "stiffer" than the other as the minimum exists but is less pronounced. Figure 8 illustrates a detailed representation of a workflow in which the operational phase, which comprises a measurement and estimation step, is represented and combined. The different steps/features are shown in the table below. The left part of the graph illustrates the measurement phase that can be started before or concurrently with the estimation phase (given in the right part of the graph). The flowchart shows a potential combination with more standard estimation workflows including discrete sensors. The boxes highlighted in bold highlight the innovative parts proposed in the embodiments of the present invention. Figure 9 shows a hardware-software system in accordance with embodiments of the present invention that can be used to implement embodiments of a method in accordance with the present invention. Figure 10 illustrates a summary of the method in accordance with embodiments of the present invention. The aforementioned description details certain embodiments of the present invention. It will be understood, however, that no matter how detailed the foregoing appears in the text, the present invention may be practiced in many ways. It should be noted that the use of particular terminology in describing certain features or aspects of the disclosure should not be taken to imply that the terminology is being redefined herein to be restricted to include any specific characteristics of the features or aspects of the disclosure with which this terminology is associated.

[0155] Although the above detailed description has shown, described and pointed out innovative features of the invention as applied to various embodiments, it will be understood that various omissions, substitutions and changes in the form and details of the illustrated device or process may be made by those skilled in the art. without moving away from the invention.

Claims (23)

1. Method for detecting a physical object, comprising: - providing a photorealistic virtual object of the physical object; - performing a measurement step, wherein the measurement step comprises recording the physical object and acquiring physical field measurements of the physical object comprising at least a two-dimensional image; - performing an estimation step, wherein the estimation step comprises applying external excitations to the photorealistic virtual object to create photorealistic virtual field measurements and comparing the photorealistic virtual field measurements with the physical field measurements and therefore detecting the physical object , characterized by the fact that the photorealistic virtual object reproduces pixel intensities of an image of the physical object and the at least one two-dimensional image is at least one two-dimensional image to provide image pixel intensities, and wherein said comparison comprises comparing intensities pixel strengths of the photorealistic virtual field measurements with corresponding pixel strengths of the physical field measurements. 2. Method according to claim 1, characterized by the fact that the method comprises using a time history of at least one two-dimensional image. 3. Method, according to any of the previous claims, characterized by the fact that detecting a physical object comprises estimating one or more of the following quantities of the physical object: • States: including, but not limited to, positions, velocities, accelerations, deformations, strain rates, currents; • Input: including, but not limited to, mechanical forces, mechanical torques, mechanical pressures, tensions; • Parameters: including, but not limited to, density, Young's moduli, Poisson's ratios, material parameters, physical dimensions, resistance, capacitance. 4. Method, according to claim 3, characterized by the fact that the method comprises providing a time history of the estimation of said quantities of the physical object. 5. Method, according to any of the previous claims, characterized by the fact that the physical object is a machine and/or electromechanical component. 6. Method according to any one of the preceding claims, characterized by the fact that providing a photorealistic virtual object comprises creating a photorealistic numerical model of the physical object. 7. Method according to any one of the preceding claims, characterized in that carrying out a measuring step comprises combining outputs from physical sensors of which at least one is an imaging device to visualize the external surface of the physical object in at least a two-dimensional image. 8. Method according to any one of the preceding claims, characterized by the fact that performing an estimation step comprises combining the photorealistic virtual field measurements with the physical field measurements to provide an estimate of the desired quantity or quantities and wherein the The estimation step is based, at least, on the use of a similarity metric between at least one two-dimensional image of the physical object or parts thereof, and the images generated by the photorealistic virtual object. 9. Method according to any of the preceding claims when dependent on claims 5 and 6, characterized by the fact that performing an estimation step comprises combining the photorealistic numerical model and the measurement step to provide an estimate of desired electromechanical quantities and wherein the estimation step is based, at least, on the use of a similarity metric between at least one two-dimensional image of the electromechanical machine, or parts thereof, and the images generated by the photorealistic numerical model. 10. Method according to any one of the preceding claims, characterized by the fact that the physical behavior of the photorealistic virtual object of the physical object is described by a static or dynamic model obtained as a discretized approximation of a system. 11. Method according to claim 10, characterized by the fact that the discretized approximation of the system is described by one or a combination of ordinary differential, partial differential or differential algebraic equations, finite element model, computational fluid dynamics model , flexible multibody model. 12. Method according to any one of the preceding claims, characterized by the fact that the photorealistic virtual object of the physical object uses one or more linear and non-linear model order reduction techniques. 13. Method according to claim 12, characterized by the fact that the one or more linear or nonlinear model order reduction techniques are one or more of component mode synthesis, Krylov-based methods, correct orthogonal decomposition , dynamic mode decomposition, balanced truncation, discrete empirical interpolation method, sampling and energy conservation weighting. 14. Method, according to any of the previous claims, characterized by the fact that the model is solved in the time domain. 15. Method, according to any of the previous claims, characterized by the fact that the estimate is obtained from dynamic filtering techniques that combine videos, images and the photorealistic numerical model. 16. Method according to claim 15, characterized by the fact that the dynamic filtering techniques comprise one or more of Kalman-based techniques, Moving Horizon Estimation techniques or Luenberger observer. 17. Method, according to any of the previous claims when dependent on claim 3, characterized by the fact that the estimation step allows evaluation of stochastic error limits of the estimated magnitudes of the physical object. 18. Method according to any one of the preceding claims when dependent on claim 3, characterized by the fact that the method comprises measuring and using a subset of the magnitudes of the physical object as a known input to the photorealistic virtual object. 19. Method according to any of the preceding claims, characterized by the fact that the method comprises measuring and using additional electromechanical quantities, in addition to videos and camera images during the estimation step. 20. Method according to any one of the preceding claims, characterized by the fact that providing a photorealistic virtual object comprises creating a numerical model of the physical object and texturing the geometry of the model with at least one image. 21. Method according to any one of the preceding claims, characterized by the fact that the physical field measurements of the physical object are deformed physical fields. 22. Device characterized by the fact that it comprises a processing system, the processing system being programmed to carry out a method as defined in any one of claims 1 to 21. 23. System for characterizing a physical object, the system being characterized by the fact that it comprises: • at least one means for measuring quantities of the physical object; • a processor adapted to receive the output of at least one means for measuring fields of the physical object and adapted to carry out the method as defined in any one of claims 1 to 21.