KR20230001370A - Real time visualization apparatus and system using augmented reality, real time visualization method using the same and computer readable recording medium for the same - Google Patents

Abstract

The present invention relates to a real-time visualization device, system, and method using augmented reality (AR). According to an embodiment of the present invention, the real-time visualization device using AR, which is able to display a visual model reflecting the physical status of a subject in real time by using AR, receives data including a reduced response acquired from a physical model showing the physical status of the subject, recovers the total response based on the received data, and generates a visual model of the subject by using an AR module from the recovered total response. The reduced response can be a response acquired by applying a model order reduction method to the physical model. Therefore, the status of a subject can be checked in real time.

Description

Real-time visualization device and system using augmented reality, real-time visualization method using the same, and computer-readable recording medium therefor

The present invention relates to a real-time visualization device and system and a method for using the same, and more particularly, to a device and system capable of visualizing the state of a structure, facility or system in real time using augmented reality, and an invention related to the method. .

Recently, smartization is actively progressing in the field of manufacturing, construction, or safety diagnosis, and there is a smart factory at the cutting edge. A smart factory is an intelligent production factory that improves productivity, quality, and customer satisfaction by applying information and communication technology (ICT) combined with digital automation solutions to production processes such as design and development, manufacturing and distribution. (IoT) is installed to collect process data in real time, analyze it, and control it on its own.

Conventionally, since automation was performed for each process in manufacturing a specific product or implementing a system, it was difficult to organically manage the entire process or the entire system, and it was difficult to grasp the current state intuitively and in real time. On the other hand, smart factories can organically link data between processes by utilizing information and communication technology, so that the entire process or system can be integrated and managed. To this end, simulation technology represented by the digital twin is attracting attention.

The digital twin is a concept advocated by General Electric (GE) of the United States, and is implemented on a computer by converting objects such as structures or systems into software. The result can be predicted in advance by simulating situations that may occur in the target object using the digital twin.

However, in the conventional visualization technology using a digital twin, it is difficult to understand the object as a whole because it measures and visualizes only the state of a local area where a sensor is installed, rather than expressing the entire state of the object. In addition, there is a problem in that a lot of communication is required to visualize data in real time, and since the entire physical quantity to be visualized is transmitted and received, it is greatly affected by the communication environment.

Korean Patent Publication No. 10-2011-0135461

The present invention is an invention to solve the above problems, and a real-time visualization device capable of visualizing the entire state of an object to deliver intuitive information in real time and reducing dependence on a communication environment through data transmission based on a reduced model, and It provides a system and a real-time visualization method.

However, these problems are exemplary and the problems to be solved in the present invention are not limited thereto.

A real-time visualization apparatus using augmented reality according to an embodiment of the present invention is an apparatus for displaying a visual model reflecting the physical state of an object in real time using augmented reality, obtained from a physical model representing the physical state of the object. Receiving data including a scaled-down response, recovering a full response based on the received data, generating a visual model of the object from the restored full response using an augmented reality module, and the scaled-down response is the physical model This is the response obtained by applying the model order reduction method to .

In the real-time visualization device using augmented reality according to an embodiment of the present invention, the reduced response is an order-reduced response from the total response by a conversion matrix according to the model order reduction method, and the real-time visualization device may recover the entire response from the reduction response using the pre-stored transition matrix.

In the real-time visualization device using augmented reality according to an embodiment of the present invention, the real-time visualization device may display the visual model when recognizing the object.

In the real-time visualization device using augmented reality according to an embodiment of the present invention, if the real-time visualization device satisfies a condition including at least one of the following (1) to (3), the photo or image of the object is displayed. A real-time visualization device using augmented reality that displays the visual model in an overlapping manner. (1) The real-time visualization device recognizes a marker attached to the object (2) The real-time visualization device recognizes the shape of the object (3) The real-time visualization device receives specific location information

In the real-time visualization device using augmented reality according to an embodiment of the present invention, the real-time visualization device communicates with the outside and includes a communication module for receiving data on the object, and an operation for restoring the entire response from the received reduction response. module, a photographing module for photographing the object, and a display module for displaying the visual model.

A real-time visualization system using augmented reality according to an embodiment of the present invention is a system that displays a visual model reflecting the physical state of an object in real time by using augmented reality, obtained from a physical model representing the physical state of the object. A real-time visualization device for receiving data including a scaled-down response, restoring an entire response based on the received data, and generating a visual model of the object using an augmented reality module from the restored entire response, disposed on the object a sensor that collects data on a physical state of the object and a server that inputs data received from the sensor into a pre-stored physical model of the object and obtains the scale-down response, wherein the scale-down response corresponds to the physical model A response obtained by applying a model order reduction method, and the real-time visualization device receives the reduction response from the server.

A real-time visualization method using augmented reality according to an embodiment of the present invention is a method of displaying a visual model reflecting the physical state of an object in real time using augmented reality, wherein a real-time visualization device monitors the physical state of the object from the outside. Receiving data including a scaled-down response obtained from an representing physical model, restoring an entire response based on the received data by the real-time visualization device, augmented reality module from the entire response recovered by the real-time visualization device generating a visual model of the object using and displaying the visual model by the real-time visualization device, wherein the reduction response is a response obtained by applying a model order reduction method to the physical model.

In the real-time visualization method using augmented reality according to an embodiment of the present invention, the reduction response is a response reduced in order from the total response by a conversion matrix according to the model order reduction method, and the step of restoring the total response The real-time visualization device may recover the entire response from the reduction response using the pre-stored transition matrix.

In the real-time visualization method using augmented reality according to an embodiment of the present invention, the displaying of the visual model may display the visual model when the real-time visualization device recognizes the object.

In the real-time visualization method using augmented reality according to an embodiment of the present invention, the step of displaying the visual model is performed when the real-time visualization device satisfies at least one of the following conditions (1) to (3). , The visual model may be overlapped and displayed on a photo or image of the object. (1) The real-time visualization device recognizes a marker attached to the object (2) The real-time visualization device recognizes the shape of the object (3) The real-time visualization device receives specific location information

In the real-time visualization method using augmented reality according to an embodiment of the present invention, before the real-time visualization device receives data, the sensor disposed on the object collects data about the physical state of the object; The method may further include acquiring the reduction response by inputting data received from the sensor to a physical model of the object stored in advance by a server, and receiving the data by the real-time visualization device from the server. The reduction response may be received.

The recording medium according to an embodiment of the present invention is computer-readable, in which a program for executing the above-described real-time visualization method is recorded.

Other aspects, features and advantages other than those described above will become apparent from the following drawings, claims and detailed description of the invention.

The real-time visualization system and method according to an embodiment of the present invention may check the state of the object in real time using data detected by a sensor in a state in which a physical model (or digital twin) of the object is constructed.

In particular, the real-time visualization system and method according to an embodiment of the present invention does not transmit and receive the entire response obtained by inputting (simulating) data into a physical model, but acquiring it in a reduced form to reduce the time and communication required for data transmission and reception. can be minimized.

In addition, the real-time visualization system and method according to an embodiment of the present invention may perform simulation in real time using data received from a sensor by using the reduction response and generate a visual model based on the real-time simulation.

1 shows a real-time visualization system according to an embodiment of the present invention.
2 shows a real-time visualization device according to an embodiment of the present invention.
3 to 5 show states in which a real-time visualization device according to an embodiment of the present invention recognizes an object.
6 shows real-time visualization results according to an embodiment of the present invention.
7 shows a block diagram of a real-time visualization system according to an embodiment of the present invention.
8 shows an operation according to a real-time visualization system according to an embodiment of the present invention.
9 shows a state of obtaining a scaled-down response according to an embodiment of the present invention.

Since the present invention can apply various transformations and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the description of the invention. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all conversions, equivalents, or substitutes included in the spirit and scope of the present invention. In describing the present invention, even though it is shown in one embodiment, the same identification code is used for the same component.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, and when describing with reference to the drawings, the same or corresponding components are assigned the same reference numerals, and overlapping descriptions thereof will be omitted. .

In the following embodiments, terms such as first and second are used for the purpose of distinguishing one component from another component without limiting meaning.

In the following examples, expressions in the singular number include plural expressions unless the context clearly dictates otherwise.

In the following embodiments, terms such as include or have mean that features or components described in the specification exist, and do not preclude the possibility that one or more other features or components may be added.

In the drawings, the size of components may be exaggerated or reduced for convenience of description. For example, since the size and thickness of each component shown in the drawings are arbitrarily shown for convenience of description, the present invention is not necessarily limited to the illustrated bar.

In the following embodiments, the x-axis, y-axis, and z-axis are not limited to the three axes of the Cartesian coordinate system, and may be interpreted in a broad sense including these. For example, the x-axis, y-axis, and z-axis may be orthogonal to each other, but may refer to different directions that are not orthogonal to each other.

When an embodiment is otherwise implementable, a specific process sequence may be performed differently from the described sequence. For example, two processes described in succession may be performed substantially simultaneously, or may be performed in an order reverse to the order described.

Terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. In this application, the terms "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

In this specification, a target object may mean a physical device, system, or process. In the present specification, a physical model (or digital twin) is an object capable of obtaining a response by inputting data by identically implementing an object in software, and may be implemented through a finite element method, a machine learning technique, and the like. In the present specification, a model order reduction method is a reduction technique for reduced order modeling (ROM), and includes both a degree-of-freedom-based reduction method, a partial structure synthesis method, and the like. In this specification, the reduced model means a model in which the order is reduced by applying the model order reduction method to the physical model. In this specification, the total response represents a result value obtained by inputting data into a physical model, that is, a physical quantity (displacement, acceleration, temperature, etc.) and state of a structure calculated from the physical model. In this specification, the reduction response refers to a response obtained by applying a model order reduction method to a physical model (for example, a response obtained by applying a model order reduction method to the entire response or a response obtained by entering data into a reduced model). do. In this specification, "obtaining a model or response" means obtaining a result value by calculating a predetermined formula.

1 shows a real-time visualization system 10 according to an embodiment of the present invention, FIG. 2 shows a real-time visualization device 100 according to an embodiment of the present invention, and FIGS. A state in which the real-time visualization apparatus 100 according to the embodiment recognizes the object S is shown.

The real-time visualization system 10 according to an embodiment of the present invention can visualize the state of the object S in real time by using augmented reality and physical data including the temperature, displacement, pressure or load of the object S. there is.

In FIG. 1, the object (S) is shown as a pipe, but the type of object (S) is not particularly limited. The target object S may include all physical devices, systems, and processes such as mechanical devices, structures, plants, or factory facilities.

The real-time visualization system 10 according to an embodiment of the present invention may include a real-time visualization device 100, a sensor 200, and a server 300.

The real-time visualization device 100 according to an embodiment of the present invention may be a device capable of displaying a visual model reflecting the physical state of an object S in real time using augmented reality.

In one embodiment, the real-time visualization device 100 may be any one of various types of user terminals such as smart phones, tablet PCs, and laptops.

In an embodiment, the real-time visualization apparatus 100 receives data including a scaled-down response obtained from a physical model representing the physical state of the object S, recovers the entire response based on the received data, and restores the entire restored overall response. A visual model of the object S may be generated from the response.

In one embodiment, the real-time visualization device 100 may include a communication module 110, a calculation module 120, an augmented reality module 130, a display module 140, and a photographing module 150.

The communication module 110 may transmit/receive data with an external device or transmit/receive data with other components inside the real-time visualization device 100 .

In one embodiment, the communication module 110 may receive data including a scaled-down response obtained from a physical model representing the physical state of the object S from the outside of the real-time visualization device 100 .

More specifically, the physical model (or digital twin) is a software-implemented model of the object (S), and appropriate governing equations according to the response type (eg temperature, displacement, speed, acceleration, stress, flow rate, etc.) to be obtained. can be implemented through The physical model of the object S may be pre-stored in an external device separate from the real-time visualization device 100, and may be pre-stored in the server 300, for example.

The physical model of the object S is a software-implemented model of the object S, and a desired response may be obtained by inputting values to variables of a governing equation. For example, a response regarding the physical state of the object S may be obtained by inputting physical data of the object S sensed by the sensor 200 into a physical model. Hereinafter, a response obtained by inputting data into the physical model of the object S and not performing any preprocessing is referred to as an overall response.

In one embodiment, the physical model is a virtual sensor, and a new type of value may be created by combining data received from the sensor 200, which is a physical sensor. For example, the physical model may represent a new type of value, such as a safety factor or a process state of the object S, based on the temperature, displacement, and the like received from the sensor 200 .

The reduction response refers to a response obtained by applying a model order reduction method to a physical model. For example, a reduction response may be obtained by inputting data about the physical state of the object S detected by the sensor 200 to a reduced model obtained by reducing the order of the physical model using a model order reduction method. Alternatively, data on the physical state of the object S detected by the sensor 200 may be input to the physical model to obtain an overall response, and a reduced response may be obtained by applying a model order reduction method to the obtained overall response.

In one embodiment, the reduction response is based on the characteristics of the physical model, when the data of the object (S) is input into the physical model, key factors affecting the response are selected, and a transformation matrix (T) is used. It can be a shortened response.

For example, by inputting the physical data (displacement, velocity, acceleration, temperature, flow rate, etc.) of the object S received by the sensor 200 into the physical model, an overall response is obtained, and preprocessing is performed on the obtained overall response. A reduced response can be computed. Here, as a model order reduction method for obtaining a reduction response, a known method such as dimensionality reduction using matrix decomposition may be used. For example, Guyan Reduction, Improved Reduction System (IRS), Craig-Bampton Method or Enhanced Craig-Bampton Method may be used.

More specifically, when it is desired to obtain the displacement of the object S as a response, an equation of motion in the mass-spring system may be used as a governing equation of the physical model as follows.

), 속도(

) 및 변위(

)의 형태를 갖는다. 또한 물리 모델은 질량 행렬(M), 감쇠 행렬(C) 및 강성 행렬(k)을 갖는다. The response obtained with the above equation is the acceleration (

), speed(

) and displacement (

) has the form of Also, the physical model has a mass matrix (M), a damping matrix (C) and a stiffness matrix (k).

A transition matrix T may be used to reduce such a physical model. The conversion matrix (T; or conversion matrix) is a matrix calculated by selecting key factors that affect the response obtained by inputting data to the physical model, and is used to reduce the order (size of the matrix or vector) of the physical model. can be used The conversion matrix T may vary according to the type of reduction method, and the aforementioned known reduction method may be used as the reduction method.

The equation reduced by the transition matrix T may have the following form.

), 속도(

) 및 변위(

)의 형태를 갖는다. 또한 축소 모델(또는 축소 행렬)은 질량 행렬(

), 감쇠 행렬(

) 및 강성 행렬(

)을 갖는다.The response obtained with the reduction equation is the acceleration (

), speed(

) and displacement (

) has the form of Also, the reduction model (or reduction matrix) is the mass matrix (

), the decay matrix (

) and the stiffness matrix (

) has

That is, a response in a reduced form may be obtained by multiplying the physical model or the entire response by the conversion matrix T and reducing the size of the matrix or vector. The real-time visualization device 100 according to an embodiment of the present invention can dramatically reduce the amount of communication compared to the case of receiving the entire response by receiving the reduced response instead of the entire response.

Unlike data compressed by a general communication data compression technique, the response reduced by the transition matrix T may include physical information (displacement, velocity, acceleration, temperature, flow rate, etc.) of the object S of the entire response. can

In an embodiment, a subject that receives physical data of the object S from the sensor 200, inputs the physical data of the object S to the physical model of the object S, and obtains a reduction response may be an external device of the real-time visualization device 100. there is. For example, the operation may be performed by a user terminal separate from the real-time visualization device 100, a computing device, or another computing device. Alternatively, the above operation may be performed by the server 300 shown in FIG. 1 .

Returning to FIG. 1 again, as described above, the communication module 110 may receive a reduction response of the object S from the outside. Here, the outside may be the server 300 or a separate user terminal or computing device.

The calculation module 120 may receive the data transmitted from the communication module 110 and recover the entire response based on the received data.

For example, the arithmetic module 120 may restore the entire response by multiplying the reduction response by the transition matrix T previously stored in the real-time visualization device 100 . The total response recovered here may be the same as the total response obtained by inputting data on the physical state of the object S sensed by the sensor 200 into the physical model.

Accordingly, the real-time visualization device 100 may receive the entire response in the form of a reduced response without having to receive the entire response, and then obtain the entire response using the transition matrix T.

In one embodiment, the calculation module 120 may quickly recover the entire response through parallel computing that decomposes the components of the reduced response and allocates work to each thread of the GPU.

The augmented reality module 130 may generate a visual model of the object S using the entire response recovered by the calculation module 120 . The augmented reality module 130 is not particularly limited, as long as it can visualize the object S from the entire response, and may be stored in the real-time visualization device 100 in the form of an application or program. For example, the augmented reality module 130 receives the physical model of the object S previously stored in the real-time visualization device 100 and the entire restored response, and generates a visual model of the object S in the form of an application or program. It can be an implemented platform.

In addition, the augmented reality module 130 may implement a visual model of the object S in augmented reality and display it along with a photo or image of the object S together with the display module 140 .

The display module 140 may display the current state of the real-time visualization system 10, the real-time visualization device 100, or the object S in real time. For example, the display module 140 is a display of the real-time visualization device 100 and may display data of the object S or a visual model of the object S received by the communication module 110 in real time.

In one embodiment, the real-time visualization apparatus 100 may display a visual model when recognizing the object S.

For example, when recognizing the marker M attached to the object S, the real-time visualization apparatus 100 may overlap and display a visual model on a photo or image of the object S. For example, as shown in FIG. 3 , when the photographing module 150 to be described later recognizes the marker M, the display module 140 displays a picture or image of the object S captured by the photographing module 150 or being photographed. A visual model may be superimposed on an image and displayed.

The type of marker M is not particularly limited, and patterns or images of various shapes may be used.

Alternatively, when recognizing the shape of the object S, the real-time visualization apparatus 100 may overlap and display a visual model on a photo or image of the object S. For example, as shown in FIG. 4 , in a state in which information on the shape (size, shape, color, texture, etc.) of the object S is previously stored in the real-time visualization device 100, the real-time visualization device 100 captures images. A photo or image of the object S captured by the module 150 may be compared with information about the shape of the object S. If the comparison results match, the display module 140 may superimpose and display the visual model on a photo or image of the object S photographed or being photographed by the photographing module 150 .

Alternatively, the real-time visualization apparatus 100 may determine the location and direction of the object S using the location information and direction, and display the visual model superimposed on a photo or image of the object S. For example, as shown in FIG. 5 , the real-time visualization device 100 detects the direction that the user or the real-time visualization device 100 is facing using a built-in sensor (eg, a gyro sensor, an acceleration sensor, or a motion sensor), and uses GPS. Receive location information using a sensor. And, if the received location information and direction coincide with the pre-stored location information and direction of the object S, the display module 140 visualizes a photo or image of the object S taken or being captured by the photographing module 150. Models can be overlaid and displayed.

According to an embodiment, the real-time visualization device 100 may overlap and display a visual model on a photo or image of the object S, if at least one of the following conditions (1) to (3) is satisfied. (1) The real-time visualization device 100 recognizes the marker M attached to the object S (2) The real-time visualization device 100 recognizes the shape of the object S (3) The real-time visualization device 100 receives specific location information

In addition to this, the real-time visualization device 100 recognizes the object S in various ways, and accordingly, a photo or image of the object S taken in advance or a photo or image currently taken by the object S is used as a visual model. can be displayed overlapping.

6 illustrates an example of a state in which the display module 140 superimposes a visual model of the object S on the image of the object S captured by the photographing module 150 in real time. As shown in FIG. 6 , the display module 140 may display the visual model on the image of the object S and display the physical quantity of the object S as numerical values and graphs. In addition, the display module 140 is a touch-type display panel capable of receiving a user's input, and displays a menu capable of receiving a user's input (eg, displaying different physical quantities (eg, pressure or displacement) according to a user's input). available menu).

For example, when the user selects the pressure menu displayed by the display module 140, the display module 140 may display a visual model representing the pressure of the object S. Alternatively, when the user selects the displacement menu displayed by the display module 140, the display module 140 may display a visual model representing the displacement of the object S.

The capturing module 150 may capture a video or image of the object S. In one embodiment, the photographing module 150 is a camera, and when the photographing module 150 captures a picture or video of the object S, the real-time visualization device 100 may superimpose the visual model on the photograph or video and display the image. there is.

Referring to FIGS. 1 and 7 , the real-time visualization system 10 according to an embodiment of the present invention may further include a sensor 200 and a server 300.

One or more sensors 200 may be disposed on the object S, detect and collect data of the object S, and transmit the data to the server 300 and/or the real-time visualization device 100. The sensor 200 does not necessarily need to be attached to the object S, and may be spaced apart from the object S.

The type of sensor 200 and the data sensed by the sensor 200 are not particularly limited. For example, the sensor 200 may be one of sensors including an infrared sensor, an ultrasonic sensor, a laser sensor, a temperature sensor, a humidity sensor, and an acceleration sensor, or a combination thereof. In addition, the sensor 200 may be disposed on the object S to measure the temperature, load, displacement, etc. of the object S.

In one embodiment, the sensor 200 may be connected to the server 300 and/or the real-time visualization device 100 wired or wirelessly. The sensor 200 may transmit the detected data of the object S to the server 300 and/or the real-time visualization device 100 in real time. For example, as shown in FIG. 7 , the sensor 200 may transmit data of the detected object S to the communication unit 310 of the server 300 .

The server 300 may obtain a result value, that is, a response, using data received from the sensor 200 . For example, the server 300 stores a physical model (or digital twin) of the object S, and may obtain a response by inputting data received from the sensor 200 to the physical model.

In one embodiment, the server 300 may include a communication unit 310 , a calculation unit 320 and a storage unit 330 .

The communication unit 310 is a part that communicates with an external device of the server 300 including the real-time visualization device 100 and the sensor 200, and the communication method is not particularly limited. For example, various wired and wireless communication methods may be used, including short-range wireless communication such as WiFi, Bluetooth, and NFC.

The calculation unit 320 inputs the data received from the sensor 200 to the physical model pre-stored in the storage unit 330 to obtain a response, performs a model order reduction operation, and obtains a reduced model from the physical model. .

In one embodiment, the physical model does not necessarily need to be pre-stored in the server 300, and may be acquired by calculation in the server 300 using external information of the object S and data received from the sensor 200.

In one embodiment, the calculation unit 320 may obtain a reduction response by inputting the data received from the sensor 200 to the physical model of the object S. That is, the calculation unit 320 may obtain a reduction response by performing reduction pre-processing on a response obtained by inputting data into the physical model.

For example, the calculation unit 320 may input the data received from the sensor 200 into the physical model of the object S to obtain an overall response, and obtain a reduced response by performing preprocessing on the obtained overall response. Here, as a preprocessing method for obtaining a reduction response, a model order reduction method as described above may be used.

Alternatively, the calculation unit 30 may obtain a reduction response by inputting the data received from the sensor 200 into the physical model of the object S using a reduction equation according to a governing equation of the physical model. Here, the physical model may be formed by the calculation unit 320 or to which a reduction equation has been applied in advance, and the calculation unit 320 may obtain a reduction response by inputting the data received from the sensor 200 into the reduced physical model. .

)에 입력하여 전체 응답(U)을 획득하고, 획득한 전체 응답에 축소 전처리를 실시하여 축소 응답(

)을 획득할 수 있다. 연산부(320)는 축소 응답을 실시간 시각화 장치(100)로 전송할 수 있다. 그리고 실시간 시각화 장치(100)는 후술하는 바와 같이 축소 변환 행렬(T)을 이용해 전체 응답을 복구할 수 있다

.More specifically, when obtaining the displacement of the target object S as a response, an equation of motion in a mass-spring system may be used as a governing equation of a physical model. The calculation unit 320 converts the data (eg, force, F) received from the sensor 200 into the entire equation of the physical model (

) to obtain a full response (U), and perform reduction preprocessing on the entire obtained response to reduce the response (

) can be obtained. The calculation unit 320 may transmit the reduction response to the real-time visualization device 100 . Also, the real-time visualization device 100 may recover the entire response using the reduced transformation matrix T as described later.

.

)를 축소 방정식(

)이 미리 적용된 형태의 물리 모델에 입력하여 축소 응답(

)을 획득할 수 있다. 여기서 센서(200)로부터 수신한 데이터 역시 축소 방정식에 대입하기 위해 적합한 형태로 전처리될 수 있다. 또는 센서(200)가 전처리를 실시하여 서버(300)로 데이터를 전송하거나, 서버(300)가 데이터를 전송받은 다음 전처리를 실시할 수 있다. 그리고 서버(300)는 축소 응답을 실시간 시각화 장치(100)로 전송하고, 실시간 시각화 장치(100)는 축소 변환 행렬(T)을 이용해 전체 응답을 복구할 수 있다

.Alternatively, the server 300 may receive data from the sensor 200 (

) to the reduction equation (

) is input to the physical model in the form of pre-applied form to reduce the response (

) can be obtained. Here, data received from the sensor 200 may also be pre-processed in a form suitable for substituting into a reduction equation. Alternatively, the sensor 200 may perform preprocessing and transmit data to the server 300, or the server 300 may perform preprocessing after receiving data. In addition, the server 300 transmits the reduction response to the real-time visualization device 100, and the real-time visualization device 100 can recover the entire response using the reduction transformation matrix T.

.

In FIG. 8 , when the real-time visualization device 100 makes a request to the server 300, the server 300 transmits a zoom response, but is not limited thereto. For example, the server 300 may transmit data in real time regardless of a request of the real-time visualization device 100, and the real-time visualization device 100 may transmit data from the sensor 200 according to a set time interval or a user's input. can be sent Alternatively, the server 300 may transmit data when there is a request signal from the real-time visualization device 100 .

In this way, the real-time visualization system 10 according to an embodiment of the present invention does not transmit the entire response itself to the real-time visualization device 100, but obtains the entire response as a compressed form of a reduced response, so that the real-time visualization device 100 It is possible to minimize the time required for data transmission and reception and the amount of communication by transmitting to .

As the governing equation of the physical model, the equation of motion in the mass-spring system has been mainly described, but is not limited thereto. For example, when a response to be obtained is another physical quantity such as temperature or pressure, a physical model to which governing equations are applied may be used.

Through this configuration, the real-time visualization system 10 according to an embodiment of the present invention transmits the data of the object S detected by the sensor 200 to the server 300 and/or the real-time visualization device 100 in real time. transmission, and based on this, a visual model of the object S may be displayed in real time.

In particular, the real-time visualization system 10 according to an embodiment of the present invention does not transmit and receive the entire response obtained from the physical model of the object S as it is, but transmits and receives data in the form of a reduced response to reduce the amount of communication and time. reduced, and through this, the object S can be visualized and displayed in real time.

9 shows a real-time visualization method according to an embodiment of the present invention.

In the real-time visualization method according to an embodiment of the present invention, the current state of the object S may be visualized and displayed in real time using the real-time visualization device 100 or the real-time visualization system 10 described above.

The real-time visualization method according to an embodiment of the present invention includes the steps of receiving, by the real-time visualization device 100, data including a scaled-down response obtained from a physical model representing the physical state of an object S from the outside, the real-time visualization device ( 100) recovering the entire response based on the received data, generating a visual model of the object S using the augmented reality module 130 from the entire response recovered by the real-time visualization device 100, and real-time visualization The device 100 may include displaying the visual model.

First, the real-time visualization device 100 receives data from the outside. Here, the data may include a reduction response obtained from a physical model representing the physical state of the object S.

More specifically, the physical model may be stored in advance in the server 300 or the like described above. The server 300 may receive data from the sensor 200 to obtain a zoom response and transmit it to the real-time visualization device 100 . Alternatively, the reduction response may not be a value calculated in real time by the server 300, but a value previously stored in an arbitrary database and received according to a request of the real-time visualization device 100.

In an embodiment, the reduction response may be a response obtained by inputting data of the object S sensed by the sensor 200 into a reduction model obtained from a physical model using a model order reduction method.

Next, the real-time visualization device 100 restores the entire response based on the received data, that is, the reduction response. For example, the real-time visualization device 100 may use the calculation module 120 to recover the entire response as a conversion matrix T used to obtain the scaled-down response.

Next, the real-time visualization device 100 generates a visual model of the object S using the augmented reality module 130 from the entire restored response. For example, the real-time visualization apparatus 100 may generate a visual model of the object S using a pre-stored physical model of the object S and a restored overall response.

Also, the display module 140 may superimpose the visual model of the object S on a photo or image of the object S and display the image.

In the step of generating the visual model according to an embodiment, when the real-time visualization device 100 recognizes the object S, the visual model may be displayed by the display module 140 .

In the step of displaying the visual model according to an embodiment, if a condition including at least one of (1) to (3) below is satisfied, the visual model may be overlapped and displayed on a photo or image of the object S. . (1) The real-time visualization device 100 recognizes the marker M attached to the object S (2) The real-time visualization device 100 recognizes the shape of the object S (3) The real-time visualization device 100 receives specific location information

For example, in the step of displaying the visual model, the photographing module 150 of the real-time visualization device 100 recognizes a marker M expressed as a pattern or color or image attached to the object S, or the object S When the shape of ) is recognized, it is superimposed on a photo or image of the object S photographed by the photographing module 150 or a photograph or image of the object S photographed by the photographing module 150 to display a visual model. can do.

Alternatively, the step of displaying the visual model may include, in a state where the real-time visualization device 100 stores the location information and direction of the object S, when receiving or detecting information corresponding to the location information and direction, the photographing module ( The visual model may be displayed by overlapping a photo or image of the object S captured by 150 or a photo or image of the object S captured by the photographing module 150 .

In an embodiment, the real-time visualization method includes the steps of collecting data on the physical state of the object S by the sensor 200 disposed on the object S before the real-time visualization device 100 receives the data, and the server Step 300 may further include obtaining a reduction response by inputting the data received from the sensor 200 into a pre-stored physical model of the object S.

For example, one or more sensors 200 are disposed on the object S to collect data (for example, displacement, speed, acceleration, temperature, flow rate, etc.) on the physical state of the object S, and collect the data on the server ( It can be transmitted to the communication unit 310 of 300).

In addition, the server 300 may obtain a reduction response by inputting the data received from the sensor 200 into a pre-stored physical model.

For example, the calculation unit 320 of the server 300 may receive data from the communication unit 310 and input the received data to a physical model stored in the storage unit 330 to obtain an overall response. In addition, the calculation unit 320 may obtain a reduced response from the entire response through a model order reduction method.

In an embodiment, the server 300 may obtain a reduced model from the physical model using the transition matrix T, and obtain a reduced response by inputting data detected by the sensor 200 into the reduced model.

In one embodiment, the real-time visualization device 100 may receive a reduction response from the server 300 . That is, the real-time visualization device 100 may receive the scaled-down response obtained by the server 300 and restore it as a full response using the conversion matrix T. In addition, the real-time visualization apparatus 100 may generate and display a visual model of the object S using the restored overall response and a pre-stored physical model of the object S.

The real-time visualization apparatus 100, system 10, and method according to an embodiment of the present invention use data detected by a sensor 200 in a state in which a physical model (or digital twin) of an object S is built, The status of (S) can be checked in real time.

In particular, the real-time visualization device 100, system 10, and method according to an embodiment of the present invention do not transmit and receive the entire response obtained by inputting (simulating) data into a physical model as it is, but acquiring it in a reduced form The time required for data transmission and reception and the amount of communication can be minimized.

In addition, the real-time visualization apparatus 100, system 10, and method according to an embodiment of the present invention use the reduction response as described above to perform simulation in real time using data received from the sensor 200, and based on this, visual model can be created.

Embodiments according to the present invention described above may be implemented in the form of a computer program that can be executed on a computer through various components, and such a computer program may be recorded on a computer-readable medium. In this case, the medium may store a program executable by a computer. Examples of the medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROM and DVD, magneto-optical media such as floptical disks, and ROM, RAM, flash memory, etc. configured to store program instructions.

Meanwhile, the computer program may be specially designed and configured for the present invention, or may be known and usable to those skilled in the art of computer software. An example of a computer program may include not only machine language code generated by a compiler but also high-level language code that can be executed by a computer using an interpreter or the like.

As such, the present invention has been described with reference to the embodiments shown in the drawings, but this is only an example. Those skilled in the art can fully understand that various modifications and equivalent embodiments are possible from the embodiments. Therefore, the true technical protection scope of the present invention should be determined based on the appended claims.

Specific technical details described in the embodiments are examples, and do not limit the technical scope of the embodiments. In order to briefly and clearly describe the description of the invention, descriptions of conventional general techniques and configurations may be omitted. In addition, the connection of lines or connection members between the components shown in the drawing is an example of functional connection and / or physical or circuit connection, which can be replaced in an actual device or additional various functional connections, physical connections, or circuit connections. In addition, if there is no specific reference such as "essential" or "important", it may not necessarily be a component necessary for the application of the present invention.

“Above” or similar designations described in the description and claims of the invention may refer to both singular and plural, unless otherwise specifically limited. In addition, when a range is described in an embodiment, it includes an invention in which individual values belonging to the range are applied (unless there is no description to the contrary), and each individual value constituting the range is described in the description of the invention. same. In addition, if there is no clear description or description of the order of steps constituting the method according to the embodiment, the steps may be performed in an appropriate order. Embodiments are not necessarily limited according to the order of description of the steps. The use of all examples or exemplary terms (eg, etc.) in the embodiments is simply to describe the embodiments in detail, and unless limited by the claims, the examples or exemplary terms limit the scope of the embodiments. It is not. In addition, those skilled in the art will appreciate that various modifications, combinations and changes may be made according to design conditions and factors within the scope of the appended claims or equivalents thereof.

10: Real-time visualization system
100: real-time visualization device
200: sensor
300: server
S: object

Claims (12)

An apparatus for displaying a visual model reflecting the physical state of an object in real time using augmented reality, comprising:
Receives data including a scaled-down response obtained from a physical model representing the physical state of the object, recovers an entire response based on the received data, and uses an augmented reality module from the restored entire response to view the object's view create a model,
The reduction response is a response obtained by applying a model order reduction method to the physical model, a real-time visualization device using augmented reality. According to claim 1,
The reduced response is an order reduced response from the total response by a conversion matrix according to the model order reduction method,
Wherein the real-time visualization device recovers the full response from the scaled-down response using the pre-stored transition matrix. According to claim 1,
The real-time visualization device using augmented reality, wherein the real-time visualization device displays the visual model when recognizing the object. According to claim 3,
When the real-time visualization device satisfies a condition including at least one of the following (1) to (3), the real-time visualization device using augmented reality to superimpose and display the visual model on a photo or image of the object.
(1) The real-time visualization device recognizes a marker attached to the object
(2) The real-time visualization device recognizes the shape of the object
(3) The real-time visualization device receives specific location information According to claim 1,
The real-time visualization device
a communication module that communicates with the outside and receives data about the object;
an arithmetic module recovering the full response from the received reduced response;
a photographing module for photographing the object; and
Real-time visualization device using augmented reality, including a; display module for displaying the visual model. A system that displays a visual model reflecting the physical state of an object in real time using augmented reality,
Receives data including a scaled-down response obtained from a physical model representing the physical state of the object, recovers an entire response based on the received data, and uses an augmented reality module from the restored entire response to view the object's view a real-time visualization device that creates a model;
a sensor disposed on the object to collect data on a physical state of the object; and
A server configured to obtain the reduction response by inputting the data received from the sensor into a previously stored physical model of the object;
The reduction response is a response obtained by applying a model order reduction method to the physical model,
Wherein the real-time visualization device receives the reduction response from the server. A method of displaying a visual model reflecting the physical state of an object in real time using augmented reality, comprising:
receiving, by a real-time visualization device, data including a scaled-down response obtained from an external physical model representing a physical state of the object;
restoring, by the real-time visualization device, an entire response based on the received data;
generating, by the real-time visualization device, a visual model of the object using an augmented reality module from the recovered responses; and
Including; displaying the visual model by the real-time visualization device;
The reduction response is a response obtained by applying a model order reduction method to the physical model, a real-time visualization method using augmented reality. According to claim 7,
The reduced response is an order reduced response from the total response by a conversion matrix according to the model order reduction method,
In the step of restoring the full response, the real-time visualization device restores the full response from the scaled-down response using the pre-stored transition matrix. According to claim 7,
The displaying of the visual model includes displaying the visual model when the real-time visualization device recognizes the object. According to claim 9,
In the displaying of the visual model, if the real-time visualization device satisfies a condition including at least one of the following (1) to (3), the visual model is overlapped and displayed on a photo or image of the object, Real-time visualization method using augmented reality.
(1) The real-time visualization device recognizes a marker attached to the object
(2) The real-time visualization device recognizes the shape of the object
(3) The real-time visualization device receives specific location information According to claim 7,
Before the real-time visualization device receives data,
collecting data on a physical state of the object by a sensor disposed on the object; and
Further comprising obtaining the reduction response by inputting data received from the sensor into a physical model of the object stored in advance by a server,
In the real-time visualization device receiving data, the real-time visualization device receives the reduction response from the server. A computer-readable recording medium recording a program executing the real-time visualization method according to any one of claims 7 to 11.

Images