KR102413958B1 - Anomaly detection method based on multi-modal sensor, and computing apparatus for performing the same - Google Patents

Abstract

In a method of performing anomaly detection using a multi-modal sensor according to an embodiment, data collected using each of a plurality of sensors of different types is reduced through synchronization and convolution operation. and combining to generate a multi-modal representation, and applying the multi-modal representation as input data to an auto encoder, and the difference between the reconstructed data output from the auto encoder and the input data and performing anomaly detection based on , wherein the auto-encoder is pre-trained using data corresponding to a normal state.

Description

Anomaly detection method using multi-modal sensor and computing device for performing the same

Embodiments disclosed in this specification relate to a method for performing anomaly detection using a multi-modal sensor and a computing device for performing the same, for example, a mobile manipulating robot using an object It relates to a method of detecting an abnormal state such as holding and dropping a robot based on data collected through various types of sensors installed in a mobile operation robot.

Recently, the form of using a robot has been gradually expanded from being fixed at a certain position and performing simple repetitive tasks to performing necessary actions according to the situation while moving by itself.

In order for the mobile operation robot to perform an operation such as gripping an object without mistakes, it must be able to determine whether the operation is successful or not based on data collected through a sensor. For example, if a specific object appears and disappears on the arm of the robot in the image captured by the camera, the mobile manipulator robot may determine that the object was grasped and then dropped.

However, the types of objects that the mobile operation robot encounters in various service environments are very diverse, and the image data collected by the robot and the detection value of the weight sensor change large enough to determine the success of a simple operation such as gripping an object. It is also not easy to judge.

Therefore, there is a need to develop a technology capable of detecting abnormalities related to the operation of the mobile operation robot with high accuracy.

On the other hand, the above-mentioned background art is technical information that the inventor possessed for the purpose of derivation of the present invention or acquired during the derivation process of the present invention, and it cannot be said that it is necessarily known technology disclosed to the general public before the filing of the present invention. .

Embodiments disclosed herein are intended to provide a method for performing anomaly detection with high accuracy based on data collected through a multi-modal sensor, and a computing device for performing the same.

As a technical means for achieving the above-described technical problem, according to an embodiment, a method for performing anomaly detection using a multi-modal sensor is to synchronize and Generating a multimodal phenotype by reducing and combining through a convolution operation, and applying the multimodal phenotype as input data to an auto-encoder, based on a difference between the restored data output from the auto-encoder and the input data. and performing detection, wherein the auto-encoder may be pre-trained using data corresponding to a steady state.

According to another embodiment, as a computer program for performing an anomaly detection method using a multi-modal sensor, the anomaly detection method using a multi-modal sensor synchronizes data collected using each of a plurality of different types of sensors. and generating a multimodal phenotype by reducing and combining through a convolution operation, and applying the multimodal phenotype as input data to an auto-encoder, based on a difference between the restored data output from the auto-encoder and the input data and performing anomaly detection, wherein the auto-encoder may be pre-learned using data corresponding to a normal state.

According to another embodiment, as a computer-readable recording medium on which a program for performing an anomaly detection method using a multi-modal sensor is recorded, the anomaly detection method using a multi-modal sensor includes each of a plurality of different types of sensors. Generating a multimodal phenotype by reducing and combining the collected data using synchronization and convolution operations, and applying the multimodal phenotype as input data to an auto-encoder, and the restored data output from the auto-encoder and performing anomaly detection based on a difference between the input data, wherein the auto-encoder may be pre-learned using data corresponding to a normal state.

According to another embodiment, a computing device for performing an anomaly detection method using a multi-modal sensor receives data collected using each of a plurality of different types of sensors, and performs anomaly detection based on the input. A control unit comprising an input/output unit for outputting a result, a storage unit storing a program for performing abnormality detection, and at least one processor, and performing abnormality detection using data received through the input/output unit by executing the program The anomaly detection model implemented by the control unit executing the program is to reduce and combine the collected data through synchronization and convolution operations to generate a multimodal phenotype, and to transmit the multimodal phenotype to an auto-encoder It is applied as input data to the input data, and anomaly detection is performed based on a difference between the restored data output from the auto-encoder and the input data, and the auto-encoder may be pre-learned using data corresponding to a normal state.

According to any one of the above-mentioned problem solving means, the performance degradation due to the signal/semantic noise that may occur in a specific sensor is alleviated by fusing the data collected through the multi-modal sensor and applying it as an input to the pre-learned auto encoder. It has the effect of performing anomaly detection with high accuracy.

Effects obtainable in the disclosed embodiments are not limited to the above-mentioned effects, and other effects not mentioned are clear to those of ordinary skill in the art to which the embodiments disclosed from the description below belong. can be understood clearly.

1 is a diagram illustrating an artificial neural network model for performing an anomaly detection method using a multi-modal sensor according to an embodiment.
2 is a diagram illustrating a configuration of a computing device for performing an anomaly detection method using a multi-modal sensor according to an embodiment.
3 to 5 are flowcharts for explaining an anomaly detection method using a multi-modal sensor according to embodiments.

Hereinafter, various embodiments will be described in detail with reference to the accompanying drawings. The embodiments described below may be modified and implemented in various different forms. In order to more clearly describe the characteristics of the embodiments, detailed descriptions of matters widely known to those of ordinary skill in the art to which the following embodiments belong are omitted. In addition, in the drawings, parts irrelevant to the description of the embodiments are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a component is said to be “connected” with another component, it includes not only a case of 'directly connected' but also a case of 'connected with another component interposed therebetween'. In addition, when a component "includes" a component, it means that other components may be further included, rather than excluding other components, unless otherwise stated.

Before describing the embodiments in detail, meanings of terms frequently used herein will be described.

A 'multi-modal sensor' means including a plurality of sensors of different types, for example, an RGB camera, a depth camera, a LiDAR sensor, a force-torque torque) sensor and a microphone.

'Multi-modal data' refers to a set of data collected by a multi-modal sensor, that is, different types of sensors.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

1 is a diagram illustrating an artificial neural network model (hereinafter referred to as an 'anomaly detection model') for performing an anomaly detection method using a multimodal sensor according to an embodiment, and FIG. 2 is an embodiment It is a diagram showing the configuration of a computing device for performing an anomaly detection method using a multi-modal sensor according to . The anomaly detection model shown in FIG. 1 may be implemented by the controller 220 of the computing device 200 of FIG. 2 executing a program stored in the storage 230 . Hereinafter, components included in the computing device 200 will be briefly described, and then, a method of performing anomaly detection through the anomaly detection model shown in FIG. 1 will be described in detail.

Referring to FIG. 2 , the computing device 200 according to an embodiment may include an input/output unit 210 , a control unit 220 , and a storage unit 230 .

The input/output unit 210 is configured to receive a user command or data related to anomaly detection using a multi-modal sensor, and output a result of performing abnormality detection. The input/output unit 210 may include various types of input devices (e.g. keyboard, touch screen, etc.) for receiving input from the user, and also transmit/receive data used for detecting anomalies and data as a result of performing abnormality detection. It may include a connection port or a communication module for

The control unit 220 is configured to include at least one processor such as a CPU, and executes a program stored in the storage unit 230 to perform abnormality detection using a multi-modal sensor according to a process presented below. In other words, the controller 220 executes the program stored in the storage 230 to implement the anomaly detection model 100 shown in FIG. 1 , and the controller 220 detects anomalies through the anomaly detection model 100 . carry out A method in which the controller 220 performs anomaly detection using a multi-modal sensor using the anomaly detection model 100 will be described in detail below with reference to FIG. 1 .

The storage unit 230 is a configuration in which files and programs can be stored, and can be configured through various types of memories. In particular, the storage unit 230 may store data and programs that enable the control unit 220 to perform an operation for detecting anomalies according to a process presented below. In addition, data corresponding to a normal state for learning the auto encoder 130 included in the anomaly detection model 100 may be stored in the storage unit 230 .

Hereinafter, a process in which the control unit 220 executes a program stored in the storage unit 230 to perform an anomaly detection method using a multi-modal sensor according to an embodiment will be described in detail with reference to FIG. 1 .

In particular, in the following embodiments, whether the mobile manipulating robot successfully performs an operation of gripping an object (normal state) or misses an object (abnormal state) is collected using a multi-modal sensor installed in a mobile operation robot Assume a situation in which judgment is made based on one data. However, it is self-evident that the anomaly detection model 100 may be used to detect anomalies in various situations other than the situations assumed above.

As described above, the anomaly detection model 100 is implemented by the control unit 220 executing a program stored in the storage unit 230, so that the operation described to be performed by the anomaly detection model 100 in subsequent embodiments or The process can be seen as being actually performed by the control unit 220 . In addition, detailed components included in the anomaly detection model 100 can be viewed as a unit of software that plays a specific function or role in the overall program for performing an anomaly detection method using a multi-modal sensor.

Referring to FIG. 1 , an anomaly detection model 100 according to an embodiment may include a synchronization module 110 , a fusion module 120 , an auto encoder 130 , and an anomaly discriminator 140 .

It is assumed that the multi-modal sensor installed in the mobile manipulator includes an RGB camera, a depth camera, a lidar sensor, a force-torque sensor, and a microphone. The multi-modal data 10 collected through the multi-modal sensor includes an RGB image, a depth image, a lidar measurement value, a force-torque measurement value, and a sound. The multi-modal data 10 collected while the mobile operation robot performs an operation of gripping an object is applied as an input to the anomaly detection model 100 .

The heterogeneous data included in the multimodal data 10 is used as an input to the auto-encoder 130 because the number of times (that is, the data collection period), size, and dimension of data collected during a unit time are different. In order to do this, a process of synchronizing and fusing data is required, and the synchronization module 110 and the fusion module 120 perform this.

The synchronization module 110 synchronizes and normalizes heterogeneous data included in the multi-modal data 10 . The synchronization module 110 samples heterogeneous data included in the multi-modal data 10 based on a specific period (synchronization), and normalizes the sampled data to values of a certain section.

To describe the process of synchronizing heterogeneous data in more detail, the synchronization module 110 records each point in time at which data is collected from the point in time when the sensors start to collect data in a timestamp, and time based on a specific period Set the index to the stamp. Then, the synchronization module 110 may sample each of the heterogeneous data by selecting only data collected at a time point closest to the index.

To illustrate with specific examples, assume that RGB images are collected at 40 Hz (i.e., data is collected 40 times per second), lidar measurements are collected at 20 Hz, and force-torque measurements are collected at 10 Hz. do. At this time, if data is sampled based on 10 Hz, the synchronization module 110 sets the index at intervals of 0.1 seconds such as 0.1 second, 0.2 second, 0.3 second, etc. on the timestamp, and only the data closest to each index for each heterogeneous data choose RGB images are collected at 0.025 sec, 0.05 sec, 0.075 sec, 0.1 sec, and 0.125 sec, respectively, and the synchronization module 110 selects only the RGB image collected at 0.1 sec among them and discards the remaining data. In addition, lidar measurement values are collected at 0.05 seconds, 0.1 seconds, and 0.15 seconds, respectively, and the synchronization module 110 selects only the lidar measurement values collected at 0.1 seconds among them and discards the remaining data. In addition, the force-torque measurement values are collected at 0.1 sec. and 0.2 sec. respectively, and since all the times at which data are collected coincide with the index, the synchronization module 110 selects all data.

The synchronization module 110 normalizes heterogeneous data synchronized through the above process. For example, in the case of an RGB image, the synchronization module 110 converts a value between 0 and 255 for each channel into a value between 0 and 1, and for each of the remaining data, the minimum and maximum values match 0 and 1, respectively. Convert it to a value between 0 and 1.

The fusion module 120 performs a process of fusion of the synchronized and normalized multimodal data 10 . Since the heterogeneous data output from the synchronization module 110 have different sizes and dimensions, the fusion module 120 reduces and unifies the dimensions of the heterogeneous data through a convolution operation, and then combines them to achieve multimodal Create a multi-modal representation (20).

The generated multimodal phenotype 20 is applied as input data to the auto-encoder 130 . The anomaly detection model 100 performs anomaly detection based on a difference between an input and an output to the auto-encoder 130. For this purpose, the auto-encoder 130 may be previously trained using data corresponding to a normal state. .

In more detail, the auto-encoder 130 is learned using data collected through the multi-modal sensor when it is in a normal state (a state in which the mobile manipulating robot stably grips an object), and thus the auto-encoder 130 is in an abnormal state (a mobile manipulating robot). If the data collected through the multi-modal sensor is input to the auto encoder 130 when the object is not grasped), the difference between the input and the output becomes larger than in the normal state. Using these characteristics, the anomaly detection model 100 calculates an error score using the difference between the input and output of the auto encoder 130, and the anomaly discriminator 140 determines that the error score is higher than a preset reference value. If it is large, it can be judged as an abnormal state, and, conversely, when the error score is less than the reference value, it can be judged as a normal state. The method of calculating the error score and setting the reference value will be described in detail below.

The error score may be calculated by various methods so as to have a value reflecting the difference between the input and output of the auto encoder 130 . A method of calculating an error score according to an exemplary embodiment will be described as follows.

Data input to the auto-encoder 130 is referred to as 'input data', and data restored and output by the auto-encoder 130 accordingly is referred to as 'restored data'. Anomaly detection model 100 is a result output from the hidden layer of the encoder included in the auto-encoder 130 when the input data is applied to the auto-encoder 130, and the auto-encoder when the restored data is applied to the auto-encoder 130 An error score may be calculated using the difference between the results output from the hidden layer of the encoder included in 130 .

라고 하고, 오토 인코더(130)에 포함된 인코더

의

번째 은닉층에서 출력되는 결과를

라고 한다. 또한, 입력 데이터

가 오토 인코더(130)를 통과하여 출력된 복원 데이터를

라고 하고, 복원 데이터

가 오토 인코더(130)에 다시 입력으로 인가되었을 때 인코더

의

번째 은닉층에서 출력되는 결과를

라고 한다. 다음의 수학식 1 및 2와 같이 인코더의 은닉층에서 출력되는 결과를 나타낼 수 있다.A detailed example is given to explain in more detail as follows. input data to the auto encoder 130

and the encoder included in the auto-encoder 130

of

The result output from the second hidden layer

It is said Also, input data

The restored data output through the auto encoder 130

and restore data

When is applied as an input to the auto-encoder 130 again, the encoder

of

The result output from the second hidden layer

It is said A result output from the hidden layer of the encoder may be represented as shown in Equations 1 and 2 below.

와

간 차이를 이용하여 아래의 수학식 3에 따라서 오차점수를 산출할 수 있다.The anomaly detection model 100 is

Wow

An error score can be calculated according to Equation 3 below using the difference between the two.

That is, the error score can be regarded as a concept in which the difference between the results output from the encoder hidden layer when the input data and the restored data are respectively applied as inputs to the auto encoder 130 is summed for all hidden layers.

Meanwhile, by controlling the operation of the mobile operation robot, data corresponding to a normal state required for learning of the auto encoder 130 may be collected. For example, the mobile manipulating robot holds the object, moves it by a predetermined distance, and then repeatedly performs an experimental operation of dropping the object, and collects data using a multi-modal sensor while the mobile manipulating robot performs the experimental operation. do. Among the multi-modal data, data collected during a state that satisfies a certain condition (e.g. from the time when the mobile robot successfully grips the object to the moment before dropping the object) can be classified as data corresponding to the normal state. .

The abnormality detection model 100 may determine a reference value as a criterion for determining whether an abnormal state exists based on an error score calculated when data corresponding to a normal state is applied to the auto encoder 130 . For example, the anomaly detection model 100 may set the reference value as the maximum value of the error points for data corresponding to the normal state, or use the maximum value among the remaining error points excluding some outliers as the reference value. can also be set to In addition, the anomaly detection model 100 may determine a reference value by using the error score in various ways to effectively detect an abnormal state.

Hereinafter, a method of performing anomaly detection using a multi-modal sensor using the computing device 200 as described above will be described. 3 to 5 are flowcharts for explaining an anomaly detection method using a multi-modal sensor according to an embodiment.

An anomaly detection method using a multi-modal sensor according to the embodiments shown in FIGS. 3 to 5 includes steps that are time-series processed by the computing device 200 shown in FIG. 2 . Therefore, even if omitted below, the contents described above with respect to the computing device 200 of FIG. 2 may also be applied to the anomaly detection method using the multi-modal sensor according to the embodiments shown in FIGS. 3 to 5 . .

Referring to FIG. 3 , in step 301, the anomaly detection model 100 reduces and combines data collected using each of a plurality of different types of sensors through synchronization and convolution operations to generate a multimodal phenotype. .

4 illustrates detailed steps included in step 301 of FIG. 3 . Referring to FIG. 4 , in step 401 , the anomaly detection model 100 samples collected data based on a preset period, and normalizes the sampled data to a value of a preset section.

In step 402, the anomaly detection model 100 reduces and unifies the dimensions of normalized data through a convolution operation, and then combines them.

3, in step 302, the anomaly detection model 100 applies the multimodal phenotype as input data to the auto-encoder 130, and based on the difference between the restored data output from the auto-encoder 130 and the input data to perform anomaly detection.

5 illustrates detailed steps included in step 302 of FIG. 3 . Referring to FIG. 5 , in step 501 , the anomaly detection module 100 outputs the result output from the hidden layer of the encoder included in the auto encoder 130 when the input data is applied to the auto encoder 130 , and the restored data is the auto encoder When applied to 130 , an error score is calculated using the difference between the results output from the hidden layer of the encoder included in the auto encoder 130 .

In step 502, the anomaly discriminator 140 of the anomaly detection module 100 determines whether the calculated error score is less than or equal to a preset reference value, and if it is less than the reference value, proceeds to step 504 to determine a normal state, and if it exceeds the reference value It proceeds to step 503 and it is determined as an abnormal state.

According to the above-described embodiments, by fusing the data collected through the multi-modal sensor and applying it as an input to the pre-learned auto encoder, performance degradation due to signal/semantic noise that may occur in a specific sensor is alleviated and high accuracy This has the effect of performing anomaly detection.

The term '~ unit' used in the above embodiments means software or hardware components such as field programmable gate array (FPGA) or ASIC, and '~ unit' performs certain roles. However, '-part' is not limited to software or hardware. '~' may be configured to reside on an addressable storage medium or may be configured to refresh one or more processors. Accordingly, as an example, '~' indicates components such as software components, object-oriented software components, class components, and task components, and processes, functions, properties, and procedures. , subroutines, segments of program patent code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables.

The functions provided in the components and '~ units' may be combined into a smaller number of elements and '~ units' or separated from additional components and '~ units'.

In addition, components and '~ units' may be implemented to play one or more CPUs in a device or secure multimedia card.

The anomaly detection method using the multi-modal sensor according to the embodiments described with reference to FIGS. 3 to 5 may also be implemented in the form of a computer-readable medium for storing instructions and data executable by a computer. In this case, the instructions and data may be stored in the form of program codes, and when executed by the processor, a predetermined program module may be generated to perform a predetermined operation. In addition, computer-readable media can be any available media that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. In addition, the computer-readable medium may be a computer recording medium, which is a volatile and non-volatile and non-volatile storage medium implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. It may include both volatile, removable and non-removable media. For example, the computer recording medium may be a magnetic storage medium such as HDD and SSD, an optical recording medium such as CD, DVD, and Blu-ray disc, or a memory included in a server accessible through a network.

Also, the abnormality detection method using the multi-modal sensor according to the embodiments described with reference to FIGS. 3 to 5 may be implemented as a computer program (or computer program product) including instructions executable by a computer. The computer program includes programmable machine instructions processed by a processor, and may be implemented in a high-level programming language, an object-oriented programming language, an assembly language, or a machine language. . In addition, the computer program may be recorded in a tangible computer-readable recording medium (eg, a memory, a hard disk, a magnetic/optical medium, or a solid-state drive (SSD), etc.).

Therefore, the abnormality detection method using the multi-modal sensor according to the embodiments described with reference to FIGS. 3 to 5 may be implemented by executing the computer program as described above by the computing device. The computing device may include at least a portion of a processor, a memory, a storage device, a high-speed interface connected to the memory and the high-speed expansion port, and a low-speed interface connected to the low-speed bus and the storage device. Each of these components is connected to each other using various buses, and may be mounted on a common motherboard or in any other suitable manner.

Here, the processor may process a command within the computing device, such as, for example, to display graphic information for providing a graphic user interface (GUI) on an external input or output device, such as a display connected to a high-speed interface. Examples are instructions stored in memory or a storage device. In other embodiments, multiple processors and/or multiple buses may be used with multiple memories and types of memory as appropriate. In addition, the processor may be implemented as a chipset formed by chips including a plurality of independent analog and/or digital processors.

Memory also stores information within the computing device. As an example, the memory may be configured as a volatile memory unit or a set thereof. As another example, the memory may be configured as a non-volatile memory unit or a set thereof. The memory may also be another form of computer readable medium, such as, for example, a magnetic or optical disk.

In addition, the storage device may provide a large-capacity storage space to the computing device. A storage device may be a computer-readable medium or a component comprising such a medium, and may include, for example, devices or other components within a storage area network (SAN), a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory, or other semiconductor memory device or device array similar thereto.

The above-described embodiments are for illustration, and those of ordinary skill in the art to which the above-described embodiments pertain can easily transform into other specific forms without changing the technical idea or essential features of the above-described embodiments. you will understand Therefore, it should be understood that the above-described embodiments are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope to be protected through this specification is indicated by the claims described below rather than the detailed description, and it should be construed to include all changes or modifications derived from the meaning and scope of the claims and their equivalents. .

100: anomaly detection model 110: synchronization module
120: fusion module 130: auto-encoder
140: anomaly discriminator 200: computing device
210: input/output unit 220: control unit
230: storage

Claims (12)

In the method of performing anomaly detection using a multi-modal sensor,
generating a multi-modal representation by reducing and combining data collected using each of a plurality of different types of sensors through synchronization and convolution operations; and
Applying the multimodal phenotype as input data to an auto encoder, and performing anomaly detection based on a difference between the restored data output from the auto encoder and the input data,
The method, characterized in that the auto-encoder is pre-learned using data corresponding to a steady state. According to claim 1,
The step of performing the anomaly detection is,
A result output from the hidden layer of the encoder included in the auto encoder when the input data is applied to the auto encoder, and a result output from the hidden layer of the encoder included in the auto encoder when the restored data is applied to the auto encoder calculating an error score (anomaly score) by using the difference; and
and determining that the calculated error score is greater than a preset reference value as an abnormal state. 3. The method of claim 2,
The reference value is determined based on an error score calculated when data corresponding to the steady state is input to the auto encoder. According to claim 1,
The step of generating the multimodal phenotype comprises:
sampling the collected data based on a preset period, and normalizing the sampled data to a value of a preset section; and
Method comprising reducing and unifying the dimensions of the normalized data through a convolution operation, and then combining them with each other. According to claim 1,
The plurality of sensors include an RGB camera, a depth camera, a LiDAR sensor and a force-torque sensor and a microphone installed in the mobile operation robot,
The generating of the multi-modal phenotype includes: RGB images, depth images, lidar measurements and force-torque measurements collected through the plurality of sensors while the mobile manipulating robot performs an operation of gripping an object; After synchronizing and normalizing the sound, the multimodal phenotype is generated by unifying and combining the dimensions,
The data corresponding to the normal state is data collected using the plurality of sensors in a state in which the mobile operation robot normally grips the object. According to claim 1,
A computer-readable recording medium in which a program for executing the method according to claim 1 is recorded on a computer. A computer program executed by a computing device and stored in a recording medium for performing the method according to claim 1 . In the computing device for performing anomaly detection using a multi-modal sensor,
an input/output unit for receiving data collected using each of a plurality of different types of sensors, and outputting a result of performing anomaly detection based thereon;
a storage unit storing a program for performing anomaly detection; and
A control unit including at least one processor and performing abnormality detection using data received through the input/output unit by executing the program,
The anomaly detection model implemented by the controller executing the program reduces and combines the collected data through synchronization and convolution operations to generate a multimodal phenotype, and uses the multimodal phenotype as input data to the auto-encoder and performs anomaly detection based on the difference between the restored data output from the auto-encoder and the input data,
The auto encoder is a computing device, characterized in that pre-trained using data corresponding to a steady state. 9. The method of claim 8,
The anomaly detection model performs the anomaly detection,
A result output from the hidden layer of the encoder included in the auto encoder when the input data is applied to the auto encoder, and a result output from the hidden layer of the encoder included in the auto encoder when the restored data is applied to the auto encoder Computing device, characterized in that calculating an error function using the difference between the two, and determining an abnormal state if the calculated error score is greater than a preset reference value. 10. The method of claim 9,
The anomaly detection model determines the reference value based on an error score calculated when data corresponding to the normal state is input to the auto encoder. 9. The method of claim 8,
The anomaly detection model in generating the multimodal phenotype,
Sample the collected data based on a preset period, normalize the sampled data to a value of a preset section, reduce and unify the dimensions of the normalized data through a convolution operation, and then combine them computing device with 9. The method of claim 8,
The plurality of sensors include an RGB camera, a depth camera, a LiDAR sensor and a force-torque sensor and a microphone installed in the mobile operation robot,
In generating the multi-modal phenotype, the anomaly detection model includes an RGB image, a depth image, a LiDAR measurement value, and a force- To generate the multimodal phenotype by synchronizing and normalizing torque measurements and sound, then unifying and combining dimensions,
The data corresponding to the normal state is data collected by using the plurality of sensors in a state in which the mobile operation robot normally grips an object.

Images