KR102112699B1 - Method for visualizing time series signal and apparatus using the same - Google Patents

Abstract

The present disclosure relates to a method for visualizing a time series signal and a device using the same. According to one embodiment of the present invention, the method for visualizing a time series signal may comprise the steps of: generating a first map for applying attention to the time series signal through reverse propagation for a reading result of a reading model; generating visualization information in which a reference area is visualized by applying the attention for the time series signal based on the first map; and outputting the visualization information.

Description

METHOD FOR VISUALIZING TIME SERIES SIGNAL AND APPARATUS USING THE SAME}

The present disclosure relates to a method for visualizing a time-series signal and an apparatus using the same.

Recently, as a method of solving the problem of classifying an input pattern that is frequently encountered in the engineering field into a specific group, studies have been actively conducted to apply an efficient pattern recognition method possessed by humans to a real computer. Among the various computer application studies, there is a study on the artificial neural network, which is an engineering model of the human brain cell structure in which efficient pattern recognition is performed.

The artificial neural network may also be used in a process of generating health information of a subject through reading a biosignal. For example, an electrocardiography, brain wave, electromyogram, etc. may be measured, and the pattern of the measured biosignal may be classified through an artificial neural network to determine the disease of the subject.

With regard to the reading of a biosignal using an artificial neural network, there is a need to increase the reliability of reading through an artificial neural network through a means capable of visualizing the basis of judgment.

Republic of Korea Registered Patent Publication No. 10-1626748

The invention described in the present disclosure, that is, the present invention is a means for visualizing an area of a biosignal on which an artificial neural network generating health information based on a biosignal collected in the form of a time series signal is a basis for generating a read result. It aims to provide.

Specifically, according to the present invention, the biosignal that is the basis of the judgment of the artificial neural network corresponds to the biosignal obtained through which measurement device (for example, a lead), and at which point in the biosignal It is an object to provide a means for outputting visualization information visualizing whether a biosignal is considered.

The characteristic configuration of the present invention for achieving the objects of the present invention as described above and realizing the characteristic effects of the present invention described below is as follows.

According to an aspect of the present invention, a time-series signal visualization method for visualizing a reference region referenced in the time-series signal in the process of reading a time-series signal through a reading model is disclosed. According to the method, Generating a first map for applying attention to the time-series signal through back propagation of the reading result of the reading model; Generating visualization information in which the reference area is visualized by applying an attention to the time-series signal based on the first map; And outputting the visualization information.

According to an aspect of the present invention, generating the visualization information may include determining identification information corresponding to the reference area based on a result of applying the attention; And generating the visualization information based on the determined identification information.

According to an aspect of the present invention, the step of generating the first map is based on the time-series signal converted into a signal in the frequency domain, and a first grade for the output of the classifier included in the readout model. Calculating a gradient; Generating a second map based on the first gradient; Calculating a second gradient for the output of the classifier based on a feature map output through a convolutional neural network included in the reading model; Generating a third map based on the second gradient; And generating the first map based on the second map and the third map.

According to an aspect of the present invention, in a process of reading a time-series signal through a reading model, a computing device that visualizes a reference area among the time-series signals includes: a communication unit that receives a time-series signal; And a processor for visualizing the reference area, wherein the processor generates a first map for applying attention to the time-series signal through back propagation of the reading result of the reading model, and based on the map, By applying the attention to the time-series signal, visualization information in which the reference region is visualized may be generated, and the visualization information may be output through a display.

According to the present invention, reliability of a readout model may be improved through visualization information visualizing a base region of a biosignal that is a basis for a determination of a readout model for reading a biosignal using an artificial neural network. That is, based on the visualization information, it is possible to visually confirm the basis of judgment of the reading model, and furthermore, it can be effectively used in a process of improving the accuracy of the judgment of the reading model. In addition, a reader (for example, a doctor) using a readout model can obtain details related to the readout result through visualization information, and through this, further diagnosis may be performed.

In addition, the invention of the present application can be used as an effective educational tool for a practitioner who trains a judgment process proceeding through a biosignal.

In addition, the present invention can provide detailed information about the determination result by performing visualization of the reference region in units of one cycle (for example, 10 seconds) of a biosignal.

The following drawings attached for use in the description of the embodiments of the present invention are only some of the embodiments of the present invention, and for those skilled in the art to which the present invention pertains, based on these drawings without efforts to reach a separate invention. Other drawings can be obtained.
1 is a view for schematically explaining a time-series signal visualization method according to the present disclosure.
2 is a conceptual diagram schematically illustrating an exemplary configuration of a computing device performing a time-series signal visualization method according to the present disclosure.
3 is an exemplary block diagram illustrating hardware or software components of a computing device performing a time-series signal visualization method according to the present disclosure.
4 is a flowchart illustrating a method for visualizing a time series signal according to the present disclosure.
FIG. 5 is a diagram for explaining a method of generating a map used for attention applied to a time-series signal in a time-series signal visualization process according to the present disclosure.
6 is a diagram illustrating an example in which a method for visualizing a lesion according to the present disclosure is applied.

For a detailed description of the present invention to be described later, reference is made to the accompanying drawings that illustrate, by way of illustration, specific embodiments in which the invention may be practiced in order to clarify the objects, technical solutions and advantages of the invention. These embodiments are described in detail enough to enable a person skilled in the art to practice the present invention.

Throughout the detailed description and claims of this disclosure, 'learning' or 'learning' is a term referring to performing machine learning through procedural computing, which refers to a mental action such as human educational activity. Those skilled in the art will understand that it is not intended to be referred to.

And throughout the detailed description and claims of this disclosure, the word 'include' and variations thereof are not intended to exclude other technical features, additions, components or steps. Also, 'one' or 'one' is used in more than one meaning, and 'another' is limited to at least a second or more.

Other objects, advantages, and characteristics of the invention will be apparent to those skilled in the art, some of which will appear from this specification and some from the practice of the invention. The examples and drawings below are provided as examples and are not intended to limit the invention. Accordingly, the details disclosed in this disclosure with respect to a particular structure or function should not be construed in a limiting sense, but merely provide guidance for a person skilled in the art to implement the invention in a variety of practically suitable details. It should be interpreted as representative basic data.

Moreover, the present invention encompasses all possible combinations of the embodiments shown in this disclosure. It should be understood that the various embodiments of the present invention are different, but need not be mutually exclusive. For example, the specific shapes, structures, and properties described herein may be implemented in other embodiments without departing from the spirit and scope of the invention in relation to one embodiment. In addition, it should be understood that the location or arrangement of individual components within each disclosed embodiment can be changed without departing from the spirit and scope of the invention. Therefore, the following detailed description is not intended to be taken in a limiting sense, and the scope of the present invention, if appropriately described, is limited only by the appended claims, along with all ranges equivalent to those claimed. In the drawings, similar reference numerals refer to the same or similar functions throughout several aspects.

Unless otherwise indicated in the present disclosure or clearly contradicted the context, an item referred to as a singular encompasses plural, unless otherwise required in that context. In addition, in describing the present invention, when it is determined that detailed descriptions of related well-known structures or functions may obscure the subject matter of the present invention, detailed descriptions thereof will be omitted.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings in order to enable a person skilled in the art to easily implement the present invention.

1 is a view for schematically explaining a time-series signal visualization method according to the present disclosure.

Recently, research into a method for obtaining health information of a subject by actively inputting a biosignal obtained in the form of a time series signal such as an electrocardiogram signal (ECG) into a readout model including a pre-trained artificial neural network, have. The readout model can generate various health information based on the pattern of the biosignal. For example, the readout model can provide information about the cardiovascular health status of the subject based on the pattern of the electrocardiogram signal. In order to improve the reliability of the health information provided by the readout model, it is necessary to visually provide a reference region of the biosignal that is the basis of the readout of the readout model.

The present invention can provide a method of visualizing a reference region of a biosignal, which is the basis of the readout, in the process of generating health information of a subject based on the biosignal in which the readout model is collected as a time-series signal. Through this, the present invention can increase the reliability of a readout model that provides health information using an artificial neural network, and furthermore, the present invention can be used as an effective educational tool in the training process of training to perform diagnosis based on a biosignal. Can be.

Referring to FIG. 1, the computing device according to the present disclosure includes time series signals 110-1, ..., 110-N and time series signals 110-1, ..., 110-N acquired from a subject. Based on the maps (120-1, ..., 120-N) applying the attention to), the reading model reads the time series signals (110-1, ..., 110-N), and the health information of the subject In the process of generating, visualization information (130-1, ..., 130-N) in which the reference region of the time-series signal referenced is visualized may be generated. The term "attention" as used herein may mean a process of determining the degree to which each region included in an input affects an output value, and the meaning of applying attention to a time-series signal through a map means a map. It may mean determining the degree of influence of each region of the time series signal on the output (health information) based on the value of each element included. The visualization information 130-1, ..., 130-N may refer to information visualized through identification information in which the extent to which each region included in the input affects the output determined through the attention.

The time series signals 110-1, ..., 110-N may include a plurality of time series signals 110-1, ..., 110-N acquired through a plurality of measurement leads. The maps 120-1, ..., 120-N may be generated corresponding to the respective time-series signals 110-1, ..., 110-N.

The visualization information (130-1, ..., 130-N) is a time series signal (110-1, ..., 110-N) of the reading model reference area and the non-referenced area through different identification information You can visualize. For example, in the time information 130-1, the area 130-1a represents the waveform of the time-series signal 110-1, and the area 130-1b indicates whether the time-series signal was referenced through a change in brightness. , It can indicate the degree of reference. For example, an area expressed in high brightness may correspond to an area having a high degree of reference by the reading model in the reading process, and an area expressed in dark form may correspond to an area having a low reference level in the reading process. The method of visualizing the reference area through the identification information is not limited to a method using the proposed brightness change, and may include a method of visualizing information read by the reading model through arbitrary identification information such as color change and corresponding numeric information. have.

Accordingly, the visualization information 130-1 corresponding to the time-series signal obtained through the first lead is generally expressed as brighter than the visualization information 130-2 corresponding to the time-series signal obtained through the second lead. As can be seen, it can be determined that the reading model refers more heavily to the reading process.

In addition, the visualization information 130-1 also shows that the time-series signals for each time correspond to different brightnesses, so it is determined that the degree to which the reference model refers to the time-series signals 110-1 for each time zone is different. Can be.

Based on the form of the visualization information (130-1, ..., 130-N), it can be effectively identified whether the readout model generated the health information of the subject by referring to the signal extracted from which lead and at what time. Can be.

Hereinafter, a method of generating visualization information 130-1, ..., 130-N is described in more detail through the accompanying drawings.

2 is a conceptual diagram schematically illustrating an exemplary configuration of a computing device performing a time-series signal visualization method according to the present disclosure.

The computing device 200 according to an embodiment of the present invention includes a communication unit 210 and a processor 220, and may directly or indirectly communicate with an external computing device (not shown) through the communication unit 210. .

Specifically, the computing device 200 includes typical computer hardware (eg, a computer processor, memory, storage, input device and output device, a device that may include components of other existing computing devices; routers, switches, and the like) Electronic communication devices; electronic information storage systems such as network-attached storage (NAS) and storage area network (SAN)) and computer software (i.e., enabling computing devices to function in a particular way) Command) to achieve desired system performance.

The communication unit 210 of the computing device can transmit and receive requests and responses with other computing devices to be interlocked. As an example, such requests and responses may be made by the same transmission control protocol (TCP) session. However, the present invention is not limited thereto. For example, it may be transmitted and received as a user datagram protocol (UDP) datagram. In addition, in a broad sense, the communication unit 210 may include a pointing device such as a keyboard and a mouse for receiving commands or instructions, other external input devices, printers, displays, and other external output devices.

In addition, the processor 220 of the computing device may include a micro processing unit (MPU), a central processing unit (CPU), a graphics processing unit (GPU) or a tensor processing unit (TPU), a cache memory, and a data bus. ). Also, it may further include a software configuration of an operating system and an application performing a specific purpose.

3 is an exemplary block diagram illustrating hardware or software components of a computing device performing a time-series signal visualization method according to the present disclosure.

Referring briefly to the configuration of the method and apparatus according to the present disclosure with reference to FIG. 3, the computing device 200 may include a time series signal acquisition module 310 as a component thereof. Since the time-series signal acquisition module 310 is configured to acquire a time-series signal to which the method according to the present disclosure is applied, the individual modules illustrated in FIG. 3 include, for example, a communication unit included in the computing device 200 ( 210) or the processor 220, or the communication unit 210 and the processor 220 may be implemented by interworking will be understood by those skilled in the art. The time-series signal may refer to a biosignal obtained in chronological order, for example, an electrocardiogram (ECG), an electroencephalograph (EGG), and a light volume pulse wave (ECG) obtainable through a plurality of measurement leads. photoplethysmography (PPG), and the like, but may include any biosignal obtained in a time sequence not limited thereto.

The time-series signal obtained through the time-series signal acquisition module 310 may be applied using the map generated through the modules described below, and the reading model may be referred to in the reading process through the time-series signal to which the attention is applied. Areas can be visualized.

The map generated through the map generation module 330 may be generated based on back propagation to an output (eg, health information of a subject) generated through the time-series signal reading module 320. The time-series signal reading module 320 may include a reading model that generates subject's health information based on the time-series signal input. The reading model may include a pre-trained artificial neural network to generate health information about a subject's health condition based on a time-series signal that is a biosignal. For example, the health information may be information about a user's cardiovascular health status using an electrocardiogram. The time-series signal reading module 320 converts the input time-series signal into a frequency domain signal, and classifies the feature map obtained by inputting the signal converted into the frequency domain into a pre-trained convolutional neural network through a classifier, The subject's health information can be generated.

The visualization information generation module 340 applies the attention to the time-series signal obtained through the time-series signal acquisition module 310 through the map generated through the map generation module 330, and reads it based on the result of the attention application The reference area referenced by the model may generate visualization information visualized. Visualization information may be displayed through the output module 350.

More specifically, the output module 350 may provide visualization information in which a reference area is visualized to an external entity through an output device.

Here, the external entity includes a user, an administrator of the computing device 200, a medical expert in charge of the subject, and the like, and if the subject needs information related to a time-series signal visualized It should be understood that any subject is included.

Although the components shown in FIG. 3 are illustrated as being realized in one computing device for convenience of description, it will be understood that the computing device 200 performing the method of the present invention may be configured such that a plurality of devices are interlocked with each other.

Now, a method for visualizing a time series signal according to the present disclosure will be described in more detail with reference to FIGS. 4 to 6.

4 is a flowchart illustrating a method for visualizing a time series signal according to the present disclosure.

Referring to FIG. 4, in step S100 of a method for visualizing a time-series signal according to the present disclosure, the map generation module 330 is configured to apply an attention to a time-series signal through back propagation of a reading result of the reading model. 1 You can create a map.

Specifically, the map generation module 330 based on the gradient of the output (reading result) of the classifier included in the readout model, information about the channel of the reference region referenced by the readout model (through some lead) A second map related to the acquired time-series signal) and a third map related to time information for the reference area (where an area of the time-series signal is referenced) may be generated. The map generation module 330 may generate a first map for applying attention based on the second map and the third map. A specific method for generating the first map, the second map, and the third map is described in more detail with reference to FIG. 5 to be attached below.

In step S200, the visualization information generation module 340 may generate visualization information visualizing the reference area referenced in the reading process by applying the attention to the time-series signal based on the generated first map. The first map includes element values that change depending on the channel (whether it is a time-series signal acquired through a node) and time (whether it corresponds to a time point of the time-series signal), and each element value corresponds to The region may indicate the degree referenced by the reference model. The meaning of applying the attention based on the first map may mean determining the degree to which each region of the time-series signal is referenced based on the element values included in the first map.

The visualization information generation module 340 may determine the degree to which each region of the time-series signal is referenced by the reference model based on the result of applying the attention based on the first map, and visualize the visualization information by visualizing the referenced degree. Can be created.

The visualization information generation module 340 may determine identification information corresponding to the reference area based on the result of applying the attention, and generate visualization information based on the determined identification information. For example, based on the result of the attention, the visualization information generation module 340 assigns a high level of brightness value to the first area when it is determined that the reference area of the first area of the time series signal is high, and The second region determined to have a low degree may generate visualization information by applying a low level brightness value to the second region. In the presented example, it was suggested to use the brightness value as the identification information, but the present invention is not limited to this, and it is referred to through arbitrary identification information, such as giving different colors to the reference area or writing different values according to the reference level. It may include a way to identify the region.

In step S300, the output module 350 may output visualization information generated through the visualization information generation module 340 through an output device. An example of output visualization information is presented through FIG. 6.

5 is a diagram for exemplarily explaining a method of generating a map used for attention applied to a time-series signal in a time-series signal visualization process according to the present disclosure.

As described above, the computing device of the present invention may generate visualization information in which an area referenced by a reading model is visualized by applying attention to a time-series signal through a map.

Referring to FIG. 5, the reading model 520 may output the health information 530 of the reader based on the time-series signal 510.

(522)로 변환할 수 있다. 주파수 영역의 신호

(522)는 예시적으로 도시된 신호(523)와 같이, 각각의 리드 별로(채널 축), 시간 축 및 주파수 축에 따른 신호의 세기가 서로 다른 밝기 값을 통해 시각적으로 표현된 형태일 수 있다. 주파수 영역의 신호

(522)는 밝게 표현될수록 더 큰 세기 값에 대응될 수 있으며, 어둡게 표현될수록 0에 가까운 세기 값에 대응될 수 있다.The reading model 520 transmits the time-series signal 510 through the frequency conversion module 521 in the frequency domain signal.

(522). Signal in the frequency domain

522 may be in the form of a signal 523 for each lead (channel axis), the intensity of the signal along the time axis and the frequency axis is visually expressed through different brightness values, such as the illustrated signal 523. . Signal in the frequency domain

522 may correspond to a larger intensity value as it is brighter, and may correspond to an intensity value closer to 0 as it is darker.

(522)에 기초하여 특징맵(525)을 생성할 수 있다. 일 실시예에 따르면, 미리 학습된 콘볼루션 뉴럴 네트워크(524)는 풀링(pooling) 방식을 이용하지 않고, 스트라이드(stride) 방식을 이용하는 콘볼루션 필터일 수 있다. 구체적으로, 미리 학습된 콘볼루션 뉴럴 네트워크(524)는 보틀넥(bottle neck)에 스트라이드 콘볼루션 필터를 포함할 수 있다.The reading model 520 is a signal in the frequency domain through a pre-trained convolutional neural network 524.

A feature map 525 may be generated based on 522. According to an embodiment, the pre-trained convolutional neural network 524 may be a convolution filter using a stride method instead of a pooling method. Specifically, the pre-learned convolutional neural network 524 may include a stride convolution filter in a bottle neck.

The reading model 520 may output the health information 530 of the subject based on the feature map 525 through the classifier 526. The classifier 526 may be an artificial neural network previously learned to output health information 530 based on the pattern of the feature map 525.

According to an embodiment, the computing device 200 may generate a first map for applying attention to a time-series signal through back propagation of the health information 530, and time-series through the generated first map By applying the attention to the signal, it is possible to generate visualization information in which the region referenced by the readout model 520 is visualized.

Hereinafter, a process in which the computing device 200 generates a first map for performing attention will be described in more detail.

(541)을 생성할 수 있다.The computing device 200 calculates the first gradient 540 for the health information 530, and based on the first gradient 540, the reading model 520 refers to the time series signal 510. A second map associated with the channel (i.e., whether it refers to a biosignal measured through the measurement lead)

(541).

Specifically, the computing device 200 may generate the first gradient 540 based on Equation (1).

는 판독 모델을 통해 출력된 건강 정보(530),

는 주파수 영역의 신호(522)를 의미할 수 있다.

The health information 530 output through the reading model,

Can mean the signal 522 in the frequency domain.

(522)와 동일한 사이즈에 대응될 수 있다. 구체적으로, 주파수 영역의 신호

(522)는 ((시간 축 사이즈)*(주파수 축 사이즈)*(채널 축 사이즈(리드의 개수에 대응))) = (x_w*x_h*x_d)에 대응되는 사이즈를 가지는데, 주파수 영역의 신호

(522)에 기초하여 생성되는 제1 그레이디언트(540) 역시 주파수 영역의 신호

(522)와 동일한 사이즈를 가질 수 있다.The first gradient 540 is a signal in the frequency domain

It may correspond to the same size as 522. Specifically, signals in the frequency domain

(522) has a size corresponding to ((time axis size) * (frequency axis size) * (channel axis size (corresponding to the number of leads))) = (x_w * x_h * x_d), the signal in the frequency domain

The first gradient 540 generated based on 522 is also a signal in the frequency domain

It may have the same size as 522.

(541)을 생성할 수 있다. 구체적으로, 컴퓨팅 장치(200)는 수학식 2에 기초하여 제2 맵

(541)을 생성할 수 있다.The computing device 200 maps the first gradient 540 to the second map based on an operation for calculating an average value for each gradient value in the time axis direction and the frequency axis direction.

(541). Specifically, the computing device 200 may map the second map based on Equation (2).

(541).

는 제2 맵(541),

는 제1 그레이디언트(540)를 의미할 수 있다.

연산은 시간 축 및 주파수 축에 대해 평균 값을 산출하는 연산자를 의미할 수 있다.

Is the second map 541,

May denote the first gradient 540.

The operation may mean an operator that calculates an average value for the time axis and frequency axis.

(541)은 수학식 2를 통한 연산에 기초하여 (1*1*x_d)의 사이즈에 대응될 수 있고, 채널의 변화에 따라 변화하는 요소 값을 포함할 수 있다.Second map

541 may correspond to the size of (1 * 1 * x_d) based on the operation through Equation 2, and may include an element value that changes according to a change in the channel.

(553)을 생성할 수 있다.The computing device 200 calculates the second gradient 550 for the health information 530, and based on the second gradient 550, the reading model 520 refers to the time series signal 510. Third map related to time corresponding to area

(553).

Specifically, the computing device 200 may generate the second gradient 550 based on Equation (3).

는 판독 모델을 통해 출력된 건강 정보(530),

는 미리 학습된 콘볼루션 뉴럴 네트워크로부터 출력된 특징맵(525)을 의미할 수 있다.

The health information 530 output through the reading model,

May denote a feature map 525 output from a previously learned convolutional neural network.

(525)와 동일한 사이즈에 대응될 수 있다. 구체적으로, 특징맵

(525)는 ((시간 축 사이즈)*(주파수 축 사이즈)*(채널 축 사이즈(리드의 개수에 대응))) = (h(x)_w*h(x)_h*h(x)_d)에 대응되는 사이즈를 가지는데, 특징맵

(525)에 기초하여 생성되는 제2 그레이디언트(550) 역시 특징맵

(525)와 동일한 사이즈를 가질 수 있다.The second gradient 550 is a feature map

It may correspond to the same size as 525. Specifically, feature map

(525) is ((time axis size) * (frequency axis size) * (channel axis size (corresponding to the number of leads))) = (h (x) _w * h (x) _h * h (x) _d) It has a size corresponding to the feature map

The second gradient 550 generated based on 525 is also a feature map

It may have the same size as (525).

(551)을 생성할 수 있다. 구체적으로, 컴퓨팅 장치(200)는 수학식 4에 기초하여 제1 중간 맵

(551)을 생성할 수 있다.The computing device 200 maps the second gradient 550 to the first intermediate map based on an operation for calculating an average value for each gradient value in the frequency axis direction and the channel axis direction.

(551). Specifically, the computing device 200 is the first intermediate map based on Equation (4)

(551).

는 제1 중간 맵(551),

는 제2 그레이디언트(550)를 의미할 수 있다.

연산은 채널 축 및 주파수 축에 대해 평균 값을 산출하는 연산을 의미할 수 있다.

Is the first intermediate map 551,

Can mean the second gradient 550.

The operation may mean an operation for calculating an average value for a channel axis and a frequency axis.

(551)은 수학식 4를 통한 연산에 기초하여 (h(x)_w*1)의 사이즈에 대응될 수 있고, 시간의 변화에 따라 변화하는 요소 값을 포함할 수 있다.First intermediate map

551 may correspond to the size of (h (x) _w * 1) based on the operation through Equation 4, and may include an element value that changes with time.

(551)에 포함되는 요소 값들의 평균 값보다 작은 값을 가지는 요소 값을 0으로 수정함으로써 제2 중간 맵

(552)를 생성할 수 있다.Computing device 200 is the first intermediate map based on Equation (5)

The second intermediate map by modifying the element value having a value smaller than the average value of the element values included in 551 to 0

(552).

는 제2 중간 맵(552),

는 제1 중간 맵(551),

는 signum 함수,

은 평균 함수를 의미할 수 있다.

Is the second intermediate map 552,

Is the first intermediate map 551,

Signum function,

Can mean the mean function.

(552)의 사이즈는 (h(x)_w*1)에 대응되는 사이즈로 유지될 수 있다.Second intermediate map

The size of 552 may be maintained at a size corresponding to (h (x) _w * 1).

(552)의 사이즈를 (x_w*1)로 변환함으로써, 제3 맵

(553)을 생성할 수 있다.Computing device 200 is the second intermediate map based on Equation (6)

By converting the size of (552) to (x_w * 1), the third map

(553).

는 제3 맵(553),

는 제2 중간 맵(552),

는 제2 중간 맵

(552)의 사이즈를 (x_w*1)로 변환하는 임의의 선형함수를 의미할 수 있다. 예를 들어, 선형 함수는 선형적으로 맵의 사이즈를 확대하는 함수일 수 있다.

Is the third map (553),

Is the second intermediate map 552,

Is the second intermediate map

It may mean an arbitrary linear function that converts the size of (552) to (x_w * 1). For example, the linear function may be a function that linearly enlarges the size of the map.

(541)의 채널 별로 대응되는 요소 값을 제3 맵

(553)에 포함된 각각의 요소 값에 곱하는 연산을 통해, 제1 맵

(560)을 생성할 수 있다. 구체적으로, 컴퓨팅 장치(200)는 수학식 7에 기초하여 제1 맵

(560)을 생성할 수 있다.According to the present disclosure, the computing device 200 is the second map

The element map corresponding to each channel of 541 is a third map

The first map through an operation of multiplying each element value included in (553)

(560). Specifically, the computing device 200 first map based on Equation (7)

(560).

는 제1 맵(560),

는 제3 맵(553),

는 제2 맵(541)을 의미할 수 있다.

Is the first map 560,

Is the third map (553),

May mean the second map 541.

(541)에 포함되는 요소 값을 제3 맵

(553)에 포함되는 각각의 요소 값에 곱하는, 수학식 7의 연산을 통해 제1 맵

(560)을 생성할 수 있다.Computing device 200 is a second map having a single element value for each channel

The element map included in (541) is a third map

The first map through the operation of equation (7), multiplied by each element value included in (553)

(560).

6 is a diagram illustrating an example of a time series signal in which a reference region is visualized according to the present disclosure.

Referring to FIG. 6, each of the visualization information 601, 602, ..., 611, and 612 is a time series signal obtained from each of the first lead, ..., and the 12th lead, and the information referenced by the reference model. You can visualize. In the visualization information (601, 602, ..., 611, 612), the total size of the horizontal axis is 2500 points, which may correspond to a time of 10 seconds.

Since the visualization information (602, 605, 606) is represented by a relatively small brightness value, the time-series signals obtained from leads 2, 5, and 6 corresponding to the visualization information (602, 605, 606) are read models. In the process of reading, it may be determined that it is rarely referenced.

The time series signal of the 4th read corresponding to the visualization information 604 that includes the region of the relatively large bright value series at a high frequency is frequently referred to by the reading model, and among the signals corresponding to the 4th read, the visualization information In 604, it may be determined that a region having a relatively large brightness value is high in reference. Based on the results in which the reference region is visualized, it is possible to visually confirm the basis on which the reading model performed the reading, and reliability of the reading result of the reading model can be provided. In addition, based on the visualized results, the practitioner can effectively practice.

Based on the description of the above embodiment, a person skilled in the art can realize that the method and / or processes of the present invention and the steps can be realized in hardware, software or any combination of hardware and software suitable for a specific application. I can understand the point clearly. The hardware may include general purpose computers and / or dedicated computing devices or specific computing devices or special features or components of a particular computing device. The processes can be realized by one or more microprocessors, microcontrollers, embedded microcontrollers, programmable digital signal processors or other programmable devices, having internal and / or external memory. Additionally, or alternatively, the processes may be configured to process application specific integrated circuit (ASIC), programmable gate array, programmable array logic (PAL) or electronic signals. Any other device or combination of devices. Moreover, the objects of the technical solution of the present invention or parts contributing to the prior art may be implemented in the form of program instructions that can be executed through various computer components and recorded in a machine-readable recording medium. The machine-readable recording medium may include program instructions, data files, data structures, or the like alone or in combination. The program instructions recorded on the machine-readable recording medium may be specially designed and configured for the present invention or may be known and usable by those skilled in the computer software field. Examples of machine-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs, DVDs, and Blu-rays, and magnetic-optical media such as floptical disks. (magneto-optical media), and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions are stored and compiled or interpreted to be executed on a machine capable of executing a heterogeneous combination of processors, processor architectures, or combinations of different hardware and software, or any other program instructions, as well as any of the devices described above. Can be made using structured programming languages such as C, object-oriented programming languages like C ++, or advanced or low-level programming languages (assemblies, hardware description languages and database programming languages and techniques), machine code, This includes not only bytecode, but also high-level language code that can be executed by a computer using an interpreter.

Thus, in one aspect according to the present disclosure, when the above-described method and combinations thereof are performed by one or more computing devices, the combination of the method and method may be implemented as executable code that performs each step. In another aspect, the method can be implemented as systems that perform the steps, and the methods can be distributed in various ways across devices or all functions can be integrated into one dedicated, standalone device or other hardware. In another aspect, means for performing the steps associated with the processes described above may include any hardware and / or software described above. All such sequential combinations and combinations are intended to fall within the scope of this disclosure.

For example, the hardware device may be configured to operate as one or more software modules to perform processing according to the present disclosure, and vice versa. The hardware device may include a processor such as an MPU, CPU, GPU, TPU, which is combined with a memory such as ROM / RAM for storing program instructions and is configured to execute instructions stored in the memory, and an external device and a signal. It may include a communication unit for sending and receiving. In addition, the hardware device may include a keyboard, a mouse, and other external input devices for receiving commands written by developers.

In the above, the present invention has been described by specific embodiments such as specific components and limited embodiments and drawings, but this is provided only to help the overall understanding of the present invention, and the present invention is not limited to the above embodiments, Any person having ordinary skill in the art to which the present invention pertains may seek various modifications and variations from these descriptions.

Therefore, the spirit of the present invention is not limited to the above-described embodiment, and should not be determined, and all modifications that are equally or equivalent to the claims, as well as the claims to be described later, fall within the scope of the spirit of the present invention. Would say

Such an equivalent or equivalent modification would include a method that is logically equivalent, which can produce, for example, the same results as implementing the method according to the present disclosure. The scope should not be limited by the examples described above, but should be understood in the broadest sense acceptable by law.

Claims (7)

In the process of reading the time-series signal through a reading model that reads the time-series signal using an artificial neural network, the reference area referenced by the artificial neural network in the time-series signal is visualized to be distinguished from other areas. A method for visualizing a time-series signal performed by a computing device, comprising:
Generating a first map for applying attention to the time-series signal through back propagation of the reading result of the reading model;
Generating visualization information in which the reference area is visualized by applying an attention to the time-series signal based on the first map; And
Outputting the visualization information
Including,
Generating the first map,
A method for visualizing a time series signal, wherein the first map is generated based on a gradient of a reading result of the reading model.
According to claim 1,
The step of generating the visualization information,
Determining identification information corresponding to the reference area based on a result of applying the attention; And
Generating the visualization information based on the determined identification information
A time-series signal visualization method comprising a.
A method for visualizing a time-series signal performed by a computing device, which visualizes a reference region referred to in the time-series signal in the process of reading a time-series signal through a reading model,
Generating a first map for applying attention to the time-series signal through back propagation of the reading result of the reading model;
Generating visualization information in which the reference area is visualized by applying an attention to the time-series signal based on the first map; And
Outputting the visualization information
Including,
Generating the first map,
Calculating a first gradient for an output of a classifier included in the reading model based on the time-series signal converted into a frequency domain signal;
Generating a second map based on the first gradient;
Calculating a second gradient for the output of the classifier based on a feature map output through a convolutional neural network included in the reading model;
Generating a third map based on the second gradient; And
Generating the first map based on the second map and the third map
A time-series signal visualization method comprising a.
According to claim 3,
Generating the second map,
With respect to the first gradient, a second map is generated by calculating an average value in a time axis direction and a frequency axis direction,
Generating the third map,
With respect to the second gradient, a third map is generated by calculating an average value in a frequency axis direction and a channel axis direction,
Generating the first map,
Generating a modified third map by modifying a value of elements corresponding to a value smaller than the average value of the elements among the elements included in the third map;
Resizing the modified third map to correspond to the size of the second map; And
Generating the first map by multiplying the value of each element included in the resized third map by the value of each element of the second map.
A time-series signal visualization method comprising a.
According to claim 3,
The feature map,
A time-series signal visualization method generated based on a signal corresponding to one period of the time-series signal.
A computer program, stored on a machine-readable non-transitory recording medium, comprising instructions implemented to cause a computing device to perform the method of claim 1.
In the process of reading the time-series signal through a reading model that reads the time-series signal using an artificial neural network, the reference area referenced by the artificial neural network among the time-series signals is visualized to be distinguished from other areas. In the computing device,
A communication unit that receives a time series signal; And
Processor to visualize the reference area
Including,
The processor,
Generating a first map for applying attention to the time-series signal through back propagation of the reading result of the reading model,
By applying the attention to the time-series signal based on the first map, the visualization information in which the reference region is visualized is generated,
The visualization information is output through a display,
The processor,
And generate the first map based on the gradient of the reading result of the reading model.

Images