KR102287430B1 - Method for evaluating test fitness of input data for neural network and apparatus thereof - Google Patents

Abstract

A method and apparatus for evaluating the test fit of input data for a neural network are disclosed. According to an embodiment, a method for evaluating test suitability of input data for a neural network includes receiving input data; acquiring activation trajectories corresponding to output patterns of neurons while the neural network is being trained; inferring a first result among inferable results by applying the input data to the neural network; inferring a second result among inferable results based on the output pattern and activation trajectory of neurons in response to the input data; and evaluating the test suitability of the input data by comparing the first result with the second result.

Description

Method for evaluating test fit of input data for neural network and device therefor

To a method and an apparatus for evaluating test suitability of input data for a neural network, for example, to a deep learning system.

Deep learning (hereinafter DL) systems have made significant advances in various fields including image recognition, speech recognition, and machine translation. Based on their ability to respond to, or even exceed, human behavior, DL systems are being adopted in key safety and security areas such as autonomous driving and malware detection.

Safety and security critical areas may need to be accurate and predictable. Although DL systems have excellent performance, they are known to exhibit unexpected behavior in certain situations. For example, in the case of an autonomous vehicle equipped with a DL system, there may be a case where another vehicle expects to yield but does not, causing a collision with the vehicle.

Because of this, the DL system needs verification of its operation and validity. However, it may be difficult to directly apply the existing software test technology to the DL system. For example, existing white-box testing techniques that increase structural coverage may not be useful in a DL system. This is because the operation of the DL system is not explicitly encoded in the control flow structure.

Two assumptions can be made for testing and verification of the DL system. The first assumption may be that if two inputs to a DL system are similar to some human sense, then their outputs should be similar. This may be a generalization of the nature of metamorphic testing. A second assumption might be that the more diverse the input set, the more effectively the DL system can be tested.

Testing and verification conducted under two assumptions is an advanced form compared to manual ad hoc testing, but its limitations still exist. Simply counting the number of neurons whose activation values satisfy a certain condition allows us to quantify the testing effect of a given set of inputs, but may convey little information about individual inputs. For example, it can be difficult to explain why an input with a higher NC should be considered better than another input with a lower NC, and why certain inputs naturally activate more neurons above a threshold than others. . For test suitability criteria to be really useful, it may need to be able to guide the selection of individual inputs.

In the present invention, a new conformance test for the DL system can be proposed. A new conformance test may be Surprise Adequacy for Deep Learning (SADL) for DL systems. A test input set suitable for a DL system may have to be systematically diversified to include inputs similar to training data to significantly different from training data.

In terms of subdividing individual inputs, SADL can measure how surprising an input is for a DL system. A measure of the actual degree of surprise may be based on the likelihood that the system saw similar inputs during training (eg, it may relate to the probability density distribution estimated during the learning process using kernel density estimation). Alternatively, the measure of actual surprise may be based on the distance (eg, Euclidean distance can be used) between the vector representing the neuron activation trace of a given input and the training data. As a result, the Surprise Adequacy (SA) of a set of test inputs can be measured through the surprise value of the individual inputs the set contains.

In the present invention, SADL, a surprise adequacy framework for a DL system that can quantitatively measure the relative surprise (SA, Surprise Adequacy) of each input with respect to learning data, can be proposed. Also, instead of measuring the number of neurons with a specific activation characteristic, a Surprise Coverage (SC) that measures the surprise range of a discrete input using SA may be proposed. SA and SC can accurately capture the surprise of the input and can be good indicators of how the DL system reacts to the unknown input. SA correlates with how the DL system looks for input and can be used to accurately classify adversarial instances. In addition, SC can be used to guide input selection for more effective re-learning of the DL system for adversarial examples as well as synthesized input.

According to an embodiment, a method for evaluating test suitability of input data for a neural network includes receiving the input data; obtaining an activation trajectory corresponding to an output pattern of the neurons while the neural network is being trained for each result that can be inferred by the neural network including a plurality of neurons; inferring a first result among the inferable results by applying the input data to the neural network; inferring a second result from among the inferable results based on an output pattern of the neurons in response to the input data and the activation trajectory; and evaluating the test suitability of the input data by comparing the first result with the second result.

According to an embodiment, the method for evaluating test suitability of input data for a neural network may further include determining how well the neural network is trained based on a result of evaluating the test suitability.

According to an embodiment, when the test suitability is less than a predetermined threshold, the test suitability evaluation method of input data for a neural network includes: acquiring a training set; and based on the acquired training set, re-training the neural network.

According to an embodiment, acquiring the training set may include determining the training set so that at least a portion of data included in the training set is associated with the input data.

According to an embodiment, inferring the first result may include obtaining a result output from the neural network in response to the input data.

According to an embodiment, in the inferring of the second result, the output pattern of the neurons in response to the input data is determined based on a probability density distribution formed by the data included in the activation trajectory. determining where on the probability density distribution it will be located.

According to an embodiment, the inferring of the second result may include: determining, among data included in the activation trajectory, data having the highest predetermined similarity to an output pattern of the neurons in response to the input data; and determining, as the second result, a value corresponding to a result output when the neural network receives the data having the highest similarity.

According to an embodiment, the similarity may be calculated by comparing individual elements of the first data and elements of the second data corresponding to the first data and second data to be compared.

According to an embodiment, the degree of similarity is a Euclidean distance ( Euclidean distance).

An apparatus for evaluating test suitability of input data for a neural network according to an embodiment includes: a memory in which a program is recorded; and a processor executing the program, wherein the program comprises: receiving the input data; obtaining an activation trajectory corresponding to an output pattern of the neurons while the neural network is being trained for each result that can be inferred by the neural network including a plurality of neurons; inferring a first result among the inferable results by applying the input data to the neural network; inferring a second result from among the inferable results based on an output pattern of the neurons in response to the input data and the activation trajectory; and evaluating the test suitability of the input data by comparing the first result with the second result.

1 is a diagram for exemplarily explaining an output aspect of a plurality of neurons included in a neural network according to an embodiment.
FIG. 2 is a diagram for explaining surprise fitness according to an embodiment and for explaining probability-based surprise fitness.
3 is a diagram for explaining distance-based surprise fit according to an embodiment.
4 is a flowchart illustrating a method for evaluating test suitability of input data for a neural network according to an exemplary embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for purposes of illustration only, and may be changed and implemented in various forms. Accordingly, the embodiments are not limited to a specific disclosure form, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical spirit.

Although terms such as first or second may be used to describe various components, these terms should be interpreted only for the purpose of distinguishing one component from another. For example, a first component may be termed a second component, and similarly, a second component may also be termed a first component.

When a component is referred to as being “connected to” another component, it may be directly connected or connected to the other component, but it should be understood that another component may exist in between.

The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, and includes one or more other features or numbers, It should be understood that the possibility of the presence or addition of steps, operations, components, parts or combinations thereof is not precluded in advance.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present specification. does not

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Like reference numerals in each figure indicate like elements.

1 is a diagram for exemplarily explaining an output aspect of a plurality of neurons included in a neural network according to an embodiment.

Referring to FIG. 1 , a neural network may include one or more layers, and each layer may include a plurality of neurons. According to an embodiment, the neural network may include a first layer 110 and a second layer 120 .

와 관련하여 그 값을 출력할 수 있다. 예를 들어, 제1 레이어(110)가 포함하는 복수의 뉴런들은 0.6, 0.2 및 0.1이라는 값들을 출력하고, 제2 레이어(120)가 포함하는 복수의 뉴런들은 0.1, 0.3 및 0.5라는 값들을 출력할 수 있다.The plurality of neurons included in the first layer 110 and the second layer 120 are individual inputs.

You can output its value in relation to . For example, the plurality of neurons included in the first layer 110 outputs values of 0.6, 0.2, and 0.1, and the plurality of neurons included in the second layer 120 outputs values of 0.1, 0.3, and 0.5. can do.

와 관련하여 이를 분류한 결과를 출력할 수 있다. 예를 들어, 입력

와 관련하여 출력된 결과는 'dog'가 될 수 있다. 이 경우, 출력된 결과는 입력

가 미리 정해진 출력의 후보들 중 '개'의 영상에 가장 유사하게 매치됨을 의미할 수 있다.Neural network input

In relation to this, it is possible to output the classification result. For example, input

The output result in relation to 'dog' may be 'dog'. In this case, the output result is the input

may mean that it most closely matches the image of 'dog' among the candidates of the predetermined output.

Values output from the plurality of neurons may be used to evaluate test suitability. Hereinafter, in FIGS. 1 to 3 , for convenience of explanation, values output from a plurality of neurons are referred to as activation traces (AT), and a value serving as a criterion for evaluation of test suitability is referred to as Surprise Adequacy. can However, in FIG. 4 , expressions such as 'values output by a plurality of neurons' and 'test suitability evaluation' may be used as they are in FIG.

Because DL systems are prone to errors with unfamiliar inputs, the diversity of test inputs in DL systems can be more meaningful in measurements related to learning systems. It is an object of the present invention to define a criterion for quantitatively measuring the behavioral difference observed in a given set of inputs with respect to learning data.

A. Activation Trace and Surprise Adequacy

으로, 입력 값의 집합을

으로 설정하자. 입력 x에 대한 단일 뉴런 n의 활성화 값을

으로 설정하자. 정렬된 뉴런의 서브 집합에 대해,

이고,

는 활성화 값의 벡터를 나타내고,

의 개별 뉴런에 대응하는 각 요소 :

의 카디널리티(cardinality)는

과 같을 수 있다.

는

의 뉴런들

의 활성화 궤적(AT, Activation Trace)일 수 있다(이하, 활성화 궤적AT로 지칭함). 유사하게,

는 일련의 입력

에 대해

의 뉴런을 통해 관찰되는 AT 집합이 되도록 할 수 있다(

). 주어진 입력에 대해 네트워크를 실행할 때마다 AT을 이용할 수 있다.The set of neurons that make up the DL system D

, the set of input values

Let's set it to The activation value of a single neuron n for an input x

Let's set it to For a subset of sorted neurons,

ego,

denotes the vector of activation values,

Each element corresponding to an individual neuron in:

The cardinality of

can be the same as

Is

neurons in

may be an activation trace (AT) of (hereinafter referred to as an activation trace AT). Similarly,

is a series of inputs

About

It can be made to be the AT set observed through the neurons of

). AT is available whenever the network is running for a given input.

와 관련하여 모든

에서 관찰된 AT은

를 이용하여 실행될 때 조사중인 DL 시스템의 동작을 완전히 캡처한다고 가정할 수 있다.Since the operation of the DL system is driven by data-flow, not control-flow,

all in relation to

AT observed in

can be assumed to fully capture the behavior of the DL system under investigation when run using

FIG. 2 is a diagram for explaining surprise fitness according to an embodiment and for explaining probability-based surprise fitness.

Referring to FIG. 2 , AT may be expressed as caption 230 and caption 240 . However, this is only shown in 2D for convenience of description, and the actual AT may have 3D or more dimensions.

Assuming that the AT and the input can be shown in two dimensions, the inputs can be shown as caption 210 and caption 220 . A caption 210 may indicate a surprising input, and a caption 220 may indicate a Not Surprising input.

를 계산할 수 있다. 이어서, 새로운 입력

가 주어지면,

의 AT을

와 비교하여 T가

에 비해 얼마나 놀라운지를 측정할 수 있다. 이 정량적 유사성이 측정된 결과가 놀라움 적합도가 될 수 있다.Surprise may correspond to the relative novelty of a given new input with respect to the input used for learning. Surprise fit (hereinafter referred to as surprise fit) may aim to measure the relative novelty (ie, surprise) of a given new input with respect to the input used for learning. Given a training set T, we first record the activation values of all neurons using all inputs from the training data set.

can be calculated. Then, a new input

is given,

AT's

compared with T

You can measure how amazing it is. The result of measuring this quantitative similarity can be a surprising fit.

와

의 유사성을 측정하는 방법이 서로 다른 SA의 두 가지 방식을 소개한다. 한 가지 방식은 도 2에서 설명되고, 나머지 한 가지 방식은 도 3에서 설명될 수 있다.below,

Wow

We introduce two methods of SA with different methods of measuring the similarity of . One scheme may be described in FIG. 2 , and the other scheme may be described in FIG. 3 .

B. Likelihood-based Surprise Adequacy (LSA)

의 각 활성화 값의 확률 밀도를 추정하고, 추정된 밀도와 관련하여 새로운 입력의 놀라움을 획득하기 위한 방법일 수 있다. 이것은 KDE를 사용하여 적대적 예시들(adversarial examples)을 탐지하는 기존 연구의 확장된 형태일 수 있다. 차원(dimensionality)과 계산 비용을 줄이려면 선택한 레이어

의 뉴런만을 고려할 수 있다. 이 경우, AT 집합

이 생성될 수 있다.In order to estimate the probability density function, Kernel Density Estimation (KDE), which is a method of estimating the probability density function of a random variable, may be used. Using the resulting density function, the relative probabilities for a particular value of a random variable can be estimated. LSA (Likelihood-based SA, Likelihood-Based Surprise Fit) uses KDE to

It may be a method for estimating the probability density of each activation value of , and obtaining a surprise of a new input in relation to the estimated density. This could be an extension of existing research using KDE to detect adversarial examples. Selected layer to reduce dimensionality and computational cost

Only neurons of In this case, the AT set

can be created.

일 수 있다. 대역폭 매트릭스 H, 가우시안 커널 함수 K, 새로운 입력

의 AT 및

가 주어지면 KDE는 아래의 수학식 1과 같이 밀도 함수

를 생성할 수 있다.In order to further reduce the computational cost, it is possible to filter out neurons whose activation values exhibit a variance lower than a predefined threshold. This neuron may not provide much information to the KDE. The cardinality of each trajectory is

can be Bandwidth matrix H, Gaussian kernel function K, new input

AT and

If is given, KDE is a density function as shown in Equation 1 below.

can create

To measure the surprise of the input x, it increases when the probability density decreases (i.e., the input is sparse compared to the training data), and decreases when the probability density increases (i.e. when the input is similar to the training data). metrics may be required.

According to an embodiment, a general approach of converting a probability density into a measure of rareness may be adopted. However, the definition method of the LSA is not necessarily limited to this example, but for convenience of explanation, examples will be described below in which an approach for converting a probability density into a sparsity measure is adopted.

In this case, LSA can be defined to be the negative value of the logarithm of the density. The result may be as in Equation 2 below.

로 교체함으로써 수행될 수 있다. 일실시예에 따르면, DL 분류기에 퍼 클래스 LSA(per-class LSA)가 이용될 수 있다.By using additional information about the input type, the LSA can be made more precise. For example, with respect to DL classifier D, inputs sharing the same class label may be expected to have similar ATs. It computes the LSA per class, and T for class c

This can be done by replacing According to an embodiment, a per-class LSA may be used for the DL classifier.

를 사용하여 분류기(classifier)를 테스트하는 경우, 입력

는 조사중인 DL 시스템에 의해 클래스 c로 분류될 수 있다. 이 경우,

의 놀라움은

에 대하여 보다 의미있게 측정될 수 있다(Tc는 구성원이 c로 분류되는 T의 서브 집합). 기본적으로, 입력이 전체의 학습 예시들(training examples)과 관련하여 놀라운 것이 아니더라도, 클래스 c의 예로서는 놀라운 것일 수가 있다.Certain types of DL tasks allow us to more accurately and meaningfully measure SA by focusing on at least a portion of the training set T. For example, a new input

When testing a classifier using

can be classified as class c by the DL system under investigation. in this case,

the surprise of

can be measured more meaningfully for (Tc is a subset of T whose members are classified as c). Basically, although the input is not surprising with respect to the training examples as a whole, it can be surprising for the example of class c.

3 is a diagram for explaining distance-based surprise fit according to an embodiment.

에 대응되고, 도 2의 캡션 220은 새로운 입력

에 대응된다고 하자. 이 경우,

에서 클래스

까지의 거리 및

에서 클래스

까지의 거리와 비교하여,

의 AT는

의 AT에 비하여 클래스

에서 더 멀리 떨어져 있을 수 있다(즉,

). 결과적으로, 클래스

과 관련하여,

이

보다 더 놀라운 것으로 결정될 수 있다.Referring to FIG. 3 , assuming that the AT and the input can be shown in two dimensions, the input with surprising inputs can be shown as caption 210 of FIG. 2 and the unsurprising input with caption 220 of FIG. 2 . Caption 210 of Figure 2 is a new input

Corresponding to , the caption 220 of FIG. 2 is a new input

Let's say it corresponds to in this case,

class in

distance to and

class in

Compared to the distance to

AT's

class compared to AT of

may be further away from (i.e.,

). As a result, the class

In relation to

this

It can be decided even more surprising.

의 AT와 학습 중에 관측된 AT 사이의 유클리드 거리를 이용하는 DSA(Distance-based SA, 거리 기반의 놀라움 적합도)가 정의될 수 있다. 거리 측정 기준 인 DSA는 입력 간 경계들(boundaries)을 활용하는 데 효과적일 수 있다. 거리

및

를 비교함으로써(다시 말해, 새로운 입력의 AT와 기준점의 거리 간의 거리를 비교함으로써),

의 학습 데이터에서 가장 가까운 AT 인

및

까지의 거리(즉, 기준점에서 측정된

까지의 거리)는 새로운 입력이 클래스 경계(class boundary)에 얼마나 가까운지를 나타낼 수 있다.As an alternative to LSA, the distance between ATs can be used simply as a measure of surprise. Here, new input

Distance-based SA (Distance-based SA, distance-based surprise fit) using the Euclidean distance between the AT of and the observed AT during training can be defined. DSA, a distance metric, can be effective in exploiting boundaries between inputs. distance

and

By comparing (i.e., by comparing the distance between the AT of the new input and the distance of the reference point),

The closest AT from the training data of

and

distance to (i.e., measured from the reference point

distance to) may indicate how close the new input is to the class boundary.

For classification problems, inputs closer to class boundaries may be more surprising and valuable in terms of test input diversity. On the other hand, it can be difficult to apply DSA for tasks where there are no boundaries between inputs, such as predicting an appropriate steering angle for an autonomous vehicle. If class boundaries do not exist, even if the AT of a new input is far from the AT of another learning input, it may not guarantee that the new input is surprising. This may be because if no class boundary exists, even if the AT of the new input is far from the AT of the other learning input, the AT of the new input may still be located in crowded parts of the AT space. . However, in the case of classification tasks, applying only DSA may still be more effective than applying LSA.

, 새로운 입력

및 새로운 입력에 대한 예측된 클래스

가 주어지는 경우, 기준점

가 동일한 클래스를 공유하는

의 가장 가까운 이웃으로 정의될 수 있다.According to an embodiment, a DL system D configured with a set of neurons N may be trained for a classification task of a class C set using the training data set T. Set of activation trajectories

, new input

and predicted classes for new inputs

If given, the reference point

share the same class

can be defined as the nearest neighbor of

는 아래의 수학식 3과 같이 계산될 수 있다.Benchmark

can be calculated as in Equation 3 below.

및

사이의 거리

는 아래의 수학식 4와 같이 계산될 수 있다.

and

distance between

can be calculated as in Equation 4 below.

이외의 클래스에서

에서 가장 가까운 이웃을 찾을 수 있다.to the next,

in a class other than

You can find the nearest neighbors in

는 아래의 수학식 5와 같이 계산될 수 있다.nearest neighbor

can be calculated as in Equation 5 below.

및

사이의 거리

는 아래의 수학식 6과 같이 계산될 수 있다.

and

distance between

can be calculated as in Equation 6 below.

의 AT로부터 자신의 클래스

에 속하는 알려진 AT까지의 거리 및

클래스의 AT와 다른 클래스인

에 알려진 AT 사이의 거리를 비교하는 것을 목표로 할 수 있다. 자신의 클래스

에 속하는 알려진 AT까지의 거리가

클래스의 AT와 다른 클래스인

에 알려진 AT 사이의 거리보다 더 큰 경우,

는 분류 DL 시스템 D의 클래스

에 대한 놀라운 입력이 될 수 있다.Intuitively, the DSA is a new input

Own class from AT

distance to a known AT belonging to and

A class different from the class AT

may aim to compare the distances between known ATs. own class

The distance to a known AT belonging to

A class different from the class AT

If greater than the distance between the known AT,

is a class of classification DL system D

can be a surprising input to

DSA according to an embodiment may be calculated as in Equation 7 below.

및

는 유클리드 거리가 아닌 다른 방법으로 계산될 수 있다. 또는, DSA는

에서

를 뺀 값으로 계산될 수도 있다.However, in addition to these embodiments, various methods for formulating the DSA may exist. E.g,

and

can be calculated in a way other than the Euclidean distance. Alternatively, the DSA is

at

It may be calculated by subtracting

4 is a flowchart illustrating a method for evaluating test suitability of input data for a neural network according to an exemplary embodiment.

Referring to FIG. 4 , the apparatus for evaluating the test suitability of input data for a neural network receives input data ( S410 ).

The apparatus for evaluating the test suitability of input data for a neural network acquires an activation trajectory corresponding to an output pattern of neurons for each result that can be inferred by a neural network including a plurality of neurons ( 420 ).

By applying the input data to the neural network, the apparatus for evaluating the test suitability of the input data for the neural network infers a first result among inferable results ( 430 ). The apparatus for evaluating the test suitability of input data for a neural network may infer the first result by obtaining a result output from the neural network in response to the input data.

Based on the output pattern and activation trajectory of the neurons in response to the input data, the test suitability evaluation apparatus for the input data for the neural network infers a second result from among the inferable results ( 440 ). According to an embodiment, the apparatus for evaluating the test suitability of input data for a neural network determines the output pattern of neurons responding to the input data based on the probability density distribution formed by the data including the activation trajectory. A second result can be inferred by determining where the image is located.

According to an embodiment, the apparatus for evaluating the test suitability of input data for a neural network determines data having the highest predetermined similarity with an output pattern of neurons responding to the input data among data included in the activation trajectory, and selects the data with the highest similarity. When the neural network receives an input, a value corresponding to an output result may be determined as the second result. In this case, the degree of similarity may be calculated by respectively comparing individual elements of the first data and elements of the corresponding second data with respect to the first data and the second data to be compared. For example, the degree of similarity may correspond to a Euclidean distance calculated by respectively comparing individual elements of the first data and corresponding elements of the second data with respect to the first data and the second data to be compared. can

By comparing the first result and the second result, the test suitability evaluation apparatus of the input data for the neural network evaluates the test suitability of the input data ( 450 ).

The apparatus for evaluating the test suitability of input data for a neural network may further determine how well the neural network is trained based on a result of evaluating the test suitability. According to an embodiment, when the test suitability is less than a predetermined threshold, the test suitability evaluation apparatus of the input data for a neural network acquires a training set, and based on the acquired training set, the neural network is trained again. can In this case, the training set may be determined such that at least a portion of data included in the training set is associated with the input data.

Hereinafter, contents not described in FIGS. 1 to 4 will be further described.

D. Surprise Coverage (SC)

의 상한과,

를 n개의 SA 세그먼트로 나누는 버킷들

이 주어지면, 입력 X 집합에 대한 SC인 SC(X)는 아래의 수학식 8과 같이 정의될 수 있다.Given a set of inputs, it is also possible to measure the range of Surprise Coverage (SC) values that the set of inputs covers. Since LSA and DSA are defined in a continuous space, it is possible to classify the space of surprise using bucketing and define Likelihood-Based Surprise Coverage (LSC) and Distance-based Surprise Coverage (DSC).

the upper limit of

buckets that divide n into n SA segments.

Given this, SC(X), which is the SC for the input X set, can be defined as in Equation 8 below.

The input set with high SC may be a set including various inputs ranging from inputs similar to training data (ie, low SA) to inputs not similar to training data (ie, high SA). As the input set to the DL system is systematically diversified in consideration of SA, it is possible to effectively check the learning result of the network through the input set.

Like C in SC, terms such as coverage or cover are used, but the meaning of SA-based coverage may be different from that of simple structural coverage. First, unlike most structural coverage criteria, there may not be a finite number of goals to cover, such as in a statement or branch coverage. At least theoretically, the degree of surprise of the input can be arbitrarily determined. However, an input with an arbitrarily high SA value may not be related to a problem domain, or may be less interesting (eg, a traffic sign image may not be related to a test of an animal photo classifier). SC can theoretically be measured only against a predefined upper bound, in the same way that a finite path range is bounded by a parameter.

Second, SC is not limited to a combinatorial set cover problem, and can be formulated based on test suite minimization. This may be because a single input only generates a single SA value and cannot belong to multiple SA buckets. As coverage criteria, redundancy for SC is weaker than structural coverage, and multiple targets can be covered with a single input.

III. Research questions

In connection with the present invention, the following questions may be posed.

RQ1. Surprise: Can SADL capture a relative surprise on the input of the DL system?

Answers to RQ1 may be provided from different perspectives. First, we can compute the SA of each test input included in the original data set and see whether the DL classifier finds the input that is much more difficult to classify the input correctly. A more surprising input can be expected to be more difficult to classify correctly. Second, since adversarial examples are expected to cause other behaviors of the DL system as well as more surprising, it can be evaluated whether adversarial examples can be detected based on the SA value. Multiple sets of adversarial instances can be generated using different techniques and compared based on SA values.

Finally, adversarial example classifiers can be trained using logistic regression on SA values. As an example, for each adversarial attack strategy, 10,000 adversarial examples are generated using 10,000 original test images provided by MNIST and CIFAR-10, and 1,000 randomly selected original test images and 1,000 adversarial examples are used. Logistic regression classifiers can be trained. Then, the trained classifier can be evaluated using the remaining 9,000 original test images and 9,000 adversarial examples. If the SA value correctly captures the behavior of the DL system, it can be expected that the SA-based classifier will successfully detect hostile instances. According to one embodiment, both true and false positive rates can be captured by using ROC-AUC (area under the curve of the Reliant Under Operator Operator characteristic) for evaluation.

RQ2. Layer Sensitivity: Does the choice of neuron layer used to calculate SA affect how accurately SA reflects the behavior of the DL system?

According to one embodiment, Bengio et al. When KDE-based adversarial examples detection technology is introduced, the deepest (ie, last hidden) layer containing the most information useful for detection may be assumed. We can evaluate this assumption in the context of SA by computing the LSA and DSA of all individual layers and then comparing the adversarial examples classifiers learned for SA in each layer.

RQ3. Correlation: Does the SC correlate with the existing coverage criteria of the DL system?

In addition to capturing input surprise, the SC may need to match existing coverage criteria based on aggregation. Otherwise, the SC may run the risk of measuring anything other than input diversity. To this end, it can be checked whether the SC is correlated with other criteria. Specifically, input diversity is controlled by accumulating inputs generated by inputs generated by different methods (i.e., different adversarial example generation techniques or input synthesis techniques), running a DL system with these inputs, SC and multiple It is possible to observe and compare changes in various coverage criteria including the existing coverage criteria. A plurality of existing coverage criteria according to an embodiment are DeepXplore's neuron coverage (NC), deep gauge introduced neuron level coverage (NLC), k-section neuron coverage (KMNC), neuron boundary coverage (NBC), and strong neuron activation. It may include at least a portion of the coverage (SNAC).

As an example, for MNIST and CIFAR-10, starting with the original test data (10,000 images) provided by the dataset, at each step 1,000 generated by FGSM, BIM-A, BIM-B, JSMA and C&W. A hostile example of dogs may be added. As another example, for Dave-2, 700 synthetic images generated by DeepXplore can be added at each step, starting from the original test data (5,614 images). As another example, in the case of Chauffeur, 1,000 composite images (Set1 to Set3) generated by applying an arbitrary number of DeepTest transforms may be added to each step.

RQ4. Guidance: Can the SA guide the retraining of the DL system to improve the accuracy of the synthetic test inputs and adversarial examples generated in DeepXplore?

를 4 개의 겹치는(overlapping) 부분 집합들로 분류될 수 있다. 구체적으로, SA의 값에 따라, 하위 25%의 SA 값들(

), 하위 50%의 SA 값들(

), 하위 75%의 SA 값들(

) 및 전체 SA 범위(

)이 부분 집합들에 포함될 수 있다.In order to evaluate whether SADL can guide further learning of DL systems to increase accuracy compared to adversarial examples, it may be important whether SA can guide input selection for further learning. As an example, in the input synthesized with adversarial examples for these models, four sets of 100 images from four different SA ranges may be selected. Assuming U as the upper bound used in RQ3 to calculate SC, the range of SA

can be classified into four overlapping subsets. Specifically, according to the value of SA, the lower 25% of SA values (

), SA values of the lower 50% (

), SA values of the lower 75% (

) and full SA coverage (

) can be included in the subsets.

The four subsets can be expected to represent increasingly diverse input sets. According to one embodiment, we set the range R to one of four subsets, randomly sample 100 images in each R, and train existing models for additional generations (e.g., 5 additional generations). can do it Finally, we can measure the performance of each model on all adversarial inputs and synthetic inputs (eg MNIST accuracy, CIFAR-10 accuracy, and Dave-2 MSE), respectively. When retraining with a more diverse subset, performance can be expected to improve.

In the present invention, SADL, a surprise adequacy framework for a DL system that can quantitatively measure the relative surprise (SA, Surprise Adequacy) of each input with respect to learning data, can be proposed. Also, instead of measuring the number of neurons with a specific activation characteristic, a Surprise Coverage (SC) that measures the surprise range of a discrete input using SA may be proposed. SA and SC can accurately capture the surprise of the input and can be good indicators of how the DL system reacts to the unknown input. SA correlates with how the DL system looks for input and can be used to accurately classify adversarial instances. In addition, SC can be used to guide input selection for more effective re-learning of the DL system for adversarial examples as well as synthesized input.

The embodiments described above may be implemented by a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the apparatus, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA) array), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (19)

In the method for evaluating the test suitability of input data for a neural network performed in a processor,
receiving the input data;
obtaining an activation trajectory corresponding to an output pattern of the neurons while the neural network is being trained for each result that can be inferred by the neural network including a plurality of neurons;
inferring a first result among the inferable results by applying the input data to the neural network;
inferring a second result from among the inferable results based on an output pattern of the neurons in response to the input data and the activation trajectory; and
evaluating the test suitability of the input data by comparing the first result with the second result;
containing,
A method for evaluating the test fit of input data for neural networks.
According to claim 1,
determining how well the neural network has been trained based on a result of evaluating the test suitability;
further comprising,
A method for evaluating the test fit of input data for neural networks.
According to claim 1,
If the test fit is less than a predetermined threshold,
obtaining a training set; and
re-training the neural network based on the acquired training set
further comprising,
A method for evaluating the test fit of input data for neural networks.
4. The method of claim 3,
The step of obtaining the training set is
determining the training set such that at least some of the data included in the training set is associated with the input data;
containing,
A method for evaluating the test fit of input data for neural networks.
According to claim 1,
The step of inferring the first result is
obtaining a result output from the neural network in response to the input data;
containing,
A method for evaluating the test fit of input data for neural networks.
According to claim 1,
The step of inferring the second result is
determining where the output pattern of the neurons in response to the input data is located on the probability density distribution based on a probability density distribution formed by data included in the activation trajectory
containing,
A method for evaluating the test fit of input data for neural networks.
According to claim 1,
The step of inferring the second result is
determining, among data included in the activation trajectory, data having the highest predetermined similarity to output patterns of the neurons in response to the input data; and
determining, as the second result, a value corresponding to a result output when the neural network receives the data having the highest similarity as the second result;
containing,
A method for evaluating the test fit of input data for neural networks.
8. The method of claim 7,
The similarity is
with respect to first data and second data to be compared, calculated by respectively comparing respective elements of the first data and corresponding elements of the second data;
A method for evaluating the test fit of input data for neural networks.
8. The method of claim 7,
The similarity is
With respect to the first data and the second data to be compared, corresponding to a Euclidean distance calculated by respectively comparing respective elements of the first data and corresponding elements of the second data,
A method for evaluating the test fit of input data for neural networks.
A computer-readable recording medium storing a program for executing the method of any one of claims 1 to 9 on a computer.
In the test suitability evaluation apparatus for input data for a neural network,
memory in which the program is recorded; and
a processor that executes the program
including,
The program is
receiving the input data;
obtaining an activation trajectory corresponding to an output pattern of the neurons while the neural network is being trained for each result that can be inferred by the neural network including a plurality of neurons;
inferring a first result among the inferable results by applying the input data to the neural network;
inferring a second result from among the inferable results based on an output pattern of the neurons in response to the input data and the activation trajectory; and
evaluating the test suitability of the input data by comparing the first result with the second result;
to do,
Test fit evaluation device for input data for neural networks.
12. The method of claim 11,
determining how well the neural network has been trained based on a result of evaluating the test suitability;
to do more,
Test fit evaluation device for input data for neural networks.
12. The method of claim 11,
If the test fit is less than a predetermined threshold,
obtaining a training set; and
re-training the neural network based on the acquired training set
to do more,
Test fit evaluation device for input data for neural networks.
14. The method of claim 13,
The step of obtaining the training set is
determining the training set such that at least some of the data included in the training set is associated with the input data;
containing,
Test fit evaluation device for input data for neural networks.
12. The method of claim 11,
The step of inferring the first result is
obtaining a result output from the neural network in response to the input data;
containing,
Test fit evaluation device for input data for neural networks.
12. The method of claim 11,
The step of inferring the second result is
determining where the output pattern of the neurons in response to the input data is located on the probability density distribution based on a probability density distribution formed by data included in the activation trajectory
containing,
Test fit evaluation device for input data for neural networks.
12. The method of claim 11,
The step of inferring the second result is
determining, among data included in the activation trajectory, data having the highest predetermined similarity to output patterns of the neurons in response to the input data; and
determining, as the second result, a value corresponding to a result output when the neural network receives the data having the highest similarity as the second result;
containing,
Test fit evaluation device for input data for neural networks.
18. The method of claim 17,
The similarity is
with respect to first data and second data to be compared, calculated by respectively comparing respective elements of the first data and corresponding elements of the second data;
Test fit evaluation device for input data for neural networks.
18. The method of claim 17,
The similarity is
With respect to the first data and the second data to be compared, corresponding to a Euclidean distance calculated by respectively comparing respective elements of the first data and corresponding elements of the second data,
Test fit evaluation device for input data for neural networks.

Images