KR20210024872A - Method for evaluating test fitness of input data for neural network and apparatus thereof - Google Patents

Abstract

Disclosed are a method and apparatus for evaluating the test fitness of input data for a neural network. The method for evaluating the test fitness of input data for a neural network according to one embodiment of the present invention comprises the steps of: receiving input data; acquiring an evaluation model corresponding to an output pattern of neurons while the neural network is trained; inferring a first result among inferable results by applying the input data to the neural network; inferring a second result among the inferable results based on the evaluation model and the output pattern of the neurons in response to the input data; and evaluating the test fitness of the input data by comparing the first result and the second result.

Description

[Method FOR EVALUATING TEST FITNESS OF INPUT DATA FOR NEURAL NETWORK AND APPARATUS THEREOF}

It relates to a method and apparatus for evaluating the test suitability of input data for neural networks, and to, for example, a deep learning system.

Deep learning (hereinafter, DL) systems have made significant advances in various fields including image recognition, speech recognition and machine translation. Based on their ability to respond to or even surpass human behavior, DL systems are being adopted in key areas of safety and security, such as autonomous driving and malware detection.

Safety and security key areas may have to be accurate and predictable. Although the DL system exhibits excellent performance, it is known to exhibit unexpected behavior in certain situations. For example, in the case of an autonomous vehicle equipped with a DL system, if another vehicle is expected to give way, but does not, there may be a case of causing a collision with the vehicle.

Because of this, the DL system needs to verify its operation and validity. However, it may be difficult to directly apply the existing software test technology to the DL system. For example, existing white-box testing techniques that increase structural coverage may not be useful in a DL system. This is because the operation of the DL system is not explicitly encoded in the control flow structure.

Two assumptions can be made for testing and verification of the DL system. The first assumption might be that if the two inputs to the DL system are similar to some human sense, then the outputs should be similar. This may be a generalization of the nature of metamorphic testing. The second assumption may be that the more diverse the input set, the more effectively the DL system can be tested.

Testing and verification under both assumptions is an advanced form compared to manual ad hoc testing, but there are still limitations. Simply counting the number of neurons whose activation value satisfies a particular condition makes it possible to quantify the testing effect of a given set of inputs, but may convey little information about individual inputs. For example, it can be difficult to explain why an input with a higher NC should be considered better than another input with a lower NC, and why certain inputs naturally activate more neurons above a threshold than other inputs. . For test suitability criteria to be really useful, it may be necessary to be able to guide the selection of individual inputs.

In the present invention, a new conformance test for the DL system may be proposed. The new conformance test may be Surprise Adequacy for Deep Learning (SADL) for the DL system. A set of test inputs suitable for a DL system may need to be systematically diversified from inputs similar to training data to inputs significantly different from training data.

When it comes to subdividing individual inputs, SADL can measure how surprising the inputs are for a DL system. A measure of the actual degree of surprise can be based on the likelihood that the system has seen similar inputs during training (for example, it can be related to the estimated probability density distribution during the learning process using kernel density estimation). Alternatively, the measurement of the actual surprise may be based on the distance between the training data and the vector representing the neuron activation trace of a given input (for example, a Euclidean distance can be used). As a result, the SA (Surprise Adequacy) of the series of test inputs can be measured through the surprise values of individual inputs included in the set.

In the present invention, SADL, which is a surprise adequacy framework for a DL system capable of quantitatively measuring the relative surprise (SA, Surprise Adequacy) of each input in relation to the learning data, may be proposed. In addition, instead of measuring the number of neurons with a specific activation characteristic, an SC (Surprise Coverage, surprise coverage) can be proposed that uses SA to measure the surprise range of a discrete input. SA and SC can accurately capture input surprises and can be good indicators of how the DL system reacts to unknown inputs. SA correlates with how the DL system finds the input, and can be used to accurately classify hostile examples. In addition, the SC can be used to guide input selection for more effective re-learning of the DL system for hostile examples as well as the synthesized input.

A method for evaluating the test suitability of input data for a neural network according to an embodiment includes the steps of: receiving the input data; Acquiring an evaluation model corresponding to an output pattern of the neurons while the neural network is being trained, for each of the results that can be inferred by the neural network including a plurality of neurons; Inferring a first result from among the inferable results by applying the input data to the neural network; Inferring a second result from among the inferable results based on the evaluation model and output patterns of the neurons in response to the input data; And comparing the first result and the second result, thereby evaluating the test suitability of the input data.

According to an embodiment, the method for evaluating the test suitability of the input data for a neural network may further include determining how well the neural network has learned based on a result of evaluating the test suitability.

According to an embodiment, when the test suitability is less than a predetermined threshold value, a method for evaluating test suitability of input data for neural networks includes: obtaining a training set; And retraining the neural network based on the acquired training set.

According to an embodiment, obtaining the learning set may include determining the learning set such that at least some of data included in the learning set is associated with the input data.

According to an embodiment, the step of inferring the first result may include obtaining a result output from the neural network in response to the input data.

According to an embodiment, the step of inferring the second result is based on a probability density distribution formed by data included in the evaluation model, and the output pattern of the neurons in response to the input data is It may include determining where to place on the probability density distribution.

According to an embodiment, the step of inferring the second result may include: determining data having the highest similarity to an output pattern of the neurons responding to the input data among data included in the evaluation model; And determining a value corresponding to a result of outputting the data having the highest similarity as the second result when the neural network is input.

According to an embodiment, the similarity may be calculated by comparing individual elements of the first data and corresponding elements of the second data with respect to the first data and the second data to be compared.

According to an embodiment, the similarity is a Euclidean distance calculated by comparing individual elements of the first data and corresponding elements of the second data in relation to the first data and second data to be compared. Euclidean distance).

An apparatus for evaluating the test suitability of input data for a neural network according to an embodiment includes a memory in which a program is recorded; And a processor that executes the program, wherein the program comprises: receiving the input data; Acquiring an evaluation model corresponding to an output pattern of the neurons while the neural network is being trained, for each of the results that can be inferred by the neural network including a plurality of neurons; Inferring a first result from among the inferable results by applying the input data to the neural network; Inferring a second result from among the inferable results based on the evaluation model and output patterns of the neurons in response to the input data; And comparing the first result and the second result, thereby evaluating the test suitability of the input data.

1 is a diagram for explaining an output pattern of a plurality of neurons included in a neural network according to an exemplary embodiment.
FIG. 2 is a diagram for describing a degree of suitability for surprise according to an exemplary embodiment, and for explaining a degree of suitability for surprise based on a possibility.
3 is a diagram for describing a degree of suitability for surprise based on a distance according to an exemplary embodiment.
4 is a flowchart illustrating a method of evaluating a test suitability of input data for a neural network according to an embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for the purpose of illustration only, and may be changed and implemented in various forms. Accordingly, the embodiments are not limited to a specific disclosure form, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea.

Although terms such as first or second may be used to describe various components, these terms should be interpreted only for the purpose of distinguishing one component from other components. For example, a first component may be referred to as a second component, and similarly, a second component may be referred to as a first component.

When a component is referred to as being "connected" to another component, it is to be understood that it may be directly connected or connected to the other component, but other components may exist in the middle.

Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, action, component, part, or combination thereof is present, but one or more other features or numbers, It is to be understood that the presence or addition of steps, actions, components, parts, or combinations thereof, does not preclude the possibility of preliminary exclusion.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the relevant technical field. Terms as defined in a commonly used dictionary should be construed as having a meaning consistent with the meaning of the related technology, and should not be interpreted as an ideal or excessively formal meaning unless explicitly defined in the present specification. Does not.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The same reference numerals shown in each drawing indicate the same members.

1 is a diagram for explaining an output pattern of a plurality of neurons included in a neural network according to an exemplary embodiment.

Referring to FIG. 1, a neural network includes one or more layers, and each layer may include a plurality of neurons. According to an embodiment, the neural network may include a first layer 110 and a second layer 120.

와 관련하여 그 값을 출력할 수 있다. 예를 들어, 제1 레이어(110)가 포함하는 복수의 뉴런들은 0.6, 0.2 및 0.1이라는 값들을 출력하고, 제2 레이어(120)가 포함하는 복수의 뉴런들은 0.1, 0.3 및 0.5라는 값들을 출력할 수 있다.A plurality of neurons included in the first layer 110 and the second layer 120 are individual inputs

In relation to, you can print the value. For example, a plurality of neurons included in the first layer 110 outputs values of 0.6, 0.2, and 0.1, and a plurality of neurons included in the second layer 120 outputs values of 0.1, 0.3, and 0.5. can do.

와 관련하여 이를 분류한 결과를 출력할 수 있다. 예를 들어, 입력

와 관련하여 출력된 결과는 'dog'가 될 수 있다. 이 경우, 출력된 결과는 입력

가 미리 정해진 출력의 후보들 중 '개'의 영상에 가장 유사하게 매치됨을 의미할 수 있다.Neural network input

In relation to, the result of classifying it can be output. For example, enter

The result output in relation to the may be'dog'. In this case, the output result is the input

It may mean that is most similarly matched to an image of'dog' among candidates of predetermined output.

Values output by a plurality of neurons may be used to evaluate test suitability. Hereinafter, in FIGS. 1 to 3, values output by a plurality of neurons are referred to as activation traces (AT) for convenience of explanation, and a value that is a standard for evaluating test suitability will be referred to as Surprise Adequacy. I can. However, in FIG. 4, in describing the flow of operation, the expressions'values output by a plurality of neurons' and'test suitability evaluation' may be used as they are in respect of the expression of the claims.

Since the DL system is prone to errors for unfamiliar inputs, the diversity of test inputs of the DL system can be more meaningful in measurements related to the learning system. The aim of the present invention is to define a criterion for quantitatively measuring behavioral differences observed in a given set of inputs with respect to training data.

A. Activation Trace and Surprise Adequacy

으로, 입력 값의 집합을

으로 설정하자. 입력 x에 대한 단일 뉴런 n의 활성화 값을

으로 설정하자. 정렬된 뉴런의 서브 집합에 대해,

이고,

는 활성화 값의 벡터를 나타내고,

의 개별 뉴런에 대응하는 각 요소 :

의 카디널리티(cardinality)는

과 같을 수 있다.

는

의 뉴런들

의 활성화 궤적(AT, Activation Trace)일 수 있다(이하, 활성화 궤적AT로 지칭함). 유사하게,

는 일련의 입력

에 대해

의 뉴런을 통해 관찰되는 AT 집합이 되도록 할 수 있다(

). 주어진 입력에 대해 네트워크를 실행할 때마다 AT을 이용할 수 있다.The set of neurons that make up the DL system D

With the set of input values

Let's set it to The activation value of a single neuron n for input x

Let's set it to For a subset of ordered neurons,

ego,

Denotes a vector of activation values,

Each element corresponding to an individual neuron in:

The cardinality of

It can be the same as

Is

Neurons

It may be an activation trace (AT, Activation Trace) of (hereinafter referred to as an activation trace AT). Similarly,

Is a series of inputs

About

It can be made to be the set of ATs observed through neurons in (

). You can use AT whenever you run the network for a given input.

와 관련하여 모든

에서 관찰된 AT은

를 이용하여 실행될 때 조사중인 DL 시스템의 동작을 완전히 캡처한다고 가정할 수 있다.Since the operation of the DL system is driven by data-flow rather than control-flow,

All in connection with

AT observed in

It can be assumed that the operation of the DL system under investigation is completely captured when executed using.

FIG. 2 is a diagram for describing a degree of suitability for surprise according to an exemplary embodiment, and for explaining a degree of suitability for surprise based on a possibility.

Referring to FIG. 2, AT may be expressed as a caption 230 and a caption 240. However, this is only illustrated in two dimensions for convenience of explanation, and the actual AT may have three dimensions or higher dimensions.

The inputs can be shown as captions 210 and 220, assuming that the AT and inputs can be shown on two dimensions. Caption 210 may indicate Surprising input, and caption 220 may indicate Not Surprising input.

를 계산할 수 있다. 이어서, 새로운 입력

가 주어지면,

의 AT을

와 비교하여 T가

에 비해 얼마나 놀라운지를 측정할 수 있다. 이 정량적 유사성이 측정된 결과가 놀라움 적합도가 될 수 있다.The surprise may correspond to the relative novelty of a new input given in relation to the input used for learning. The surprise fit (hereinafter, the surprise fit is referred to as SA) may aim to measure the relative novelty (ie, surprise) of a given new input in relation to the input used for learning. Given the training set T, we first record the activation values of all neurons using all the inputs in the training data set.

Can be calculated. Subsequently, a new input

Is given,

AT

Compared with T

You can measure how surprising it is. The result of this quantitative similarity measurement can be a surprising fit.

와

의 유사성을 측정하는 방법이 서로 다른 SA의 두 가지 방식을 소개한다. 한 가지 방식은 도 2에서 설명되고, 나머지 한 가지 방식은 도 3에서 설명될 수 있다.Below,

Wow

Here are two ways of measuring the similarity of SAs. One method may be described in FIG. 2, and the other method may be described in FIG. 3.

B. Likelihood-based Surprise Adequacy (LSA)

의 각 활성화 값의 확률 밀도를 추정하고, 추정된 밀도와 관련하여 새로운 입력의 놀라움을 획득하기 위한 방법일 수 있다. 이것은 KDE를 사용하여 적대적 예시들(adversarial examples)을 탐지하는 기존 연구의 확장된 형태일 수 있다. 차원(dimensionality)과 계산 비용을 줄이려면 선택한 레이어

의 뉴런만을 고려할 수 있다. 이 경우, AT 집합

이 생성될 수 있다.In order to estimate the probability density function, Kernel Density Estimation (KDE), which is a method of estimating the probability density function of a random variable, may be used. Using the resulting density function, the relative likelihood of a random variable for a specific value can be estimated. Likelihood-based SA (LSA) uses KDE to

It may be a method for estimating the probability density of each activation value of, and obtaining a surprise of a new input in relation to the estimated density. This could be an extended form of existing research that uses KDE to detect adversarial examples. Selected layer to reduce dimensionality and computational cost

We can only consider the neurons of In this case, the AT set

Can be generated.

일 수 있다. 대역폭 매트릭스 H, 가우시안 커널 함수 K, 새로운 입력

의 AT 및

가 주어지면 KDE는 아래의 수학식 1과 같이 밀도 함수

를 생성할 수 있다.In order to further reduce the computational cost, neurons having an activation value lower than a predefined threshold value may be filtered out. These neurons may not provide much information to the KDE. The cardinality of each trajectory is

Can be Bandwidth matrix H, Gaussian kernel function K, new input

AT and

Given is, KDE is a density function as shown in Equation 1 below.

Can be created.

To measure the surprise of the input x, it increases when the probability density decreases (i.e., the input is rare compared to the training data), and decreases when the probability density increases (i.e., the input is similar to the training data). You may need metics to do it.

According to one embodiment, a general approach to converting the probability density to a measure of rareness can be adopted. However, the definition method of the LSA is not necessarily limited to this example, but for convenience of description, embodiments in which an approach method of converting a probability density into a scarcity measure will be described below.

In this case, the LSA can be defined to be a negative logarithmic value for density. The result may be as in Equation 2 below.

로 교체함으로써 수행될 수 있다. 일실시예에 따르면, DL 분류기에 퍼 클래스 LSA(per-class LSA)가 이용될 수 있다.The LSA can be made more precise with additional information about the input type. For example, with respect to DL classifier D, inputs sharing the same class label can be expected to have similar ATs. This computes the LSA per class, and T for class c

It can be done by replacing with. According to an embodiment, a per-class LSA (LSA) may be used for the DL classifier.

를 사용하여 분류기(classifier)를 테스트하는 경우, 입력

는 조사중인 DL 시스템에 의해 클래스 c로 분류될 수 있다. 이 경우,

의 놀라움은

에 대하여 보다 의미있게 측정될 수 있다(Tc는 구성원이 c로 분류되는 T의 서브 집합). 기본적으로, 입력이 전체의 학습 예시들(training examples)과 관련하여 놀라운 것이 아니더라도, 클래스 c의 예로서는 놀라운 것일 수가 있다.SA can be measured more accurately and meaningfully by focusing on at least a part of the learning set T through a specific type of DL task. For example, new input

If you are testing a classifier using the input

Can be classified as class c by the DL system under investigation. in this case,

The surprise of

Can be measured more meaningfully for (Tc is a subset of T whose members are classified as c). Basically, although the input isn't surprising with respect to the overall training examples, it can be surprising as an example of class c.

3 is a diagram for describing a degree of suitability for surprise based on a distance according to an exemplary embodiment.

에 대응되고, 도 2의 캡션 220은 새로운 입력

에 대응된다고 하자. 이 경우,

에서 클래스

까지의 거리 및

에서 클래스

까지의 거리와 비교하여,

의 AT는

의 AT에 비하여 클래스

에서 더 멀리 떨어져 있을 수 있다(즉,

). 결과적으로, 클래스

과 관련하여,

이

보다 더 놀라운 것으로 결정될 수 있다.Referring to FIG. 3, assuming that the AT and the input can be shown in two dimensions, the inputs may be shown as the caption 210 of FIG. 2 and the non-surprising input may be shown as caption 220 of FIG. Caption 210 of Figure 2 is a new input

Corresponds to, and the caption 220 of FIG. 2 is a new input

Let's say it corresponds to. in this case,

In class

Distance to and

In class

Compared to the distance to,

AT

Class compared to AT of

Can be further away from (i.e.

). As a result, the class

In relation to,

this

It can be determined to be even more surprising.

의 AT와 학습 중에 관측된 AT 사이의 유클리드 거리를 이용하는 DSA(Distance-based SA, 거리 기반의 놀라움 적합도)가 정의될 수 있다. 거리 측정 기준 인 DSA는 입력 간 경계들(boundaries)을 활용하는 데 효과적일 수 있다. 거리

및

를 비교함으로써(다시 말해, 새로운 입력의 AT와 기준점의 거리 간의 거리를 비교함으로써),

의 학습 데이터에서 가장 가까운 AT 인

및

까지의 거리(즉, 기준점에서 측정된

까지의 거리)는 새로운 입력이 클래스 경계(class boundary)에 얼마나 가까운지를 나타낼 수 있다.As an alternative to the LSA, the distance between ATs can be used simply as a measure of surprise. Here, the new input

DSA (Distance-based SA, distance-based surprise fit) using the Euclidean distance between the AT of the AT and the AT observed during learning can be defined. DSA, a distance measurement criterion, can be effective in utilizing boundaries between inputs. Street

And

By comparing (that is, by comparing the distance between the AT of the new input and the distance of the reference point),

Which is the closest AT in the training data of

And

Distance to (i.e., measured from the reference point)

Distance) can indicate how close the new input is to the class boundary.

For classification problems, inputs closer to class boundaries can be more surprising and valuable in terms of test input diversity. On the other hand, it may be difficult to apply DSA in the case of tasks without boundaries between inputs, such as predicting an appropriate steering angle for an autonomous vehicle. If the class boundary does not exist, even if the AT of the new input is far from the AT of the other learning input, it may not be guaranteed that the new input is surprising. This may be because if the class boundary does not exist, even if the AT of the new input is far from the AT of the other learning input, the AT of the new input may still be located in the crpowded parts of the AT space. . However, in the case of classification tasks, still applying only DSA may be more effective than applying LSA.

, 새로운 입력

및 새로운 입력에 대한 예측된 클래스

가 주어지는 경우, 기준점

가 동일한 클래스를 공유하는

의 가장 가까운 이웃으로 정의될 수 있다.According to an embodiment, a DL system D composed of a set of neurons N may be trained for classification of a class C set using a training data set T. Set of activation trajectories

, New input

And predicted class for new input

If is given, the reference point

Share the same class

Can be defined as the nearest neighbor of.

는 아래의 수학식 3과 같이 계산될 수 있다.Benchmark

Can be calculated as in Equation 3 below.

및

사이의 거리

는 아래의 수학식 4와 같이 계산될 수 있다.

And

Distance between

Can be calculated as in Equation 4 below.

이외의 클래스에서

에서 가장 가까운 이웃을 찾을 수 있다.to the next,

In a class other than

You can find the nearest neighbors in.

는 아래의 수학식 5와 같이 계산될 수 있다.Nearest neighbor

Can be calculated as in Equation 5 below.

및

사이의 거리

는 아래의 수학식 6과 같이 계산될 수 있다.

And

Distance between

Can be calculated as in Equation 6 below.

의 AT로부터 자신의 클래스

에 속하는 알려진 AT까지의 거리 및

클래스의 AT와 다른 클래스인

에 알려진 AT 사이의 거리를 비교하는 것을 목표로 할 수 있다. 자신의 클래스

에 속하는 알려진 AT까지의 거리가

클래스의 AT와 다른 클래스인

에 알려진 AT 사이의 거리보다 더 큰 경우,

는 분류 DL 시스템 D의 클래스

에 대한 놀라운 입력이 될 수 있다.Intuitively, the DSA is a new input

Own class from AT

Distance to a known AT belonging to and

Which is a different class from AT of the class

You can aim to compare the distances between the ATs known to. Own class

The distance to a known AT belonging to

Which is a different class from AT of the class

If greater than the distance between ATs known to,

Class of Classification DL System D

It can be an amazing input for.

DSA according to an embodiment may be calculated as in Equation 7 below.

및

는 유클리드 거리가 아닌 다른 방법으로 계산될 수 있다. 또는, DSA는

에서

를 뺀 값으로 계산될 수도 있다.However, in addition to these embodiments, there may be various methods for formulating DSA. E.g,

And

Can be calculated in a way other than the Euclidean distance. Or, DSA

in

It can also be calculated by subtracting.

4 is a flowchart illustrating a method of evaluating a test suitability of input data for a neural network according to an embodiment.

Referring to FIG. 4, the apparatus for evaluating the test suitability of input data for neural networks receives input data (410).

The apparatus for evaluating the test suitability of input data for a neural network acquires an evaluation model corresponding to an output pattern of neurons for each result that can be inferred by a neural network including a plurality of neurons (420 ).

By applying the input data to the neural network, the apparatus for evaluating the test suitability of the input data for the neural network infers a first result from among the inferable results (430). The apparatus for evaluating the test suitability of input data for neural networks may infer a first result by acquiring a result output from the neural network in response to the input data.

The apparatus for evaluating the test suitability of input data for neural networks infers a second result from among the inferable results based on the output pattern and the evaluation model of the neurons responding to the input data (440). According to an embodiment, the apparatus for evaluating the test suitability of input data for neural networks is based on a probability density distribution formed by data included in the evaluation model, and the output pattern of neurons responding to the input data is a probability density distribution. The second result can be inferred by deciding where to place the image.

According to an embodiment, the apparatus for evaluating the test suitability of input data for neural networks determines the output patterns of neurons responding to the input data and data with the highest similarity, among data included in the evaluation model, and determines the data with the highest similarity. When the neural network receives an input, a value corresponding to the output result may be determined as the second result. In this case, the degree of similarity may be calculated by comparing individual elements of the first data and corresponding elements of the second data with respect to the first data and the second data to be compared. For example, the similarity may correspond to a Euclidean distance calculated by comparing individual elements of the first data and elements of the corresponding second data in relation to the first data and the second data to be compared. I can.

By comparing the first result and the second result, the apparatus for evaluating the test suitability of the input data for neural networks evaluates the test suitability of the input data (450).

The apparatus for evaluating the test suitability of the input data for neural networks may further determine how well the neural network is trained based on the result of evaluating the test suitability. According to an embodiment, when the test suitability is less than a predetermined threshold, the apparatus for evaluating test suitability of input data for neural networks acquires a training set, and retrains the neural network based on the acquired training set. I can. In this case, the training set may be determined such that at least some of the data included in the training set is associated with the input data.

Hereinafter, contents not described in FIGS. 1 to 4 will be further described.

D. Surprise Coverage (SC)

의 상한과,

를 n개의 SA 세그먼트로 나누는 버킷들

이 주어지면, 입력 X 집합에 대한 SC인 SC(X)는 아래의 수학식 8과 같이 정의될 수 있다.Given an input set, it is also possible to measure the range of SC (Surprise Coverage) values that the input set covers. Since LSA and DSA are defined in a continuous space, the space of surprise can be classified using bucketing and LSC (Likelihood-Based Surprise Coverage) and DSC (Distance-based Surprise Coverage) can be defined.

With the upper limit of,

Buckets dividing into n SA segments

Given is given, SC(X), which is the SC for the input X set, may be defined as in Equation 8 below.

The input set having a high SC may be a set including various inputs ranging from an input similar to the training data (ie, when the SA is low) to an input that is not similar to the training data (ie, when the SA is high). As the input set for the DL system is systematically diversified in consideration of the SA, the learning result of the network can be effectively checked through the input set.

Like C in SC, terms such as coverage or cover are used, but the meaning of SA-based coverage may be different from that of simple structural coverage. First, unlike most structural coverage criteria, there may not be a finite number of targets to be covered, such as in statement or branch coverage. At least in theory, the degree of surprise of the input can be arbitrarily determined. However, an input with an arbitrarily high SA value may not be related to the problem domain or may be less interesting (e.g., traffic sign images may not be related to the test of an animal photo classifier). SC can only be measured against a predefined upper limit, in the same way that the theoretically finite path range is limited by parameters.

Second, the SC is not limited to the combinatorial set cover problem, but can be formulated based on test suite minimization. This may be because a single input generates only a single SA value and cannot belong to multiple SA buckets. Redundancy for SC as coverage criteria is weaker than structural coverage, and multiple targets can be covered with a single input.

III. Research Questions

In connection with the present invention, the following questions may be presented.

RQ1. Surprise: Can SADL capture a relative surprise on the input of the DL system?

Answers to RQ1 can be provided from different perspectives. First, we can calculate the SA of each test input included in the original data set, and determine whether the DL classifier finds an input that is much more difficult to accurately classify the input. More surprising inputs can be expected to be more difficult to classify correctly. Second, since adversarial examples are expected to be more surprising and cause other behaviors of the DL system, it is possible to evaluate whether or not adversarial examples can be detected based on the SA value. Several sets of hostile examples can be created using different techniques and compared based on the SA value.

Finally, adversarial example classifiers may be trained using logistic regression on SA values. As an example, 10,000 hostile examples were generated using 10,000 original test images provided by MNIST and CIFAR-10 for each adversarial attack strategy, and 1,000 original test images randomly selected and 1,000 hostile examples were used. Logistic regression classifiers can be learned. The trained classifier can then be evaluated using the remaining 9,000 original test images and 9,000 hostile examples. If the SA value correctly captures the operation of the DL system, it can be expected that the SA-based classifier will successfully detect hostile examples. According to one embodiment, by using ROC-AUC (area under the curve of the Reliant Under Operator Operator characteristic) for evaluation, both true and false positive rates can be captured.

RQ2. Layer Sensitivity: Does the selection of neuronal layers used in SA calculations affect how accurately the SA reflects the behavior of the DL system?

According to one embodiment, Bengio et al. When a KDE-based hostile example detection technology is introduced, it is possible to assume the deepest (ie, the last hidden) layer that contains the most information useful for detection. We can evaluate this assumption in the context of the SA by calculating the LSA and DSA of all individual layers and then comparing the classifiers of the hostile examples learned for the SA at each layer.

RQ3. Correlation: Does the SC correlate with the existing coverage criteria of the DL system?

In addition to capturing the input surprise, the SC may have to match existing coverage criteria based on aggregation. If not, the SC may risk measuring anything other than input diversity. To this end, it can be checked whether the SC is correlated with other criteria. Specifically, inputs generated by inputs generated in different ways (i.e., different hostile examples generation techniques or input synthesis techniques) are accumulated to control input diversity, execute a DL system with these inputs, and SC and multiple inputs. It is possible to observe and compare changes in various coverage criteria, including three existing coverage criteria. A plurality of existing coverage criteria according to an embodiment include Neuron Coverage (NC) of DeepXplore, Neuron Level Coverage (NLC) introduced by Deep Gauge, k-section Neuron Coverage (KMNC), Neuron Boundary Coverage (NBC), and strong neuron activation. It may include at least some of the coverage (SNAC).

As an example, in the case of MNIST and CIFAR-10, 1,000 generated by FGSM, BIM-A, BIM-B, JSMA and C&W in each step, starting with the original test data (10,000 images) provided by the dataset. Hostile examples of dogs may be added. As another example, in the case of Dave-2, 700 composite images generated by DeepXplore may be added for each step starting from the original test data (5,614 images). As another example, in the case of Chauffeur, 1,000 composite images (Set1 to Set3) generated by applying an arbitrary number of DeepTest transforms may be added to each step.

RQ4. Guidance: Can the SA guide retraining of the DL system to improve the accuracy of the hostile examples and synthetic test inputs generated in DeepXplore?

를 4 개의 겹치는(overlapping) 부분 집합들로 분류될 수 있다. 구체적으로, SA의 값에 따라, 하위 25%의 SA 값들(

), 하위 50%의 SA 값들(

), 하위 75%의 SA 값들(

) 및 전체 SA 범위(

)이 부분 집합들에 포함될 수 있다.In order to evaluate whether SADL can guide additional learning of DL systems in order to increase accuracy compared to hostile examples, it may be important whether the SA can guide input selection for further learning. As an example, in an input synthesized with hostile examples for these models, four sets of 100 images in four different SA ranges may be selected. Assuming U as the upper bound used in RQ3 to calculate SC, the range of SA

Can be classified into four overlapping subsets. Specifically, according to the value of SA, SA values of the lower 25% (

), the bottom 50% of SA values (

), SA values of the lower 75% (

) And the full SA range (

) Can be included in subsets.

The four subsets can be expected to represent an increasingly diverse set of inputs. According to one embodiment, set the range R to one of four subsets, randomly sample 100 images from each R, and train existing models for additional generations (e.g., 5 additional generations). I can make it. Finally, we can measure the performance of each model (e.g., MNIST's accuracy, CIFAR-10's accuracy, and Dave-2's MSE) for all hostile and synthetic inputs. When retraining with a more diverse subset, performance can be expected to improve.

In the present invention, SADL, which is a surprise adequacy framework for a DL system capable of quantitatively measuring the relative surprise (SA, Surprise Adequacy) of each input in relation to the learning data, may be proposed. In addition, instead of measuring the number of neurons with a specific activation characteristic, an SC (Surprise Coverage, surprise coverage) can be proposed that uses SA to measure the surprise range of a discrete input. SA and SC can accurately capture input surprises and can be good indicators of how the DL system reacts to unknown inputs. SA correlates with how the DL system finds the input, and can be used to accurately classify hostile examples. In addition, the SC can be used to guide input selection for more effective re-learning of the DL system for hostile examples as well as the synthesized input.

The embodiments described above may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the devices, methods, and components described in the embodiments are, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. Further, the processing device may access, store, manipulate, process, and generate data in response to the execution of software. For the convenience of understanding, although it is sometimes described that one processing device is used, one of ordinary skill in the art, the processing device is a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to operate as desired or processed independently or collectively. You can command the device. Software and/or data may be interpreted by a processing device or, to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. , Or may be permanently or temporarily embodyed in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

As described above, although the embodiments have been described by the limited drawings, a person of ordinary skill in the art can apply various technical modifications and variations based on the above. For example, the described techniques are performed in a different order from the described method, and/or components such as systems, structures, devices, circuits, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and those equivalent to the claims also fall within the scope of the claims to be described later.

Claims (19)

In the test suitability evaluation method of input data for neural networks,
Receiving the input data;
Acquiring an evaluation model corresponding to an output pattern of the neurons while the neural network is being trained, for each of the results inferred by the neural network including a plurality of neurons;
Inferring a first result from among the inferable results by applying the input data to the neural network;
Inferring a second result from among the inferable results based on the evaluation model and output patterns of the neurons in response to the input data; And
Evaluating the test suitability of the input data by comparing the first result and the second result
Containing,
A method for evaluating the test fit of input data for neural networks.
The method of claim 1,
Determining how well the neural network has learned based on the result of evaluating the test suitability
Further comprising,
A method for evaluating the test fit of input data for neural networks.
The method of claim 1,
If the test suitability is less than a predetermined threshold value,
Obtaining a training set; And
Based on the acquired training set, retraining the neural network
Further comprising,
A method for evaluating the test fit of input data for neural networks.
The method of claim 3,
Obtaining the learning set
Determining the training set such that at least some of the data included in the training set is associated with the input data
Containing,
A method for evaluating the test fit of input data for neural networks.
The method of claim 1,
Inferring the first result
Obtaining a result output from the neural network in response to the input data
Containing,
A method for evaluating the test fit of input data for neural networks.
The method of claim 1,
Inferring the second result
Determining where the output patterns of the neurons responding to the input data will be located on the probability density distribution based on a probability density distribution formed by data included in the evaluation model
Containing,
A method for evaluating the test fit of input data for neural networks.
The method of claim 1,
Inferring the second result
Determining data having the highest similarity to an output pattern of the neurons responding to the input data among data included in the evaluation model; And
Determining a value corresponding to the output result when the neural network receives the data having the highest similarity as the second result
Containing,
A method for evaluating the test fit of input data for neural networks.
The method of claim 7,
The similarity is
With respect to the first data and the second data to be compared, calculated by comparing each individual element of the first data and the corresponding element of the second data,
A method for evaluating the test fit of input data for neural networks.
The method of claim 7,
The similarity is
Corresponding to the Euclidean distance calculated by comparing respective elements of the first data and corresponding elements of the second data with respect to the first data and second data to be compared,
A method for evaluating the test fit of input data for neural networks.
A computer-readable recording medium containing a program for performing the method of claim 1.
In the test suitability evaluation device for input data for neural networks,
A memory in which a program is recorded; And
Processor that executes the above program
Including,
The above program,
Receiving the input data;
Acquiring an evaluation model corresponding to an output pattern of the neurons while the neural network is being trained, for each of the results inferred by the neural network including a plurality of neurons;
Inferring a first result from among the inferable results by applying the input data to the neural network;
Inferring a second result from among the inferable results based on the evaluation model and output patterns of the neurons in response to the input data; And
Evaluating the test suitability of the input data by comparing the first result and the second result
To do,
A device for evaluating the test suitability of input data for neural networks.
The method of claim 11,
Determining how well the neural network has learned based on the result of evaluating the test suitability
To do more,
A device for evaluating the test suitability of input data for neural networks.
The method of claim 11,
If the test suitability is less than a predetermined threshold value,
Obtaining a training set; And
Based on the acquired training set, retraining the neural network
To do more,
A device for evaluating the test suitability of input data for neural networks.
The method of claim 13,
Obtaining the learning set
Determining the training set such that at least some of the data included in the training set is associated with the input data
Containing,
A device for evaluating the test suitability of input data for neural networks.
The method of claim 11,
Inferring the first result
Obtaining a result output from the neural network in response to the input data
Containing,
A device for evaluating the test suitability of input data for neural networks.
The method of claim 11,
Inferring the second result
Determining where the output patterns of the neurons responding to the input data will be located on the probability density distribution based on a probability density distribution formed by data included in the evaluation model
Containing,
A device for evaluating the test suitability of input data for neural networks.
The method of claim 11,
Inferring the second result
Determining data having the highest similarity to an output pattern of the neurons responding to the input data among data included in the evaluation model; And
Determining a value corresponding to the output result when the neural network receives the data having the highest similarity as the second result
Containing,
A device for evaluating the test suitability of input data for neural networks.
The method of claim 17,
The similarity is
With respect to the first data and the second data to be compared, calculated by comparing each individual element of the first data and the corresponding element of the second data,
A device for evaluating the test suitability of input data for neural networks.
The method of claim 17,
The similarity is
Corresponding to the Euclidean distance calculated by comparing respective elements of the first data and corresponding elements of the second data with respect to the first data and second data to be compared,
A device for evaluating the test suitability of input data for neural networks.

Images