CN110363243A - The appraisal procedure and device of disaggregated model - Google Patents

Abstract

This application involves a kind of appraisal procedure of disaggregated model and devices, which comprises obtains sample to be tested；Sample to be tested includes normal sample and corresponding to resisting sample；Sample to be tested is input to disaggregated model, obtains the prediction result of disaggregated model output；Prediction result includes sample class and corresponding forecast confidence；Obtain the confidence level vector of normal sample and the confidence level vector to resisting sample；According to normal sample with to the respective confidence level vector of resisting sample, confidence similarity is determined；According to confidence similarity, the assessment result of disaggregated model is obtained.Scheme provided by the present application, which can solve, can not assess the problem of disaggregated model is to robustness to resisting sample due to that can not learn internal structure and relevant parameter.

Description

The appraisal procedure and device of disaggregated model

Technical field

This application involves field of artificial intelligence, more particularly to a kind of appraisal procedure of disaggregated model, device, calculating Machine readable storage medium storing program for executing and computer equipment.

Background technique

With the development of artificial intelligence (Artificial Intelligence, AI) technology, there are various classification moulds The processing of recognition of face, unmanned, illegal picture filtering etc. may be implemented based on the classification results of disaggregated model output in type. However, in practical applications, also there is the malicious attack for disaggregated model.The mode of one of malicious attack is confrontation Sample attack, for example, criminal adds naked eyes in illegal picture for the disaggregated model for carrying out illegal picture filtration treatment The fine noise that can not be differentiated, forms confrontation samples pictures, and confrontation samples pictures may be then predicted as normogram by disaggregated model Piece is not filtered.To bypass the filtration treatment of disaggregated model.

As a result, when assessing disaggregated model, in addition to assessing its accuracy predicted, also to assess it and resist to resisting sample The defence capability of interference in other words needs to carry out the robustness of disaggregated model comprehensive, comprehensive assessment.

At present to the appraisal procedure of disaggregated model, commented primarily directed to the internal structure and relevant parameter of disaggregated model Estimate.However, the internal structure and relevant parameter of disaggregated model are protected in actual assessment scene, user can not simultaneously be obtained To the internal structure and relevant parameter of disaggregated model, therefore, it is impossible to effectively assess what resisting sample was interfered in disaggregated model resistance Robustness.

Therefore, the appraisal procedure of traditional disaggregated model, there is can not effectively assess disaggregated model to resist confrontation sample The problem of robustness of this interference.

Summary of the invention

Based on this, it is necessary to resist asking for the robustness interfered resisting sample for can not effectively assess disaggregated model Topic, provides appraisal procedure, device, computer readable storage medium and the computer equipment of a kind of disaggregated model.

A kind of appraisal procedure of disaggregated model, comprising:

Obtain sample to be tested；The sample to be tested includes normal sample and corresponding to resisting sample；

The sample to be tested is input to the disaggregated model, obtains the prediction result of the disaggregated model output；It is described Prediction result includes sample class and corresponding forecast confidence；

Obtain the confidence level vector of the normal sample and the confidence level vector to resisting sample；The confidence level vector by The corresponding forecast confidence composition of multiple sample class；

According to the normal sample and described to the respective confidence level vector of resisting sample, confidence similarity is determined；

According to the confidence similarity, the assessment result of the disaggregated model is obtained.

A kind of assessment device of disaggregated model, comprising:

Sample acquisition module, for obtaining sample to be tested；The sample to be tested includes normal sample and corresponding confrontation sample This；

It is defeated to obtain the disaggregated model for the sample to be tested to be input to the disaggregated model for sample input module Prediction result out；The prediction result includes sample class and corresponding forecast confidence；

Vector obtains module, for obtain the normal sample confidence level vector and to the confidence level of resisting sample to Amount；The confidence level vector is made of the corresponding forecast confidence of multiple sample class；

Similarity determining module, for according to the normal sample and described to the respective confidence level vector of resisting sample, really Fixation believes similarity；

Evaluation module, for obtaining the assessment result of the disaggregated model according to the confidence similarity.

A kind of computer readable storage medium is stored with computer program, when the computer program is executed by processor, So that the processor executes following steps:

Obtain sample to be tested；The sample to be tested includes normal sample and corresponding to resisting sample；

The sample to be tested is input to the disaggregated model, obtains the prediction result of the disaggregated model output；It is described Prediction result includes sample class and corresponding forecast confidence；

Obtain the confidence level vector of the normal sample and the confidence level vector to resisting sample；The confidence level vector by The corresponding forecast confidence composition of multiple sample class；

According to the normal sample and described to the respective confidence level vector of resisting sample, confidence similarity is determined；

According to the confidence similarity, the assessment result of the disaggregated model is obtained.

A kind of computer equipment, including memory and processor, the memory are stored with computer program, the calculating When machine program is executed by the processor, so that the processor executes following steps:

Obtain sample to be tested；The sample to be tested includes normal sample and corresponding to resisting sample；

The sample to be tested is input to the disaggregated model, obtains the prediction result of the disaggregated model output；It is described Prediction result includes sample class and corresponding forecast confidence；

Obtain the confidence level vector of the normal sample and the confidence level vector to resisting sample；The confidence level vector by The corresponding forecast confidence composition of multiple sample class；

According to the normal sample and described to the respective confidence level vector of resisting sample, confidence similarity is determined；

According to the confidence similarity, the assessment result of the disaggregated model is obtained.

Appraisal procedure, device, computer readable storage medium and the computer equipment of above-mentioned disaggregated model, by will be normal Sample and disaggregated model is input to resisting sample, the prediction result of disaggregated model output is obtained, according to samples multiple in prediction result The corresponding forecast confidence of this classification, obtains normal sample and to the respective confidence level vector of resisting sample, according to normal sample and To the respective confidence level vector of resisting sample, confidence similarity is obtained, disaggregated model resistance pair can be assessed according to confidence similarity The robustness of the interference of resisting sample, and the internal structure and relevant parameter of disaggregated model need not be depended on.Therefore, above-mentioned assessment side Thinking of the method based on Black-box Testing obtains using forecast confidence provided by disaggregated model and can reflect out disaggregated model institute By the confidence similarity of the annoyance level to resisting sample, and according to the confidence similarity assessment disaggregated model, solve due to It can not learn internal structure and relevant parameter and the problem of disaggregated model is to robustness to resisting sample can not be assessed.

Detailed description of the invention

Fig. 1 is a kind of applied environment figure of the appraisal procedure of disaggregated model in one embodiment；

Fig. 2 is a kind of flow diagram of the appraisal procedure of disaggregated model in one embodiment；

Fig. 3 is the normal sample image of one embodiment and the comparison schematic diagram of confrontation sample image；

Fig. 4 is a kind of schematic diagram of the frame structure of service valuation system of one embodiment；

Fig. 5 is a kind of flow diagram of indicating risk step in one embodiment；

Fig. 6 is the flow diagram of the appraisal procedure of another disaggregated model in one embodiment；

Fig. 7 is a kind of structural block diagram of the assessment device of disaggregated model in one embodiment；

Fig. 8 is the structural block diagram of the assessment device of another disaggregated model in one embodiment；

Fig. 9 is the structural block diagram of computer equipment in one embodiment.

Specific embodiment

It is with reference to the accompanying drawings and embodiments, right in order to which the objects, technical solutions and advantages of the application are more clearly understood The application is further elaborated.It should be appreciated that specific embodiment described herein is only used to explain the application, and It is not used in restriction the application.

Fig. 1 is a kind of applied environment figure of the appraisal procedure of disaggregated model in one embodiment.The assessment of the disaggregated model Method can be applied to service valuation system.The service valuation system includes server 110, user terminal 120 and classified service end 130。

Wherein, server 110 and user terminal 120 pass through network connection.User terminal 120 specifically can be terminal console or shifting Dynamic terminal, mobile terminal specifically can be at least one of mobile phone, tablet computer, laptop etc..Server 110 can be used The server cluster of independent server either multiple servers composition is realized.Classified service end 130 can be to provide point The server of class service, or the server cluster being made of multiple servers for providing classified service, classified service end 130 By disaggregated model, the service of recognition of face, unmanned, illegal picture filtering etc. is provided a user.

For example, filtering in the service in illegal picture, the picture that user can be published on forum by classified service end 130 is defeated Enter to disaggregated model, disaggregated model can classify to the picture of input, classification belonging to picture be predicted, if picture category In the classification of illegal picture, then the image filtering is fallen, prevents its publication in forum.

In an actual application scenarios, user needs to select illegal picture filtering services, it is therefore desirable to take to classification The disaggregated model that business end 130 is used to provide illegal picture filtering services is assessed, to judge the classification mould according to assessment result Whether the robustness of type meets user demand.

User can obtain the service provision interface at classified service end 130 by user terminal 120, then be mentioned according to the service For interface, service valuation request is initiated to server 110.Sample can be input to by server 110 according to service provision interface The disaggregated model at classified service end 130 exports prediction result by the disaggregated model at classified service end 130.Server 110 is according to defeated Prediction result out is assessed, and assessment result is fed back to user terminal 120.

As shown in Fig. 2, in one embodiment, providing a kind of appraisal procedure of disaggregated model.The present embodiment mainly with This method is applied to the server 110 in above-mentioned Fig. 1 to illustrate.Referring to Fig. 2, the appraisal procedure of the disaggregated model is specifically wrapped Include following steps:

S202 obtains sample to be tested；Sample to be tested includes normal sample and corresponding to resisting sample.

Wherein, sample to be tested can for will be input to disaggregated model, to obtain sample class and forecast confidence Sample.The sample type of sample to be tested can be image pattern, video sample, audio sample or samples of text.

Wherein, normal sample can be for without added with to antimierophonic sample.

It wherein, can be that the sample formed to antinoise is added with to normal sample to resisting sample.

In the specific implementation, server 110 can be set normal sample database, machine learning model library and to resisting sample life At algorithms library.Server 110 can choose a normal sample from normal sample database.For the normal sample, use Each machine learning model in machine learning model library, to each pair of resisting sample generating algorithm in resisting sample generating algorithm library And different confrontation noise intensities, generate multiple pairs of resisting samples of the normal sample.Server 110 has obtained normally as a result, Sample and corresponding largely to resisting sample, as sample to be tested.

Fig. 3 is the normal sample image of one embodiment and the comparison schematic diagram of confrontation sample image.As shown, left side Image be normal sample image, disaggregated model predict its sample type be " panda " (panda), forecast confidence It (confidence) is 57.7%.Intermediate image is that will be superimposed upon normal sample image to antinoise to antinoise, and obtain the right side The confrontation sample image of side.The naked eyes of the mankind can not tell the technicality of confrontation sample image and normal sample image, Apparently, two kinds of images are not different the naked eyes of the mankind, are all " pandas ".However, the sample of disaggregated model prediction confrontation sample image This type is " long ape and monkey " (gibbon), forecast confidence 99.3%.It can be seen that subtle by being added in normal sample To antinoise, disaggregated model can be made to make the prediction to make mistake.

Sample to be tested is input to disaggregated model by S204, obtains the prediction result of disaggregated model output；Prediction result includes Sample class and corresponding forecast confidence.

Wherein, sample to be tested can be predicted as specific sample classification and be predicted as specific by prediction result for disaggregated model The forecast confidence of sample class.

Wherein, sample class can be classification belonging to the content of sample.For example, a content is the picture sample of panda This, sample class is then panda.

Wherein, forecast confidence can be the credibility that is specific sample classification by sample predictions.Credibility is usual Belong to the probability of some classification by some sample to express, the higher letter represented by sample predictions as some sample class of probability The heart is bigger.For example, image content is the bear shape animal of black and white hair, the forecast confidence that sample class is predicted as panda is 90%, the forecast confidence for being predicted as grizzly bear is 10%.

In the specific implementation, sample to be tested can be passed through the service provision interface at classified service end 130 by server 110, it is defeated Enter to the disaggregated model at classified service end 130, disaggregated model can predict sample class belonging to sample to be tested, and determination is to be measured Sample belongs to the forecast confidence of each sample class.

Sample class belonging to the sample to be tested that classified service end 130 exports disaggregated model and corresponding prediction confidence Degree, feeds back to server 110 as prediction result.Server has obtained the pre- of the normal sample that disaggregated model is exported as a result, Survey result and the prediction result to resisting sample.

S206 obtains the confidence level vector of normal sample and the confidence level vector to resisting sample；Confidence level vector is by more The corresponding forecast confidence composition of a sample class.

Wherein, confidence level vector can be the vector of the corresponding forecast confidence composition of multiple sample class.

It obtains normal sample and is belonging respectively to the prediction of each sample class setting in the specific implementation, server 110 is available Reliability.Normal sample is belonging respectively to the forecast confidence of each sample class, forms the confidence level vector of normal sample.

For example, in prediction result, X to be predicted as to the forecast confidence of n sample class respectively for normal sample X For X₁、X₂…X_n, the confidence level vector A of normal sample X_X={ X₁、X₂…X_n}。

Similarly, the available forecast confidence for obtaining being belonging respectively to resisting sample each sample class of server 110. It will be belonging respectively to the forecast confidence of each sample class to resisting sample, form the confidence level vector to resisting sample.

For example, in prediction result, X` to be predicted as to the prediction confidence of n sample class respectively for resisting sample X` Degree is X`<sub>1</sub>、X`₂…X`<sub>n</sub>, to the confidence level vector B of resisting sample X`<sub>X`</sub>={ X`₁、X`<sub>2</sub>…X`_n}。

In practical application, the quantity of sample class to be predicted can be more, and server 110 can extract specific one Or multiple sample class, confidence level is formed using forecast confidence corresponding to target sample classification as target sample classification Vector, and confidence level vector need not be formed using the corresponding forecast confidence of sample class all in prediction result.

S208 determines confidence similarity according to institute's normal sample with to the respective confidence level vector of resisting sample.

Wherein, confidence similarity can be for the confidence level vector of normal sample and between the confidence level vector of resisting sample Similarity degree.

In the specific implementation, server 110 can compare the confidence level vector of normal sample and the confidence level to resisting sample to It measures, the similarity degree between determination will be similar between the confidence level vector of normal sample and the confidence level vector to resisting sample Degree, as confidence similarity.

For example, calculating the confidence level vector of normal sample and to the vector cosine value between the confidence level vector of resisting sample, Using the vector cosine value as confidence similarity.

In practical application, those skilled in the art can also determine confidence similarity using other modes.For example, it is also possible to The confidence level vector of normal sample is calculated by Euclidean distance and to the similarity degree between the confidence level vector of resisting sample.

S210 obtains the assessment result of disaggregated model according to confidence similarity.

Wherein, assessment result can be the result in multiple dimensions assessment disaggregated model for the robustness to resisting sample. The assessment result can be used for assessing disaggregated model and resist to the interference of resisting sample, be accurate sample class by confrontation sample predictions Ability.Assessment result, which can specifically include, comments resisting sample defence assessed value, the distribution of confrontation sample pattern, confrontation algorithm defence At least one of valuation, antagonistic intensity defence assessed value.

Wherein, robustness can remain to export correct data when being abnormal state or abnormal data occur for system Performance.

In the specific implementation, server 110 can resist the interference to resisting sample to disaggregated model according to confidence similarity Ability carries out various dimensions, comprehensive assessment, to obtain above-mentioned assessment result.Server 110 can send out assessment result It send to user terminal 120, shows assessment result to user for user terminal 120.User can judge the classification mould according to assessment result Whether type meets its user demand to the robustness to resisting sample, alternatively, being selected in multiple disaggregated models according to assessment result Meet the disaggregated model of its user demand.

It should be noted that confidence similarity can reflect the ability for the interference that disaggregated model is resisted to resisting sample.If Disaggregated model predicts that the forecast confidence that normal sample is specific sample type is higher, and predicting is same specific sample to resisting sample The forecast confidence of this type is lower, and calculating obtained confidence similarity then can be lower, lower confidence similarity, shows point Class model is larger by the interference to resisting sample, and disaggregated model is being resisted weaker to the ability of the interference of resisting sample, will fight Sample predictions are that the risk of error sample classification is higher.Conversely, higher confidence similarity, shows disaggregated model by confrontation sample This interference is smaller, and disaggregated model is stronger to the ability of the interference of resisting sample in resistance, is error sample by confrontation sample predictions The risk of classification is lower.According to confidence similarity, then can resist from multiple dimension comprehensive assessment disaggregated models to resisting sample Robustness.

For example, the assessment result of one of dimension, can be disaggregated model defends assessed value to resisting sample.Specifically Ground can average the confidence similarity of multiple pairs of resisting samples, obtain making to resisting sample defence assessed value for disaggregated model For assessment result.The assessment result can synthetically assess disaggregated model for the defence capability of a variety of different pairs of resisting samples. It is higher to resisting sample defence assessed value, represent, disaggregated model resistance robustness to resisting sample stronger to resisting sample defence capability Better.

In another example the assessment result of one of dimension, can be the confrontation algorithm defence assessed value of disaggregated model.Specifically The confidence similarity of multiple pairs of resisting samples can be grouped by ground according to the corresponding confrontation sample algorithm of multiple pairs of resisting samples, Each group of confidence similarity corresponds to the same confrontation sample algorithm and obtains for each group of confidence similarity calculation average value Disaggregated model defends assessed value as assessment result the confrontation algorithm of each confrontation sample algorithm.The assessment result can be commented Disaggregated model is estimated for a variety of different confrontation sample algorithms defence capability generated to resisting sample.Confrontation algorithm defence is commented Valuation is higher, i.e. confrontation algorithm defence capability is stronger, disaggregated model resist that some confrontation sample algorithm generates to resisting sample Robustness is better.

In another example assessed value and confrontation algorithm can also will be defendd to defend assessed value two assessed value conducts resisting sample Assessment result, synthetically to assess disaggregated model for the defence capability of a variety of different pairs of resisting samples and right from two dimensions In a variety of different confrontation sample algorithms defence capability generated to resisting sample.

Certainly, those skilled in the art can obtain the assessment of disaggregated model according to confidence similarity according to actual needs As a result.Above-mentioned example is merely to illustrate Shandong according to the available multiple dimensions of confidence similarity, for assessing disaggregated model Stick as a result, not the particular content of assessment result is restricted.For example, it is also possible to determine the confidence of multiple pairs of resisting samples Minimum similarity degree in similarity, then, it is determined that determining resisting sample for generating the confrontation sample corresponding to minimum similarity degree This confrontation sample algorithm, it is generated to the confrontation sample algorithm weaker to the defence capability of resisting sample to obtain disaggregated model Assessment result.

It should be noted that it is traditional when assessing disaggregated model, usually pass through white-box testing (White-box Test mode) is assessed.The mode of white-box testing needs to rely on the internal structure and relevant parameter of disaggregated model.And this Apply for that the appraisal procedure of the disaggregated model provided, the internal structure and relevant parameter for being not based on disaggregated model are assessed, it should Assessment mode is also referred to as Black-box Testing (Black-box Test).

It should be further noted that the appraisal procedure of above-mentioned disaggregated model, to be applied to carry out disaggregated model The application scenarios of line assessment are illustrated.In practical applications, above-mentioned appraisal procedure can be also used for surveying inside disaggregated model In the application scenarios of examination.Research staff can carry out close beta to disaggregated model by above-mentioned appraisal procedure, with assessment point Class model improves disaggregated model to the robustness to resisting sample, and according to assessment result, to promote the Shandong of disaggregated model Stick.

The appraisal procedure of above-mentioned disaggregated model is obtained by being input to disaggregated model by normal sample and to resisting sample The prediction result of disaggregated model output obtains normal sample according to the corresponding forecast confidence of sample class multiple in prediction result Originally and confidence is obtained according to normal sample and to resisting sample respective confidence level vector to the respective confidence level vector of resisting sample Similarity, the robustness for the interference that disaggregated model is resisted to resisting sample can be assessed according to confidence similarity, and need not be depended on The internal structure and relevant parameter of disaggregated model.Therefore, thinking of the above-mentioned appraisal procedure based on Black-box Testing, utilizes disaggregated model Provided forecast confidence, obtain can reflect out it is similar to the confidence of the annoyance level of resisting sample suffered by disaggregated model Degree, and according to the confidence similarity assessment disaggregated model, solve due to that can not learn internal structure and relevant parameter and can not Assess the problem of disaggregated model is to robustness to resisting sample.

Moreover, the appraisal procedure of above-mentioned disaggregated model, be not based on disaggregated model internal structure and relevant parameter into Row assessment, is either directed to the disaggregated model of image pattern, audio-video sample or samples of text, can apply above-mentioned Appraisal procedure assesses disaggregated model, and is not limited to the disaggregated model of specific internal and relevant parameter.Therefore, above-mentioned point The appraisal procedure of class model has the versatility of assessment object.

In one embodiment, step S202 can be specifically included:

It chooses sample and generates parameter；The selection sample generates parameter: in N_mIn a confrontation sample pattern, Target Countermeasure sample pattern is selected, and, in N_aIn a confrontation sample algorithm, Target Countermeasure sample algorithm is selected, and, Noise intensity ε is fought in maximum_maxNoise intensity ε is fought with minimum_minBetween, select Target Countermeasure noise intensity；By target Sample pattern, Target Countermeasure sample algorithm and Target Countermeasure noise intensity are fought as sample and generates parameter；Obtain Target Countermeasure The model parameter of sample pattern；Pass through the original sample noise of Target Countermeasure sample algorithm computation model parameter；Using target pair Antinoise intensity adjusts the noise intensity of original sample noise, is adjusted rear sample noise；Sample noise is superimposed after adjusting To normal sample, obtain initially to resisting sample；Target Countermeasure sample pattern initially will be input to resisting sample, obtains Target Countermeasure The prediction result of sample pattern output；When the prediction result mistake, initially resisting sample will be used as to resisting sample, and be back to The step of sample generates parameter is chosen, until obtaining the N of normal sample_gIt is a to resisting sample；Wherein, N_g=N_m*N_a*(ε_max-ε_min+ 1)。

Wherein, it can be for generating the relevant parameter to resisting sample that sample, which generates parameter,.Sample, which generates parameter, to be had Body is the parameter for fighting sample pattern, confrontation sample algorithm, confrontation noise intensity etc..

Wherein, confrontation sample pattern can be for for generating the machine learning model to resisting sample.

Wherein, confrontation sample algorithm can be for for generating the algorithm to resisting sample.

Wherein, confrontation noise intensity can be the intensity of noise added by confrontation sample.

Wherein, sample noise can be the noise for classification of disturbance model prediction sample class on confrontation sample.Example Such as, for image pattern, sample noise can be some pixel.

The appraisal procedure that the application is deeply understood for the ease of those skilled in the art, below with reference to a specific clothes The internal structure of business assessment system is illustrated.Fig. 4 is a kind of showing for the frame structure of service valuation system of one embodiment It is intended to.

As shown, sample generation module and service valuation can be deployed in the frame structure of service valuation system Module.Sample generation module is mainly used for generating sample to be tested, and service valuation module is then mainly used for according to disaggregated model end 130 The prediction result of feedback generates the assessment result of disaggregated model.

Specifically, sample generation module can be used for safeguarding normal sample database, machine learning model library and confrontation sample This generating algorithm library.

Wherein, normal sample database includes the normal sample of various sample types, such as image, video, audio, text Deng sample type.

Wherein, machine learning model library includes common machine learning model, as ResNet (residual error neural network), A kind of machine learning models such as Inception (neural network).

It wherein, include commonly to resisting sample generating algorithm, such as FGSM (Fast to resisting sample generating algorithm library Gradient Sign Method, Fast Field descent algorithm), BIM (Basic Iterative Methods, primary iteration calculate Method), C&W (Carlini&Wagner, a kind of pair of resisting sample generating algorithm), DeepFool (fascination learning algorithm) etc. is to resisting sample Generating algorithm.

A variety of normal or abnormal samples can be generated to assess disaggregated model in sample generation module, guarantee sample Coverage, so as to from multiple dimensions assessment disaggregated model to the robustness of the interference to resisting sample.

Sample generation module can randomly select a normal sample X from normal sample database.Then, sample is being chosen It, can be from the N in machine learning model library when this generation parameter_mA machine learning model { M₁、M₂…M_iAmong, select one A machine learning model, as generating the Target Countermeasure sample pattern M to resisting sample_i.It can also be generated to resisting sample The N of algorithms library_aIt is a to resisting sample generating algorithm { A₁、A₂…A_jAmong, select one to resisting sample generating algorithm, as with In generation to the Target Countermeasure sample algorithm A of resisting sample_j.It can also be in [ε_min, ε_max] section in, select a numerical value, As Target Countermeasure noise intensity ε_tar。

In practical application, since too small confrontation noise intensity can not effectively interfere disaggregated model, most Small confrontation noise intensity ε_minIt can be set to 1.And excessive confrontation noise intensity will to produce resisting sample and normal sample Raw biggish difference, can also accurately predict even if the weaker disaggregated model of robustness, assessment can not be effectively performed, because This, maximum confrontation noise intensity ε_maxIt can be set to 32.

By the above-mentioned means, obtaining Target Countermeasure sample pattern, Target Countermeasure sample algorithm and Target Countermeasure noise intensity Parameter is generated as above-mentioned sample.Then, sample generation module can determine the model parameter of Target Countermeasure sample pattern, adopt With the algorithmic formula of Target Countermeasure sample algorithm, an original sample noise is calculated.By the noise of the original sample noise Intensity is adjusted to Target Countermeasure noise intensity, is adjusted rear sample noise.Sample noise after the adjustment is superimposed to normal sample In sheet, obtain initially to resisting sample.

Target Countermeasure sample pattern initially will be input to resisting sample, Target Countermeasure sample pattern to initially to resisting sample into Row prediction, and prediction result is exported, which is initially to the sample class of resisting sample.When Target Countermeasure sample pattern is defeated Prediction result mistake out shows that the sample has played the role of interference prediction accuracy, therefore, can be initial right by this Resisting sample is as assessing to resisting sample.

Then, it is back to and chooses the step of sample generates parameter, choose another Target Countermeasure sample pattern, Target Countermeasure Sample algorithm or Target Countermeasure noise intensity, until obtaining the N of normal sample X_gIt is a to resisting sample X`.For N<sub>m</sub>It is a to resisting sample Model, N<sub>a</sub>A confrontation sample algorithm and [ε<sub>min</sub>, ε<sub>max</sub>] several confrontation noise intensities in section, it is freely combined To obtain N<sub>g</sub>=N<sub>m</sub>*N<sub>a</sub>*(ε<sub>max</sub>-ε<sub>min</sub>+ 1) a to resisting sample X`, in other words, for a normal sample, available (N_g+ 1) a sample to be tested.

The sample to be tested that sample generation module can will acquire is input to disaggregated model, and disaggregated model can be with feedback forecasting As a result, being assessed by service valuation module according to prediction result.

In practical applications, it can be assessed using multiple and different normal sample X.For example, choosing sample generates ginseng When number, the normal sample quantity N for being assessed can be chosen_e, normal sample quantity N_eMinimum value can be 1, maximum value It can be the total quantity of the normal sample in normal sample database.N is generated when being directed to a normal sample_gA confrontation sample This, then choose next normal sample and generate N_gIt is a to resisting sample.For N_eA normal sample can then recycle above-mentioned steps N_e It is secondary, until obtaining N_e*(N_g+ 1) a sample to be tested.

The appraisal procedure of above-mentioned disaggregated model, by using multiple confrontation sample patterns, multiple confrontation sample algorithms and Multiple confrontation noise intensities are generated multiple pairs of resisting samples of normal sample, are assessed using multiple pair of resisting sample, so as to The comprehensive assessment of various dimensions is carried out in confrontation sample pattern, confrontation sample algorithm, confrontation noise intensity to disaggregated model.

In one embodiment, step S208 can be specifically included:

It calculates the confidence level vector of normal sample and to the vector cosine value between the confidence level vector of resisting sample, is set Believe similarity.

In the specific implementation, obtaining the confidence level vector A of normal sample X_X={ X₁、X₂…X_n, and, to resisting sample X`'s Confidence level vector B<sub>X`</sub>={ X`<sub>1</sub>、X`₂…X`_n, it can be by following formula, vector cosine value between calculating:

Using the vector cosine value being calculated as above-mentioned confidence similarity, to be classified using confidence similarity The assessment result of model.

The appraisal procedure of above-mentioned disaggregated model, by calculate normal sample with to the respective confidence level vector of resisting sample it Between vector cosine value, can lead to too small amount of calculation amount obtain reflection normal sample with to the respective confidence level vector of resisting sample Similarity degree numerical value, save the spent process resource of assessment.

In one embodiment, step S210 can be specifically included:

The average value for calculating each confidence similarity defends assessed value as to resisting sample；Generate assessment result；Assessment knot Fruit includes defending assessed value to resisting sample.

Wherein, assessed value is defendd to resisting sample to be correct sample class for assessing disaggregated model for sample predictions are fought Ability numerical value.It can be to resist a variety of different pairs of resisting samples for assessing disaggregated model to resisting sample defence assessed value The integration capability of interference.

In the specific implementation, server 110 can calculate the sum of each confidence similarity, the sum of each confidence similarity is removed With the quantity of each confidence similarity, the average value of each confidence similarity is obtained, which is the confrontation of disaggregated model Sample defends assessed value.

For example, with reference to Fig. 4, confrontation Samples Estimates module in service valuation module, available N_gIt is a to resisting sample Confidence similarity is respectively Sim_N₁、Sim_N₂…Sim_N_g, { Sim_N is calculated to resisting sample module₁、Sim_N₂…Sim_N_g? Average value defends assessed value to resisting sample as above-mentioned.Confrontation Samples Estimates module exports this and defends assessed value to resisting sample, Assessment result as disaggregated model.

It should be noted that if disaggregated model prediction is the forecast confidence of some sample class to resisting sample, and it is pre- Survey normal sample be the sample class forecast confidence it is more similar, show even if use to resisting sample to disaggregated model into Row interference, disaggregated model is to the forecast confidence to resisting sample, still close with the forecast confidence of normal sample, mould of classifying Type is there is no by the interference to resisting sample, alternatively, the influence to the interference of resisting sample to the prediction result of disaggregated model is smaller.

, whereas if disaggregated model prediction is the forecast confidence of some sample class to resisting sample, with prediction normal sample This is that the forecast confidence of the sample class is dissimilar, shows to work as to use and interfere disaggregated model resisting sample, classifies Model differs greatly to the forecast confidence to resisting sample with the forecast confidence of normal sample, and disaggregated model receives pair The interference of resisting sample.

Therefore, it is a variety of different right to reflect disaggregated model resistance for the average value of the forecast confidence of each pair of resisting sample The interference of resisting sample will resist the integration capability that sample predictions are correct sample class.

The average value is bigger, shows that disaggregated model resistance is higher to the defence capability of the interference of resisting sample, will be to resisting sample The risk for being predicted as error sample classification is lower, that is, disaggregated model is preferable to the robustness of resisting sample in resistance.

The average value is smaller, shows that disaggregated model resistance is lower to the defence capability of the interference of resisting sample, will be to resisting sample The risk for being predicted as error sample classification is higher, that is, disaggregated model is poor to the robustness of resisting sample in resistance.

In practical application, assessment result can also include indicating risk.Specifically, a defence capability threshold can be preset Value generates indicating risk, then when being calculated to resisting sample defence assessed value lower than the defence capability threshold value to prompt user It is higher that disaggregated model will fight the risk that sample predictions are error sample classification.

The appraisal procedure of above-mentioned disaggregated model, the average value by calculating each confidence similarity, which is used as, defends resisting sample Assessed value solves due to that can not learn internal structure and relevant parameter and can not assess disaggregated model to a variety of different right The problem of robustness of resisting sample.

Moreover, the appraisal procedure of above-mentioned disaggregated model, with disaggregated model to the defence capability of a variety of different pairs of resisting samples As assessment dimension, based on the robustness of assessment dimension assessment disaggregated model, so as to more fully to disaggregated model Robustness is assessed.

In one embodiment, step S210 can be specifically included:

Determine the corresponding confrontation sample algorithm of confidence similarity and confrontation sample pattern；It is described to resisting sample according to described right Resisting sample algorithm and the confrontation sample pattern are generated；Each confidence similarity is grouped, multiple same algorithm similarities are obtained Set；Correspond to the same confrontation sample algorithm with each confidence similarity in algorithm similarity set；It determines respectively multiple With the minimum similarity degree in algorithm similarity set；The Target Countermeasure sample mould of multiple same algorithm similarity set is determined respectively Type；Target Countermeasure sample pattern is confrontation sample pattern corresponding with minimum similarity degree；Count multiple Target Countermeasure sample patterns Frequency of occurrence；The number is the number of multiple same algorithm similarity set corresponding to the same Target Countermeasure sample pattern Amount；Confrontation sample pattern distribution is generated, as assessment result；Fighting sample pattern distribution includes each Target Countermeasure sample pattern And corresponding frequency of occurrence.

It wherein, can be the collection of the corresponding identical confidence similarity of confrontation sample algorithm with algorithm similarity set It closes.

Wherein, Target Countermeasure sample pattern can be for the same as corresponding to the smallest confidence similarity in algorithm similarity set Sample pattern is fought, Target Countermeasure sample pattern has the risk of the internal model structure of leakage disaggregated model.

Wherein, the distribution of confrontation sample pattern can be the distribution of the frequency of occurrence of each Target Countermeasure sample pattern, confrontation Sample pattern is distributed the disclosure risk for assessing the internal model structure of disaggregated model.Confrontation sample pattern distribution can use The various ways such as histogram, lines figure, pie chart are presented.

In the specific implementation, server 110 can determine the corresponding confrontation sample algorithm of confidence similarity, and, determination is set Believe the corresponding confrontation sample pattern of similarity.Each confidence similarity is grouped by server 110 according to confrontation sample algorithm, Multiple groups are obtained with algorithm similarity set.

Then, the smallest confidence similarity is determined respectively in algorithm similarity set in multiple groups.Due to each confidence phase Have like degree corresponding to resisting sample, and has confrontation sample pattern corresponding, for generating this to resisting sample to resisting sample, because This, can correspondingly determine confrontation sample pattern corresponding to the smallest confidence similarity, as above-mentioned Target Countermeasure sample Model.

When determining that certain group with the Target Countermeasure sample pattern of algorithm similarity set, is then recorded, to determine the target Fight the frequency of occurrence of sample pattern.Determine that each group with the Target Countermeasure sample pattern of algorithm similarity set, then can count The frequency of occurrence of each Target Countermeasure sample pattern out.Finally, generating confrontation sample pattern distribution, the assessment as disaggregated model As a result.

For example, with reference to Fig. 4, confrontation sample pattern evaluation module in service valuation module can will be to resisting sample { X`<sub>11</sub>, X`₁₂…X`<sub>1j</sub>…X`_i1, X`<sub>i2</sub>…X`_ijCorresponding confidence similarity { Sim₁₁, Sim₁₂…Sim_1j…Sim_i1, Sim_i2… Sim_ij, according to confrontation sample algorithm { A₁、A₂…A_jBe grouped, multiple groups are obtained with algorithm similarity set { Sim₁₁, Sim₂₁…Sim_i1}、{Sim₁₂, Sim₂₂…Sim_i2}…{Sim_1j, Sim_2j…Sim_ij, and determine each group with algorithm similarity collection The smallest confidence similarity Sim in conjunction_{min_1}、Sim_{min_2}…Sim_{min_j}.Then, it is determined that each the smallest confidence similarity institute Corresponding confrontation sample pattern as Target Countermeasure sample pattern, and records the frequency of occurrence of Target Countermeasure sample pattern.

For example, in confrontation sample pattern { M₁、M₂…M_iIn, Target Countermeasure sample mould of certain group with algorithm similarity set Type is M₂, to M₂Frequency of occurrence then add 1.So analogize, when having N group with the Target Countermeasure sample pattern of algorithm similarity set It is M₂, M₂Frequency of occurrence be then N.

According to the frequency of occurrence of each Target Countermeasure sample pattern, confrontation sample pattern distribution is generated, sample pattern is fought Evaluation module exports confrontation sample pattern distribution, as assessment result.

For example, specific confrontation sample pattern distribution can be with are as follows: M₁Frequency of occurrence be 2, M₂Frequency of occurrence be 6, M₃Frequency of occurrence be 12 ... M_iFrequency of occurrence be 5.

It should be noted that passing through confrontation sample pattern distribution, it can be estimated that whether the internal model structure of disaggregated model It has a risk of leakage.If the frequency of occurrence of some Target Countermeasure sample pattern is more in confrontation sample pattern distribution, namely It is to say, it is generated to resisting sample according to some confrontation sample pattern in different confrontation sample algorithms, to disaggregated model Prediction causes biggish interference.It is therefore shown that the disaggregated model, which has higher possibility, to be constructed based on the confrontation sample pattern Internal model structure, so that there are disclosure risks for the internal model structure of disaggregated model.

If the internal model structure of disaggregated model is revealed, criminal can be directed to the internal model knot of disaggregated model Structure, generate it is various can not be classified that model accurately predicts to resisting sample, to carry out malicious attack to disaggregated model.

For example, criminal can be according to the internal model structure of disaggregated model, non-for illegal picture filtering services Addition may will be added with to antinoise, illegal picture filtering services and be predicted as closing to antimierophonic illegal image on method image Method image, there is no being filtered to it, so that illegal picture filtration inefficiencies.

, whereas if the distribution of each Target Countermeasure sample pattern is relatively uniform, criminal learns the interior of disaggregated model A possibility that portion's model structure, is lower, and the disclosure risk of the internal model structure of disaggregated model is lower.

The appraisal procedure of above-mentioned disaggregated model, by being divided each confidence similarity according to confrontation sample algorithm Group determines minimum similarity degree for each group confidence similarity, Target Countermeasure sample pattern is determined according to minimum similarity degree, according to mesh The frequency of occurrence of mark confrontation sample pattern obtains confrontation sample pattern distribution, so as to utilize confrontation sample pattern distribution assessment The disclosure risk of the internal model structure of disaggregated model is solved due to that can not learn internal structure and relevant parameter and can not be commented The problem of estimating the disclosure risk of disaggregated model internal model structure.

Moreover, the appraisal procedure of above-mentioned disaggregated model, using the disclosure risk of the internal model structure of disaggregated model as commenting Dimension is estimated, based on the robustness of assessment dimension assessment disaggregated model, so as to more fully to the robustness of disaggregated model It is assessed.

In one embodiment, as shown in figure 5, above-mentioned appraisal procedure can be with further include:

S502 determines maximum frequency of occurrence in the frequency of occurrence of each Target Countermeasure sample pattern；

S504 calculates the average value of the frequency of occurrence of each Target Countermeasure sample pattern, obtains frequency of occurrence mean value；

S506 calculates the number difference of maximum frequency of occurrence and frequency of occurrence mean value；

S508, when number difference is greater than preset threshold, generation disclosure risk prompt.

Wherein, disclosure risk prompt is for prompting the internal model structure of disaggregated model to have a risk of leakage.

In the specific implementation, server 110 can more each Target Countermeasure sample pattern frequency of occurrence, determine that maximum goes out Occurrence number.In addition, server 110 can also calculate the average value of the frequency of occurrence of each Target Countermeasure sample pattern, gone out Occurrence number mean value.Then, the difference for calculating maximum frequency of occurrence and frequency of occurrence mean value, obtains number difference.The number is poor Value is compared with preset threshold value, if number difference is greater than threshold value, shows the frequency of occurrence point of Target Countermeasure sample pattern Cloth is uneven, and the frequency of occurrence of some Target Countermeasure sample pattern is more, and there are leakages for the internal model structure of disaggregated model Therefore risk generates disclosure risk prompt, to prompt the internal model structure of user's disaggregated model to have higher leakage wind Danger.If number difference is less than threshold value, show that the frequency of occurrence of Target Countermeasure sample pattern is distributed relatively uniform, the disaggregated model Internal model structure disclosure risk it is lower.

The appraisal procedure of above-mentioned disaggregated model, it is poor by calculating the maximum frequency of occurrence number average with frequency of occurrence Value generates disclosure risk prompt according to number difference, user is allowed to learn whether the internal model structure of disaggregated model deposits In disclosure risk, the safety of disaggregated model is judged according to disclosure risk prompt convenient for user.

In one embodiment, step S210 can be specifically included:

The average value for calculating separately each confidence similarity in multiple same algorithm similarity set, it is anti-as confrontation algorithm Imperial assessed value；Generate assessment result；Assessment result includes that confrontation algorithm defends assessed value.

Wherein, confrontation algorithm defence assessed value is the confrontation that will be generated according to confrontation sample algorithm for assessing disaggregated model Sample predictions are the numerical value of the ability of correct sample class.Confrontation algorithm defence assessed value can be used for assessing disaggregated model resistance According to the ability of a variety of different confrontation sample algorithm interference generated to resisting sample.

In the specific implementation, server 110 can be asked for each group with each confidence similarity in algorithm similarity set With by the sum of confidence similarity each in every group of set divided by the quantity of each confidence similarity in the group set, obtain the group With the average value of each confidence similarity in algorithm similarity set, which is that the confrontation algorithm defence of disaggregated model is commented Valuation.For the average value that each group is calculated with algorithm similarity set, as disaggregated model calculates resisting sample for different The confrontation algorithm of method defends assessed value.

For example, with reference to Fig. 4, confrontation sample algorithm evaluation module in service valuation module is similar with algorithm for multiple groups Degree set { Sim₁₁, Sim₂₁…Sim_i1}、{Sim₁₂, Sim₂₂…Sim_i2}…{Sim_1j, Sim_2j…Sim_ij, calculate separately each group With the average value of the confidence similarity of algorithm similarity setObtain each confrontation sample algorithm { A₁、A₂… A_jCorresponding to confrontation algorithm defend assessed value

In practical application, assessment result can also include indicating risk.Specifically, a defence capability threshold can be preset Value generates indicating risk, then to prompt user when the confrontation algorithm defence assessed value being calculated is lower than the defence capability threshold value Disaggregated model is higher by the risk that some confrontation sample algorithm confrontation sample predictions generated is error sample classification.

The appraisal procedure of above-mentioned disaggregated model, by calculating each group with the flat of the confidence similarity in algorithm similarity set Mean value defends assessed value as confrontation algorithm, solves due to that can not learn internal structure and relevant parameter and can not assess point The problem of class model is to a variety of different confrontation sample algorithms robustness generated to resisting sample.

Moreover, the appraisal procedure of above-mentioned disaggregated model, generates a variety of different confrontation sample algorithms with disaggregated model To the defence capability of resisting sample as assessment dimension, based on the robustness of assessment dimension assessment disaggregated model, so as to More fully the robustness of disaggregated model is assessed.

In one embodiment, step S210 can be specifically included:

Determine the corresponding confrontation noise intensity of confidence similarity；Fight noise intensity be in resisting sample to antimierophonic Intensity；Each confidence similarity is grouped, multiple same intensity similarity set are obtained；It is set with each in intensity similarity set Believe that similarity corresponds to the same confrontation noise intensity；The each confidence calculated separately in multiple same intensity similarity set is similar The average value of degree defends assessed value as antagonistic intensity；Generate assessment result；Assessment result includes antagonistic intensity defence assessment Value.

It wherein, can be the collection of the corresponding identical confidence similarity of confrontation noise intensity with intensity similarity set It closes.

Wherein, antagonistic intensity defence assessed value is the confrontation that will be generated according to confrontation noise intensity for assessing disaggregated model Sample predictions are the numerical value of the ability of correct sample class.Antagonistic intensity defence assessed value can be used for assessing disaggregated model resistance The ability of the interference to resisting sample of a variety of different confrontation noise intensities.

In the specific implementation, server 110 can determine the corresponding confrontation noise intensity of each confidence similarity, set each Letter similarity is grouped according to confrontation noise intensity, obtains multiple groups with intensity similarity set.It is similar with intensity for each group Each confidence similarity summation in degree set, by the sum of confidence similarity each in every group of set divided by each in the group set The quantity of confidence similarity obtains the group with the average value of confidence similarity each in intensity similarity set, which is Assessed value is defendd for the antagonistic intensity of disaggregated model.For the average value that each group is calculated with intensity similarity set, as Disaggregated model defends assessed value for the antagonistic intensity of different confrontation noise intensities.

For example, with reference to Fig. 4, confrontation sample intensity evaluation module in service valuation module can will be to resisting sample { X`<sub>11</sub>, X`₁₂…X`<sub>1j</sub>…X`_i1, X`<sub>i2</sub>…X`_ijCorresponding confidence similarity { Sim₁₁₁, Sim₁₂₁…Sim_1j1…Sim₁₁₂, Sim₁₂₂…Sim_1j2, Sim_i1k, Sim_i2k…Sim_ijk, according to confrontation noise intensity { ε₁、ε₂…ε_kBe grouped, obtain multiple groups With intensity similarity set.Each group is calculated separately with the average value of the confidence similarity of intensity similarity setObtain each confrontation noise intensity { ε₁、ε₂…ε_kCorresponding to antagonistic intensity defend assessed value

In practical application, assessment result can also include indicating risk.Specifically, a defence capability threshold can be preset Value generates indicating risk, then to prompt user when the antagonistic intensity defence assessed value being calculated is lower than the defence capability threshold value Disaggregated model is higher by the risk that the confrontation sample predictions of some confrontation noise intensity are error sample classification.

The appraisal procedure of above-mentioned disaggregated model, by calculating each group with the flat of the confidence similarity in intensity similarity set Mean value defends assessed value as antagonistic intensity, solves due to that can not learn internal structure and relevant parameter and can not assess point Class model to a variety of different antagonistic intensities defence assessed value to the robustness of resisting sample the problem of.

Moreover, a variety of different antagonistic intensities are defendd assessed value with disaggregated model by the appraisal procedure of above-mentioned disaggregated model To the defence capability of resisting sample as assessment dimension, based on the robustness of assessment dimension assessment disaggregated model, so as to More fully the robustness of disaggregated model is assessed.

In one embodiment, above-mentioned appraisal procedure can be with further include:

Determine the authentic specimen classification of normal sample；By the sample class of the authentic specimen classification of normal sample and prediction result It is not matched；Statistical forecast accurate quantity；Predict that accurate quantity is authentic specimen categorical match in the normal sample of sample class This quantity；The ratio for calculating the total amount of prediction accurate quantity and normal sample, obtains predictablity rate.

For example, with reference to Fig. 4, normal sample evaluation module in service valuation module, according to the sample label of normal sample The authentic specimen classification for determining normal sample matches authentic specimen classification with the sample class in prediction result, if Matching shows that prediction is correct, adds 1 to prediction right amount.So analogize, finally obtains prediction right amount M, i.e. authentic specimen classification It is matched with the quantity of the normal sample of the sample class of prediction result.Calculate the total amount N of prediction right amount M and normal sample_e's Ratio obtains predictablity rate.

The appraisal procedure of above-mentioned disaggregated model, by being combined on the basis of disaggregated model is to robustness to resisting sample point The predictablity rate of class model, so as to more fully assess disaggregated model.

In one embodiment, sample to be tested may include image pattern, video sample, audio sample, in samples of text At least one.

In the specific implementation, the appraisal procedure of the application, can be applied not only to commenting for the disaggregated model for being directed to image pattern Estimate, the assessment of the disaggregated model for video sample, audio sample or samples of text can also be applied to.Correspondingly, it is directed to Different disaggregated models, sample to be tested can be image pattern, video sample, audio sample or samples of text.For example, being directed to Unpiloted disaggregated model, sample to be tested can be video sample.

In one embodiment, as shown in fig. 6, providing a kind of appraisal procedure of disaggregated model, the present embodiment mainly with The user terminal 120 that this method is applied in above-mentioned Fig. 1 comes for example, the appraisal procedure of the disaggregated model specifically includes following step It is rapid:

S602 sends service valuation and requests to server；Server extremely divides for obtaining sample to be tested, input sample to be tested Class server-side obtains classified service end and passes through the prediction result that disaggregated model is exported；Prediction result includes sample class and right The forecast confidence answered；Sample to be tested includes normal sample and corresponding to resisting sample；Server is also used to obtain normal sample Confidence level vector sum to the confidence level vector of resisting sample；Confidence level vector is by the corresponding prediction confidence of multiple sample class Degree composition；According to normal sample and confidence similarity is determined to the respective confidence level vector of resisting sample, and according to confidence similarity Obtain assessment result；

S604, the assessment result of display server feedback；The disaggregated model that assessment result is used to assess classified service end supports The anti-robustness to resisting sample.

In the specific implementation, user can obtain the service provision interface at classified service end 130 by user terminal 120, then According to the service provision interface, service valuation request is initiated to server 110.Server 110 can according to service provision interface, Sample to be tested is input to the disaggregated model at classified service end 130, classified service end 130 by disaggregated model to sample to be tested into Row classification, exports prediction result.Server 110 is assessed according to the prediction result that classified service end 130 exports, and assessment is tied Fruit feeds back to user terminal 120.

Server 110 has been described in the above-described embodiments according to the detailed process that prediction result exports assessment result, This is repeated no more.

The appraisal procedure of above-mentioned disaggregated model, by initiating service valuation request to server, server is in response to asking It asks, is input to normal sample and to resisting sample the disaggregated model at classified service end, obtain the prediction result of disaggregated model output, According to the corresponding forecast confidence of sample class multiple in prediction result, normal sample is obtained and to the respective confidence level of resisting sample Vector obtains confidence similarity, according to confidence similarity according to normal sample and to the respective confidence level vector of resisting sample The disaggregated model for assessing classified service end resists the robustness of the interference to resisting sample, and need not be dependent on point at classified service end The internal structure and relevant parameter of class model.Therefore, thinking of the above-mentioned appraisal procedure based on Black-box Testing utilizes disaggregated model institute The forecast confidence of offer obtains the suffered annoyance level to resisting sample of the disaggregated model that can reflect out classified service end Confidence similarity, and according to the disaggregated model at the confidence similarity assessment classified service end, it solves due to that can not learn inside Structure and relevant parameter and the problem of disaggregated model at classified service end is to robustness to resisting sample can not be assessed.

Moreover, user can assess the disaggregated model at classified service end to the robustness to resisting sample according to assessment result, Effective reference information is provided for the suitable disaggregated model of user's selection.

It should be understood that although each step in the flow chart of Fig. 2, Fig. 5 and Fig. 6 is successively shown according to the instruction of arrow Show, but these steps are not that the inevitable sequence according to arrow instruction successively executes.Unless expressly state otherwise herein, this There is no stringent sequences to limit for the execution of a little steps, these steps can execute in other order.Moreover, Fig. 2, Fig. 5 and At least part step in Fig. 6 may include that perhaps these sub-steps of multiple stages or stage be not necessarily for multiple sub-steps It is so to execute completion in synchronization, but can execute at different times, these sub-steps or stage execute sequence Also it is not necessarily and successively carries out, but can be at least part of the sub-step or stage of other steps or other steps It executes in turn or alternately.

In one embodiment, as shown in fig. 7, providing a kind of assessment device 700 of disaggregated model, comprising:

Sample acquisition module 702, for obtaining sample to be tested；Sample to be tested includes normal sample and corresponding confrontation sample This；

Sample input module 704 obtains the prediction knot of disaggregated model output for sample to be tested to be input to disaggregated model Fruit；Prediction result includes sample class and corresponding forecast confidence；

Vector obtains module 706, for obtaining the confidence level vector of normal sample and to the confidence level vector of resisting sample； Confidence level vector is made of the corresponding forecast confidence of multiple sample class；

Similarity determining module 708, for, with to the respective confidence level vector of resisting sample, determining confidence according to normal sample Similarity；

Evaluation module 710, for obtaining the assessment result of disaggregated model according to confidence similarity.

The assessment device of above-mentioned disaggregated model is obtained by being input to disaggregated model by normal sample and to resisting sample The prediction result of disaggregated model output obtains normal sample according to the corresponding forecast confidence of sample class multiple in prediction result Originally and confidence is obtained according to normal sample and to resisting sample respective confidence level vector to the respective confidence level vector of resisting sample Similarity, the robustness for the interference that disaggregated model is resisted to resisting sample can be assessed according to confidence similarity, and need not be depended on The internal structure and relevant parameter of disaggregated model.Therefore, thinking of the above-mentioned appraisal procedure based on Black-box Testing, utilizes disaggregated model Provided forecast confidence, obtain can reflect out it is similar to the confidence of the annoyance level of resisting sample suffered by disaggregated model Degree, and according to the confidence similarity assessment disaggregated model, solve due to that can not learn internal structure and relevant parameter and can not Assess the problem of disaggregated model is to robustness to resisting sample.

In one embodiment, evaluation module 710 is specifically used for:

The average value for calculating each confidence similarity defends assessed value as to resisting sample；Assessed value is defendd to resisting sample For the numerical value that will fight the ability that sample predictions are correct sample class for assessing disaggregated model；Generate assessment result；Assessment It as a result include that assessed value is defendd to resisting sample.

In one embodiment, evaluation module 710 is specifically used for:

Determine the corresponding confrontation sample algorithm of confidence similarity and confrontation sample pattern；It is described to resisting sample according to described right Resisting sample algorithm and the confrontation sample pattern are generated；Each confidence similarity is grouped, multiple same algorithm similarities are obtained Set；Correspond to the same confrontation sample algorithm with each confidence similarity in algorithm similarity set；It determines respectively multiple With the minimum similarity degree in algorithm similarity set；The Target Countermeasure sample mould of multiple same algorithm similarity set is determined respectively Type；Target Countermeasure sample pattern is confrontation sample pattern corresponding with minimum similarity degree；Count multiple Target Countermeasure sample patterns Frequency of occurrence；Frequency of occurrence is the quantity of the same algorithm similarity set corresponding to the same Target Countermeasure sample pattern；It is raw Pairs of resisting sample model profile, as assessment result；Fighting sample pattern distribution includes each Target Countermeasure sample pattern and right The frequency of occurrence answered.

In one embodiment, further includes:

Maximum times determining module, in the frequency of occurrence of each Target Countermeasure sample pattern, determining maximum appearance Number；

Mean value computation module, the average value of the frequency of occurrence for calculating each Target Countermeasure sample pattern, is occurred Number mean value；

Difference calculating module, for calculating the number difference of maximum frequency of occurrence Yu frequency of occurrence mean value；

Cue module generates disclosure risk prompt for being greater than preset threshold when number difference.

In one embodiment, evaluation module 710 is specifically used for:

The average value for calculating separately each confidence similarity in multiple same algorithm similarity set, it is anti-as confrontation algorithm Imperial assessed value；Generate assessment result；Assessment result includes that confrontation algorithm defends assessed value.

In one embodiment, evaluation module 710 is specifically used for:

Determine the corresponding confrontation noise intensity of confidence similarity；The confrontation noise intensity is pair in resisting sample Antimierophonic intensity；Each confidence similarity is grouped, multiple same intensity similarity set are obtained；With in intensity similarity set Each confidence similarity correspond to the same confrontation noise intensity；It calculates separately each in multiple same intensity similarity set The average value of confidence similarity defends assessed value as antagonistic intensity；Generate assessment result；Assessment result includes that antagonistic intensity is anti- Imperial assessed value.

In one embodiment, device further include:

True classification obtains module, for determining the authentic specimen classification of normal sample；

Matching module, for matching the authentic specimen classification of normal sample with the sample class of prediction result；

Quantity statistics module is used for statistical forecast accurate quantity；Predict that accurate quantity is authentic specimen categorical match in sample The quantity of the normal sample of this classification；

Ratio calculation module, the ratio of the total amount for calculating prediction accurate quantity and normal sample, it is accurate to obtain prediction Rate.

In one embodiment, similarity determining module 708 is specifically used for:

It calculates the confidence level vector of normal sample and to the vector cosine value between the confidence level vector of resisting sample, is set Believe similarity.

In one embodiment, sample acquisition module 702 is specifically used for:

It chooses sample and generates parameter；Choosing sample generation parameter further comprises: in N_mIn a confrontation sample pattern, choose Target Countermeasure sample pattern out, and, in N_aIn a confrontation sample algorithm, Target Countermeasure sample algorithm is selected, and, most Big confrontation noise intensity ε_maxNoise intensity ε is fought with minimum_minBetween, select Target Countermeasure noise intensity；By Target Countermeasure Sample pattern, Target Countermeasure sample algorithm and Target Countermeasure noise intensity generate parameter as sample；

Obtain the model parameter of Target Countermeasure sample pattern；

Pass through the original sample noise of Target Countermeasure sample algorithm computation model parameter；

Using the noise intensity of Target Countermeasure noise intensity adjustment original sample noise, it is adjusted rear sample noise；

Sample noise after adjustment is superimposed to the normal sample, is obtained initially to resisting sample；

The Target Countermeasure sample pattern initially will be input to resisting sample, obtains the pre- of Target Countermeasure sample pattern output Survey result；

When prediction result mistake, initially resisting sample will be used as to resisting sample, and be back to and choose sample generation parameter Step, until obtaining the N of normal sample_gIt is a to resisting sample；Wherein, N_g=N_m*N_a*(ε_max-ε_min+1)。

In one embodiment, sample to be tested include image pattern, video sample, audio sample, in samples of text extremely Few one kind.

In one embodiment, as shown in figure 8, providing a kind of assessment device 800 of disaggregated model, comprising:

Sending module 802 is requested for sending service valuation to server；Server is inputted for obtaining sample to be tested Sample to be tested obtains classified service end and passes through the prediction result that disaggregated model is exported to classified service end；Prediction result includes Sample class and corresponding forecast confidence；Sample to be tested includes normal sample and corresponding to resisting sample；Server is also used to Obtain the confidence level vector described in the confidence level vector sum of normal sample to resisting sample；Confidence level vector is by multiple sample class point Not corresponding forecast confidence composition；According to normal sample and confidence similarity is determined to the respective confidence level vector of resisting sample, And assessment result is obtained according to confidence similarity；

Display module 804, the assessment result for display server feedback；Assessment result is for assessing classified service end Disaggregated model resists the robustness to resisting sample.

Fig. 9 shows the internal structure chart of computer equipment in one embodiment.The computer equipment specifically can be Fig. 1 In server 110 or user terminal 120.As shown in figure 9, it includes total by system that the computer equipment, which includes the computer equipment, Processor, memory, network interface, input unit and the display screen of line connection.Wherein, memory includes that non-volatile memories are situated between Matter and built-in storage.The non-volatile memory medium of the computer equipment is stored with operating system, can also be stored with computer journey Sequence when the computer program is executed by processor, may make processor to realize the appraisal procedure of disaggregated model.In the built-in storage Computer program can also be stored, when which is executed by processor, processor may make to execute commenting for disaggregated model Estimate method.The display screen of computer equipment can be liquid crystal display or electric ink display screen, the input of computer equipment Device can be the touch layer covered on display screen, be also possible to the key being arranged on computer equipment shell, trace ball or touching Plate is controlled, can also be external keyboard, Trackpad or mouse etc..

It will be understood by those skilled in the art that structure shown in Fig. 9, only part relevant to application scheme is tied The block diagram of structure does not constitute the restriction for the computer equipment being applied thereon to application scheme, specific computer equipment It may include perhaps combining certain components or with different component layouts than more or fewer components as shown in the figure.

In one embodiment, the assessment device of disaggregated model provided by the present application can be implemented as a kind of computer program Form, computer program can run in computer equipment as shown in Figure 9.Group can be stored in the memory of computer equipment At each program module of the assessment device of the disaggregated model, for example, sample acquisition module shown in Fig. 7 702, sample input mould Block 704, vector obtain module 706, similarity determining module 708 and evaluation module 710.The computer that each program module is constituted Step in the appraisal procedure for the disaggregated model that program makes processor execute each embodiment of the application described in this specification Suddenly.

For example, computer equipment shown in Fig. 9 can pass through the sample in the assessment device of disaggregated model as shown in Figure 7 It obtains module 702 and executes acquisition sample to be tested.Computer equipment can be executed by sample input module 704 and input sample to be tested To the disaggregated model, the prediction result of disaggregated model output is obtained.

In one embodiment, a kind of computer equipment, including memory and processor are provided, memory is stored with meter Calculation machine program, when computer program is executed by processor, so that the step of processor executes the appraisal procedure of above-mentioned disaggregated model. The step of appraisal procedure of disaggregated model can be the step in the appraisal procedure of the disaggregated model of above-mentioned each embodiment herein.

In one embodiment, a kind of computer readable storage medium is provided, computer program, computer journey are stored with When sequence is executed by processor, so that the step of processor executes the appraisal procedure of above-mentioned disaggregated model.Disaggregated model is commented herein The step of estimating method can be the step in the appraisal procedure of the disaggregated model of above-mentioned each embodiment.

Those of ordinary skill in the art will appreciate that realizing all or part of the process in above-described embodiment method, being can be with Relevant hardware is instructed to complete by computer program, the program can be stored in a non-volatile computer and can be read In storage medium, the program is when being executed, it may include such as the process of the embodiment of above-mentioned each method.Wherein, provided herein Each embodiment used in any reference to memory, storage, database or other media, may each comprise non-volatile And/or volatile memory.Nonvolatile memory may include that read-only memory (ROM), programming ROM (PROM), electricity can be compiled Journey ROM (EPROM), electrically erasable ROM (EEPROM) or flash memory.Volatile memory may include random access memory (RAM) or external cache.By way of illustration and not limitation, RAM is available in many forms, such as static state RAM (SRAM), dynamic ram (DRAM), synchronous dram (SDRAM), double data rate sdram (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronization link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) directly RAM (RDRAM), straight Connect memory bus dynamic ram (DRDRAM) and memory bus dynamic ram (RDRAM) etc..

Each technical characteristic of above embodiments can be combined arbitrarily, for simplicity of description, not to above-described embodiment In each technical characteristic it is all possible combination be all described, as long as however, the combination of these technical characteristics be not present lance Shield all should be considered as described in this specification.

The several embodiments of the application above described embodiment only expresses, the description thereof is more specific and detailed, but simultaneously The limitation to the application the scope of the patents therefore cannot be interpreted as.

Claims (13)

1. a kind of appraisal procedure of disaggregated model characterized by comprising

Obtain sample to be tested；The sample to be tested includes normal sample and corresponding to resisting sample；

The sample to be tested is input to the disaggregated model, obtains the prediction result of the disaggregated model output；The prediction It as a result include sample class and corresponding forecast confidence；

Obtain the confidence level vector of the normal sample and the confidence level vector to resisting sample；The confidence level vector is by multiple The corresponding forecast confidence composition of the sample class；

According to the normal sample and described to the respective confidence level vector of resisting sample, confidence similarity is determined；

According to the confidence similarity, the assessment result of the disaggregated model is obtained.

2. obtaining the classification the method according to claim 1, wherein described according to the confidence similarity The assessment result of model, comprising:

The average value for calculating each confidence similarity defends assessed value as to resisting sample；

Generate the assessment result；The assessment result includes described to resisting sample defence assessed value.

3. obtaining the classification the method according to claim 1, wherein described according to the confidence similarity The assessment result of model, comprising:

Determine the corresponding confrontation sample algorithm of the confidence similarity and confrontation sample pattern；It is described to resisting sample according to described right Resisting sample algorithm and the confrontation sample pattern are generated；

Each confidence similarity is grouped, multiple same algorithm similarity set are obtained；In the same algorithm similarity set Each confidence similarity correspond to the same confrontation sample algorithm；

The minimum similarity degree in multiple same algorithm similarity set is determined respectively；

The Target Countermeasure sample pattern of multiple same algorithm similarity set is determined respectively；The Target Countermeasure sample pattern is Confrontation sample pattern corresponding with the minimum similarity degree；

Count the frequency of occurrence of multiple Target Countermeasure sample patterns；The frequency of occurrence is corresponding to the same target Fight the quantity of multiple same algorithm similarity set of sample pattern；

Confrontation sample pattern distribution is generated, as the assessment result；The confrontation sample pattern distribution includes each mesh Mark confrontation sample pattern and corresponding frequency of occurrence.

4. according to the method described in claim 3, it is characterized by further comprising:

In the frequency of occurrence of each Target Countermeasure sample pattern, maximum frequency of occurrence is determined；

The average value for calculating the frequency of occurrence of each Target Countermeasure sample pattern, obtains frequency of occurrence mean value；

Calculate the number difference of the maximum frequency of occurrence and the frequency of occurrence mean value；

When the number difference is greater than preset threshold, generation disclosure risk prompt.

5. according to the method described in claim 3, obtaining the classification it is characterized in that, described according to the confidence similarity The assessment result of model, comprising:

The average value for calculating separately each confidence similarity in multiple same algorithm similarity set, it is anti-as confrontation algorithm Imperial assessed value；

Generate the assessment result；The assessment result includes the confrontation algorithm defence assessed value.

6. obtaining the classification the method according to claim 1, wherein described according to the confidence similarity The assessment result of model, comprising:

Determine the corresponding confrontation noise intensity of the confidence similarity；The confrontation noise intensity is pair in resisting sample Antimierophonic intensity；

Each confidence similarity is grouped, multiple same intensity similarity set are obtained；In the same intensity similarity set Each confidence similarity correspond to the same confrontation noise intensity；

The average value for calculating separately each confidence similarity in multiple same intensity similarity set, it is anti-as antagonistic intensity Imperial assessed value；

Generate the assessment result；The assessment result includes the antagonistic intensity defence assessed value.

7. the method according to claim 1, wherein the method also includes:

Determine the authentic specimen classification of the normal sample；

The authentic specimen classification of the normal sample is matched with the sample class of the prediction result；

Statistical forecast accurate quantity；The prediction accurate quantity be the authentic specimen categorical match in the sample class just The quantity of normal sample；

The ratio for calculating the total amount of the prediction accurate quantity and the normal sample, obtains predictablity rate.

8. the method according to claim 1, wherein it is described according to the normal sample with it is described each to resisting sample From confidence level vector, determine confidence similarity, comprising:

The vector cosine value between the confidence level vector of the normal sample and the confidence level vector to resisting sample is calculated, is obtained To the confidence similarity.

9. the method according to claim 1, wherein the acquisition sample to be tested, comprising:

It chooses sample and generates parameter；The selection sample generates parameter: in N_mIn a confrontation sample pattern, choose Target Countermeasure sample pattern out, and, in N_aIn a confrontation sample algorithm, Target Countermeasure sample algorithm is selected, and, most Big confrontation noise intensity ε_maxNoise intensity ε is fought with minimum_minBetween, select Target Countermeasure noise intensity；By the target Sample pattern, the Target Countermeasure sample algorithm and the Target Countermeasure noise intensity are fought as the sample and generates parameter；

Obtain the model parameter of the Target Countermeasure sample pattern；

The original sample noise of the model parameter is calculated by the Target Countermeasure sample algorithm；

The noise intensity that the original sample noise is adjusted using the Target Countermeasure noise intensity is adjusted rear sample and made an uproar Sound；

Sample noise after the adjustment is superimposed to the normal sample, is obtained initially to resisting sample；

The Target Countermeasure sample pattern initially is input to resisting sample by described, obtains the Target Countermeasure sample pattern output Prediction result；

When the prediction result mistake, using it is described initially to resisting sample as described to resisting sample, and be back to the selection sample The step of this generation parameter, until obtaining the N of the normal sample_gIt is a to resisting sample；Wherein, N_g=N_m*N_a*(ε_max-ε_min+ 1)。

10. the method according to claim 1, wherein the sample to be tested include image pattern, video sample, At least one of audio sample, samples of text.

11. a kind of assessment device of disaggregated model characterized by comprising

Sample acquisition module, for obtaining sample to be tested；The sample to be tested includes normal sample and corresponding to resisting sample；

Sample input module obtains the disaggregated model output for the sample to be tested to be input to the disaggregated model Prediction result；The prediction result includes sample class and corresponding forecast confidence；

Vector obtains module, for obtaining the confidence level vector of the normal sample and to the confidence level vector of resisting sample；Institute Confidence level vector is stated to be made of the corresponding forecast confidence of multiple sample class；

Similarity determining module is used for according to the normal sample with described to the respective confidence level vector of resisting sample, and determination is set Believe similarity；

Evaluation module, for obtaining the assessment result of the disaggregated model according to the confidence similarity.

12. a kind of computer readable storage medium is stored with computer program, when the computer program is executed by processor, So that the processor is executed such as the step of any one of claims 1 to 10 the method.

13. a kind of computer equipment, including memory and processor, the memory is stored with computer program, the calculating When machine program is executed by the processor, so that the processor is executed such as any one of claims 1 to 10 the method Step.