WO2021135546A1 - 深度神经网络的解释方法、装置、终端及存储介质 - Google Patents

深度神经网络的解释方法、装置、终端及存储介质 Download PDF

Info

Publication number
WO2021135546A1
WO2021135546A1 PCT/CN2020/123196 CN2020123196W WO2021135546A1 WO 2021135546 A1 WO2021135546 A1 WO 2021135546A1 CN 2020123196 W CN2020123196 W CN 2020123196W WO 2021135546 A1 WO2021135546 A1 WO 2021135546A1
Authority
WO
WIPO (PCT)
Prior art keywords
explanatory
category
factor
prediction
target
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2020/123196
Other languages
English (en)
French (fr)
Inventor
陈筱
周细文
庄伯金
王少军
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ping An Technology Shenzhen Co Ltd
Original Assignee
Ping An Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ping An Technology Shenzhen Co Ltd filed Critical Ping An Technology Shenzhen Co Ltd
Publication of WO2021135546A1 publication Critical patent/WO2021135546A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent

Definitions

  • This application relates to the field of deep learning technology of artificial intelligence, and in particular to an interpretation method, device, terminal and storage medium of a deep neural network.
  • Neural network algorithms have been a research hotspot that has emerged in the field of artificial intelligence since the 1980s. It abstracts the neuron network from the perspective of information processing, establishes a certain simple model, and composes different networks according to different connection methods. It has an autonomous learning function, which can slowly learn to recognize and predict through training; Lenovo storage function, with high algorithm robustness; high parallelism, has the ability to find optimized solutions at a high speed, and can quickly find optimized solutions for complex problems with big data; It has strong plasticity and can fully approximate arbitrarily complex nonlinear relationships; it has strong information synthesis capabilities, can process quantitative and qualitative information at the same time, and is suitable for multi-information fusion and multimedia technology. In recent years, the research work of neural networks has continued to deepen, and great progress has been made.
  • Feature visualization is a commonly used interpretation method of neural networks. It searches for the influence of each area in the model input on the model prediction and visualizes it. It can explain the focus of the model to a certain extent. However, in practical applications, the inventor found that only It is difficult to quantitatively explain the model through feature visualization, nor can it reflect the main factors that affect the judgment of the model, and it is impossible to explain the prediction results of the model based on which factors.
  • This application provides a deep neural network interpretation method, device, terminal, and storage medium to solve the problem of difficulty in determining factors that affect the prediction result of the deep neural network.
  • this application provides a deep neural network interpretation method, which includes: inputting multiple data into the trained deep neural network model to obtain the first prediction category corresponding to each data; when the first prediction category is When the target category is preset, the target data corresponding to the first prediction category is input into the explanatory factor prediction model to obtain the explanatory factor corresponding to the preset target category.
  • the explanatory factor prediction model is based on the preset target number and preset target category
  • the first predictive value of is obtained through training; after counting the number of occurrences of each explanatory factor and sorting, the number of explanatory factors with the highest ranking target is selected as the target explanatory factor of the preset target category.
  • this application also provides a deep neural network interpretation device, including: a first input module for inputting multiple data into the trained deep neural network model to obtain the first data corresponding to each data. Prediction category; the second input module is used to input the target data corresponding to the first prediction category into the explanatory factor prediction model when the first prediction category is the preset target category to obtain the explanatory factor corresponding to the preset target category,
  • the explanatory factor prediction model is trained according to the preset number of targets and the first predicted value of the preset target category;
  • the filtering module is used to count the number of occurrences of each explanatory factor and sort it, and then filter the number of the highest-ranked explanatory factors As the target explanatory factor of the preset target category.
  • the present application also provides a terminal, including a memory, a processor, and a program file stored in the memory and capable of running on the processor, wherein the processor implements the following steps when executing the program file: input multiple Transfer the data to the trained deep neural network model to obtain the first prediction category corresponding to each data; when the first prediction category is the preset target category, input the target data corresponding to the first prediction category into the explanatory factor prediction model , In order to obtain the explanatory factor corresponding to the preset target category, the explanatory factor prediction model is trained according to the preset target number and the first predicted value of the preset target category; after counting the number of occurrences of each explanatory factor and sorting, the ranking is filtered The highest number of target explanatory factors are used as the target explanatory factors of the preset target category.
  • this application also provides a storage medium, in which a program file capable of realizing the interpretation method of the deep neural network is stored.
  • the program file is executed by the processor, the following steps are implemented: input multiple data to the trained In the deep neural network model, the first prediction category corresponding to each data is obtained; when the first prediction category is the preset target category, the target data corresponding to the first prediction category is input into the explanatory factor prediction model to obtain the preset
  • the explanatory factor corresponding to the target category the explanatory factor prediction model is trained according to the preset number of targets and the first predicted value of the preset target category; after counting the number of occurrences of each explanatory factor and sorting, the number of targets with the highest ranking is selected
  • the explanatory factor is used as the target explanatory factor of the preset target category.
  • the interpretation method of the deep neural network proposed in this application is to input the target data corresponding to the first preset category into the trained explanatory factor prediction model when the first prediction category output by the deep neural network is the preset target category.
  • Obtain the explanatory factors corresponding to the preset target category enter different target data to obtain multiple sets of explanatory factors, and then count the number of occurrences of each explanatory factor, and filter out the highest-ranking target number and explanatory factors as the preset target
  • the target explanatory factor of the category, the target explanatory factor will be used as the main influencing factor of the deep neural network to classify the target data output result into the preset target category, which is convenient for users to find the influencing factors that affect the classification and prediction of the deep neural network model.
  • the deep neural network interpretation method of this application can also be applied to smart government affairs/smart city management/smart communities/smart security/smart logistics/smart medical treatment/smart education/smart environmental protection/smart transportation scenarios to promote the construction of smart cities .
  • FIG. 1 is a schematic flowchart of a first embodiment of a method for explaining a deep neural network according to this application;
  • FIG. 2 is a schematic diagram of the training process of the explanatory factor prediction model in an embodiment of the explanatory method of the deep neural network of this application;
  • Figure 3 is a schematic diagram of the structure of a deep neural network
  • FIG. 4 is a schematic flowchart of a second embodiment of a method for explaining a deep neural network according to this application;
  • FIG. 5 is a schematic diagram of functional modules of an embodiment of an explanation device for a deep neural network according to this application;
  • FIG. 6 is a schematic structural diagram of an embodiment of a terminal of this application.
  • FIG. 7 is a schematic structural diagram of an embodiment of a storage medium of this application.
  • first”, “second”, and “third” in this application are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined with “first”, “second”, and “third” may explicitly or implicitly include at least one of the features.
  • "a plurality of” means at least two, such as two, three, etc., unless otherwise specifically defined. All directional indicators (such as up, down, left, right, front, back%) in the embodiments of this application are only used to explain the relative positional relationship between the components in a specific posture (as shown in the drawings) , Movement status, etc., if the specific posture changes, the directional indication will also change accordingly.
  • FIG. 1 is a schematic flowchart of a deep neural network interpretation method according to the first embodiment of the present application. It should be noted that if there is substantially the same result, the method of the present application is not limited to the sequence of the process shown in FIG. 1. As shown in Figure 1, the method includes steps:
  • Step S101 Input a plurality of data into the trained deep neural network model, and obtain the first prediction category corresponding to each data.
  • a deep neural network is a neural network algorithm that is widely used in image processing, pattern recognition, intelligent robots, automatic control, predictive estimation, biology, medicine, economics and other fields. It usually includes an input layer, a hidden layer, and an output. Please refer to Figure 2 together. Generally, the first layer (input layer) is the input layer, the last layer (output layer) is the output layer, and the number of layers in the middle (hidden layer) are all hidden layers. It is fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer, and there can be one or more neurons in the output layer, and each neuron corresponds to an output result.
  • input data into the deep neural network model to obtain the predicted value of each category, and then use the category with the largest predicted value as the output result of the deep neural network for example, take animal classification as an example
  • a trained deep neural network to identify an animal in a picture, it can obtain the predicted value when the animal is a cat, dog, cow, sheep, etc., and then select the animal with the largest predicted value as the output result.
  • Step S102 When the first prediction category is the preset target category, input the target data corresponding to the first prediction category into the explanatory factor prediction model to obtain the explanatory factor corresponding to the preset target category.
  • the explanatory factor prediction model is trained according to the preset number of targets and the first prediction value of the preset target category.
  • the target number refers to the number of target explanatory factors, which can be preset by the user;
  • the preset target category is a category preset by the user;
  • the explanatory factor refers to the factors that affect the judgment of the deep neural network model, for example, through deep neural network
  • the network model recognizes what animal the animal in a picture is.
  • the factors that affect the judgment of the deep neural network model can include the number of animals’ feet, hair growth, eye size, etc. These factors may cause the output of the deep neural network model. influences.
  • each category is affected by different explanatory factors.
  • explanatory factors for the predicted category can include full-body hair, walking on four legs, large eyes, beards, hidden paws, and pads on the soles of the feet, etc.
  • the explanatory factors for the predicted category of "dog” can include full-body hair, walking on four legs, likes Squatting, showing paws, no pads on the soles of the feet, etc. This application aims to determine the main factors that affect the output results when using a deep neural network.
  • step S102 whenever the first prediction category is a preset target category, the target data corresponding to the first prediction category is input into the explanatory factor prediction model to obtain an explanatory factor corresponding to the preset target category.
  • the explanatory factors corresponding to the multiple sets of preset target categories may be the same or different, for example: take animal classification as an example ,
  • the prediction results of the two target data of A and B are cats, but the target data of A is predicted by big eyes, four legs, and full-body hair, while the target data of B is predicted by four legs, full-body hair, and hair with markings.
  • the explanatory factors of the two are only the same for the four legs and the whole body hair.
  • step S102 specifically includes: inputting the target data corresponding to the preset target category into the explanatory factor prediction model corresponding to the preset target category to obtain the second predicted value of each explanatory factor;
  • the explanatory factor value of the threshold is set to 1, and used as the explanatory factor corresponding to the preset target category.
  • the target data is input into the explanatory factor prediction model to obtain the second predicted value of each explanatory factor, and then according to the magnitude of the second predicted value, the value of the explanatory factor whose second predicted value exceeds the preset threshold is set as 1. The values of the remaining explanatory factors are set to 0.
  • step S102 the target data corresponding to the first prediction class is input Before the explanatory factor prediction model, it also includes: obtaining the explanatory factor prediction model corresponding to the preset target category.
  • Step S103 After counting the number of occurrences of each explanatory factor and sorting, the explanatory factors with the highest number of targets are selected as the target explanatory factors of the preset target category.
  • step S103 by counting the number of occurrences of each explanatory factor among the explanatory factors corresponding to the multiple target data, and sorting, the explanatory factors with the highest number of targets are selected from the explanatory factors as the target explanatory factors of the preset target category. For example, taking animal classification as an example, the total number of explanatory factors is 100. Through the above explanatory factor prediction model, the 10 most relevant explanatory factors that classify animals as cats are selected from the 100 explanatory factors as the target explanatory factors.
  • the target data corresponding to the first preset category is input into the trained explanatory factor prediction model to obtain the interpretation corresponding to the preset target category Factors, by entering different target data, multiple sets of explanatory factors are obtained, and then the number of occurrences of each explanatory factor is counted, and the number of explanatory factors with the highest ranking target is selected as the target explanatory factor of the preset target category.
  • the target The explanatory factor will be the main influencing factor for the deep neural network to classify the output result of the target data into the preset target category, which is convenient for users to find the influencing factors that affect the classification and prediction of the deep neural network model.
  • the deep neural network interpretation method of this application can also be applied to scenarios such as smart city management/smart community/smart security/smart logistics, thereby promoting the construction of smart cities.
  • the explanatory factor prediction model corresponding to each preset target category is obtained through pre-training.
  • the training steps of the explanatory factor prediction model specifically include:
  • Step S201 Input samples one by one into the trained deep neural network model, and obtain the prediction category and the first prediction value corresponding to each sample.
  • the output of a deep neural network model usually includes multiple prediction categories, each prediction category corresponds to a prediction value, and the result with the highest prediction value is used as the final output result.
  • the method further includes: acquiring at least one preset target category input by the user in advance, and the target quantity of the target interpretation factor corresponding to each preset target category.
  • the user can select the type of training required according to the needs, and the user can also set the target number of explanatory factors to be finally output.
  • the user can set the target number of target explanatory factors for cats in the animal classification to 10.
  • Step S202 When the predicted category is the preset target category, input the target sample corresponding to the predicted category into the explanatory factor prediction model to obtain the second predicted value of each explanatory factor.
  • Step S203 Construct an explanatory factor label according to the first predicted value, the second predicted value and the target quantity.
  • the step of constructing an explanatory factor label specifically includes:
  • ceil represents rounding up
  • m represents the number of targets
  • Sk represents the first predicted value
  • explanatory factor label is as follows:
  • q 1 , q 2 ,..., q w refer to the index of the w explanatory factors with the largest second predicted value.
  • m 10
  • Sk 80%
  • w 8 calculated according to the above formula.
  • input the target sample into the explanatory factor prediction model to get The second predicted value of the 30 explanatory factors, and then select the 8 explanatory factors with the highest second predicted value from them, set their value to 1, and set the value of the remaining 22 explanatory factors to 0 to obtain the predicted category The corresponding explanatory factor label.
  • the explanatory factor label of each predicted category is constructed based on the current first predicted value of the predicted category. Therefore, each time the target sample is input to obtain the predicted category, the explanatory factor label needs to be constructed once.
  • Step S204 update the explanatory factor prediction model based on the explanatory factor label and the second predicted value back propagation.
  • step S204 the method of back-propagating and updating the explanatory factor prediction model by comparing the explanatory factor label with the second predicted value of each explanatory factor is the same as the back-propagation updating process during deep neural network training.
  • Step S205 Repeat the above steps until the explanatory factor prediction model converges, and bind the converged explanatory factor prediction model to the preset target category.
  • the explanatory factor prediction model is repeatedly iteratively trained until the explanatory factor prediction model converges, stop training and converge the explanatory factor prediction model, and converge
  • the explanatory factor prediction model is bound to the preset target category.
  • the label of the explanatory factor is dynamically obtained by combining the first predicted value of the predicted category, and the explanatory factor prediction model is trained using the label, so that the explanatory factor corresponding to the preset target category will have a greater impact on the preset target category. Take the larger value of 1 for the sample, and take the smaller value of 0 for other types of samples.
  • FIG. 4 shows a schematic flow chart of the deep neural network interpretation method of the second embodiment of the present application. It should be noted that if there is substantially the same result, the method of the present application is not limited to the sequence of the process shown in FIG. 4. As shown in Figure 4, the method includes the steps:
  • Step S301 Obtain at least one preset target category input by the user in advance, and each preset target category corresponds to an explanatory factor prediction model pre-trained to a convergent state.
  • the preset target category can be pre-designated by the user, and the user can specify one or more preset target categories at a time, and each preset target category uses the explanatory factor prediction model to confirm the target explanatory factor.
  • the explanatory factor prediction model corresponding to each preset target category is obtained without interfering with each other.
  • Step S302 Input a plurality of data into the trained deep neural network model, and obtain the first prediction category corresponding to each data.
  • step S302 in FIG. 4 is similar to step S101 in FIG.
  • Step S303 When the first prediction category is the preset target category, input the target data corresponding to the first prediction category into the explanatory factor prediction model to obtain the explanatory factor corresponding to the preset target category.
  • step S303 in FIG. 4 is similar to step S102 in FIG.
  • Step S304 After counting the number of occurrences of each explanatory factor and sorting, the explanatory factors with the highest number of targets are selected as the target explanatory factors of the preset target category.
  • step S304 in FIG. 4 is similar to step S103 in FIG.
  • the preset target category can be specified by the user.
  • the user can determine the main influencing factors for the deep neural network model to output the preset target category, and the user can once Enter multiple preset target categories to help users quickly confirm the main influencing factors of each preset target category.
  • Fig. 5 shows a schematic diagram of functional modules of a deep neural network interpretation device according to an embodiment of the present application.
  • the interpretation device 10 of the deep neural network includes a first input module 11, a second input module 12 and a screening module 13.
  • the first input module 11 is used to input multiple data into the trained deep neural network model to obtain the first prediction category corresponding to each data;
  • the second input module 12 is used to input the target data corresponding to the first prediction category into the explanatory factor prediction model when the first prediction category is the preset target category to obtain the explanatory factor corresponding to the preset target category, the explanatory factor
  • the prediction model is trained according to the preset number of targets and the first prediction value of the preset target category;
  • the screening module 13 is used for counting the number of occurrences of each explanatory factor and sorting, and then screening the explanatory factors with the highest number of targets as the target explanatory factors of the preset target category.
  • the second input module 12 before the second input module 12 inputs the target data corresponding to the first prediction category to the explanatory factor prediction model, it further includes pre-training the explanatory factor prediction model, including: inputting samples one by one into the trained deep neural network model, Obtain the predicted category and the first predicted value corresponding to each sample; when the predicted category is the preset target category, input the target sample corresponding to the predicted category into the explanatory factor prediction model to obtain the second predicted value of each explanatory factor; Construct explanatory factor labels based on the first predicted value, second predicted value, and target number; update explanatory factor prediction models based on the explanatory factor labels and second predicted values back propagation; repeat the above steps until the explanatory factor prediction model converges and will converge The explanatory factor prediction model is bound to the preset target category.
  • the operation of constructing an explanatory factor label using the first predicted value, the second predicted value of each explanatory factor, and the target quantity can also be: calculating all explanatory factors with a value of 1 according to the first predicted value and the target quantity
  • the number of explanatory factors w, w ceil(m ⁇ S k ), where ceil means rounding up, m means the number of targets, and Sk means the first predicted value; according to the magnitude of the second predicted value of each explanatory factor, Set the value of the w explanatory factors with the largest second predicted value to 1, and set the values of the remaining explanatory factors to 0 to obtain the explanatory factor label.
  • the trained deep neural network model before the operation of inputting samples one by one to the trained deep neural network model, it further includes: obtaining at least one preset target category input by the user in advance, and the target quantity of the target explanatory factor corresponding to each preset target category.
  • the first input module 11 before the first input module 11 inputs multiple data into the trained deep neural network model, it is also used to obtain at least one preset target category input in advance by the user, and each preset target category corresponds to a pre-trained The explanatory factor prediction model to the convergence state.
  • the second input module 12 inputs the target data corresponding to the first prediction category into the explanatory factor prediction model to obtain the explanatory factor corresponding to the preset target category.
  • the operation may also be: setting the target corresponding to the preset target category
  • the data is input into the explanatory factor prediction model corresponding to the preset target category, and the second predicted value of each explanatory factor is obtained; the value of the explanatory factor whose second predicted value exceeds the preset threshold is taken as 1, and used as the preset target category The corresponding explanatory factor.
  • the method before the operation of inputting the target data corresponding to the first prediction category into the explanatory factor prediction model by the first input module 11, the method further includes: obtaining an explanatory factor prediction model corresponding to the preset target category.
  • FIG. 6 is a schematic structural diagram of a terminal according to an embodiment of the application.
  • the terminal 20 includes a processor 200 and a memory 201 coupled to the processor 200.
  • the memory 201 stores program instructions for implementing the deep neural network interpretation method described in any of the above embodiments.
  • the processor 200 is configured to execute the program instructions stored in the memory 201 to train to obtain the target explanatory factor of the preset target category.
  • the processor 200 may also be referred to as a CPU (Central Processing Unit, central processing unit).
  • the processor 200 may be an integrated circuit chip with signal processing capabilities.
  • the processor 200 may also be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component .
  • DSP digital signal processor
  • ASIC application specific integrated circuit
  • FPGA field programmable gate array
  • the general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like.
  • FIG. 7 is a schematic structural diagram of a storage medium according to an embodiment of the application.
  • the storage medium of the embodiment of the present application stores a program file 30 that can implement all the above methods.
  • the program file 30 can be stored in the above storage medium in the form of a software product, and includes several instructions to enable a computer device (which can A personal computer, a server, or a network device, etc.) or a processor (processor) executes all or part of the steps of the methods described in the various embodiments of the present application.
  • the aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disks or optical disks and other media that can store program codes. , Or terminal devices such as computers, servers, mobile phones, and tablets.
  • the computer-readable storage medium may be non-volatile or volatile
  • the disclosed terminal, device, and method may be implemented in other ways.
  • the device embodiments described above are merely illustrative, for example, the division of units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components can be combined or integrated. To another system, or some features can be ignored, or not implemented.
  • the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms.
  • the functional units in the various embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit.
  • the above-mentioned integrated unit can be implemented in the form of hardware or software functional unit. The above are only implementations of this application, and do not limit the scope of this application. Any equivalent structure or equivalent process transformation made using the content of the description and drawings of this application, or directly or indirectly applied to other related technical fields, The same reasoning is included in the scope of patent protection of this application.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Health & Medical Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Image Analysis (AREA)

Abstract

一种深度神经网络的解释方法、装置、终端及存储介质，涉及人工智能技术领域，具体公开了通过深度神经网络输出的结果从输入的多个数据中确定所有的目标数据，再将目标数据分别输入至解释因子预测模型中，得到每个目标数据对应的一组解释因子，再统计多组解释因子中每个解释因子的出现次数并排序后，筛选排名最高的目标数量个解释因子作为预设目标类别的目标解释因子。该方法通过预先训练好的解释因子预测模型得到预测结果的解释因子，以解释影响深度神经网络预测结果主要因素，其可应用于智慧政务/智慧城管/智慧社区/智慧安防/智慧物流/智慧医疗/智慧教育/智慧环保/智慧交通场景中，从而推动智慧城市的建设。

Description

深度神经网络的解释方法、装置、终端及存储介质
本申请要求于2020年07月17日提交中国专利局、申请号为202010690459.4，发明名称为“深度神经网络的解释方法、装置、终端及存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及人工智能的深度学习技术领域，尤其涉及一种深度神经网络的解释方法、装置、终端及存储介质。
背景技术
神经网络算法，是20世纪80年代以来人工智能领域兴起的研究热点。它从信息处理角度对神经元网络进行抽象，建立某种简单模型，按不同的连接方式组成不同的网络。其具有自主学习功能，可通过训练慢慢学会识别和预测；联想存储功能，具有很高算法健壮性；并行度高，具有高速寻找优化解的能力，可针对大数据复杂问题快速找到优化解；可塑性强，可充分逼近任意复杂的非线性关系；具有很强的信息综合能力，能同时处理定量和定性的信息，适用于多信息融合和多媒体技术。近年来，神经网络的研究工作不断深入，已经取得了很大的进展，其在图像处理、模式识别、智能机器人、自动控制、预测估计、生物、医学、经济等领域已成功地解决了许多传统计算机方法难以解决的实际问题，表现出了良好的智能特性，推动着信息处理技术和人工智能不断地发展。
深度神经网络模型的可解释性一直是业内的重点关注领域，模型的可解释性对于模型改进和模型可信性与透明度都有重要作用。特征可视化是目前常用的神经网络的解释方法，其寻找模型输入中各个区域对模型预测的影响并进行可视化表示，可以在一定程度上说明模型的关注点，但是在实际应用中，发明人发现仅通过特征可视化很难对模型进行定量的解释，也不能反映影响模型判断的主要因素，无法很好的解释模型是根据哪些因子得到的预测结果。
发明内容
本申请提供一种深度神经网络的解释方法、装置、终端及存储介质，以解决难以确定影响深度神经网络预测结果的因素的问题。
为实现上述目的，本申请提供一种深度神经网络的解释方法，包括：输入多个数据至训练好的深度神经网络模型中，得到每一个数据对应的第一预测类别；当第一预测类别为预设目标类别时，将第一预测类别对应的目标数据输入至解释因子预测模型中，以得到预设目标类别对应的解释因子，解释因子预测模型根据预先设定的目标数量和预设目标类别的第一预测值训练得到；统计每一个解释因子的出现次数并排序后，筛选排名最高的目标数量个解释因子作为预设目标类别的目标解释因子。
为实现上述目的，本申请还提供了一种深度神经网络的解释装置，包括：第一输入模块，用于输入多个数据至训练好的深度神经网络模型中，得到每一个数据对应的第一预测类别；第二输入模块，用于当第一预测类别为预设目标类别时，将第一预测类别对应的目标数据输入至解释因子预测模型中，以得到预设目标类别对应的解释因子，解释因子预测模型根据预先设定的目标数量和预设目标类别的第一预测值训练得到；筛选模块，用于统计每一个解释因子的出现次数并排序后，筛选排名最高的目标数量个解释因子作为预设目标类别的目标解释因子。
为实现上述目的，本申请还提供了一种终端，包括存储器、处理器以及存储在存储器上并可在处理器上运行的程序文件，其中，处理器执行程序文件时实现以下步骤：输入多 个数据至训练好的深度神经网络模型中，得到每一个数据对应的第一预测类别；当第一预测类别为预设目标类别时，将第一预测类别对应的目标数据输入至解释因子预测模型中，以得到预设目标类别对应的解释因子，解释因子预测模型根据预先设定的目标数量和预设目标类别的第一预测值训练得到；统计每一个解释因子的出现次数并排序后，筛选排名最高的目标数量个解释因子作为预设目标类别的目标解释因子。
为实现上述目的，本申请还提供了一种存储介质，其中，存储有能够实现深度神经网络的解释方法的程序文件，程序文件被处理器执行时实现以下步骤：输入多个数据至训练好的深度神经网络模型中，得到每一个数据对应的第一预测类别；当第一预测类别为预设目标类别时，将第一预测类别对应的目标数据输入至解释因子预测模型中，以得到预设目标类别对应的解释因子，解释因子预测模型根据预先设定的目标数量和预设目标类别的第一预测值训练得到；统计每一个解释因子的出现次数并排序后，筛选排名最高的目标数量个解释因子作为预设目标类别的目标解释因子。
本申请提出的深度神经网络的解释方法，通过当深度神经网络输出的第一预测类别为预设目标类别时，将第一预设类别对应的目标数据输入至训练好的解释因子预测模型中，得到预设目标类别对应的解释因子，通过输入不同的目标数据，从而得到多组解释因子，再统计每个解释因子的出现次数，从中筛选出排名最高的目标数量个解释因子作为该预设目标类别的目标解释因子，该目标解释因子将作为深度神经网络将目标数据的输出结果划分为预设目标类别的主要影响因素，方便用户找出影响深度神经网络模型进行分类预测的影响因子。此外，本申请的深度神经网络的解释方法还可应用于智慧政务/智慧城管/智慧社区/智慧安防/智慧物流/智慧医疗/智慧教育/智慧环保/智慧交通场景中，从而推动智慧城市的建设。
附图说明
图1为本申请深度神经网络的解释方法第一实施例的流程示意图；
图2为本申请深度神经网络的解释方法实施例的解释因子预测模型训练的流程示意图；
图3为深度神经网络的结构示意图；
图4为本申请深度神经网络的解释方法第二实施例的流程示意图；
图5为本申请深度神经网络的解释装置实施例的功能模块示意图；
图6为本申请终端实施例的结构示意图；
图7为本申请存储介质实施例的结构示意图。
本申请目的的实现、功能特点及优点将结合实施例，参照附图做进一步说明。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅是本申请的一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
本申请中的术语“第一”、“第二”、“第三”仅用于描述目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”、“第三”的特征可以明示或者隐含地包括至少一个该特征。本申请的描述中，“多个”的含义是至少两个，例如两个，三个等，除非另有明确具体的限定。本申请实施例中所有方向性指示(诸如上、下、左、右、前、后……)仅用于解释在某一特定姿态(如附图所示)下各部件之间的相对位置关系、运动情况等，如果该特定姿态发生改变时，则该方向性指示也相应地随之改变。此外，术语“包括”和“具有”以及它们任何变形，意图在于覆盖不排他的包含。例如包含了一系列步骤或单元的过程、方法、系统、产品或设备没有限定于已列出的步骤 或单元，而是可选地还包括没有列出的步骤或单元，或可选地还包括对于这些过程、方法、产品或设备固有的其它步骤或单元。
在本文中提及“实施例”意味着，结合实施例描述的特定特征、结构或特性可以包含在本申请的至少一个实施例中。在说明书中的各个位置出现该短语并不一定均是指相同的实施例，也不是与其它实施例互斥的独立的或备选的实施例。本领域技术人员显式地和隐式地理解的是，本文所描述的实施例可以与其它实施例相结合。
图1是本申请第一实施例的深度神经网络的解释方法的流程示意图。需注意的是，若有实质上相同的结果，本申请的方法并不以图1所示的流程顺序为限。如图1所示，该方法包括步骤：
步骤S101：输入多个数据至训练好的深度神经网络模型中，得到每一个数据对应的第一预测类别。
需要说明的是，深度神经网络是一种广泛应用于图像处理、模式识别、智能机器人、自动控制、预测估计、生物、医学、经济等领域神经网络算法，其通常包括输入层、隐藏层和输出层，请一并参阅图2，一般第一层(input layer)是输入层，最后一层(output layer)是输出层，而中间的层数(hidden layer)都是隐藏层，层与层之间是全连接的，即第i层的任意一个神经元一定与第i+1层的任意一个神经元相连，输出层的神经元可以为一个或多个，每个神经元对应一个输出结果。在使用深度神经网络对处理分类任务时，将数据输入至深度神经网络模型，得到每个类别的预测值，再将预测值最大的类别作为深度神经网络的输出结果，例如，以动物分类为例，使用训练好的深度神经网络识别一张图片中的动物时，其可得到该动物为猫、狗、牛、羊等动物时的预测值，然后从中选取预测值最大的动物作为输出结果。
步骤S102：当第一预测类别为预设目标类别时，将第一预测类别对应的目标数据输入至解释因子预测模型中，以得到预设目标类别对应的解释因子。
需要说明的是，解释因子预测模型根据预先设定的目标数量和预设目标类别的第一预测值训练得到。其中，目标数量是指目标解释因子的个数，可由用户预先设定；该预设目标类别为用户预先设定的类别；解释因子是指影响深度神经网络模型判断的因素，例如，通过深度神经网络模型识别一张图片中的动物是什么动物，其影响深度神经网络模型判断的因素可以包括动物的脚的数量、毛发生长情况、眼睛大小等，这些因素均可能对深度神经网络模型的输出造成影响。
需要理解的是，在使用深度神经网络模型对物品进行分类时，每一个类别均受到不同解释因子的影响，例如：以动物分类为例，通过深度神经网络模型识别动物时，“猫”这一预测类别的解释因子可以包括全身毛发、四条腿行走、眼睛大、长有胡须、隐藏爪子、脚掌有肉垫等，“狗”这一预测类别的解释因子可以包括全身毛发、四条腿行走、喜欢蹲坐、显示爪子、脚掌无肉垫等，本申请旨在使用深度神经网络时确定影响输出结果的主要因素。
在步骤S102中，每当第一预测类别为预设目标类别时，将该第一预测类别对应的目标数据输入至解释因子预测模型中，以得到预设目标类别对应的解释因子。通过输入多个目标数据至解释因子预测模型，即可得到多组预设目标类别对应的解释因子，而该多组预设目标类别对应的解释因子可能相同或不同，例如：以动物分类为例，A、B两个目标数据的预测结果均为猫，但是A目标数据是通过大眼睛、四条腿、全身毛发进行预测的，而B目标数据是通过四条腿、全身毛发、毛发上带有斑纹进行预测的，两者的解释因子只有四条腿、全身毛发相同。
进一步的，步骤S102具体包括：将预设目标类别对应的目标数据输入至与预设目标类别对应的解释因子预测模型中，得到每个解释因子的第二预测值；将第二预测值超过预设阈值的解释因子的值取1，并作为预设目标类别对应的解释因子。
具体地，将目标数据输入至解释因子预测模型，从而得到每个解释因子的第二预测值， 再根据第二预测值的大小，将第二预测值超过预设阈值的解释因子的值设置为1，其余解释因子的值设置为0。
进一步的，需要理解的是，不同的预设目标类通过利用不同的样本训练得到，其对应的解释因子预测模型各不相同，因此，步骤S102中，在将第一预测类别对应的目标数据输入至解释因子预测模型之前，还包括：获取与预设目标类别对应的解释因子预测模型。
步骤S103：统计每一个解释因子的出现次数并排序后，筛选排名最高的目标数量个解释因子作为预设目标类别的目标解释因子。
在步骤S103中，通过统计多个目标数据对应的解释因子中，每个解释因子的出现次数，并进行排序，从中筛选出排名最高的目标数量个解释因子作为预设目标类别的目标解释因子。例如，以动物分类为例，解释因子的总数量为100个，通过上述解释因子预测模型，从100个解释因子中筛选将动物划分为猫的10个相关性最高的解释因子作为目标解释因子。
本实施例通过当深度神经网络输出的第一预测类别为预设目标类别时，将第一预设类别对应的目标数据输入至训练好的解释因子预测模型中，得到预设目标类别对应的解释因子，通过输入不同的目标数据，从而得到多组解释因子，再统计每个解释因子的出现次数，从中筛选出排名最高的目标数量个解释因子作为该预设目标类别的目标解释因子，该目标解释因子将作为深度神经网络将目标数据的输出结果划分为预设目标类别的主要影响因素，方便用户找出影响深度神经网络模型进行分类预测的影响因子。此外，本申请的深度神经网络的解释方法还可应用于智慧城管/智慧社区/智慧安防/智慧物流等场景中，从而推动智慧城市的建设。
进一步的，上述实施例中，每个预设目标类别对应的解释因子预测模型预先训练得到，如图3所示，解释因子预测模型的训练步骤具体包括：
步骤S201：逐个输入样本至训练好的深度神经网络模型中，得到每个样本对应的预测类别以及第一预测值。
需要说明的是，深度神经网络模型的输出通常包括多个预测类别，每个预测类别对应一个预测值，其中预测值最高的结果作为最终输出结果。
进一步的，在步骤S201之前，还包括：获取用户预先输入的至少一个预设目标类别，以及每个预设目标类别对应的目标解释因子的目标数量。
本实施例中，用户可根据需求选择需要训练的类别，并且，用户还可设定最终输出的解释因子的目标数量。例如：用户可以设定动物分类中猫这一类别的目标解释因子的目标数量为10个。
步骤S202：当预测类别为预设目标类别时，将预测类别对应的目标样本输入至解释因子预测模型中，得到每个解释因子的第二预测值。
步骤S203：根据第一预测值、第二预测值和目标数量构建解释因子标签。
本实施例中，该构建解释因子标签的步骤具体包括：
1、根据第一预测值和目标数量计算所有的解释因子中取值为1的解释因子的数目w。计算公式为：
w＝ceil(m×S k)；
其中，ceil表示向上取整，m表示目标数量，S k表示第一预测值。
2、根据每个解释因子的第二预测值的大小，将第二预测值最大的w个解释因子的值设置为1，将其余的解释因子的值设置为0，得到解释因子标签。
具体地，该解释因子标签具体如下：
Figure PCTCN2020123196-appb-000001
A i＝0,i≤m且
Figure PCTCN2020123196-appb-000002
其中，q 1,q 2,…,q w是指第二预测值最大的w个解释因子的角标。
例如：以一个例子进行说明，m为10，S k为80％，根据上述公式计算得到w为8，假设所有的解释因子共有30个，将目标样本输入至解释因子预测模型中，即可得到该30个解释因子的第二预测值，再从中挑选出第二预测值最高的8个解释因子，将其值设置为1，其余的22个解释因子的值设置为0，从而得到该预测类别对应的解释因子标签。
需要理解的是，每一个预测类别的解释因子标签均是基于预测类别当前的第一预测值构建，因此，每次输入目标样本得到预测类别时，均需构建一次解释因子标签。
步骤S204：基于解释因子标签和第二预测值反向传播更新解释因子预测模型。
在步骤S204中，通过比对解释因子标签与每个解释因子的第二预测值反向传播更新解释因子预测模型的方式与深度神经网络训练时的反向传播更新过程相同。
步骤S205：循环执行上述步骤直至解释因子预测模型收敛，并将收敛的解释因子预测模型与预设目标类别绑定。
具体地，通过循环输入样本至训练好的深度神经网络模型，从而对解释因子预测模型进行反复迭代训练，直至解释因子预测模型收敛时，停止训练并将收敛的解释因子预测模型，并将收敛的解释因子预测模型与预设目标类别绑定。
本实施例中，通过结合预测类别的第一预测值动态得到解释因子的标签，并利用该标签对解释因子预测模型进行训练，从而使得预设目标类别对应的解释因子将对于预设目标类别的样本取较大值1，对于其他类别样本取较小值0。
图4展示了本申请第二实施例的深度神经网络的解释方法的流程示意图。需注意的是，若有实质上相同的结果，本申请的方法并不以图4所示的流程顺序为限。如图4所示，该方法包括步骤：
步骤S301：获取用户预先输入的至少一个预设目标类别，每个预设目标类别均对应有预先训练至收敛状态的解释因子预测模型。
在步骤S301中，预设目标类别可由用户预先指定，用户可以一次性指定一个或多个预设目标类别，而每个预设目标类别在分别利用解释因子预测模型确认目标解释因子时，分别通过每个预设目标类别对应的解释因子预测模型得到，相互之间互不干扰。
步骤S302：输入多个数据至训练好的深度神经网络模型中，得到每一个数据对应的第一预测类别。
在本实施例中，图4中的步骤S302和图1中的步骤S101类似，为简约起见，在此不再赘述。
步骤S303：当第一预测类别为预设目标类别时，将第一预测类别对应的目标数据输入至解释因子预测模型中，以得到预设目标类别对应的解释因子。
在本实施例中，图4中的步骤S303和图1中的步骤S102类似，为简约起见，在此不再赘述。
步骤S304：统计每一个解释因子的出现次数并排序后，筛选排名最高的目标数量个解释因子作为预设目标类别的目标解释因子。
在本实施例中，图4中的步骤S304和图1中的步骤S103类似，为简约起见，在此不再赘述。
本实施例中，预设目标类别可以由用户自行指定，通过获取用户预先输入预设目标类别即可为用户确定该深度神经网络模型输出该预设目标类别的主要影响因素，并且，用户可以一次性输入多个预设目标类别，帮助用户快速确认每个预设目标类别的主要影响因素。
图5展示了本申请实施例深度神经网络的解释装置的功能模块示意图。如图5所示，该深度神经网络的解释装置10包括第一输入模块11、第二输入模块12和筛选模块13。
第一输入模块11，用于输入多个数据至训练好的深度神经网络模型中，得到每一个数据对应的第一预测类别；
第二输入模块12，用于当第一预测类别为预设目标类别时，将第一预测类别对应的 目标数据输入至解释因子预测模型中，以得到预设目标类别对应的解释因子，解释因子预测模型根据预先设定的目标数量和预设目标类别的第一预测值训练得到；
筛选模块13，用于统计每一个解释因子的出现次数并排序后，筛选排名最高的目标数量个解释因子作为预设目标类别的目标解释因子。
可选地，第二输入模块12将第一预测类别对应的目标数据输入至解释因子预测模型之前，还包括预先训练解释因子预测模型，包括：逐个输入样本至训练好的深度神经网络模型中，得到每个样本对应的预测类别以及第一预测值；当预测类别为预设目标类别时，将预测类别对应的目标样本输入至解释因子预测模型中，得到每个解释因子的第二预测值；根据第一预测值、第二预测值和目标数量构建解释因子标签；基于解释因子标签和第二预测值反向传播更新解释因子预测模型；循环执行上述步骤直至解释因子预测模型收敛，并将收敛的解释因子预测模型与预设目标类别绑定。
可选地，利用第一预测值、每个解释因子的第二预测值和目标数量构建解释因子标签操作还可以为：根据第一预测值和目标数量计算所有的解释因子中取值为1的解释因子的数目w，w＝ceil(m×S k)，其中，ceil表示向上取整，m表示目标数量，S k表示第一预测值；根据每个解释因子的第二预测值的大小，将第二预测值最大的w个解释因子的值设置为1，将其余的解释因子的值设置为0，得到解释因子标签。
可选地，逐个输入样本至训练好的深度神经网络模型的操作之前还包括：获取用户预先输入的至少一个预设目标类别，以及每个预设目标类别对应的目标解释因子的目标数量。
可选地，第一输入模块11输入多个数据至训练好的深度神经网络模型中之前，还用于获取用户预先输入的至少一个预设目标类别，每个预设目标类别均对应有预先训练至收敛状态的解释因子预测模型。
可选地，第二输入模块12将第一预测类别对应的目标数据输入至解释因子预测模型中，以得到预设目标类别对应的解释因子的操作还可以为：将预设目标类别对应的目标数据输入至与预设目标类别对应的解释因子预测模型中，得到每个解释因子的第二预测值；将第二预测值超过预设阈值的解释因子的值取1，并作为预设目标类别对应的解释因子。
可选地，第一输入模块11将第一预测类别对应的目标数据输入至解释因子预测模型的操作之前，还包括：获取与预设目标类别对应的解释因子预测模型。
请参阅图6，图6为本申请实施例的终端的结构示意图。如图6所示，该终端20包括处理器200及和处理器200耦接的存储器201。
存储器201存储有用于实现上述任一实施例所述的深度神经网络的解释方法的程序指令。
处理器200用于执行存储器201存储的程序指令以训练得到预设目标类别的目标解释因子。
其中，处理器200还可以称为CPU(Central Processing Unit，中央处理单元)。处理器200可能是一种集成电路芯片，具有信号的处理能力。处理器200还可以是通用处理器、数字信号处理器(DSP)、专用集成电路(ASIC)、现场可编程门阵列(FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。
参阅图7，图7为本申请实施例的存储介质的结构示意图。本申请实施例的存储介质存储有能够实现上述所有方法的程序文件30，其中，该程序文件30可以以软件产品的形式存储在上述存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)或处理器(processor)执行本申请各个实施方式所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、磁碟或者光盘等各种可以存储程序代码的介质，或者是计算机、服务器、手机、平板等终端设备。所述计算机可读 存储介质可以是非易失性，也可以是易失性
在本申请所提供的几个实施例中，应该理解到，所揭露的终端，装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。以上仅为本申请的实施方式，并非因此限制本申请的专利范围，凡是利用本申请说明书及附图内容所作的等效结构或等效流程变换，或直接或间接运用在其他相关的技术领域，均同理包括在本申请的专利保护范围内。

Claims (22)

  1. 一种深度神经网络的解释方法，其中，包括：
    输入多个数据至训练好的深度神经网络模型中，得到每一个数据对应的第一预测类别；
    当所述第一预测类别为预设目标类别时，将所述第一预测类别对应的目标数据输入至解释因子预测模型中，以得到所述预设目标类别对应的解释因子，所述解释因子预测模型根据预先设定的目标数量和所述预设目标类别的第一预测值训练得到；
    统计每一个解释因子的出现次数并排序后，筛选排名最高的所述目标数量个解释因子作为所述预设目标类别的目标解释因子。
  2. 根据权利要求1所述的方法，其中，还包括预先训练所述解释因子预测模型，包括：
    逐个输入样本至所述训练好的深度神经网络模型中，得到每个样本对应的预测类别以及第一预测值；
    当所述预测类别为预设目标类别时，将所述预测类别对应的目标样本输入至所述解释因子预测模型中，得到每个解释因子的第二预测值；
    根据所述第一预测值、所述第二预测值和所述目标数量构建解释因子标签；
    基于所述解释因子标签和所述第二预测值反向传播更新所述解释因子预测模型；
    循环执行上述步骤直至所述解释因子预测模型收敛，并将收敛的解释因子预测模型与所述预设目标类别绑定。
  3. 根据权利要求2所述的方法，其中，所述利用所述第一预测值、所述每个解释因子的第二预测值和所述目标数量构建解释因子标签，包括：
    根据所述第一预测值和所述目标数量计算所有的解释因子中取值为1的解释因子的数目w，w＝ceil(m×S k)，其中，ceil表示向上取整，m表示所述目标数量，S k表示所述第一预测值；
    根据所述每个解释因子的第二预测值的大小，将所述第二预测值最大的所述w个解释因子的值设置为1，将其余的解释因子的值设置为0，得到所述解释因子标签。
  4. 根据权利要求2所述的方法，其中，所述逐个输入样本至所述训练好的深度神经网络模型中之前，还包括：
    获取用户预先输入的至少一个预设目标类别，以及每个预设目标类别对应的目标解释因子的目标数量。
  5. 根据权利要求1所述的方法，其中，所述输入多个数据至训练好的深度神经网络模型中之前，还包括：
    获取用户预先输入的至少一个预设目标类别，每个所述预设目标类别均对应有预先训练至收敛状态的解释因子预测模型。
  6. 根据权利要求1所述的方法，其中，所述将所述第一预测类别对应的目标数据输入至解释因子预测模型中，以得到所述预设目标类别对应的解释因子，包括：
    将所述预设目标类别对应的目标数据输入至与所述预设目标类别对应的解释因子预测模型中，得到所述每个解释因子的第二预测值；
    将所述第二预测值超过预设阈值的解释因子的值取1，并作为所述预设目标类别对应的解释因子。
  7. 根据权利要求1所述的方法，其中，所述将所述第一预测类别对应的目标数据输入至解释因子预测模型中之前，还包括：
    获取与所述预设目标类别对应的解释因子预测模型。
  8. 一种深度神经网络的解释装置，其中，包括：
    第一输入模块，用于输入多个数据至训练好的深度神经网络模型中，得到每一个数据 对应的第一预测类别；
    第二输入模块，用于当所述第一预测类别为预设目标类别时，将所述第一预测类别对应的目标数据输入至解释因子预测模型中，以得到所述预设目标类别对应的解释因子，所述解释因子预测模型根据预先设定的目标数量和所述预设目标类别的第一预测值训练得到；
    筛选模块，用于统计每一个解释因子的出现次数并排序后，筛选排名最高的所述目标数量个解释因子作为所述预设目标类别的目标解释因子。
  9. 一种终端，包括存储器、处理器以及存储在存储器上并可在处理器上运行的程序文件，其中，所述处理器执行所述程序文件时实现以下步骤：
    输入多个数据至训练好的深度神经网络模型中，得到每一个数据对应的第一预测类别；
    当所述第一预测类别为预设目标类别时，将所述第一预测类别对应的目标数据输入至解释因子预测模型中，以得到所述预设目标类别对应的解释因子，所述解释因子预测模型根据预先设定的目标数量和所述预设目标类别的第一预测值训练得到；
    统计每一个解释因子的出现次数并排序后，筛选排名最高的所述目标数量个解释因子作为所述预设目标类别的目标解释因子。
  10. 根据权利要求9所述的终端，其中，所述处理器执行所述程序文件时还用于预先训练所述解释因子预测模型，包括：
    逐个输入样本至所述训练好的深度神经网络模型中，得到每个样本对应的预测类别以及第一预测值；
    当所述预测类别为预设目标类别时，将所述预测类别对应的目标样本输入至所述解释因子预测模型中，得到每个解释因子的第二预测值；
    根据所述第一预测值、所述第二预测值和所述目标数量构建解释因子标签；
    基于所述解释因子标签和所述第二预测值反向传播更新所述解释因子预测模型；
    循环执行上述步骤直至所述解释因子预测模型收敛，并将收敛的解释因子预测模型与所述预设目标类别绑定。
  11. 根据权利要求10所述的终端，其中，所述利用所述第一预测值、所述每个解释因子的第二预测值和所述目标数量构建解释因子标签，包括：
    根据所述第一预测值和所述目标数量计算所有的解释因子中取值为1的解释因子的数目w，w＝ceil(m×S k)，其中，ceil表示向上取整，m表示所述目标数量，S k表示所述第一预测值；
    根据所述每个解释因子的第二预测值的大小，将所述第二预测值最大的所述w个解释因子的值设置为1，将其余的解释因子的值设置为0，得到所述解释因子标签。
  12. 根据权利要求10所述的终端，其中，所述逐个输入样本至所述训练好的深度神经网络模型中之前，还包括：
    获取用户预先输入的至少一个预设目标类别，以及每个预设目标类别对应的目标解释因子的目标数量。
  13. 根据权利要求9所述的终端，其中，所述输入多个数据至训练好的深度神经网络模型中之前，还包括：
    获取用户预先输入的至少一个预设目标类别，每个所述预设目标类别均对应有预先训练至收敛状态的解释因子预测模型。
  14. 根据权利要求9所述的终端，其中，所述将所述第一预测类别对应的目标数据输入至解释因子预测模型中，以得到所述预设目标类别对应的解释因子，包括：
    将所述预设目标类别对应的目标数据输入至与所述预设目标类别对应的解释因子预测模型中，得到所述每个解释因子的第二预测值；
    将所述第二预测值超过预设阈值的解释因子的值取1，并作为所述预设目标类别对应 的解释因子。
  15. 根据权利要求9所述的终端，其中，所述将所述第一预测类别对应的目标数据输入至解释因子预测模型中之前，还包括：
    获取与所述预设目标类别对应的解释因子预测模型。
  16. 一种存储介质，其中，存储有能够实现深度神经网络的解释方法的程序文件，所述程序文件被处理器执行时实现以下步骤：
    输入多个数据至训练好的深度神经网络模型中，得到每一个数据对应的第一预测类别；
    当所述第一预测类别为预设目标类别时，将所述第一预测类别对应的目标数据输入至解释因子预测模型中，以得到所述预设目标类别对应的解释因子，所述解释因子预测模型根据预先设定的目标数量和所述预设目标类别的第一预测值训练得到；
    统计每一个解释因子的出现次数并排序后，筛选排名最高的所述目标数量个解释因子作为所述预设目标类别的目标解释因子。
  17. 根据权利要求16所述的存储介质，其中，所述程序文件被执行时还用于预先训练所述解释因子预测模型，包括：
    逐个输入样本至所述训练好的深度神经网络模型中，得到每个样本对应的预测类别以及第一预测值；
    当所述预测类别为预设目标类别时，将所述预测类别对应的目标样本输入至所述解释因子预测模型中，得到每个解释因子的第二预测值；
    根据所述第一预测值、所述第二预测值和所述目标数量构建解释因子标签；
    基于所述解释因子标签和所述第二预测值反向传播更新所述解释因子预测模型；
    循环执行上述步骤直至所述解释因子预测模型收敛，并将收敛的解释因子预测模型与所述预设目标类别绑定。
  18. 根据权利要求17所述的存储介质，其中，所述利用所述第一预测值、所述每个解释因子的第二预测值和所述目标数量构建解释因子标签，包括：
    根据所述第一预测值和所述目标数量计算所有的解释因子中取值为1的解释因子的数目w，w＝ceil(m×S k)，其中，ceil表示向上取整，m表示所述目标数量，S k表示所述第一预测值；
    根据所述每个解释因子的第二预测值的大小，将所述第二预测值最大的所述w个解释因子的值设置为1，将其余的解释因子的值设置为0，得到所述解释因子标签。
  19. 根据权利要求17所述的存储介质，其中，所述逐个输入样本至所述训练好的深度神经网络模型中之前，还包括：
    获取用户预先输入的至少一个预设目标类别，以及每个预设目标类别对应的目标解释因子的目标数量。
  20. 根据权利要求16所述的存储介质，其中，所述输入多个数据至训练好的深度神经网络模型中之前，还包括：
    获取用户预先输入的至少一个预设目标类别，每个所述预设目标类别均对应有预先训练至收敛状态的解释因子预测模型。
  21. 根据权利要求16所述的存储介质，其中，所述将所述第一预测类别对应的目标数据输入至解释因子预测模型中，以得到所述预设目标类别对应的解释因子，包括：
    将所述预设目标类别对应的目标数据输入至与所述预设目标类别对应的解释因子预测模型中，得到所述每个解释因子的第二预测值；
    将所述第二预测值超过预设阈值的解释因子的值取1，并作为所述预设目标类别对应的解释因子。
  22. 根据权利要求16所述的存储介质，其中，所述将所述第一预测类别对应的目标数据输入至解释因子预测模型中之前，还包括：
    获取与所述预设目标类别对应的解释因子预测模型。
PCT/CN2020/123196 2020-07-17 2020-10-23 深度神经网络的解释方法、装置、终端及存储介质 Ceased WO2021135546A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010690459.4A CN111832707A (zh) 2020-07-17 2020-07-17 深度神经网络的解释方法、装置、终端及存储介质
CN202010690459.4 2020-07-17

Publications (1)

Publication Number Publication Date
WO2021135546A1 true WO2021135546A1 (zh) 2021-07-08

Family

ID=72923963

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/123196 Ceased WO2021135546A1 (zh) 2020-07-17 2020-10-23 深度神经网络的解释方法、装置、终端及存储介质

Country Status (2)

Country Link
CN (1) CN111832707A (zh)
WO (1) WO2021135546A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115705584A (zh) * 2021-08-10 2023-02-17 中国移动通信集团浙江有限公司 看家产品需求的预测方法、装置及可读存储介质
WO2024198754A1 (zh) * 2023-03-24 2024-10-03 腾讯科技（深圳）有限公司 一种数据处理方法、装置、设备以及可读存储介质

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220138510A1 (en) * 2020-10-30 2022-05-05 Robert Bosch Gmbh Deep neural networks via prototype factorization
CN112101574B (zh) * 2020-11-20 2021-03-02 成都数联铭品科技有限公司 一种机器学习有监督模型解释方法、系统及设备
CN112861933B (zh) * 2021-01-22 2022-08-12 中国科学院信息工程研究所 一种深度神经网络模型可视化解释方法及系统
CN114170485B (zh) * 2021-11-23 2024-09-10 北京航空航天大学 深度学习可解释方法及设备、存储介质和程序产品
CN114331292A (zh) * 2022-01-05 2022-04-12 成都以专信息技术有限公司 一种用于研究生招生考试试卷分拣的管理信息系统
CN114663694B (zh) * 2022-01-20 2025-04-01 中央财经大学 一种卷积神经网络隐层解释方法
CN116167417A (zh) * 2023-03-21 2023-05-26 上海金仕达软件科技股份有限公司 一种深度神经网络模型的预测结果的解释方法及系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170315960A1 (en) * 2014-11-19 2017-11-02 Nec Corporation Factor analysis apparatus, factor analysis method and recording medium, and factor analysis system
CN107688870A (zh) * 2017-08-15 2018-02-13 中国科学院软件研究所 一种基于文本流输入的深度神经网络的分层因素可视化分析方法及装置
CN109492760A (zh) * 2018-12-21 2019-03-19 北京航空航天大学 基于深度模糊认知图模型的可解释预测方法
CN110414627A (zh) * 2019-08-07 2019-11-05 北京嘉和海森健康科技有限公司 一种模型的训练方法及相关设备
CN111353603A (zh) * 2020-03-05 2020-06-30 深圳索信达数据技术有限公司 一种深度学习模型个体预测解释方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170315960A1 (en) * 2014-11-19 2017-11-02 Nec Corporation Factor analysis apparatus, factor analysis method and recording medium, and factor analysis system
CN107688870A (zh) * 2017-08-15 2018-02-13 中国科学院软件研究所 一种基于文本流输入的深度神经网络的分层因素可视化分析方法及装置
CN109492760A (zh) * 2018-12-21 2019-03-19 北京航空航天大学 基于深度模糊认知图模型的可解释预测方法
CN110414627A (zh) * 2019-08-07 2019-11-05 北京嘉和海森健康科技有限公司 一种模型的训练方法及相关设备
CN111353603A (zh) * 2020-03-05 2020-06-30 深圳索信达数据技术有限公司 一种深度学习模型个体预测解释方法

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115705584A (zh) * 2021-08-10 2023-02-17 中国移动通信集团浙江有限公司 看家产品需求的预测方法、装置及可读存储介质
WO2024198754A1 (zh) * 2023-03-24 2024-10-03 腾讯科技（深圳）有限公司 一种数据处理方法、装置、设备以及可读存储介质

Also Published As

Publication number Publication date
CN111832707A (zh) 2020-10-27

Similar Documents

Publication Publication Date Title
WO2021135546A1 (zh) 深度神经网络的解释方法、装置、终端及存储介质
CN111259142B (zh) 基于注意力编码和图卷积网络的特定目标情感分类方法
JP7403909B2 (ja) 系列マイニングモデルの訓練装置の動作方法、系列データの処理装置の動作方法、系列マイニングモデルの訓練装置、系列データの処理装置、コンピュータ機器、及びコンピュータプログラム
WO2022042002A1 (zh) 一种半监督学习模型的训练方法、图像处理方法及设备
Jia et al. Visual concept learning: Combining machine vision and bayesian generalization on concept hierarchies
Wang et al. Multitask learning for object localization with deep reinforcement learning
CN111125422A (zh) 一种图像分类方法、装置、电子设备及存储介质
WO2021136131A1 (zh) 一种信息推荐方法以及相关设备
JP7457436B2 (ja) 少数ショット時間的行動局所化を容易化するシステム、方法、プログラム
CN107851198A (zh) 媒体分类
CN111027622B (zh) 图片标签生成方法、装置、计算机设备及存储介质
CN112749737A (zh) 图像分类方法及装置、电子设备、存储介质
US20220076129A1 (en) Method of training a deep neural network to classify data
CN113609337A (zh) 图神经网络的预训练方法、训练方法、装置、设备及介质
CN112364912B (zh) 信息分类方法、装置、设备及存储介质
Kalita Machine learning: Theory and practice
CN111783473B (zh) 医疗问答中最佳答案的识别方法、装置和计算机设备
US20250342215A1 (en) Content search method and apparatus, electronic device, storage medium, and program product
Uke et al. An enhanced artificial neural network for hand gesture recognition using multi-modal features
CN115131604A (zh) 一种多标签图像分类方法、装置、电子设备及存储介质
Bi et al. A new artificial intelligent approach to buoy detection for mussel farming
Refat et al. SentiNet: a nonverbal facial sentiment analysis using convolutional neural network
CN114863560A (zh) 手势识别方法、装置、电子设备及计算机可读存储介质
Bello et al. Cattle Instance Segmentation by Transfer Learning Approach Using Deep Learning Models for Sustainable Livestock Farming
WO2024055677A1 (zh) 一种深度聚类的方法、装置及系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20910119

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20910119

Country of ref document: EP

Kind code of ref document: A1