US20220358695A1 - Method for generating detailed visualization of machine learning model behavior - Google Patents
Method for generating detailed visualization of machine learning model behavior Download PDFInfo
- Publication number
- US20220358695A1 US20220358695A1 US17/302,592 US202117302592A US2022358695A1 US 20220358695 A1 US20220358695 A1 US 20220358695A1 US 202117302592 A US202117302592 A US 202117302592A US 2022358695 A1 US2022358695 A1 US 2022358695A1
- Authority
- US
- United States
- Prior art keywords
- image
- heatmap
- cropped
- visualization
- selected area
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/20—Drawing from basic elements, e.g. lines or circles
- G06T11/206—Drawing of charts or graphs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/09—Supervised learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/04—Inference or reasoning models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2210/00—Indexing scheme for image generation or computer graphics
- G06T2210/22—Cropping
Definitions
- This disclosure relates generally to machine learning, and more particularly, to a method for generating a more detailed visualization of a behavior of a machine learning (ML) model.
- ML machine learning
- Machine learning is becoming more widely used in many of today's applications, such as applications involving forecasting and classification.
- a ML model is trained, at least partly, before it is used.
- Training data is used for training a ML model.
- Machine learning models may be classified by how they are trained.
- Supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning are examples of training techniques.
- Machine learning models have been used to classify images.
- Machine learning models have also been used in object detection to locate and classify various objects in an image.
- object detection each object in an image may be labeled so that an image may include more than one label.
- Object detection is useful in automotive applications such as autonomous driving where the automobile needs to locate objects in the images captured by cameras on the automobile.
- object detectors typically need to operate in a low-latency online fashion and to preserve the privacy of the user.
- FIG. 1 illustrates a simplified system for training and using a ML model in accordance with an embodiment.
- FIG. 2 illustrates a flow diagram of a method for generating a higher resolution visualization.
- FIG. 3 illustrates a flowchart of a method for a machine learning model (ML) in accordance with an embodiment.
- FIG. 4 illustrates a data processing system useful for implementing an embodiment of the present invention.
- a method for providing a more detailed visualization for explaining the behavior of a ML model first generates a first heat map to find an area in an input sample above a predetermined threshold that indicates the most important area for a specified target classification.
- the area is cropped from the input sample and upscaled to, for example, the input resolution.
- a second heatmap is generated of the cropped area to allow a more detailed explanation of the influence this area had on the classification result. Because the area is upscaled, the heatmap can provide a more detailed visualization and provide better insight into why the ML model made the classification it made.
- the method may be used in, for example, classifiers and object detectors.
- a method for generating a visualization for explaining a behavior of a machine learning (ML) model including: inputting an image into a ML model for an inference operation; generating a first heatmap for the image using a first visualization method; selecting an area of highest attention on the first heatmap using a predetermined threshold; cropping the selected area from the image; upscaling the cropped selected area; generating a second heatmap for the cropped and upscaled selected area of the image; and presenting a final visualization for analysis.
- Generating the second heatmap for the cropped and upscaled selected area of the image may further include generating the second heatmap for the cropped and upscaled selected area of the image using a second visualization method.
- the first and second visualization methods may each be the GradCAM visualization method.
- the first visualization method may be a GradCAM visualization method and the second visualization method may be a visualization method different from the GradCAM visualization method.
- Upscaling the cropped selected area may further include one of either zooming in the cropped selected area or using the cropped selected area from the image when the image has a higher resolution than the cropped image.
- Selecting the area of highest attention on the first heatmap using a predetermined threshold may further include selecting multiple areas of highest attention that exceed the predetermined threshold.
- Generating a first heatmap for the image using a first visualization method may further include generating the first heatmap for the image using GradCAM on a last convolutional layer of a neural network of the ML model.
- the predetermined threshold may be equal to a 75 th percentile of heatmap values.
- Generating a second heatmap for the cropped and upscaled selected area of the image may further include generating the second heatmap for the cropped and upscaled selected area using GradCAM on a last convolutional layer of a neural network of the ML model.
- a computer program comprising instructions executable by a processor, for executing a method for generating a visualization for explaining a behavior of a machine learning (ML) model, the executable instructions including: instructions for inputting an image into a ML model for an inference operation; instructions for generating a first heatmap for the image using a first visualization method; instructions for selecting an area of highest attention on the first heatmap using a predetermined threshold; instructions for cropping the selected area from the image; instructions for upscaling the cropped selected area; instructions for generating a second heatmap for the cropped and upscaled selected area of the image; and instructions for presenting a final visualization for analysis.
- ML machine learning
- the instructions for generating the second heatmap for the cropped and upscaled selected area of the image may further include instructions for generating the second heatmap for the cropped and upscaled selected area of the image using a second visualization method.
- the first and second visualization methods may each be the GradCAM visualization method.
- the first visualization method may be a GradCAM visualization method and the second visualization method may be a visualization method different from the GradCAM visualization method.
- the instructions for upscaling the cropped selected area may further include one of either zooming in the cropped selected area or using the cropped selected area from an original image when the original image has a higher resolution than the cropped image.
- the instructions for selecting the area of highest attention on the first heatmap using a predetermined threshold may further include instructions for selecting multiple areas of highest attention that exceed the predetermined threshold.
- the instructions for generating the first heatmap for the image using a first visualization method may further include instructions for generating the first heatmap for the image using GradCAM on a last convolutional layer of a neural network of the ML model.
- the predetermined threshold may be equal to a 75 th percentile of heatmap values.
- the instructions for generating a second heatmap for the cropped and upscaled selected area of the image may further include instructions for generating the second heatmap for the cropped and upscaled selected area using GradCAM on a last convolutional layer of a neural network of the ML model.
- FIG. 1 illustrates a simplified system for training and using a ML model in accordance with an embodiment.
- System 10 includes labeled set of ML training data 12 , model training block 14 , and resulting trained ML model 16 .
- system 10 is implemented as a computer program stored on a non-transitory medium comprising instructions that are executable on a processor.
- One example embodiment includes a neural network (NN) algorithm used to classify images as trained using a training data set 12 .
- training data set 12 may be used for object detection in an image.
- Various training data sets can be acquired, such as for example, the CIFAR10 data set for image classification and the PASCAL Visual Object Classes (VOC) data set for object classification.
- a ML algorithm for a vision-based classification problem only returns a single output prediction (a class label) for an input image.
- a ML algorithm for an object detection problem which is commonly referred to as an object detector can, on the other hand, return multiple output predictions for a single input image, one output prediction for every identified object. Every output prediction consists of a class label as well as a bounding box that contains the object.
- the ML model is based on a neural network (NN) algorithm and the training data includes a plurality of images. In other embodiments, the ML algorithm and training data may be different.
- the ML model is trained in model training 14 using training data 12 comprising a plurality of labeled images.
- the training data may include images that are labeled by classes, such as for example, bird, cat, dog, bus, traffic light, etc. If the ML model is to be used as an object detector, the training data includes images with more than one labeled object.
- the trained ML model 16 can be used for inference operations. In an inference operation, one or more input samples labeled “INPUT SAMPLES” are input into ML Model 16 and an output classification labeled “OUTPUT” is provided by the model.
- FIG. 2 illustrates a flow diagram of method 20 for generating a higher resolution visualization.
- the currently available methods for providing interpretability and explainability are considered to generate good explanations in general and are relatively computationally inexpensive, but their ability to explain model behavior is severely limited by their low resolution compared to the input resolution.
- the low resolution is a direct result of the trade-off that is made when selecting a convolutional layer for the visualization. For example, more and more noise is added and heatmaps get less semantically meaningful towards the beginning of the convolutional neural network (CNN) while the resolution is increasingly reduced towards the end of the network. This means that in order to generate the most meaningful visualizations the layer that is typically selected is very close to the end of the network which results in a very low resolution.
- CNN convolutional neural network
- input images may have a resolution of 224 ⁇ 224 pixels.
- the resolution of the generated visualizations may be reduced to only 7 ⁇ 7 pixels. This low resolution makes it hard to interpret the visualizations in many cases, especially if smaller objects are involved or the classification decision of the model depends on finer details of the input.
- the method disclosed herein improves generated visualizations by focusing on a smaller area of interest so that the full resolution of the visualization method is available for that area.
- a first visualization method is used to find an area of interest in an input image for a neural network (i.e. an area that is the most influential for a given target classification as determined by the model). This area of interest is then cropped out of the image and scaled up to the original input resolution.
- a second visualization method is then used to visualize the influence of each individual input pixel or groups of pixels on the target classification. The resulting visualization can then be shown directly, overlayed over the cropped image or overlayed over the original image to enhance the explainability of the decisions made by the CNN.
- CNNs convolutional neural networks
- GradCAM gradient-weighted class activation mapping
- GradCAM generates heatmaps showing the most influential areas of the input for a target classification based on a weighted sum of the activation maps of a selected convolutional layer. For weights it uses the gradients for the target class as an estimation for the importance of each activation map for the target class.
- a trade-off is made when selecting a convolutional layer for GradCAM as the highest resolution is retained towards the beginning of the CNN but as more noise is added the heatmaps get semantically less meaningful.
- the convolutional layer that is typically selected is very close to the end of the network, i.e. the last convolutional layer.
- an image is input to the ML model.
- a first GradCAM heatmap is generated to find the area in the input image that is seen by the model as the most important for the classification as the target class (i.e. the area that has the highest model attention for the target class).
- the area of interest is chosen by the developer and can be any part of the image.
- the area is chosen based on a threshold.
- the selected area is a rectangle and surrounds a sun symbol. The sun symbol is representative of any portion of an image that is of interest for analysis, e.g., a dog, cat, bus, traffic light, etc.
- the first GradCAM heatmap shows the area with the highest model attention based on a threshold.
- the smallest rectangle such that all areas where the heatmap value is greater than or equal to the threshold are inside the rectangle.
- This threshold can be a fixed value or dynamically calculated based on the heatmap.
- a dynamic threshold equal to the 75th percentile of the heatmap values reliably selected an area of interest that contained the object representing the target class while at the same time being small enough to benefit from the gain in resolution when focusing on the selected area only.
- a heatmap of an image typically appears as a very out of focus version of the image and doesn't necessarily look like the visual image.
- the selected area of highest attention is cropped from the input image and scaled up to the original input resolution of the model using linear interpolation.
- linear interpolation delivers good results but any other method for upscaling is also possible.
- the input sample from the dataset has a higher resolution than can be input into the ML model, so the input is actually downscaled before providing the input to the model. In this case, upscaling may not be necessary because the higher resolution input sample can be used.
- a second GradCAM heatmap is generated of the cropped and upscaled selected area.
- the second GradCAM visualization differs from the first visualization in that the whole resolution of the visualization is used only for the selected area of interest to get more information on finer details of the selected area.
- a final visualization is then presented to the user, which can be either just showing the heatmap itself, overlaying the heatmap over the cropped image or overlaying the heatmap over the original image.
- the user may be a human user or a computer system.
- overlaying the heatmap over the cropped image the heatmap is scaled to the original image size, if necessary.
- overlaying the heatmap over the complete original image the heatmap is scaled to the original width and height of the selected area and only overlayed over this area, alternatively both the first and second heatmap can be combined before overlaying them over the original image.
- more than one area of highest model attention can be selected, cropped, visualized and combined to one heatmap.
- the disclosed embodiments improve the resolution of the visualizations used to visualize the influence or importance of individual pixels or groups of pixels of the input image. This allows the user to visualize finer details that help explain model behavior in cases where the original resolution is not enough or that help to increase confidence in the interpretation of model behavior (e.g. when examining misclassifications there are many cases where the visualizations for correct class and predicted class are very close to each other).
- the use of the first visualization method to select the area of highest model attention allows this method to automatically select the most important areas of the input image from the perspective of the model.
- the disclosed method can also be used to verify the correctness of generated visualizations in an automated way. Instead of cropping the selected area of highest attention, the area is covered by a patch (e.g. by a solid grey patch) and run through the model again. If the expected significant decrease in confidence of the target class is indeed observed, the visualization is valid, otherwise it's probably not valid.
- a patch e.g. by a solid grey patch
- the method can also be used to examine the influence of the background itself or other objects in the background on the classification decision of the model.
- the cropped area of highest attention is run through the model and confidence in the target class decreases, then there may be a bias in the model towards the target class in combination with a certain background or another object in the background.
- the confidence value of the target class increases, there may be a negative bias caused by the background or some other objects in the background.
- FIG. 3 illustrates a flowchart of method 40 for analyzing a machine learning model (ML) in accordance with an embodiment.
- Method 40 begins at step 42 .
- an image is input into an ML model for an inference operation.
- a first heatmap is generated for the image using a first visualization method.
- an area of highest attention is selected from the first heatmap using a predetermined threshold.
- the selected area is cropped from the image.
- the cropped selected area is upscaled.
- a second heatmap is generated for the cropped and upscaled selected area of the image.
- a final visualization is presented for analysis.
- FIG. 4 illustrates data processing system 60 useful for implementing an embodiment of the present invention.
- Data processing system 60 may be implemented on one or more integrated circuits and may be used in an implementation of the described embodiments.
- Data processing system 60 includes bus 62 .
- the one or more processor cores 64 may include any hardware device capable of executing instructions stored in memory 76 or instruction memory 70 .
- processor cores 64 may execute the machine learning algorithms used for training and operating the ML model.
- Processor cores 64 may be used to execute the method for explainability described herein.
- Processor cores 64 may be, for example, a microprocessor, field programmable gate array (FPGA), application-specific integrated circuit (ASIC), or similar device.
- Processor cores 64 may be implemented in a secure hardware element and may be tamper resistant.
- Memory 66 may be any kind of memory, such as for example, L1, L2, or L3 cache or system memory.
- Memory 66 may include volatile memory such as static random-access memory (SRAM) or dynamic RAM (DRAM), or may include non-volatile memory such as flash memory, read only memory (ROM), or other volatile or non-volatile memory.
- SRAM static random-access memory
- DRAM dynamic RAM
- non-volatile memory such as flash memory, read only memory (ROM), or other volatile or non-volatile memory.
- memory 66 may be implemented in a secure hardware element. Alternately, memory 66 may be a hard drive implemented externally to data processing system 60 . In one embodiment, memory 66 may be used to store weight matrices for the ML model.
- User interface 68 may be connected to one or more devices for enabling communication with a user such as an administrator.
- user interface 68 may be enabled for coupling to a display, a mouse, a keyboard, or other input/output device.
- Network interface 72 may include one or more devices for enabling communication with other hardware devices.
- network interface 72 may include, or be coupled to, a network interface card (NIC) configured to communicate according to the Ethernet protocol.
- NIC network interface card
- network interface 72 may implement a TCP/IP stack for communication according to the TCP/IP protocols. Data samples for classification may be input via network interface 72 , or similar interface.
- Various other hardware or configurations for communicating are available.
- Instruction memory 70 may include one or more machine-readable storage media for storing instructions for execution by processor cores 64 .
- both memories 66 and 70 may store data upon which processor cores 64 may operate.
- Memories 66 and 70 may also store, for example, encryption, decryption, and verification applications.
- Memories 66 and 70 may be implemented in a secure hardware element and be tamper resistant.
- Non-transitory machine-readable storage medium including any mechanism for storing information in a form readable by a machine, such as a personal computer, laptop computer, file server, smart phone, or other computing device.
- the non-transitory machine-readable storage medium may include volatile and non-volatile memories such as read only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage medium, flash memory, and the like.
- ROM read only memory
- RAM random access memory
- magnetic disk storage media such as magnetic disks, optical storage medium, flash memory, and the like.
- the non-transitory machine-readable storage medium excludes transitory signals.
Abstract
Description
- This disclosure relates generally to machine learning, and more particularly, to a method for generating a more detailed visualization of a behavior of a machine learning (ML) model.
- Machine learning (ML) is becoming more widely used in many of today's applications, such as applications involving forecasting and classification. Generally, a ML model is trained, at least partly, before it is used. Training data is used for training a ML model. Machine learning models may be classified by how they are trained. Supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning are examples of training techniques.
- Machine learning models have been used to classify images. Machine learning models have also been used in object detection to locate and classify various objects in an image. In object detection, each object in an image may be labeled so that an image may include more than one label. Object detection is useful in automotive applications such as autonomous driving where the automobile needs to locate objects in the images captured by cameras on the automobile. In autonomous driving, for example, object detectors typically need to operate in a low-latency online fashion and to preserve the privacy of the user.
- In ML, improving human interpretability and explainability of results is an important problem. A lack of understanding how a ML model derives its conclusions makes it difficult to verify that the ML model is working as expected and no significant flaws of the model are overlooked. The lack of understanding can cause mistrust and security concerns that hinder the use of ML for important tasks.
- Many different approaches exist to generate visualizations that show the user which parts of the input were the most important for the model to derive its conclusion. When used on a model for image classification for example, these visualizations show the influence of each individual input pixel or groups of pixels on the classification result. Similar visualizations can also be applied to models used for object detection. All existing approaches have limitations that limit their use for explaining model behavior. Specifically, for convolutional neural networks (CNN), several variants of visualization methods have been developed (e.g. GradCAM (gradient-weighted class activation mapping) and AblationCAM) that generate heatmaps showing the most influential areas of the input for a target classification based on the activation maps of a selected convolutional layer of the CNN. These methods are considered to generate good explanations in general and are relatively computationally inexpensive, but their ability to explain model behavior is severely limited by their relatively low resolution.
- The present invention is illustrated by way of example and is not limited by the accompanying figures, in which like references indicate similar elements. Elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale.
-
FIG. 1 illustrates a simplified system for training and using a ML model in accordance with an embodiment. -
FIG. 2 illustrates a flow diagram of a method for generating a higher resolution visualization. -
FIG. 3 illustrates a flowchart of a method for a machine learning model (ML) in accordance with an embodiment. -
FIG. 4 illustrates a data processing system useful for implementing an embodiment of the present invention. - Generally, there is provided, a method for providing a more detailed visualization for explaining the behavior of a ML model. The method first generates a first heat map to find an area in an input sample above a predetermined threshold that indicates the most important area for a specified target classification. The area is cropped from the input sample and upscaled to, for example, the input resolution. A second heatmap is generated of the cropped area to allow a more detailed explanation of the influence this area had on the classification result. Because the area is upscaled, the heatmap can provide a more detailed visualization and provide better insight into why the ML model made the classification it made.
- The method may be used in, for example, classifiers and object detectors.
- In accordance with an embodiment, there is provided, a method for generating a visualization for explaining a behavior of a machine learning (ML) model, the method including: inputting an image into a ML model for an inference operation; generating a first heatmap for the image using a first visualization method; selecting an area of highest attention on the first heatmap using a predetermined threshold; cropping the selected area from the image; upscaling the cropped selected area; generating a second heatmap for the cropped and upscaled selected area of the image; and presenting a final visualization for analysis. Generating the second heatmap for the cropped and upscaled selected area of the image may further include generating the second heatmap for the cropped and upscaled selected area of the image using a second visualization method. The first and second visualization methods may each be the GradCAM visualization method. The first visualization method may be a GradCAM visualization method and the second visualization method may be a visualization method different from the GradCAM visualization method. Upscaling the cropped selected area may further include one of either zooming in the cropped selected area or using the cropped selected area from the image when the image has a higher resolution than the cropped image. Selecting the area of highest attention on the first heatmap using a predetermined threshold may further include selecting multiple areas of highest attention that exceed the predetermined threshold. Generating a first heatmap for the image using a first visualization method may further include generating the first heatmap for the image using GradCAM on a last convolutional layer of a neural network of the ML model. The predetermined threshold may be equal to a 75th percentile of heatmap values. Generating a second heatmap for the cropped and upscaled selected area of the image may further include generating the second heatmap for the cropped and upscaled selected area using GradCAM on a last convolutional layer of a neural network of the ML model.
- In accordance with another embodiment, there is provided, a computer program comprising instructions executable by a processor, for executing a method for generating a visualization for explaining a behavior of a machine learning (ML) model, the executable instructions including: instructions for inputting an image into a ML model for an inference operation; instructions for generating a first heatmap for the image using a first visualization method; instructions for selecting an area of highest attention on the first heatmap using a predetermined threshold; instructions for cropping the selected area from the image; instructions for upscaling the cropped selected area; instructions for generating a second heatmap for the cropped and upscaled selected area of the image; and instructions for presenting a final visualization for analysis. The instructions for generating the second heatmap for the cropped and upscaled selected area of the image may further include instructions for generating the second heatmap for the cropped and upscaled selected area of the image using a second visualization method. The first and second visualization methods may each be the GradCAM visualization method. The first visualization method may be a GradCAM visualization method and the second visualization method may be a visualization method different from the GradCAM visualization method. The instructions for upscaling the cropped selected area may further include one of either zooming in the cropped selected area or using the cropped selected area from an original image when the original image has a higher resolution than the cropped image. The instructions for selecting the area of highest attention on the first heatmap using a predetermined threshold may further include instructions for selecting multiple areas of highest attention that exceed the predetermined threshold. The instructions for generating the first heatmap for the image using a first visualization method may further include instructions for generating the first heatmap for the image using GradCAM on a last convolutional layer of a neural network of the ML model. The predetermined threshold may be equal to a 75th percentile of heatmap values. The instructions for generating a second heatmap for the cropped and upscaled selected area of the image may further include instructions for generating the second heatmap for the cropped and upscaled selected area using GradCAM on a last convolutional layer of a neural network of the ML model.
-
FIG. 1 illustrates a simplified system for training and using a ML model in accordance with an embodiment.System 10 includes labeled set ofML training data 12,model training block 14, and resulting trained MLmodel 16. In one embodiment,system 10 is implemented as a computer program stored on a non-transitory medium comprising instructions that are executable on a processor. - One example embodiment includes a neural network (NN) algorithm used to classify images as trained using a
training data set 12. In another embodiment,training data set 12 may be used for object detection in an image. Various training data sets can be acquired, such as for example, the CIFAR10 data set for image classification and the PASCAL Visual Object Classes (VOC) data set for object classification. A ML algorithm for a vision-based classification problem only returns a single output prediction (a class label) for an input image. A ML algorithm for an object detection problem, which is commonly referred to as an object detector can, on the other hand, return multiple output predictions for a single input image, one output prediction for every identified object. Every output prediction consists of a class label as well as a bounding box that contains the object. For discussion purposes, the ML model is based on a neural network (NN) algorithm and the training data includes a plurality of images. In other embodiments, the ML algorithm and training data may be different. - The ML model is trained in
model training 14 usingtraining data 12 comprising a plurality of labeled images. The training data may include images that are labeled by classes, such as for example, bird, cat, dog, bus, traffic light, etc. If the ML model is to be used as an object detector, the training data includes images with more than one labeled object. After training, the trainedML model 16 can be used for inference operations. In an inference operation, one or more input samples labeled “INPUT SAMPLES” are input intoML Model 16 and an output classification labeled “OUTPUT” is provided by the model. -
FIG. 2 illustrates a flow diagram ofmethod 20 for generating a higher resolution visualization. The currently available methods for providing interpretability and explainability are considered to generate good explanations in general and are relatively computationally inexpensive, but their ability to explain model behavior is severely limited by their low resolution compared to the input resolution. The low resolution is a direct result of the trade-off that is made when selecting a convolutional layer for the visualization. For example, more and more noise is added and heatmaps get less semantically meaningful towards the beginning of the convolutional neural network (CNN) while the resolution is increasingly reduced towards the end of the network. This means that in order to generate the most meaningful visualizations the layer that is typically selected is very close to the end of the network which results in a very low resolution. For example, when using the neural network architecture MobileNetv2, input images may have a resolution of 224×224 pixels. By the time the processing moves to one of the last convolutional layers the resolution of the generated visualizations may be reduced to only 7×7 pixels. This low resolution makes it hard to interpret the visualizations in many cases, especially if smaller objects are involved or the classification decision of the model depends on finer details of the input. - The method disclosed herein improves generated visualizations by focusing on a smaller area of interest so that the full resolution of the visualization method is available for that area.
- Generally, in the method for improving explainability, a first visualization method is used to find an area of interest in an input image for a neural network (i.e. an area that is the most influential for a given target classification as determined by the model). This area of interest is then cropped out of the image and scaled up to the original input resolution. A second visualization method is then used to visualize the influence of each individual input pixel or groups of pixels on the target classification. The resulting visualization can then be shown directly, overlayed over the cropped image or overlayed over the original image to enhance the explainability of the decisions made by the CNN.
- The disclosed embodiments are applicable to all types of ML where images (or video, i.e. a stream of multiple images) are used as input or where the input is treated as if it were an image, for example in image classification or image object detection use cases. For these applications, convolutional neural networks (CNNs) are the most commonly used type of ML models. As mentioned previously, to improve explainability of the model behavior, many different approaches exist to generate visualizations that show the user which parts of the input are the most important for the model to derive its conclusion. In case of an image classifier the visualization shows which parts of the input image are seen as the most important by the model for a given target class (which does not necessarily have to be the actual predicted class).
- One widely used example for these visualization methods is GradCAM (gradient-weighted class activation mapping). GradCAM generates heatmaps showing the most influential areas of the input for a target classification based on a weighted sum of the activation maps of a selected convolutional layer. For weights it uses the gradients for the target class as an estimation for the importance of each activation map for the target class. A trade-off is made when selecting a convolutional layer for GradCAM as the highest resolution is retained towards the beginning of the CNN but as more noise is added the heatmaps get semantically less meaningful. In order to generate the most meaningful visualizations the convolutional layer that is typically selected is very close to the end of the network, i.e. the last convolutional layer.
- As shown in
FIG. 2 , instep 22, an image is input to the ML model. Instep 24, a first GradCAM heatmap is generated to find the area in the input image that is seen by the model as the most important for the classification as the target class (i.e. the area that has the highest model attention for the target class). In one embodiment, the area of interest is chosen by the developer and can be any part of the image. In another embodiment, the area is chosen based on a threshold. For illustration purposes, inFIG. 2 , the selected area is a rectangle and surrounds a sun symbol. The sun symbol is representative of any portion of an image that is of interest for analysis, e.g., a dog, cat, bus, traffic light, etc. The first GradCAM heatmap shows the area with the highest model attention based on a threshold. Atstep 26, the smallest rectangle such that all areas where the heatmap value is greater than or equal to the threshold are inside the rectangle. This threshold can be a fixed value or dynamically calculated based on the heatmap. In experiments, a dynamic threshold equal to the 75th percentile of the heatmap values reliably selected an area of interest that contained the object representing the target class while at the same time being small enough to benefit from the gain in resolution when focusing on the selected area only. As shown insteps step 28, the selected area of highest attention is cropped from the input image and scaled up to the original input resolution of the model using linear interpolation. Using linear interpolation delivers good results but any other method for upscaling is also possible. In many cases the input sample from the dataset has a higher resolution than can be input into the ML model, so the input is actually downscaled before providing the input to the model. In this case, upscaling may not be necessary because the higher resolution input sample can be used. Atstep 30, a second GradCAM heatmap is generated of the cropped and upscaled selected area. The second GradCAM visualization differs from the first visualization in that the whole resolution of the visualization is used only for the selected area of interest to get more information on finer details of the selected area. Instep 32, a final visualization is then presented to the user, which can be either just showing the heatmap itself, overlaying the heatmap over the cropped image or overlaying the heatmap over the original image. The user may be a human user or a computer system. When overlaying the heatmap over the cropped image the heatmap is scaled to the original image size, if necessary. When overlaying the heatmap over the complete original image, the heatmap is scaled to the original width and height of the selected area and only overlayed over this area, alternatively both the first and second heatmap can be combined before overlaying them over the original image. Alternatively, more than one area of highest model attention can be selected, cropped, visualized and combined to one heatmap. - The disclosed embodiments improve the resolution of the visualizations used to visualize the influence or importance of individual pixels or groups of pixels of the input image. This allows the user to visualize finer details that help explain model behavior in cases where the original resolution is not enough or that help to increase confidence in the interpretation of model behavior (e.g. when examining misclassifications there are many cases where the visualizations for correct class and predicted class are very close to each other). The use of the first visualization method to select the area of highest model attention allows this method to automatically select the most important areas of the input image from the perspective of the model.
- The disclosed method can also be used to verify the correctness of generated visualizations in an automated way. Instead of cropping the selected area of highest attention, the area is covered by a patch (e.g. by a solid grey patch) and run through the model again. If the expected significant decrease in confidence of the target class is indeed observed, the visualization is valid, otherwise it's probably not valid.
- The method can also be used to examine the influence of the background itself or other objects in the background on the classification decision of the model. When the cropped area of highest attention is run through the model and confidence in the target class decreases, then there may be a bias in the model towards the target class in combination with a certain background or another object in the background. When the confidence value of the target class increases, there may be a negative bias caused by the background or some other objects in the background.
-
FIG. 3 illustrates a flowchart ofmethod 40 for analyzing a machine learning model (ML) in accordance with an embodiment.Method 40 begins atstep 42. Atstep 42, an image is input into an ML model for an inference operation. Atstep 44, a first heatmap is generated for the image using a first visualization method. Atstep 48, an area of highest attention is selected from the first heatmap using a predetermined threshold. Atstep 50, the selected area is cropped from the image. Atstep 52, the cropped selected area is upscaled. Atstep 54, a second heatmap is generated for the cropped and upscaled selected area of the image. Atstep 56, a final visualization is presented for analysis. -
FIG. 4 illustratesdata processing system 60 useful for implementing an embodiment of the present invention.Data processing system 60 may be implemented on one or more integrated circuits and may be used in an implementation of the described embodiments.Data processing system 60 includesbus 62. Connected tobus 62 is one ormore processor cores 64,memory 66,user interface 68,instruction memory 70, andnetwork interface 72. The one ormore processor cores 64 may include any hardware device capable of executing instructions stored in memory 76 orinstruction memory 70. For example,processor cores 64 may execute the machine learning algorithms used for training and operating the ML model.Processor cores 64 may be used to execute the method for explainability described herein.Processor cores 64 may be, for example, a microprocessor, field programmable gate array (FPGA), application-specific integrated circuit (ASIC), or similar device.Processor cores 64 may be implemented in a secure hardware element and may be tamper resistant. -
Memory 66 may be any kind of memory, such as for example, L1, L2, or L3 cache or system memory.Memory 66 may include volatile memory such as static random-access memory (SRAM) or dynamic RAM (DRAM), or may include non-volatile memory such as flash memory, read only memory (ROM), or other volatile or non-volatile memory. Also,memory 66 may be implemented in a secure hardware element. Alternately,memory 66 may be a hard drive implemented externally todata processing system 60. In one embodiment,memory 66 may be used to store weight matrices for the ML model. -
User interface 68 may be connected to one or more devices for enabling communication with a user such as an administrator. For example,user interface 68 may be enabled for coupling to a display, a mouse, a keyboard, or other input/output device.Network interface 72 may include one or more devices for enabling communication with other hardware devices. For example,network interface 72 may include, or be coupled to, a network interface card (NIC) configured to communicate according to the Ethernet protocol. Also,network interface 72 may implement a TCP/IP stack for communication according to the TCP/IP protocols. Data samples for classification may be input vianetwork interface 72, or similar interface. Various other hardware or configurations for communicating are available. -
Instruction memory 70 may include one or more machine-readable storage media for storing instructions for execution byprocessor cores 64. In other embodiments, bothmemories processor cores 64 may operate.Memories Memories - Various embodiments, or portions of the embodiments, may be implemented in hardware or as instructions on a non-transitory machine-readable storage medium including any mechanism for storing information in a form readable by a machine, such as a personal computer, laptop computer, file server, smart phone, or other computing device. The non-transitory machine-readable storage medium may include volatile and non-volatile memories such as read only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage medium, flash memory, and the like. The non-transitory machine-readable storage medium excludes transitory signals.
- Although the invention is described herein with reference to specific embodiments, various modifications and changes can be made without departing from the scope of the present invention as set forth in the claims below. Accordingly, the specification and figures are to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present invention. Any benefits, advantages, or solutions to problems that are described herein with regard to specific embodiments are not intended to be construed as a critical, required, or essential feature or element of any or all the claims.
- Furthermore, the terms “a” or “an,” as used herein, are defined as one or more than one. Also, the use of introductory phrases such as “at least one” and “one or more” in the claims should not be construed to imply that the introduction of another claim element by the indefinite articles “a” or “an” limits any particular claim containing such introduced claim element to inventions containing only one such element, even when the same claim includes the introductory phrases “one or more” or “at least one” and indefinite articles such as “a” or “an.” The same holds true for the use of definite articles.
- Unless stated otherwise, terms such as “first” and “second” are used to arbitrarily distinguish between the elements such terms describe. Thus, these terms are not necessarily intended to indicate temporal or other prioritization of such elements.
Claims (18)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/302,592 US11688112B2 (en) | 2021-05-07 | 2021-05-07 | Method for generating detailed visualization of machine learning model behavior |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/302,592 US11688112B2 (en) | 2021-05-07 | 2021-05-07 | Method for generating detailed visualization of machine learning model behavior |
Publications (2)
Publication Number | Publication Date |
---|---|
US20220358695A1 true US20220358695A1 (en) | 2022-11-10 |
US11688112B2 US11688112B2 (en) | 2023-06-27 |
Family
ID=83901496
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/302,592 Active 2041-08-14 US11688112B2 (en) | 2021-05-07 | 2021-05-07 | Method for generating detailed visualization of machine learning model behavior |
Country Status (1)
Country | Link |
---|---|
US (1) | US11688112B2 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20230010239A1 (en) * | 2021-07-06 | 2023-01-12 | GM Global Technology Operations LLC | Perception system with attention module for processing visual data |
CN116452923A (en) * | 2023-06-16 | 2023-07-18 | 安徽大学 | Cooperative defense strategy and system for attack resistance |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120328210A1 (en) * | 2010-01-28 | 2012-12-27 | Yissum Research Development Company Of The Hebrew University Of Jerusalem | Method and system for generating an output image of increased pixel resolution from an input image |
US20200357117A1 (en) * | 2018-11-21 | 2020-11-12 | Enlitic, Inc. | Heat map generating system and methods for use therewith |
US11023730B1 (en) * | 2020-01-02 | 2021-06-01 | International Business Machines Corporation | Fine-grained visual recognition in mobile augmented reality |
US20210247735A1 (en) * | 2018-10-29 | 2021-08-12 | Hewlett-Packard Development Company, L.P. | Thermal mapping |
US20210406678A1 (en) * | 2020-06-30 | 2021-12-30 | EMC IP Holding Company LLC | Predicting failures with code flow tensors |
US20220138505A1 (en) * | 2020-10-30 | 2022-05-05 | Cho Ho LAM | Methods, systems, and media for deep neural network interpretation via rule extraction |
US20220147751A1 (en) * | 2020-11-12 | 2022-05-12 | Samsung Electronics Co., Ltd. | Region of interest selection for object detection |
Family Cites Families (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108475331B (en) | 2016-02-17 | 2022-04-05 | 英特尔公司 | Method, apparatus, system and computer readable medium for object detection |
US10586350B2 (en) | 2017-12-03 | 2020-03-10 | Facebook, Inc. | Optimizations for dynamic object instance detection, segmentation, and structure mapping |
US10402692B1 (en) | 2019-01-22 | 2019-09-03 | StradVision, Inc. | Learning method and learning device for fluctuation-robust object detector based on CNN using target object estimating network adaptable to customers' requirements such as key performance index, and testing device using the same |
-
2021
- 2021-05-07 US US17/302,592 patent/US11688112B2/en active Active
Patent Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20120328210A1 (en) * | 2010-01-28 | 2012-12-27 | Yissum Research Development Company Of The Hebrew University Of Jerusalem | Method and system for generating an output image of increased pixel resolution from an input image |
US20210247735A1 (en) * | 2018-10-29 | 2021-08-12 | Hewlett-Packard Development Company, L.P. | Thermal mapping |
US20200357117A1 (en) * | 2018-11-21 | 2020-11-12 | Enlitic, Inc. | Heat map generating system and methods for use therewith |
US11023730B1 (en) * | 2020-01-02 | 2021-06-01 | International Business Machines Corporation | Fine-grained visual recognition in mobile augmented reality |
US20210406678A1 (en) * | 2020-06-30 | 2021-12-30 | EMC IP Holding Company LLC | Predicting failures with code flow tensors |
US20220138505A1 (en) * | 2020-10-30 | 2022-05-05 | Cho Ho LAM | Methods, systems, and media for deep neural network interpretation via rule extraction |
US20220147751A1 (en) * | 2020-11-12 | 2022-05-12 | Samsung Electronics Co., Ltd. | Region of interest selection for object detection |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20230010239A1 (en) * | 2021-07-06 | 2023-01-12 | GM Global Technology Operations LLC | Perception system with attention module for processing visual data |
CN116452923A (en) * | 2023-06-16 | 2023-07-18 | 安徽大学 | Cooperative defense strategy and system for attack resistance |
Also Published As
Publication number | Publication date |
---|---|
US11688112B2 (en) | 2023-06-27 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20210319559A1 (en) | Deep convolutional neural network with self-transfer learning | |
US20210350113A1 (en) | Computer-implemented method, computer program product and system for analysis of cell images | |
Samek et al. | Evaluating the visualization of what a deep neural network has learned | |
Fong et al. | Explanations for attributing deep neural network predictions | |
US20170147941A1 (en) | Subspace projection of multi-dimensional unsupervised machine learning models | |
US11688112B2 (en) | Method for generating detailed visualization of machine learning model behavior | |
CN111052144A (en) | Attribute-aware zero-sample machine vision system by joint sparse representation | |
US20210319340A1 (en) | Machine learning model confidence score validation | |
US11461992B2 (en) | Region of interest selection for object detection | |
CN113837205B (en) | Method, apparatus, device and medium for image feature representation generation | |
CN116686017A (en) | Time bottleneck attention architecture for video action recognition | |
US20230186100A1 (en) | Neural Network Model for Image Segmentation | |
Zhang et al. | Motion-excited sampler: Video adversarial attack with sparked prior | |
US20220076119A1 (en) | Device and method of training a generative neural network | |
US11636331B2 (en) | User explanation guided machine learning | |
US20230040470A1 (en) | Method for generating a detailed visualization of machine learning model behavior | |
Gu et al. | Contextual prediction difference analysis for explaining individual image classifications | |
US20230351203A1 (en) | Method for knowledge distillation and model genertation | |
Alashhab et al. | Precise ship location with cnn filter selection from optical aerial images | |
US20230134508A1 (en) | Electronic device and method with machine learning training | |
US20210365735A1 (en) | Computer-implemented training method, classification method and system and computer-readable recording medium | |
Negri et al. | Examining region-based methods for land cover classification using stochastic distances | |
US20210406693A1 (en) | Data sample analysis in a dataset for a machine learning model | |
US11961314B2 (en) | Method for analyzing an output of an object detector | |
US11699208B2 (en) | Method for watermarking a machine learning model |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: NXP B.V., NETHERLANDS Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:DOLIWA, PETER;REEL/FRAME:056180/0760 Effective date: 20210507 |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |