CN115392483A - Deep learning algorithm visualization method and picture visualization method - Google Patents

Abstract

The invention relates to a depth algorithm visualization method, which comprises the following steps: providing a visualization module comprising a single or a plurality of visualizers and a visualization backend, wherein the visualizers are used for visualization, the visualization backend is used for storing visualization content, and each visualizer is in communication with the single or the plurality of visualization backend; determining the called visualizer and the visualization rear end, and configuring parameters of the visualizer and parameters of the visualization rear end; initializing a visualizer and a visualization back end according to configuration parameters; storing and visualizing a configuration file of a deep learning algorithm; visualizing the training data; visualizing a deep learning model structure diagram; and determining the current deep learning model in a training stage or after training, and performing corresponding visualization. The visualization method can perform visualization analysis on all links of the deep learning algorithm in operation in a uniform calling mode, and helps to quickly analyze the deep learning model. The invention also relates to a picture visualization method.

Description

Deep learning algorithm visualization method and picture visualization method

Technical Field

The invention relates to the technical field of deep learning and visualization, in particular to a deep learning algorithm visualization method and a picture visualization method.

Background

The operation of the deep learning algorithm comprises the processes of data processing, neural network model reasoning, neural network model parameter updating and the like. Each process involves complex calculations, and visual analysis of the results of the calculations produced by each process may help understand and analyze the details of each process and the characteristics of the neural network model.

The operation process of the different deep learning algorithm can involve different input and output and diversified visualization requirements. The existing visualization method mainly has the following defects that firstly, a used visualization tool is designed and realized aiming at a specific deep learning algorithm and is difficult to be reused by other deep learning algorithms; secondly, the existing visualization tool can only visualize basic data such as pictures, texts and the like, and is difficult to directly meet the visualization requirements of the diversification of the deep learning algorithm. A new research idea and solution are needed to solve the above problems.

Disclosure of Invention

The invention aims to provide a deep learning algorithm visualization method and a picture visualization method, wherein the visualization method utilizes a uniform visualizer module for visualization, can perform visualization analysis on each link during the operation of a deep learning algorithm in a uniform calling mode, helps to quickly analyze a deep learning model, and supports various visualization rear ends and stores visualization analysis results into the corresponding visualization rear ends.

In a first aspect of the present invention, to solve the problems in the prior art, the present invention provides a depth algorithm visualization method, including:

providing a visualization module comprising a single or a plurality of visualizers and a visualization backend, wherein the visualizers are used for visualization, the visualization backend is used for storing visualization content, and each visualizer is in communication with the single or the plurality of visualization backend;

determining the called visualizer and the visualization rear end, and configuring parameters of the visualizer and parameters of the visualization rear end;

initializing a visualizer and a visualization back end according to configuration parameters;

storing and visualizing a configuration file of a deep learning algorithm;

visualizing the training data by a corresponding visualizer, determining whether pictures and labels in the training data are correct or not, and storing the pictures and the labels at the visualized back end;

the deep learning model structure diagram is visualized by a corresponding visualizer and is stored at the corresponding visualization back end; and

and if the current state is the training stage of the deep learning model or after the training, carrying out model parameter training and loss calculation, visualizing the loss, the learning rate and the feature map in the training by the corresponding visualizer, and storing the feature map in the corresponding visualization rear end, and visualizing the prediction inference result of the deep learning model and the performance index of the deep learning model by the corresponding visualizer and storing the prediction inference result and the performance index in the corresponding visualization rear end.

In one embodiment of the invention, single or multiple visualizers are determined according to tasks, wherein the visualizers comprise a drawing interface, a backend interface and an OpenMMLab data format visualization interface.

In one embodiment of the invention, the visualizer communicates with the single or multiple visualization back-ends through the back-end interface, so as to call the single or multiple visualization back-ends and store the visualized content in the visualization back-ends.

In an embodiment of the present invention, the OpenMMLab data format visualization interface is configured to render and store data samples in OpenMMLab format.

In one embodiment of the invention, the drawing interface is configured to draw detection boxes, masks, text, points, lines, and feature maps.

In one embodiment of the invention, the predictive inference result of the visualized deep learning model comprises drawing one or more of a detection box, a mask, text, a point and a line by using a drawing interface of the visualizer.

In an embodiment of the invention, when the data format of the prediction inference result of the deep learning model is OpenMMLab data format, one or more of a detection box, a mask, a text, a point and a line are drawn by the OpenMMLab data format visualization interface, and the drawing result is displayed by a corresponding visualizer or stored in the OpenMMLab data format visualization interface.

In a second aspect of the present invention, the present invention provides a picture visualization method, including:

providing a visualization module comprising a single or a plurality of visualizers and a visualization backend, wherein the visualizers are used for visualization, the visualization backend is used for storing visualization content, and each visualizer is in communication with the single or the plurality of visualization backend;

determining the called visualizer and the visualization rear end, and configuring parameters of the visualizer and parameters of the visualization rear end;

initializing a visualizer and a visualization back end according to configuration parameters;

preparing an original picture to be visualized and related picture data;

calling a drawing interface provided by a visualizer to draw an original picture to be drawn; and

and storing the drawn result to a corresponding visualization back end through a visualizer.

In one embodiment of the invention, the visualizer comprises a rendering interface, a backend interface and an OpenMMLab data format visualization interface;

the visualizer communicates with one or more visualization backend through the backend interface so as to call one or more visualization backend and store the visualized content in the visualization backend;

the OpenMMLab data format visualization interface is used for drawing and storing data samples in an OpenMMLab format.

In one embodiment of the invention, one or more of a detection box, a mask, text, a point, a line and a feature map are drawn using a drawing interface of the visualizer.

The invention has at least the following beneficial effects: the invention discloses a deep learning algorithm visualization method and a picture visualization method, wherein the visualization method utilizes a unified visualizer module for visualization, can perform visualization analysis on each link during the operation of a deep learning algorithm in a unified calling mode, helps to quickly analyze a deep learning model, and simultaneously supports various visualization rear ends and stores visualization analysis results into the corresponding visualization rear ends; the visualization module is utilized to visualize the deep learning algorithm, help to quickly analyze the deep learning model, unify the use modes of the visualization functions of various deep learning algorithms, support the arbitrary combination and expansion of visualization analysis and visualization back-end storage, improve the flexibility of the visualizer, and enable a user to conveniently analyze each process of the deep learning; the visualizer of the deep learning algorithm supports various structured and unstructured data such as visual pictures, texts, a deep learning model structure diagram, a detection box, a segmentation mask and the like; the method supports various visual back ends, has expansibility, and supports the storage and display of visual contents in the back ends; the web end visualization and storage display functions are supported; and supporting the expansion of the visualizer and the visualization back end.

Drawings

To further clarify the above and other advantages and features of embodiments of the present invention, a more particular description of embodiments of the invention will be rendered by reference to the appended drawings. It is appreciated that these drawings depict only typical embodiments of the invention and are therefore not to be considered limiting of its scope.

FIG. 1 is a schematic diagram illustrating a connection relationship between a visualizer and a visualization backend in a visualization module according to an embodiment of the present invention;

FIG. 2 illustrates a process flow for deep learning algorithm visualization using a visualization module according to an embodiment of the present invention; and

fig. 3 illustrates a process for visualizing picture data using a visualization module according to an embodiment of the present invention.

Detailed Description

It should be noted that the components in the figures may be exaggerated and not necessarily to scale for illustrative purposes.

In the present invention, the embodiments are only intended to illustrate the aspects of the present invention, and should not be construed as limiting.

In the present invention, the terms "a" and "an" do not exclude the presence of a plurality of elements, unless otherwise specified.

It is further noted herein that in embodiments of the present invention, only a portion of the components or assemblies may be shown for clarity and simplicity, but those of ordinary skill in the art will appreciate that, given the teachings of the present invention, required components or assemblies may be added as needed in a particular scenario.

It is also noted herein that, within the scope of the present invention, the terms "same", "equal", and the like do not mean that the two values are absolutely equal, but allow some reasonable error, that is, the terms also encompass "substantially the same", "substantially equal".

It should also be noted herein that in the description of the present invention, the terms "central", "longitudinal", "lateral", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate orientations or positional relationships based on those shown in the drawings, and are only for convenience of description and simplicity of description, and do not indicate or imply that the device or element being referred to must have a particular orientation, be constructed and operated in a particular orientation, and thus, should not be construed as limiting the present invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.

The numbering of the steps of the methods of the present invention does not limit the order of execution of the steps of the methods. Unless specifically stated, the method steps may be performed in a different order.

In order to meet the input and output requirements and diversified visualization requirements related to the running process of the deep learning algorithm, the visualization module is provided, has a uniform use mode, supports visualization of various data and supports storage in back-end software. The visualization module is used for visualization of deep learning tasks and visualization in other application scenes, and redundant visualization codes in a code library of each deep learning algorithm can be greatly reduced. The visualization module supports the visualization and storage display functions of the web end, also supports the expansion of the visualizer and the visualization rear end, and adds the visualizer and the visualization rear end according to task requirements.

Fig. 1 shows a schematic connection relationship between a visualizer and a visualization backend in a visualization module according to an embodiment of the present invention.

The Visualizer (Visualizer) is a module which can be applied to various visual tasks, and various visual requirements can be realized by calling a visual interface. The Visualization Backend (Visualization Backend) is a storage module which can support a plurality of Backend, and the Visualization Backend can store the analysis result of the visualizer into the designated Backend through the storage module.

As shown in fig. 1, the visualization module includes a plurality of single or multiple visualizers and a visualization backend, each of which may be in communication with any single or multiple visualization backend. Visualizers contain multiple categories, such as detection local visualizers, detection wandb visualizers, and the like. The visualization backend also contains multiple categories such as local visualization backend, tensorboard visualization backend, wandb visualization backend. The wandb visual rear end is a visual rear end capable of displaying and recording various format data in a browser, all data can be sent to a cloud end, and the data can be conveniently checked at any time and any place. the tensorboard visual backend and the wandb visual backend are similar in function, except that the data format and display function are different.

The visualizer comprises a drawing interface, a back-end interface and an OpenMMLab data format visualization interface. The drawing interface is used for drawing detection boxes, masks, text, points, lines, feature maps and the like. The number of drawing interfaces is single or plural. The back-end interface is used for communication between the visualizer and the visualization back-end, and the visualization back-end or the visualization back-ends can be called by calling the interface once through the visualizer. Each visualizer communicates with one or more visualization backend through one backend interface, so as to call one or more visualization backend, and send the visualized content to the visualization backend for storage.

And drawing and storing the data sample in the OpenMMLab format by the OpenMMLab data format visualization interface. And the picture drawn by the OpenMMLab data format visualization interface is directly displayed on the visualizer or stored in the OpenMMLab data format visualization interface.

The details of the OpenMMLab data format visualization interface are as follows:

and (3) interface parameter description: and drawing and storing the input custom data sample.

List of interface parameters:

name: identifier

image: picture data with drawing

data _ sample input custom data sample

Interface return value: and if not, directly displaying or storing the drawn picture.

Optionally, since the wandb visualizer cannot acquire the drawn picture data through the OpenMMLab data format visualization interface, the wandb visualizer has specificity because the wandb visualizer can only be connected with the wandb visualization back end, but the wandb visualizer still has a consistent use mode with other visualizers.

FIG. 2 shows a process for deep learning algorithm visualization using a visualization module according to an embodiment of the invention.

As shown in fig. 2, the process of performing deep learning algorithm visualization by using the visualization module provided by the present invention includes:

step 1, determining a called visualizer and a visualization rear end according to a task, and configuring visualizer parameters and visualization rear end parameters. The visualizer is used for visualization, and the visualization back-end is used for storing the analysis result (visualization content) of the visualizer. The number of the visualizers and the visualization back ends can be single or multiple, each visualizer can be communicated with any single or multiple visualization back ends, and the called visualizers and the called visualization back ends are determined according to specific visualization tasks.

And 2, initializing the visualizer and the visualization back end according to the configuration parameters.

And 3, storing and visualizing the configuration file of the deep learning algorithm. And storing and visualizing the configuration file of the deep learning algorithm by the corresponding visualizer and the visualization back end, wherein the configuration file is a training configuration parameter of the deep learning algorithm.

And 4, visualizing the training data by the corresponding visualizer, determining whether the pictures and the labels in the training data are correct or not, and storing the pictures and the labels at the corresponding visualization rear end.

And 5, visualizing the deep learning model structure chart by a corresponding visualizer and storing the deep learning model structure chart at the corresponding visualization rear end.

And 6, determining whether the current state is a training stage of the deep learning model or after training, if so, carrying out model parameter training and loss calculation, visualizing the loss, the feature map, the learning rate and the like in the training by a corresponding visualizer and storing the loss, the feature map, the learning rate and the like in the training at the corresponding visualization rear end, and if so, visualizing the prediction inference result of the deep learning model and the performance index of the deep learning model by the corresponding visualizer and storing the prediction inference result and the performance index in the corresponding visualization rear end. Visualizing the predictive reasoning result for the deep learning model includes drawing one or more of a detection box, a mask, text, a point, and a line with a drawing interface of the visualizer. And calling a plurality of drawing interfaces of a plurality of visualizers in any sequence and times to perform superposition drawing on the prediction inference result of the deep learning model. If the data format of the prediction inference result of the deep learning model is the OpenMMLab data format, drawing one or more of a detection box, a mask, a text, a point and a line by using the OpenMMLab data format visualization interface, and directly displaying the drawing result by using a corresponding visualizer or storing the drawing result in the OpenMMLab data format visualization interface.

Fig. 3 illustrates a process for visualizing picture data using a visualization module according to an embodiment of the present invention.

The visualization module can visualize the deep learning algorithm and can also directly visualize the pictures. As shown in fig. 3, the process of using the visualization module to visualize the original picture needing to be visualized is as follows:

step 1, determining a called visualizer and a visualization rear end according to a task, and configuring visualizer parameters and visualization rear end parameters. The visualization back end may be configured singly or plurally.

And 2, initializing the visualizer and the single or multiple visualization back ends according to the configuration parameters.

And 3, preparing an original picture to be visualized and related picture data, such as picture labels, detection frames and the like.

And 4, calling a single drawing interface or a plurality of drawing interfaces provided by the visualizer, and calling the drawing interfaces in any sequence and times to perform superposition drawing on all data. The drawing interface draws one or more of a detection box, a mask, text, a point, a line, a feature map, and the like. The dotted line in fig. 3 indicates that any object may be drawn as well as a plurality of times.

And 5, after all the drawing functions are finished, storing the result in different visualization back ends in the visualizer. The results are stored to different visualization backend through the backend interface.

The visualization method, the picture visualization method and the visualization module of the deep learning algorithm obtained by the technical scheme of the invention can be used for realizing the following technical effects in the deep learning field and the picture visualization field: the method has the advantages of providing abundant ready-to-use drawing interfaces, reducing the difficulty of the visual deep learning algorithm, and meeting the visual requirements of different input and output and diversification in the running process of different deep learning algorithms. The principle is as follows: the visualizer specifies the calling interface and the input parameters, and a user can quickly understand the functional meaning of the interface through the interface name. The system has the advantages that a large number of commonly used bottom layer drawing functions are provided functionally, a user can call various drawing interfaces with strong functions and excellent display effects without modification, meanwhile, openMMLab data format high-level drawing interfaces related to tasks, common analysis and evaluation index storage interfaces in various training processes and the like are also provided, the user can easily perform visualization in the deep learning algorithm process, and the visualization requirements of various tasks can be met by easily expanding the system on the basis of the prior art.

Embodiments may be provided as a computer program product that may include one or more machine-readable media having stored thereon machine-executable instructions that, when executed by one or more machines such as a computer, network of computers, or other electronic devices, may result in the one or more machines carrying out operations in accordance with embodiments of the present invention. The machine-readable medium may include, but is not limited to, floppy diskettes, optical disks, CD-ROMs (compact disc read-only memories), and magneto-optical disks, ROMs (read-only memories), RAMs (random access memories), EPROMs (erasable programmable read-only memories), EEPROMs (electrically erasable programmable read-only memories), magnetic or optical cards, flash memory, or other type of media/machine-readable medium suitable for storing machine-executable instructions.

Moreover, embodiments may be downloaded as a computer program product, wherein the program may be transferred from a remote computer (e.g., a server) to a requesting computer (e.g., a client) by way of one or more data signals embodied in and/or modulated by a carrier wave or other propagation medium via a communication link (e.g., a modem and/or network connection). Accordingly, a machine-readable medium as used herein may include, but is not required to be, such a carrier wave.

The invention has at least the following beneficial effects: the deep learning algorithm visualization method utilizes a uniform visualizer module for visualization, can perform visual analysis on each link during the operation of the deep learning algorithm in a uniform calling mode, helps to quickly analyze the deep learning model, and supports various visual back ends and stores visual analysis results into the corresponding visual back ends; the visualization module is used for visualizing the deep learning algorithm, helps to quickly analyze the deep learning model, unifies the use modes of the visualization functions of various deep learning algorithms, supports the arbitrary combination and expansion of visualization analysis and visualization back-end storage, improves the flexibility of the visualizer, and enables a user to conveniently analyze each process of deep learning; the visualizer of the deep learning algorithm supports various structured and unstructured data such as visual pictures, texts, a deep learning model structure diagram, a detection box, a segmentation mask and the like; the system supports various visual back ends, has expansibility, and supports the storage and display of visual contents in the back ends; the web end visualization and storage display functions are supported; and supporting the expansion of the visualizer and the visualization back end.

Although some embodiments of the present invention have been described herein, those skilled in the art will appreciate that they have been presented by way of example only. Numerous variations, substitutions and modifications will occur to those skilled in the art in light of the teachings of the present invention without departing from the scope thereof. It is intended that the following claims define the scope of the invention and that methods and structures within the scope of these claims and their equivalents be covered thereby.

Claims (10)

1. A depth algorithm visualization method, comprising:

providing a visualization module comprising a single or a plurality of visualizers and a visualization backend, wherein the visualizers are used for visualization, the visualization backend is used for storing visualized content, and each visualizer is in communication with the single or the plurality of visualization backend;

determining a called visualizer and a visualization back end, and configuring parameters of the visualizer and parameters of the visualization back end;

initializing a visualizer and a visualization back end according to configuration parameters;

storing and visualizing a configuration file of a deep learning algorithm;

visualizing the training data by a corresponding visualizer, determining whether pictures and labels in the training data are correct or not, and storing the pictures and the labels at a visualized rear end;

visualizing the deep learning model structure chart by a corresponding visualizer and storing the deep learning model structure chart at the corresponding visualization back end; and

and if the current state is the training stage of the deep learning model or after the training, carrying out model parameter training and loss calculation, visualizing the loss, the learning rate and the feature map in the training by the corresponding visualizer, and storing the feature map in the corresponding visualization rear end, and visualizing the prediction inference result of the deep learning model and the performance index of the deep learning model by the corresponding visualizer and storing the prediction inference result and the performance index in the corresponding visualization rear end.

2. The depth algorithm visualization method according to claim 1, wherein the one or more visualizers are determined according to tasks, wherein the visualizers comprise a drawing interface, a backend interface and an OpenMMLab data format visualization interface.

3. The deep learning algorithmic visualization method of claim 2 wherein the visualizer communicates with the single or multiple visualization backend through the backend interface to invoke the single or multiple visualization backend to store the visualized content to the visualization backend.

4. The deep learning algorithm visualization method according to claim 2, wherein the OpenMMLab data format visualization interface is configured to render and store OpenMMLab-formatted data samples.

5. The deep learning algorithm visualization method according to claim 2, wherein the drawing interface is configured to draw a detection box, a mask, text, a point, a line, and a feature map.

6. The deep learning algorithm visualization method according to claim 2, wherein the predictive inference result of the visualized deep learning model comprises drawing one or more of a detection box, a mask, a text, a point and a line with a drawing interface of a visualizer.

7. The deep learning algorithm visualization method according to claim 2, wherein when the data format of the prediction inference result of the deep learning model is OpenMMLab data format, one or more of a detection box, a mask, a text, a point and a line are drawn by the OpenMMLab data format visualization interface, and the drawing result is displayed by a corresponding visualizer or stored in the OpenMMLab data format visualization interface.

8. A picture visualization method is characterized by comprising the following steps:

providing a visualization module comprising a single or a plurality of visualizers and a visualization backend, wherein the visualizers are used for visualization, the visualization backend is used for storing visualization content, and each visualizer is in communication with the single or the plurality of visualization backend;

determining a plurality of called visualizers and visualization rear ends, and configuring parameters of the visualizers and parameters of the visualization rear ends;

initializing a visualizer and a visualization back end according to configuration parameters;

preparing an original picture to be visualized and related picture data;

calling a drawing interface provided by a visualizer to draw an original picture to be drawn; and

and storing the drawn result to a corresponding visualization rear end through a visualizer.

9. The picture visualization method according to claim 8, wherein the visualizer comprises a rendering interface, a backend interface, and an OpenMMLab data format visualization interface;

the visualizer communicates with one or more visualization back ends through the back-end interface so as to call the one or more visualization back ends and send the content needing visualization to the visualization back ends for storage;

the OpenMMLab data format visualization interface is used for drawing and storing data samples in an OpenMMLab format.

10. The picture visualization method according to claim 9, wherein one or more of a detection box, a mask, a text, a point, a line and a feature map are drawn by using a drawing interface of the visualizer.

Images