WO2022007434A1

WO2022007434A1 - Visualization method and related device

Info

Publication number: WO2022007434A1
Application number: PCT/CN2021/082348
Authority: WO
Inventors: 朱雁博
Original assignee: 上海商汤智能科技有限公司
Priority date: 2020-07-09
Filing date: 2021-03-23
Publication date: 2022-01-13
Also published as: JP2022543180A; KR20220011134A; CN111797294A

Abstract

A visualization method and a related device. The method comprises: a user equipment sending a visualization request to a cloud server, wherein the visualization request is used for requesting visualization of target training data of a target deep learning model; the user equipment receiving a visualization response sent by the cloud server, wherein the visualization response carries visualization information of the target training data; and the user equipment displaying the visualization information. By means of the present method, training data of a deep learning model can be visualized, thereby facilitating the timely determination of the feasibility of the current training strategy, and providing a basis for decisions such as stopping early.

Description

Visualization method and related equipment

This application claims the priority of the Chinese patent application with the application number of 202010656553.8 and the application title of "Visualization Method and Related Equipment" filed with the China Patent Office on July 9, 2020, the entire contents of which are incorporated into this application by reference.

technical field

The present application relates to the field of computer technology, and in particular, to a visualization method and related equipment.

Background technique

In recent years, deep learning models have been widely used in the field of image and video processing. Generally, deep learning models are obtained through training. However, the training process of deep learning models involves complex and esoteric calculations, and needs to be implemented through multiple iterations. The training time is long, and developers can only evaluate the performance of the deep learning model after training.

SUMMARY OF THE INVENTION

Embodiments of the present application provide a visualization method and related equipment, which are used to visualize training data of a deep learning model.

In a first aspect, an embodiment of the present application provides a visualization method, which is applied to a user equipment, and the method includes:

sending a visualization request to the cloud server, where the visualization request is used to request visualization of the target training data of the target deep learning model;

receiving a visualization response sent by the cloud server, where the visualization response carries the visualization information of the target training data;

The visualization information is displayed.

In a second aspect, an embodiment of the present application provides a visualization apparatus, which is applied to a user equipment, and the apparatus includes:

a sending unit, configured to send a visualization request to the cloud server, where the visualization request is used to request visualization of the target training data of the target deep learning model;

a receiving unit, configured to receive a visualization response sent by the cloud server, where the visualization response carries the visualization information of the target training data;

a display unit for displaying the visual information.

In a third aspect, an embodiment of the present application provides a visualization method, which is applied to a cloud server, and the method includes:

receiving a visualization request from the user equipment, where the visualization request is used to request visualization of the target training data of the target deep learning model;

obtaining target training data corresponding to the visualization request;

Preprocessing the target training data to obtain visualization information;

A visualization response is sent to the user equipment, and the visualization response carries the visualization information.

In a fourth aspect, an embodiment of the present application provides a visualization device, which is applied to a cloud server, and the device includes:

a receiving unit, configured to receive a visualization request from the user equipment, where the visualization request is used to request visualization of the target training data of the target deep learning model;

an obtaining unit for obtaining target training data corresponding to the visualization request;

a determining unit for preprocessing the target training data to obtain visualization information;

A sending unit, configured to send a visualization response to the user equipment, where the visualization response carries the visualization information.

In a fifth aspect, the present application provides a computer device comprising a processor, a memory, a communication interface, and one or more programs, wherein the one or more programs are stored in the memory and configured by The above-mentioned processor is executed, and the above-mentioned program includes instructions for executing the steps in the method described in the first aspect or the third aspect of the embodiments of the present application.

In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for electronic data exchange, wherein the computer program enables a computer to execute the computer program as described in the first embodiment of the present application. Some or all of the steps described in the method of the first aspect or the third aspect.

In a seventh aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute as implemented in the present application. For example, some or all of the steps described in the method described in the first aspect or the third aspect. The computer program product may be a software installation package.

It can be seen that, in this embodiment of the present application, the user equipment first sends a visualization request to the cloud server, then receives the visualization response sent by the cloud server, the visualization response carries the visualization information of the target training data, and finally displays the visualization information. The visualization information of the data can intuitively understand the training status of the deep learning model, which helps to judge the feasibility of the current training strategy in time, and provides a basis for decision-making such as early stopping.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.

1 is a schematic diagram of the architecture of a visualization system provided by an embodiment of the present application;

2 is a schematic flowchart of a visualization method provided by an embodiment of the present application;

3 is a schematic structural diagram of a computer device provided by an embodiment of the present application;

FIG. 4 is a schematic structural diagram of a visualization device provided by an embodiment of the present application;

FIG. 5 is a schematic structural diagram of another visualization apparatus provided by an embodiment of the present application.

detailed description

In order to make those skilled in the art better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only Embodiments are part of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

Each of them will be described in detail below.

The terms "first", "second", "third" and "fourth" in the description and claims of the present invention and the accompanying drawings are used to distinguish different objects, rather than to describe a specific order. . Furthermore, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or modules is not limited to the listed steps or modules, but optionally also includes unlisted steps or modules, or optionally also includes Other steps or modules inherent to these processes, methods, products or devices.

Reference herein to an "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present invention. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor a separate or alternative embodiment that is mutually exclusive of other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

Please refer to FIG. 1 . FIG. 1 is a schematic diagram of the architecture of a visualization system provided by an embodiment of the application, where the visualization system includes a supercomputing cluster, a cloud server, a cloud storage, and user equipment. It should be noted that the shapes and numbers of supercomputing clusters, cloud servers, cloud storages, and user equipment shown in FIG. 1 are only for examples, and do not constitute limitations to the embodiments of the present application. Among them, the visualization system can realize the visualization of multi-dimensional and multi-class intermediate training data based on javascript and svg technology, and supports fast and efficient visualization rendering of a large amount of intermediate training data.

Among them, the visualization system is based on the kubernetes container orchestration system, which provides stable visualization system deployment at the production level and provides dynamic and controllable service capabilities. The kubernetes container is used to manage containerized applications on multiple hosts in the cloud service.

The cloud server provides visualization services, and the visualization services are used to perform data operations such as preprocessing on the acquired intermediate training data, and visualize and render the intermediate training data after data operations to obtain visualization information.

Optionally, the cloud server may further provide at least one of the following services: a service gateway, a training management service, a data storage service, and a permission service.

Wherein, the service gateway can be used as the entrance of at least one of the following services: the entrance of training management service, data storage service, data visualization service and authority service.

Optionally, the service gateway can be an application.

Optionally, the service gateway may have a current limiting function.

Among them, the training management service can provide at least one of the following services: monitoring whether the training tasks of the deep learning model are completed, recording, indexing and searching for abnormal training tasks, comparing the training process of multiple training tasks of the same deep learning model, and Share the training results of deep learning models.

The data storage service can provide at least one of the following services: data persistence, using data threads to store intermediate training data generated by deep learning model training, and using cache to store intermediate training data generated by deep learning model training.

Among them, the rights management service is used to ensure the security of the visualization system, and the visualization system can only be used after passing the rights management service. In addition, shared training is also authenticated through the rights management service, which supports unified authentication account login and/or ordinary There are two types of login methods to register.

Among them, the cloud server can interact with the developer toolkit through HyperText Transfer Protocol (HTTP), Google Remote Procedure Call (GRPC) protocol or other protocols.

Among them, the cloud server can directly store the training data in the cloud storage.

Among them, the supercomputing cluster provides development tools such as python and other programming language toolkits by running the developer toolkit for users to use with tensorflow, pytorch, caffe frameworks, etc., and/or open a new thread as a dedicated thread for data uploading, and The intermediate training data generated by the training of the deep learning model is uploaded to the cloud server through the HTTP communication protocol and the GRPC protocol on the dedicated thread for data uploading. The uploaded intermediate training data types can include vectors, scalars, pictures, videos, audios, etc. Data is stored in the developer kit's data cache.

Among them, opening a new thread as a dedicated thread for data uploading can ensure the computing speed of the deep learning model.

Among them, cloud storage can optionally be distributed storage, which is used for unified management of intermediate training data generated by deep learning models, providing massive storage functions for intermediate training data, and as the amount of intermediate training data increases, it can dynamically The capacity can be expanded to meet the storage requirements of intermediate training data and provide follow-up guarantee for the growth of visualization business.

Optionally, the cloud storage may also be data block-level cloud storage, file-level cloud storage, object-level cloud storage and/or other forms of cloud storage.

The user equipment may communicate with the cloud server through preset application software, or communicate with the cloud server through a preset network address.

Optionally, the user equipment may include various handheld devices with communication functions, vehicle-mounted devices, wearable devices, computing devices or other processing devices connected to the wireless modem, and various forms of user equipment (User Equipment, UE), Mobile station (Mobile Station, MS), terminal device (terminal device) and so on.

Please refer to FIG. 2. FIG. 2 is a schematic flowchart of a visualization method provided by an embodiment of the present application, which is applied to the above-mentioned user equipment and cloud server, and specifically includes the following steps:

Step 201: The user equipment sends a visualization request to the cloud server, where the visualization request is used to request visualization of the target training data of the target deep learning model.

Step 202: The cloud server receives a visualization request from the user equipment, where the visualization request is used to request visualization of the target training data of the target deep learning model.

In a possible implementation manner, the visualization request carries the training task identifier of the target deep learning model. After receiving the visualization request, the cloud server can obtain the relevant training data of the training task based on the training task identifier of the target deep learning model. As an example, the stored training data related to the training tasks of each deep learning model may be associated with the training task identifier of the deep learning model. For example, training data may be stored according to training tasks, and relevant training data of different training tasks are stored in different locations. For another example, training data may be stored according to deep learning models or data types.

Among them, the data type can be a two-dimensional heat map, a three-dimensional histogram, a feature map, a call sequence diagram, a scalar line graph, a directed acyclic graph, and so on.

Among them, different data types correspond to different visualization icons.

Among them, the training tasks are all located on the supercomputing cluster, and the training of the deep learning model can be accelerated by using the central processing unit of the supercomputing cluster.

At this time, optionally, the training data may be stored in association with the task identifier of the training task to which the training data belongs, and so on, which is not limited in this embodiment of the present disclosure.

In a possible implementation, the visualization request carries the training task identification of the target deep learning model and the index information of the target training data, and the index information is used to find the target training data to determine the storage location of the target training data. The index information may be implemented in multiple ways. For example, the index information may include any one or more of data creation time, data type, and data label. For another example, the index information includes data type and data label.

Among them, the same data type can have multiple data labels, and the data labels can be customized.

For example, if the data type is a scalar line chart, the data labels can be exact values, loss values, and so on.

The visualization request may be a request to sample all the data in the intermediate data set corresponding to the index information or randomly sample the data in the intermediate data set according to the training task identifier and the index information.

The target deep learning model includes at least one deep neural network, and the target training data may be intermediate data generated by one of the deep neural networks in the target deep learning model, or may be intermediate data generated by multiple deep neural networks in the target deep learning model. .

Optionally, the visualization request may be a request to visualize the intermediate data generated by the target deep learning model in the first period of time, wherein the duration of the first period of time may be other values such as 3min, 5min, 9min, and 15min, and the termination time of the first moment may be is the current time, it can be earlier than the current time, or it can be later than the current time.

Among them, the target training data is stored in cloud storage.

In a possible implementation manner, the target training data includes at least one of the following: model performance change trend information, model loss trend information, model parameter distribution information, model processing intermediate results, model structure information, current progress information of model training, Comparing information for different trainings of the same model, and time series of scheduling time information for various processes.

In a possible implementation manner, when the target training data is at least one of model performance change trend information, model loss trend information, current progress information of model training, and comparison information for different trainings of the same model, the visualization The category of information is a scalar line chart.

Wherein, in the case where the target training data is the model performance change trend information, it can be determined whether the performance change trend satisfies the first preset condition based on the scalar line graph, and when the first preset condition is met, the model training is stopped, and when the first preset condition is met, the model training is stopped. When the first preset condition is met, model training is continued.

Wherein, when the target training data is the model loss trend information, it can be determined whether the loss trend satisfies the second preset condition based on the scalar line graph, and when the second preset condition is met, the model training is stopped, and when the preset condition is not met, the model training is stopped. Continue model training if conditions are met.

In the case where the target training data is the current progress information of model training, it can be determined whether the training progress meets the preset progress based on the scalar line graph, and if the preset progress is met, continue model training, and if the preset conditions are not met case, stop model training.

Among them, when the target training data is the comparison information of different trainings of the same model, it can be determined whether the features extracted by different trainings are the same based on the scalar line graph. If the features are the same, continue the model training. If the features are different , stop model training.

Wherein, when the target training data is model parameter distribution information, the category of visualization information is a three-dimensional histogram, and it can be determined whether the model parameter distribution is abnormal through the three-dimensional histogram. If the model parameter distribution is abnormal, the model training is stopped.

Among them, when the target training data is the intermediate result of the model processing, the category of the visual information is the feature map, and through the feature map, it can be determined whether the features extracted by the model are correct, and if the extracted features are wrong, rebuild the model.

Among them, when the target training data is the model structure information, the category of the visualization information is a directed acyclic graph, and the directed acyclic graph can determine whether the structure of the model is correct, and if the structure of the model is wrong, rebuild the The structure of the model.

Among them, the architecture of the deep learning model supported by the directed acyclic graph can be a standard architecture such as Open Neural Network Exchange (ONNX), or other types of architectures, in which ONNX does not need to perform intermediate training data. Do data processing to obtain a directed acyclic graph, while some architectures need to preprocess the intermediate training data to obtain a directed acyclic graph.

Among them, when the target training data is the scheduling time information of various processes, the types of visualization information are the calling sequence diagrams of various processes, and the time-consuming situation of each operator can be determined through the calling sequence diagram. When the time consumption is greater than the preset time, the operators whose time consumption is greater than the preset time are optimized.

Among them, in the case where the target training data is an intermediate result of model processing, the category of the visualization information is a two-dimensional heat map. Through the heat map, it can be determined whether the features extracted by the model are correct. Build the model.

Among them, the feature map and the heat map are different representations of the target training data for the model to process the intermediate results.

The types of target training data include at least one of the following: scalar, vector, picture, video, and audio.

Step 203: The cloud server obtains target training data corresponding to the visualization request.

In a possible implementation manner, the acquiring target training data corresponding to the visualization request includes:

Search the index database based on the training task identifier and index information carried in the visualization request to obtain the storage location information of the target training data;

The target training data stored in the cloud storage is acquired based on the storage location information of the target training data.

Wherein, a list of intermediate training data is obtained based on the training task identification and index information, and the intermediate training data is obtained in cloud storage based on the list of intermediate training data.

Optionally, the visualization request carries the identification information of the target training data of the target deep learning model, and the identification information is used by the cloud server to obtain the target training data from the cloud storage; wherein, the identification information is preset, and the identification Information is unique in the cloud storage.

For example, the training tasks of the target deep learning model include training task A and training task B, the training data generated by training task A includes training data A1 and training data A2, and the training data generated by training task B includes training data B1 and training data B2, training data A1, training data A2, training data B1 and training data B2 are all stored in the cloud platform, the identification information of training data A1 is 1, the identification information of training data A2 is 2, and the identification information of training data B1 is 3 , the identification information of the training data B2 is 4, if the identification information is 1, the target training data obtained by the cloud server from the cloud storage is the training data A1; if the identification information is 2, the target training data obtained by the cloud server from the cloud storage The data is training data A2; if the identification information is 3, the target training data obtained by the cloud server from the cloud storage is training data B1; if the identification information is 4, the target training data obtained by the cloud server from the cloud storage is training data B2.

Step 204: The cloud server preprocesses the target training data to obtain visualization information.

Step 205: The cloud server sends a visualization response to the user equipment, where the visualization response carries the visualization information.

Step 206: The user equipment receives a visualization response sent by the cloud server, where the visualization response carries the visualization information of the target training data.

Step 207: The user equipment displays the visualized information.

In a possible implementation manner, the visualization information includes at least one of the following categories: directed acyclic graphs, three-dimensional histograms, call sequence diagrams of various processes, feature maps, two-dimensional heat maps, and scalar line graphs.

Optionally, when the category of the visualization information is a directed acyclic graph, the visualization information includes the following information: topology information of at least a part of the target deep learning model, wherein at least a part of the target deep learning model includes: multiple modules and/or multiple operators of the target deep learning model; resource occupation information of each operator in at least one operator included in the target deep learning model.

Optionally, the topology information of multiple modules includes at least one of the following: identification information of the multiple modules, dependencies between the multiple modules, data size of each module in the multiple modules, Information of at least one operator included in each of the modules.

The identification information of the module is unique and can be preset.

Optionally, the information of the operators includes at least one of the following: identification information of the operators, dependencies between operators, and data size of the operators.

Among them, the dependency relationship can be a sequential relationship or a parallel relationship.

Optionally, the resource occupation information is determined by the cloud server based on at least one of the data type of the operator, the input data information of the operator, and the output data information of the operator.

The input of the operator can be a picture, video, audio, scalar, vector, etc., and different inputs correspond to different dimensions. For example, a picture is equivalent to a two-dimensional matrix, and an audio is equivalent to a one-dimensional matrix.

Among them, after the operation of the operator, the dimension of the input information of the operator will change.

The data type of the operator can be a double-precision type or a single-precision type.

Among them, the operator can be at least one of the following: convolution, batch normalization (BatchNorm, BN), full link, pooling, matrix multiplication and division, dropout (DropOut), activation, etc.

The input data information may be the size of the input data, and the output data information may be the size of the output data.

Optionally, in the case where the category of the visualization information is a scalar line graph, the visualization information includes the performance change trend information of the target deep model, the loss trend information of the target deep model, the training progress information of the target deep learning model, and the difference in the target deep learning model. At least one of the training comparison information.

Optionally, when the category of the visualization information is a feature map or a two-dimensional heat map, the visualization information includes features extracted by the target deep learning model.

Optionally, when the category of the visualization information is a three-dimensional histogram, the visualization information includes the parameter distribution of each operator in the plurality of operators.

Optionally, in the case where the category of the visualization information is the calling sequence diagram of multiple processes, the visualization information includes the running time of each operator in at least one process of the multiple operators.

Optionally, the process may be at least one of an interpreted language process, a local process, and an AI chip process.

Among them, the interpreted language process, the local process, and the AI chip process alternately execute the calling sequence diagram according to time.

It can be seen that, in this embodiment of the present application, the user equipment first sends a visualization request to the cloud server, then receives the visualization response sent by the cloud server, the visualization response carries the visualization information of the target training data, and finally displays the visualization information. The visual information of the data can intuitively understand the training status of the deep learning model, which helps to improve the feasibility of timely judgment of the current training strategy, and provides a basis for decision-making such as early stopping.

In an implementation manner of the present application, the displaying the visualized information includes:

displaying the visual information in a collapsed display mode or a revealing display mode, wherein,

In the folding display mode, a module in the target deep learning model is used as the minimum display unit, wherein the module includes at least one operator; and/or

In the expanded display mode, the operator in the target deep learning model is used as the minimum display unit.

Among them, when the category of the visualization information is a directed acyclic graph, the visualization information can be displayed in the collapsed display mode, and the visualization information can also be displayed in the expanded display mode. , which displays visual information by expanding the display mode.

The expanded display mode may be an operator in the expanded display module after the module receives an operation instruction.

Wherein, when the category of the visualization information is a three-dimensional histogram, the expanded display mode may be to directly display the parameter distribution of each operator in each module.

Wherein, when the category of the visual information is a call sequence diagram, the expanded display mode may be to directly display the running time of each operator in each module in at least one process.

For example, assuming that the category of visualization information is a directed acyclic graph, the target deep learning model includes 2 modules (module A and module B), and module A includes 3 operators (A1, A2, and A3), the model B includes 2 operators (B1 and B2). If the visualized information is displayed in the folded display mode, module A and module B are displayed; if the visualized information is displayed in the expanded display mode, after the module A receives the operation instruction, the operators A1, A2 and A3 in the module A are displayed, and the module B receives the operation After the instruction, the operators B1 and B2 in module B are displayed.

For example, assuming that the type of visualization information is a three-dimensional histogram, the target deep learning model includes 2 modules (module A and module B), module A includes 3 operators (A1, A2, and A3), and model B includes Including two operators (B1 and B2), module A corresponds to 3D histogram 1, and module B corresponds to 3D histogram B, then 3D histogram 1 includes the parameter distribution of A1, the parameter distribution of A2 and the parameter distribution of A3 The three-dimensional histogram 2 includes the parameter distribution of B1 and the parameter distribution of B2.

For example, assuming that the category of visualization information is the call sequence diagram, the target deep learning model includes 2 modules (module A and module B) and 2 processes (process C1 and C2), and module A includes 2 operators ( A1 and A2), model B includes 2 operators (B1 and B2), module A corresponds to call sequence diagram 1, and module B corresponds to call sequence diagram B, then the call sequence diagram 1 includes the running time of A1 in process C1 , the running duration of A1 in process C2, including the running duration of A2 in process C1 and the running duration of A2 in process C2; the calling sequence diagram 2 includes the running duration of B1 in process C1 and the running duration of B1 in process C2 The running duration includes the running duration of B2 in process C1 and the running duration of B2 in process C2.

Optionally, a display mode of visualized information may be preset, which may be to display the visualized information in a folded display mode first, and then display the visualized information in an expanded display mode after a first duration, which may be preset.

Optionally, when the type of visualization information is a scalar line graph, there can be multiple line graphs, which can be zoomed or panned after receiving an operation instruction, and the expanded display mode of the scalar line graph can be to directly display the performance change of the target depth model. At least one of the trend, the loss trend of the target deep learning model, the training progress of the target deep learning model, and the features extracted by different trainings in the target deep learning model.

Optionally, when the category of the visual information is a feature map or a two-dimensional heat map, the expandable display mode may be to directly display the features extracted by the target deep learning model.

Among them, in the two-dimensional heat map, the higher the heat, the greater the probability of being a feature.

It can be seen that, in the embodiments of the present application, the user equipment displays visual information in different ways, which is beneficial to improve the application scope of the user equipment.

In an implementation manner of the present application, before the sending the visualization request to the cloud server, the method further includes:

The intermediate training data of the target deep learning model is sent to the cloud server through a data upload thread, and the intermediate training data is used by the cloud server to obtain the index information and training task identifier of the target training data, and based on the The training task identifier stores the index information in a database, and stores the target training data in cloud storage based on the index information.

The data uploading thread is a newly opened thread.

The training task identification is unique, and the training task identification may be determined after the intermediate training data is generated, or may be determined at the beginning of training.

The intermediate training data includes target training data, training task identifiers, and index information.

Optionally, the intermediate training data of the target deep learning model is sent to the cloud server through a data upload thread, and the intermediate data is used for the cloud server to obtain the identification information of the target training data, and to store the identification information in a database. , the target training data is stored in cloud storage based on the identification information.

Among them, the identification information is unique.

Optionally, before sending the intermediate training data of the target deep learning model to the cloud server through a data upload thread, the method further includes:

determining the data volume of the target training data in the intermediate training data;

In the case where the data amount is greater than the preset data amount, determine the training task identifier and the index information of the target training data, and carry the training task information and the index information of the target training data in the intermediate training data;

In the case that the data amount is less than the preset data amount, the identification information of the target training data is determined, and the identification information is carried on the intermediate training data.

Optionally, when the cloud server receives the identification information, the identification information may be stored in the first buffer area of the database, and the target training data may be stored in the second buffer area of the database based on the identification information.

It can be seen that, in the embodiment of the present application, the target data is stored in the cloud storage by the cloud server, which is beneficial for the cloud server to quickly obtain the target training data after receiving the visualization request.

In an implementation manner of the present application, the preprocessing of the target training data to obtain visualization information includes:

Perform at least one preprocessing on the target training data to obtain preprocessing data;

Perform rendering processing on the preprocessed data to obtain the visualization information.

Wherein, if the data type of the target training data is a picture or a video, the preprocessing of the picture or video includes one or more of the following processes: insensitive area removal processing, image precision enhancement processing, image noise reduction processing and image processing Binarization processing.

Among them, if the data type of the target training data is text, the text preprocessing includes one or more of the following processes: document segmentation, text segmentation, and removal of stop words (including punctuation, numbers, monads, and other Meaningless words) processing, text feature extraction, word frequency statistics processing and text vectorization processing.

Among them, the visual rendering is to assemble the preprocessed data into a hypertext markup language (Hypertext Markup Language, HTML).

Among them, the type of visualization information obtained by preprocessing is non-directed acyclic graph.

It can be seen that, in the embodiment of the present application, the target training data is preprocessed to obtain preprocessed data, and then the preprocessed data is rendered to obtain visual information, which is beneficial to the target deep learning model and analysis based on the visual information.

When the training task of the target deep learning model is executed under the target framework, the target training data is parsed through a binary tree to obtain the visualization information.

Among them, the type of the target deep learning model that processes the target training data through the binary tree is parrots.

The type of visualization information obtained by parsing the target training data through a binary tree is a directed acyclic graph.

It can be seen that, in the embodiment of the present application, the visualization information is obtained by analyzing the target training data, which is beneficial to analyze and analyze the target deep learning model based on the visualization information.

Please refer to FIG. 3. FIG. 3 is a schematic structural diagram of a computer device provided by an embodiment of the present application. As shown in the figure, the computer device includes a processor, a memory, a communication interface, and one or more programs, wherein one or more of the above A plurality of programs are stored in the above-mentioned memory, and are configured to be executed by the above-mentioned processor.

In one implementation, the computer device is user equipment, and the above program includes instructions for performing the following steps:

The visualization information is displayed.

Optionally, the visualization information includes at least one of the following categories: directed acyclic graph, three-dimensional histogram, call sequence diagram of various processes, feature map, two-dimensional heat map, and scalar line graph; and/or

The target training data includes at least one of the following: model performance change trend information, model loss trend information, model parameter distribution information, model processing intermediate results, model structure information, current progress information of model training, and information on different trainings for the same model. Comparison information, scheduling time information of various processes.

Optionally, the visualization information includes at least one of the following information:

topology information of at least a part of the target deep learning model, wherein at least a part of the target deep learning model includes: multiple modules and/or multiple operators of the target deep learning model;

Resource occupation information of each operator in the at least one operator included in the target deep learning model.

Optionally, the topology information of the multiple modules includes at least one of the following:

The identification information of the multiple modules, the dependencies between the multiple modules, the data size of each module in the multiple modules, and the information of at least one operator included in each of the modules.

Optionally, the visualization request carries a training task identifier of the target deep learning model and index information of the target training data, where the index information includes: data type and data label.

Optionally, in terms of displaying the visualized information, the above-mentioned program includes instructions for executing the following steps:

In the embodiments of the present disclosure, the display of visual data in different modes is supported, wherein, in some embodiments, the visual information may be displayed in a folding display mode or a presentation display mode, or other display modes may also be defined. Embodiments are not limited thereto.

In some embodiments, in the folding display mode, a module in the target deep learning model is used as the smallest display unit, wherein the module includes at least one operator. The modules here may be divided in a default division manner, or may also be set by a user, which is not limited in this embodiment of the present disclosure.

In some embodiments, in the expanded display mode, the operator in the target deep learning model is used as the minimum display unit. At this point, the information of all operators in the model can be displayed.

In some embodiments, it can also be displayed in a mixed display mode, that is, some modules are folded and displayed, and other modules are displayed in an expanded manner, which can optionally be displayed based on user settings, which is not covered by the embodiments of the present disclosure. Do limit.

Optionally, before sending the visualization request to the cloud server, the above program includes an instruction for performing the following steps:

In another implementation, the computer device is a cloud server, and the above program includes instructions for performing the following steps:

obtaining target training data corresponding to the visualization request;

Preprocessing the target training data to obtain visualization information;

Optionally, in terms of preprocessing the target training data to obtain visualization information, the above program includes instructions for executing the following steps:

Determine each operator in the at least one operator based on at least one of the data type of the at least one operator in the target deep learning model, the input data information of the operator, and the output data information of the operator Child resource occupancy information.

Optionally, the visualization request carries a training task identifier of the target deep learning model and index information of the target training data, where the index information includes: data type and data label;

In terms of acquiring the target training data corresponding to the visualization request, the above-mentioned program includes an instruction for executing the following steps: searching an index database based on the training task identifier and index information carried in the visualization request, so as to obtain the target training data. storage location information; based on the storage location information of the target training data, obtain the target training data stored in the cloud storage.

It should be noted that, for the specific implementation process of this embodiment, reference may be made to the specific implementation process described in the foregoing method embodiment, which is not described herein again.

Please refer to FIG. 4. FIG. 4 is a visualization apparatus provided by an embodiment of the present application, applied to user equipment, and the apparatus includes:

A sending unit 401, configured to send a visualization request to a cloud server, where the visualization request is used to request visualization of target training data of a target deep learning model;

A receiving unit 402, configured to receive a visualization response sent by the cloud server, where the visualization response carries the visualization information of the target training data;

The display unit 403 is configured to display the visualized information.

In an implementation of the present application, the visualization information includes at least one of the following categories: directed acyclic graph, three-dimensional histogram, call sequence diagram of various processes, feature map, two-dimensional heat map, and scalar polyline Figures; and/or

In an implementation manner of the present application, the visualized information includes at least one of the following information:

In an implementation manner of the present application, the topology information of the multiple modules includes at least one of the following:

In an implementation manner of the present application, the resource occupation information is at least one of the cloud server based on the data type of the operator, the input data information of the operator, and the output data information of the operator definite.

In an implementation manner of the present application, the visualization request carries a training task identifier of the target deep learning model and index information of the target training data, where the index information includes: data type and data label.

In an implementation manner of the present application, in terms of displaying the visualized information, the above-mentioned display unit 403 is specifically used for:

In an implementation manner of the present application, the above-mentioned sending unit 401 is specifically further configured to send the intermediate training data of the target deep learning model to the cloud server through a data upload thread before sending the visualization request to the cloud server, and the The intermediate training data is used for the cloud server to obtain the index information and training task identifier of the target training data;

The apparatus further includes: a data storage unit 404, configured to store the index information in a database based on the training task identifier, and store the target training data in cloud storage based on the index information.

It should be noted that the sending unit 401 and the receiving unit 402 can be implemented through a communication interface, the display unit 403 can be implemented through a display screen, and the data storage unit 404 can be implemented through a processor screen.

Please refer to FIG. 5. FIG. 5 is a visualization device provided by an embodiment of the present application, applied to a cloud server, and the device includes:

A receiving unit 501, configured to receive a visualization request from a user equipment, where the visualization request is used to request visualization of target training data of a target deep learning model;

an obtaining unit 502, configured to obtain target training data corresponding to the visualization request;

Determining unit 503, configured to preprocess the target training data to obtain visualization information;

The sending unit 504 is configured to send a visualization response to the user equipment, where the visualization response carries the visualization information.

In an implementation manner of the present application, in terms of preprocessing the target training data to obtain visualization information, the above program includes instructions for executing the following steps:

In an implementation manner of the present application, the visualization request carries a training task identifier of the target deep learning model and index information of the target training data, where the index information includes: data type and data label;

In terms of acquiring the target training data corresponding to the visualization request, the above acquiring unit 502 is specifically used for:

In an implementation manner of the present application, in terms of preprocessing the target training data to obtain visualization information, the above determining unit 503 is specifically configured to:

It should be noted that, the receiving unit 501 and the sending unit 504 may be implemented by a communication interface, and the acquiring unit 502 and the determining unit 503 may be implemented by a processor.

Embodiments of the present application further provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program for electronic data exchange, wherein the computer program causes a computer to execute the electronic Some or all of the steps described by the device or cloud server.

Embodiments of the present application further provide a computer program product, wherein the computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause the computer to execute the electronic method as described above. Some or all of the steps described by the device or cloud server. The computer program product may be a software installation package.

The steps of the method or algorithm described in the embodiments of the present application may be implemented in a hardware manner, or may be implemented in a manner in which a processor executes software instructions. Software instructions can be composed of corresponding software modules, and software modules can be stored in random access memory (Random Access Memory, RAM), flash memory, read only memory (Read Only Memory, ROM), erasable programmable read only memory ( Erasable Programmable ROM, EPROM), Electrically Erasable Programmable Read-Only Memory (Electrically EPROM, EEPROM), registers, hard disk, removable hard disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, such that the processor can read information from, and write information to, the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium may reside in an ASIC. Additionally, the ASIC may reside in access network equipment, target network equipment or core network equipment. Of course, the processor and the storage medium may also exist in the access network device, the target network device or the core network device as discrete components.

Those skilled in the art should realize that, in one or more of the above examples, the functions described in the embodiments of the present application may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general purpose computer, special purpose computer, computer network, or other programmable device. The computer instructions may be stored in or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions may be downloaded from a website site, computer, server or data center Transmission to another website site, computer, server or data center via wired (eg coaxial cable, optical fiber, Digital Subscriber Line, DSL) or wireless (eg infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes an integration of one or more available media. The available media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, Digital Video Disc (DVD)), or semiconductor media (eg, Solid State Disk (SSD)) )Wait.

The specific embodiments described above further describe in detail the purposes, technical solutions and beneficial effects of the embodiments of the present application. It should be understood that the above descriptions are only specific implementations of the embodiments of the present application, and are not intended to be used for The protection scope of the embodiments of the present application is limited, and any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the embodiments of the present application should be included within the protection scope of the embodiments of the present application.

Claims

A visualization method, characterized in that, applied to a user equipment, the method comprising:

sending a visualization request to the cloud server, where the visualization request is used to request visualization of the target training data of the target deep learning model;

receiving a visualization response sent by the cloud server, where the visualization response carries the visualization information of the target training data;

The visualization information is displayed.
The method according to claim 1, wherein the visualization information includes at least one of the following categories: directed acyclic graph, three-dimensional histogram, calling sequence diagram of various processes, feature map, two-dimensional thermal graphs and scalar line graphs; and/or

The target training data includes at least one of the following: model performance change trend information, model loss trend information, model parameter distribution information, model processing intermediate results, model structure information, current progress information of model training, and information on different trainings for the same model. Comparison information, scheduling time information of various processes.
The method according to claim 1 or 2, the visualization information includes at least one of the following information:

topology information of at least a part of the target deep learning model, wherein at least a part of the target deep learning model includes: multiple modules and/or multiple operators of the target deep learning model;

Resource occupation information of each operator in the at least one operator included in the target deep learning model.
The method according to claim 3, wherein the topology information of the multiple modules includes at least one of the following:

The identification information of the multiple modules, the dependencies between the multiple modules, the data size of each module in the multiple modules, and the information of at least one operator included in each of the modules.
The method according to claim 3 or 4, wherein the resource occupation information is based on the data type of the operator, the input data information of the operator, and the output data of the operator by the cloud server. At least one of the information is determined.
The method according to any one of claims 1 to 5, wherein the visualization request carries a training task identifier of the target deep learning model and index information of the target training data, wherein the index information Includes: data types and data labels.
The method according to any one of claims 1-6, wherein the displaying the visualized information comprises:

displaying the visual information in a collapsed display mode or a revealing display mode, wherein,

In the folding display mode, a module in the target deep learning model is used as the minimum display unit, wherein the module includes at least one operator; and/or

In the expanded display mode, the operator in the target deep learning model is used as the minimum display unit.
The method according to any one of claims 1-7, wherein before the sending the visualization request to the cloud server, the method further comprises:

The intermediate training data of the target deep learning model is sent to the cloud server through a data upload thread, and the intermediate training data is used by the cloud server to obtain the index information and training task identifier of the target training data, and based on the The training task identifier stores the index information in an index database, and stores the target training data in cloud storage based on the index information.
A visualization method, characterized in that, applied to a cloud server, the method comprising:

receiving a visualization request from the user equipment, where the visualization request is used to request visualization of the target training data of the target deep learning model;

obtaining target training data corresponding to the visualization request;

Preprocessing the target training data to obtain visualization information;

A visualization response is sent to the user equipment, and the visualization response carries the visualization information.
The method according to claim 9, wherein the visualization information includes at least one of the following categories: directed acyclic graph, three-dimensional histogram, calling sequence diagram of various processes, feature map, two-dimensional thermal graphs and scalar line graphs; and/or

The target training data includes at least one of the following: model performance change trend information, model loss trend information, model parameter distribution information, model processing intermediate results, model structure information, current progress information of model training, and information on different trainings for the same model. Comparison information, scheduling time information of various processes.
The method according to claim 9 or 10, the visualization information includes at least one of the following information:

topology information of at least a part of the target deep learning model, wherein at least a part of the target deep learning model includes: multiple modules and/or multiple operators of the target deep learning model;

Resource occupation information of each operator in the at least one operator included in the target deep learning model.
The method according to claim 11, wherein the topology information of the plurality of modules comprises at least one of the following:

The identification information of the multiple modules, the dependencies between the multiple modules, the data size of each module in the multiple modules, and the information of at least one operator included in each of the modules.
The method according to any one of claims 9 to 12, wherein the preprocessing of the target training data to obtain visualization information includes:

Determine each operator in the at least one operator based on at least one of the data type of the at least one operator in the target deep learning model, the input data information of the operator, and the output data information of the operator Child resource occupancy information.
The method according to any one of claims 9 to 13, wherein the visualization request carries a training task identifier of the target deep learning model and index information of the target training data, and the index information includes: data types and data labels;

The acquiring target training data corresponding to the visualization request includes:

Search the index database based on the training task identifier and index information carried in the visualization request to obtain the storage location information of the target training data;

The target training data stored in the cloud storage is acquired based on the storage location information of the target training data.
The method according to any one of claims 9 to 14, wherein the preprocessing of the target training data to obtain visualization information includes:

Perform at least one preprocessing on the target training data to obtain preprocessing data;

Perform rendering processing on the preprocessed data to obtain the visualization information.
The method according to any one of claims 9 to 15, wherein the preprocessing of the target training data to obtain visualization information includes:

When the training task of the target deep learning model is executed under the target framework, the target training data is parsed through a binary tree to obtain the visualization information.
A visualization device, characterized in that, applied to user equipment, the device comprising:

a sending unit, configured to send a visualization request to the cloud server, where the visualization request is used to request visualization of the target training data of the target deep learning model;

a receiving unit, configured to receive a visualization response sent by the cloud server, where the visualization response carries the visualization information of the target training data;

a display unit for displaying the visual information.
A visualization device, characterized in that, applied to a cloud server, the device comprising:

a receiving unit, configured to receive a visualization request from the user equipment, where the visualization request is used to request visualization of the target training data of the target deep learning model;

an obtaining unit for obtaining target training data corresponding to the visualization request;

a determining unit for preprocessing the target training data to obtain visualization information;

A sending unit, configured to send a visualization response to the user equipment, where the visualization response carries the visualization information.
A computer device, characterized in that the user equipment includes a processor, a memory, a communication interface, and one or more programs, the one or more programs being stored in the memory and configured by the The processor executes the program comprising instructions for performing the steps in the method of any of claims 1-8 or 9-16.
A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, wherein the computer program is processed to execute the method according to any one of claims 1-8 or 9-16 .