CN117635083A - Intelligent hardware chip model integrated development system and method for population intelligent scene - Google Patents
Intelligent hardware chip model integrated development system and method for population intelligent scene Download PDFInfo
- Publication number
- CN117635083A CN117635083A CN202410110225.6A CN202410110225A CN117635083A CN 117635083 A CN117635083 A CN 117635083A CN 202410110225 A CN202410110225 A CN 202410110225A CN 117635083 A CN117635083 A CN 117635083A
- Authority
- CN
- China
- Prior art keywords
- model
- module
- development
- training
- hardware chip
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000011161 development Methods 0.000 title claims abstract description 131
- 238000000034 method Methods 0.000 title claims abstract description 33
- 238000012549 training Methods 0.000 claims abstract description 66
- 230000006870 function Effects 0.000 claims abstract description 20
- 230000008569 process Effects 0.000 claims abstract description 19
- 230000003993 interaction Effects 0.000 claims abstract description 16
- 238000004458 analytical method Methods 0.000 claims abstract description 14
- 238000012800 visualization Methods 0.000 claims abstract description 13
- 230000002452 interceptive effect Effects 0.000 claims abstract description 5
- 238000007726 management method Methods 0.000 claims description 65
- 238000004088 simulation Methods 0.000 claims description 39
- 238000012545 processing Methods 0.000 claims description 23
- 230000000875 corresponding effect Effects 0.000 claims description 18
- 238000012544 monitoring process Methods 0.000 claims description 7
- 238000006243 chemical reaction Methods 0.000 claims description 6
- 238000012217 deletion Methods 0.000 claims description 6
- 230000037430 deletion Effects 0.000 claims description 6
- 238000005457 optimization Methods 0.000 claims description 6
- 238000012358 sourcing Methods 0.000 claims description 5
- 230000005540 biological transmission Effects 0.000 claims description 4
- 230000002688 persistence Effects 0.000 claims description 4
- 238000007781 pre-processing Methods 0.000 claims description 4
- 238000011084 recovery Methods 0.000 claims description 4
- 230000002596 correlated effect Effects 0.000 claims description 3
- 230000006378 damage Effects 0.000 claims description 3
- 238000013515 script Methods 0.000 claims description 3
- 230000001360 synchronised effect Effects 0.000 claims description 3
- 238000009960 carding Methods 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 4
- 230000009286 beneficial effect Effects 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 3
- 238000013473 artificial intelligence Methods 0.000 description 1
- 239000000969 carrier Substances 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000013079 data visualisation Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011165 process development Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 238000007794 visualization technique Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Landscapes
- Stored Programmes (AREA)
Abstract
The invention discloses an intelligent hardware chip model integrated development system and method of a population intelligent scene, wherein the system comprises the following components: the model management module is used for managing the processes of adding, deleting, modifying and checking the whole life cycle of the model; the role management module is used for managing authority and interaction functions of different types of roles in the development system; the model integrated development module is used for realizing integrated development and training of the model; the multi-role concurrent interaction module is used for realizing concurrent interactive development of multiple roles on the model at the same time; the file storage module is used for butting the distributed file system and providing files required by model development; the visualization module is used for displaying the performance analysis result of the model and reporting the training result in real time. The invention can realize the development of the hardware chip model of multi-role concurrent development, realize the same-end development in the full life cycle of the chip development field, improve the efficiency and the flexibility of the development of the hardware chip model, meet the requirements of different application scenes and have good popularization and application prospects.
Description
Technical Field
The invention relates to the field of hardware chip model development, in particular to an intelligent hardware chip model integrated development system and method for a population intelligent scene.
Background
With the development of the emerging technologies such as the Internet of things, artificial intelligence and big data, the demand of a hardware chip model is continuously increasing, wherein the hardware chip model development based on the crowd-sourced collaboration scene has good application prospects in the fields of embedded equipment, intelligent home, industrial control and the like. The hardware chip model development based on crowd-sourced cooperation can provide a more convenient and efficient same-end development environment for a developer, and reduce cross-end work of the developer; meanwhile, developers can rapidly and accurately develop a hardware chip model meeting market demands through shared resources and co-collaboration, so that development period and cost are greatly reduced, the intelligent and resource of each party can be better exerted through the mode of crowd-sourced collaboration, and the efficiency and quality of model development are improved. In the development process of a hardware chip model, the steps of model creation, data set import, integrated development environment (Integrated Development Environment, IDE) editing, model training result display and the like are usually required to be carried out. The traditional development process usually requires a plurality of independent tools and platforms for operation, and is complex in operation, low in efficiency and incapable of being cooperated with each other among engineers. Therefore, an integrated system and method are needed to simplify the process of hardware chip model development, promote the group collaboration among engineers, improve the development efficiency and convenience, and finally realize the group intelligence co-creation in the chip development field.
Disclosure of Invention
The invention aims to provide an intelligent hardware chip model integrated development system and method for a population intelligent scene aiming at the defects of the prior art.
The aim of the invention is realized by the following technical scheme: the first aspect of the embodiment of the invention provides an intelligent hardware chip model integrated development system of a population intelligent scene, which comprises the following components:
the model management module is used for managing the full quantity model and the single model, and is used for managing the processes of adding, deleting, modifying and checking the whole life cycle of the hardware chip model, and specifically comprises the functions related to model building, model deleting, version management, model batch processing, data set processing and model overall management;
the role management module is used for managing authority and interaction functions of different types of roles in the development system;
the model integrated development module comprises a model development and training module, a model compiling module and a model simulation module, wherein the model integrated development module is used for realizing integrated development and training of a hardware chip model and specifically comprises model creation, model development and training, model compiling and model simulation functions;
the multi-role concurrent interaction module is used for realizing concurrent interactive development of the hardware chip model by multiple roles at the same time;
the file storage module is used for butting the distributed file system and providing file management service and files required in code editing; and
and the visualization module is used for displaying the performance analysis result and the reported training result in the training process of the model in real time.
Further, the managing of the full-scale model includes: model overview including model storage space, model carding, and type distribution overview, and batch processing including deletion and download;
the management of the single model includes: modifying the associated dataset, version history management, and developer rights management.
Further, the implementation of the model management module specifically includes: dynamically generating a corresponding model structure or selecting a predefined model template according to the model name, type, task and label input by a user; designing an interface for automation and user-friendly interaction of model creation, basic information editing and deletion; importing a data set, preprocessing the data set, and performing automatic format conversion on data in the data set to meet the requirement of model input; to perform full lifecycle management for all model development tasks.
Further, the types of the roles include an administrator, a developer and a tester;
the rights include creating, training, compiling, and simulating editing processing rights.
Further, the model integrated development module is used for detailed editing, debugging and optimizing of the model and specifically comprises a code editor, a debugger, a performance analyzer and an optimizing tool;
the code editor realizes the editing of model codes based on an open-source integrated development environment jupyter lab, and specifically comprises grammar highlighting, automatic completion, code navigation and code reconstruction functions, and periodically checks the state of each item under each user id so as to perform resource recovery after data persistence on the items which are not used for more than 1 day;
the debugger is used for helping developers and testers find out and solve problems in model codes in the model development process, and particularly comprises breakpoint setting, step execution and variable monitoring functions;
the performance analyzer is used for helping a developer analyze performance bottleneck of the model code and finding and optimizing performance problems in the model code, and specifically comprises code execution time analysis and memory use condition analysis functions;
the optimizing tool is used for optimizing the model according to the debugging result of the debugger and the analysis result of the performance analyzer.
Further, the implementation of the multi-role concurrent interaction module specifically includes: and synchronizing the processing measures of the users to the model to the cloud in real time through network transmission, and synchronizing the processing measures to the terminals of all related users to cooperate with version management in real time through the cloud so as to realize multi-role synchronization for model development.
Further, the file storage module uses a shared-storage network file system as an implementation scheme of file storage, and synchronizes files required in the file management service and code editing before the container is started and the container is destroyed to a distributed file system or other backup storage positions;
the use process of the file storage module specifically comprises the following steps:
network file system using shared storage: storing all model codes and data sets on a network file system, wherein the network file system shares storage space among a plurality of servers through a network;
setting a size threshold for each item: creating independent folders for each item on a network file system, limiting the size of each item, setting a size threshold for each item, and monitoring the size of the folders in real time;
container startup is preceded to a network file system: synchronizing files required in file management services and code editing from a local or other storage location into corresponding project folders on a network file system using a file synchronization tool prior to launching the container; wherein the file required in the file management service and code editing includes all model codes and data in the dataset;
synchronizing to a distributed file system after destroying the container: after the container is destroyed, the files required in the file management service and code editing are synchronized to a distributed file system or other backup storage location by writing scripts or using file synchronization tools.
Further, the development system also comprises a log module, wherein the log module is used for managing the log of model training.
The second aspect of the embodiment of the invention provides a development method of an intelligent hardware chip model integrated development system based on the crowd-sourcing scenario, which comprises the following steps:
model creation phase: corresponding creation, training, compiling and simulation authorities are given to different types of roles through the role management module; a person with the creation authority creates a model in the model management module, edits basic information of the model and a data set required by the model to initialize an editable code item; wherein the code and the data set are stored in a file storage module;
model development and training phase: the personnel with training authority edits the model codes in the model development and training module of the model integrated development module so as to complete the development of the model; editing training environment and model training parameters, training the developed model, acquiring a training report, and displaying in real time through a visualization module; optimizing the model according to the training result in the training report;
model compiling stage: the personnel with compiling authority configures parameters required by a model compiling environment in a model compiling module of the model integrated development module so as to compile a model after training and optimizing, and outputs a compiled model machine code;
model simulation stage: a person with model simulation authority edits a simulation environment in a model simulation module of the model integrated development module, simulates a compiled model by adjusting simulation parameters, generates a corresponding simulation report, and simultaneously displays the simulation report in real time by a visualization module; wherein the simulation parameters include hardware parameters of the hardware chip.
Further, the basic information of the model comprises the name, task type and label of the model;
the data set specifically comprises: the data sets required for the model are uploaded locally or correlated from a list of data sets.
The invention integrates the chip model development flow which is originally developed across platforms and tools into a system, and realizes the management and development of the full life cycle of hardware chip model development; the invention can realize the model development of multi-role concurrent development, promote the group cooperation among developers, improve the efficiency and flexibility of hardware chip model development and meet the requirements of different application scenes.
Drawings
FIG. 1 is a functional framework diagram of an intelligent hardware chip model integrated development system for a crowd-sourced scenario, provided by an embodiment of the present invention;
FIG. 2 is a flowchart of an integrated development method for intelligent hardware chip models of a crowd-sourcing scenario according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a data set management flow according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a character management module according to an embodiment of the present invention;
fig. 5 is a schematic diagram of a file storage module according to an embodiment of the invention.
Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. It is obvious that the drawings used in the following description are only some embodiments of the present invention, and that other drawings may be obtained from them without inventive faculty for a person skilled in the art. The implementations described in the following exemplary examples do not represent all implementations consistent with the invention. Rather, they are merely examples of apparatus and methods consistent with aspects of the invention as detailed in the accompanying claims.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any or all possible combinations of one or more of the associated listed items.
It should be understood that although the terms first, second, third, etc. may be used herein to describe various information, these information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the invention. The word "if" as used herein may be interpreted as "at … …" or "at … …" or "responsive to a determination", depending on the context.
The present invention will be described in detail with reference to the accompanying drawings. The features of the examples and embodiments described below may be combined with each other without conflict.
In order to realize the whole life cycle management and the whole process development of the whole hardware chip model, the invention provides an intelligent hardware chip model integrated development system of a population intelligent scene, which is mainly characterized by module composition and standardized development process.
Referring to fig. 1, the intelligent hardware chip model integrated development system of the crowd-sourced scenario specifically comprises a model management module, a role management module, a model integrated development module, a multi-role concurrent interaction module, a file storage module and a visualization module.
In this embodiment, the model management module is used for managing the full-scale model and the single model, and is used for managing the processes of adding, deleting, modifying and checking the whole life cycle of the hardware chip model, and specifically includes the functions related to model building, model deleting, version management, model batch processing, data set processing and the like.
Further, the managing of the full-quantity model includes: model overview including model storage space, model sorting, and type distribution overview, and batch processing including deletion, downloading, and the like. The management of the single model includes: modifying the associated dataset, version history management, developer rights management, and the like.
Further, the implementation of the model management module specifically includes: dynamically generating a corresponding model structure or selecting a predefined model template according to the model name, type, task and label input by a user; designing an interface for automation and user-friendly interaction of model creation, basic information editing and deletion; importing a data set, preprocessing the data set, and performing automatic format conversion on data in the data set to meet the requirement of model input; etc. to perform full lifecycle management for all model development tasks.
It should be understood that the model management module may be implemented as a set of management systems capable of performing full lifecycle management on all model development tasks, including specific or batch processing of model specific tasks, such as adding, deleting, modifying, and checking.
It should be noted that, preprocessing of the data set is format processing, a programmer customizes different types of data conversion algorithms, automatically formats the data set to meet the requirement of model input, specifically, as shown in fig. 3, a management flow of the data set is displayed, the data set is first imported, then parameter configuration is performed, a file is uploaded, whether the data in the data set needs to be converted or not is judged, if the format needs to be converted, the data set is converted by adopting the corresponding type of data conversion algorithm, and the data set is stored after conversion; if the format does not need to be converted, the data set is directly stored, so that the data set is convenient for the direct use in the subsequent process.
In this embodiment, the role management module is configured to manage rights and interactive functions of different roles in the development system.
Further, the types of roles include various roles such as manager, developer and tester.
Further, the rights include editing processing rights such as creation, training, compiling, simulation, and the like.
Specifically, the role management module manages different types of roles participating in the development of the hardware chip model in the development system, such as developers, testers, administrators and the like, and gives corresponding editing processing authorities for creation, training, compiling, simulation and the like to the different types of roles. It should be understood that a complete process includes creation, training, compiling and simulation of a model, and the role management module gives rights to different types of roles according to the project, for example, a developer has the development rights of the project a, and then also has all the rights in the application development link; the same is true of the test person.
Illustratively, as shown in FIG. 4, the implementation of the role management module may include: adding users, deleting users and updating users, wherein each user (user) comprises a user id, a name (name), a password (passwd), a type (type) and a user code (user code); such as when a user is newly added, the above information may be added to the user. According to actual needs, roles are allocated to users, different types of roles are bound to different users, and corresponding rights are bound to the corresponding users according to the different types of roles; for example, the role type of the developer is bound to the user 1, the role type of the tester is bound to the user 2, the corresponding permissions such as creation, training, compiling and simulation are bound to the user 1, the training permission is bound to the user 2, the user id and the name of the user 1 are added in the role (role) of the developer, the user id and the name of the user 2 are added in the role of the tester, the user id and the name of the user 3 can be added to complete the addition of the role, and similarly, the role can be deleted. When binding the rights to the roles, the corresponding user id, name and key (key) can be added to the rights (permission), and the rights can be added or deleted in the same way.
In this embodiment, the model integrated development module includes a model development and training module, a model compiling module and a model simulation module, where the model integrated development module is used to implement integrated development and training of a hardware chip model, and specifically includes functions of model creation, model development and training, model compiling, and model simulation.
Further, the model integrated development module is a set of integrated development environment and is used for detailed editing, debugging and optimizing of the model, and specifically comprises a code editor, a debugger, a performance analyzer, an optimizing tool and the like.
The code editor realizes the editing of model codes based on an open-source integrated development environment jupyter lab, and particularly comprises the functions of grammar highlighting, automatic completion, code navigation, code reconstruction and the like, so that the development efficiency of the model can be effectively improved, and meanwhile, the state of each project under each user id is periodically checked to perform resource recovery after data persistence on the projects which are not used for more than 1 day. In addition, the code editor can integrate a model development environment, so that a developer can conveniently perform visual editing of the model. It should be understood that each item under each user id is a jupyter instance, so that to prevent resource waste, the state of each item under each user id needs to be checked regularly, and the unused items are more than 1 day to recover resources and ensure data persistence before recovery.
The debugger is used for helping developers and testers find out and solve the problems in model codes in the model development process, and particularly comprises functions of breakpoint setting, stepping execution, variable monitoring and the like, so that the developers and testers can be helped to debug the codes.
The performance analyzer is used for helping developers analyze performance bottlenecks of model codes, find and optimize performance problems in the model codes, and particularly comprises functions of code execution time analysis, memory use condition analysis and the like, and is beneficial to the developers to optimize the performance of the model codes.
The optimization tool is used for optimizing the model according to the debugging result of the debugger and the analysis result of the performance analyzer, for example, a training report is displayed in a data visualization mode, a plurality of reports are supported to be selected to generate a performance comparison report, and the like, so that a developer is helped to perform model optimization, such as automatic parameter adjustment, model compression, and the like, an optimization algorithm is provided for automatic parameter adjustment, and the optimization efficiency is improved.
In this embodiment, the multi-role concurrent interaction module is configured to implement concurrent interactive development of multiple roles on the hardware chip model at the same time, which is beneficial to improving development efficiency and flexibility.
It should be understood that the model creation, training, compiling, simulation and other steps included in the model integrated development module can be simultaneously and concurrently interacted by multiple roles of the multi-role concurrent interaction module, so that the development efficiency and flexibility of the hardware chip model can be effectively improved.
Further, the implementation of the multi-role concurrent interaction module specifically includes: and synchronizing the processing measures of the users to the model to the cloud in real time through network transmission, and synchronizing the processing measures to the terminals of all related users to cooperate with version management in real time through the cloud so as to realize multi-role synchronization for model development.
It should be understood that different users have different roles, the rights of the different roles are different, the different rights represent different processing results of the model, and the processing measures of the users on the model are synchronized to the cloud in real time through network transmission, so that the operation results of the users, such as creation, editing, compiling, training, simulation and the like, can be saved, and meanwhile, the history record of the user operation is also supported to be saved.
In this embodiment, the file storage module is used to interface with the distributed file system (FastDFS) to provide general file management services and files required for code editing.
It should be appreciated that FastDFS is an open source lightweight storage system that manages files, the functions of which include: file storage, file synchronization, file access (file uploading, file downloading) and the like, and solves the problems of large-capacity storage and load balancing; the method is particularly suitable for online services taking files as carriers, such as album websites, video websites and the like. The FastDFS is customized for the Internet, fully considers mechanisms such as redundancy backup, load balancing, linear capacity expansion and the like, pays attention to indexes such as high availability, high performance and the like, and a set of high-performance file server clusters are easily built by using the FastDFS to provide services such as file uploading, downloading and the like.
Further, the file storage module uses a shared storage network file system (Network File System, NFS) as an implementation of file storage, and synchronizes files required in the container startup preamble file management service and code editing, and after container destruction, to FastDFS or other backup storage locations, so that reliability and consistency of data can be ensured, and data loss can be prevented.
It should be appreciated that NFS allows resources to be shared between computers in a network, and in NFS applications, a client application of local NFS can transparently read and write files located on a remote NFS server, just as if accessing local files. The containers refer to a docker container, and the containers can be started, stopped and deleted, each container is a platform which is isolated from the other and ensures safety, and the containers can be regarded as a simple Linux environment and an application program running in the Linux environment.
Specifically, as shown in fig. 5, the use process of the file storage module specifically includes:
(a) NFS using shared storage: all model codes and data sets are stored on NFS, which shares storage space between multiple servers over a network, which is beneficial to ensuring data consistency and accessibility.
(b) Setting a size threshold for each item: by creating an independent folder for each item on NFS, limiting the size of each item, setting a size threshold for each item, for example, limiting the size threshold of each item to 1GB, and monitoring the size of the folder in real time helps to control the use of storage space.
It should be appreciated that the size threshold for each item may be determined based on the device resources and the number of users, and set to other values.
(c) Container startup preamble to NFS: prior to launching the container, the file synchronization tool is used to synchronize files required in the file management service and code editing from a local or other storage location into the corresponding project folder on NFS. Among them, files required in file management service and code editing include, but are not limited to: all model code and data in the dataset, etc.
Further, the file synchronization tools include rsync tools, scp tools, and the like.
(d) After container destruction, synchronizing to FastDFS: after the container is destroyed, the relevant data can be ensured to be backed up or moved to a reliable storage location after the container is destroyed by writing a script or synchronizing files required in file management services and code editing to the FastDFS or other backup storage locations using a file synchronization tool.
In this embodiment, the visualization module is configured to display, in real time, a performance analysis result and a reported training result in a training process of the model, for example, dynamic change of training loss, real-time monitoring of performance indexes, and the like.
Further, the visualization module is implemented by a set of visualization methods, and the performance analysis results and the reported training results need to be organized into a report form which is easy to understand, and the presentation forms of the visualization module include, but are not limited to: the graph display form, the performance evaluation index form, the key statistical data form and the like, and the expandability and the customization of the report are also required to be considered.
In some other embodiments, the development system further includes a log module for managing a log of model training.
It should be noted that the embodiment of the invention also provides a development method, which is realized based on the intelligent hardware chip model integrated development system of the crowd scene in the embodiment, and can realize the development of the whole flow, multiple roles and concurrent interaction of the hardware chip model.
Referring to fig. 2, the development method of the present invention specifically includes the following steps:
(1) Model creation phase: corresponding creation, training, compiling and simulation authorities are given to different types of roles through the role management module; a person with the creation authority creates a model in the model management module, edits basic information of the model and a data set required by the model to initialize an editable code item; wherein the code and data sets are stored in a file storage module.
Further, the basic information of the model includes the name, task type, label, and the like of the model.
Further, the data set specifically includes: the data sets required for the model are uploaded locally or correlated from a list of data sets. The data set is stored in the data set list after being uploaded locally.
(2) Model development and training phase: the personnel with training authority edits the model codes in the model development and training module of the model integrated development module so as to complete the development of the model; editing training environment and model training parameters, training the developed model, acquiring a training report, and displaying in real time through a visualization module; and optimizing the model according to the training result in the training report.
(3) Model compiling stage: and the personnel with compiling authority configures parameters required by the model compiling environment in a model compiling module of the model integrated development module so as to compile the model after training and optimizing, and outputs a compiled model machine code.
(4) Model simulation stage: a person with model simulation authority edits a simulation environment in a model simulation module of the model integrated development module, simulates a compiled model by adjusting simulation parameters, generates a corresponding simulation report, and simultaneously displays the simulation report in real time by a visualization module; wherein the simulation parameters include hardware parameters of the hardware chip, etc.
The invention integrates the chip model development flow which is originally developed across platforms and tools into a system, and realizes the management and development of the full life cycle of hardware chip model development; the invention can realize the model development of multi-role concurrent development, promote the group cooperation among developers, improve the efficiency and flexibility of hardware chip model development and meet the requirements of different application scenes.
The above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. An intelligent hardware chip model integrated development system of a population intelligence scene, which is characterized by comprising:
the model management module is used for managing the full quantity model and the single model, and is used for managing the processes of adding, deleting, modifying and checking the whole life cycle of the hardware chip model, and specifically comprises the functions related to model building, model deleting, version management, model batch processing, data set processing and model overall management;
the role management module is used for managing authority and interaction functions of different types of roles in the development system;
the model integrated development module comprises a model development and training module, a model compiling module and a model simulation module, wherein the model integrated development module is used for realizing integrated development and training of a hardware chip model and specifically comprises model creation, model development and training, model compiling and model simulation functions;
the multi-role concurrent interaction module is used for realizing concurrent interactive development of the hardware chip model by multiple roles at the same time;
the file storage module is used for butting the distributed file system and providing file management service and files required in code editing; and
and the visualization module is used for displaying the performance analysis result and the reported training result in the training process of the model in real time.
2. The intelligent hardware chip model integrated development system of a crowd-sourced scenario of claim 1, wherein the management of the full-scale model comprises: model overview including model storage space, model carding, and type distribution overview, and batch processing including deletion and download;
the management of the single model includes: modifying the associated dataset, version history management, and developer rights management.
3. The intelligent hardware chip model integrated development system of a crowd-sourcing scenario of claim 1, wherein the implementation of the model management module specifically comprises: dynamically generating a corresponding model structure or selecting a predefined model template according to the model name, type, task and label input by a user; designing an interface for automation and user-friendly interaction of model creation, basic information editing and deletion; importing a data set, preprocessing the data set, and performing automatic format conversion on data in the data set to meet the requirement of model input; to perform full lifecycle management for all model development tasks.
4. The intelligent hardware chip model integrated development system of a crowd-sourced scenario of claim 1, wherein the types of roles include administrators, developers, and testers;
the rights include creating, training, compiling, and simulating editing processing rights.
5. The intelligent hardware chip model integrated development system of a crowd-sourcing scenario of claim 1, wherein the model integrated development module is used for detailed editing, debugging and optimization of models, and specifically comprises a code editor, a debugger, a performance analyzer and an optimization tool;
the code editor realizes the editing of model codes based on an open-source integrated development environment jupyter lab, and specifically comprises grammar highlighting, automatic completion, code navigation and code reconstruction functions, and periodically checks the state of each item under each user id so as to perform resource recovery after data persistence on the items which are not used for more than 1 day;
the debugger is used for helping developers and testers find out and solve problems in model codes in the model development process, and particularly comprises breakpoint setting, step execution and variable monitoring functions;
the performance analyzer is used for helping a developer analyze performance bottleneck of the model code and finding and optimizing performance problems in the model code, and specifically comprises code execution time analysis and memory use condition analysis functions;
the optimizing tool is used for optimizing the model according to the debugging result of the debugger and the analysis result of the performance analyzer.
6. The intelligent hardware chip model integrated development system of a crowd-sourced scenario according to claim 1, wherein the implementation of the multi-persona concurrent interaction module specifically comprises: and synchronizing the processing measures of the users to the model to the cloud in real time through network transmission, and synchronizing the processing measures to the terminals of all related users to cooperate with version management in real time through the cloud so as to realize multi-role synchronization for model development.
7. The intelligent hardware chip model integrated development system of a crowd-sourced scenario of claim 1, wherein the file storage module uses a shared-storage network file system as an implementation of file storage and synchronizes files required in file management services and code editing before container startup and after container destruction to a distributed file system or other backup storage locations;
the use process of the file storage module specifically comprises the following steps:
network file system using shared storage: storing all model codes and data sets on a network file system, wherein the network file system shares storage space among a plurality of servers through a network;
setting a size threshold for each item: creating independent folders for each item on a network file system, limiting the size of each item, setting a size threshold for each item, and monitoring the size of the folders in real time;
container startup is preceded to a network file system: synchronizing files required in file management services and code editing from a local or other storage location into corresponding project folders on a network file system using a file synchronization tool prior to launching the container; wherein the file required in the file management service and code editing includes all model codes and data in the dataset;
synchronizing to a distributed file system after destroying the container: after the container is destroyed, the files required in the file management service and code editing are synchronized to a distributed file system or other backup storage location by writing scripts or using file synchronization tools.
8. The intelligent hardware chip model integrated development system of a crowd-sourced scenario of claim 1, further comprising a log module for managing a log of model training.
9. A method for developing an integrated development system for intelligent hardware chip models based on a crowd-sourcing scenario as claimed in any one of claims 1-8, comprising:
model creation phase: corresponding creation, training, compiling and simulation authorities are given to different types of roles through the role management module; a person with the creation authority creates a model in the model management module, edits basic information of the model and a data set required by the model to initialize an editable code item; wherein the code and the data set are stored in a file storage module;
model development and training phase: the personnel with training authority edits the model codes in the model development and training module of the model integrated development module so as to complete the development of the model; editing training environment and model training parameters, training the developed model, acquiring a training report, and displaying in real time through a visualization module; optimizing the model according to the training result in the training report;
model compiling stage: the personnel with compiling authority configures parameters required by a model compiling environment in a model compiling module of the model integrated development module so as to compile a model after training and optimizing, and outputs a compiled model machine code;
model simulation stage: a person with model simulation authority edits a simulation environment in a model simulation module of the model integrated development module, simulates a compiled model by adjusting simulation parameters, generates a corresponding simulation report, and simultaneously displays the simulation report in real time by a visualization module; wherein the simulation parameters include hardware parameters of the hardware chip.
10. The development method according to claim 9, wherein the basic information of the model includes a name, a task type, and a label of the model;
the data set specifically comprises: the data sets required for the model are uploaded locally or correlated from a list of data sets.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410110225.6A CN117635083A (en) | 2024-01-26 | 2024-01-26 | Intelligent hardware chip model integrated development system and method for population intelligent scene |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410110225.6A CN117635083A (en) | 2024-01-26 | 2024-01-26 | Intelligent hardware chip model integrated development system and method for population intelligent scene |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117635083A true CN117635083A (en) | 2024-03-01 |
Family
ID=90036310
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410110225.6A Pending CN117635083A (en) | 2024-01-26 | 2024-01-26 | Intelligent hardware chip model integrated development system and method for population intelligent scene |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117635083A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112394907A (en) * | 2020-10-15 | 2021-02-23 | 北京仿真中心 | Container-based delivery system construction method, application delivery method and delivery system |
| CN112529023A (en) * | 2019-09-18 | 2021-03-19 | 上海钛空猫智能科技有限公司 | Configured artificial intelligence scene application research and development method and system |
| CN113760464A (en) * | 2021-09-14 | 2021-12-07 | 中化现代农业有限公司 | Artificial intelligence model development platform based on deep learning |
| CN114781635A (en) * | 2022-06-22 | 2022-07-22 | 国汽智控(北京)科技有限公司 | Model deployment method, device, equipment and medium |
| CN116339752A (en) * | 2022-12-01 | 2023-06-27 | 航天时代飞鸿技术有限公司 | Automatic deployment method and system for autonomous controllable chip model based on multistage intermediate representation |
-
2024
- 2024-01-26 CN CN202410110225.6A patent/CN117635083A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112529023A (en) * | 2019-09-18 | 2021-03-19 | 上海钛空猫智能科技有限公司 | Configured artificial intelligence scene application research and development method and system |
| CN112394907A (en) * | 2020-10-15 | 2021-02-23 | 北京仿真中心 | Container-based delivery system construction method, application delivery method and delivery system |
| CN113760464A (en) * | 2021-09-14 | 2021-12-07 | 中化现代农业有限公司 | Artificial intelligence model development platform based on deep learning |
| CN114781635A (en) * | 2022-06-22 | 2022-07-22 | 国汽智控(北京)科技有限公司 | Model deployment method, device, equipment and medium |
| CN116339752A (en) * | 2022-12-01 | 2023-06-27 | 航天时代飞鸿技术有限公司 | Automatic deployment method and system for autonomous controllable chip model based on multistage intermediate representation |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10540335B2 (en) | Solution to generate a scriptset for an automated database migration | |
| US20180293287A1 (en) | Automating extract, transform, and load job testing | |
| US11042471B2 (en) | System and method for providing a test manager for use with a mainframe rehosting platform | |
| US20190034189A1 (en) | Systems and methods for transformation of reporting schema | |
| US10366112B2 (en) | Compiling extract, transform, and load job test data cases | |
| US11151025B1 (en) | Generating software test plans based at least in part on monitored traffic of a production application | |
| CN114912255A (en) | On-line simulation experiment system and method | |
| CN114546563A (en) | Multi-tenant page access control method and system | |
| US11200052B1 (en) | Documentation enforcement during compilation | |
| CN115996230B (en) | Cross-cloud data synchronization method and device, computer equipment and readable storage medium | |
| CN117635083A (en) | Intelligent hardware chip model integrated development system and method for population intelligent scene | |
| Wannipurage et al. | A Framework to capture and reproduce the Absolute State of Jupyter Notebooks | |
| CN110750582A (en) | Data processing method, device and system | |
| CN115757090A (en) | Software testing system, method, device and medium | |
| CN113220592A (en) | Processing method and device for automated testing resources, server and storage medium | |
| Riahi et al. | Using Hadoop File System and MapReduce in a small/medium Grid site | |
| Shih | MASS HDFS: multi-agent spatial simulation hadoop distributed file system | |
| Stonozhenko et al. | Automated Object Storage Management Approach with Operator SDK and Custom Resource Definition | |
| CN114500268B (en) | Method and device for deploying char resources, server and storage medium | |
| Zhang et al. | Intelligent Grid Operation and Maintenance Management and Command Platform Based on Computer Distributed Network | |
| Aven | Hadoop in 24 Hours, Sams Teach Yourself | |
| Bemm | Extending the Kubernetes operator Kubegres to handle database restoration from dump files | |
| Correia | Entrega Contínua de Aplicações em Ambientes Híbridos | |
| Chenshuo | DEVELOPING THE DESIGN OF AN ONLINE REAL-TIME NOTES APPLICATION | |
| Leo | Achieving a Reusable Reference Architecture for Microservices in Cloud Environments |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |