CN117389838A - Operation and maintenance data combined acquisition and integration method and system for super calculation and intelligent calculation - Google Patents

Operation and maintenance data combined acquisition and integration method and system for super calculation and intelligent calculation Download PDF

Info

Publication number
CN117389838A
CN117389838A CN202311548433.6A CN202311548433A CN117389838A CN 117389838 A CN117389838 A CN 117389838A CN 202311548433 A CN202311548433 A CN 202311548433A CN 117389838 A CN117389838 A CN 117389838A
Authority
CN
China
Prior art keywords
data
computing
class
computing environment
acquisition task
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311548433.6A
Other languages
Chinese (zh)
Inventor
杨林泉
刘鹏
项君
陈越
易国栋
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Xiangjiang Laboratory
Changsha Xinchuang Digital Intelligence Technology Co ltd
Original Assignee
Xiangjiang Laboratory
Changsha Xinchuang Digital Intelligence Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Xiangjiang Laboratory, Changsha Xinchuang Digital Intelligence Technology Co ltd filed Critical Xiangjiang Laboratory
Priority to CN202311548433.6A priority Critical patent/CN117389838A/en
Publication of CN117389838A publication Critical patent/CN117389838A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/3089—Monitoring arrangements determined by the means or processing involved in sensing the monitored data, e.g. interfaces, connectors, sensors, probes, agents
    • G06F11/3093—Configuration details thereof, e.g. installation, enabling, spatial arrangement of the probes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25—Integrating or interfacing systems involving database management systems
    • G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • Quality & Reliability (AREA)
  • Data Mining & Analysis (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention relates to a method and a system for joint collection and integration of operation and data oriented to super calculation and intelligent calculation, wherein the method comprises the following steps: generating a first class of operation and maintenance data acquisition task; uniformly packaging and instantiating the operator container for the first class operation and data acquisition task, and judging whether the first class operation and data acquisition task needs to interoperate across computing environments; if so, submitting the first type of operation and maintenance acquisition task to a second computing environment through the heterogeneous resource engine so as to generate a second type of operation and maintenance data acquisition task; the second class operation and data acquisition task is unified in an ultra-calculation and intelligent calculation combined computing environment to carry out operator container packaging and instantiation to form a second computing environment to obtain a result operation and data set; submitting the first class of operation and maintenance data acquisition tasks through the heterogeneous resource engine; and obtaining a joint calculation result operation and maintenance data set. The scheme can provide basic capability guarantee for the operation and maintenance data combined acquisition and integration of the super-calculation and intelligent calculation combined computing environment.

Description

Operation and maintenance data combined acquisition and integration method and system for super calculation and intelligent calculation
Technical Field
The invention belongs to the field of intelligent computation, and particularly relates to a method and a system for operation and maintenance data combined acquisition and integration for super computation and intelligent computation.
Background
The super computer (super calculation) is a computer cluster which consists of thousands of CPUs and can calculate large-scale complex problems which cannot be completed by common computers, has extremely critical supporting function in a large number of scientific calculation and great engineering design fields, and the super calculation performance is also an important expression of national science and technology strength. In the past, the performance improvement of the super-computing system is mainly achieved by means of the improvement of hardware capability, and in recent years, with the breakthrough development of Artificial Intelligence (AI) technology, the acceleration of the scientific calculation of the super-computing system by an AI intelligent computing system has become a new way for improving the performance of the existing super-computing system.
However, after the two heterogeneous technology systems of the super-computing and the intelligent computing are fused, when the support is provided for unified service application, how to effectively develop the operation and maintenance guarantee of a cross-system, and the provision of the unified intelligent operation and maintenance capability is an important sign of the mature trend of the super-computing-intelligent computing fusion technology.
Disclosure of Invention
In order to solve the technical problems, the invention provides a technical scheme of a joint collection and integration method of operation and data oriented to super calculation and intelligent calculation, so as to solve the technical problems.
A joint collection and integration method of operation and data oriented to super calculation and intelligent calculation, the method comprising:
arranging and generating a first class of operation data acquisition tasks on a first computing environment;
the first type operation and data acquisition task is unified in an supercomputing and intelligent computing combined computing environment to carry out operator container packaging and instantiation, and according to the resource configuration condition of the first computing environment, based on a global resource discovery mechanism of a platform engine, whether the first type operation and data acquisition task needs to interoperate across the computing environment is judged by inquiring unified operation and data cache information;
if the operation and maintenance data resources required to be acquired by the first type of operation and maintenance data acquisition task need to be interoperated across computing environments, submitting the first type of operation and maintenance acquisition task to a second computing environment through a heterogeneous resource engine based on a data interoperation mechanism and a scene so as to generate a second type of operation and maintenance data acquisition task;
the second class operation and maintenance data acquisition task is unified in an ultra-calculation and intelligent calculation combined computing environment to carry out operator container packaging and instantiation; interoperating the instantiated second class operation data acquisition task and the resources of the second computing environment through a heterogeneous computing model to form a complete operation data acquisition task on the second computing environment, and computing and integrating the complete operation data acquisition task to obtain a result operation data set;
performing data bridging, type system conversion and dynamic metadata adaptation on the result operation and maintenance data set, and submitting the result operation and maintenance data set to a first computing environment through a heterogeneous resource engine; and (3) carrying out calculation integration with the first class of operation and data acquisition task to obtain a final joint calculation result operation and data set.
In one preferred embodiment, the submitting, by the heterogeneous resource engine, the first type of operation and maintenance collection task to a second computing environment based on a data interoperation mechanism and a scenario to generate a second type of operation and maintenance data collection task includes:
based on a data interoperation mechanism and a scene, pushing a computing environment through a heterogeneous resource engine to analyze an instantiated first class of operation and data acquisition task;
extracting deployment to the second computing environment through a platform preloading mode for computing resource preparation on the second computing environment;
and submitting the parsed first class operation and data acquisition task to a second computing environment through the heterogeneous resource engine to generate a second class operation and data acquisition task.
In one preferred embodiment, the step of packaging and instantiating the first-class operation and data collection task in an operator container in a super computing and intelligent computing combined computing environment comprises the following steps:
carrying out unified operator container packaging on the first type of operation and maintenance data acquisition tasks to form a first type of acquisition task operators;
submitting the first-class collection task operators to a unified operator execution engine for operator instantiation to obtain instantiated first-class operation data collection tasks.
In one preferred embodiment, the step of encapsulating and instantiating the second-class operation data collection task into an operator container in a joint computing environment of super computing and intelligent computing comprises the following steps:
carrying out unified operator container packaging on the second-class operation data acquisition task to form a second-class acquisition task operator;
submitting the second type acquisition task operators to a unified operator execution engine for operator instantiation to obtain instantiated second type acquisition tasks.
In one preferred embodiment, the interoperating the instantiated second class of operation data collection tasks and the resources of the second computing environment via the heterogeneous computing model includes:
the instantiated second-class operation data acquisition task discovers second-class data resources required for preparing the second-class operation data acquisition task through the data resources in the second computing environment.
In one preferred embodiment, the second type of data resource performs data resource registration through unified registration management of distributed resource data tags of heterogeneous resource engines.
In one preferred embodiment, the second type of data resource performs data resource registration through unified registration management of distributed resource data tags of heterogeneous resource engines, including:
the instantiated first-class operation and data acquisition task discovers first-class data resources required by the preparation of the first operation task through data resources on a first computing environment, and simultaneously introduces data resource registration information prepared by the second-class operation and data acquisition task through a distributed resource data tag unified registry.
In one preferred embodiment, the first-type operation and data collection task through data resource discovery on the first computing environment includes:
interoperating the instantiated first class operation and data acquisition task and the first class data resource through a heterogeneous computing model to form a local first class operation and data acquisition task.
The embodiment of the invention supports the joint calculation scheduling of the super-computing system by utilizing the unified encapsulation of the super-computing and intelligent computing joint computing environment resource container, the global registration and management of the operation and maintenance data model, the key management and control mechanism and the realization method, and eliminates the running time difference of the heterogeneous clusters by utilizing the heterogeneous resource engine container technology and the heterogeneous computing model expansion and labeling technology, thereby providing the realization possibility of extending expansion and splicing through unified arrangement and scheduling for the interoperation of heterogeneous computing tasks of the cross-computing environment. The scheme provided by the invention can provide basic capability guarantee for the operation and data combined acquisition and integration of the super-computing and intelligent computing combined computing environment.
An operation and maintenance data combined acquisition and integration system for super calculation and intelligent calculation comprises:
the first type acquisition task generation module is used for arranging and generating first type operation data acquisition tasks on a first computing environment;
the computing environment judging module is used for uniformly packaging and instantiating the first-class operation and data acquisition task in the super computing and intelligent computing combined computing environment, and judging whether the first-class operation and data acquisition task needs to interoperate across the computing environment by inquiring the uniform operation and data cache information based on a global resource discovery mechanism of the platform engine according to the resource configuration condition of the first computing environment;
the second class acquisition task generation module is used for submitting the first class operation and maintenance acquisition task to a second computing environment through the heterogeneous resource engine based on a data interoperation mechanism and a scene if operation and maintenance data resources required to be acquired by the first class operation and maintenance data acquisition task need to be interoperated across the computing environment so as to generate the second class operation and maintenance data acquisition task;
the computing integration module is used for uniformly packaging and instantiating the second class operation data acquisition task in an ultra-computing and intelligent computing combined computing environment; interoperating the instantiated second class operation data acquisition task and the resources of the second computing environment through a heterogeneous computing model to form a complete operation data acquisition task on the second computing environment, and computing and integrating the complete operation data acquisition task to obtain a result operation data set;
the data set acquisition module is used for carrying out data bridging, type system conversion and dynamic metadata adaptation on the result operation and maintenance data set and submitting the result operation and maintenance data set to a first computing environment through a heterogeneous resource engine; and calculating the first class operation and data acquisition task after the integrated submission to obtain a final joint calculation result operation and data set.
The embodiment of the invention supports the joint computation scheduling by utilizing the unified encapsulation of the super computation and intelligent computation joint computation environment resource container, the global registration and management of the operation and maintenance data model, the key management and control mechanism and the realization method, and eliminates the running time difference of the heterogeneous clusters by utilizing the heterogeneous resource engine container technology and the heterogeneous computation model expansion and labeling technology, thereby providing the realization possibility of extension expansion and splicing through unified arrangement and scheduling for the interoperation of heterogeneous computation tasks of the cross computation environment. The scheme provided by the invention can provide basic capability guarantee for the operation and data combined acquisition and integration of the super-computing and intelligent computing combined computing environment.
A computer storage medium having a computer program stored thereon, which when executed by a processor, is a joint collection and integration method of operation and dimension data for super and intellectual computing as described above.
The embodiment of the invention supports the joint computation scheduling of the cross-super computing system by utilizing the unified encapsulation of the super computing and intelligent computing joint computing environment resource container, the global registration and management of the operation and maintenance data model, the key management and control mechanism and the realization method, and eliminates the running time difference of the heterogeneous clusters by utilizing the heterogeneous resource engine container technology and the heterogeneous computing model expansion and labeling technology, thereby providing the realization possibility of extending expansion and splicing through unified arrangement and scheduling for the interoperation of the heterogeneous computing tasks of the cross computing environment. The scheme provided by the invention can provide basic capability guarantee for the operation and data combined acquisition and integration of the super-computing and intelligent computing combined computing environment.
Drawings
FIG. 1 is a flow chart of a method for joint collection and integration of operation and data for super calculation and intelligent calculation in a first preferred embodiment of the invention;
fig. 2 is a schematic block diagram of a system for joint collection and integration of operational and dimensional data for super-computing and intelligent computing in a second preferred embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
It will be understood that when an element is referred to as being "disposed on" another element, it can be directly on the other element or intervening elements may also be present. When an element is referred to as being "connected" to another element, it can be directly connected to the other element or intervening elements may also be present. The terms "vertical," "horizontal," "left," "right," and the like are used herein for illustrative purposes only and are not meant to be the only embodiment.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and/or" as used herein includes any and all combinations of one or more of the associated listed items.
As shown in fig. 1, in a preferred embodiment of the present invention, a method for joint collection and integration of operation and data oriented to super calculation and intelligent calculation is disclosed, which includes:
s10: arranging and generating a first class of operation data acquisition tasks on a first computing environment;
s20: the first type operation and data acquisition task is unified in an supercomputing and intelligent computing combined computing environment to carry out operator container packaging and instantiation, and according to the resource configuration condition of the first computing environment, based on a global resource discovery mechanism of a platform engine, whether the first type operation and data acquisition task needs to interoperate across the computing environment is judged by inquiring unified operation and data cache information;
in this step S20, the step of packaging and instantiating the operator container for the first-class operation and data collection task in the super-computing and intelligent computing combined computing environment includes:
carrying out unified operator container packaging on the first type of operation and maintenance data acquisition tasks to form a first type of acquisition task operators;
submitting the first-class collection task operators to a unified operator execution engine for operator instantiation to obtain instantiated first-class operation data collection tasks.
S30: if the operation and maintenance data resources required to be acquired by the first type of operation and maintenance data acquisition task need to be interoperated across computing environments, submitting the first type of operation and maintenance acquisition task to a second computing environment through a heterogeneous resource engine based on a data interoperation mechanism and a scene so as to generate a second type of operation and maintenance data acquisition task;
specifically, in the step S30, the following subdivision steps may be specifically included:
s31: based on a data interoperation mechanism and a scene, pushing a computing environment through a heterogeneous resource engine to analyze an instantiated first class of operation and data acquisition task;
s32: extracting the computing resources deployed in the second computing environment through a platform preloading mode to prepare the computing resources in the second computing environment;
s33: and submitting the parsed first class operation and data acquisition task to a second computing environment through the heterogeneous resource engine to generate a second class operation and data acquisition task.
S40: the second class operation and maintenance data acquisition task is unified in an ultra-calculation and intelligent calculation combined computing environment to carry out operator container packaging and instantiation; interoperating the instantiated second class operation data acquisition task and the resources of the second computing environment through a heterogeneous computing model to form a complete operation data acquisition task on the second computing environment, and computing and integrating the complete operation data acquisition task to obtain a result operation data set;
in this step, the step of performing the operator container packaging and instantiation on the second class of operation data collection task in the super-computing and intelligent computing combined computing environment includes:
carrying out unified operator container packaging on the second-class operation data acquisition task to form a second-class acquisition task operator;
submitting the second type acquisition task operators to a unified operator execution engine for operator instantiation to obtain instantiated second type acquisition tasks.
Specifically, in one of the preferred embodiments, the interoperating the instantiated second class of operation data collection tasks and the resources of the second computing environment via the heterogeneous computing model includes:
the instantiated second-class operation data acquisition task discovers second-class data resources required for preparing the second-class operation data acquisition task through the data resources in the second computing environment.
In one preferred embodiment, the second type of data resource performs data resource registration through unified registration management of distributed resource data tags of heterogeneous resource engines.
In one preferred embodiment, the second type of data resource performs data resource registration through unified registration management of distributed resource data tags of heterogeneous resource engines, including:
the instantiated first-class operation and data acquisition task discovers first-class data resources required by the first-class operation and data acquisition task through the data resources on the first computing environment, and simultaneously introduces data resource registration information prepared by the second-class operation and data acquisition task through a distributed resource data tag unified registry.
In one preferred embodiment, the first-type operation and data collection task through the data resource discovery in the first computing environment, the first-type data resource required for preparing the first-type operation and data collection task includes:
interoperating the instantiated first class operation and data acquisition task and the first class data resource through a heterogeneous computing model to form a local first class operation and data acquisition task.
S50: performing data bridging, type system conversion and dynamic metadata adaptation on the result operation and maintenance data set, and submitting the result operation and maintenance data set to a first type operation and maintenance data acquisition task through a heterogeneous resource engine; and calculating the first class operation and data acquisition task after the integrated submission to obtain a final joint calculation result operation and data set.
The embodiment of the invention supports the joint computation scheduling by utilizing the unified encapsulation of the super computation and intelligent computation joint computation environment resource container, the global registration and management of the operation and maintenance data model, the key management and control mechanism and the realization method, and eliminates the running time difference of the heterogeneous clusters by utilizing the heterogeneous resource engine container technology and the heterogeneous computation model expansion and labeling technology, thereby providing the realization possibility of extension expansion and splicing through unified arrangement and scheduling for the interoperation of heterogeneous computation tasks of the cross computation environment. The scheme provided by the invention can provide basic capability guarantee for the operation and data combined acquisition and integration of the super-computing and intelligent computing combined computing environment.
As shown in fig. 2, in a second preferred embodiment of the present invention, a system 100 for joint collection and integration of operation and data oriented to super calculation and intelligent calculation is disclosed, where the system 100 includes a first type collection task generating module 110, a computing environment judging module 120, a second type collection task generating module 130, a computing integration module 140 and a data set obtaining module 150.
The first type acquisition task generating module 110 is configured to schedule and generate a first type of operation data acquisition task on a first computing environment;
the computing environment judging module 120 is configured to perform operator container encapsulation and instantiation on the first type of operation and data acquisition task in a joint computing environment of super computing and intelligent computing, and judge whether the first type of operation and data acquisition task needs inter-operation across computing environments by querying the cache information of the unified operation and data acquisition task according to the resource configuration condition of the first computing environment and based on the global resource discovery mechanism of the platform engine;
the computing environment determining module 120 may specifically further include: carrying out unified operator container packaging on the first type of operation and maintenance data acquisition tasks to form a first type of acquisition task operators; submitting the first-class collection task operators to a unified operator execution engine for operator instantiation to obtain instantiated first-class operation data collection tasks.
The second type of collection task generating module 130 is configured to submit the first type of operation and maintenance collection task to a second computing environment through a heterogeneous resource engine based on a data interoperability mechanism and a scenario if operation and maintenance data resources required to be collected by the first type of operation and maintenance data collection task need to be interoperated across the computing environment, so as to generate a second type of operation and maintenance data collection task;
specifically, the second type acquisition task generating module 130 may further include: based on a data interoperation mechanism and a scene, pushing a computing environment through a heterogeneous resource engine to analyze an instantiated first class of operation and data acquisition task; extracting the computing resources deployed in the second computing environment through a platform preloading mode to prepare the computing resources in the second computing environment; and submitting the parsed first class operation and data acquisition task to a second computing environment through the heterogeneous resource engine to generate a second class operation and data acquisition task. The second class operation and maintenance data acquisition task is unified in an ultra-calculation and intelligent calculation combined computing environment to carry out operator container packaging and instantiation; interoperating the instantiated second class operation and data acquisition task and the resources of the second computing environment through a heterogeneous computing model to form a complete operation and data acquisition task on the second computing environment, and computing and integrating the complete operation and data acquisition task to obtain a result operation and data set;
the computing integration module 140 is configured to perform operator container packaging and instantiation on the second class of operation data collection tasks in a joint computing environment of super computing and intelligent computing; interoperating the instantiated second class operation data acquisition task and the resources of the second computing environment through a heterogeneous computing model to form a complete operation data acquisition task on the second computing environment, and computing and integrating the complete operation data acquisition task to obtain a result operation data set;
specifically, the computing integration module 140 includes: carrying out unified operator container packaging on the second-class operation data acquisition task to form a second-class acquisition task operator; submitting the second-class collection task operator to a unified operator execution engine for operator instantiation to obtain an instantiated second-class operation data collection task.
Specifically, in one of the preferred embodiments, the interoperating the instantiated second class of operation data collection task and the computing resources of the second environment through the heterogeneous computing model includes:
the instantiated second-class operation data acquisition task discovers second-class data resources required for preparing the second-class operation data acquisition task through the data resources in the second computing environment.
In one preferred embodiment, the second type of data resource performs data resource registration through unified registration management of distributed resource data tags of heterogeneous resource engines.
In one preferred embodiment, the second type of data resource performs data resource registration through unified registration management of distributed resource data tags of heterogeneous resource engines, including:
the instantiated first-class operation and data acquisition task discovers first-class data resources required by the preparation of the first operation task through data resources on a first computing environment, and simultaneously introduces data resource registration information prepared by the second-class operation and data acquisition task through a distributed resource data tag unified registry.
In one preferred embodiment, the first-type operation and data collection task through data resource discovery on the first computing environment includes:
interoperating the instantiated first class operation and data acquisition task and the first class data resource through a heterogeneous computing model to form a local first class operation and data acquisition task.
The data set obtaining module 150 is configured to perform data bridging, type system conversion and dynamic metadata adaptation on the result operation and maintenance data set, and submit the result operation and maintenance data set to a first type operation and maintenance data collection task through a heterogeneous resource engine; and calculating the first class operation and data acquisition task after the integrated submission to obtain a final joint calculation result operation and data set.
The embodiment of the invention supports the joint computation scheduling of the cross-super computing system by utilizing the unified encapsulation of the super computing and intelligent computing joint computing environment resource container, the global registration and management of the operation and maintenance data model, the key management and control mechanism and the realization method, and eliminates the running time difference of the heterogeneous clusters by utilizing the heterogeneous resource engine container technology and the heterogeneous computing model expansion and labeling technology, thereby providing the realization possibility of extending expansion and splicing through unified arrangement and scheduling for the interoperation of the heterogeneous computing tasks of the cross computing environment. The scheme provided by the invention can provide basic capability guarantee for the operation and data combined acquisition and integration of the super-computing and intelligent computing combined computing environment.
In another preferred embodiment of the present invention, a computer storage medium is disclosed, where a computer program is stored, where the computer program, when executed by a processor, is a method for jointly collecting and integrating operation and dimension data for super-computing and intelligent computing.
The embodiment of the invention supports the cross-super computing system joint computing scheduling by utilizing the unified packaging of the super computing and intelligent computing joint computing environment resource container, the global registration and management of the operation and maintenance data model, the key management and control mechanism and the realization method, and eliminates the running time difference of the heterogeneous clusters by utilizing the heterogeneous resource engine container technology and the heterogeneous computing model expansion and labeling technology, thereby providing the realization possibility of extending expansion and splicing through unified arrangement and scheduling for the interoperation of the heterogeneous computing tasks of the cross computing environment. The scheme provided by the invention can provide basic capability guarantee for the operation and data combined acquisition and integration of the super-computing and intelligent computing combined computing environment.
It should be noted that the computer storage medium described in the present disclosure may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer storage medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer storage medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, fiber optic cables, RF (radio frequency), and the like, or any suitable combination of the foregoing.
In some implementations, the clients, servers may communicate using any currently known or future developed network protocol, such as HTTP (Hyper Text Transfer Protocol ), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the internet (e.g., the internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed networks.
The computer storage medium may be contained in the electronic device; or may exist alone without being incorporated into the electronic device.
The computer storage medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:
the technical features of the above embodiments may be arbitrarily combined, and for brevity, all of the possible combinations of the technical features of the above embodiments are not described, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples illustrate only a few embodiments of the invention, which are described in detail and are not to be construed as limiting the scope of the invention. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the invention, which are all within the scope of the invention. Accordingly, the scope of protection of the present invention is to be determined by the appended claims.

Claims (9)

1. The operation and maintenance data combined acquisition and integration method for super calculation and intelligent calculation is characterized by comprising the following steps of:
arranging and generating a first class of operation data acquisition tasks on a first computing environment;
the first type operation and data acquisition task is unified in an supercomputing and intelligent computing combined computing environment to carry out operator container packaging and instantiation, and according to the resource configuration condition of the first computing environment, based on a global resource discovery mechanism of a platform engine, whether the first type operation and data acquisition task needs to interoperate across the computing environment is judged by inquiring unified operation and data cache information;
if the operation and maintenance data resources required to be acquired by the first type of operation and maintenance data acquisition task need to be interoperated across computing environments, submitting the first type of operation and maintenance acquisition task to a second computing environment through a heterogeneous resource engine based on a data interoperation mechanism and a scene so as to generate a second type of operation and maintenance data acquisition task;
the second class operation and maintenance data acquisition task is unified in an ultra-calculation and intelligent calculation combined computing environment to carry out operator container packaging and instantiation; interoperating the instantiated second class operation data acquisition task and the resources of the second computing environment through a heterogeneous computing model to form a complete operation data acquisition task on the second computing environment, and computing and integrating the complete operation data acquisition task to obtain a result operation data set;
performing data bridging, type system conversion and dynamic metadata adaptation on the result operation and maintenance data set, and submitting the result operation and maintenance data set to a first computing environment through a heterogeneous resource engine; and (3) carrying out calculation integration with the first class of operation and data acquisition task to obtain a final joint calculation result operation and data set.
2. The super and intelligent computing oriented operation and data joint collection and integration method according to claim 1, wherein the submitting the first kind of operation and data collection task to a second computing environment through a heterogeneous resource engine based on a data interoperation mechanism and a scene to generate a second kind of operation and data collection task comprises:
based on a data interoperation mechanism and a scene, pushing a computing environment through a heterogeneous resource engine to analyze an instantiated first class of operation and data acquisition task;
extracting deployment to the second computing environment through a platform preloading mode to prepare computing resources in the second computing environment;
and submitting the parsed first class operation and data acquisition task to a second computing environment through the heterogeneous resource engine to generate a second class operation and data acquisition task.
3. The joint collection and integration method of operation and data for super computing and intelligent computing according to claim 1, wherein the unifying the operation and data collection tasks of the first type in the super computing and intelligent computing joint computing environment comprises:
carrying out unified operator container packaging on the first type of operation and maintenance data acquisition tasks to form a first type of acquisition task operators;
submitting the first type of collection task operators to a unified operator execution engine for operator instantiation to obtain instantiated first type of operation data collection tasks.
4. The joint collection and integration method of operation and data for super computing and intelligent computing according to claim 1, wherein said unifying the operation and data collection tasks of the second class in the super computing and intelligent computing joint computing environment comprises:
carrying out unified operator container packaging on the second-class operation data acquisition task to form a second-class acquisition task operator;
submitting the second type acquisition task operators to a unified operator execution engine for operator instantiation to obtain instantiated second type acquisition tasks.
5. The joint collection and integration method of operation and data oriented to super computing and intelligent computing according to claim 1, wherein the interoperating the instantiated second type of operation and data collection task and the resource of the second computing environment through the heterogeneous computing model comprises:
the instantiated second-class operation data acquisition task discovers second-class data resources required for preparing the second-class operation data acquisition task through the data resources in the second computing environment.
6. The joint collection and integration method of operation and data oriented to super computing and intelligent computing according to claim 5, wherein the second type of data resources are subjected to data resource registration through unified registration management of distributed resource data tags of heterogeneous resource engines.
7. The method for joint collection and integration of operation and data oriented to super computing and intelligent computing according to claim 6, wherein the data resource registration of the second class of data resources through the distributed resource data tag unified registration management of heterogeneous resource engine comprises:
the instantiated first-class operation and data acquisition task discovers first-class data resources required by the first-class operation and data acquisition task through the data resources on the first computing environment, and simultaneously introduces data resource registration information prepared by the second-class operation and data acquisition task through the distributed resource data tag unified registry.
8. The super and intelligent oriented operation and data combined collection and integration method according to claim 7, wherein the instantiated operation and data collection task of the first type discovers the first type data resources required for preparing the operation and data collection task of the first type through the data resources on the first computing environment, and comprises the following steps:
interoperating the instantiated first class operation and data acquisition task and the first class data resource through a heterogeneous computing model to form a local first class operation and data acquisition task.
9. The utility model provides a fortune dimension joint collection and integrated system towards super calculation and intelligent calculation which characterized in that includes:
the first type acquisition task generation module is used for arranging and generating first type operation data acquisition tasks on a first computing environment;
the computing environment judging module is used for uniformly packaging and instantiating the first-class operation and data acquisition task in the super computing and intelligent computing combined computing environment, and judging whether the first-class operation and data acquisition task needs to interoperate across the computing environment by inquiring the uniform operation and data cache information based on a global resource discovery mechanism of the platform engine according to the resource configuration condition of the first computing environment;
the second class acquisition task generation module is used for submitting the first class operation and maintenance acquisition task to a second computing environment through the heterogeneous resource engine based on a data interoperation mechanism and a scene if operation and maintenance data resources required to be acquired by the first class operation and maintenance data acquisition task need to be interoperated across the computing environment so as to generate the second class operation and maintenance data acquisition task;
the computing integration module is used for uniformly packaging and instantiating the second class operation data acquisition task in an ultra-computing and intelligent computing combined computing environment; interoperating the instantiated second class operation data acquisition task and the resources of the second computing environment through a heterogeneous computing model to form a complete operation data acquisition task on the second computing environment, and computing and integrating the complete operation data acquisition task to obtain a result operation data set;
the data set acquisition module is used for carrying out data bridging, type system conversion and dynamic metadata adaptation on the result operation and maintenance data set and submitting the result operation and maintenance data set to a first computing environment through a heterogeneous resource engine; and calculating the first class operation and data acquisition task after the integrated submission to obtain a final joint calculation result operation and data set.
CN202311548433.6A 2023-11-20 2023-11-20 Operation and maintenance data combined acquisition and integration method and system for super calculation and intelligent calculation Pending CN117389838A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311548433.6A CN117389838A (en) 2023-11-20 2023-11-20 Operation and maintenance data combined acquisition and integration method and system for super calculation and intelligent calculation

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311548433.6A CN117389838A (en) 2023-11-20 2023-11-20 Operation and maintenance data combined acquisition and integration method and system for super calculation and intelligent calculation

Publications (1)

Publication Number Publication Date
CN117389838A true CN117389838A (en) 2024-01-12

Family

ID=89439298

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311548433.6A Pending CN117389838A (en) 2023-11-20 2023-11-20 Operation and maintenance data combined acquisition and integration method and system for super calculation and intelligent calculation

Country Status (1)

Country Link
CN (1) CN117389838A (en)

Similar Documents

Publication Publication Date Title
CN109240821B (en) Distributed cross-domain collaborative computing and service system and method based on edge computing
Khriji et al. Design and implementation of a cloud-based event-driven architecture for real-time data processing in wireless sensor networks
CN112511218B (en) Satellite ground station monitoring system based on microservice
Ahmed et al. Benefits and challenges of internet of things for telecommunication networks
CN114189274A (en) Satellite ground station monitoring system based on microservice
CN111885439A (en) Optical network integrated management and duty management system
Babu et al. Fog computing Qos review and open challenges
Nkenyereye et al. MEIX: Evolving multi-access edge computing for industrial Internet-of-Things services
Xiao et al. A hybrid task crash recovery solution for edge computing in IoT-based manufacturing
Shah Multi-agent cognitive architecture-enabled IoT applications of mobile edge computing
Kruger et al. Towards the integration of digital twins and service-oriented architectures
Nakamura et al. FUDGE: A frugal edge node for advanced IoT solutions in contexts with limited resources
Alaya et al. Frameself: A generic context-aware autonomic framework for self-management of distributed systems
CN112486666A (en) Model-driven reference architecture method and platform
CN112039985A (en) Heterogeneous cloud management method and system
Zhang et al. Efficient online surveillance video processing based on spark framework
CN116755799A (en) Service arrangement system and method
Baldoni et al. Zenoh-based dataflow framework for autonomous vehicles
CN117389838A (en) Operation and maintenance data combined acquisition and integration method and system for super calculation and intelligent calculation
CN114884830B (en) Distributed parallel simulation deduction system based on wide area network
Rekik et al. Application of a CAN BUS transport for DDS middleware
Badidi Towards a message broker based platform for real-time streaming of urban iot data
Wang et al. Design of satellite ground management system based on microservices
Singh et al. Developing novagenesis architecture for internet of things services: Observation, challenges and itms application
Xu et al. Software defined industrial network: architecture and edge offloading strategy

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination