CN115202874A - Method and storage medium for scheduling data transmission system resources based on declarative API - Google Patents
Method and storage medium for scheduling data transmission system resources based on declarative API Download PDFInfo
- Publication number
- CN115202874A CN115202874A CN202210792245.7A CN202210792245A CN115202874A CN 115202874 A CN115202874 A CN 115202874A CN 202210792245 A CN202210792245 A CN 202210792245A CN 115202874 A CN115202874 A CN 115202874A
- Authority
- CN
- China
- Prior art keywords
- resource
- service
- resources
- data transmission
- dts
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/547—Messaging middleware
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method for scheduling data transmission system resources based on an declarative API (application program interface) and a storage medium, belonging to the technical field of resource scheduling in a Kubernets environment, aiming at solving the technical problem of how to realize convenient and rapid scheduling of data transmission system resources by utilizing Kubernets resource scheduling capability, and adopting the technical scheme that: the method comprises the following specific steps: custom API resource type CRD: respectively defining resource types CRD of acquisition end resources, warehousing end resources and message middleware resources of a data transmission system; wherein the resource type CRD group is dts; scheduling the acquisition end resource, the warehousing end resource and the message middleware resource by using the resource description file corresponding to the resource type CRD, and capturing the applied resource file by a user-defined controller to realize corresponding resource scheduling work; the abstraction of bottom physical resources and service logic resources is completed through the definition of resource files, and the scheduling of the service resources is realized through a user-defined controller.
Description
Technical Field
The invention relates to the technical field of resource scheduling in a Kubernetes environment, in particular to a data transmission system resource scheduling method based on an declarative API and a storage medium.
Background
kubernets, K8s for short, is an abbreviation resulting from 8 replacing 8 characters "ubernet". The Kubernets are open-source and used for managing containerized applications on a plurality of hosts in a cloud platform, the goal of the Kubernets is to make the deployment of containerized applications simple and efficient (powerfull), and the Kubernets provide a mechanism for application deployment, planning, updating and maintenance. A traditional deployment of applications is to install the applications through plug-ins or scripts. The disadvantage of this is that the running, configuration, management, and all life cycles of the application will be bound to the current operating system, which is not beneficial to the upgrade update/rollback and other operations of the application, and certainly, some functions can be implemented by creating a virtual machine, but the virtual machine is very heavy and not beneficial to portability. The new mode is realized by deploying containers, each container is isolated from each other, each container has a file system, processes among the containers cannot influence each other, and computing resources can be distinguished. Compared with a virtual machine, the container can be rapidly deployed, and due to the fact that the container is decoupled from underlying facilities and a machine file system, the container can be migrated between different clouds and different versions of operating systems. The container occupies less resources and is fast to deploy, each application can be packaged into a container mirror image, the container has greater advantages due to the one-to-one relationship between each application and the container, and the container mirror image can be created for the application at the stage of build or release by using the container, because each application does not need to be combined with the rest of application stacks and does not depend on the production environment infrastructure, and a consistent environment can be provided from research and development to test and production. Similarly, containers are lighter weight, more "transparent" than virtual machines, which is more convenient to monitor and manage.
Kubernetes is a container orchestration engine for Google open sources that supports automated deployment, large-scale scalable, application containerization management. When an application is deployed in a production environment, multiple instances of the application are typically deployed to load balance application requests. In kubernets, a plurality of containers can be created, each container runs an application instance, and then management, discovery and access of the group of application instances are realized through a built-in load balancing strategy, and the details do not need operation and maintenance personnel to perform complicated manual configuration and processing.
At ordinary times, when a simple Webserver is deployed in a kubernets cluster, a Deployment controller needs to be written first, then a Service object is created, association is performed through a label of a Pod, and finally a Service is exposed through Ingress or type = nodoport type Service, and each time, the operation needs to be slightly troublesome.
Therefore, how to realize convenient and rapid scheduling of data transmission system resources by using the kubernets resource scheduling capability is a technical problem to be solved urgently at present.
Disclosure of Invention
The technical task of the invention is to provide a data transmission system resource scheduling method and a storage medium based on an declarative API, so as to solve the problem of how to realize convenient and rapid scheduling of data transmission system resources by utilizing Kubernets resource scheduling capability.
The technical task of the invention is realized in the following way, a method for scheduling data transmission system resources based on an declarative API, which comprises the following steps:
custom API resource type CRD: respectively defining resource types CRD of acquisition end resources, warehousing end resources and message middleware resources of the data transmission system; wherein the resource type CRD group is dts;
scheduling the acquisition end resource, the warehousing end resource and the message middleware resource by using the resource description file corresponding to the resource type CRD, and capturing the applied resource file by a user-defined controller to realize corresponding resource scheduling work;
the abstraction of bottom physical resources and service logic resources is completed through the definition of resource files, and the scheduling of the service resources is realized through a user-defined controller.
Preferably, the data transmission system comprises an acquisition component, a message middleware and a storage component, wherein the acquisition component reads data and writes the data into the message middleware, the message middleware caches the read data to ensure the integrity of the data, and the acquisition component and the message are arranged in a pod; the acquisition component and the message middleware respectively define a resource type CRD of type dtsReder, and the warehousing component defines a resource type CRD of type dtsWriter;
the acquisition assembly is used for reading data from a source database;
the message middleware is used for storing and transmitting data;
and the warehousing component is used for writing the acquired data into a target database.
Preferably, the fields of the resource type CRD of dtsReader and dtsWriter are as follows:
taskId: unique identification of the data transmission task;
version: the version number of the product, the version numbers of the background database acquisition component and the warehousing component are consistent with the version number of the product;
isAvailable: whether a resource is available;
dbInfo: data source information, connection information of a source end or a destination end database;
task: task configuration information, wherein the task configuration information comprises table information, field information and related transmission strategy parameters;
resources: and the resource information comprises cpu information, memory information and storage information of the pod needing to be pulled up.
Preferably, the self-defined controller is that a corresponding CDR controller is created for the resource type CRD, so that when the CRD object changes, a service logic code in the CRD controller is triggered; the method comprises the following specific steps:
a data transmission system creates a custom controller dts-Operator;
a dts-Operator of the custom controller acquires and monitors the change of an object instance by using a ListAndWatch method of a Reflector;
when any one instance is changed, the Reflector receives the event notification and stores the combination of the event and the object corresponding to the object instance into a first-in first-out queue after receiving the notification;
the custom controller dts-Operator uses the Informer to continuously read the object from the first-in first-out queue, and judges whether the type of the event is dtsReader or dtsWriter:
and after the type of the event is determined, creating resources required by dtsReder or dtsWriter according to the type of the event.
Preferably, the resources required to create dtsReader are specifically as follows:
a custom controller dts-Operator creates pod and service resources of a message middleware nat on which an acquisition component depends;
creating a jetstream and a consumer resource in a message middleware nats;
after the nats resources of the message middleware are deployed, deploying the pod, the service and the storage resources of the acquisition component;
by monitoring corresponding events, physical resources and service configuration can be processed according to self-defined logic;
when the processing is finished, the acquisition assembly required by the data transmission task can work normally;
the resources required to create dtsWriter are specifically as follows:
a custom controller dts-Operator creates pod and service physical resources required by the warehousing component and generates a configuration file related to a data transmission task;
writing information related to the message middleware nat into a configuration file;
by monitoring the corresponding events, the physical resources and the service configuration can be processed according to the self-defined logic;
and after the processing is finished, the warehousing component required by the data transmission task can work normally.
Preferably, the custom controller dts-Operator obtains a dtsWriter object and a dtsReder object from an APIServer of Kubernets by means of a code block of the Informer, the informers correspond to the API objects one by one, and the dts-Operator is an Informer of a Network object and is transmitted to the custom controller dts-Operator in the data transmission process;
when the Informer factory is created, a Network client is transmitted to the Informer factory, the Network Informer uses the Network client to establish connection with the APIServer, a ListAndWatch mechanism in a Reflector package used by the Informer is used for maintaining the connection between the Network Informer and the APIServer, and the ListAndWatch mechanism is used for acquiring and monitoring the change of a Network object instance;
under a ListAndWatch mechanism, once a new Network instance is created, deleted or updated at an APIServer end, a Reflector receives an event notification corresponding to the Network instance, an event and an API object combination corresponding to the event are called increment Delta and are put into an increment first-in first-out Queue (Delta FIFO Queue); meanwhile, the Inform continuously reads the increment from the increment first-in first-out queue, and every time an increment is read, the Inform judges the event type in the increment and creates or updates the buffer of the local object.
More preferably, the Informer has the following functions:
(1) synchronizing the local cache; the Informer is provided with a cache and an index mechanism; triggering a client library of a Handler, wherein a local cache is called Store in Kubernets, and an Index is called Index; the Informer uses a Reflector packet, and the Reflector packet acquires and monitors client encapsulation of API object change through a ListAndWatch mechanism; the Reflector packet and the Informer are coordinated by an increment first-in first-out queue;
(2) triggering and realizing the registered ResourceEventHandler (resource event handler) according to the event type; the Informer updates the local cache in real time through monitored event changes and calls an eventlandler (event handler) corresponding to the event, and the local cache maintained by the Informer is forcibly updated once by using a result returned by the latest LIST every time the resynperiod passes, so that the effectiveness of the cache is ensured; when a custom controller dts-Operator is created, the custom controller dts-Operator is registered to an Informer corresponding to the ResourceEventHandler;
before compiling a control cycle, an actor uses a work queue to carry out cooperation; in practical applications, all codes except the control loop are actually automatically generated by Kubernetes; wherein, the logic of the inner former control loop is as follows:
waiting for an inform to finish a data synchronization operation of a local cache;
starting one or more endless loop tasks by the goroutine;
in each cycle, executing the data transmission service logic codes;
if the control loop does not acquire the information of the corresponding object in the Informer cache, the control loop indicates that the deleting logic is to be executed;
if the object information is retrieved from the cache, the controller mode is executed to compare the expected state with the actual state, the expected state coming from the etcd actual state coming from the cluster itself.
Preferably, the custom API resource type CRD further includes the following:
carrying out Initializer operation and Validation operation of the API object: validation operation verifies the validity of each field in the object, the validity is stored in a Registry data structure after verification, and if the definition of an API object can be found in the Registry, the API is correspondingly an effective k8sAPI object;
and converting the superversion object into the object serialization of the user submitted version, and storing the object serialization into the etcd.
Preferably, the data transmission system defines a management end Service dts-Service, the management end Service dts-Service is used for receiving a user instruction sent by a front end, and the management end Service dts-Service interacts with a back end custom controller dts-Operator in an audible API mode to realize the creation of data transmission task resources and the operation logic control of tasks;
the creation of the resource is specifically as follows:
a user creates a task on a front-end interface, and the front-end interface calls a management end Service dts-Service to create a data transmission task method through an http request;
the management end Service dts-Service sequentially applies the data transmission tasks dtsReader and the resource types CRD of the dtsWriter by accessing the API of Kubernets, and the name fields in the resource types CRD are consistent with the task numbers to distinguish the resource types CRD of different tasks;
the resource type CRD of the application can be monitored by a custom controller dts-Operator and corresponding resource creating logic is executed;
after the resources are created, the acquisition component, the warehousing component and the message middleware nats of the database automatically complete initialization work;
after detecting that the resources of the acquisition component and the warehousing component of the task are successfully distributed and initialized normally, the management end Service dts-Service returns a message of successful creation to the user;
the operation logic control of the service is specifically as follows:
the user starts, stops, pauses, continues to transmit, changes the password and deletes the data transmission task through the front-end interface; the front-end interface calls a corresponding method in the management-end Service dts-Service through an http request;
the management end Service dts-Service respectively assembles resource types CRD of dtsReder and dtsWriter corresponding to the data transmission task, and the name field of the resource type CRD is consistent with the task number and is used for distinguishing the resource types CRD of different tasks;
the operation field in the assembled resource type CRD corresponds to the operation of the user, for example, the value corresponding to the operation field when the operation is stopped is stopTask;
the management end Service dts-Service respectively applies the task dtsReader and the resource type CRD of the dtsWriter by accessing the API of Kubernets;
and after monitoring the CRD version change of the resource type, the management end service dts-Operator analyzes the content of the operation field to judge the operation of the user, and calls a grpc interface of the acquisition component and a corresponding operation of the library entry component respectively according to the operation type, thereby completing the operation of the user.
A computer readable storage medium having stored thereon a computer program executable by a processor for implementing a method for declarative API-based scheduling of data transmission system resources as described above.
The method for scheduling the resources of the data transmission system based on the declarative API and the storage medium have the following advantages:
the data transmission system uses CRD to define the resource corresponding to the service model, captures the applied resource file through the user-defined controller, and the user-defined controller completes the scheduling of the required resource according to the written rule;
the invention utilizes the strong Kubernetes resource arrangement ability of the declarative API to realize the abstraction and definition of the service resources of the data transmission system and schedule the resources required by the data transmission system;
the invention realizes the resource scheduling of the data transmission system service by using the declarative API, can better describe the service resource by self-defining the resource, can better and efficiently use the Kubernets project arrangement capability by a self-defining controller, and is mainly embodied in the following aspects:
(1) the invention allows a plurality of API write terminals to modify the API object in a PATCH way without concerning the content of the local original YAML file;
(2) the invention can complete the tuning (Reconfile) process of the actual state and the expected state based on the addition, deletion, modification and check of the API object under the condition of completely not needing external intervention.
Drawings
The invention is further described below with reference to the accompanying drawings.
Fig. 1 is a schematic structural diagram of a method for scheduling resources of a data transmission system based on an declarative API.
Detailed Description
The method and storage medium for scheduling resources of a data transmission system based on an declarative API of the invention are described in detail below with reference to the accompanying drawings and embodiments.
Example 1:
the embodiment provides a method for scheduling resources of a data transmission system based on an declarative API, which specifically includes the following steps:
s1, self-defining API resource type CRD: respectively defining resource types CRD of acquisition end resources, warehousing end resources and message middleware resources of a data transmission system; wherein the resource type CRD group is dts;
s2, scheduling the acquisition end resource, the warehousing end resource and the message middleware resource by applying the resource description file corresponding to the resource type CRD, and capturing the applied resource file by a user-defined controller to realize corresponding resource scheduling work;
s3, abstracting bottom layer physical resources and business logic resources through the definition of the resource files, and scheduling the business resources through a custom controller.
The data transmission system in the embodiment comprises an acquisition component, a message middleware and a storage component, wherein the acquisition component reads data and writes the data into the message middleware, the message middleware caches the read data to ensure the integrity of the data, and the acquisition component and the message are deployed in one pod; the acquisition component and the message middleware respectively define a resource type CRD of a type dtsReader, and the warehousing component defines a resource type CRD of a type dtsWriter;
the acquisition component is used for reading data from a source database;
the message middleware is used for storing and transmitting data;
and the warehousing component is used for writing the acquired data into a target database.
The fields of resource types CRD of dtsReader and dtsWriter in this embodiment are specifically as follows:
taskId: unique identification of the data transmission task;
version: the version number of the product, the version numbers of the background database acquisition component and the warehousing component are consistent with the version number of the product;
isAvailable: whether a resource is available;
dbInfo: data source information, connection information of a source end or a destination end database;
task: task configuration information, wherein the task configuration information comprises table information, field information and related transmission strategy parameters;
resources: and the resource information comprises cpu information, memory information and storage information of the pod needing to be pulled up.
The self-defined controller in the embodiment is to create a corresponding CDR controller for the resource type CRD, so that when a CRD object changes, a service logic code in the CRD controller is triggered; the method comprises the following specific steps:
the method comprises the steps that firstly, a data transmission system creates a custom controller dts-Operator;
secondly, the custom controller dts-Operator uses a ListAndWatch method of the Reflector to acquire and monitor the change of the object instance;
when any one instance is changed, the Reflector receives the event notification and stores the combination of the event and the object corresponding to the object instance into a first-in first-out queue after receiving the notification;
and (IV) continuously reading the object from the first-in first-out queue by using an Informer by the custom controller dts-Operator, and judging whether the type of the event is dtsReader or dtsWriter:
and (V) after the type of the event is determined, creating resources required by dtsReder or dtsWriter according to the type of the event.
In this embodiment, the resources required to create dtsReader are specifically as follows:
(1) The custom controller dts-Operator creates pod and service resources of the message middleware nat on which the acquisition component depends;
(2) Creating jetstream and consumer resources in the message middleware nats;
(3) After the nats resources of the message middleware are deployed, deploying the pod, the service and the storage resources of the acquisition component;
(4) The physical resources and the service configuration can be processed according to the self-defined logic by monitoring the corresponding events;
(5) When the processing is finished, the acquisition assembly required by the data transmission task can work normally;
in this embodiment, the resources required to create dtsWriter are specifically as follows:
(1) The custom controller dts-Operator creates pod and service physical resources required by the warehousing component and generates a configuration file related to a data transmission task;
(2) Writing the information related to the message middleware nat into a configuration file;
(3) The physical resources and the service configuration can be processed according to the self-defined logic by monitoring the corresponding events;
(4) And after the processing is finished, the warehousing assembly required by the data transmission task can work normally.
In the embodiment, a dts-Operator of a custom controller acquires a dtsWriter object and a dtsReder object from an APIServer of Kubernets by means of a code block of the Informer, wherein the informers correspond to the API objects one by one, and the dts-Operator is an Informer of a Network object and is transmitted to the custom controller in a data transmission process;
when the Informer factory is created, a Network client is transmitted to the Informer factory, the Network Informer uses the Network client to establish connection with the APIServer, a ListAndWatch mechanism in a Reflector package used by the Informer is used for maintaining the connection between the Network Informer and the APIServer, and the ListAndWatch mechanism is used for acquiring and monitoring the change of a Network object instance;
under a ListAndWatch mechanism, once a new Network instance is created, deleted or updated at an APIServer end, a Reflector receives an event notification corresponding to the Network instance, an event and an API object combination corresponding to the event are called increment Delta and are put into an increment first-in first-out Queue (Delta FIFO Queue); meanwhile, the Inform continuously reads the increment from the increment first-in first-out queue, and every time an increment is read, the Inform judges the event type in the increment and creates or updates the buffer of the local object.
The Informer in this embodiment has the following functions:
(1) synchronizing the local cache; the Informer is provided with a cache and an index mechanism; triggering a client library of a Handler, wherein a local cache is called Store in Kubernets, and an Index is called Index; the Informer uses a Reflector package, and the Reflector package acquires and monitors client encapsulation of API object change through a ListAndWatch mechanism; the Reflector packet and the Informer are coordinated by an increment first-in first-out queue;
(2) triggering and realizing the registered ResourceEventHandler (resource event handler) according to the event type; the Informer updates the local cache in real time through monitored event changes and calls an eventlandler (event handler) corresponding to the event, and the local cache maintained by the Informer is forcibly updated once by using a result returned by the latest LIST every time the resynperiod passes, so that the effectiveness of the cache is ensured; when a custom controller dts-Operator is created, the custom controller dts-Operator is registered to an Informer corresponding to the ResourceEventHandler;
before compiling a control cycle, an informer uses a work queue for cooperation; in practical applications, all codes except the control loop are actually automatically generated by kubernets; wherein, the logic of the inner former control loop is as follows:
(1) Waiting for the inform to finish the data synchronization operation of the local cache;
(2) Starting one or more endless loop tasks by the goroutine;
(3) In each cycle, the data transmission service logic code is executed:
(1) if the control loop does not acquire the information of the corresponding object in the Informer cache, the control loop indicates that the deletion logic is to be executed;
(2) and if the object information is acquired from the cache, comparing the expected state and the actual state in the execution controller mode, wherein the expected state comes from the etcd actual state and comes from the cluster.
In this embodiment, the custom API resource type CRD further includes the following contents:
(1) and performing Initializer operation and Validation operation of the API object: validation operation verifies the validity of each field in the object, the validity is stored in a Registry data structure after verification, and if the definition of an API object can be found in the Registry, the API is correspondingly an effective k8sAPI object;
(2) and converting the superversion object into an object serialization of a user submitted version, and storing the object serialization into the etcd.
As shown in fig. 1, the data transmission system in this embodiment defines a management-side Service dts-Service, where the management-side Service dts-Service is configured to receive a user instruction sent by a front end, and interact with a back-end custom controller dts-Operator in an explicit API manner to implement creation of data transmission task resources and operation logic control of tasks;
the creation of resources in this embodiment is specifically as follows:
(1) The user creates a task on the front-end interface, and the front-end interface calls a management end Service dts-Service to create a data transmission task method through an http request;
(2) The management end Service dts-Service sequentially applies the data transmission tasks dtsReader and the resource types CRD of the dtsWriter by accessing the API of Kubernets, and the name fields in the resource types CRD are consistent with the task numbers to distinguish the resource types CRD of different tasks;
(3) The resource type CRD of the application can be monitored by a custom controller dts-Operator and corresponding resource creating logic is executed;
(4) After the resources are established, automatically finishing initialization work by an acquisition component, a warehousing component and a message middleware (nat) of the database;
(5) After detecting that the resources of the acquisition component and the warehousing component of the task are successfully distributed and initialized normally, the management end Service dts-Service returns a message of successful creation to the user;
the operation logic control of the service in this embodiment is specifically as follows:
(1) The user starts, stops, pauses, continues to transmit, changes the password and deletes the data transmission task through the front-end interface; the front-end interface calls a corresponding method in the management end Service dts-Service through an http request;
(2) The management end Service dts-Service respectively assembles resource types CRD of dtsReder and dtsWriter corresponding to the data transmission task, and the name field of the resource type CRD is consistent with the task number and is used for distinguishing the resource types CRD of different tasks;
(3) The operation field in the assembled resource type CRD corresponds to the operation of the user, for example, the value corresponding to the operation field when the operation is stopped is stopTask;
(4) The management end Service dts-Service respectively applies the task dtsReader and the resource type CRD of the dtsWriter by accessing the API of Kubernetes;
(5) And after monitoring the change of the CRD version of the resource type, the management end service dts-Operator analyzes the content of the operation field to judge the operation of the user, and respectively calls grpc interfaces of the acquisition component and the library entry component for corresponding operation according to the operation type, thereby completing the operation of the user.
Example 2:
the present embodiments also provide a computer-readable storage medium having stored thereon a plurality of instructions, which are loadable by a processor and cause the processor to execute the method for declarative API-based scheduling of data transmission system resources in any of the embodiments of the invention. Specifically, a system or an apparatus equipped with a storage medium on which software program codes that realize the functions of any of the above-described embodiments are stored may be provided, and a computer (or a CPU or MPU) of the system or the apparatus is caused to read out and execute the program codes stored in the storage medium.
In this case, the program code itself read from the storage medium can realize the functions of any of the above-described embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.
Examples of the storage medium for supplying the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (e.g., CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD + RW), a magnetic tape, a nonvolatile memory card, and a ROM. Alternatively, the program code may be downloaded from a server computer by a communications network.
Further, it should be clear that the functions of any one of the above-described embodiments may be implemented not only by executing the program code read out by the computer, but also by causing an operating system or the like operating on the computer to perform a part or all of the actual operations based on instructions of the program code.
Further, it is to be understood that the program code read out from the storage medium is written to a memory provided in an expansion board inserted into the computer or to a memory provided in an expansion unit connected to the computer, and then causes a CPU or the like mounted on the expansion board or the expansion unit to perform part or all of the actual operations based on instructions of the program code, thereby realizing the functions of any of the above-described embodiments.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. A method for scheduling data transmission system resources based on an declarative API is characterized in that the method specifically comprises the following steps:
custom API resource type CRD: respectively defining resource types CRD of acquisition end resources, warehousing end resources and message middleware resources of a data transmission system; wherein the resource type CRD group is dts;
scheduling the acquisition end resource, the warehousing end resource and the message middleware resource by using the resource description file corresponding to the resource type CRD, and capturing the applied resource file by a user-defined controller to realize corresponding resource scheduling work;
the abstraction of bottom physical resources and service logic resources is completed through the definition of resource files, and the scheduling of the service resources is realized through a user-defined controller.
2. The method for scheduling resources of a data transmission system based on an declarative API as recited in claim 1, wherein the data transmission system includes an acquisition component, a message middleware, and a storage component, the acquisition component reads data and writes the data into the message middleware, the message middleware caches the read data to ensure the integrity of the data, and the acquisition component and the message are intermediately deployed in one pod; the acquisition component and the message middleware respectively define a resource type CRD of type dtsReder, and the warehousing component defines a resource type CRD of type dtsWriter;
the acquisition assembly is used for reading data from a source database;
the message middleware is used for storing and transmitting data;
and the warehousing component is used for writing the acquired data into a target database.
3. The method according to claim 2, wherein the fields of the resource type CRD of dtsReader and dtsWriter are as follows:
taskId: unique identification of the data transmission task;
version: the version number of the product is consistent with the version numbers of the background database acquisition component and the warehousing component;
isAvailable: whether a resource is available;
dbInfo: data source information, connection information of a source end or a destination end database;
task: task configuration information, wherein the task configuration information comprises table information, field information and related transmission strategy parameters;
resources: and the resource information comprises cpu information, memory information and storage information of the pod needing to be pulled up.
4. The method for scheduling resources of a data transmission system based on an declarative API as claimed in claim 1, wherein the custom controller means that a corresponding CDR controller is created for a resource type CRD to implement triggering a service logic code in the CRD controller when a CRD object changes; the method comprises the following specific steps:
a data transmission system creates a custom controller dts-Operator;
a dts-Operator of the custom controller acquires and monitors the change of an object instance by using a ListAndWatch method of a Reflector;
when any one instance has any change, the Reflector receives the event notification and stores the combination of the event and the object corresponding to the object instance into a first-in first-out queue after receiving the notification;
the custom controller dts-Operator uses the Informer to continuously read the object from the first-in first-out queue, and judges whether the type of the event is dtsReader or dtsWriter:
and after the type of the event is determined, creating resources required by dtsReder or dtsWriter according to the type of the event.
5. The method for data transmission system resource scheduling based on the declarative API of claim 4, wherein the resources required to create dtsReder are specifically as follows:
a custom controller dts-Operator creates pod and service resources of a message middleware nat on which an acquisition component depends;
creating a jetstream and a consumer resource in a message middleware nats;
after the message middleware nats resource deployment is finished, deploying the pod, the service and the storage resource of the acquisition component;
by monitoring the corresponding events, the physical resources and the service configuration can be processed according to the self-defined logic;
when the processing is finished, the acquisition assembly required by the data transmission task can work normally;
the resources required to create dtsWriter are specifically as follows:
a custom controller dts-Operator creates pod and service physical resources required by a warehousing component and generates a configuration file related to a data transmission task;
writing information related to the message middleware nat into a configuration file;
by monitoring the corresponding events, the physical resources and the service configuration can be processed according to the self-defined logic;
and after the processing is finished, the warehousing component required by the data transmission task can work normally.
6. The method for scheduling resources of a data transmission system based on an declarative API as in claim 4 or 5, wherein a custom controller dts-Operator obtains a dtsWriter object and a dtsReder object from an APIServer of Kubernetets by means of a code block of an inform, the inform corresponds to the API objects one by one, and the inform of a Network object is transmitted to the custom controller dts-Operator in the data transmission process;
when the Informer factory is established, a Network client is transmitted to the Informer factory, the Network Informer uses the Network client to establish connection with an APIServer, a ListAndWatch mechanism in a Reflector package used by the Informer is used for maintaining the connection between the Network Informer and the APIServer, and the ListAndWatch mechanism is used for acquiring and monitoring the change of a Network object instance;
under a ListAndWatch mechanism, once a new Network instance is created, deleted or updated at an APIServer end, a Reflector receives an event notification corresponding to the Network instance, and an API object combination corresponding to the event is called increment Delta and is put into an increment first-in first-out queue; meanwhile, the Inform continuously reads the increment from the increment first-in first-out queue, and every time when the increment is read, the Inform judges the event type in the increment and creates or updates the buffer of the local object.
7. The method for declarative API-based resource scheduling of a data transmission system as recited in claim 6, wherein the inform mer has the following functions:
(1) synchronizing the local cache; the Informer is provided with a cache and an index mechanism; triggering a client library of a Handler, wherein a local cache is called Store in Kubernets, and an Index is called Index; the Informer uses a Reflector package, and the Reflector package acquires and monitors client encapsulation of API object change through a ListAndWatch mechanism; the Reflector packet and the Informer are coordinated by an increment first-in first-out queue;
(2) triggering and realizing the registered ResourceEventHandler according to the event type; the inform updates the local cache in real time through monitored event changes, and invokes EventHandler corresponding to the event, and the local cache maintained by the inform is forcibly updated once by using the result returned by the latest LIST every time the resynPeriod is appointed, so that the effectiveness of the cache is ensured; when a custom controller dts-Operator is created, the custom controller dts-Operator is registered to an Informer corresponding to a ResourceEventHandler;
before compiling a control cycle, an actor uses a work queue to carry out cooperation; wherein, the logic of the inner former control loop is as follows:
waiting for the inform to finish the data synchronization operation of the local cache;
starting one or more endless loop tasks by the goroutine;
in each cycle, executing the data transmission service logic codes;
if the control loop does not acquire the information of the corresponding object in the Informer cache, the control loop indicates that the deleting logic is to be executed;
if the object information is obtained from the cache, the execution controller mode compares the expected state and the actual state, wherein the expected state comes from the etcd actual state and comes from the cluster.
8. The method for declarative API-based data transmission system resource scheduling of claim 1, wherein custom API resource type CRD further includes the following:
carrying out Initializer operation and Validation operation of the API object: validation operation verifies the validity of each field in the object, the validity is stored in a Registry data structure after verification, and if the definition of an API object can be found in the Registry, the API is correspondingly an effective k8sAPI object;
and converting the superversion object into the object serialization of the user submitted version, and storing the object serialization into the etcd.
9. The method for scheduling resources of a data transmission system based on an declarative API of claim 1, wherein the data transmission system defines a management-side Service dts-Service, the management-side Service dts-Service is used for receiving a user instruction sent by a front end, and the management-side Service dts-Service interacts with a back-end custom controller dts-Operator in an declarative API manner to implement creation of data transmission task resources and operation logic control of tasks;
the creation of the resource is specifically as follows:
a user creates a task on a front-end interface, and the front-end interface calls a management end Service dts-Service to create a data transmission task method through an http request;
the management end Service dts-Service sequentially applies the data transmission task dtsReader and the resource type CRD of the dtsWriter by accessing the API of Kubernets, and the name field in the resource type CRD is consistent with the task number and is used for distinguishing the resource types CRD of different tasks;
the resource type CRD of the application can be monitored by a custom controller dts-Operator and corresponding resource creating logic is executed;
after the resources are created, the acquisition component, the warehousing component and the message middleware nats of the database automatically complete initialization work;
after detecting that the resources of the acquisition component and the warehousing component of the task are successfully distributed and normally initialized, the management end Service dts-Service returns a message of successful creation to the user;
the operation logic control of the service is specifically as follows:
the user starts, stops, pauses, continues to transmit, changes the password and deletes the data transmission task through the front-end interface; the front-end interface calls a corresponding method in the management end Service dts-Service through an http request;
the management end Service dts-Service respectively assembles resource types CRD of dtsReder and dtsWriter corresponding to the data transmission task, and the name field of the resource type CRD is consistent with the task number and is used for distinguishing the resource types CRD of different tasks;
the operation field in the assembled resource type CRD corresponds to the operation of the user;
the management end Service dts-Service respectively applies the task dtsReader and the resource type CRD of the dtsWriter by accessing the API of Kubernets;
after monitoring the change of the CRD version of the resource type, the management-side service dts-Operator analyzes the content of the operation field to judge the operation of the user, and calls grpc interfaces of corresponding operations of the acquisition component and the library entry component respectively according to the operation type, so that the operation of the user is completed.
10. A computer-readable storage medium, in which a computer program is stored which is executable by a processor for implementing the method for declarative API-based scheduling of data transmission system resources as claimed in any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210792245.7A CN115202874A (en) | 2022-07-07 | 2022-07-07 | Method and storage medium for scheduling data transmission system resources based on declarative API |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210792245.7A CN115202874A (en) | 2022-07-07 | 2022-07-07 | Method and storage medium for scheduling data transmission system resources based on declarative API |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115202874A true CN115202874A (en) | 2022-10-18 |
Family
ID=83580468
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210792245.7A Pending CN115202874A (en) | 2022-07-07 | 2022-07-07 | Method and storage medium for scheduling data transmission system resources based on declarative API |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115202874A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115357369A (en) * | 2022-10-21 | 2022-11-18 | 之江实验室 | CRD application integration calling method and device in k8s container cloud platform |
| CN115915457A (en) * | 2023-01-30 | 2023-04-04 | 阿里巴巴（中国）有限公司 | Resource scheduling method, vehicle control method, device and system |
| CN116974703A (en) * | 2023-09-22 | 2023-10-31 | 山东未来网络研究院(紫金山实验室工业互联网创新应用基地) | Kubernetes application resource management method and system |
-
2022
- 2022-07-07 CN CN202210792245.7A patent/CN115202874A/en active Pending
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115357369A (en) * | 2022-10-21 | 2022-11-18 | 之江实验室 | CRD application integration calling method and device in k8s container cloud platform |
| CN115357369B (en) * | 2022-10-21 | 2023-03-24 | 之江实验室 | CRD application integration calling method and device in k8s container cloud platform |
| CN115915457A (en) * | 2023-01-30 | 2023-04-04 | 阿里巴巴（中国）有限公司 | Resource scheduling method, vehicle control method, device and system |
| CN116974703A (en) * | 2023-09-22 | 2023-10-31 | 山东未来网络研究院(紫金山实验室工业互联网创新应用基地) | Kubernetes application resource management method and system |
| CN116974703B (en) * | 2023-09-22 | 2024-01-02 | 山东未来网络研究院(紫金山实验室工业互联网创新应用基地) | Kubernetes application resource management method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10171383B2 (en) | Methods and systems for portably deploying applications on one or more cloud systems | |
| CN115202874A (en) | Method and storage medium for scheduling data transmission system resources based on declarative API | |
| US9851989B2 (en) | Methods and apparatus to manage virtual machines | |
| US9047160B2 (en) | Designing and building virtual images using semantically rich composable software image bundles | |
| US7519711B2 (en) | Method for middleware assisted system integration in a federated environment | |
| US8607200B2 (en) | Executing a web application at different stages in the application life cycle | |
| US11099909B2 (en) | Methods and apparatus for adaptive workflow adjustment during resource provisioning using meta-topics | |
| US9003389B2 (en) | Generating an encoded package profile based on executing host processes | |
| US20180167275A1 (en) | Methods, systems and apparatus to propagate node configuration changes to services in a distributed environment | |
| CN113569987A (en) | Model training method and device | |
| CN110109649A (en) | For container control method, device and the containment system of Web service | |
| US20190349428A1 (en) | Systems and methods for update propagation between nodes in a distributed system | |
| US20220091830A1 (en) | System and method for a hybrid development platform | |
| US11550615B2 (en) | Kubernetes resource policy enforcement | |
| US20200394049A1 (en) | Support for third-party kernel modules on host operating systems | |
| CN116028163A (en) | Method, device and storage medium for scheduling dynamic link library of container group | |
| CN110494849B (en) | System and method for determining success of cross-platform application migration | |
| US20220156097A1 (en) | Systems and methods to pre-provision sockets for serverless functions | |
| CN111522623B (en) | Modularized software multi-process running system | |
| CN114787836A (en) | System and method for remotely executing one or more arbitrarily defined workflows | |
| CN111147291A (en) | Service maintenance method and device | |
| US20180276079A1 (en) | System and method for determining the success of a cross-platform application migration | |
| CN114860203A (en) | Project creation method, project creation device, server and storage medium | |
| US20240012758A1 (en) | Hash optimized composition cache for isolated execution environments | |
| woon Ahn et al. | Mirra: Rule-based resource management for heterogeneous real-time applications running in cloud computing infrastructures |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |