CN110572436A - Multi-place cross-cluster server deployment method and system - Google Patents
Multi-place cross-cluster server deployment method and system Download PDFInfo
- Publication number
- CN110572436A CN110572436A CN201910739457.7A CN201910739457A CN110572436A CN 110572436 A CN110572436 A CN 110572436A CN 201910739457 A CN201910739457 A CN 201910739457A CN 110572436 A CN110572436 A CN 110572436A
- Authority
- CN
- China
- Prior art keywords
- cluster
- deployment
- project
- server
- deployed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
- H04L67/025—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP] for remote control or remote monitoring of applications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/104—Peer-to-peer [P2P] networks
- H04L67/1044—Group management mechanisms
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/104—Peer-to-peer [P2P] networks
- H04L67/1044—Group management mechanisms
- H04L67/1046—Joining mechanisms
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/104—Peer-to-peer [P2P] networks
- H04L67/1044—Group management mechanisms
- H04L67/1053—Group management mechanisms with pre-configuration of logical or physical connections with a determined number of other peers
- H04L67/1055—Group management mechanisms with pre-configuration of logical or physical connections with a determined number of other peers involving connection limits
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
Abstract
the invention relates to a multi-place cross-cluster server deployment method and a system, wherein the method comprises the following steps: performing unified configuration at a cloud end, and generating a plurality of configuration files containing configuration information according to project information to be issued and server information to be deployed, wherein the configuration files correspond to the servers to be deployed one by one; receiving a deployment instruction, acquiring a project code according to the issued project information after receiving the deployment instruction, and compiling and packaging the project code; and selecting a target server according to the configuration file, and synchronizing the compiled code to the target server. The invention can realize multi-place cross-network deployment, can compile and distribute for multiple times for multi-provincial environments at one time, improves the deployment efficiency, and ensures the deployment code version to be uniform through the system.
Description
Technical Field
The invention relates to the technical field of server cluster deployment, in particular to a server deployment method and a server deployment system supporting interfacial deployment and performing cross-region and cross-cluster on a plurality of server clusters based on a public cloud network.
background
At present, based on multiple factors such as disaster recovery in different places, improvement of user access speed in different areas, service deployment cost and the like, server clusters are built in multiple places such as multiple provinces, each province is equivalent to an independent cluster environment, internal services need to be accessed internally, and the server clusters in all the places are located in the same public cloud network so as to be convenient for management. At present, some scripts are usually deployed manually, so that the following problems occur: because deployment of the server needs to have a certain operating system foundation, a plurality of front-end personnel do not have the skill, and if the operation and maintenance personnel operate in a unified manner, on one hand, communication cost is increased, so that the operation and maintenance personnel become a resource bottleneck, and on the other hand, the instability of the system caused by multi-user operation can also be caused. At present, a set of environment has hundreds of micro services including java, C + +, Golang and front-end resources, the workload of updating and restarting every day is large, and complicated tasks are added for operation, maintenance and development projects.
for the above problems, there is also a cluster server deployment method in the prior art that can implement efficient and automatic deployment of cluster servers, but the methods in the prior art are all used for deploying the same server cluster, and do not solve the problem of efficient and automatic deployment of cross-regional and cross-cluster server clusters. For deployment across regions and clusters, compared with server deployment of a single cluster, in addition to differences in configuration information settings included in configuration files, problems such as compatibility of the method with different server clusters need to be considered.
Disclosure of Invention
The object of the present invention is to solve at least one of the above problems.
therefore, one objective of the present invention is to provide a server deployment method that can efficiently and automatically deploy servers across multiple clusters.
In order to achieve the above object, according to the method for deploying a multi-place cross-cluster server in an embodiment of the first aspect of the present invention, a deployment service is executed in a central computer room, and the method includes the following steps:
performing unified configuration at a cloud end, and generating a plurality of configuration files containing configuration information according to project information to be issued and server information to be deployed, wherein the configuration files correspond to the servers to be deployed one by one;
receiving a deployment instruction, acquiring a project code according to the issued project information after receiving the deployment instruction, and compiling and packaging the project code;
Selecting a target server according to the configuration file, synchronizing the compiled code to the target server,
wherein:
The method comprises the steps that a remote management protocol supporting file transmission and remote command execution is pre-configured for each server cluster in the system, and the remote management protocol used by the cluster where a selected target server is located is determined before code synchronization;
For a target server supporting the ssh protocol, code synchronization is carried out based on the ssh protocol;
for target servers supporting the private protocol, code synchronization is performed based on the long connection mode.
optionally, the method further comprises the steps of:
judging whether a new server cluster is deployed or not, and if so, providing an interface for initiating a one-key offline deployment request to a user;
The new cluster needs to configure a machine list in advance, and each machine deploys needed basic software and application services;
after receiving a one-key offline deployment request, selecting micro-services to be installed according to a function range configured in advance by the new cluster, uniformly pulling, collecting and packaging the selected micro-services by the deployment service, extracting an execution script from each micro-service, and uniformly generating an executable script, thereby generating a large number of offline installation packages and execution scripts of the micro-services for operation and maintenance personnel who are installed on the offline. When a cluster in a new region (for example, a new province) needs to deploy a whole set of service, the whole network is generally in a closed state to the outside at the beginning, an external network cannot enter, and the inside cannot come out, so that the central deployment service cannot be directly connected with the cluster, and at the moment, operation and maintenance personnel need to perform door-to-door installation and deployment, so that a key offline deployment function is provided for facilitating the deployment of hundreds of micro-services, and offline installation packages and execution scripts of hundreds of micro-services can be generated for the operation and maintenance personnel who perform the door-to-door installation offline.
Optionally, the to-be-published item information may be determined through multiple publishing state comparisons, specifically:
When the function of another cluster needs to be updated according to the cluster which is already deployed, the deployment states of part of the services of the two clusters are compared so as to judge which services are dependent on the publication of the functional module which needs to be updated, and therefore the services are selected as the information of the items to be published. The distribution difference between the clusters of the two regions, information related to the generated configuration file, such as the final distribution system of each service, a distributor, functional point descriptions distributed at that time, and the like, can be known through comparison of multiple distribution conditions.
Optionally, the method further comprises the steps of:
and determining whether the current release relates to different clusters in multiple places according to the configuration file, and if the current release relates to different clusters in multiple places, sequentially synchronizing the clusters according to a preset release sequence.
optionally, the method further comprises the steps of:
and in the process of synchronizing the target server, switching the external traffic of the target server to other servers of the cluster to which the external traffic belongs, and ensuring that other requests are not accessed in the period.
optionally, long connection is realized by adopting a user-defined ace mode, and the ace mode adopts the following mode to connect a cloud center machine room, a server cluster and specific machines in the server cluster:
Executing a PubServer service in a central machine room of a cloud, wherein the PubServer service is used for heel deployment service communication on one hand, and sending a corresponding instruction to a corresponding cluster agent machine after scheduling the instruction of the deployment service on the other hand;
the method comprises the steps that an agent service ClientAgent is deployed for each cluster supporting a private protocol, the ClientAgents are automatically connected to a PubServer service of a center after being started, and are mainly responsible for communication with the PubServer and transmitting instructions to specific machines under the cluster;
The method comprises the steps that a proxy service machine agent is installed on each specific machine in a cluster with a ClientAgent service, the machine agent is automatically connected to the ClientAgents of the current cluster after being started, so that each machine of each cluster is connected through a PubServer, and the machine agent is responsible for receiving and finally executing instructions from the ClientAgents.
Optionally, the cloud center machine room provides an interfacing operation window.
Optionally, for a user requirement for deploying a project deployed on one cluster to another cluster, at least a source code of the project to be published is directly acquired from the deployed cluster.
optionally, the information of the items to be published includes publishing type information, and for pre-specified items of a specific publishing type, the number of corresponding configuration information items is greater. For example, the following configurations exist for Java, Golang deployments: health check url, compiling script, initializing script and synchronizing subsequent script of file.
Optionally, the method further comprises the steps of:
If one step of performing the synchronization of the compiled code to the target server is in error, a quick roll-back to the previous step to redeploy occurs.
optionally, the method further comprises the steps of:
and configuring the authority of an operator, and if the current operator does not have the authority of direct release for the project to be deployed, the deployment of the project needs the corresponding project responsible person to examine and approve, and the code synchronization is performed only after the examination and approval by the examining and approving person.
An embodiment of the second aspect of the present invention provides a terminal, comprising a processor and a memory, the memory storing a computer program supporting the terminal to execute the above method, the computer program comprising program instructions, and the processor being configured to call the program instructions to execute the method of the first aspect.
Embodiments of the third aspect of the invention provide a computer readable storage medium having stored thereon a computer program comprising program instructions which, when executed by a processor, cause the processor to perform the method of the first aspect described above.
an embodiment of a fourth aspect of the invention provides a computer program comprising program instructions which, when executed by a processor, cause the processor to perform the method of the first aspect described above.
The invention also aims to provide a multi-place cross-cluster server deployment system which can realize the method.
To achieve the above object, a multi-place cross-cluster server deployment system according to an embodiment of a fifth aspect of the present invention includes:
the configuration unit is arranged on the central server and used for generating a plurality of configuration files containing configuration information according to the project information to be issued and the server information to be deployed, and the configuration files correspond to the servers to be deployed one by one;
the to-be-issued project code acquisition unit is used for receiving the deployment instruction, acquiring the project code according to the issued project information after receiving the deployment instruction, and compiling and packaging the project code;
And the release unit is used for selecting a target server according to the configuration file, synchronizing the compiled code to the target server, firstly determining a remote management protocol used by a cluster where the selected target server is located before code synchronization, and selecting a corresponding mode according to different protocols for synchronization.
optionally, the multi-place and cross-cluster server deployment system includes a cross-cluster issuing order management unit, configured to determine whether the present issuing involves different clusters in multiple places according to the configuration file, and if the present issuing involves different clusters in multiple places, determine, according to a preset deployment policy, that the issuing orders of the clusters are sequentially synchronized.
Optionally, the system includes a smooth deployment unit, configured to switch external traffic of the target server to another server of the cluster to which the external traffic belongs in a process of synchronizing the target server, and ensure that no other request is accessed during the period.
optionally, the cloud center system comprises an ace mode implementation unit, configured to execute a PubServer service in a center machine room, deploy a proxy service ClientAgent in each cluster supporting a private protocol, install a local service machine in each specific machine in the cluster in which the ClientAgent service is deployed, and implement connection between the cloud center machine room, the server cluster, and the specific machine in the server cluster through the PubServer service, the ClientAgent service, and the machine agent service.
Optionally, a user interaction unit is included for providing an interfacing operation window.
Optionally, an item rollback unit is included for quickly rolling back to a previous step for redeployment after a step of performing the synchronization process is in error.
Optionally, the system includes a pre-release verification unit, configured to deploy to a pre-release environment for full verification before the service is on-line according to a requirement initiated by a user, where the pre-release environment is consistent with the on-line environment but only visible to internal testers.
The embodiment of the invention realizes multi-place cross-network deployment by uniformly configuring and managing the central machine room and uniformly compiling, packaging and distributing, can compile once and distribute for multiple times for multi-provincial environments, improves the deployment efficiency, and ensures the deployment code version to be uniform through the system. And different connection modes are adopted aiming at network restrictions possibly encountered by cross-cluster deployment, so that not only is a common ssh protocol supported, but also the cluster with special network restrictions can be deployed in a distributed manner, and the barrier of the network restrictions is opened.
drawings
FIG. 1 is a system architecture based on which the multi-place cross-cluster server deployment method of the present invention is implemented.
fig. 2 is a specific example of the interfacing operation window after information input is completed.
FIG. 3 is a configuration example of configuring a cross-cluster deployment.
Fig. 4 is a communication example of the Ace mode according to the embodiment of the present invention.
FIG. 5 is a journal interface of an embodiment of the present invention.
FIG. 6 is a diagram illustrating a user's history of operation.
FIG. 7 is a deployment example interface.
Detailed Description
The embodiments of the present invention will be described in detail below with reference to the accompanying drawings, and it should be noted that the embodiments described below by referring to the drawings are exemplary only for the purpose of explaining the present invention, and are not to be construed as limiting the present invention.
first, technical terms related to this section will be explained:
SSH, which is a network protocol for encrypted login between computers, is currently a standard configuration for linux operating systems.
rsync and linux synchronization tools, which can mirror the whole directory tree and files, perform corresponding operations according to parameters, and transmit functions of compression, archiving mode, subdirectory recursion, rule file omission and the like
git, an open source distributed version control system, can handle efficiently and at high speed the management of project versions from very small to very large.
shell, a script file under the operating system, is relative to the kernel because it is a user-oriented representation based on the kernel.
ace, an RPC framework developed internally for transmission between different languages.
Vue, a progressive front end framework for building data driven web interfaces. The goal is to implement the data binding and combined view components of the response through as simple an API as possible. It is not only easy to get a hand, but also convenient to integrate with third party libraries or existing projects.
Maven, a Project Management tool, contains a Project Object Model (Project Object Model), a set of criteria, a Project Lifecycle (Project Lifecycle), a Dependency Management System (Dependency Management System), and logic for running plug-in objects (loads) defined in the Lifecycle phase
first, a system architecture based on which the multi-place cross-cluster server deployment method of the present invention is implemented is described with reference to fig. 1, where the system architecture includes a central machine room and a plurality of server clusters, each server cluster is distributed in different regions, for example, the server clusters in this embodiment are respectively deployed in different provinces, and the central machine room performs unified management on the multi-place server clusters.
the method for deploying the multi-place cross cluster server comprises the following steps.
Step S1, performing unified configuration in the central machine room of the cloud, and generating a plurality of configuration files containing configuration information according to the information of the project to be issued and the information of the servers to be deployed, which are acquired from the front end, wherein the configuration files correspond to the servers to be deployed one by one.
Firstly, the information of a project to be published and the information of a server to be deployed (namely, a target server) are obtained from a front end. In a specific embodiment, the central computer room provides an interface operation window for interacting with a user, and an operator inputs specified information at a corresponding position of the interface operation window. And acquiring the information of the release project and the information of the server to be deployed according to the information in the interface operation window. The input mode can be a common mode such as manual input, drop-down list input and the like. After the information input is complete, the system runs a deployment service in response to the operator's request. After the deployment service is executed, a plurality of configuration files containing configuration information are generated according to the project information to be issued and the server information to be deployed, and the configuration files correspond to the servers to be deployed one by one.
Fig. 2 is a specific example of the interface operation window after information input is completed, which may include items related to item summaries: such as environment, application name, project description, git address, release type, whether auditing is needed, whether unified configuration is needed, project principal, test principal, etc.; and project resource-related: such as publisher source code directory, file directory requiring synchronization, target machine directory, target machine login user, pre-published machine address, online machine address, publisher compile log path, application log path, etc.
For a practical scenario that projects need to be deployed across clusters (e.g., across provinces), it is often necessary to rapidly deploy all projects of one province to another province, and for this scenario, entry using a single project case is troublesome, so that an operator may select a source site (province or cluster), a project type, and a project that needs to be migrated as shown in fig. 3, and then fill in original values and target values of replacement variables (such as machine ip needs to be changed, file path, and the like), and add a plurality of projects at will, so as to implement one-time migration of a plurality of projects.
And the number of corresponding configuration information items may also differ according to different release types. For example, for a release type of Java or Golang resource, the release of which has specific requirements, such as the need for health check, the corresponding configuration information includes health check related information.
For convenience of management, the issue of the project generally needs to consider the operation authority of the operator, and therefore, the configuration information includes authority information of the operator, and whether the current operating user has the authority to deploy the project is determined through the authority information. In this embodiment, the setting information related to the operation authority includes: system user roles, user identities, project user roles, and assignment of permissions for individual projects that may be configured, published, approved, etc. The system user roles include: common users, administrators, system administrators; the user identities are: the system comprises a server, a front end, a test, a product and an operation; project user roles include: a general user, a project administrator. An operator who performs project deployment needs to operate after a user logs in, and after the deployment service is started, the information needs to be checked to determine whether the current login user has corresponding operation authority.
in an embodiment, the release sheet is one of configuration files, and the configuration information includes information related to a source of a project to be released, a site of a server to be deployed, an ip of the server to be deployed, and the like. The site of the server to be deployed refers to site information of a cluster where the server to be deployed is located, and generally includes a cluster number, the area and the like.
After configuration is completed, initialization items and some basic configuration checks, such as checking whether the configuration items are correct or not and checking whether the ip path is normal or not, need to be performed. These are the basic operations of project deployment and are not expanded in detail here.
Step S2, the project code is acquired, compiled and packaged, and is ready to be distributed to the target server.
After the step S1 is completed, the address where the source code of the item to be released is located can be already determined according to the configuration information included in the configuration file, and the item code is obtained from the address. For a scenario in which projects need to be deployed across clusters (e.g., across provinces), it is often necessary to rapidly deploy all projects of one province to another province, and for convenience, migration of multiple projects may be performed at one time, in this case, the to-be-published project information obtained from the front end in step S1 includes a storage location of a source code of the to-be-published project in a specific cluster, and the source code of the to-be-published project may be directly obtained from the storage location. For example, in one embodiment, the issue list described in step S1 includes git branch information, and after the deployment service is executed, a number of shell scripts are run on the deployment machine in the central computer room, where the shell scripts include git.sh scripts, and git.sh can obtain corresponding codes under the git branch according to the git branch in the issue list. The description of fig. 3 above is combined, so that the items of one cluster can be quickly copied to another cluster, and the configuration items that need to be changed, such as ip, directory path, etc., are modified by one key.
Of course, in the embodiment, the source code of the project to be issued may also be directly uploaded to the central computer room by the operator. And compiling and packaging after acquiring the project codes, and preparing to issue to a target server.
And step S3, selecting a target server according to the configuration file, and synchronizing the compiled project code to the selected target server.
The configuration file comprises information of the server to be deployed, and relevant information of the corresponding position is directly extracted from the corresponding configuration file. In some embodiments, before performing the synchronization, it is also necessary to verify the authority information of the current operator to confirm whether the current operator has the operation authority to deploy the project. For example, operators, i.e., deployment engineers, have different operation permissions on different projects, and when a specific project is deployed, project code synchronization operation can be performed after approval by a responsible person of the project; and if the current operator has the direct deployment authority on the selected project, directly performing subsequent project code synchronization operation. In this embodiment, a project that needs to be approved is subjected to pre-release verification processing, where the pre-release verification refers to issuing the project to be deployed to a pre-release environment for commissioning, and the pre-release environment is the last verification environment before online and has an environment that is substantially the same as that of the online system, but is invisible to external users and only visible to internal testers. After the pre-sending verification is passed, the examination and approval task flow of the current project to be issued is transferred to the examination and approval personnel with corresponding authority, and the corresponding examination and approval personnel are informed, so that the current project can be reminded in a common message reminding mode such as mail.
The compiled project code then needs to be synchronized to the selected target server. Due to the uniform deployment by the central office, the problem of remote communication between the server cluster and the central office is involved. Each server cluster in the system is preconfigured with a remote management protocol for supporting file transmission and remote command execution, and the cluster in the embodiment is divided into two communication modes: SSH protocol supported and SSH protocol not supported; the remote management protocol used by the cluster in which the selected target server resides is first determined prior to code synchronization.
SSH is a security protocol built on an application layer basis, which is currently a more reliable protocol that translates into telnet sessions and other web services to provide security. In the method, the ssh mode is provided for the server cluster supporting the ssh protocol for synchronization. The SSH mode precondition is that the network between the trans-provincial cluster and the deployment machine needs to be opened, and the cluster system must support the SSH protocol.
For a server cluster which does not support ssh protocol, an ACE long connection mode is adopted to complete the work of file transmission, remote command execution and the like. After each new cluster is started, whether the access mode is used or not is configured in the center. The allocation of the walk mode is not different from the ssh mode in user operation, but the communication mode of the bottom layer is different. Specifically, as shown in fig. 4:
in the Ace mode, a PubServer service is started in the central computer room, and is used for deploying service communication at the heel on one hand, and sending a corresponding instruction to a corresponding cluster agent after scheduling the instruction for deploying the service on the other hand.
Each cluster is provided with a ClientAgent service, the ClientAgents are automatically connected to a PubServer service of the center after being started, and the ClientAgents are mainly responsible for communication with the PubServer and transmitting instructions to specific machines under the cluster.
Each machine is provided with a machine agent and is automatically connected to the ClientAgents of the current cluster after being started, so that each machine of each cluster is connected through a PubServer. The MachineAgent is responsible for receiving an instruction which is used as a Clientagent and finally executing the instruction.
The pubServer determines which cluster the instruction is sent to by the site ID (cluster ID representing a province), and the ClientAgent determines which machine the instruction is sent to by ip. The PubServer and the ClientAgents not only serve as transmission channels but also serve as routing functions.
The machine agent is a local service installed in each machine, is responsible for receiving instructions, executing commands and returning execution results to the ClientAgents, and mainly comprises operations of command shell commands and file adding, deleting, modifying and checking commands. The multimedia agent only communicates with the ClientAgents, and all information interacts with the cloud through the ClientAgents.
One ClientAgents is installed in one cluster and used as a communication bridge between an internal machine and a cloud system, and data are transferred through an ACE protocol.
The PubServer is deployed at the cloud, communicates with the ClientAgents of each province only externally, receives the instructions of the deployment system, and sends specific instructions to the final machine agent for execution according to the cluster ID and the internal ip as the distribution condition.
In the embodiment, the three modules, namely the machine agent, the ClientAgent and the PubServer, are developed by C + +, the execution instruction directly calls an operating system bottom library, and the internal communication goes away from the ACE protocol.
In step S3, the connection is managed using Ngnix, and it is determined that ssh mode or ace mode needs to be adopted currently based on Ngnix. The ssh mode will be taken first for clusters that are to get through ssh. The invention not only supports the common ssh protocol, but also can perform distributed deployment on the cluster with special network restriction, thereby breaking through the barrier of the network restriction.
And after the connection mode needing to be selected is determined, synchronizing the compiled file to the target server. For example, in an embodiment of executing the deployment method based on the linux system, synchronizing the deployment machine file to the target machine is realized through an own rsync command. Based on the steps, the invention can realize multi-place cross-network deployment without deploying service independently in each cluster, uniformly configuring and managing by a central machine room, and uniformly compiling, packaging and distributing.
In an actual deployment scenario, for the deployment of servers that are already online (for example, service upgrade), since the service restart needs time, some external requests during the restart process may have an impact, and therefore, the deployment method of the multi-span cluster server further includes a step of smooth deployment. The premise of the smooth deployment is as follows: the cluster in which the target server is located is built by adopting a distributed architecture. Then, in the process of synchronizing the target server, the external traffic of the target server may be switched to other servers of the cluster to which the target server belongs, and it is ensured that no other requests are accessed during the period. The specific implementation manner in the embodiment may be that, for the external Http service, restarting generally has a time period influence, and the system makes a call to the flow control policy of the service end and nginx, so that it can be ensured that all external services do not access the service being restarted, and the restarting in the release process has zero influence on the line.
It will be appreciated by those skilled in the art that, in the above steps, before the synchronous file (write) is executed, some conventional commands are generally executed, for example, the commands are used to perform ending work before deployment; after the synchronization file is executed, it is also generally necessary to execute some conventional commands, for example, by which the start script of the service is executed.
The method further comprises the following steps: if one step of performing the synchronization of the compiled code to the target server is in error, a quick roll-back to the previous step to redeploy occurs. Based on this, after the target server deployment is completed, the original package of the service deployed this time is backed up for use in rollback. The rollback is the rollback of all codes, and in the embodiment, the rollback may be performed through the git source of the codes or through a certain release order. Wherein, the rolling back of the Git source is equivalent to repackaging and deploying a code of rolling back the service to a certain time stamp as a reference; the reference to a distribution list is equivalent to a reference to a packet for rolling back a service to a previous distribution list.
For the smooth processing, health check is needed after the synchronous file is completed, whether the service is started successfully is judged, and flow re-cut is performed after the health check is passed. If rollback is required, a rollback action is taken, which may also be performed at any time after deployment is complete.
In an actual scenario of cross-province arrangement, there are usually many deployments for the same project, respectively, and the deployed codes are the same, except for configuration information. Based on the configuration, each cluster is independently used as a configuration center, and the deployment process is packaged, compiled and distributed for multiple times. As the deployment example shown in fig. 7, after one-time packing and compiling is completed, a cluster in, for example, zhejiang, hunan, sichuan, etc. may be selected for distribution and deployment. The method also comprises the steps of determining whether the current release relates to different clusters in multiple places according to the configuration file, and if the current release relates to different clusters in multiple places, sequentially synchronizing the clusters according to a preset release sequence. In this embodiment, the release order is manually determined by an operation and maintenance person or a developer, or may be determined according to a certain deployment policy, for example, according to busy and idle states of different servers. The service deployment is executed according to the steps S1-S3, and after one cluster deployment is completed, it is determined whether there are any other clusters to be deployed until all the clusters to be deployed are completely deployed. Therefore, the deployment efficiency is improved, and the deployment code version is ensured to be uniform.
the method further comprises the step of generating corresponding logs for compiling and starting the deployment service in the release process and for historical operation behaviors of a certain release list so as to facilitate positioning problems by operators. FIG. 5 is a log interface according to an embodiment, which includes three tab pages, one of which is used to show the execution log of the current issue ticket, and to inform the operator of the operation result of each step; the other label page is a log for displaying the compiler when the server is issued and is used for checking whether an exception occurs in the compiling process; the last tab page is the boot log for a certain target machine. And as shown in fig. 6, each operation of the operator can be traced back afterwards, and the execution result at that time can be checked.
It should be noted that, those skilled in the art implement the above method for multiple locations to be deployed across cluster servers through a computer program, and therefore, in order to implement the above method, it is necessary to provide a storage medium on which the computer program is stored, and the program is executed by a processor to implement the method for multiple locations to be deployed across cluster servers as described above. Accordingly, there is also a need to provide an apparatus comprising a processor and a storage medium having stored thereon a computer program for execution by the processor to implement the method for multi-place deployment across cluster servers as described above.
Claims (12)
1. a method for multi-place cross-cluster server deployment, which is characterized in that a deployment service is operated in a central machine room in response to a user request, and comprises the following steps:
Performing unified configuration at a cloud end, and generating a plurality of configuration files containing configuration information according to the information of the project to be issued and the information of the server to be deployed, wherein the configuration files correspond to the servers to be deployed one by one;
Acquiring a project code, compiling and packaging the project code, and preparing to issue the project code to a target server;
selecting a target server according to the configuration file, synchronizing the compiled code to the selected target server,
Wherein:
the method comprises the steps that a remote management protocol for supporting file transmission and remote command execution is pre-configured for each server cluster in the system, and the remote management protocol used by the cluster where a selected target server is located is firstly determined before code synchronization;
For a target server supporting the ssh protocol, performing code synchronization based on the ssh mode;
And for the target server which does not support the ssh protocol, carrying out code synchronization based on the self-defined ace long connection mode.
2. The method for deploying servers across clusters in multiple places according to claim 1, further comprising the steps of:
Judging whether a new server cluster is deployed or not, and if so, providing an interface for initiating a one-key offline deployment request to a user;
The new cluster needs to configure a machine list in advance, and each machine deploys needed basic software and application services;
after receiving a one-key offline deployment request, selecting micro-services to be installed according to a function range configured in advance by the new cluster, uniformly pulling, collecting and packaging the selected micro-services by the deployment service, extracting an execution script from each micro-service, and uniformly generating an executable script, thereby generating a large number of offline installation packages and execution scripts of the micro-services for operation and maintenance personnel who are installed on the offline.
3. The multi-place cross-cluster server deployment method according to claim 1, wherein the to-be-published item information can be determined by multi-place publication state comparison, specifically:
When the function of another cluster needs to be updated according to the cluster which is already deployed, the deployment states of part of the services of the two clusters are compared so as to judge which services are dependent on the publication of the functional module which needs to be updated, and therefore the services are selected as the information of the items to be published.
4. The method of claim 1, further comprising the steps of:
And determining whether the current release relates to different clusters in multiple places according to the configuration file, and if the current release relates to different clusters in multiple places, sequentially synchronizing the clusters according to a preset release sequence.
5. The method of claim 1, further comprising the steps of:
And in the process of synchronizing the target server, switching the external traffic of the target server to other servers of the cluster to which the external traffic belongs, and ensuring that other requests are not accessed in the period.
6. The method of claim 1, wherein the customized ace long connection mode is used for connecting a cloud center machine room, a server cluster and specific machines in the server cluster in the following manner:
Executing a PubServer service in a central machine room of a cloud, wherein the PubServer service is used for heel deployment service communication on one hand, and sending a corresponding instruction to a corresponding cluster agent machine after scheduling the instruction of the deployment service on the other hand;
The method comprises the steps that an agent service ClientAgent is deployed for each cluster supporting a private protocol, the ClientAgents are automatically connected to a PubServer service of a center after being started, and are mainly responsible for communication with the PubServer and transmitting instructions to specific machines under the cluster;
each specific machine in the cluster deployed with the ClientAgents is provided with a local service Machine agent, the Machine agents are automatically connected to the ClientAgents of the current cluster after being started, so that each machine of each cluster is connected through a PubServer, and the Machine agents are responsible for receiving and finally executing instructions from the ClientAgents.
7. The method of claim 1, wherein cloud-centric machine room provides an interfacing operational window for user interaction.
8. The method of claim 1, wherein for a user's requirement to deploy a project deployed on one cluster to another cluster, at least a source code of the project to be published is directly obtained from the deployed cluster.
9. The method according to claim 1, wherein the information about the items to be published includes publishing type information, and the number of items of configuration information of the items to be published differs for items of different publishing types.
10. The method of claim 1, further comprising the steps of:
If one step of performing the synchronization of the compiled code to the target server is in error, a quick roll-back to the previous step to redeploy occurs.
11. The method of claim 1, further comprising the steps of:
And configuring the authority of an operator, and if the current operator does not have the authority of direct release for the project to be deployed, the deployment of the project needs the corresponding project responsible person to examine and approve, and the code synchronization is performed only after the examination and approval by the examining and approving person.
12. A multi-place cross-cluster server deployment system, comprising:
the configuration unit is arranged on the central server and used for generating a plurality of configuration files containing configuration information according to the project information to be issued and the server information to be deployed, and the configuration files correspond to the servers to be deployed one by one;
The to-be-issued project code acquisition unit is used for acquiring the project codes according to the issued project information, and compiling and packaging the project codes;
And the release unit is used for selecting a target server according to the configuration file, synchronizing the compiled code to the target server, firstly determining a remote management protocol used by a cluster where the selected target server is located before code synchronization, and selecting a corresponding mode according to different protocols for synchronization.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910739457.7A CN110572436B (en) | 2019-08-12 | 2019-08-12 | Multi-place cross-cluster server deployment method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910739457.7A CN110572436B (en) | 2019-08-12 | 2019-08-12 | Multi-place cross-cluster server deployment method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110572436A true CN110572436A (en) | 2019-12-13 |
| CN110572436B CN110572436B (en) | 2020-09-22 |
Family
ID=68775087
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910739457.7A Active CN110572436B (en) | 2019-08-12 | 2019-08-12 | Multi-place cross-cluster server deployment method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110572436B (en) |
Cited By (25)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109871241A (en) * | 2019-01-02 | 2019-06-11 | 石化盈科信息技术有限责任公司 | A kind of configuration method of Cross-environment application server |
| CN111273912A (en) * | 2020-01-19 | 2020-06-12 | 深圳创维－Rgb电子有限公司 | Method for distributing independent environment, intelligent terminal and storage medium |
| CN111338643A (en) * | 2020-02-26 | 2020-06-26 | 北京思特奇信息技术股份有限公司 | Method and system for synchronizing codes across hosts and networks |
| CN111367533A (en) * | 2020-02-16 | 2020-07-03 | 深圳市策维软件技术有限公司 | Full life cycle management method for EPOS (Ethernet Passive optical System) industrial APP |
| CN111475225A (en) * | 2020-03-31 | 2020-07-31 | 招商局金融科技有限公司 | Microservice configuration method, system, apparatus and computer readable storage medium |
| CN112181439A (en) * | 2020-09-24 | 2021-01-05 | 中国建设银行股份有限公司 | Micro-service deployment method and device, electronic equipment and readable storage medium |
| CN112256403A (en) * | 2020-11-11 | 2021-01-22 | 网易（杭州）网络有限公司 | Internal server deployment method, device, framework, medium and equipment |
| CN112272221A (en) * | 2020-10-19 | 2021-01-26 | 随锐科技集团股份有限公司 | Batch deployment method and system of server cluster and computer readable storage medium |
| CN112650480A (en) * | 2020-12-21 | 2021-04-13 | 上海多维度网络科技股份有限公司 | Code distribution method and phase device |
| CN112698839A (en) * | 2020-12-30 | 2021-04-23 | 深圳前海微众银行股份有限公司 | Data center node deployment method, device and system and computer storage medium |
| CN112749085A (en) * | 2020-12-31 | 2021-05-04 | 北京知因智慧科技有限公司 | Cross-cluster task issuing method and device, electronic equipment and storage medium |
| CN112769922A (en) * | 2020-12-31 | 2021-05-07 | 南京视察者智能科技有限公司 | Device and method for self-starting micro-service cluster |
| CN112799681A (en) * | 2021-02-02 | 2021-05-14 | 无锡车联天下信息技术有限公司 | Task deployment method, system, electronic equipment and storage medium |
| CN112988571A (en) * | 2021-02-09 | 2021-06-18 | 中国建设银行股份有限公司 | Automated compiling and deploying method and system for optimizing CC codes |
| CN113360164A (en) * | 2021-05-27 | 2021-09-07 | 上海信宝博通电子商务有限公司 | Method, device and storage medium for rapidly deploying application |
| CN113504928A (en) * | 2021-07-27 | 2021-10-15 | 银清科技有限公司 | Automatic application deployment method and device |
| CN113535186A (en) * | 2020-04-15 | 2021-10-22 | 北京京东振世信息技术有限公司 | Service application deployment method and device |
| CN113553075A (en) * | 2021-07-28 | 2021-10-26 | 用友汽车信息科技（上海）股份有限公司 | Deployment method, deployment system, computer device and storage medium for micro-services |
| CN113553163A (en) * | 2021-08-31 | 2021-10-26 | 重庆允成互联网科技有限公司 | Scheduler-based Jenkins application deployment method and device |
| CN113839821A (en) * | 2021-10-14 | 2021-12-24 | 京东科技信息技术有限公司 | Method, device, system, equipment and medium for deploying cluster and constructing infrastructure |
| CN113946350A (en) * | 2021-10-28 | 2022-01-18 | 苏州万店掌网络科技有限公司 | Deployment method and system of shared working space |
| CN114070889A (en) * | 2021-11-10 | 2022-02-18 | 北京百度网讯科技有限公司 | Configuration method, traffic forwarding method, device, storage medium, and program product |
| CN114389936A (en) * | 2021-11-26 | 2022-04-22 | 上海景域智能科技有限公司 | Cross-cloud multi-cluster deployment operation and maintenance method, system, processor and storage medium |
| CN117591132A (en) * | 2024-01-12 | 2024-02-23 | 杭州谐云科技有限公司 | Service release method and release system |
| CN117591132B (en) * | 2024-01-12 | 2024-04-26 | 杭州谐云科技有限公司 | Service release method and release system |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040093465A1 (en) * | 2002-10-28 | 2004-05-13 | Quicksilver Technology, Inc. | Cache for instruction set architecture |
| CN101727316A (en) * | 2008-10-22 | 2010-06-09 | 上海超级计算中心 | Parallel program integrated development system suitable for clustering system as well as application implementation method thereof |
| CN102148712A (en) * | 2011-04-21 | 2011-08-10 | 天讯天网(福建)网络科技有限公司 | Cloud computing-based service management system |
| CN103701788A (en) * | 2013-12-20 | 2014-04-02 | 深圳市金宏威技术股份有限公司 | Cross-platform flexible bus communication device and flexible bus |
| CN106549857A (en) * | 2016-11-30 | 2017-03-29 | 济南中维世纪科技有限公司 | A kind of method and system of trans-regional PUSH message |
| CN108052358A (en) * | 2017-12-12 | 2018-05-18 | 上海天旦网络科技发展有限公司 | A kind of system and method for distributed deployment |
| CN108924217A (en) * | 2018-06-29 | 2018-11-30 | 中山大学 | A kind of distribution cloud system Automation arranging method |
| CN108958741A (en) * | 2017-05-25 | 2018-12-07 | 中兴通讯股份有限公司 | A kind of controller cluster installation method and device |
-
2019
- 2019-08-12 CN CN201910739457.7A patent/CN110572436B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20040093465A1 (en) * | 2002-10-28 | 2004-05-13 | Quicksilver Technology, Inc. | Cache for instruction set architecture |
| CN101727316A (en) * | 2008-10-22 | 2010-06-09 | 上海超级计算中心 | Parallel program integrated development system suitable for clustering system as well as application implementation method thereof |
| CN102148712A (en) * | 2011-04-21 | 2011-08-10 | 天讯天网(福建)网络科技有限公司 | Cloud computing-based service management system |
| CN103701788A (en) * | 2013-12-20 | 2014-04-02 | 深圳市金宏威技术股份有限公司 | Cross-platform flexible bus communication device and flexible bus |
| CN106549857A (en) * | 2016-11-30 | 2017-03-29 | 济南中维世纪科技有限公司 | A kind of method and system of trans-regional PUSH message |
| CN108958741A (en) * | 2017-05-25 | 2018-12-07 | 中兴通讯股份有限公司 | A kind of controller cluster installation method and device |
| CN108052358A (en) * | 2017-12-12 | 2018-05-18 | 上海天旦网络科技发展有限公司 | A kind of system and method for distributed deployment |
| CN108924217A (en) * | 2018-06-29 | 2018-11-30 | 中山大学 | A kind of distribution cloud system Automation arranging method |
Non-Patent Citations (1)
| Title |
|---|
| 李宜达: "基于ACE的分布式服务器集群系统框架的设计和实现", 《中国优秀硕士学位论文全文数据库信息科技辑》 * |
Cited By (33)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109871241A (en) * | 2019-01-02 | 2019-06-11 | 石化盈科信息技术有限责任公司 | A kind of configuration method of Cross-environment application server |
| CN111273912A (en) * | 2020-01-19 | 2020-06-12 | 深圳创维－Rgb电子有限公司 | Method for distributing independent environment, intelligent terminal and storage medium |
| CN111273912B (en) * | 2020-01-19 | 2024-04-02 | 深圳创维－Rgb电子有限公司 | Independent environment distribution method, intelligent terminal and storage medium |
| CN111367533A (en) * | 2020-02-16 | 2020-07-03 | 深圳市策维软件技术有限公司 | Full life cycle management method for EPOS (Ethernet Passive optical System) industrial APP |
| CN111338643A (en) * | 2020-02-26 | 2020-06-26 | 北京思特奇信息技术股份有限公司 | Method and system for synchronizing codes across hosts and networks |
| CN111475225A (en) * | 2020-03-31 | 2020-07-31 | 招商局金融科技有限公司 | Microservice configuration method, system, apparatus and computer readable storage medium |
| CN113535186A (en) * | 2020-04-15 | 2021-10-22 | 北京京东振世信息技术有限公司 | Service application deployment method and device |
| CN113535186B (en) * | 2020-04-15 | 2023-09-22 | 北京京东振世信息技术有限公司 | Service application deployment method and device |
| CN112181439A (en) * | 2020-09-24 | 2021-01-05 | 中国建设银行股份有限公司 | Micro-service deployment method and device, electronic equipment and readable storage medium |
| CN112272221A (en) * | 2020-10-19 | 2021-01-26 | 随锐科技集团股份有限公司 | Batch deployment method and system of server cluster and computer readable storage medium |
| CN112272221B (en) * | 2020-10-19 | 2023-12-22 | 随锐科技集团股份有限公司 | Batch deployment method and system for server cluster and computer readable storage medium |
| CN112256403A (en) * | 2020-11-11 | 2021-01-22 | 网易（杭州）网络有限公司 | Internal server deployment method, device, framework, medium and equipment |
| CN112650480A (en) * | 2020-12-21 | 2021-04-13 | 上海多维度网络科技股份有限公司 | Code distribution method and phase device |
| CN112698839B (en) * | 2020-12-30 | 2024-04-12 | 深圳前海微众银行股份有限公司 | Data center node deployment method, device and system and computer storage medium |
| CN112698839A (en) * | 2020-12-30 | 2021-04-23 | 深圳前海微众银行股份有限公司 | Data center node deployment method, device and system and computer storage medium |
| CN112769922A (en) * | 2020-12-31 | 2021-05-07 | 南京视察者智能科技有限公司 | Device and method for self-starting micro-service cluster |
| CN112749085A (en) * | 2020-12-31 | 2021-05-04 | 北京知因智慧科技有限公司 | Cross-cluster task issuing method and device, electronic equipment and storage medium |
| CN112749085B (en) * | 2020-12-31 | 2023-10-24 | 北京知因智慧科技有限公司 | Cross-cluster task publishing method and device, electronic equipment and storage medium |
| CN112799681A (en) * | 2021-02-02 | 2021-05-14 | 无锡车联天下信息技术有限公司 | Task deployment method, system, electronic equipment and storage medium |
| CN112988571A (en) * | 2021-02-09 | 2021-06-18 | 中国建设银行股份有限公司 | Automated compiling and deploying method and system for optimizing CC codes |
| CN113360164A (en) * | 2021-05-27 | 2021-09-07 | 上海信宝博通电子商务有限公司 | Method, device and storage medium for rapidly deploying application |
| CN113504928A (en) * | 2021-07-27 | 2021-10-15 | 银清科技有限公司 | Automatic application deployment method and device |
| CN113553075A (en) * | 2021-07-28 | 2021-10-26 | 用友汽车信息科技（上海）股份有限公司 | Deployment method, deployment system, computer device and storage medium for micro-services |
| CN113553163A (en) * | 2021-08-31 | 2021-10-26 | 重庆允成互联网科技有限公司 | Scheduler-based Jenkins application deployment method and device |
| CN113553163B (en) * | 2021-08-31 | 2023-07-07 | 重庆允丰科技有限公司 | Deployment method and device for Jenkins application based on scheduler |
| CN113839821A (en) * | 2021-10-14 | 2021-12-24 | 京东科技信息技术有限公司 | Method, device, system, equipment and medium for deploying cluster and constructing infrastructure |
| CN113946350B (en) * | 2021-10-28 | 2022-08-19 | 苏州万店掌网络科技有限公司 | Deployment method and system of shared working space |
| CN113946350A (en) * | 2021-10-28 | 2022-01-18 | 苏州万店掌网络科技有限公司 | Deployment method and system of shared working space |
| CN114070889B (en) * | 2021-11-10 | 2023-11-14 | 北京百度网讯科技有限公司 | Configuration method, traffic forwarding device, storage medium, and program product |
| CN114070889A (en) * | 2021-11-10 | 2022-02-18 | 北京百度网讯科技有限公司 | Configuration method, traffic forwarding method, device, storage medium, and program product |
| CN114389936A (en) * | 2021-11-26 | 2022-04-22 | 上海景域智能科技有限公司 | Cross-cloud multi-cluster deployment operation and maintenance method, system, processor and storage medium |
| CN117591132A (en) * | 2024-01-12 | 2024-02-23 | 杭州谐云科技有限公司 | Service release method and release system |
| CN117591132B (en) * | 2024-01-12 | 2024-04-26 | 杭州谐云科技有限公司 | Service release method and release system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110572436B (en) | 2020-09-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110572436B (en) | Multi-place cross-cluster server deployment method and system | |
| CN106991035B (en) | Host monitoring system based on micro-service architecture | |
| US7774762B2 (en) | System including run-time software to enable a software application to execute on an incompatible computer platform | |
| US10942724B2 (en) | Release lifecycle management system for multi-node application | |
| US8151277B2 (en) | Method and system for dynamic remote injection of in-process agents into virtual machine based applications | |
| US20150112829A1 (en) | System and method for deploying and maintaining software applications | |
| US8719782B2 (en) | Integrated package development and machine configuration management | |
| US9081747B1 (en) | Computer program deployment to one or more target devices | |
| JP6329547B2 (en) | System and method for providing a service management engine for use in a cloud computing environment | |
| CN111989681A (en) | Automatically deployed Information Technology (IT) system and method | |
| US20170132031A1 (en) | Method, system and device for managing virtual machine software in cloud environment | |
| US20150121375A1 (en) | System and method for providing a physical plugin for use in a cloud platform environment | |
| US20050289538A1 (en) | Deploying an application software on a virtual deployment target | |
| EP3201772A1 (en) | Methods and systems for portably deploying applications on one or more cloud systems | |
| US20030217131A1 (en) | Processing distribution using instant copy | |
| JP2005502117A (en) | System and method for automatic installation and configuration of operating systems | |
| US10402216B1 (en) | Live support integration in a virtual machine based development environment | |
| CN1271893A (en) | System and method for improving management and usability of Java environment | |
| US20070101328A1 (en) | Sequencing a single task sequence across multiple operating environments | |
| US20190349428A1 (en) | Systems and methods for update propagation between nodes in a distributed system | |
| TW202105221A (en) | Automatically deployed information technology (it) system and method with enhanced security | |
| CN111949276B (en) | System and method for automatically deploying application programs based on container mode | |
| US20210149682A1 (en) | System and method for implementing a filesystem agent management solution | |
| US20170031667A1 (en) | Managing application lifecycles within a federation of distributed software applications | |
| Sethi et al. | Rapid deployment of SOA solutions via automated image replication and reconfiguration |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |