CN117544697A - Big data access method for protocol adaptation and scheduling in earthquake industry - Google Patents
Big data access method for protocol adaptation and scheduling in earthquake industry Download PDFInfo
- Publication number
- CN117544697A CN117544697A CN202311653155.0A CN202311653155A CN117544697A CN 117544697 A CN117544697 A CN 117544697A CN 202311653155 A CN202311653155 A CN 202311653155A CN 117544697 A CN117544697 A CN 117544697A
- Authority
- CN
- China
- Prior art keywords
- data
- client
- protocol
- scheduling
- service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 47
- 230000006978 adaptation Effects 0.000 title claims abstract description 27
- 238000012360 testing method Methods 0.000 claims abstract description 23
- 238000011084 recovery Methods 0.000 claims abstract description 9
- 238000012544 monitoring process Methods 0.000 claims abstract description 8
- 230000005540 biological transmission Effects 0.000 claims description 11
- 230000008569 process Effects 0.000 claims description 11
- 238000009826 distribution Methods 0.000 claims description 8
- 230000036541 health Effects 0.000 claims description 8
- 238000005096 rolling process Methods 0.000 claims description 6
- 239000000523 sample Substances 0.000 claims description 6
- 238000004140 cleaning Methods 0.000 claims description 5
- 238000012546 transfer Methods 0.000 claims description 5
- 230000002776 aggregation Effects 0.000 claims description 4
- 238000004220 aggregation Methods 0.000 claims description 4
- 238000004891 communication Methods 0.000 description 17
- 238000012545 processing Methods 0.000 description 11
- 238000004590 computer program Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 238000003860 storage Methods 0.000 description 4
- 238000006467 substitution reaction Methods 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000010354 integration Effects 0.000 description 3
- 230000003993 interaction Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 210000001503 joint Anatomy 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000003032 molecular docking Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/08—Protocols for interworking; Protocol conversion
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0654—Management of faults, events, alarms or notifications using network fault recovery
- H04L41/0663—Performing the actions predefined by failover planning, e.g. switching to standby network elements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0893—Assignment of logical groups to network elements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/16—Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]
- H04L69/161—Implementation details of TCP/IP or UDP/IP stack architecture; Specification of modified or new header fields
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/22—Parsing or analysis of headers
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Computer Security & Cryptography (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention relates to a big data access method for protocol adaptation and scheduling in the earthquake industry, which belongs to the technical field of big data in the earthquake, and comprises the following steps: performing protocol adaptation; automatically pulling up the client strategy scheduling; controlling through a client plug-in; receiving the seismic data; pushing the seismic data; and performing fault recovery and smooth upgrading. Compared with the prior art, which only can manually judge whether resources need to be added or not and has no standard or strategy, according to the availability requirement of different seismic data access and the availability information of each resource, the access tasks of different data types are divided into different priorities, and according to the total number of instrument terminals which need to be accessed and the number of instrument terminals actually supported by a single client in a cloud platform and the resource conditions consumed by different protocol data through monitoring test, the load threshold is configured to be 0.7, and the method can automatically judge whether the resources need to be added or not as a scheduling triggering condition.
Description
Technical Field
The invention belongs to the technical field of seismic big data, and particularly relates to a big data access method for protocol adaptation and scheduling in the seismic industry.
Background
In the seismic industry, real-time acquisition and stable access of seismic data are of great significance to subsequent data processing and data analysis. The large data access challenge in the earthquake industry mainly comprises the problems of huge data volume, various data sources, protocol differences, high data access stability requirement, non-uniform data formats and the like. Firstly, the data volume in the earthquake industry is huge, and various types of data such as earthquake observation data, state data, control data and the like are huge in scale and require an effective and stable access mode; secondly, the data sources in the earthquake industry are very various, and various data acquisition modes such as earthquake monitoring stations, earthquake instruments, satellite remote sensing and the like are included, and the diversity of the data sources and the difference of docking protocols bring certain complexity to data access; then, most of data in the earthquake industry is accessed in real time, the stability requirement on data access is extremely high, and the condition of data loss is avoided as much as possible; finally, the data format in the earthquake industry is also in a non-uniform condition, and different data sources adopt different data formats and protocols, so that the difficulty of data access is increased.
The traditional data access method only supports specific protocols and data formats, each protocol needs to be adapted and converted, the selection and data access capability of a data source are limited, and meanwhile, as the traditional data access service single machine is deployed, the coupling with the server environment resources is strong, and the risks of single-point faults and data loss exist.
Disclosure of Invention
In view of the shortcomings of the prior art, the invention aims to provide a large data access method for protocol adaptation and scheduling in the earthquake industry, which ensures support of multi-protocol instrument terminal data butt joint through multi-protocol plug-in integration, configures an availability sensing priority algorithm strategy according to availability requirements of different earthquake data accesses and availability information of each resource in an actual environment, configures a load threshold as a scheduling trigger condition, performs service pull-up and task distribution of a scheduler in combination with Kubernetes, distributes instrument terminals to different clients, and ensures high stability of data access through configuration of health check and automatic failover in the Kubernetes.
The invention provides a big data access method for protocol adaptation and scheduling in the earthquake industry, which comprises the following steps:
s1, performing protocol adaptation;
s2, automatically pulling up the strategy scheduling of the client;
s3, controlling through a client plug-in;
s4, receiving the seismic data;
s5, pushing the seismic data;
s6, fault recovery and smooth upgrading are carried out.
Further, the protocol adaptation specifically includes the following steps:
s11, inputting information of an access instrument terminal;
s12, performing plug-in test;
s13, issuing a test through a protocol adaptation instruction.
Further, the automatic pull-up client policy scheduling specifically includes the following steps:
s21, grouping instrument terminals with different protocol types, wherein service protocols in each group belong to the same protocol type;
s22, dividing tasks into different priorities according to availability requirements of different seismic data access and availability information of each resource;
s23, carrying out service pull-up and task distribution by setting available sensing priority algorithm strategies on the total number of instrument terminals to be accessed, the number of instrument terminals actually supported by a single client in a cloud platform, the passing monitoring test and the resource conditions consumed by different protocol data;
s24, setting a load threshold to be 70%, then carrying out test judgment, judging that the system is stable in operation if the resource usage is within a range of 70%, automatically calling up the client service when more instrument terminals need to be accessed and configuring relevant information when the load threshold exceeds 70%, and completing terminal data access.
Further, the controlling by the client plug-in specifically includes the following steps:
s31, after the client service is successfully pulled up, service registration is carried out in the aggregation gateway, and the service name, ip address information, type and externally exposed interface information of the client service in the cloud platform are stored in a database;
s32, after the client service finishes registration, sending heartbeat information to a convergence gateway, and recording the current client identification id, the total load of the client, the current load of the client, the number of currently accessed terminals, the number of normally accessed terminals, the load condition and the state of each plug-in unit;
s33, instruction control is carried out on the service management module through actual service needs and specific client service, and operation when the plug-in the service is connected with the terminal is controlled.
Further, the receiving of the seismic data specifically includes the following steps:
s41, establishing network connection between a client and a server, and transmitting data through a TCP/IP protocol;
s42, analyzing a data format through a protocol plug-in, and dividing the data into different parts according to protocol definition;
s43, analyzing the data and verifying the integrity of the data;
s44, decoding the parsed data according to an encoding mode defined by the seismic data transmission protocol, and further cleaning the parsed and decoded seismic data;
s45, after the data transmission is completed, the connection between the client and the server is closed, and resources and network ports are released.
Further, the pushing the seismic data specifically includes the following steps:
s51, according to specific subsystem data requirements, specifying the naming standards of topic in Pulsar, and automatically creating the topic through a Tenant, namespace, topic three-level limiting principle;
s52, after the client protocol plug-in analyzes the packet header information of the access data, pushing different subsystem data into the corresponding topic for use by a subsequent system.
Further, the fault recovery and smooth upgrade specifically includes the following steps:
s61, in the Pod configuration of the application program, performing definition health check, configuring a liveness probe and a readiness probe, and periodically checking the running state of the container;
s62, when a certain Pod fails, the Kubernetes automatically transfers the Pod on the node to other healthy nodes;
s63, triggering smooth upgrade by updating the mirror image version in the deployment resource or using a kubectl command, gradually starting the Pod of the new version by Kubernetes, and gradually stopping the Pod of the old version;
s64, when errors or other problems are detected in the upgrading process, automatically rolling back to the previous version, rolling back to the old version of deployment by updating the mirror version of the deployment resource or using a kubectl command, and rolling back to the previous state by the Kubernetes.
Further, in S44, the cleaning process includes deduplicating, sorting, and formatting the data.
Further, in S43, a hash method is used to verify the integrity of the data.
Further, in S33, the operation includes registration, test, start, stop.
The invention has the following beneficial effects:
1. compared with the prior art, which only can manually judge whether resources need to be added, has no standard and no strategy, the method has the advantages that access tasks of different data types are divided into different priorities according to the availability requirements of different seismic data accesses and the availability information of each resource, wherein the observation data needs to be accessed in real time, the real-time requirements are extremely high, and the priorities are set to be high; the state data and the control data need to be accessed in real time, but delay in a certain range is allowed, and the priority is set as the middle; the real-time requirement of the offline data is not high, only the timing synchronization is needed, the priority is set to be low, and according to the total number of instrument terminals to be accessed and the number of instrument terminals actually supported by a single client in the cloud platform and the resource conditions consumed by different protocol data through monitoring test, the load threshold is configured to be 0.7, and the scheduling trigger condition is used for automatically judging whether the resource needs to be added.
2. Compared with the prior art that when the data access service is required to be added, the service can only be started through background deployment, the service is pulled up through the set available sensing priority algorithm strategy, the scheduler performs task distribution, all instrument terminals to be connected are distributed to different clients, the instrument terminal access number of each client is maintained to a certain range, the consumed system resources are maintained to a certain range, more instrument terminals are required to be accessed, the client service can be automatically controlled and invoked, and relevant information is configured, so that the terminal data access is completed.
3. Aiming at the requirements of the earthquake industry, various related communication protocols, such as SeedLink, ntrip, HTTP, MQTT, coAP and the like, are packaged in a plug-in integrated mode, interfaces of different protocols are unified by utilizing an adapter mode, corresponding services are required to be started when data of different protocols are accessed, multiple types of protocols cannot be compatible at the same time, and compared with the modes of supporting multiple data formats, the data interaction and communication among different protocols can be realized, elastic expansion can be supported, and adapter examples can be dynamically increased or reduced according to requirements so as to meet different access requirements.
4. Compared with the prior art that services are often deployed into a server in a single node mode, once faults occur and are required to be restarted manually, and disaster recovery processing is not required, in the Kubernetes, health inspection is defined, a liveness probe and a readinessProbe are configured, whether a container normally operates is checked regularly, automatic fault transfer is configured, when a client service in a certain Pod is down, and when data access fails, the Kubernetes automatically transfer the Pod on the node to other healthy nodes, and high stability of the data access is ensured.
Drawings
The drawings are only for purposes of illustrating particular embodiments and are not to be construed as limiting the invention, like reference numerals being used to refer to like parts throughout the several views. It is apparent that the drawings in the following description are only some of the embodiments described in the embodiments of the present invention, and that other drawings may be obtained from these drawings by those of ordinary skill in the art.
FIG. 1 is a flow chart of an embodiment of the present invention;
FIG. 2 is a functional architecture diagram of a data access portion according to an embodiment of the present invention;
FIG. 3 is a flow chart of a protocol adaptation connectivity test according to an embodiment of the present invention;
fig. 4 is a flowchart of a client policy scheduling pull-up according to an embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the embodiments of the present invention better understood by those skilled in the art, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments. It should be understood that the description is only illustrative and is not intended to limit the scope of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, shall fall within the scope of the invention.
In addition, in the following description, descriptions of well-known structures and techniques are omitted so as not to unnecessarily obscure the concepts of the present disclosure.
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the invention. Rather, they are merely examples of methods and systems that are consistent with aspects of the invention as detailed in the accompanying claims.
The invention provides a large data access method for protocol adaptation and scheduling in the earthquake industry, which is used for solving the problems of large protocol difference, various data formats and high data access stability requirements faced by data access in the earthquake industry. And according to the availability requirement of data access and the availability information of each resource, an availability sensing priority algorithm strategy is made, a load threshold is configured as a scheduling triggering condition, service pull-up and scheduler task distribution automation processing are realized by combining with Kubernetes, instrument terminals are distributed to different clients, and the instrument terminal access number and system resource consumption of each client are maintained within a certain range. High stability of data access is ensured through configuration of health checks and automatic failover.
Method embodiment
In the invention, as shown in fig. 1-4, a big data access method for protocol adaptation and scheduling in the earthquake industry is provided, which comprises the following steps:
s1, performing protocol adaptation;
s2, automatically pulling up the strategy scheduling of the client;
s3, controlling through a client plug-in;
s4, receiving the seismic data;
s5, pushing the seismic data;
s6, fault recovery and smooth upgrading are carried out.
In addition, in the invention, aiming at the stable access of a large amount of data of instrument terminals with different protocol types in the earthquake industry, the invention has the following advantages:
(1) through multi-protocol plug-in integration, the data butt joint of the multi-class protocol instrument terminal is ensured to be supported;
(2) monitoring resources, and ensuring sufficient resources when the client service is to be pulled up;
(3) according to the availability requirements of different seismic data access and the availability information of each resource in the actual environment, an availability sensing priority algorithm strategy is configured;
(4) the load threshold is configured as a dispatching triggering condition, service pull-up and dispatcher task distribution are carried out by combining with Kubernetes, and instrument terminals are distributed to different clients;
(5) instruction control is carried out aiming at specific client service, and operations such as registration, test, start and stop are controlled when a plug-in the service is connected with a terminal;
(6) after the packet header information of the access data is analyzed by the client protocol plug-in, pushing different subsystem data into a topic corresponding to a message middleware Pulsar according to the specification for use by a subsequent system;
(7) high stability of data access is ensured through the configuration of health check and automatic failover in Kubernetes.
Regarding the above protocol adaptation, in the present invention, the following detailed flow is provided:
a. and the access instrument terminal information is input, according to requirements, the type and the data format of the seismic data transmission protocol which are required to be adapted are definitely recorded, the instrument terminals which are required to be accessed under different protocols are subjected to database table input, and input fields need to comprise the basic information such as the ip address, the port, the user name, the password, the station information, the station network information, the equipment type, the equipment model, the sub-sensor associated information, the observation data communication account number, the observation data communication password and the like of the terminal equipment, and meanwhile, the type distinguishing field is added, so that the multi-protocol plug-in unit of the client is convenient to identify and start.
b. And (3) testing the plug-ins, wherein different protocols acquire communication information of the plug-ins according to the protocol types and the instrument terminals, and select one to carry out a connection test so as to ensure that the protocol analysis function of the protocol plug-ins is normal.
c. The protocol adaptation instruction issues a test, interfaces of different protocols are packaged and distributed through a uniformly defined inlet, data formats are distinguished, an automatic adaptation protocol can be realized, data access connection test of an instrument terminal is carried out, plug-in packaging is carried out on related multiple protocols such as SeedLink, ntrip, HTTP, MQTT, coAP common protocols according to the requirements of the earthquake industry, interfaces of different protocols are unified through the uniformly defined interfaces and the uniformly defined data formats, and data interaction and communication between different protocols can be realized.
Regarding the above-described auto-pull client policy scheduling, in the present invention, the following detailed flow is provided:
a. for instrument terminals with different protocol types, the services in each group should belong to the same protocol type, and provide similar functions, 100 are taken as a batch to perform performance pressure test, and the monitoring tool Metrics Server of Kubernetes is utilized to collect index data of single-node resource use conditions, including CPU utilization rate, memory use amount, network bandwidth, data throughput and the like.
b. The tasks are divided into different priorities according to the availability requirements of different seismic data accesses and the availability information of each resource. For example, the tasks are divided into three priorities of high, medium and low, wherein the observation data needs to be accessed in real time and the real-time requirement is extremely high, and the priorities are set to be high; the state data and the control data need to be accessed in real time, but delay in a certain range is allowed, and the priority is set as the middle; the real-time requirement of the offline data is not high, only timing synchronization is needed, and the priority is set to be low.
c. According to the total number of instrument terminals to be accessed and the number of instrument terminals actually supported by a single client in a cloud platform, and the resource conditions consumed by different protocol data through monitoring tests, service pull-up and task distribution are carried out through a set available sensing priority algorithm strategy, all instrument terminals to be accessed are distributed to different clients, the instrument terminal access number of each client is maintained to a certain range, and the consumed system resources are maintained to a certain range.
d. At present, through testing, the resource use is in the range of 70%, and the system operates more stably, so that the threshold value is set to be 0.7. When more instrument terminals need to be accessed and the load threshold exceeds 0.7, the client service can be automatically called up, and related information is configured to complete terminal data access.
And the instrument terminal access number of each agent is maintained to a certain range through a set scheduling strategy according to the total number of instrument terminals which need to be accessed and the number of instrument terminals actually supported by a single client in the cloud platform. When more instrument terminals need to be accessed, the new client service can be automatically called up, and related information is configured to complete terminal data access.
In summary, in the invention, package of multiple communication protocols is realized by using a plug-in integration mode, and data interaction and communication between different protocols are realized by using an adapter mode to perform unified interface design.
In addition, various protocol plug-ins are integrated into a client, and the client simultaneously has the capability of pushing the accessed data to a message middleware Pulsar in a data packet mode in real time, so that real-time data access is completed.
Regarding the above control by the client plug-in, in the present invention, the following detailed flow is provided:
a. after the client service is successfully pulled up, service registration is required to be carried out in the aggregation gateway, and the service name, ip address information, type and externally exposed interface information of the client service in the cloud platform are stored in a database.
b. After the client service is registered, heartbeat information is continuously sent to the aggregation gateway, and the current client identification id, the total load of the client, the current load of the client, the number of currently accessed terminals, the number of normally accessed terminals, the load condition and the state of each plug-in unit are recorded.
c. The service management module can control the operations of registration, test, start, stop and the like when the plug-in the service is connected with the terminal according to the actual service requirement and aiming at the specific client service.
In summary, in the invention, an availability sensing priority algorithm strategy is made according to the availability requirement of data access and the availability information of each resource, a load threshold is configured as a scheduling trigger condition, service pull-up and scheduler task distribution are carried out in combination with Kubernetes, instrument terminals are distributed to different clients, the access number of the instrument terminals of each client and the consumption of system resources are ensured to be within a certain range, and finally the automatic control and high stability of the service are ensured.
And, the client uses stateless services as a carrier to create a container image containing service codes and running environments. The image may be built using a Docker or other containerization technique and uploaded into a container image repository.
Regarding the above-described reception of seismic data, in the present invention, the following detailed flow is provided:
a. protocol plug-in network connections. The client and the server establish network connection, data transmission is carried out by using TCP/IP protocol, and protocol handshake is carried out between the client and the server so as to ensure that both parties can correctly understand and analyze the seismic data transmission protocol, and the handshake process comprises sending and receiving information such as version numbers, handshake marks and the like.
b. The protocol plug-in parses the data format. And the client analyzes the received data according to the format of the seismic data transmission protocol. According to the protocol definition, the data is divided into different parts, such as a data head, a data body and the like, and the parsing process needs to identify and extract different information according to the structure and identification fields of the data.
c. And verifying the data integrity. When the data is analyzed, a hash method is used for verifying the integrity of the data, and according to the protocol definition, the data contains a verified field for ensuring that the data is not damaged or lost in the transmission process.
d. Data decoding and processing. And decoding the analyzed data according to the coding mode defined by the seismic data transmission protocol. And then, further cleaning the parsed and decoded seismic data, including performing operations such as de-duplication, sequencing and formatting on the data so as to meet application requirements.
f. Closing the connection. After the data transmission is completed, the connection between the client and the server can be closed, and the resources and the network ports are released.
In summary, in the invention, instruction control is performed for specific client service, so as to control operations such as registration, test, start and stop when a plug-in the service is connected with a terminal, and ensure controllability of the service.
Regarding the pushing of the seismic data, in the present invention, the following detailed flow is provided:
a. according to specific subsystem data requirements, the naming standards of topic in Pulsar are agreed, and the topic is automatically created by utilizing Tenant, namespace, topic three-level limiting principle.
b. After the packet header information of the access data is analyzed by the client protocol plug-in, different subsystem data are pushed into corresponding topic according to the specification so as to be used by a subsequent system.
In summary, in the present invention, after the client protocol plug-in analyzes the header information of the access data, different subsystem data is pushed to the corresponding topic according to the specification, so as to ensure the subsequent system use.
Regarding the above-described fault recovery and smoothing upgrade, in the present invention, the following detailed flow is provided:
a. in the Pod configuration of the application program, a health check is defined, and the liveness probe and readinessProbe are configured to periodically check whether the container is operating normally.
b. Automatic failover is configured. When a Pod fails, kubernetes will automatically transfer the Pod on that node to other healthy nodes. By using the cooperation of the container runtime (Container Runtime) and Control Plane (Control Plane), kubernetes can quickly resume service when a failure occurs.
c. A smooth upgrade is performed. Smooth upgrades are triggered by updating the image version in the deployment resource or using kubcctl commands. Kubernetes will gradually start a new version of Pod and gradually stop the old version of Pod. During the whole upgrading process, the clusters will remain available and no interruption of service will be perceived.
d. Fault tolerance and rollback. If an error or problem is detected during an upgrade, a rollback to the previous version may be performed. Rollback to the old version deployment is done by updating the mirrored version of the deployment resource or using the kubctl command. Kubernetes will start the old version of Pod and stop the new version of Pod to roll back to the previous state.
Regarding the configuration of Pod described above, in the present invention, a Pod description file is created, and the configuration of Pod is defined, including a container image name, a resource requirement, an environment variable, a port map, and the like. Creating a Service object to expose Pod to other services or users inside or outside the cluster, service being able to provide stable network access address for Pod and distribute traffic to the Pod of the backend according to load balancing algorithm
In summary, the present invention ensures high stability of data access and disaster recovery when a failure occurs through configuration of health check and automatic failover in Kubernetes.
Based on the same inventive concept, another embodiment of the present invention provides an electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface, the memory complete communication with each other through the communication bus,
a memory for storing a computer program;
and the processor is used for realizing the big data access method for the protocol adaptation and the scheduling of the earthquake industry when executing the program stored in the memory.
The communication bus mentioned by the above terminal may be a peripheral component interconnect standard (Peripheral Component Interconnect, abbreviated as PCI) bus or an extended industry standard architecture (Extended Industry Standard Architecture, abbreviated as EISA) bus, etc. The communication bus may be classified as an address bus, a data bus, a control bus, or the like. For ease of illustration, the figures are shown with only one bold line, but not with only one bus or one type of bus. The communication interface is used for communication between the terminal and other devices. The memory may include random access memory (Random Access Memory, RAM) or non-volatile memory (non-volatile memory), such as at least one disk memory. Optionally, the memory may also be at least one storage system located remotely from the aforementioned processor.
The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a network processor (Network Processor, NP for short), etc.; but also digital signal processors (Digital Signal Processing, DSP for short), application specific integrated circuits (Application Specific Integrated Circuit, ASIC for short), field-programmable gate arrays (Field-Programmable Gate Array, FPGA for short) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
In addition, in order to achieve the above objective, the embodiments of the present invention further provide a computer readable storage medium storing a computer program, where the computer program when executed by a processor implements the big data access method of the seismic industry protocol adaptation and scheduling of the embodiments of the present invention.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, embodiments of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the invention may take the form of a computer program product on one or more computer-usable vehicles having computer-usable program code embodied therein, including but not limited to disk storage, CD-ROM, optical storage, and the like.
Embodiments of the present invention are described with reference to flowchart illustrations and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, create a system for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Finally, it is further noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. "" and/or "" "means either or both of these can be selected. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or terminal. Without further limitation, an element defined by the statement "" comprising one … … "", does not exclude the presence of other identical elements in a process, method, article or terminal device comprising the element.
The present invention is not limited to the above embodiments, and various equivalent modifications and substitutions can be easily made by those skilled in the art within the technical scope of the present invention, and these modifications and substitutions are intended to be included in the scope of the present invention. Therefore, the protection scope of the invention is subject to the protection scope of the claims.
Finally, it should be noted that the above embodiments are merely for illustrating the technical solution of the embodiments of the present invention, and are not limiting. Although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the embodiments of the invention, and any changes and substitutions that would be apparent to one skilled in the art are intended to be included within the scope of the present invention.
Claims (10)
1. The big data access method for the protocol adaptation and scheduling of the earthquake industry is characterized by comprising the following steps:
s1, performing protocol adaptation;
s2, automatically pulling up the strategy scheduling of the client;
s3, controlling through a client plug-in;
s4, receiving the seismic data;
s5, pushing the seismic data;
s6, fault recovery and smooth upgrading are carried out.
2. The method for accessing big data for adapting and scheduling protocols in the earthquake industry according to claim 1, wherein the protocol adaptation comprises the following steps:
s11, inputting information of an access instrument terminal;
s12, performing plug-in test;
s13, issuing a test through a protocol adaptation instruction.
3. The big data access method for seismic industry protocol adaptation and scheduling according to claim 1, wherein the automatic pull-up client policy scheduling specifically comprises the following steps:
s21, grouping instrument terminals with different protocol types, wherein service protocols in each group belong to the same protocol type;
s22, dividing tasks into different priorities according to availability requirements of different seismic data access and availability information of each resource;
s23, carrying out service pull-up and task distribution by setting available sensing priority algorithm strategies on the total number of instrument terminals to be accessed, the number of instrument terminals actually supported by a single client in a cloud platform, the passing monitoring test and the resource conditions consumed by different protocol data;
s24, setting a load threshold to be 70%, then carrying out test judgment, judging that the system is stable in operation if the resource usage is within a range of 70%, automatically calling up the client service when more instrument terminals need to be accessed and configuring relevant information when the load threshold exceeds 70%, and completing terminal data access.
4. A method for accessing big data for adapting and scheduling a seismic industry protocol according to claim 3, wherein the controlling by a client plug-in specifically comprises the following steps:
s31, after the client service is successfully pulled up, service registration is carried out in the aggregation gateway, and the service name, ip address information, type and externally exposed interface information of the client service in the cloud platform are stored in a database;
s32, after the client service finishes registration, sending heartbeat information to a convergence gateway, and recording the current client identification id, the total load of the client, the current load of the client, the number of currently accessed terminals, the number of normally accessed terminals, the load condition and the state of each plug-in unit;
s33, instruction control is carried out on the service management module through actual service needs and specific client service, and operation when the plug-in the service is connected with the terminal is controlled.
5. The method for accessing big data for adapting and scheduling the seismic industry protocol according to claim 1, wherein the receiving of the seismic data comprises the following steps:
s41, establishing network connection between a client and a server, and transmitting data through a TCP/IP protocol;
s42, analyzing a data format through a protocol plug-in, and dividing the data into different parts according to protocol definition;
s43, analyzing the data and verifying the integrity of the data;
s44, decoding the parsed data according to an encoding mode defined by the seismic data transmission protocol, and further cleaning the parsed and decoded seismic data;
s45, after the data transmission is completed, the connection between the client and the server is closed, and resources and network ports are released.
6. The method for accessing big data for adapting and scheduling the seismic industry protocol according to claim 5, wherein pushing the seismic data comprises the following steps:
s51, according to specific subsystem data requirements, specifying the naming standards of topic in Pulsar, and automatically creating the topic through a Tenant, namespace, topic three-level limiting principle;
s52, after the client protocol plug-in analyzes the packet header information of the access data, pushing different subsystem data into the corresponding topic for use by a subsequent system.
7. The method for accessing big data for adapting and scheduling protocols in the earthquake industry according to claim 5, wherein the fault recovery and smooth upgrade steps comprise the following steps:
s61, in the Pod configuration of the application program, performing definition health check, configuring a liveness probe and a readiness probe, and periodically checking the running state of the container;
s62, when a certain Pod fails, the Kubernetes automatically transfers the Pod on the current node to other healthy nodes;
s63, triggering smooth upgrade by updating the mirror image version in the deployment resource or using a kubectl command, gradually starting the Pod of the new version by Kubernetes, and gradually stopping the Pod of the old version;
s64, when errors or other problems are detected in the upgrading process, automatically rolling back to the previous version, rolling back to the old version of deployment by updating the mirror version of the deployment resource or using a kubectl command, and rolling back to the previous state by the Kubernetes.
8. The method for large data access for seismic industry protocol adaptation and scheduling according to claim 5, wherein in S44, the cleaning process comprises de-duplication, ordering and formatting of the data.
9. The method for accessing large data for protocol adaptation and scheduling in the earthquake industry according to claim 5, wherein in S43, the method for verifying the integrity of the data uses a hash method.
10. The method for accessing big data for adaptation and scheduling of a seismic industry protocol according to claim 4, wherein in S33, the operations comprise registering, testing, starting and stopping.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311653155.0A CN117544697A (en) | 2023-12-05 | 2023-12-05 | Big data access method for protocol adaptation and scheduling in earthquake industry |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311653155.0A CN117544697A (en) | 2023-12-05 | 2023-12-05 | Big data access method for protocol adaptation and scheduling in earthquake industry |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117544697A true CN117544697A (en) | 2024-02-09 |
Family
ID=89785938
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311653155.0A Pending CN117544697A (en) | 2023-12-05 | 2023-12-05 | Big data access method for protocol adaptation and scheduling in earthquake industry |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117544697A (en) |
-
2023
- 2023-12-05 CN CN202311653155.0A patent/CN117544697A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9641449B2 (en) | Variable configurations for workload distribution across multiple sites | |
| US9331870B2 (en) | Switch, information processing apparatus, and information processing system | |
| CN113630372A (en) | Cloud edge coordination system for edge computing | |
| US10461990B2 (en) | Diagnostic traffic generation for automatic testing and troubleshooting | |
| CN112988485A (en) | Simulation test method and device for power Internet of things equipment | |
| CN105052076B (en) | Network element management system and network element management method based on cloud computing | |
| US11044188B2 (en) | Automated testing of push notification service in cloud computing system | |
| US9104565B2 (en) | Fault tracing system and method for remote maintenance | |
| CN102594618A (en) | Method and device for realizing storage device test of storage area network (SAN) | |
| CN106776346B (en) | Test method and device of CCMTS (China center testing System) equipment | |
| CN104283780A (en) | Method and device for establishing data transmission route | |
| CN113572726A (en) | Multi-mode network control-data plane consistency verification method and device | |
| CN102291274A (en) | System and method for realizing automatic adaptation of network testing instrument | |
| CN113703997A (en) | Bidirectional asynchronous communication middleware system integrating multiple message agents and implementation method | |
| CN112584398B (en) | Data processing method, device and storage medium | |
| CN103677983A (en) | Scheduling method and device of application | |
| CN110138753B (en) | Distributed message service system, method, apparatus, and computer-readable storage medium | |
| JP2013539644A (en) | Dynamic configuration of interconnected devices for measuring performance characteristics in a network | |
| CN115705190A (en) | Method and device for determining dependence degree | |
| CN117544697A (en) | Big data access method for protocol adaptation and scheduling in earthquake industry | |
| CN110825538A (en) | MQ-based method for dynamically interacting data by self-defining data types | |
| US9614891B1 (en) | Assembling communications based on captured packets | |
| CN114338477B (en) | Communication link monitoring method, device, equipment and storage medium | |
| CN111399749A (en) | Data processing system and method | |
| CN113032054B (en) | Service execution method and device, storage medium and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |