CN118277060A - Distributed high-availability rainfall forecast data analysis method, system, terminal and medium - Google Patents
Distributed high-availability rainfall forecast data analysis method, system, terminal and medium Download PDFInfo
- Publication number
- CN118277060A CN118277060A CN202410508182.7A CN202410508182A CN118277060A CN 118277060 A CN118277060 A CN 118277060A CN 202410508182 A CN202410508182 A CN 202410508182A CN 118277060 A CN118277060 A CN 118277060A
- Authority
- CN
- China
- Prior art keywords
- data
- program
- node
- meteorological
- data receiving
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007405 data analysis Methods 0.000 title claims abstract description 70
- 238000000034 method Methods 0.000 title claims abstract description 50
- 238000012545 processing Methods 0.000 claims abstract description 31
- 230000002159 abnormal effect Effects 0.000 claims description 12
- 238000004891 communication Methods 0.000 claims description 10
- 238000012544 monitoring process Methods 0.000 claims description 6
- 230000003993 interaction Effects 0.000 claims description 4
- 238000012216 screening Methods 0.000 claims description 3
- 230000006870 function Effects 0.000 description 15
- 238000010586 diagram Methods 0.000 description 10
- 238000004458 analytical method Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 5
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 239000000243 solution Substances 0.000 description 3
- 239000000872 buffer Substances 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000002265 prevention Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000001556 precipitation Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000004904 shortening Methods 0.000 description 1
- 239000012086 standard solution Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention relates to the field of rainfall forecast, in particular to a distributed high-availability rainfall forecast data analysis method, a system, a terminal and a medium, wherein an original meteorological data processing single program is split into a data receiving program and a data analysis program of meteorological data; deploying the data receiving program to at least one data receiving node, and deploying the data analyzing program to at least one data analyzing node; selecting a data receiving node, executing a data receiving program on the data receiving node to receive meteorological data, and storing the meteorological data in a distributed object storage system; dividing meteorological data into N pieces of data to be analyzed, and creating N thread tasks, wherein each thread task corresponds to one piece of data to be analyzed; and distributing the thread tasks to each data analysis node, and executing the thread tasks by the data analysis program. The invention decouples the data processing program, builds a high-availability distributed architecture, simultaneously executes multi-thread operation, and greatly improves the timeliness of weather forecast.
Description
Technical Field
The invention relates to the field of rainfall forecast, in particular to a distributed high-availability rainfall forecast data analysis method, a system, a terminal and a medium.
Background
The main characteristics of the monsoon climate area are that the annual change rate of the precipitation amount is high, drought and flood problems are easy to occur, life and property safety of people is threatened, flood prevention and drought resistance pressures are huge, the rainfall condition can be predicted in advance, early discovery, early warning and early scheduling are achieved, loss caused by disasters is reduced to the greatest extent, and the method plays an important role in flood prevention and drought resistance.
The weather forecast is effectively applied to flood control and fighting, forecast the rainfall condition in the future in advance, and provide key basis for the decision of flood fighting and other activities, can greatly enhance the management capability of flood control work, and has important significance for reducing unnecessary economic loss and safety accidents.
In the water conservancy industry, the method is mainly applied to the rainfall forecast of the grid of 1*1 kilometers in the future 72 hours and the rainfall forecast of 5 minutes in the future 2 hours, the data sources are mainly to purchase the meteorological data from a third party company or a meteorological office, the corresponding meteorological data is received at fixed time, the meteorological data of the structure is analyzed and processed, the future rainfall data of each grid is rendered, the processed meteorological forecast service is provided for the outside, and the service can be called in the system to be displayed on a map or be displayed independently.
The current initial weather data receiving and processing program is a traditional single machine architecture, and the rainfall forecast data (about 14 hundred million grid data) is processed once for 72 hours in the future in the country under a single 2-core 8G server, which requires about 90 minutes, so that the timeliness of weather forecast is seriously affected, and the value of weather forecast is reduced.
Disclosure of Invention
In order to solve the problems, the invention provides a distributed high-availability rainfall forecast data analysis method, a system, a terminal and a medium, which are used for decoupling a data processing program, constructing a high-availability distributed architecture, executing multi-thread operation at the same time and greatly improving the timeliness of weather forecast.
In a first aspect, the present invention provides a distributed high-availability rainfall forecast data analysis method, including the following steps:
splitting an original meteorological data processing single program into a data receiving program and a data analyzing program of meteorological data;
Deploying the data receiving program to at least one node, and deploying the data analyzing program to at least one node; the node with the data receiving program is recorded as a data receiving node, and the node with the data analyzing program is recorded as a data analyzing node;
Selecting a data receiving node, executing a data receiving program on the data receiving node to receive meteorological data, and storing the meteorological data in a distributed object storage system;
dividing meteorological data into N pieces of data to be analyzed, and creating N thread tasks, wherein each thread task corresponds to one piece of data to be analyzed;
and distributing the thread tasks to each data analysis node, and executing the thread tasks by the data analysis program.
In an alternative embodiment, the original meteorological data processing monomer program is split into a data receiving program and a data analyzing program of meteorological data, which specifically comprise:
determining a target split point according to the function and business logic of the original meteorological data processing single program;
splitting an original meteorological data processing single program into a data receiving program and a data analyzing program of meteorological data at a target splitting point;
Configuring a communication interface of a data receiving program and a data analyzing program;
the data interaction formats of the data receiving program and the data analyzing program are configured.
In an alternative embodiment, the distributed object storage system is Minio distributed object storage systems.
In an alternative embodiment, selecting a data receiving node specifically includes:
monitoring heartbeat of each data receiving node by using a zookeeper component;
If the heartbeat state is normal, judging that the data receiving node operates normally; if the heartbeat state is abnormal, judging that the data receiving node is abnormal in operation;
detecting the load state of each data receiving node which operates normally;
and screening out the data receiving node with the minimum current load as a target data receiving node.
In an alternative embodiment, the allocation of the thread task to each data analysis node specifically includes:
Thread tasks are distributed to the data parsing nodes according to a polling algorithm using a zookeeper component.
In an alternative embodiment, the allocation of the thread tasks to the respective data parsing nodes further comprises:
Monitoring the heartbeat of each data analysis node in real time by using a zookeeper component;
if the heartbeat state is normal, judging that the data analysis node operates normally; if the heartbeat state is abnormal, judging that the operation of the data analysis node is abnormal;
if the data analysis node runs abnormally, the thread task on the data analysis node is distributed to other data analysis nodes which run normally.
In an optional implementation manner, dividing the meteorological data into N parts of data to be analyzed specifically includes:
dividing the meteorological data into N parts of data to be analyzed according to the hour.
In a second aspect, the present invention provides a distributed high availability rainfall forecast data analysis system, comprising,
Program splitting module: splitting an original meteorological data processing single program into a data receiving program and a data analyzing program of meteorological data;
program deployment module: deploying the data receiving program to at least one node, and deploying the data analyzing program to at least one node; the node with the data receiving program is recorded as a data receiving node, and the node with the data analyzing program is recorded as a data analyzing node;
and a data receiving module: selecting a data receiving node, executing a data receiving program on the data receiving node to receive meteorological data, and storing the meteorological data in a distributed object storage system;
and a task splitting module: dividing meteorological data into N pieces of data to be analyzed, and creating N thread tasks, wherein each thread task corresponds to one piece of data to be analyzed;
and a data analysis module: and distributing the thread tasks to each data analysis node, and executing the thread tasks by the data analysis program.
In a third aspect, a technical solution of the present invention provides a terminal, including:
the storage is used for storing a distributed high-availability rainfall forecast data analysis program;
And the processor is used for realizing the steps of the distributed high-availability rainfall forecast data analysis method when executing the distributed high-availability rainfall forecast data analysis program.
In a fourth aspect, the present invention provides a computer readable storage medium, on which a distributed high availability rainfall forecast data analysis program is stored, where the distributed high availability rainfall forecast data analysis program when executed by a processor implements the steps of the distributed high availability rainfall forecast data analysis method according to any one of the above.
The distributed high-availability rainfall forecast data analysis method, system, terminal and medium provided by the invention have the following beneficial effects compared with the prior art: decoupling a data processing program, constructing a high-availability distributed architecture, simultaneously executing multi-thread operation, shortening the processing time of rainfall forecast data of 72 hours in the future to 3 minutes under the condition of three servers of 2 cores 8G, improving the performance by 40 times, greatly improving the timeliness of weather forecast, solving the problem of slow processing of large data amount weather data, processing the received data in a short time and providing services for the outside; the program is deployed on a plurality of nodes, so that the stability problem of the weather service is solved, the high availability of the weather service is ensured, and the problem of one node is solved, so that the normal use of the weather service is not influenced; and storing the data into a distributed object storage system, solving the problem of calling the meteorological data across servers, and ensuring that the distributed cluster nodes corresponding to the meteorological analysis service can acquire the corresponding meteorological data.
Drawings
For a clearer description of embodiments of the invention or of the prior art, the drawings that are used in the description of the embodiments or of the prior art will be briefly described, it being apparent that the drawings in the description below are only some embodiments of the invention, and that other drawings can be obtained from them without inventive effort for a person skilled in the art.
Fig. 1 is a schematic flow chart of a method for analyzing distributed high-availability rainfall forecast data according to an embodiment of the present invention.
FIG. 2 is a program split diagram.
Fig. 3 is a schematic drawing of the lead-in Minio.
Fig. 4 is a schematic diagram of a high availability deployment of data reception procedures.
FIG. 5 is a task multithreading split schematic.
Fig. 6 is a schematic diagram of distributed distribution of analytic tasks.
Fig. 7 is a schematic block diagram of a distributed high-availability rainfall forecast data analysis system according to an embodiment of the present invention.
Fig. 8 is a schematic structural diagram of a terminal according to an embodiment of the present invention.
Detailed Description
In order to better understand the aspects of the present invention, the present invention will be described in further detail with reference to the accompanying drawings and detailed description. It will be apparent that the described embodiments are only some, but not all, embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.
The following explains key terms appearing in the present invention.
Minio: minio is an object storage item based on APACHE LICENSE V2.0 open source protocol, implemented using Golang, and the client supports Java, python, javacript, golang language. The main goal of its design is a standard solution stored as a private cloud object. The method is mainly used for storing massive pictures, videos, documents and the like. It is well suited for storing large volumes of unstructured data, such as pictures, videos, log files, backup data, and container/virtual machine images, etc., while an object file may be any size, varying from a few kb up to a maximum of 5T.
ZooKeeper: the ZooKeeper is a distributed application coordination service of open source codes, is a realization of Chubby of Google as an open source, and is an important component of Hadoop and Hbase. It is a software providing a consistency service for distributed applications, the provided functions include: configuration maintenance, domain name service, distributed synchronization, group service, etc.
Thread: a thread is the smallest unit that an operating system can perform operational scheduling. It is included in the process and is the actual unit of operation in the process. One thread refers to a single sequential control flow in a process, and multiple threads can be concurrent in a process, each thread executing different tasks in parallel.
The method has high availability: high availability (High Availability, HA for short) refers to systems that are specifically designed to reduce downtime, thereby maintaining a high availability of services. The method is one of factors which must be considered in the design of the distributed system architecture, and aims to ensure the continuity of the service, namely, the service always provides services normally and externally in view of users.
Distributed computing: distributed computing is a research direction in computer science that studies how to divide a problem that requires a very large amount of computing power into many small parts, then distribute the parts to multiple computers for processing, and finally integrate the results of the computations to obtain the final result.
Fig. 1 is a schematic flow chart of a method for analyzing distributed high-availability rainfall forecast data according to an embodiment of the present invention. The execution subject of fig. 1 may be a distributed high-availability rainfall forecast data analysis method system. The distributed high-availability rainfall forecast data analysis method provided by the embodiment of the invention is executed by the computer equipment, and correspondingly, the distributed high-availability rainfall forecast data analysis method system is operated in the computer equipment. The order of the steps in the flow chart may be changed and some may be omitted according to different needs.
As shown in fig. 1, the method includes the following steps.
S1, splitting an original meteorological data processing single program into a data receiving program and a data analyzing program of meteorological data.
In this embodiment, the weather forecast service is divided according to functions, and the original single program splits the weather data receiving program and the weather data analyzing program to implement the decoupling of the program, as shown in fig. 2, which is a program splitting schematic diagram, and specifically includes the following steps.
S1.1, determining a target split point according to the function and business logic of an original meteorological data processing single program.
The program structure is first understood and the split point is determined on this basis. The source codes of the single program are deeply analyzed, the overall architecture and the functions of each module are understood, and the functional modules or business logic which can be independent and used as independent programs are identified. And determining a proper split point according to the functions and business logic of the program and combining specific requirements.
S1.2, splitting the original meteorological data processing single program into a data receiving program and a data analyzing program of meteorological data at a target splitting point.
The code is split into two separate programs according to the determined split point, ensuring that each program contains complete business logic and necessary data processing functions.
S1.3, configuring communication interfaces of a data receiving program and a data analyzing program.
S1.4, configuring data interaction formats of a data receiving program and a data analyzing program.
Considering how to interact with another program, a communication interface between the two programs needs to be configured, which may be a RESTful API, a message queue, a shared database, etc., and determines a format of data interaction, such as JSON, XML, or a protocol buffer (Protocol Buffers).
S2, deploying the data receiving program to at least one node, and deploying the data analyzing program to at least one node; the node with the data receiving program is recorded as a data receiving node, and the node with the data analyzing program is recorded as a data analyzing node.
In this embodiment, in order to ensure accurate reception of meteorological data, the problem that the meteorological data cannot be acquired on time due to a fault of a reception program does not occur. The receiving program is modified with high availability, and the data receiving program is deployed on a plurality of nodes. Meanwhile, in order to improve the processing analysis efficiency, the data analysis program also performs multi-node deployment.
It should be noted that a certain node may be both a data receiving node and a data parsing node.
S3, selecting a data receiving node, executing a data receiving program on the data receiving node to receive meteorological data, and storing the meteorological data in a distributed object storage system.
Firstly, when a single node is used, the meteorological data and the meteorological data analysis program are both on one server, and data reading is not problematic, but when a cluster is deployed, the received meteorological data and the received meteorological data analysis program are possibly not on one server, and the data cannot be acquired. The embodiment changes the original weather data storage mode from original local storage to distributed object storage system storage, and the distributed object storage system can be Minio. Thus, all servers can remotely invoke the meteorological data in Minio, as shown in FIG. 3 as an import Minio.
In particular, a data receiving node is selected, and a data receiving program on the node is executed to receive weather data and store the weather data in Minio. A schematic of a high availability deployment of data reception procedures is shown in fig. 4.
In the embodiment, the zookeeper component monitors the heartbeat of each data receiving node, selects a receiving program on a normal data receiving node, completes the reception of meteorological data, and writes the meteorological data into the Minio distributed object storage system.
Wherein, selecting a data receiving node specifically comprises the following steps.
And S3.1, monitoring heartbeat of each data receiving node by using a zookeeper component.
S3.2, if the heartbeat state is normal, judging that the data receiving node operates normally; and if the heartbeat state is abnormal, judging that the data receiving node is abnormal in operation.
And S3.3, detecting the load state of each data receiving node which operates normally.
And S3.4, screening out the data receiving node with the minimum current load as a target data receiving node.
The embodiment adopts Minio distributed object storage system to store meteorological data, so that the storage and management of the data are more efficient and reliable. The Minio distributed object storage system has the characteristics of high availability, high reliability, high expansibility and the like, can ensure the safety and stability of data, and simultaneously provides flexible data access and management functions.
Furthermore, the embodiment realizes dynamic selection and task allocation of the data receiving node by using the zookeeper component to monitor node heartbeat and detect load state. The method not only ensures the high availability of the system, namely, the task can be automatically transferred to other normally operated nodes when the nodes are in fault, but also realizes the reasonable allocation of resources through a load balancing mechanism, and avoids the overload or idle of the nodes.
And S4, dividing the meteorological data into N pieces of data to be analyzed, and creating N thread tasks, wherein each thread task corresponds to one piece of data to be analyzed.
The original 72-hour rainfall forecast data in the future is analyzed once, and according to the characteristics of the 72-hour rainfall forecast data in the future, the data is three-dimensional data of 72 x 4156 x 4536, 72 represents the data in the future of 72 hours, and 4156 x 4536 represents the grid data in one hour. In this embodiment, in order to further improve the processing and analysis efficiency, the meteorological data is divided into N parts of data to be analyzed, which can be divided according to hours and split into 72 parts of threads for execution, as shown in fig. 5, which is a task multithreading split schematic diagram.
And S5, distributing the thread tasks to each data analysis node, and executing the data analysis program.
In the embodiment, tasks are split according to the hour dimension, so that a distributed architecture is introduced for further improving the processing analysis efficiency, and a data analysis program is deployed by multiple nodes. The task distribution scheduling is realized by adopting a zookeeper component.
Specifically, using a zookeeper component, thread tasks are distributed to each data analysis node according to a polling algorithm, and a analysis task distributed distribution diagram is shown in fig. 6.
When the analysis tasks are distributed to each data analysis node, the heartbeat of the data analysis node is monitored, and if the data analysis node is disconnected, the tasks are distributed to other data analysis nodes to operate, so that high availability is ensured on the premise of ensuring the efficiency.
The method comprises the following steps of distributing thread tasks to each data analysis node.
S5.1, monitoring the heartbeat of each data analysis node in real time by using a zookeeper component.
S5.2, if the heartbeat state is normal, judging that the data analysis node operates normally; and if the heartbeat state is abnormal, judging that the data analysis node is abnormal in operation.
And S5.3, if the data analysis node runs abnormally, distributing the thread task on the data analysis node to other data analysis nodes which run normally.
According to the embodiment, the meteorological data are divided into a plurality of pieces of data to be analyzed according to the hour, and corresponding thread tasks are created for parallel processing, so that the data analysis efficiency is further improved. The parallel processing mode can fully utilize system resources and shorten data processing time, so that accurate data support is provided for rainfall forecast in time.
The embodiment of the distributed high-availability rainfall forecast data analysis method is described in detail above, and the distributed high-availability rainfall forecast data analysis system corresponding to the method is also provided based on the distributed high-availability rainfall forecast data analysis method described in the embodiment.
Fig. 7 is a schematic block diagram of a distributed high-availability rainfall forecast data analysis system according to an embodiment of the present invention, where the distributed high-availability rainfall forecast data analysis system 700 may be divided into a plurality of functional modules according to the functions performed by the distributed high-availability rainfall forecast data analysis system, as shown in fig. 7. The functional module may include: program splitting module 710, program deployment module 720, data receiving module 730, task splitting module 740, data parsing module 750. The module referred to in the present invention refers to a series of computer program segments capable of being executed by at least one processor and of performing a fixed function, stored in a memory.
Program splitting module 710: splitting an original meteorological data processing single program into a data receiving program and a data analyzing program of meteorological data.
Program deployment module 720: deploying the data receiving program to at least one node, and deploying the data analyzing program to at least one node; the node with the data receiving program is recorded as a data receiving node, and the node with the data analyzing program is recorded as a data analyzing node.
The data receiving module 730: a data receiving node is selected, a data receiving program on the data receiving node is executed to receive meteorological data and store the meteorological data in a distributed object storage system.
Task splitting module 740: dividing meteorological data into N pieces of data to be analyzed, and creating N thread tasks, wherein each thread task corresponds to one piece of data to be analyzed.
The data parsing module 750: and distributing the thread tasks to each data analysis node, and executing the thread tasks by the data analysis program.
The distributed high availability rainfall forecast data analysis system of this embodiment is used to implement the foregoing distributed high availability rainfall forecast data analysis method, so that the specific implementation of this system can be seen from the foregoing example part of the distributed high availability rainfall forecast data analysis method, so that the specific implementation thereof can refer to the description of the corresponding examples of each part, and will not be described herein.
In addition, since the distributed high-availability rainfall forecast data analysis system of the present embodiment is used to implement the foregoing distributed high-availability rainfall forecast data analysis method, the functions thereof correspond to those of the foregoing method, and will not be described herein.
Fig. 8 is a schematic structural diagram of a terminal 800 according to an embodiment of the present invention, including: processor 810, memory 820, and communication unit 830. The processor 810 is configured to implement the following steps when implementing the distributed high availability rainfall forecast data analysis program stored in the memory 820:
splitting an original meteorological data processing single program into a data receiving program and a data analyzing program of meteorological data;
Deploying the data receiving program to at least one node, and deploying the data analyzing program to at least one node; the node with the data receiving program is recorded as a data receiving node, and the node with the data analyzing program is recorded as a data analyzing node;
Selecting a data receiving node, executing a data receiving program on the data receiving node to receive meteorological data, and storing the meteorological data in a distributed object storage system;
dividing meteorological data into N pieces of data to be analyzed, and creating N thread tasks, wherein each thread task corresponds to one piece of data to be analyzed;
and distributing the thread tasks to each data analysis node, and executing the thread tasks by the data analysis program.
The terminal 800 includes a processor 810, a memory 820, and a communication unit 830. The components may communicate via one or more buses, and it will be appreciated by those skilled in the art that the configuration of the server as shown in the drawings is not limiting of the invention, as it may be a bus-like structure, a star-like structure, or include more or fewer components than shown, or may be a combination of certain components or a different arrangement of components.
The memory 820 may be implemented by any type of volatile or non-volatile memory terminal or combination thereof, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk, among other things, for storing instructions for execution by the processor 810. The execution of the instructions in memory 820, when executed by processor 810, enables terminal 800 to perform some or all of the steps in the method embodiments described below.
The processor 810 is a control center of the storage terminal, connects various parts of the entire electronic terminal using various interfaces and lines, and performs various functions of the electronic terminal and/or processes data by running or executing software programs and/or modules stored in the memory 820, and invoking data stored in the memory. The processor may be comprised of an integrated circuit (INTEGRATED CIRCUIT, simply referred to as an IC), for example, a single packaged IC, or may be comprised of multiple packaged ICs connected to one another for the same function or for different functions. For example, the processor 810 may include only a central processing unit (Central Processing Unit, CPU for short). In the embodiment of the invention, the CPU can be a single operation core or can comprise multiple operation cores.
A communication unit 830, configured to establish a communication channel, so that the storage terminal may communicate with other terminals. Receiving user data sent by other terminals or sending the user data to other terminals.
The invention also provides a computer storage medium, which can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (random access memory, RAM) and the like.
The computer storage medium stores a distributed high-availability rainfall forecast data analysis program, and the distributed high-availability rainfall forecast data analysis program realizes the following steps when being executed by a processor:
splitting an original meteorological data processing single program into a data receiving program and a data analyzing program of meteorological data;
Deploying the data receiving program to at least one node, and deploying the data analyzing program to at least one node; the node with the data receiving program is recorded as a data receiving node, and the node with the data analyzing program is recorded as a data analyzing node;
Selecting a data receiving node, executing a data receiving program on the data receiving node to receive meteorological data, and storing the meteorological data in a distributed object storage system;
dividing meteorological data into N pieces of data to be analyzed, and creating N thread tasks, wherein each thread task corresponds to one piece of data to be analyzed;
and distributing the thread tasks to each data analysis node, and executing the thread tasks by the data analysis program.
It will be apparent to those skilled in the art that the techniques of embodiments of the present invention may be implemented in software plus a necessary general purpose hardware platform. Based on such understanding, the technical solution in the embodiments of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium such as a U-disc, a mobile hard disc, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk or an optical disk, etc. various media capable of storing program codes, including several instructions for causing a computer terminal (which may be a personal computer, a server, or a second terminal, a network terminal, etc.) to execute all or part of the steps of the method described in the embodiments of the present invention.
In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.
The foregoing disclosure is merely illustrative of the preferred embodiments of the invention and the invention is not limited thereto, since modifications and variations may be made by those skilled in the art without departing from the principles of the invention.
Claims (10)
1. The distributed high-availability rainfall forecast data analysis method is characterized by comprising the following steps of:
splitting an original meteorological data processing single program into a data receiving program and a data analyzing program of meteorological data;
Deploying the data receiving program to at least one node, and deploying the data analyzing program to at least one node; the node with the data receiving program is recorded as a data receiving node, and the node with the data analyzing program is recorded as a data analyzing node;
Selecting a data receiving node, executing a data receiving program on the data receiving node to receive meteorological data, and storing the meteorological data in a distributed object storage system;
dividing meteorological data into N pieces of data to be analyzed, and creating N thread tasks, wherein each thread task corresponds to one piece of data to be analyzed;
and distributing the thread tasks to each data analysis node, and executing the thread tasks by the data analysis program.
2. The method for analyzing the distributed high-availability rainfall forecast data according to claim 1, wherein the single program for processing the original meteorological data is split into a data receiving program and a data analyzing program of the meteorological data, and the method specifically comprises the following steps:
determining a target split point according to the function and business logic of the original meteorological data processing single program;
splitting an original meteorological data processing single program into a data receiving program and a data analyzing program of meteorological data at a target splitting point;
Configuring a communication interface of a data receiving program and a data analyzing program;
the data interaction formats of the data receiving program and the data analyzing program are configured.
3. The distributed high availability rainfall forecast data analysis method of claim 1, wherein the distributed object storage system is a Minio distributed object storage system.
4. A distributed high availability rainfall forecast data analysis method according to claim 1 or 3, characterized in that selecting a data receiving node specifically comprises:
monitoring heartbeat of each data receiving node by using a zookeeper component;
If the heartbeat state is normal, judging that the data receiving node operates normally; if the heartbeat state is abnormal, judging that the data receiving node is abnormal in operation;
detecting the load state of each data receiving node which operates normally;
and screening out the data receiving node with the minimum current load as a target data receiving node.
5. A distributed high availability rainfall forecast data analysis method according to claim 1 or 3, wherein the allocation of thread tasks to each data analysis node comprises:
Thread tasks are distributed to the data parsing nodes according to a polling algorithm using a zookeeper component.
6. The distributed high availability rainfall forecast data analysis method of claim 5, wherein the allocation of thread tasks to the individual data analysis nodes further comprises:
Monitoring the heartbeat of each data analysis node in real time by using a zookeeper component;
if the heartbeat state is normal, judging that the data analysis node operates normally; if the heartbeat state is abnormal, judging that the operation of the data analysis node is abnormal;
if the data analysis node runs abnormally, the thread task on the data analysis node is distributed to other data analysis nodes which run normally.
7. The method for analyzing distributed high-availability rainfall forecast data according to claim 6, wherein the dividing of the meteorological data into N pieces of data to be analyzed specifically comprises:
dividing the meteorological data into N parts of data to be analyzed according to the hour.
8. A distributed high-availability rainfall forecast data analysis system is characterized by comprising,
Program splitting module: splitting an original meteorological data processing single program into a data receiving program and a data analyzing program of meteorological data;
program deployment module: deploying the data receiving program to at least one node, and deploying the data analyzing program to at least one node; the node with the data receiving program is recorded as a data receiving node, and the node with the data analyzing program is recorded as a data analyzing node;
and a data receiving module: selecting a data receiving node, executing a data receiving program on the data receiving node to receive meteorological data, and storing the meteorological data in a distributed object storage system;
and a task splitting module: dividing meteorological data into N pieces of data to be analyzed, and creating N thread tasks, wherein each thread task corresponds to one piece of data to be analyzed;
and a data analysis module: and distributing the thread tasks to each data analysis node, and executing the thread tasks by the data analysis program.
9. A terminal, comprising:
the storage is used for storing a distributed high-availability rainfall forecast data analysis program;
A processor for implementing the steps of the distributed high availability rainfall forecast data parsing method according to any of claims 1-7 when executing the distributed high availability rainfall forecast data parsing program.
10. A computer readable storage medium, wherein a distributed high availability rainfall forecast data parsing program is stored on the readable storage medium, which when executed by a processor, implements the steps of the distributed high availability rainfall forecast data parsing method according to any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410508182.7A CN118277060A (en) | 2024-04-25 | 2024-04-25 | Distributed high-availability rainfall forecast data analysis method, system, terminal and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410508182.7A CN118277060A (en) | 2024-04-25 | 2024-04-25 | Distributed high-availability rainfall forecast data analysis method, system, terminal and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN118277060A true CN118277060A (en) | 2024-07-02 |
Family
ID=91633923
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410508182.7A Pending CN118277060A (en) | 2024-04-25 | 2024-04-25 | Distributed high-availability rainfall forecast data analysis method, system, terminal and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118277060A (en) |
-
2024
- 2024-04-25 CN CN202410508182.7A patent/CN118277060A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111049705B (en) | Method and device for monitoring distributed storage system | |
| US11755452B2 (en) | Log data collection method based on log data generated by container in application container environment, log data collection device, storage medium, and log data collection system | |
| US8375200B2 (en) | Embedded device and file change notification method of the embedded device | |
| CN109245908B (en) | Method and device for switching master cluster and slave cluster | |
| CN110334072A (en) | A kind of distributed file system, file updating method and device | |
| US20170033979A1 (en) | Restoration of consistent regions within a streaming environment | |
| US11277316B2 (en) | Testing and maintaining outage resilience of server resources | |
| CN112579288A (en) | Cloud computing-based intelligent security data management system | |
| US10331484B2 (en) | Distributed data platform resource allocator | |
| Wladdimiro et al. | Disaster management platform to support real-time analytics | |
| CN113971098A (en) | RabbitMQ consumption management method and system | |
| CN113660136A (en) | Method, device, equipment and storage medium for checking availability of ports of two-way switch | |
| Imran et al. | Cloud-niagara: A high availability and low overhead fault tolerance middleware for the cloud | |
| CN110807058B (en) | Method and system for exporting data | |
| CN116260703A (en) | Distributed message service node CPU performance fault self-recovery method and device | |
| CN118277060A (en) | Distributed high-availability rainfall forecast data analysis method, system, terminal and medium | |
| Meza et al. | Defcon: Preventing Overload with Graceful Feature Degradation | |
| CN114756301B (en) | Log processing method, device and system | |
| CN111124669A (en) | Operation method, system, terminal and storage medium of distributed SaaS software | |
| CN115514618A (en) | Alarm event processing method and device, electronic equipment and medium | |
| CN113032000A (en) | Intelligent operation data management device and method and computer system | |
| CN114201508A (en) | Data processing method, data processing apparatus, electronic device, and storage medium | |
| CN114070889A (en) | Configuration method, traffic forwarding method, device, storage medium, and program product | |
| CN110493071B (en) | Message system resource balancing device, method and equipment | |
| CN111722986A (en) | Software performance monitoring method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |