CN113742416A - Data processing method, device, system and storage medium - Google Patents
Data processing method, device, system and storage medium Download PDFInfo
- Publication number
- CN113742416A CN113742416A CN202010477580.9A CN202010477580A CN113742416A CN 113742416 A CN113742416 A CN 113742416A CN 202010477580 A CN202010477580 A CN 202010477580A CN 113742416 A CN113742416 A CN 113742416A
- Authority
- CN
- China
- Prior art keywords
- data
- scheduling
- node
- storage
- server cluster
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2219—Large Object storage; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application provides a data processing method, data processing equipment and a storage medium. In some embodiments of the present application, a scheduling node in a cluster of data scheduling servers is utilized to receive data of a user terminal associated with the scheduling node; updating real-time data in a local database of a scheduling node by using data sent by terminal equipment associated with the scheduling node in the data scheduling server cluster; according to the mapping relation between the storage nodes and the scheduling nodes sent by the data management server, each scheduling node is controlled to send respective historical data to the corresponding storage node of the data storage server cluster for storage, and the scheduling nodes and the storage nodes perform data processing in parallel, so that the data processing efficiency is improved; the scheduling nodes in the data scheduling server cluster store real-time data sent by the user terminal, and the storage nodes in the data storage server cluster store historical data of users, so that access to the data storage server is reduced, and system performance is improved.
Description
Technical Field
The present application relates to the field of data processing technologies, and in particular, to a data processing method, device, system, and storage medium.
Background
The existing data storage system needs to collect and store a large amount of real-time data, a plurality of user terminals are connected to the data storage system, and then summary data and detailed data of the data storage system are transmitted to the data storage system, wherein the summary data is about hundreds of data points, the detailed data is about thousands of data points, the uploading time interval of the summary data points is required to be 10 seconds, the uploading time interval of the detailed data points is required to be 30 seconds, the system needs to store all the data, and user real-time data query and historical data query are supported, so that the system performance requirement is high.
At present, after a data gateway of a data storage system receives various data transmitted by a user terminal, the data gateway immediately executes an insertion operation to a corresponding data table established in advance in a database, reads the latest data in the data table when inquiring real-time data, and filters the data table according to a time field when inquiring historical data to obtain corresponding data. When a large number of user terminals are accessed, the operations of inserting, updating and reading the same database and even the same data table are performed frequently, which causes great burden to the storage system.
Disclosure of Invention
Aspects of the present application provide a data processing method, apparatus, system, and storage medium to provide data processing efficiency of a data storage system.
An embodiment of the present application provides a data processing method, including:
receiving data of the user terminal associated with the scheduling node by using the scheduling node in the data scheduling server cluster according to a mapping relation between the scheduling node and the user terminal sent by the data management server; updating real-time data in a local database of a scheduling node by using data sent by terminal equipment associated with the scheduling node in the data scheduling server cluster; and controlling each scheduling node to send respective historical data to corresponding storage nodes of the data storage server cluster for storage according to the mapping relation between the storage nodes and the scheduling nodes sent by the data management server.
Further, an inquiry node in the data scheduling server cluster receives a real-time data inquiry request sent by an inquiry terminal and sends a real-time data acquisition instruction to a target scheduling node in the data scheduling server cluster; receiving real-time data returned by the target scheduling node; sending the real-time data to the query terminal;
or, the query node in the data scheduling server cluster receives the historical data query request sent by the query terminal, and sends a historical data acquisition instruction to the target storage node in the data storage server cluster, so that the target storage node returns the queried historical data to the query terminal.
Further, the method also comprises the following steps: an inquiry node in a data management server receives a real-time data inquiry request sent by an inquiry terminal and sends a real-time data acquisition instruction to a target scheduling node in a data scheduling server cluster; receiving real-time data returned by the target scheduling node; sending the real-time data to the query terminal;
or, the query node in the data management server receives the historical data query request sent by the query terminal, and sends a historical data acquisition instruction to the target storage node in the data storage server cluster, so that the target storage node returns the queried historical data to the query terminal.
Further, receiving data of the user terminal associated with the scheduling node by using the scheduling node in the data scheduling server cluster includes: sending an access instruction to a user terminal, and receiving data of the user terminal associated with a scheduling node if the user terminal is connected with the scheduling node associated with the user terminal; or monitoring whether the user terminal is accessed, and receiving data of the user terminal associated with the scheduling node if the connection between the user terminal and the scheduling node associated with the user terminal is monitored.
Further, the method also comprises the following steps: and sending the connection state of the user terminal and the scheduling node associated with the user terminal to a data management server.
An embodiment of the present application further provides a data processing method, which is applicable to a data management server, and includes:
receiving fault information sent by a storage node with a fault in a data storage server cluster;
updating the mapping relation between the storage node and the scheduling node according to the fault information;
and sending the updated mapping relation between the storage node and the scheduling node to the changed scheduling node in the data scheduling server cluster, so that the scheduling node in the data scheduling server cluster stores data to other storage nodes according to the updated mapping relation between the storage node and the scheduling node, wherein the other storage nodes are storage nodes except the storage node with the fault in the data storage server cluster.
Further, the method further comprises: receiving a fault recovery message sent by a storage node with a fault; and sending the historical data of the other storage nodes to the storage node with the fault, and updating the mapping relation between the storage node and the scheduling node.
An embodiment of the present application further provides a data processing system, including: the system comprises a data management server, a data scheduling server cluster and a data storage server cluster;
the data management server is used for storing the mapping relation between the scheduling node and the user terminal and the mapping relation between the storage node and the scheduling node;
the data scheduling server cluster comprises a plurality of scheduling nodes, a local database and a plurality of historical data storage nodes, wherein the scheduling nodes are used for receiving data of user terminals associated with the scheduling nodes, updating real-time data in the local database of the scheduling nodes in the data scheduling server cluster, and controlling the scheduling nodes to send respective historical data to corresponding storage nodes of the data storage server cluster for storage according to a mapping relation between the storage nodes and the scheduling nodes sent by the data management server;
the data storage server cluster comprises a plurality of storage nodes and is used for receiving data sent by scheduling nodes of the data scheduling server cluster.
Further, the system further comprises a query node, and the query node is configured to: receiving a real-time data query request sent by a query terminal, and sending a real-time data acquisition instruction to a target scheduling node in a data scheduling server cluster; receiving real-time data returned by the target scheduling node; sending the real-time data to the query terminal; the query node is further configured to: receiving a historical data query request sent by a query terminal, and sending a historical data acquisition instruction to a target storage node in a data storage server cluster; receiving historical data returned by the target storage node; and sending the historical data to the inquiry terminal.
Further, the query node receives a query request sent by a query terminal, the query request includes user terminal information and a data type identifier which need to be queried, whether the query request is a real-time data request or a historical data query request is determined according to the data type identifier, if the query request is the real-time data request, the query node finds a target scheduling node corresponding to the user terminal according to a mapping relation between a scheduling node and the user terminal in the data management server based on the user terminal information, queries the user terminal real-time information from the target scheduling node, and returns the user terminal real-time information; if the data type identification is a historical data query request, the query node finds a target scheduling node corresponding to the user terminal according to the mapping relation between the scheduling node and the user terminal in the data management server based on the user terminal information, finds a corresponding target storage node according to the mapping relation between the storage node and the scheduling node based on the target scheduling node, sends a historical data acquisition instruction to the target storage node in the data storage server cluster, and receives historical data returned by the target storage node; and sending the historical data to the inquiry terminal.
Further, the scheduling node sends an access instruction to the user terminal associated with the scheduling node, and if the user terminal is connected with the associated scheduling node, the scheduling node receives data of the user terminal associated with the scheduling node; or, the scheduling node monitors whether the user terminal associated with the scheduling node is accessed, and if the user terminal is monitored to be accessed to the scheduling node, the scheduling node receives data of the user terminal associated with the scheduling node; and the scheduling node sends the connection state of the user terminal and the scheduling node associated with the user terminal to the data management server.
Further, the data management server receives fault information sent by a storage node with a fault in the data storage server cluster; updating the mapping relation between the storage node and the scheduling node according to the fault information; and sending the updated mapping relation between the storage node and the scheduling node to the changed scheduling node in the data scheduling server cluster, so that the scheduling node in the data scheduling server cluster stores data to other storage nodes according to the updated mapping relation between the storage node and the scheduling node, wherein the other storage nodes are storage nodes except the storage node with the fault in the data storage server cluster.
An embodiment of the present application further provides a master control device, a master control node is deployed on the master control device, the master control device includes: one or more processors and one or more memories storing a computer program, the one or more processors to execute the computer program to:
receiving data of the user terminal associated with the scheduling node by using the scheduling node in the data scheduling server cluster according to a mapping relation between the scheduling node and the user terminal sent by the data management server;
updating real-time data in a local database of a scheduling node by using data sent by terminal equipment associated with the scheduling node in the data scheduling server cluster;
and controlling each scheduling node to send respective historical data to corresponding storage nodes of the data storage server cluster for storage according to the mapping relation between the storage nodes and the scheduling nodes sent by the data management server.
Embodiments of the present application also provide a computer-readable storage medium storing a computer program, which, when executed by one or more processors, causes the one or more processors to perform the steps of the above-mentioned method.
An embodiment of the present application further provides a data management server, including: one or more processors and one or more memories storing a computer program, the one or more processors to execute the computer program to:
receiving fault information sent by a storage node with a fault in a data storage server cluster;
updating the mapping relation between the storage node and the scheduling node according to the fault information;
and sending the updated mapping relation between the storage node and the scheduling node to the changed scheduling node in the data scheduling server cluster, so that the scheduling node in the data scheduling server cluster stores data to other storage nodes according to the updated mapping relation between the storage node and the scheduling node, wherein the other storage nodes are storage nodes except the storage node with the fault in the data storage server cluster.
Embodiments of the present application also provide a computer-readable storage medium storing a computer program, which, when executed by one or more processors, causes the one or more processors to perform the steps of the above-mentioned method.
In some embodiments of the present application, a data scheduling server cluster receives data of a user terminal associated with a scheduling node by using the scheduling node in the data scheduling server cluster according to a mapping relationship between the scheduling node and the user terminal sent by a data management server; updating real-time data in a local database of a scheduling node by using data sent by terminal equipment associated with the scheduling node in the data scheduling server cluster; according to the mapping relation between the storage nodes and the scheduling nodes sent by the data management server, each scheduling node is controlled to send respective historical data to the corresponding storage node of the data storage server cluster for storage, and the scheduling nodes and the storage nodes perform data processing in parallel, so that the data processing efficiency is improved; the scheduling nodes in the data scheduling server cluster store real-time data sent by the user terminal, and the storage nodes in the data storage server cluster store historical data of users, so that access to the data storage server is reduced, and system performance is improved.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the application and not to limit the application. In the drawings:
FIG. 1a is a block diagram of a data processing system according to an exemplary embodiment of the present application;
FIG. 1b is a schematic flow chart illustrating operation of a data processing system according to an exemplary embodiment of the present application;
FIG. 1c is a block diagram of a data processing system according to an exemplary embodiment of the present application;
FIG. 2 is a flowchart of a method of data processing according to an exemplary embodiment of the present application;
FIG. 3 is a method flow diagram of a data processing method exemplary provided by the present application;
fig. 4 is a schematic structural diagram of a master device according to an exemplary embodiment of the present application;
fig. 5 is a schematic structural diagram of a data management server according to an exemplary embodiment of the present application.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the technical solutions of the present application will be described in detail and completely with reference to the following specific embodiments of the present application and the accompanying drawings. It should be apparent that the described embodiments are only some of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
In the prior art, a data distribution server, a data storage server and a data query server are established, data of each terminal device is collected to the data distribution server by using a data gateway, then the data is distributed to the data storage server, and corresponding terminal data is read by the data query server. However, this method is not flexible enough and has poor scalability, which is specifically as follows: 1. the corresponding table relation between the terminal and the gateway is stored in respective servers, and is inconvenient to modify; 2. the data distribution server can only distribute data to the designated data storage server, and when the pressure of the designated data storage server is high, the data storage speed and the response speed are influenced; 3. the system coupling is strong, the data distribution server and the data storage server are in a one-to-one relationship, and when a new data gateway is added, the data distribution server and the data storage server must be correspondingly added, so that the system cost is increased; 4. when the query node reads the real-time data of the terminal, the latest data still needs to be accessed to the data storage server according to time, and the frequency of reading the real-time data is very high, so that the pressure of the data storage server is greatly increased.
In some embodiments of the present application, a data scheduling server cluster receives data of a user terminal associated with a scheduling node by using the scheduling node in the data scheduling server cluster according to a mapping relationship between the scheduling node and the user terminal sent by a data management server; updating real-time data in a local database of a scheduling node by using data sent by terminal equipment associated with the scheduling node in the data scheduling server cluster; according to the mapping relation between the storage nodes and the scheduling nodes sent by the data management server, each scheduling node is controlled to send respective historical data to the corresponding storage node of the data storage server cluster for storage, and the scheduling nodes and the storage nodes perform data processing in parallel, so that the data processing efficiency is improved; the scheduling nodes in the data scheduling server cluster store real-time data sent by the user terminal, and the storage nodes in the data storage server cluster store historical data of users, so that access to the data storage server is reduced, and system performance is improved.
The technical solutions provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings.
Fig. 1a is a schematic diagram of a data processing system according to an exemplary embodiment of the present application. As shown in FIG. 1a, the data processing system includes a cluster of data scheduling servers, a cluster of data management servers, and a cluster of data storage servers. The data scheduling server cluster comprises a plurality of scheduling nodes, and each scheduling node in the data scheduling server cluster receives data uploaded by a user terminal associated with the scheduling node; after each scheduling node receives data uploaded by a user terminal associated with the scheduling node, on one hand, the data uploaded by the user terminal is used for updating the data of a local database to be used as real-time data; and on the other hand, the data uploaded by the user terminal is sent to the corresponding storage node in the data storage server cluster for storage.
In this embodiment, the data management server is a management device of the entire system, and mainly stores system metadata, which includes key information such as a data distribution rule, a scheduling rule, terminal information, and states of storage nodes of the data storage server. The data distribution rule refers to a terminal data storage distribution rule and comprises a mapping relation between storage nodes and scheduling nodes, and the scheduling rule refers to a rule for scheduling data storage when a certain storage node is crashed or has no response, and supports secondary configuration on the certain storage node; the terminal information comprises the serial number of the scheduling node to which the user terminal belongs, namely the mapping relation between the scheduling node and the user terminal, the online state of the user terminal and the terminal data transmission mode (sub-terminal active reporting and passive reporting); the state of each storage node of the data storage server is as follows: on one hand, the storage node actively reports the state to the data management server, and on the other hand, the data scheduling server reports the state to the management server according to the operation result of the data scheduling server on the data management server.
In this embodiment, the data storage server cluster is an actual data storage area of the whole system, and includes a plurality of storage nodes, each of which runs and manages one database instance, receives data from the scheduling nodes in the scheduling server cluster, and executes the decomposed query tasks, and the execution results are returned to the query nodes through the scheduling nodes. The number of storage nodes is limited only by the hard conditions such as ethernet bandwidth and physical conditions of a computer room. The storage nodes only store the data belonging to the corresponding partitions according to the data distribution rule, and are logically equivalent.
In this embodiment, the data scheduling server cluster includes a query node and a plurality of scheduling nodes, where the query node is responsible for processing and decomposing the query request and collecting and returning the result. The scheduling nodes have three functions, namely a data collection processing function which is used as a data input analysis point, and each scheduling node is responsible for data collection and analysis of a user terminal connected with the scheduling node according to terminal information configuration in the data management server; secondly, the terminal real-time data cache area is used for storing the analyzed terminal data in a local database, so that a user can conveniently and quickly inquire the terminal real-time data; and thirdly, a data distribution function, namely distributing data to the storage nodes of the corresponding data storage server cluster according to the data distribution rule of the data management server. When the data scheduling server performs data storage and query access on a plurality of scheduling nodes, concurrent processing can achieve parallel processing of multiple paths of data requests on the plurality of storage nodes, and therefore efficient data storage and access are achieved.
In the embodiment of the system, the query nodes are arranged in the data scheduling server cluster, and in order to ensure that the data import directions are consistent, the uniform data import direction can regard the whole system as a complete logic whole for the outside world.
FIG. 1b is a flowchart illustrating operation of a data processing system according to an exemplary embodiment of the present application. As shown in fig. 1b, the data processing system operates as follows:
firstly, a data management server is started, after the data management server is started, a data distribution rule, a scheduling rule, terminal information configuration and a data storage server state are initialized, and then information interaction of the data scheduling server is responded and the data storage server state information is monitored.
Then, starting the data storage server cluster, starting a state synchronization timer, synchronizing the state of the storage node to the data management server at regular time, and responding to the data distributed by the scheduling server and the decomposed query task.
Then, starting a data scheduling server cluster, starting a query node to respond to the query request, then initializing a real-time data buffer area by the scheduling node, reading terminal information configuration in the data management server, and obtaining corresponding terminal information and a data transmission mode of each terminal: if the data transmission mode of the user terminal is a passive reporting mode, the scheduling node starts an active connection thread and gets contact with the corresponding user terminal; and if the data transmission mode of the user terminal is an active reporting mode, the scheduling node starts a monitoring thread and waits for the access of the user terminal. After user terminal data are imported, updating the running state of the user terminal to a data management server regularly, wherein the running state comprises information such as online or offline of the user terminal, IP addresses and the like, analyzing the data, comparing the analyzed data with the data stored in the buffer area, changing corresponding data in the buffer area if the data are changed, or not performing updating operation, and then sending the current real-time data to a corresponding storage node regularly by a timer for storing according to a data distribution rule (a mapping relation between the storage node and a scheduling node).
Finally, when the query node receives a query request, the query request comprises user terminal information (terminal number and the like) to be queried, data to be queried, queried data type identification (comprising real-time data type identification and historical data type identification) and time for querying the data, whether the query request is a real-time data request or a historical data query request is determined according to the data type identification, if the data type identification is the real-time data request, the query node finds a target scheduling node corresponding to the user terminal according to terminal information configuration (mapping relation between the scheduling node and the user terminal) in the data management server based on the user terminal information, and queries the user terminal real-time information from the target scheduling node and returns the user terminal real-time information; if the data type identification is a historical data query request, decomposing the data type identification into corresponding sub-queries according to a data distribution rule in the data management server, and sending the sub-queries to a corresponding data storage server to obtain historical data of a corresponding time period, namely, a query node finds a target scheduling node corresponding to a user terminal according to the mapping relation between a scheduling node and the user terminal in the data management server based on user terminal information, finds a corresponding target storage node according to the mapping relation between the storage node and the scheduling node based on the target scheduling node, sends a historical data obtaining instruction to the target storage node in the data storage server cluster, and receives the historical data returned by the target storage node; and sending the historical data to the inquiry terminal. Because most of the queries are the queries of the current state and the current data of the user terminal, namely the real-time data queries, the real-time data queries are separated from the historical data queries, the scheduling nodes store the current states and the current data of the corresponding user terminals, and then the target scheduling nodes are subjected to the real-time data queries through the data type identifiers (the default can be the real-time data queries), so that the query efficiency is greatly improved. In addition, when a storage node crashes in the data storage server cluster, the data management server changes the data distribution rule according to the scheduling rule and informs the corresponding scheduling node.
Further, in the above embodiment, the data transmission manner of the ue is divided into active reporting and passive reporting, and for the active reporting situation of the ue, the IP address of the ue is not fixed, and the ue actively connects to its belonging scheduling node and reports data; aiming at the situation that the user terminal reports passively, the IP address of the user terminal is fixed, and the scheduling node can actively connect with the user node according to the IP address. When the scheduling node can not be connected with the corresponding user terminal for a long time, the data management server is informed, and the data management server regularly prompts an administrator.
FIG. 1c is a block diagram of a data processing system according to an exemplary embodiment of the present application. The data processing system of the embodiment of the present application is further described with reference to fig. 1a and 1 b.
In this embodiment, each scheduling node in the cluster of the data scheduling server is connected to a plurality of user terminals through a gateway, the cluster of the data scheduling server includes a plurality of scheduling nodes, and each scheduling node is a small server running a background service program and a local database. The background service program is used for data collection, data distribution, response of real-time data reading and other operations, and the local database is used for storing real-time data of each terminal. When receiving data imported by a user terminal, a scheduling node updates the terminal running state (including information such as online information, offline information and IP addresses) to a data management server, analyzes the data, compares the analyzed data with the data stored in a buffer area, changes the corresponding data in the buffer area and a local database if the data changes, or does not perform updating operation, and then a timer sends the current real-time data to a corresponding storage node for storage according to a data distribution rule. The query node is responsible for interacting with the query terminal of the user, and is essentially a background service for processing and decomposing query requests and result collection and return.
In this embodiment, the data management server is a manager of the entire system, and mainly stores metadata information of the system, and is also a server itself, and runs a background service program and a local database, where key information such as a data distribution rule, a scheduling rule, a terminal information configuration, and states of storage nodes of the data storage server are stored in the local database.
In this embodiment, the data storage server cluster is an actual data storage area of the whole system, and includes a plurality of storage nodes, each of which runs and manages a database instance, receives data from the scheduling node, executes a query task decomposed by the query node, and returns an execution result to the query node.
With reference to fig. 1a, 1b, and 1c, a specific scenario illustrates the work flow of a data processing system:
starting a data management server, a data storage server cluster and a data scheduling server cluster in sequence, and waiting for initialization of each server to be completed;
when a user terminal A (which belongs to a scheduling node B and adopts a data transmission mode of a terminal active reporting mode) is started, the user terminal A is actively connected with the scheduling node B according to a server IP address, after the connection is successful, the scheduling node B starts to collect various data information to the terminal A, updates the state information of the user terminal A to a data management server C, and simultaneously starts a historical data storage timer. Analyzing the reported data, comparing the data in the cache corresponding to the user terminal A, updating the data in the buffer area and the data in the local database, and distributing the data to the corresponding storage node E according to the data distribution rule after the timer is triggered;
a user D checks the real-time information of the user terminal A, an inquiry node inquires the online state of the user terminal A from a data management server C, if the user terminal A is online, a scheduling node B corresponding to the user terminal A is found according to the configuration of the terminal information, and the user D asks for the real-time data of the terminal and returns the real-time data; and the user D checks the historical information of the user terminal A, finds the scheduling node B corresponding to the user terminal A according to the terminal information configuration, finds the storage node E corresponding to the scheduling node B according to the data distribution rule in the data management server C, and asks for the historical information of the user terminal A and returns the historical information.
If the storage node E is paralyzed or the pressure is high, the data management server C compares the states of the storage nodes of the data storage server C, selects the storage node F with low busy degree to temporarily replace the storage node E, informs the scheduling node B to execute the scheduling rule, distributes newly generated historical data of the user terminal A to the storage node F, temporarily changes the data distribution rule, waits for the recovery of the storage node E (the activity state and the busy degree of the storage node B are reported to the data management server regularly after the node is recovered), and reads the historical information of the user terminal A according to the latest data distribution rule (the mapping relation between the updated storage node and the scheduling node) when the user D checks the historical information of the user terminal A. After the storage node E is restored, the data of the user terminal a stored in the storage node F needs to be synchronized to the storage node E, and the data management server C changes the data distribution rule. The storage node E may be automatically restored after the stress is relieved, or may be manually restored, and if there is data loss, the data is restored based on the backup data, and the data backup scheme may have various schemes, which are not described herein again for the prior art. The data processing system in the embodiment of the application is divided into three parts: the data management server cluster, the data management server cluster and the data storage server cluster divide and refine functions of the whole data storage system, separate system management, data acquisition and analysis, data storage and data reading, introduce technologies such as distributed management and scheduling management, realize multi-node scheduling for data acquisition and analysis and data storage, increase stability and expandability of the system and reduce system coupling. The concrete embodiment is as follows: 1. various management contents such as mapping relations, distribution rules, scheduling rules and the like are stored in a data management server which is a control center of the whole system, so that a user can conveniently control the whole system; 2. according to the scheduling rule and the distribution rule, the many-to-many relation between the data scheduling server and the data storage server can be realized, the coupling of the system is reduced, and the stability of the system can be improved; 3. the data scheduling server can store real-time data of the equipment, and the real-time data is separated from historical data, so that the data reading efficiency and the system response speed are improved.
In another data processing system, the difference with the above data processing system is only that the query node is moved to the data management server, and after the query node receives the sent real-time data query request, the real-time data is obtained from the corresponding scheduling node according to the mapping relation between the scheduling node and the user terminal and returned; and when the query node receives the sent historical data query request, decomposing the historical data query request into corresponding sub-queries according to the mapping relation between the storage node and the scheduling node, and sending the sub-queries to the corresponding storage nodes in the corresponding data storage server cluster to obtain the historical data and returning the historical data. For the data storage manner, reference may be made to the description of the foregoing system embodiment, and details are not repeated here.
Fig. 2 is a flowchart of a method of a data processing method according to an exemplary embodiment of the present application, where the data processing method is not limited to the foregoing data processing system, and may also be applied to other data processing systems, as shown in fig. 2, from the perspective of a data scheduling server cluster, the data processing method includes:
s201: receiving data of the user terminal associated with the scheduling node by using the scheduling node in the data scheduling server cluster according to a mapping relation between the scheduling node and the user terminal sent by the data management server;
s202: updating real-time data in a local database of a scheduling node by using data sent by terminal equipment associated with the scheduling node in the data scheduling server cluster;
s203: and controlling each scheduling node to send respective historical data to corresponding storage nodes of the data storage server cluster for storage according to the mapping relation between the storage nodes and the scheduling nodes sent by the data management server.
In this embodiment, the method may be executed on the data processing system in the foregoing system embodiment, and for the storage manner of the user terminal data, reference may be made to the description of the foregoing system embodiment, which is not described herein again.
In this embodiment, the data transmission mode of the ue is divided into active reporting and passive reporting. For the situation of passive reporting of data of the user terminal, an optional implementation manner is that the scheduling node sends an access instruction to the user terminal, the user terminal establishes a connection with the scheduling node after receiving the access instruction, and the scheduling node receives data of the user terminal associated with the scheduling node. For the situation of active reporting by the user terminal, one way to implement is that the scheduling node monitors whether the user terminal is accessed, and if so, receives the data of the user terminal. In addition, the connection state of the user terminal and the scheduling node associated with the user terminal is sent to the data management server.
If the query node is arranged in the data scheduling server cluster, one realizable way for a user to query the real-time data and the historical data is that the query node in the data scheduling server cluster receives a real-time data query request sent by a query terminal and sends a real-time data acquisition instruction to a target scheduling node in the data scheduling server cluster; receiving real-time data returned by a target scheduling node; sending the real-time data to an inquiry terminal; or, the query node in the data scheduling server cluster receives the historical data query request sent by the query terminal, and sends a historical data acquisition instruction to the target storage node in the data storage server cluster, so that the target storage node returns the queried historical data to the query terminal.
If the query node is arranged in the data management server, one realizable way for a user to query the real-time data and the historical data is that the query node in the data management server receives a real-time data query request sent by a query terminal and sends a real-time data acquisition instruction to a target scheduling node in a data scheduling server cluster; receiving real-time data returned by a target scheduling node; sending the real-time data to an inquiry terminal; or, the query node in the data management server receives the historical data query request sent by the query terminal, and sends a historical data acquisition instruction to the target storage node in the data storage server cluster, so that the target storage node returns the queried historical data to the query terminal.
Fig. 3 is a flowchart of a method of a data processing method exemplarily provided in the present application, in view of a situation that a storage node in a data storage server cluster may have a failure. As shown in fig. 3, from the perspective of the data management server, the method includes:
s301: receiving fault information sent by a storage node with a fault in a data storage server cluster;
s302: updating the mapping relation between the storage node and the scheduling node according to the fault information;
s303: and sending the updated mapping relation between the storage node and the scheduling node to the changed scheduling node in the data scheduling server cluster, so that the scheduling node in the data scheduling server cluster stores data to other storage nodes according to the updated mapping relation between the storage node and the scheduling node, wherein the other storage nodes are storage nodes except the storage node with the fault in the data storage server cluster.
In this embodiment, if a storage node in the data storage server cluster fails, failure information is sent to the data management server, and after receiving the failure information, the data management server changes a scheduling rule (including a mapping relationship between the storage node and a scheduling node), and sends the updated mapping relationship between the storage node and the scheduling node to the changed scheduling node in the data scheduling server cluster, so that the scheduling node in the data scheduling server cluster stores data to other storage nodes, and it is ensured that the service is uninterrupted.
After the storage node with the fault is recovered to be normal, the data management server receives a fault recovery message sent by the storage node with the fault; and sending the historical data of other storage nodes to the storage node with the fault, and updating the mapping relation between the storage node and the scheduling node.
Fig. 4 is a schematic structural diagram of a master device according to an exemplary embodiment of the present application. As shown in fig. 4, the master device includes: a memory 401 and a processor 402. In addition, the master device also includes necessary components such as a communication component 403 and a power component 404.
The memory 401 is used to store computer programs and may be configured to store other various data to support operations on the content update apparatus. Examples of such data include instructions for any application or method operating on the master device.
The memory 401 may be implemented by any type or combination of volatile or non-volatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.
A communication component 403 for establishing a communication connection with other devices;
Optionally, the processor 402 may be further configured to:
an inquiry node in a data scheduling server cluster receives a real-time data inquiry request sent by an inquiry terminal and sends a real-time data acquisition instruction to a target scheduling node in the data scheduling server cluster; receiving real-time data returned by a target scheduling node; sending the real-time data to an inquiry terminal;
alternatively, the first and second electrodes may be,
and the query node in the data scheduling server cluster receives the historical data query request sent by the query terminal and sends a historical data acquisition instruction to the target storage node in the data storage server cluster so that the target storage node can return the queried historical data to the query terminal.
Optionally, the processor 402 further includes:
an inquiry node in a data management server receives a real-time data inquiry request sent by an inquiry terminal and sends a real-time data acquisition instruction to a target scheduling node in a data scheduling server cluster; receiving real-time data returned by a target scheduling node; sending the real-time data to an inquiry terminal;
alternatively, the first and second electrodes may be,
a query node in a data management server receives a historical data query request sent by a query terminal and sends a historical data acquisition instruction to a target storage node in a data storage server cluster; receiving historical data returned by the target storage node; and sending the historical data to the inquiry terminal.
Optionally, the processor 402 receives, by using a scheduling node in the data scheduling server cluster, data of a user terminal associated with the scheduling node, and is specifically configured to: sending an access instruction to a user terminal, and receiving data of the user terminal associated with a scheduling node if the user terminal is connected with the scheduling node associated with the user terminal; or monitoring whether the user terminal is accessed, and receiving data of the user terminal associated with the scheduling node if the connection between the user terminal and the scheduling node associated with the user terminal is monitored.
Optionally, the processor 402 may be further configured to: and sending the connection state of the user terminal and the scheduling node associated with the user terminal to a data management server.
Correspondingly, the embodiment of the application also provides a computer readable storage medium storing the computer program. The computer-readable storage medium stores a computer program, and the computer program, when executed by one or more processors, causes the one or more processors to perform the steps in the method embodiment shown in fig. 2.
Fig. 5 is a schematic structural diagram of a data management server according to an exemplary embodiment of the present application. As shown in fig. 5, the data management server includes: a memory 501 and a processor 502. In addition, the data management server includes necessary components such as a communication component 503 and a power component 504.
The memory 501 is used to store computer programs and may be configured to store other various data to support operations on the content update apparatus. Examples of such data include instructions for any application or method operating on the master device.
The memory 501, which may be implemented by any type of volatile or non-volatile memory device or combination thereof, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk.
A communication component 503 for establishing a communication connection with other devices;
the processor 502, which may execute computer instructions stored in the memory 501, is configured to: receiving fault information sent by a storage node with a fault in a data storage server cluster; updating the mapping relation between the storage node and the scheduling node according to the fault information; and sending the updated mapping relation between the storage nodes and the scheduling nodes to the changed scheduling nodes in the data scheduling server cluster so that the scheduling nodes in the data scheduling server cluster can store the data to other storage nodes, wherein the other storage nodes are storage nodes except the storage nodes with faults in the data scheduling server cluster.
Correspondingly, the embodiment of the application also provides a computer readable storage medium storing the computer program. The computer-readable storage medium stores a computer program, and the computer program, when executed by one or more processors, causes the one or more processors to perform the steps in the method embodiment shown in fig. 3.
The communication components of fig. 4 and 5 described above are configured to facilitate wired or wireless communication between the device in which the communication component is located and other devices. The device in which the communication component is located may access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
The power supply components of fig. 4 and 5 described above provide power to the various components of the device in which the power supply components are located. The power components may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device in which the power component is located.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
The above description is only an example of the present application and is not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (10)
1. A data processing method, comprising:
receiving data of the user terminal associated with the scheduling node by using the scheduling node in the data scheduling server cluster according to a mapping relation between the scheduling node and the user terminal sent by the data management server;
updating real-time data in a local database of a scheduling node by using data sent by terminal equipment associated with the scheduling node in the data scheduling server cluster;
and controlling each scheduling node to send respective historical data to corresponding storage nodes of the data storage server cluster for storage according to the mapping relation between the storage nodes and the scheduling nodes sent by the data management server.
2. The method of claim 1, further comprising:
an inquiry node in a data scheduling server cluster receives a real-time data inquiry request sent by an inquiry terminal and sends a real-time data acquisition instruction to a target scheduling node in the data scheduling server cluster; receiving real-time data returned by the target scheduling node; sending the real-time data to the query terminal;
alternatively, the first and second electrodes may be,
and the query node in the data scheduling server cluster receives the historical data query request sent by the query terminal and sends a historical data acquisition instruction to the target storage node in the data storage server cluster so that the target storage node can return the queried historical data to the query terminal.
3. The method of claim 1, further comprising:
an inquiry node in a data management server receives a real-time data inquiry request sent by an inquiry terminal and sends a real-time data acquisition instruction to a target scheduling node in a data scheduling server cluster; receiving real-time data returned by the target scheduling node; sending the real-time data to the query terminal;
alternatively, the first and second electrodes may be,
and the query node in the data management server receives the historical data query request sent by the query terminal and sends a historical data acquisition instruction to the target storage node in the data storage server cluster, so that the target storage node returns the queried historical data to the query terminal.
4. The method of claim 1, wherein receiving data of a user terminal associated with a scheduling node by using the scheduling node in the cluster of data scheduling servers comprises:
sending an access instruction to a user terminal, and receiving data of the user terminal associated with a scheduling node if the user terminal is connected with the scheduling node associated with the user terminal;
or monitoring whether the user terminal is accessed, and receiving data of the user terminal associated with the scheduling node if the connection between the user terminal and the scheduling node associated with the user terminal is monitored.
5. A data processing method is suitable for a data management server, and is characterized by comprising the following steps:
receiving fault information sent by a storage node with a fault in a data storage server cluster;
updating the mapping relation between the storage node and the scheduling node according to the fault information;
and sending the updated mapping relation between the storage node and the scheduling node to the changed scheduling node in the data scheduling server cluster, so that the scheduling node in the data scheduling server cluster stores data to other storage nodes according to the updated mapping relation between the storage node and the scheduling node, wherein the other storage nodes are storage nodes except the storage node with the fault in the data storage server cluster.
6. A data processing system, comprising: the system comprises a data management server, a data scheduling server cluster and a data storage server cluster;
the data management server is used for storing the mapping relation between the scheduling node and the user terminal and the mapping relation between the storage node and the scheduling node;
the data scheduling server cluster comprises a plurality of scheduling nodes, a local database and a plurality of historical data storage nodes, wherein the scheduling nodes are used for receiving data of user terminals associated with the scheduling nodes, updating real-time data in the local database of the scheduling nodes in the data scheduling server cluster, and controlling the scheduling nodes to send respective historical data to corresponding storage nodes of the data storage server cluster for storage according to a mapping relation between the storage nodes and the scheduling nodes sent by the data management server;
the data storage server cluster comprises a plurality of storage nodes and is used for receiving data sent by scheduling nodes of the data scheduling server cluster.
7. The master control device is characterized in that a master control node is deployed on the master control device, and the master control device comprises: one or more processors and one or more memories storing a computer program, the one or more processors to execute the computer program to:
receiving data of the user terminal associated with the scheduling node by using the scheduling node in the data scheduling server cluster according to a mapping relation between the scheduling node and the user terminal sent by the data management server;
updating real-time data in a local database of a scheduling node by using data sent by terminal equipment associated with the scheduling node in the data scheduling server cluster;
and controlling each scheduling node to send respective historical data to corresponding storage nodes of the data storage server cluster for storage according to the mapping relation between the storage nodes and the scheduling nodes sent by the data management server.
8. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by one or more processors, causes the one or more processors to perform the steps of the method of any one of claims 1-5.
9. A data management server, comprising: one or more processors and one or more memories storing a computer program, the one or more processors to execute the computer program to:
receiving fault information sent by a storage node with a fault in a data storage server cluster;
updating the mapping relation between the storage node and the scheduling node according to the fault information;
and sending the updated mapping relation between the storage nodes and the scheduling nodes to the changed scheduling nodes in the data scheduling server cluster so that the scheduling nodes in the data scheduling server cluster can store the data to other storage nodes, wherein the other storage nodes are storage nodes except the storage nodes with faults in the data scheduling server cluster.
10. A computer-readable storage medium storing a computer program, which when executed by one or more processors causes the one or more processors to perform the steps of the method of claim 6 or 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010477580.9A CN113742416A (en) | 2020-05-29 | 2020-05-29 | Data processing method, device, system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010477580.9A CN113742416A (en) | 2020-05-29 | 2020-05-29 | Data processing method, device, system and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113742416A true CN113742416A (en) | 2021-12-03 |
Family
ID=78724815
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010477580.9A Pending CN113742416A (en) | 2020-05-29 | 2020-05-29 | Data processing method, device, system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113742416A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114629783A (en) * | 2022-03-14 | 2022-06-14 | 上海英方软件股份有限公司 | State monitoring method, system, equipment and computer readable storage medium |
-
2020
- 2020-05-29 CN CN202010477580.9A patent/CN113742416A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114629783A (en) * | 2022-03-14 | 2022-06-14 | 上海英方软件股份有限公司 | State monitoring method, system, equipment and computer readable storage medium |
| CN114629783B (en) * | 2022-03-14 | 2024-03-26 | 上海英方软件股份有限公司 | State monitoring method, system, equipment and computer readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20190213175A1 (en) | Data migration method and system | |
| CN106888245B (en) | Data processing method, device and system | |
| CN107368369B (en) | Distributed container management method and system | |
| CN102291416A (en) | Two-way synchronizing method and system of client-side and server-side | |
| CN107404509B (en) | Distributed service configuration system and information management method | |
| CN101242302A (en) | Data synchronization method, device and system | |
| US20160344582A1 (en) | Call home cluster | |
| CN103346910A (en) | Method and device for managing available mode of servers | |
| CN110611603B (en) | Cluster network card monitoring method and device | |
| CN105701099A (en) | Method and device used for executing task in distributed environment, and distributed task execution system | |
| CN112543222A (en) | Data processing method and device, computer equipment and storage medium | |
| CN110213359B (en) | Internet of vehicles networking data pushing system and method based on D2D | |
| CN109560903B (en) | Vehicle-mounted command communication system for complete disaster recovery | |
| CN113742416A (en) | Data processing method, device, system and storage medium | |
| EP3570169B1 (en) | Method and system for processing device failure | |
| CN113301558B (en) | Message transmission method, device, system and storage medium | |
| CN113420029A (en) | Global ID generation method, device, equipment and medium for distributed system | |
| CN111404978B (en) | Data storage method and cloud storage system | |
| CN112214377A (en) | Equipment management method and system | |
| CN116346834A (en) | Session synchronization method, device, computing equipment and computer storage medium | |
| CN113824801B (en) | Intelligent integration terminal unified access management component system | |
| CN105959427B (en) | DNS server automatic expansion method | |
| CN111400110B (en) | Database access management system | |
| US11381642B2 (en) | Distributed storage system suitable for sensor data | |
| CN110099208B (en) | Method and device for cross-level and cross-domain upgrading of camera in video monitoring system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |