CN108804711B - Data processing method and device and computer readable storage medium - Google Patents
Data processing method and device and computer readable storage medium Download PDFInfo
- Publication number
- CN108804711B CN108804711B CN201810678182.6A CN201810678182A CN108804711B CN 108804711 B CN108804711 B CN 108804711B CN 201810678182 A CN201810678182 A CN 201810678182A CN 108804711 B CN108804711 B CN 108804711B
- Authority
- CN
- China
- Prior art keywords
- data
- instruction
- server
- client
- program
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
Abstract
The embodiment of the invention discloses a data processing method, a data processing device and a computer readable storage medium, wherein a corresponding client program and a corresponding server program are set for each node, and the client program can provide a service interface for a user so that the user can input corresponding instructions according to requirements. The server program can realize the query and reading of data, and the client program can interact with the server programs of different nodes. The client program can send a command corresponding to the instruction to the corresponding server according to the received instruction, so that a response result fed back by the server is obtained, and a corresponding script program is called to process the data according to the data carried in the response result. Compare and handle data according to in the manual work completely among the prior art, at this in-process, the user only need input the instruction alright in order to realize the automated processing of data, promoted the efficiency of data processing in the kvstore to the effectual accuracy of guaranteeing data processing.
Description
Technical Field
The present invention relates to the field of distributed file system technologies, and in particular, to a method and an apparatus for data processing, and a computer-readable storage medium.
Background
The management module, namely the Mon module in the distributed file system plays the role of a manager, maintains the state of the whole cluster, ensures that related components of the cluster can reach the same consistency at the same time, and is equivalent to a leader layer in the cluster. When the Mon module manages each node, the Mon module stores data information of each node in the form of a graph, i.e., a map table, where the type of the map may include osdmap, monmap, mdsmap, authmap, logmap, and the like, and the data information is stored in a Key-Value Store (kvstore).
In the distributed file system, each node has a corresponding kvstore, and when a manager needs to acquire data in the kvstore, the manager needs to first fetch the data by using a data reading tool (kvstore-tool). Since the data in the kvstore is serialized data, after the data is taken out, the manager needs to perform deserialization processing on the serialized data by using another decoder tool, so that the data which is convenient for the manager to recognize can be obtained.
The data in each kvstore should remain consistent in the distributed file system. In order to verify whether the data in each kvstore are consistent, the data of the corresponding node needs to be acquired according to the data query mode, and whether the data on each node are consistent is determined through a manual comparison mode. In the prior art, no matter data is queried or the consistency of the data on different nodes is checked, manual operation is needed, and the processing efficiency is low.
Therefore, how to improve the efficiency of data processing in kvstore is an urgent problem to be solved by those skilled in the art.
Disclosure of Invention
Embodiments of the present invention provide a data processing method, an apparatus, and a computer-readable storage medium, which can improve the efficiency of data processing in a kvstore.
In order to solve the above technical problem, an embodiment of the present invention provides a data processing method, where a corresponding client program and a server program are set for each node in a distributed file system, and the method includes:
the client sends a command corresponding to the instruction to a corresponding server according to the received instruction;
the client receives a response result fed back by the server;
and the client calls a corresponding script program to process the data according to the data carried in the response result so as to realize the automatic processing of the data.
Optionally, the sending, by the client, a command corresponding to the instruction to the corresponding server according to the received instruction includes:
the client receives the instruction;
the client inquires the corresponding relation between the host name and the IP address according to the host name carried in the command so as to acquire the IP address corresponding to the host;
and the client establishes connection with the corresponding server according to the IP address and sends a command corresponding to the instruction to the server.
Optionally, the instruction includes an information comparison instruction and a data query instruction; the information comparison instruction carries a comparison identifier and host names of a plurality of nodes; the data query instruction carries a host name of at least one node.
Optionally, when the instruction is an information comparison instruction, the command corresponding to the instruction is a command for acquiring all data information in the server;
correspondingly, the step of calling a corresponding script program to process the data by the client according to the data carried in the response result so as to realize the automatic processing of the data comprises the following steps:
the client receives a mapping table fed back by each server; the mapping table stores the corresponding relation between the data type and the data version;
the client compares all the mapping tables to obtain the unique data types and data versions of the mapping tables;
the client receives the key value table fed back by each server, calls an deserialization script to perform deserialization processing on target data in the key value table, and stores the deserialized key value table; the key value table stores the corresponding relationship between each data version and the corresponding target data.
The embodiment of the invention also provides a data processing device, which is used for setting a corresponding client program and a server program for each node in the distributed file system and comprises a sending unit, a receiving unit and a processing unit;
the sending unit is used for sending a command corresponding to the instruction to the corresponding server according to the received instruction;
the receiving unit is configured to receive a response result fed back by the server;
and the processing unit is used for calling a corresponding script program to process the data according to the data carried in the response result so as to realize the automatic processing of the data.
Optionally, the sending unit includes a receiving subunit, an inquiring subunit, and an establishing subunit;
the receiving subunit is configured to receive the instruction;
the inquiry subunit is configured to inquire, according to the host name carried in the instruction, a correspondence between the host name and the IP address, so as to obtain an IP address corresponding to the host;
and the establishing subunit is used for establishing connection with a corresponding server according to the IP address and sending a command corresponding to the instruction to the server.
Optionally, the instruction includes an information comparison instruction and a data query instruction; the information comparison instruction carries a comparison identifier and host names of a plurality of nodes; the data query instruction carries the host name of at least one node.
Optionally, when the instruction is an information contrast instruction,
correspondingly, the processing unit comprises a receiving subunit, a comparing subunit and a storing subunit;
the receiving subunit is configured to receive a mapping table fed back by each server; the mapping table stores the corresponding relation between the data type and the data version;
the comparison subunit is configured to compare all the mapping tables to obtain a data type and a data version that are unique to each mapping table;
and the storage subunit is used for receiving the key value table fed back by each server, calling an deserializing script to perform deserialization processing on data in the key value table, and storing the deserialized key value table.
An embodiment of the present invention further provides a data processing apparatus, including:
a memory for storing a computer program;
a processor for executing the computer program to implement the steps of the method of data processing as described above.
An embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the computer program implements the steps of the data processing method described above.
According to the technical scheme, the corresponding client program and the corresponding server program are set for each node, and the client program can provide a service interface for a user so that the user can input corresponding instructions according to requirements. The server program can realize the query and reading of data, and the client program can interact with the server programs of different nodes. The client program can send a command corresponding to the instruction to the corresponding server according to the received instruction, so that a response result fed back by the server is obtained, and a corresponding script program is called to process the data according to the data carried in the response result, so that the automatic processing of the data is realized. Compare and handle data in prior art according to the manual work completely, at this in-process, the user only needs the input instruction, alright in order to realize the automated processing of data, has promoted the efficiency of data processing in the kvstore to the effectual accuracy of guaranteeing data processing.
Drawings
In order to illustrate the embodiments of the present invention more clearly, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings can be obtained by those skilled in the art without inventive effort.
Fig. 1 is a flowchart of a data processing method according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention;
fig. 3 is a schematic hardware structure diagram of a data processing apparatus according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present invention without any creative work belong to the protection scope of the present invention.
In order that those skilled in the art will better understand the disclosure, the invention will be described in further detail with reference to the accompanying drawings and specific embodiments.
Next, a method for processing data provided by the embodiment of the present invention is described in detail. Fig. 1 is a flowchart of a data processing method provided in an embodiment of the present invention, where a corresponding client program and a server program are set for each node in a distributed file system, and the method includes:
s101: and the client sends a command corresponding to the instruction to the corresponding server according to the received instruction.
In the embodiment of the invention, in order to realize automatic and convenient processing of data, a corresponding client program and a server program are set for each node. The client program and the server program may adopt Python language.
In practical application, the server program of each node is equivalent to a background program and needs to be run in the background all the time. The client program is started when the client program needs to run the instruction, that is, the client program is started only after receiving the instruction. For convenience of introduction, the client program may be referred to as a client for short, and the server program may be referred to as a server for short.
The client is used for receiving instructions, establishing connection with the server, analyzing and processing received data and the like. And the server is used for reading the data in the kvstore of the node of the server and feeding the data back to the client.
The data in each node kvstore may be stored in the form of a prefix (prefix) -key (key) -key value (value), where the prefix specifically may include data types such as osdmap, trimap, mdsmap, authmap, logmap, etc., the key may be a data version corresponding to each data type, and the value may be specific information corresponding to the corresponding key.
For example, osdmap is 128, osdmap is prefix,128 is key, i.e., osdmap has a version number of 128. And the corresponding value is the specific information of osdmap version 128.
In practical application, a user may input a corresponding instruction on a relevant interface of any node, for example, on the node 1, at this time, a client on the node 1 is started, and the client may query a corresponding relationship between a host name and an IP address according to the host name carried in the instruction to obtain an IP address corresponding to the host, thereby establishing a connection with a corresponding server according to the IP address, and sending the instruction corresponding to the instruction to the server.
The host name is used to represent the name of the node. In the embodiment of the invention, each node has a unique host name, and different nodes can be distinguished according to the host name.
Each server has an IP address corresponding to it. The client can establish communication connection with the server pointed by the IP address according to the IP address.
S102: and the client receives the response result fed back by the server.
After receiving the command sent by the client, the server can read the data corresponding to the command in the kvstore of the node of the server, and feed back the data as a response result to the client.
S103: and the client calls a corresponding script program to process the data according to the data carried in the response result so as to realize the automatic processing of the data.
The server connected to the client and the command sent to the server are different according to different instructions input by the user, correspondingly, data carried in response results fed back by the server are different, and the processing modes of the client are different for different data.
The flow of data processing in a particular form will now be described by way of example.
The user can input different forms of instructions on the node for different operation requirements. In the embodiment of the invention, the instruction can be divided into two types, wherein one type is an information comparison instruction, and the other type is a data query instruction.
The information comparison instruction is used for verifying the consistency of data in the kvstore of different nodes. The information comparison instruction may carry a comparison identifier and host names of a plurality of nodes.
The data query instruction is used for querying the data stored in the kvstore of a certain node or certain nodes so as to obtain the required data. The host name of at least one node may be carried in the data query.
In order to ensure query regularity and avoid confusion of query information, a node is generally used as a processing unit of query, that is, a host name of a node is carried in a data query instruction.
When the instruction is an information comparison instruction, the instruction corresponding to the instruction is an instruction for acquiring all data information in the server. Specifically, the data carried in the response result fed back by the server may include a mapping table and a key value table. The mapping table stores the corresponding relation between the data type and the data version; the key value table stores the corresponding relationship between each data version and the corresponding target data. The target data is the specific data information corresponding to the data version.
Correspondingly, the client can receive the mapping tables fed back by each server and compare all the mapping tables to obtain the unique data types and data versions of the mapping tables. The client receives the key value table fed back by each server, calls the deserialization script to deserialize the target data in the key value table, and stores the deserialized key value table.
For example, assuming that the user needs to compare the data in kvstore of node 1 and node 2, the specific process flow is as follows,
(1) The user may run the instruction "store-tool-client autocontrorast host1 host2" on any one node, e.g., node 1, to start the client on node 1.
Wherein, the "store-tool-client" represents a client program, the autocontrost is a command parameter, the host1 is the host name of the node 1, and the host2 is the host name of the node 2.
Because data in kvstore of two nodes needs to be acquired, in a specific implementation, a client may first establish a connection with a server of one of the nodes to acquire the data in the kvstore of the node. And then establishing connection with a server of another node to acquire data in the kvstore of the node. Taking the example of first establishing a connection with the server of the node 1, the client may enter (2).
(2) And the client acquires the IP address of the node 1 according to the corresponding relation between the host name and the IP address. According to the IP address, a connection can be established with the server on the node 1, and at this time, the client can send a command to query all prefixes to the server.
(3) After receiving the command, the server of the node 1 first lists prefixes in the local kvstore, and then queries keys corresponding to different prefixes one by one.
A node's kvstore may include a plurality of prefixes, each prefix having its corresponding key and value corresponding to the key.
The server side can firstly send a mapping table of prefix-key to the client side, then pack the query results of different prefixes into network messages, and send the network messages to the client side one by one.
The query result may be a key-value table, which is a key-value table obtained by the server. Each prefix has its corresponding key-value table.
(4) After receiving the mapping table of prefix-key sent back by the server of the node 1, the client stores the mapping table. And then all the packed key-value tables of different prefixes sent back by the server side are processed.
The key-value table fed back by the server is serialized data. The client can call an deserializing script program to deserialize the data, and then the deserialized data is stored. Specifically, the file name of the data storage may be in a format of hostname + prefix.
(5) The client can perform corresponding processing on the node 2 according to the processing flows of the steps (2), (3) and (4).
(6) The client compares the mapping tables of the prefix-key of the node 1 and the node 2 and outputs the prefix and the key unique to the two nodes.
In a specific implementation, the output result after comparison may be stored or displayed, and a user may directly know whether the data in the kvstore of the node 1 and the node 2 are consistent according to the output result.
When node 1 and node 2 have no unique prefix and key, it means that the data in kvstore of these two nodes are consistent, and the synchronization of data is realized.
When unique prefix and key exist in the node 1 or the opposite node 2, the situation that the data in the kvstore of the two nodes are inconsistent is shown, and the synchronization of the data is not realized.
Compared with the steps of manual query, manual comparison and the like in the prior art, in the embodiment of the invention, the user can obtain the corresponding output result only by inputting one instruction. The operation flow of the user is simplified, and the data processing efficiency is improved.
When the instruction is a data query instruction, the data query instruction may include 4 types of instructions according to the difference of data information that a user needs to obtain in actual operation: list, dump, dumpAll, and get.
Wherein, list represents all prefixes and keys in kvstore of the query node.
Taking the client on the initiating node 1 as an example, the list instruction is implemented as follows,
(1) The user can run the instruction "tore-tool-client list host1" on host1 to start the client.
(2) And the client acquires the IP address of the node 1 according to the corresponding relation between the host name and the IP address. According to the IP address, connection can be established with the server on the node 1, and at this time, the client can send a list command to the server.
(3) After receiving the list command, the service end of host1 inquires all prefixes of the local kvstore and sends the prefix-key mapping table to the client.
(4) The client may output the received prefix-key mapping table to a screen for easy viewing by a user.
dump represents all key-values of a certain prefix in the kvstore of the query node.
Taking the client on the initiating node 1 as an example, the dump instruction is implemented as follows,
(1) The instruction "store-tool-client dump host1 prefix" is run on host1 to start the client.
The prefix may be a specific data type, for example, the specific instruction of osdmap querying host1 is: store-tool-client dump host1 osdmap.
(2) And the client acquires the IP address of the node 1 according to the corresponding relation between the host name and the IP address. According to the IP address, a connection can be established with the server on the node 1, and at this time, the client can send a dump command to the server.
(3) And after receiving the dump command, the server queries all keys and values of the specified prefix of the local kvstore, packages the query result into a network message and sends the network message to the client.
(4) After receiving the message fed back by the server, the client can call a deserialization script program to deserialize the data in the message, and then store the deserialized data.
dumpul represents all prefixes in the kvstore of the query node, and all their corresponding key-values.
Taking the client on the boot-strap node 1 as an example, the dumplall instruction is implemented as follows,
(1) The instruction "store-tool-client dumpALL host1" is run on host1 to start the client.
(2) And the client acquires the IP address of the node 1 according to the corresponding relation between the host name and the IP address. Based on the IP address, a connection may be established with the server on node 1, and at this time, the client may send a dumplall command to the server.
(3) After receiving the dumpALL command, the server side inquires all prefixes in the local kvstore and the key-value corresponding to each prefix, packs the key-values of different prefixes into network messages and sends the network messages to the client side one by one.
(4) After receiving the message fed back by the server, the client can call a deserialization script program to deserialize the data in the message, and then store the deserialized data.
get represents the specified key in the specified prefix in the kvstore of the query node.
Taking the client on the initiating node 1 as an example, the get instruction is implemented as follows,
(1) The instruction "store-tool-client get host1 prefix key out" is run on host1 to start the client.
(2) And the client acquires the IP address of the node 1 according to the corresponding relation between the host name and the IP address. According to the IP address, connection can be established with a server on the node 1, and at this time, the client can send a get command to the server.
(3) And after receiving the get command, the server inquires a key corresponding to the prefix in the local kvstore, and sends the value of the key to the client.
(4) And calling a deserialization script program by the client to perform deserialization processing on the received value, and then outputting the deserialized value to a screen so as to be convenient for a user to view.
It should be noted that, in addition to implementing information interaction with the server of the node itself, the client on one node may also implement information interaction with the servers on other nodes, thereby implementing a function of cross-node query.
For example, after a user inputs a corresponding instruction on a related interface of the node 1, the client on the node 1 is started, and assuming that the host name of the node 2 is carried in the instruction, the client may obtain the IP address corresponding to the host name according to a corresponding relationship between the host name and the IP address, where the IP address points to the server on the node 2, and at this time, the client on the node 1 may establish a communication connection with the server on the node 2. The server on the node 2 can read the data in the kvstore of the node 2 and feed back the read data to the client. Therefore, a user can acquire data in the node 2 by inputting an instruction on the node 1, and cross-node query is realized.
According to the technical scheme, the corresponding client program and the corresponding server program are set for each node, and the client program can provide a service interface for a user so that the user can input corresponding instructions according to requirements. The server program can realize the query and reading of data, and the client program can interact with the server programs of different nodes. The client program can send a command corresponding to the instruction to the corresponding server according to the received instruction, so as to obtain a response result fed back by the server, and call the corresponding script program to process the data according to the data carried in the response result, so as to realize the automatic processing of the data. Compare and handle data according to in the manual work completely among the prior art, at this in-process, the user only needs input instruction, alright in order to realize the automated processing of data, has promoted the efficiency of data processing among the kvstore to the effectual accuracy of guaranteeing data processing.
Fig. 2 is a schematic structural diagram of a data processing apparatus according to an embodiment of the present invention, where a corresponding client program and a server program are set for each node in a distributed file system, and the apparatus includes a sending unit 21, a receiving unit 22, and a processing unit 23;
a sending unit 21, configured to send a command corresponding to the instruction to a corresponding server according to the received instruction;
a receiving unit 22, configured to receive a response result fed back by the server;
and the processing unit 23 is configured to invoke a corresponding script program to process the data according to the data carried in the response result, so as to implement automatic processing of the data.
Optionally, the sending unit includes a receiving subunit, an inquiring subunit, and an establishing subunit;
a receiving subunit, configured to receive an instruction;
the query subunit is used for querying the corresponding relation between the host name and the IP address according to the host name carried in the instruction so as to acquire the IP address corresponding to the host;
and the establishing subunit is used for establishing connection with the corresponding server according to the IP address and sending a command corresponding to the instruction to the server.
Optionally, the instruction includes an information comparison instruction and a data query instruction; the information comparison instruction carries a comparison identifier and host names of a plurality of nodes; the data query instruction carries the host name of at least one node.
Optionally, when the instruction is an information contrast instruction,
correspondingly, the processing unit comprises a receiving subunit, a comparing subunit and a storing subunit;
the receiving subunit is used for receiving the mapping table fed back by each server; the mapping table stores the corresponding relation between the data type and the data version;
the comparison subunit is used for comparing all the mapping tables to obtain the unique data types and data versions of the mapping tables;
and the storage subunit is used for receiving the key value table fed back by each service end, calling the deserializing script to deserialize the data in the key value table, and storing the deserialized key value table. The description of the features in the embodiment corresponding to fig. 2 may refer to the related description of the embodiment corresponding to fig. 1, and is not repeated here.
According to the technical scheme, the corresponding client program and the corresponding server program are set for each node, and the client program can provide a service interface for a user so that the user can input corresponding instructions according to requirements. The server program can realize the query and reading of data, and the client program can interact with the server programs of different nodes. The client program can send a command corresponding to the instruction to the corresponding server according to the received instruction, so that a response result fed back by the server is obtained, and a corresponding script program is called to process the data according to the data carried in the response result, so that the automatic processing of the data is realized. Compare and handle data according to in the manual work completely among the prior art, at this in-process, the user only needs input instruction, alright in order to realize the automated processing of data, has promoted the efficiency of data processing among the kvstore to the effectual accuracy of guaranteeing data processing.
Fig. 3 is a schematic hardware structure diagram of a data processing apparatus 30 according to an embodiment of the present invention, where the apparatus 30 includes:
a memory 31 for storing a computer program;
a processor 32 for executing a computer program to implement the steps of the method of data processing as described above.
The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method for processing data are implemented.
The above detailed description is provided for a data processing method, apparatus and computer-readable storage medium according to the embodiments of the present invention. The embodiments are described in a progressive manner in the specification, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description. It should be noted that, for those skilled in the art, it is possible to make various improvements and modifications to the present invention without departing from the principle of the present invention, and those improvements and modifications also fall within the scope of the claims of the present invention.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the technical solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
Claims (6)
1. A method for processing data, wherein a corresponding client program and a server program are provided for each node in a distributed file system, the method comprising:
the client sends a command corresponding to the instruction to a corresponding server according to the received instruction;
the client receives a response result fed back by the server;
the client calls a corresponding script program to process the data according to the data carried in the response result so as to realize the automatic processing of the data;
the instruction comprises an information comparison instruction and a data query instruction; the information comparison instruction carries a comparison identifier and host names of a plurality of nodes; the data query instruction carries a host name of at least one node;
when the instruction is an information comparison instruction, the command corresponding to the instruction is a command for acquiring all data information in the server;
correspondingly, the step of calling a corresponding script program to process the data by the client according to the data carried in the response result so as to realize the automatic processing of the data comprises the following steps:
the client receives a mapping table fed back by each server; the mapping table stores the corresponding relation between the data type and the data version;
the client compares all the mapping tables to obtain the unique data types and data versions of the mapping tables;
the client receives the key value table fed back by each server, calls an deserialization script to perform deserialization processing on target data in the key value table, and stores the deserialized key value table; and the key value table stores the corresponding relation between each data version and the corresponding target data.
2. The method of claim 1, wherein the client sending a command corresponding to the instruction to the corresponding server according to the received instruction comprises:
the client receives the instruction;
the client inquires the corresponding relation between the host name and the IP address according to the host name carried in the command so as to acquire the IP address corresponding to the host;
and the client establishes connection with the corresponding server according to the IP address and sends a command corresponding to the instruction to the server.
3. The device for processing the data is characterized in that a corresponding client program and a corresponding server program are set for each node in a distributed file system, and the device comprises a sending unit, a receiving unit and a processing unit;
the sending unit is used for sending a command corresponding to the instruction to the corresponding server according to the received instruction;
the receiving unit is used for receiving a response result fed back by the server;
the processing unit is used for calling a corresponding script program to process the data according to the data carried in the response result so as to realize the automatic processing of the data;
the instruction comprises an information comparison instruction and a data query instruction; the information comparison instruction carries a comparison identifier and host names of a plurality of nodes; the data query instruction carries a host name of at least one node;
when the instruction is an information contrast instruction,
correspondingly, the processing unit comprises a receiving subunit, a comparing subunit and a saving subunit;
the receiving subunit is configured to receive mapping tables fed back by the servers; the mapping table stores the corresponding relation between the data type and the data version;
the comparison subunit is configured to compare all the mapping tables to obtain a data type and a data version that are unique to each mapping table;
and the storage subunit is used for receiving the key value table fed back by each server, calling an deserializing script to perform deserialization processing on data in the key value table, and storing the deserialized key value table.
4. The apparatus of claim 3, wherein the sending unit comprises a receiving subunit, an inquiring subunit, and an establishing subunit;
the receiving subunit is configured to receive the instruction;
the inquiry subunit is used for inquiring the corresponding relation between the host name and the IP address according to the host name carried in the instruction so as to acquire the IP address corresponding to the host;
and the establishing subunit is used for establishing connection with a corresponding server according to the IP address and sending a command corresponding to the instruction to the server.
5. An apparatus for data processing, comprising:
a memory for storing a computer program;
a processor for executing the computer program to carry out the steps of the method of data processing according to any one of claims 1 to 2.
6. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a processor, carries out the steps of the method of data processing according to any one of claims 1 to 2.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810678182.6A CN108804711B (en) | 2018-06-27 | 2018-06-27 | Data processing method and device and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810678182.6A CN108804711B (en) | 2018-06-27 | 2018-06-27 | Data processing method and device and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108804711A CN108804711A (en) | 2018-11-13 |
| CN108804711B true CN108804711B (en) | 2022-12-06 |
Family
ID=64071206
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810678182.6A Active CN108804711B (en) | 2018-06-27 | 2018-06-27 | Data processing method and device and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108804711B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109739828B (en) * | 2018-12-29 | 2021-06-29 | 咪咕文化科技有限公司 | Data processing method and device and computer readable storage medium |
| CN110718218B (en) * | 2019-09-12 | 2022-08-23 | 百度在线网络技术（北京）有限公司 | Voice processing method, device, equipment and computer storage medium |
| CN110718219B (en) * | 2019-09-12 | 2022-07-22 | 百度在线网络技术（北京）有限公司 | Voice processing method, device, equipment and computer storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104333578A (en) * | 2014-10-23 | 2015-02-04 | 秦科 | Distributed data exchange system and method |
| CN106302607A (en) * | 2015-06-05 | 2017-01-04 | 腾讯科技（深圳）有限公司 | It is applied to block storage system and the method for cloud computing |
| CN107203594A (en) * | 2017-04-28 | 2017-09-26 | 努比亚技术有限公司 | A kind of data processing equipment, method and computer-readable recording medium |
| CN107256269A (en) * | 2013-05-03 | 2017-10-17 | 脸谱公司 | Method, computer-readable non-transitory storage medium and system |
| CN108171473A (en) * | 2017-12-26 | 2018-06-15 | 北京九章云极科技有限公司 | A kind of Data Analysis Services system and data analysis processing method |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103581229B (en) * | 2012-07-26 | 2018-06-15 | 腾讯科技（深圳）有限公司 | Distributed file system, file access method and client |
-
2018
- 2018-06-27 CN CN201810678182.6A patent/CN108804711B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107256269A (en) * | 2013-05-03 | 2017-10-17 | 脸谱公司 | Method, computer-readable non-transitory storage medium and system |
| CN104333578A (en) * | 2014-10-23 | 2015-02-04 | 秦科 | Distributed data exchange system and method |
| CN106302607A (en) * | 2015-06-05 | 2017-01-04 | 腾讯科技（深圳）有限公司 | It is applied to block storage system and the method for cloud computing |
| CN107203594A (en) * | 2017-04-28 | 2017-09-26 | 努比亚技术有限公司 | A kind of data processing equipment, method and computer-readable recording medium |
| CN108171473A (en) * | 2017-12-26 | 2018-06-15 | 北京九章云极科技有限公司 | A kind of Data Analysis Services system and data analysis processing method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108804711A (en) | 2018-11-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108804711B (en) | Data processing method and device and computer readable storage medium | |
| CN110519409B (en) | Domain name dynamic configuration method, device, equipment and storage medium | |
| CN112506617A (en) | Mirror image updating method and device for sidecar container in Kubernetes cluster | |
| CN110460967B (en) | Short message platform test method and device, computer equipment and storage medium | |
| CN106789166B (en) | Method and device for network element batch configuration | |
| CN113014681A (en) | Network card binding method and device of multi-network card server, electronic equipment and storage medium | |
| CN110704376A (en) | Log file saving method and device | |
| CN114185577A (en) | Configuration management system, configuration management method, server, and storage medium | |
| CN113626225B (en) | Program calling method, terminal device and medium | |
| CN113688183A (en) | Multi-type data source extraction method, system, terminal equipment and storage medium | |
| CN113938391A (en) | Network distribution method and system for gateway equipment and electronic equipment | |
| CN110955460B (en) | Service process starting method and device, electronic equipment and storage medium | |
| CN107425994B (en) | Method, terminal and server for realizing remote parameter management | |
| CN109495319B (en) | Method, device and equipment for determining fault information of CDN node | |
| CN113127335A (en) | System testing method and device | |
| CN113660359A (en) | Management method and device for domain name resolution record, storage medium and electronic equipment | |
| CN112131263B (en) | Software package acquisition method, system, device, electronic equipment and storage medium | |
| CN108052380B (en) | Business function docking method and device, computer equipment and storage medium | |
| CN111294377A (en) | Network request sending method of dependency relationship, terminal device and storage medium | |
| CN114328026A (en) | Virtual disk backup method, device, equipment and medium | |
| CN109698808A (en) | A kind of method, equipment and device loading template message | |
| CN109104499B (en) | Session establishing method, device, equipment and storage medium | |
| CN114443603A (en) | Transaction log playback method and device, computer equipment and medium | |
| CN111722883A (en) | Method and device for updating interface address and computer readable storage medium | |
| CN110162250B (en) | Screenshot method, system and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |