CN112988343A - Batch data fragmentation method, system, computer equipment and storage medium - Google Patents
Batch data fragmentation method, system, computer equipment and storage medium Download PDFInfo
- Publication number
- CN112988343A CN112988343A CN202110161596.3A CN202110161596A CN112988343A CN 112988343 A CN112988343 A CN 112988343A CN 202110161596 A CN202110161596 A CN 202110161596A CN 112988343 A CN112988343 A CN 112988343A
- Authority
- CN
- China
- Prior art keywords
- client
- node
- zookeeper
- slicing
- batch data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013467 fragmentation Methods 0.000 title claims abstract description 36
- 238000006062 fragmentation reaction Methods 0.000 title claims abstract description 36
- 238000000034 method Methods 0.000 title claims abstract description 32
- 238000012163 sequencing technique Methods 0.000 claims abstract description 22
- 230000002085 persistent effect Effects 0.000 claims description 31
- 238000012544 monitoring process Methods 0.000 claims description 12
- 238000012545 processing Methods 0.000 abstract description 28
- 238000011161 development Methods 0.000 abstract description 16
- 239000012634 fragment Substances 0.000 abstract description 7
- 238000004891 communication Methods 0.000 description 17
- 230000000694 effects Effects 0.000 description 6
- 238000004364 calculation method Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 230000008859 change Effects 0.000 description 4
- 230000001960 triggered effect Effects 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 238000005034 decoration Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000010267 cellular communication Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
- G06F16/24554—Unary operations; Data partitioning operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/542—Event management; Broadcasting; Multicasting; Notifications
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Multimedia (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a batch processing data fragmentation method, a system, computer equipment and a storage medium, wherein the method comprises the following steps: when a client is started, registering service to a Zookeeper; receiving a broadcast sent by the Zookeeper to the newly registered message; inquiring the updated node list of the Zookeeper; obtaining the current sequencing position of the node corresponding to the client in the node list and the total registration number; and calculating to obtain the slicing coefficient corresponding to the client according to the current sorting position and the total registration number. According to the scheme, dynamic and real-time capacity expansion of the batch processing data fragments can be realized through the Zookeeper, so that the execution efficiency of batch processing tasks can be improved, and development access cost and development difficulty can be reduced.
Description
Technical Field
The present invention relates to the field of data fragmentation technologies, and in particular, to a method, a system, a computer device, and a storage medium for batch processing data fragmentation.
Background
Zookeeper is a distributed, open-source distributed application coordination service, is an open-source implementation of Chubby of Google, and is an important component of Hadoop and Hbase. It is a software that provides a consistent service for distributed applications, and the functions provided include: configuration maintenance, domain name service, distributed synchronization, group service, etc. The Zookeeper aims to package complex and error-prone key services and provide a simple and easy-to-use interface and a system with high performance and stable functions for users.
With the wider application of the existing distributed system and the larger and larger data volume to be processed, the tasks for processing the batch processing in the traditional mode are usually single-machine deployment, so that a bottleneck exists, and the execution efficiency of the batch processing tasks is not improved. Therefore, a batch processing data fragmentation method based on Zookeeper, which can support dynamic and real-time capacity expansion to improve the execution efficiency of batch processing tasks, is needed.
Disclosure of Invention
The invention aims to provide a batch processing data fragmentation method, a system, computer equipment and a storage medium, and the scheme can realize dynamic and real-time capacity expansion of batch processing data fragmentation through Zookeeper, thereby improving the execution efficiency of batch processing tasks and being beneficial to reducing development access cost and development difficulty.
The technical scheme provided by the invention is as follows:
the invention provides a batch data fragmentation method, which comprises the following steps:
when a client is started, registering service to a Zookeeper;
receiving a broadcast sent by the Zookeeper to the newly registered message;
inquiring the updated node list of the Zookeeper;
obtaining the current sequencing position of the node corresponding to the client in the node list and the total registration number;
and calculating to obtain the slicing coefficient corresponding to the client according to the current sorting position and the total registration number.
When a client is started, the SDK registers service to the Zookeeper, the Zookeeper can update a node list according to the new registration service and send broadcast to newly registered messages, the SDK can inquire the node list updated by the Zookeeper after receiving the broadcast, so that the current sequencing position and the total registration number of the node corresponding to the client in the node list are determined, and the slicing coefficient corresponding to the client can be calculated and obtained according to the current sequencing position and the total registration number, so that the effect of data slicing is achieved. According to the scheme, dynamic and real-time capacity expansion of the batch processing data fragments can be realized through the Zookeeper, so that the execution efficiency of batch processing tasks is improved, and the development access cost and the development difficulty are favorably reduced.
Further, registering the service to the Zookeeper when the client is started specifically includes:
initializing a Zookeeper connection;
establishing connection with the Zookeeper through an open source framework;
and monitoring the real-time condition of the self-defined persistent node in the Zookeeper.
Specifically, the open source frame encapsulates a whole set of bottom-layer api connected with the Zookeeper, so that the connection process can be greatly simplified, the stability of a link can be ensured, and a heartbeat detection mechanism is encapsulated inside the open source frame; monitoring a custom persistent node/monitor-folder in the Zookeeper (if the custom persistent node/monitor-folder does not exist, the custom persistent node/monitor-folder is created first), and any change under the monitoring node can be notified to a monitor by utilizing the Watch characteristic in the Zookeeper; creating (EPHEMERAL _ SEQUENTIAL) temporary sequence nodes/monitor-folders/node. If the client disconnects from the Zookeeper, the temporary sequence node is automatically deleted.
When a new adding or deleting operation is carried out under the monitored persistent node/monitor-folder, the monitor is triggered to notify each client (monitor), and the client immediately initiates query of the persistent node/monitor-folder after receiving the monitoring notification to acquire all temporary nodes under the persistent node. The position of the current client can be judged and the slicing coefficient can be calculated through the temporary node name created at the moment.
Further, after the calculating obtains the slicing coefficient corresponding to the client, the method further includes the steps of:
and sending the slicing coefficients to the client, so that the client queries data of different slicing areas in a database through the slicing coefficients.
In addition, the invention also provides another batch processing data fragmentation method, which comprises the following steps:
receiving registration service sent by the SDK when the client is started;
updating nodes in a node list according to the registration service;
after the node is established, broadcasting the newly registered message;
and receiving a node list inquiry request sent by the SDK after receiving the broadcast, so that the SDK calculates and obtains the fragmentation coefficient corresponding to the client according to the current sequencing position and the total registration number of the node corresponding to the client in the node list.
The Zookeeper can update the nodes in the node list by receiving the registration service sent by the SDK when the client is started, and broadcast the newly registered message after the node is established, the SDK can inquire the updated node list of the Zookeeper after receiving the broadcast, thereby determining the current sequencing position and the total registration number of the nodes corresponding to the client in the node list, and according to the current sequencing position and the total registration number, the fragmentation coefficient corresponding to the client can be calculated and obtained, thereby achieving the effect of data fragmentation. According to the scheme, dynamic and real-time capacity expansion of the batch processing data fragments can be realized through the Zookeeper, so that the execution efficiency of batch processing tasks is improved, and the development access cost and the development difficulty are favorably reduced.
Further, the updating the node in the node list according to the registration service specifically includes:
creating a user-defined persistent node;
and creating a temporary sequence node under the user-defined persistent node, wherein the sequence number of the temporary sequence node is automatically generated according to the sequence generated by the node.
Further, still include: deleting the corresponding temporary sequence node after the connection with the client is disconnected;
and reordering the temporary sequence nodes under the user-defined persistent nodes.
Specifically, a custom persistent node/monitor-folder is created in the Zookeeper, and any change under the monitoring node is notified to the listener by using the Watch characteristic in the Zookeeper; creating (EPHEMERAL _ SEQUENTIAL) temporary sequence nodes/monitor-folders/node. If the client disconnects from the Zookeeper, the temporary sequence node is automatically deleted.
When a new adding or deleting operation is carried out under the monitored persistent node/monitor-folder, the monitor is triggered to notify each client (monitor), and the client immediately initiates query of the persistent node/monitor-folder after receiving the monitoring notification to acquire all temporary nodes under the persistent node. The position of the current client can be judged and the slicing coefficient can be calculated through the temporary node name created at the moment.
In addition, the present invention also provides a batch data slicing system, comprising:
the registration module is used for registering the service to the Zookeeper when the client is started;
a receiving module, configured to receive a broadcast sent by the Zookeeper to a newly registered message;
the query module is used for querying the updated node list of the Zookeeper to obtain the current sequencing position of the node corresponding to the client in the node list and the total registration number;
and the computing module is used for computing and obtaining the slicing coefficient corresponding to the client according to the current sorting position and the total registration number.
Further, still include: and the sending module is used for sending the slicing coefficients to the client so that the client queries data of different areas in a database through the slicing coefficients.
In addition, the present invention also provides a computer device comprising:
the memory is used for storing the running program;
and the processor is used for executing the running program stored in the memory and realizing the operation executed by the two batch data fragmentation methods.
In addition, the present invention also provides a storage medium, where at least one instruction is stored, where the instruction is loaded and executed by a processor to implement the operations performed by the two batch data fragmentation methods.
According to the batch data fragmentation method, the system, the computer equipment and the storage medium provided by the invention, when a client is started, the client registers service to the Zookeeper through the SDK, the Zookeeper can update the node list according to the new registration service and send broadcast to the newly registered message, the SDK can inquire the updated node list of the Zookeeper after receiving the broadcast, so that the current sequencing position and the total registration number of the node corresponding to the client in the node list are determined, and the fragmentation coefficient corresponding to the client can be calculated and obtained according to the current sequencing position and the total registration number, so that the effect of data fragmentation is achieved. According to the scheme, dynamic and real-time capacity expansion of the batch processing data fragments can be realized through the Zookeeper, so that the execution efficiency of batch processing tasks is improved, and the development access cost and the development difficulty are favorably reduced.
Drawings
The foregoing features, technical features, advantages and embodiments of the present invention will be further explained in the following detailed description of the preferred embodiments, which is to be read in connection with the accompanying drawings.
FIG. 1 is a schematic overall flow diagram of one embodiment of the present invention;
FIG. 2 is a schematic overall flow diagram of another embodiment of the present invention;
FIG. 3 is a schematic diagram of an SDK timing sequence according to an embodiment of the present invention;
FIG. 4 is a schematic view of a Zookeeper node directory according to an embodiment of the present invention;
FIG. 5 is a block diagram of an embodiment of the invention;
FIG. 6 is a schematic illustration of batch data slicing according to an embodiment of the present invention;
fig. 7 is a schematic system structure according to an embodiment of the present invention.
Fig. 8 is a schematic structural diagram of a computer device according to an embodiment of the present invention.
Reference numbers in the figures: 1-a registration module; 2-a receiving module; 3-a query module; 4-a calculation module; 5-a sending module; 100-a memory; 200-a processor; 300-a communication interface; 400-a communication bus; 500-input/output interface.
Detailed Description
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following description will be made with reference to the accompanying drawings. It is obvious that the drawings in the following description are only some examples of the invention, and that for a person skilled in the art, other drawings and embodiments can be derived from them without inventive effort.
For the sake of simplicity, the drawings only schematically show the parts relevant to the present invention, and they do not represent the actual structure as a product. In addition, in order to make the drawings concise and understandable, components having the same structure or function in some of the drawings are only schematically illustrated or only labeled. In this document, "one" means not only "only one" but also a case of "more than one".
Example 1
One embodiment of the present invention, as shown in fig. 1 and 3, provides a batch data slicing method, including the steps of:
s1, when the client is started, registering the service to the Zookeeper.
Preferably, when the client is started, registering the service to the Zookeeper specifically includes:
initializing a Zookeeper connection; establishing connection with the Zookeeper through an open source frame (Curator frame); and monitoring the real-time condition of the self-defined persistent node in the Zookeeper so as to receive the broadcast of the Zookeeper to the newly registered message in time.
Specifically, the open source framework encapsulates a whole set of bottom-layer api connected with the Zookeeper, so that the connection process can be greatly simplified, the stability of a link can be ensured, and a heartbeat detection mechanism is encapsulated inside the open source framework.
And S2, receiving the broadcast sent by the Zookeeper to the newly registered message.
And S3, inquiring the node list updated by the Zookeeper.
S4, obtaining the current sequencing position of the node corresponding to the client in the node list and the total registration number;
and S5, calculating and obtaining the slicing coefficient corresponding to the client according to the current sorting position and the total registration number.
Preferably, after the slicing coefficient corresponding to the client is obtained through calculation, the method further includes the steps of:
and S6, sending the slicing coefficients to the client, so that the client queries data of different areas in the database through the slicing coefficients.
Specifically, a user-defined persistent node/monitor-folder in the Zookeeper is monitored (created if the user-defined persistent node/monitor-folder does not exist), and any change under the monitoring node is notified to a monitor by using the Watch characteristic in the Zookeeper; creating (EPHEMERAL _ SEQUENTIAL) a temporary sequence node/monitor-folder/node. If the client disconnects from the Zookeeper, the temporary sequence node is automatically deleted.
When a new adding or deleting operation is carried out under the monitored persistent node/monitor-folder, the monitor is triggered to notify each client (monitor), and the client immediately initiates query of the persistent node/monitor-folder after receiving the monitoring notification to acquire all temporary nodes under the persistent node. The position of the current client can be judged and the slicing coefficient can be calculated through the temporary node name created at the moment.
When a client is started, registering service to the Zookeeper through an SDK (software development kit), wherein the Zookeeper can update a node list according to new registration service and send broadcast to newly registered messages, and the SDK can query the updated node list of the Zookeeper after receiving the broadcast, so that the current sequencing position and the total registration number of the node corresponding to the client in the node list are determined, and the slicing coefficient corresponding to the client can be calculated and obtained according to the current sequencing position and the total registration number, so that the effect of data slicing is achieved. According to the scheme, dynamic and real-time capacity expansion of the batch processing data fragments can be realized through the Zookeeper, so that the execution efficiency of batch processing tasks is improved, and the development access cost and the development difficulty are favorably reduced.
As shown in fig. 5 and fig. 6, in the present solution, 4 clients register to the Zookeeper, and the parameters of each client can be obtained according to the above method:
client-1, node name (node 0000000001), node total (node total 4), and node location (index 0);
client-2, node name (node 0000000002), node total (node total 4), and node location (index 1);
client-3, node name (node 0000000003), node total (node total 4), and node location (index 2);
client-4, node name (node 0000000004), node total (node total 4), and node location (index 3).
During service processing, each client reads data in the database DB, and different clients determine data to be processed according to respective fragmentation coefficients (indexes) and node total numbers (nodeTotal), so that the purpose of data fragmentation is achieved.
Example 2
In an embodiment of the present invention, as shown in fig. 2 and 3, the present invention further provides another batch data fragmentation method, including the steps of:
and S10, receiving the registration service sent by the SDK when the client is started.
And S11, updating the nodes in the node list according to the registration service.
And S12, after the node is established, broadcasting the newly registered message.
And S13, receiving a node list query request sent by the SDK after receiving the broadcast, and enabling the SDK to calculate and obtain the fragmentation coefficient corresponding to the client according to the current sequencing position and the total registration number of the node corresponding to the client in the node list.
The Zookeeper can update the nodes in the node list by receiving the registration service sent by the SDK when the client is started, and broadcast the newly registered message after the node is established, the SDK can inquire the updated node list of the Zookeeper after receiving the broadcast, thereby determining the current sequencing position and the total registration number of the nodes corresponding to the client in the node list, and according to the current sequencing position and the total registration number, the fragmentation coefficient corresponding to the client can be calculated and obtained, thereby achieving the effect of data fragmentation. According to the scheme, dynamic and real-time capacity expansion of the batch processing data fragments can be realized through the Zookeeper, so that the execution efficiency of batch processing tasks is improved, and the development access cost and the development difficulty are favorably reduced.
Preferably, updating the nodes in the node list according to the registration service specifically includes:
creating a user-defined persistent node; and creating a temporary sequence node under the user-defined persistent node, wherein the sequence number of the temporary sequence node is automatically generated according to the sequence generated by the node.
In addition, the batch processing data fragmentation method of the scheme further comprises the following steps: deleting the corresponding temporary sequence node after the connection with the client is disconnected; and reordering the temporary sequence nodes under the user-defined persistent nodes.
Specifically, a custom persistent node/monitor-folder is created in the Zookeeper, and any change under the monitoring node is notified to the listener by using the Watch characteristic in the Zookeeper; creating (EPHEMERAL _ SEQUENTIAL) a temporary sequence node/monitor-folder/node. If the client disconnects from the Zookeeper, the temporary sequence node is automatically deleted.
When a new adding or deleting operation is carried out under the monitored persistent node/monitor-folder, the monitor is triggered to notify each client (monitor), and the client immediately initiates query of the persistent node/monitor-folder after receiving the monitoring notification to acquire all temporary nodes under the persistent node. The position of the current client can be judged and the slicing coefficient can be calculated through the temporary node name created at the moment.
Example 3
An embodiment of the present invention, as shown in fig. 7, further provides a batch data slicing system, which includes a registration module 1, a receiving module 2, an inquiry module 3, a calculation module 4, and a sending module 5.
The registration module 1 is used for registering service to the Zookeeper when the client is started; the receiving module 2 is used for receiving the broadcast sent by the Zookeeper to the newly registered message; the query module 3 is used for querying the updated node list of the Zookeeper, and obtaining the current sequencing position of the node corresponding to the client in the node list and the total registration number; the calculation module 4 is used for calculating and obtaining the slicing coefficient corresponding to the client according to the current sorting position and the total registration number; the sending module 5 is configured to send the fragmentation coefficient to the client, so that the client queries data of different fragmentation areas in the database through the fragmentation coefficient.
When a client is started, the service is registered to the Zookeeper through the registration module 1, the Zookeeper can update the node list according to the new registration service and send a broadcast to a newly registered message, the SDK can query the updated node list of the Zookeeper through the query module 3 after receiving the broadcast through the receiving module 2, so that the current sequencing position and the total registration number of the node corresponding to the client in the node list are determined, and the calculating module 4 can calculate and obtain the slicing coefficient corresponding to the client according to the current sequencing position and the total registration number, so that the effect of data slicing is achieved. According to the scheme, dynamic and real-time capacity expansion of the batch processing data fragments can be realized through the Zookeeper, so that the execution efficiency of batch processing tasks is improved, and the development access cost and the development difficulty are favorably reduced.
Example 4
In addition, as shown in fig. 8, the present invention further provides a computer device, which includes a memory 100 and a processor 200, where the memory 100 is used to store an execution program, and the processor 200 is used to execute the execution program stored in the memory, so as to implement the operations performed by the batch data fragmentation method according to any one of embodiment 1 and/or embodiment 2.
Specifically, the computer device may further include a communication interface 300, a communication bus 400 and an input/output interface 500, wherein the processor 200, the memory 100, the input/output interface 500 and the communication interface 300 are communicated with each other through the communication bus 400.
A communication bus 400 is a circuit that connects the elements described and enables transmission between these elements. For example, the processor 200 receives commands from other elements through the communication bus 400, decrypts the received commands, and performs calculations or data processing according to the decrypted commands. The memory 100 may include program modules such as a kernel (kernel), middleware (middleware), an Application Programming Interface (API), and applications. The program modules may be comprised of software, firmware or hardware, or at least two of the same. The input/output interface 500 forwards commands or data entered by a user via an input/output device (e.g., sensor, keyboard, touch screen). The communication interface 300 connects the computer device with other network devices, user devices, networks. For example, the communication interface 300 may be connected to a network by wire or wirelessly to connect to external other network devices or user devices. The wireless communication may include at least one of: wireless fidelity (WiFi), Bluetooth (BT), Near Field Communication (NFC), Global Positioning Satellite (GPS) and cellular communications, among others. The wired communication may include at least one of: universal Serial Bus (USB), high-definition multimedia interface (HDMI), asynchronous transfer standard interface (RS-232), and the like. The network may be a telecommunications network and a communications network. The communication network may be a computer network, the internet of things, a telephone network. The computer device may connect to the network through the communication interface 300, and a protocol by which the computer device communicates with other network devices may be supported by at least one of an application, an Application Programming Interface (API), middleware, a kernel, and a communication interface.
Example 5
In addition, the present invention further provides a storage medium, where at least one instruction is stored, and the instruction is loaded and executed by a processor to implement the operation performed by the batch data fragmentation method according to any one of embodiment 1 and/or embodiment 2. For example, the computer readable storage medium may be a read-only memory (ROM), a random-access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like. They may be implemented in program code that is executable by a computing device such that it may be stored in a memory device and executed by the computing device, or they may be separately fabricated into various integrated circuit modules, or multiple modules or steps thereof may be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.
It should be noted that the above embodiments can be freely combined as necessary. The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
Claims (10)
1. A method for batch data fragmentation, comprising the steps of:
when a client is started, registering service to a Zookeeper;
receiving a broadcast sent by the Zookeeper to the newly registered message;
inquiring the updated node list of the Zookeeper;
obtaining the current sequencing position of the node corresponding to the client in the node list and the total registration number;
and calculating to obtain the slicing coefficient corresponding to the client according to the current sorting position and the total registration number.
2. The method according to claim 1, wherein registering a service to a Zookeeper when a client is started specifically comprises:
initializing a Zookeeper connection;
establishing connection with the Zookeeper through an open source framework;
and monitoring the real-time condition of the self-defined persistent node in the Zookeeper.
3. The method for fragmenting batch data according to claim 1, wherein after said calculating obtains the fragmentation coefficient corresponding to said client, further comprising the steps of:
and sending the slicing coefficients to the client, so that the client queries data of different slicing areas in a database through the slicing coefficients.
4. A method for batch data fragmentation, comprising the steps of:
receiving registration service sent by the SDK when the client is started;
updating nodes in a node list according to the registration service;
after the node is established, broadcasting the newly registered message;
and receiving a node list inquiry request sent by the SDK after receiving the broadcast, so that the SDK calculates and obtains the fragmentation coefficient corresponding to the client according to the current sequencing position and the total registration number of the node corresponding to the client in the node list.
5. The method according to claim 4, wherein the updating the nodes in the node list according to the registration service specifically comprises:
creating a user-defined persistent node;
and creating a temporary sequence node under the user-defined persistent node, wherein the sequence number of the temporary sequence node is automatically generated according to the sequence generated by the node.
6. The batch data fragmentation method of claim 5, further comprising: deleting the corresponding temporary sequence node after the connection with the client is disconnected;
and reordering the temporary sequence nodes under the user-defined persistent nodes.
7. A batch data slicing system, comprising:
the registration module is used for registering the service to the Zookeeper when the client is started;
a receiving module, configured to receive a broadcast sent by the Zookeeper to a newly registered message;
the query module is used for querying the updated node list of the Zookeeper to obtain the current sequencing position of the node corresponding to the client in the node list and the total registration number;
and the computing module is used for computing and obtaining the slicing coefficient corresponding to the client according to the current sorting position and the total registration number.
8. The batch data slicing system of claim 7, further comprising: and the sending module is used for sending the slicing coefficients to the client so that the client queries data of different areas in a database through the slicing coefficients.
9. A computer device, comprising:
the memory is used for storing the running program;
a processor for executing the running program stored in the memory to implement the operations executed by the batch data slicing method according to any one of claims 1 to 3;
and/or;
operations for implementing the batch data slicing method of any one of claim 4 to claim 6.
10. A storage medium, characterized by: the storage medium has stored therein at least one instruction that is loaded and executed by a processor to perform an operation performed by the batch data fragmentation method of any of claims 1 to 3;
and/or;
operations for implementing the batch data slicing method of any one of claim 4 to claim 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110161596.3A CN112988343A (en) | 2021-02-05 | 2021-02-05 | Batch data fragmentation method, system, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110161596.3A CN112988343A (en) | 2021-02-05 | 2021-02-05 | Batch data fragmentation method, system, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112988343A true CN112988343A (en) | 2021-06-18 |
Family
ID=76348130
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110161596.3A Pending CN112988343A (en) | 2021-02-05 | 2021-02-05 | Batch data fragmentation method, system, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112988343A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113722099A (en) * | 2021-09-01 | 2021-11-30 | 烽火通信科技股份有限公司 | Ten million-level data fragment processing method and device |
| CN115033562A (en) * | 2022-05-10 | 2022-09-09 | 兴业银行股份有限公司 | Dynamic data fragment processing method, system, medium and device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109144731A (en) * | 2018-08-31 | 2019-01-04 | 中国平安人寿保险股份有限公司 | Data processing method, device, computer equipment and storage medium |
| CN110399394A (en) * | 2018-04-16 | 2019-11-01 | 北京京东尚科信息技术有限公司 | The data traversal method, apparatus and calculate node of calculate node |
| WO2020024405A1 (en) * | 2018-08-03 | 2020-02-06 | 平安科技（深圳）有限公司 | Test method, device, server and storage medium based on distributed coordination |
| CN111381972A (en) * | 2018-12-27 | 2020-07-07 | 北京奇虎科技有限公司 | Distributed task scheduling method, device and system |
-
2021
- 2021-02-05 CN CN202110161596.3A patent/CN112988343A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110399394A (en) * | 2018-04-16 | 2019-11-01 | 北京京东尚科信息技术有限公司 | The data traversal method, apparatus and calculate node of calculate node |
| WO2020024405A1 (en) * | 2018-08-03 | 2020-02-06 | 平安科技（深圳）有限公司 | Test method, device, server and storage medium based on distributed coordination |
| CN109144731A (en) * | 2018-08-31 | 2019-01-04 | 中国平安人寿保险股份有限公司 | Data processing method, device, computer equipment and storage medium |
| CN111381972A (en) * | 2018-12-27 | 2020-07-07 | 北京奇虎科技有限公司 | Distributed task scheduling method, device and system |
Non-Patent Citations (1)
| Title |
|---|
| 周晓场: "基于Redis的分布式Key-Value系统的优化研究", 《中国优秀硕士学位论文全文数据库 信息科技辑》, pages 3 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113722099A (en) * | 2021-09-01 | 2021-11-30 | 烽火通信科技股份有限公司 | Ten million-level data fragment processing method and device |
| CN115033562A (en) * | 2022-05-10 | 2022-09-09 | 兴业银行股份有限公司 | Dynamic data fragment processing method, system, medium and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6275837B2 (en) | Method and apparatus for performing subscription notification | |
| US20190357104A1 (en) | Communication system, base station, communication method, and non- transitory computer readable medium storing program | |
| CN110881207B (en) | Network slice selection method and related product | |
| WO2020147030A1 (en) | Cell registration method and terminal device | |
| CN110489474B (en) | Data processing method, device, medium and electronic equipment | |
| US10887751B1 (en) | Systems and methods for facilitating network function device discovery | |
| CN112988343A (en) | Batch data fragmentation method, system, computer equipment and storage medium | |
| CN112134784A (en) | Message display method, device, terminal, server, system and storage medium | |
| CN111835802B (en) | Communication method and device | |
| WO2012019391A1 (en) | System and method for obtaining detail information of number | |
| WO2015168981A1 (en) | Attribute operating method and apparatus | |
| CN106028311B (en) | A kind of terminal registration method and device | |
| CN112235378B (en) | Data updating method and device, storage medium and electronic equipment | |
| CN112995005B (en) | Virtual network data exchange method and device | |
| CN108696855B (en) | Information management method, session management function and mobility management function | |
| CN113553206B (en) | Data event execution method and device, electronic equipment and computer readable medium | |
| CN110661857B (en) | Data synchronization method and device | |
| CN113472565B (en) | Method, apparatus, device and computer readable medium for expanding server function | |
| CN114915617B (en) | HTTP updating method and communication device | |
| CN111314457B (en) | Method and device for setting virtual private cloud | |
| WO2024169556A1 (en) | Network element selection method and apparatus | |
| WO2024146596A1 (en) | Method and apparatus for subscription and notification | |
| CN111866700B (en) | Method, equipment and device for selecting positioning information | |
| WO2020258986A1 (en) | Method and device for obtaining radio capability information, storage medium, and electronic device | |
| WO2022003542A1 (en) | Update service operation for short message service (sms) service |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |