CN118069625A - Data processing method, system, electronic equipment and storage medium - Google Patents
Data processing method, system, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN118069625A CN118069625A CN202410227307.9A CN202410227307A CN118069625A CN 118069625 A CN118069625 A CN 118069625A CN 202410227307 A CN202410227307 A CN 202410227307A CN 118069625 A CN118069625 A CN 118069625A
- Authority
- CN
- China
- Prior art keywords
- data
- region server
- data application
- store
- name
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 22
- 230000008859 change Effects 0.000 claims abstract description 42
- 230000000977 initiatory effect Effects 0.000 claims abstract description 16
- 238000000034 method Methods 0.000 claims description 20
- 238000012545 processing Methods 0.000 claims description 10
- 238000004891 communication Methods 0.000 claims description 2
- 230000008569 process Effects 0.000 description 11
- 238000013461 design Methods 0.000 description 10
- 230000003993 interaction Effects 0.000 description 5
- 230000006870 function Effects 0.000 description 3
- 230000002688 persistence Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000006978 adaptation Effects 0.000 description 1
- 230000003139 buffering effect Effects 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a data processing method, a system, electronic equipment and a storage medium, wherein a HMaster address is obtained from a ZooKeeper when a user is detected to initiate a data application; if the data application is a write-in data application, acquiring corresponding Region Server information from the Redis single node/cluster according to HMaster addresses and the write-in data application; establishing connection with a Region Server indicated by the Region Server information, and writing data indicated by the written data application into a first store; the first stoer is stoer corresponding to a store name indicated by the write data application, and is a store in a Region maintained by the Region Server; after the data writing is completed, storing the generated change information to a Redis single node/cluster, wherein the change information comprises metadata with data change; if the data application is a read data application, acquiring corresponding metadata from the Redis single node/cluster according to HMaster addresses and the read data application; the metadata at least comprises a target Region Server name, a Region name and a store name; initiating a read request to a target Region Server corresponding to the target Region Server name, so that the target Region Server acquires corresponding data from the second store according to the read request; the read request is generated according to the read data application, and the second store is a store corresponding to a store name in the Region corresponding to the Region name.
Description
Technical Field
The present invention relates to the field of computers, and more particularly, to a data processing method, system, electronic device, and storage medium.
Background
With the rapid development of computer technology, HBase is widely used in various enterprises. Wherein, HBase is a distributed NoSQL database for storing large-scale semi-structured data and unstructured data.
In the prior art, the HBase uses a Region Server address of a ZooKeeper storage metadata table, and uses the Region Server to store the metadata table, so that when a client performs a data reading and writing request, the ZooKeeper and the Region Server need to interact in sequence, the reading and writing efficiency of metadata is low, and the reading and writing efficiency of the data is low, thereby influencing the performance of the whole HBase cluster; besides, the ZooKeeper and the Region Server need to reserve a memory to store the metadata of the HBase besides storing the corresponding metadata table and the specific information of the metadata, so that the design structure of the ZooKeeper and the Region Server is complex, long in time consumption and high in cost.
Disclosure of Invention
In view of this, the present invention provides a data processing method, system, electronic device and storage medium, so as to improve the read-write efficiency of metadata and data, improve the performance of HBase clusters, simplify the design structure of the ZooKeeper and Region Server, reduce the time consumption and reduce the cost.
The first aspect of the present invention provides a data processing method, applied to a client, the method comprising:
When detecting that a user initiates a data application, acquiring HMaster addresses from a ZooKeeper;
If the data application is a write-in data application, acquiring corresponding Region Server information from the Redis single node/cluster according to the HMaster address and the write-in data application;
Establishing connection with a Region Server indicated by the Region Server information, and writing data indicated by a written data application into a first store; the first stoer is stoer corresponding to a store name indicated by a write data application, and is a store in a Region maintained by the Region Server;
after the data writing is completed, storing the generated change information to a Redis single node/cluster, wherein the change information comprises metadata with data change;
If the data application is a read data application, acquiring corresponding metadata from the Redis single node/cluster according to the HMaster address and the read data application; the metadata at least comprises a target Region Server name, a Region name and a store name;
Initiating a read request to a target Region Server corresponding to the target Region Server name, so that the target Region Server acquires corresponding data from a second store according to the read request; and the read request is generated according to the read data application, and the second store is a store corresponding to the store name in the Region corresponding to the Region name.
Optionally, if the data application is a write data application, according to the HMaster address and the write data application, obtaining the corresponding Region Server information from the Redis single node/cluster includes:
if the data application is a write data application, using the HMaster address call HMaster to access the Redis single node/cluster, and according to the write data application, obtaining the Region Server information corresponding to the Redis single node/cluster.
Optionally, after the data writing is completed, the storing the generated change information to the Redis single node/cluster includes:
After the data writing is completed, generating corresponding change information according to the Region Server name of the Region Server, the Region name of the Region maintained by the Region Server and the store name of the first store;
and storing the change information to the Redis single node/cluster.
Optionally, the read data application at least includes a target metadata table name and a first data reading range;
If the data application is a read data application, according to the HMaster address and the read data application, obtaining corresponding metadata from the Redis single node/cluster includes:
And accessing a Redis single node/cluster by utilizing the HMaster address call HMaster, and acquiring metadata corresponding to the first data reading range from a metadata table corresponding to the target metadata table name stored in the Redis single node/cluster.
Optionally, the initiating a read request to the target Region Server corresponding to the target Region Server name, so that the target Region Server obtains corresponding data from the second store according to the read request, including:
And initiating a read request to a target Region Server corresponding to the target Region Server name, so that the target Region Server reads data corresponding to a second data read range indicated by the read request from a second store.
A second aspect of the present invention provides a data processing system for use with a client, the system comprising:
HMaster address obtaining unit, configured to obtain HMaster address from the ZooKeeper when detecting that the user initiates the data application;
The Region Server information acquisition unit is used for acquiring corresponding Region Server information from the Redis single node/cluster according to the HMaster address and the write-in data application if the data application is the write-in data application;
The data writing unit is used for establishing connection with the Region Server indicated by the Region Server information and writing the data indicated by the written data application into the first store; the first stoer is stoer corresponding to a store name indicated by a write data application, and is a store in a Region maintained by the Region Server;
The storage unit is used for storing the generated change information to the Redis single node/cluster after the data writing is completed, wherein the change information comprises metadata with data change;
The metadata acquisition unit is used for acquiring corresponding metadata from the Redis single node/cluster according to the HMaster address and the read data application if the data application is a read data application; the metadata at least comprises a target Region Server name, a Region name and a store name;
the data reading unit is used for initiating a reading request to a target Region Server corresponding to the target Region Server name, so that the target Region Server acquires corresponding data from a second store according to the reading request; and the read request is generated according to the read data application, and the second store is a store corresponding to the store name in the Region corresponding to the Region name.
Optionally, the Region Server information obtaining unit includes:
And the Region Server information acquisition subunit is used for calling HMaster to access the Redis single node/cluster by using the HMaster address if the data application is a write-in data application, and according to the write-in data application, accessing the Region Server information corresponding to the Redis single node/cluster.
Optionally, the storage unit includes:
The change information generating unit is used for generating corresponding change information according to the Region Server name of the Region Server, the Region name of the Region maintained by the Region Server and the store name of the first store after the data writing is completed;
and the storage subunit is used for storing the change information to the Redis single node/cluster.
A third aspect of the present invention provides an electronic apparatus, comprising: the device comprises a processor and a memory, wherein the processor and the memory are connected through a communication bus; the processor is used for calling and executing the program stored in the memory; the memory is used for storing a program for implementing the data processing method provided by the first aspect of the invention.
A fourth aspect of the present invention provides a computer readable storage medium having stored therein computer executable instructions for performing the data processing method provided in the first aspect of the present invention.
The invention provides a data processing method, a system, electronic equipment and a storage medium, wherein a HMaster address is obtained from a ZooKeeper when a user is detected to initiate a data application; if the data application is a write-in data application, acquiring corresponding Region Server information from the Redis single node/cluster according to HMaster addresses and the write-in data application; establishing connection with a Region Server indicated by the Region Server information, and writing data indicated by the written data application into a first store; after the data writing is completed, storing the generated change information to a Redis single node/cluster; if the data application is a read data application, acquiring corresponding metadata from the Redis single node/cluster according to HMaster addresses and the read data application; and initiating a read request to the target Region Server corresponding to the target Region Server name, so that the target Region Server acquires corresponding data from the second store according to the read request. According to the technical scheme provided by the invention, all metadata of the HBase are stored by using the Redis single node/cluster, the metadata can be directly obtained from the Redis single node/cluster without interaction between a ZooKeeper and a Region Server in sequence, so that the read-write efficiency of the metadata is low, the read-write efficiency of the data is low, and the performance of the whole HBase cluster is improved; and the ZooKeeper and Region Server do not need to reserve a memory to store the metadata of HBase, so that the design structure of the ZooKeeper and Region Server is simplified, the time consumption is reduced, and the cost is reduced.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present invention, and that other drawings can be obtained according to the provided drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a data processing method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a data processing system according to an embodiment of the present invention;
fig. 3 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The term "including" and variations thereof as used herein are intended to be open-ended, i.e., including, but not limited to. The term "based on" is based at least in part on. The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments. Related definitions of other terms will be given in the description below.
It should be noted that the terms "first," "second," and the like in this disclosure are merely used for distinguishing between different devices, modules, or units and not for limiting the order or interdependence of the functions performed by these devices, modules, or units.
It should be noted that references to "one" or "a plurality" in this disclosure are intended to be illustrative rather than limiting, and those of ordinary skill in the art will appreciate that "one or more" is intended to be interpreted as "one or more" unless the context clearly indicates otherwise.
It should be noted that, in the technical solution of the present disclosure, the processes of acquiring, collecting, storing, using, processing, transmitting, providing, disclosing, applying, etc. data all conform to the rules of the related laws and regulations, and necessary security measures are adopted without violating the public order harmony.
For a better understanding of the present invention, technical terms related to the present invention are explained below:
HBase: HBase is a highly reliable, high performance, column-oriented, scalable distributed storage system for storing large amounts of structured or semi-structured, unstructured data.
HMaster: the primary server in the HBase cluster is responsible for monitoring all RegionServer in the cluster and is the interface for all metadata changes.
ZooKeeper: a distributed application coordination service of distributed open source codes is Chubby of Google, and is an important component of Hadoop and Hbase. It is a software providing a consistency service for distributed applications, the provided functions include: configuration maintenance, domain name service, distributed synchronization, group service, etc.
Metadata table of HBase: the metadata table of the HBase stores metadata related to a custom table, including table names, which column clusters the table has, which regions the table has, where each Region is stored, each Region is managed by which Region Server, the table is also stored in a Region, and the metadata table is only managed by one Region Server.
Region Server: region Server is a service that HBase clusters run on each working node. The system is a key point of the whole HBase system, on one hand, the system maintains the state of the Region, and provides management and service for the Region; on the other hand, the system interacts with the Master to participate in the distributed coordination management of the Master.
Region: region is the minimum unit of distributed storage and load balancing in HBase, is the basic element of table acquisition and distribution, and consists of one or more stores per column family.
Store: a Store stores a column family in the data table, consisting of one MemStore and 0 to multiple StoreFile.
StoreFile: files storing ordered KeyValue are stored on HDFS.
MemStore: the write cache, keyValue, orders MemStore, and flushes to StoreFile after the threshold is reached, each flush generating a new StoreFile.
BlockCache read buffering, each time the newly queried data is buffered in BlockCache.
RDB persistence: and writing the data set snapshot in the memory into the disk in a specified time interval, wherein the actual operation process is a subprocess of fork, writing the data set into a temporary file, replacing the previous file after the writing is successful, and storing by binary compression.
AOF persistence: each writing and deleting operation processed by the server is recorded in a log mode, the inquiring operation is not recorded, the writing and deleting operation is recorded in a text mode, and the detailed operation record can be seen by opening the file.
In the prior art, the existing data reading process may specifically be: firstly, a client acquires Region Server node information where a metadata table is located from a ZooKeeper; secondly, the client accesses a Region Server where the metadata table is located according to the Region Server node information to acquire Region Server information where the Region is located; and finally, the client determines the corresponding Region and stoer according to the Region Server information, and initiates a read request to the Region Server so that the Region Server acquires the data to be read from the corresponding store in the corresponding Region.
The existing data writing process specifically can be: the client acquires Region Server node information of the metadata table from the ZooKeeper; secondly, the client accesses a Region Server (Region Server needing to write data) where the metadata table is located according to the Region Server node information, acquires Region Server information where the Region is located, and determines corresponding regions and stoer according to the Region Server information; finally, the client establishes a connection with Region Serve to be written with data, writes HLog the data to be written, and writes MemStore the data to the store module of the corresponding Region.
Therefore, the current data read-write process needs to be interacted by the ZooKeeper and the Region Server in sequence, the read-write efficiency of metadata is low, the read-write efficiency of the data is low, and the performance of the whole HBase cluster is affected; besides, the ZooKeeper and the Region Server need to reserve a memory to store the metadata of the HBase besides storing the corresponding metadata table and the specific information of the metadata, so that the design structure of the ZooKeeper and the Region Server is complex, long in time consumption and high in cost.
Therefore, the invention provides a data processing method, a system, electronic equipment and a storage medium, wherein the metadata of the HBase is stored by using a Redis single node/cluster, the metadata can be directly obtained from the Redis single node/cluster without interaction with a ZooKeeper and a Region Server in sequence, so that the read-write efficiency of the metadata is improved, the read-write efficiency of the data is improved, and the performance of the whole HBase cluster is improved; and the ZooKeeper and the Region Server do not need to reserve a memory to store the metadata of the HBase, so that the design structure of the ZooKeeper and the Region Server is simplified, the interaction time with the ZooKeeper is reduced, and the design cost of the ZooKeeper and the Region Server is reduced.
Referring to fig. 1, a flow chart of a data processing method provided by an embodiment of the present invention is shown, where the data processing method is applied to a client, and the data processing method specifically includes the following steps:
S101: when it is detected that the user initiates a data application, the HMaster address is obtained from the ZooKeeper.
In the embodiment of the invention, when a user wants to perform corresponding data read-write operation, a corresponding data application can be initiated on a corresponding client, so that the client can acquire a corresponding HMaster address from a ZooKeeper when detecting the data application initiated by the user.
It should be noted that the data application may be a write data application or a read data application, and the embodiments of the present invention are not limited herein.
It should be further noted that, if the data application is a write data application, the data application (write data application) may include a Region Server name of a Region Server that requests write data, a Region name of a Region maintained by the Region Server, and a store name that requests write data, which are not limited in this embodiment of the present invention.
If the data application is a read data application, the data application (read data application) may include a target HBase metadata table name, a first data read range, and a second data read range.
In some embodiments, the first data read range may include a data range of data requested to be read, with specific content regarding the first data read range and the second data read range; the second data reading range indicates a data range in the HBase metadata table corresponding to the name of the target HBase metadata table requesting access, which may be set according to practical applications, and the embodiment of the present invention is not limited herein.
S102: judging whether the data application is a written data application or a read data application; if the data application is a write data application, executing step S103; if the data application is a read data application, step S106 is performed.
S103: and acquiring corresponding Region Server information from the Redis single node/cluster according to HMaster addresses and the written data application.
In the embodiment of the invention, each metadata table of the HBase can be stored in the Redis single node/cluster in advance, so that when the data application is determined to be the writing application, each metadata table stored in the Redis single node/cluster in advance is accessed based on the obtained HMaster address range, and corresponding Region Server information is obtained from each metadata table according to the writing data application.
Optionally, according to HMaster addresses and write-in data applications, the process of obtaining the corresponding Region Server information from the Redis single node/cluster may specifically be: and using HMaster address call HMaster to access the Redis single node/cluster, and applying for corresponding Region Server information from the Redis single node/cluster according to the written data.
The Region Server information may include a location IP address of the Region Server that requests write data indicated in the write data application.
In some embodiments, the Redis single node/cluster may be accessed using HMaster address call HMaster to obtain Region Server information corresponding to the Region Server name of the Region Server for which the write data application indicates write data from the respective metadata tables stored in the Redis single node/cluster.
S104: and establishing connection with the Region Server indicated by the Region Server information, and writing the data indicated by the written data application into the first store.
In the embodiment of the invention, after the corresponding Region Server information is acquired, connection can be established according to the position IP address in the Region Server information and the Region Server indicated by the Region Server information, so that the client can start writing data under the condition of being connected with the Region Server indicated by the Region Server information.
Specifically, in the case that the client is connected to the Region Server indicated by the Region Server information, the data requested to be written in the data writing application is written HLog first, then the data is written MemStore in the first store, and when the data writing is completed in the first store and HLog, it can be determined that the data writing of the client is completed.
S105: after the data writing is completed, the generated change information is stored to the Redis single node/cluster, wherein the change information comprises metadata of which the data is changed.
In the embodiment of the invention, after determining that the data writing is completed, the client can further acquire metadata such as the name of the first store where the data change occurs, the name of the Region where the first store belongs to, the name of the Region Server where the Region Server is maintained, and the like, namely, the name of the Region Server indicated by the Region Server information, the name of the Region maintained by the Region Server, and the name of the store of the first store; and generating corresponding change information according to the acquired Region Server name, the acquired Region name and the store name of the first store, and finally storing the generated change information into the Redis single node/cluster.
In sum, the Redis single node/cluster is used as the storage of the HBase metadata, so that the writing flow of the HBase can be optimized, and the performance and reliability of the whole HBase cluster are improved.
In some embodiments, a metadata table corresponding to a Region Server name of a Region Server where the write data application indicates write data may be determined from among the respective metadata tables stored by the Redis single node/cluster, and metadata in the generated change information may be stored into the determined metadata table.
S106: acquiring corresponding metadata from the Redis single node/cluster according to HMaster addresses and the read data application; the metadata at least comprises a target Region Server name, a Region name and a store name.
In the process of specifically executing step S106, in the case where the data application is determined to be the read data application, each metadata table stored in advance in the Redis single node/cluster may be accessed based on the obtained HMaster address, so as to obtain corresponding metadata from each metadata table according to the read data application.
The metadata at least comprises a target Region Server name, a Region name and a store name.
Optionally, according to HMaster addresses and the application for reading data, the process of obtaining the corresponding metadata from the Redis single node/cluster may specifically be: and accessing the Redis single node/cluster by utilizing HMaster address call HMaster, and acquiring metadata corresponding to the first data reading range from a metadata table corresponding to the target metadata table name stored in the Redis single node/cluster.
S107: and initiating a read request to the target Region Server corresponding to the target Region Server name, so that the target Region Server acquires corresponding data from the second store according to the read request.
In the specific execution process of step S107, after obtaining the corresponding metadata, the client may determine, according to the target Region Server name in the metadata, that the read data is in the Region Server; after the target Region Server to be accessed is determined according to the name of the target Region Server, a corresponding read request can be generated according to the second data read range in the read data application so as to initiate the read request to the target Region Server, and after the target Region Server receives the corresponding read request, the target Region Server can start reading data from the second store based on the received read request. The second store is a store corresponding to a store name in the Region corresponding to the Region name.
Optionally, the process of initiating a read request to the target Region Server corresponding to the target Region Server name, so that the target Region Server obtains the corresponding data from the second store according to the read request may specifically be: and initiating a read request to the target Region Server corresponding to the target Region Server name, so that the target Region Server reads data corresponding to a second data read range indicated by the read request from the second store.
In some embodiments, after the client knows that the data to be read is on the target Region Server, the client may initiate a read request to the target Region Server, so that the target Region Server starts reading the data from the second store based on the read request, and the specific data reading sequence may be: memStore- - > BlockCache- - > StoreFile- - > Hfile.
In sum, the Redis single node/cluster is used for storing all metadata of the HBase, the ZooKeeper is only used for high availability of the HBase, the metadata of the HBase is not stored, and the Region Server is not stored, so that the performance and reliability of the HBase can be improved, and the processing mode of the metadata of the HBase can be optimized.
In addition, the ZooKeeper and the Region Server do not need to reserve a memory to store the metadata of the HBase, so that the design structures of the ZooKeeper and the Region Server are simplified, the read-write efficiency of the metadata can be improved to a certain extent, and the read-write efficiency of the data is further improved; the Region Server does not need to reserve a memory to store metadata of the HBase, so that the memory pressure of the Region Server can be reduced, the Region Server has single function, only needs to pay attention to data access, the memory does not become a bottleneck of metadata management, and the transverse expansion can be conveniently carried out.
Furthermore, the embodiment of the invention can also replace the metadata backup mode in the HBase prior art by using two persistence modes of RDB and AOF of the Redis single node/cluster.
The invention provides a data processing method, when detecting that a user initiates a data application, a HMaster address is obtained from a ZooKeeper; if the data application is a write-in data application, acquiring corresponding Region Server information from the Redis single node/cluster according to HMaster addresses and the write-in data application; establishing connection with a Region Server indicated by the Region Server information, and writing data indicated by the written data application into a first store; after the data writing is completed, storing the generated change information to a Redis single node/cluster; if the data application is a read data application, acquiring corresponding metadata from the Redis single node/cluster according to HMaster addresses and the read data application; and initiating a read request to the target Region Server corresponding to the target Region Server name, so that the target Region Server acquires corresponding data from the second store according to the read request. According to the technical scheme provided by the invention, all metadata of the HBase are stored by using the Redis single node/cluster, the metadata can be directly obtained from the Redis single node/cluster without interaction between a ZooKeeper and a Region Server in sequence, so that the read-write efficiency of the metadata is low, the read-write efficiency of the data is low, and the performance of the whole HBase cluster is improved; and the ZooKeeper and Region Server do not need to reserve a memory to store the metadata of HBase, so that the design structure of the ZooKeeper and Region Server is simplified, the time consumption is reduced, and the cost is reduced.
Based on the data processing method provided by the embodiment of the present invention, correspondingly, the embodiment of the present invention also provides a data processing system, as shown in fig. 2, applied to a client, where the data processing system includes:
HMaster an address obtaining unit 21, configured to obtain, when detecting that a user initiates a data application, a HMaster address from a ZooKeeper;
The Region Server information obtaining unit 22 is configured to obtain corresponding Region Server information from the Redis single node/cluster according to the HMaster address and the write data application if the data application is the write data application;
A data writing unit 23, configured to establish a connection with a Region Server indicated by the Region Server information, and write data indicated by the written data application into the first store; the first stoer is stoer corresponding to a store name indicated by the write data application, and is a store in a Region maintained by the Region Server;
A storage unit 24, configured to store the generated change information to the Redis single node/cluster after the data writing is completed, where the change information includes metadata in which data change occurs;
The metadata obtaining unit 25 is configured to obtain corresponding metadata from the Redis single node/cluster according to the HMaster address and the read data application if the data application is the read data application; the metadata at least comprises a target Region Server name, a Region name and a store name;
The data reading unit 26 is configured to initiate a read request to a target Region Server corresponding to the target Region Server name, so that the target Region Server obtains corresponding data from the second store according to the read request; the read request is generated according to the read data application, and the second store is a store corresponding to a store name in the Region corresponding to the Region name.
The specific principle and execution process of each unit in the data processing system disclosed in the above embodiment of the present invention are the same as those of the data processing method disclosed in the above embodiment of the present invention, and may refer to the corresponding parts in the data processing method disclosed in the above embodiment of the present invention, and will not be described in detail here.
The invention provides a data processing system, when detecting that a user initiates a data application, a HMaster address is obtained from a ZooKeeper; if the data application is a write-in data application, acquiring corresponding Region Server information from the Redis single node/cluster according to HMaster addresses and the write-in data application; establishing connection with a Region Server indicated by the Region Server information, and writing data indicated by the written data application into a first store; after the data writing is completed, storing the generated change information to a Redis single node/cluster; if the data application is a read data application, acquiring corresponding metadata from the Redis single node/cluster according to HMaster addresses and the read data application; and initiating a read request to the target Region Server corresponding to the target Region Server name, so that the target Region Server acquires corresponding data from the second store according to the read request. According to the technical scheme provided by the invention, all metadata of the HBase are stored by using the Redis single node/cluster, the metadata can be directly obtained from the Redis single node/cluster without interaction between a ZooKeeper and a Region Server in sequence, so that the read-write efficiency of the metadata is low, the read-write efficiency of the data is low, and the performance of the whole HBase cluster is improved; and the ZooKeeper and Region Server do not need to reserve a memory to store the metadata of HBase, so that the design structure of the ZooKeeper and Region Server is simplified, the time consumption is reduced, and the cost is reduced.
Optionally, the Region Server information obtaining unit includes:
And the Region Server information acquisition subunit is used for accessing the Region single node/cluster by utilizing HMaster address call HMaster if the data application is a write-in data application, and according to the write-in data application, the Region Server information corresponding to the Region single node/cluster is obtained.
Optionally, the storage unit includes:
The change information generating unit is used for generating corresponding change information according to the Region Server name of the Region Server, the Region name of the Region maintained by the Region Server and the store name of the first store after the data writing is completed;
and the storage subunit is used for storing the change information to the Redis single node/cluster.
Optionally, the read data application at least includes a target metadata table name and a first data reading range; a metadata acquisition unit comprising:
The metadata obtaining subunit is configured to use HMaster address call HMaster to access the Redis single node/cluster, and obtain metadata corresponding to the first data reading range from the HBase metadata table corresponding to the target metadata table name stored in the Redis single node/cluster.
Optionally, the data reading unit includes:
And the data reading subunit is used for initiating a reading request to the target Region Server corresponding to the target Region Server name so that the target Region Server reads the data corresponding to the second data reading range indicated by the reading request from the second store.
An embodiment of the present application provides an electronic device, as shown in fig. 3, where the electronic device includes a processor 301 and a memory 302, where the memory 302 is configured to store program codes and data for data processing, and the processor 301 is configured to invoke program instructions in the memory to execute steps for implementing the data processing method in the embodiment described above.
The embodiment of the application provides a storage medium, which comprises a storage program, wherein the storage medium is controlled to be located in equipment to execute the personalized social method of the meta universe shown in the embodiment when the program runs.
In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for a system or system embodiment, since it is substantially similar to a method embodiment, the description is relatively simple, with reference to the description of the method embodiment being made in part. The systems and system embodiments described above are merely illustrative, wherein elements illustrated as separate elements may or may not be physically separate, and elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative elements and steps are described above generally in terms of functionality in order to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing is merely a preferred embodiment of the present invention and it should be noted that modifications and adaptations to those skilled in the art may be made without departing from the principles of the present invention, which are intended to be comprehended within the scope of the present invention.
Claims (10)
1. A data processing method, applied to a client, the method comprising:
When detecting that a user initiates a data application, acquiring HMaster addresses from a ZooKeeper;
If the data application is a write-in data application, acquiring corresponding Region Server information from the Redis single node/cluster according to the HMaster address and the write-in data application;
Establishing connection with a Region Server indicated by the Region Server information, and writing data indicated by a written data application into a first store; the first stoer is stoer corresponding to a store name indicated by a write data application, and is a store in a Region maintained by the Region Server;
after the data writing is completed, storing the generated change information to a Redis single node/cluster, wherein the change information comprises metadata with data change;
If the data application is a read data application, acquiring corresponding metadata from the Redis single node/cluster according to the HMaster address and the read data application; the metadata at least comprises a target Region Server name, a Region name and a store name;
Initiating a read request to a target Region Server corresponding to the target Region Server name, so that the target Region Server acquires corresponding data from a second store according to the read request; and the read request is generated according to the read data application, and the second store is a store corresponding to the store name in the Region corresponding to the Region name.
2. The method of claim 1, wherein if the data application is a write data application, obtaining corresponding Region Server information from the dis single node/cluster according to the HMaster address and the write data application, comprises:
if the data application is a write data application, using the HMaster address call HMaster to access the Redis single node/cluster, and according to the write data application, obtaining the Region Server information corresponding to the Redis single node/cluster.
3. The method of claim 1, wherein storing the generated change information to the Redis single node/cluster after the data writing is completed comprises:
After the data writing is completed, generating corresponding change information according to the Region Server name of the Region Server, the Region name of the Region maintained by the Region Server and the store name of the first store;
and storing the change information to the Redis single node/cluster.
4. The method of claim 1, wherein the read data application includes at least a target metadata table name and a first data read range;
If the data application is a read data application, according to the HMaster address and the read data application, obtaining corresponding metadata from the Redis single node/cluster includes:
And accessing a Redis single node/cluster by utilizing the HMaster address call HMaster, and acquiring metadata corresponding to the first data reading range from a metadata table corresponding to the target metadata table name stored in the Redis single node/cluster.
5. The method of claim 1, wherein the initiating a read request to the target Region Server corresponding to the target Region Server name, causing the target Region Server to obtain corresponding data from the second store according to the read request, comprises:
And initiating a read request to a target Region Server corresponding to the target Region Server name, so that the target Region Server reads data corresponding to a second data read range indicated by the read request from a second store.
6. A data processing system for application to a client, the system comprising:
HMaster address obtaining unit, configured to obtain HMaster address from the ZooKeeper when detecting that the user initiates the data application;
The Region Server information acquisition unit is used for acquiring corresponding Region Server information from the Redis single node/cluster according to the HMaster address and the write-in data application if the data application is the write-in data application;
The data writing unit is used for establishing connection with the Region Server indicated by the Region Server information and writing the data indicated by the written data application into the first store; the first stoer is stoer corresponding to a store name indicated by a write data application, and is a store in a Region maintained by the Region Server;
The storage unit is used for storing the generated change information to the Redis single node/cluster after the data writing is completed, wherein the change information comprises metadata with data change;
The metadata acquisition unit is used for acquiring corresponding metadata from the Redis single node/cluster according to the HMaster address and the read data application if the data application is a read data application; the metadata at least comprises a target Region Server name, a Region name and a store name;
the data reading unit is used for initiating a reading request to a target Region Server corresponding to the target Region Server name, so that the target Region Server acquires corresponding data from a second store according to the reading request; and the read request is generated according to the read data application, and the second store is a store corresponding to the store name in the Region corresponding to the Region name.
7. The system of claim 6, wherein the Region Server information acquisition unit comprises:
And the Region Server information acquisition subunit is used for calling HMaster to access the Redis single node/cluster by using the HMaster address if the data application is a write-in data application, and according to the write-in data application, accessing the Region Server information corresponding to the Redis single node/cluster.
8. The system of claim 6, wherein the storage unit comprises:
The change information generating unit is used for generating corresponding change information according to the Region Server name of the Region Server, the Region name of the Region maintained by the Region Server and the store name of the first store after the data writing is completed;
and the storage subunit is used for storing the change information to the Redis single node/cluster.
9. An electronic device, comprising: the device comprises a processor and a memory, wherein the processor and the memory are connected through a communication bus; the processor is used for calling and executing the program stored in the memory; the memory for storing a program for implementing the data processing method according to any one of claims 1 to 5.
10. A computer readable storage medium having stored therein computer executable instructions for performing the data processing method of any of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410227307.9A CN118069625A (en) | 2024-02-29 | 2024-02-29 | Data processing method, system, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410227307.9A CN118069625A (en) | 2024-02-29 | 2024-02-29 | Data processing method, system, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN118069625A true CN118069625A (en) | 2024-05-24 |
Family
ID=91105425
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410227307.9A Pending CN118069625A (en) | 2024-02-29 | 2024-02-29 | Data processing method, system, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118069625A (en) |
-
2024
- 2024-02-29 CN CN202410227307.9A patent/CN118069625A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11394625B2 (en) | Service level agreement based storage access | |
| US11711420B2 (en) | Automated management of resource attributes across network-based services | |
| US20200344322A1 (en) | Resource scheduling method, apparatus, device and system | |
| US8990243B2 (en) | Determining data location in a distributed data store | |
| CN110597739A (en) | Configuration management method, system and equipment | |
| US10860604B1 (en) | Scalable tracking for database udpates according to a secondary index | |
| US10970311B2 (en) | Scalable snapshot isolation on non-transactional NoSQL | |
| US10191663B1 (en) | Using data store accelerator intermediary nodes and write control settings to identify write propagation nodes | |
| US9910881B1 (en) | Maintaining versions of control plane data for a network-based service control plane | |
| US10579597B1 (en) | Data-tiering service with multiple cold tier quality of service levels | |
| US9009196B2 (en) | Discovery and client routing to database nodes | |
| US10262024B1 (en) | Providing consistent access to data objects transcending storage limitations in a non-relational data store | |
| US20170153909A1 (en) | Methods and Devices for Acquiring Data Using Virtual Machine and Host Machine | |
| US20220188340A1 (en) | Tracking granularity levels for accessing a spatial index | |
| CN112685499A (en) | Method, device and equipment for synchronizing process data of work service flow | |
| US11157456B2 (en) | Replication of data in a distributed file system using an arbiter | |
| CN112148745B (en) | Multi-HBase cluster access method, device and storage medium | |
| CN113407356A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN114840562B (en) | Distributed caching method and device for business data, electronic equipment and storage medium | |
| CN110457307B (en) | Metadata management system, user cluster creation method, device, equipment and medium | |
| CN115981559A (en) | Distributed data storage method and device, electronic equipment and readable medium | |
| CN118069625A (en) | Data processing method, system, electronic equipment and storage medium | |
| US11386072B1 (en) | Automatic consistency for database write forwarding | |
| CN115098469A (en) | Database migration method and device, electronic equipment and readable storage medium | |
| CN111680069B (en) | Database access method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication |