CN110928954A - HBase index synchronization method, HBase index synchronization device, computer equipment and storage medium - Google Patents
HBase index synchronization method, HBase index synchronization device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN110928954A CN110928954A CN201911229207.5A CN201911229207A CN110928954A CN 110928954 A CN110928954 A CN 110928954A CN 201911229207 A CN201911229207 A CN 201911229207A CN 110928954 A CN110928954 A CN 110928954A
- Authority
- CN
- China
- Prior art keywords
- data
- hbase
- index
- synchronized
- needing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 57
- 230000001360 synchronised effect Effects 0.000 claims abstract description 100
- 238000004590 computer program Methods 0.000 claims description 21
- 238000012545 processing Methods 0.000 claims description 14
- 230000015572 biosynthetic process Effects 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 claims description 3
- 238000010276 construction Methods 0.000 claims description 2
- 238000010586 diagram Methods 0.000 description 9
- 239000000306 component Substances 0.000 description 8
- 238000013500 data storage Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000009545 invasion Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 239000008358 core component Substances 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/221—Column-oriented storage; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2272—Management thereof
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a method, a device, computer equipment and a storage medium for synchronizing HBase indexes, wherein the method comprises the steps of obtaining HBase data to be synchronized; obtaining Rowkey in HBase; establishing an index according to HBase data needing to be synchronized to obtain a target index; and storing the target index and Rowkey into a search system for being used as a query index when reading data. According to the method, the target index of the HBase data to be synchronized is constructed in a field selection mode after the HBase data to be synchronized is obtained, and is stored in a search system in a Rowkey combination mode according to the index type corresponding to the target index, configuration does not need to be changed, real-time synchronization of the HBase index is achieved, and the system is small in invasiveness.
Description
Technical Field
The invention relates to a data processing method, in particular to an HBase index synchronization method, an HBase index synchronization device, computer equipment and a storage medium.
Background
HBase is a distributed database based on column storage, serves as a core component of an open-source distributed batch processing frame Hadoop ecological circle, plays a key role in a storage framework of a plurality of internet enterprises with good writing performance, excellent expandability and stable data storage, and is an ideal storage medium for mass data.
The HBase is suitable for a real-time high-concurrency random reading scene, in order to accelerate the reading speed of the HBase, if the Rowkey of the HBase can be quickly obtained, corresponding data can be obtained through a get method, the reading speed of the HBase is greatly improved, meanwhile, the HBase has good support for data storage, but the complex query support for the data is poor, in order to solve the problems, indexes of the corresponding data are generally established through an enterprise-level search application server or an Elasticissearch, the complex query is firstly carried out through a search frame, then the Rowkey of the corresponding HBase is obtained, and therefore the data can be quickly obtained from the HBase, but the indexes need to be synchronously processed through a coprocessor of the HBase, and the invasion to a system is large due to the need of configuration change and the like.
Therefore, it is necessary to design a new method to implement real-time synchronization of HBase indexes with less intrusiveness to the system.
Disclosure of Invention
The invention aims to overcome the defects of the prior art and provides an HBase index synchronization method, an HBase index synchronization device, computer equipment and a storage medium.
In order to achieve the purpose, the invention adopts the following technical scheme: the HBase index synchronization method comprises the following steps:
acquiring HBase data to be synchronized;
obtaining Rowkey in HBase;
establishing an index according to HBase data needing to be synchronized to obtain a target index;
and storing the target index and Rowkey into a search system for being used as a query index when reading data.
The further technical scheme is as follows: the acquiring HBase data needing to be synchronized comprises the following steps:
judging whether HBase data to be synchronized is stock data;
if the HBase data to be synchronized is stock data, reading the total HBase data by a MapReduce method, and acquiring a field needing to establish an index and a value corresponding to the field needing to establish the index so as to obtain the HBase data to be synchronized;
if the HBase data needing to be synchronized is not stock data, acquiring the data needing to be read and sending the data to kafka to obtain a data forming result;
judging whether the data forming result is a formable data stream result;
if the data forming result is a data stream forming result, receiving the kafka message through spark line or flex stream processing to form HBase data to be synchronized;
and if the data forming result is not the result of forming the data stream, importing the data required to be read, and reading the field needing to establish the index in the Hbase to obtain the HBase data required to be synchronized.
The further technical scheme is as follows: the importing the data required to be read and reading the fields required to establish indexes in the Hbase to obtain the HBase data tool required to be synchronized comprises the following steps:
and importing data to be read, and reading a field needing to establish an index in the Hbase through at least one of SPARK and Flink to obtain an HBase data tool needing to be synchronized.
The further technical scheme is as follows: the establishing of the index according to the HBase data needing to be synchronized to obtain the target index comprises the following steps:
and selecting a field of the HBase second index according to the HBase data to be synchronized to obtain a target index.
The further technical scheme is as follows: the number of the fields of the HBase second index is at least one.
The further technical scheme is as follows: the storing the target index and the Rowkey into a search system for being used as a query index when reading data comprises:
acquiring a type corresponding to the target index from a preset type table according to the target index to obtain an index type;
and storing the target index to a corresponding position in the search system according to the index type and the Rowkey so as to be used as a query index when data is read.
The invention also provides an HBase index synchronization device, which comprises:
the data acquisition unit is used for acquiring HBase data needing synchronization;
the first acquisition unit is used for acquiring Rowkey in HBase;
the index construction unit is used for establishing an index according to HBase data needing to be synchronized so as to obtain a target index;
and the index storage unit is used for storing the target index and the Rowkey into the search system so as to be used as a query index when data is read.
The further technical scheme is as follows: the data acquisition unit includes:
the stock judging subunit is used for judging whether the HBase data needing to be synchronized is stock data;
the stock obtaining subunit is used for reading the HBase full data by a MapReduce method and obtaining a field needing to establish an index and a value corresponding to the field needing to establish the index to obtain the HBase data needing to be synchronized if the HBase data needing to be synchronized is stock data;
the issuing subunit is used for acquiring the data to be read and sending the data to kafka to obtain a data forming result if the HBase data to be synchronized is not stock data;
a data flow judgment subunit, configured to judge whether the data formation result is a formable data flow result;
the processing subunit is used for receiving the kafka message through spark line or flash stream processing to form HBase data to be synchronized if the data forming result is a data stream forming result;
and the importing subunit is used for importing the data to be read if the data forming result is not the result capable of forming the data stream, and reading the field needing to establish the index in the Hbase through spark or flush combined with a memory calculation tool to obtain the HBase data needing to be synchronized.
The invention also provides computer equipment which comprises a memory and a processor, wherein the memory is stored with a computer program, and the processor realizes the method when executing the computer program.
The invention also provides a storage medium storing a computer program which, when executed by a processor, is operable to carry out the method as described above.
Compared with the prior art, the invention has the beneficial effects that: according to the method, the target index of the HBase data to be synchronized is constructed in a field selection mode after the HBase data to be synchronized is obtained, and is stored in a search system in a Rowkey combination mode according to the index type corresponding to the target index, configuration does not need to be changed, real-time synchronization of the HBase index is achieved, and the system is small in invasiveness.
The invention is further described below with reference to the accompanying drawings and specific embodiments.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic view of an application scenario of an HBase index synchronization method according to an embodiment of the present invention;
fig. 2 is a schematic flow chart of an HBase index synchronization method according to an embodiment of the present invention;
fig. 3 is a schematic sub-flow diagram of an HBase index synchronization method according to an embodiment of the present invention;
fig. 4 is a schematic sub-flow diagram of an HBase index synchronization method according to an embodiment of the present invention;
fig. 5 is a schematic block diagram of an HBase index synchronization apparatus according to an embodiment of the present invention;
fig. 6 is a schematic block diagram of a data obtaining unit of the HBase index synchronization apparatus according to the embodiment of the present invention;
fig. 7 is a schematic block diagram of an index saving unit of the HBase index synchronization apparatus according to the embodiment of the present invention;
FIG. 8 is a schematic block diagram of a computer device provided by an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the specification of the present invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
Referring to fig. 1 and fig. 2, fig. 1 is a schematic view of an application scenario of the HBase index synchronization method according to the embodiment of the present invention. Fig. 2 is a schematic flow chart of an HBase index synchronization method according to an embodiment of the present invention. The HBase index synchronization method is applied to a server, the server can perform data interaction with a terminal, and data to be synchronized are transmitted to the server through the terminal so that the server can establish a corresponding index, the index can be inquired when the terminal inquires the data again, and the inquiry speed is improved.
Fig. 2 is a schematic flow chart of an HBase index synchronization method according to an embodiment of the present invention. As shown in fig. 2, the method includes the following steps S110 to S150.
And S110, acquiring HBase data needing synchronization.
In this embodiment, the HBase data to be synchronized refers to data that needs to be stored in the HBase and is input by the terminal.
Acquiring HBase data to be synchronized comprises two forms, wherein one form is data acquisition aiming at real-time index creation, namely, before the Hbase data is inserted, data is synchronously sent to kafka through a certain method such as prePut and the like to form a data stream, and a message of the kafka is received through spark streaming or flink streaming; data imported through a certain tool such as sqoop, because a data stream cannot be formed, reading Hbase through at least one of spark and flush by a memory computing method to obtain a field needing index establishment;
the other method is to acquire data of the stock index, namely, for the stock data, reading Hbase full-size data by a MapReduce method, and acquiring a field needing to establish the index and a corresponding value thereof.
In an embodiment, referring to fig. 3, the step S110 may include steps S111 to S116.
And S111, judging whether the HBase data needing to be synchronized is stock data.
In the present embodiment, the stock data refers to the total amount data in Hbase that can be read by the MapReduce method.
S112, if the HBase data to be synchronized is stock data, reading the total HBase data by a MapReduce method, and acquiring a field needing to establish an index and a value corresponding to the field needing to establish the index so as to obtain the HBase data to be synchronized;
s113, if the HBase data needing to be synchronized is not stock data, acquiring the data needing to be read and sending the data to kafka to obtain a data forming result;
the data forming result includes a result that a data stream can be formed and also includes a result that a data stream cannot be formed.
S114, judging whether the data forming result is a data stream forming result or not;
s115, if the data forming result is a data stream forming result, receiving a kafka message through spark line or flash stream processing to form HBase data needing to be synchronized;
and S116, if the data forming result is not the result capable of forming the data stream, importing the data required to be read, and reading the field needing to establish the index in the Hbase to obtain the HBase data required to be synchronized.
Specifically, data required to be read is imported, and fields needing index establishment in Hbase are read through at least one of SPARK and Flink, so that an HBase data tool needing synchronization is obtained.
HBase data is synchronized in real time, for example, the HBase data to be synchronized is obtained through a message middleware, and then the message in the message middleware can be synchronously obtained through spark or flash in real time to form the HBase data to be synchronized; if the HBase data can not be acquired to the message source, reading the HBase data needing to be synchronized from the HBase through memory computing tools such as spark or flink.
And S120, obtaining Rowkey in HBase.
In this embodiment, the Rowkey is used to search the records in the table, and after the Rowkey is obtained, the corresponding data can be obtained by the get method.
And S130, establishing an index according to HBase data needing to be synchronized to obtain a target index.
In this embodiment, the target index refers to a condition for referring to HBase data to be synchronized.
Specifically, a field of the HBase second index is selected according to the HBase data needing to be synchronized, so that a target index is obtained.
Specifically, the number of fields of the HBase second index is at least one.
When the target index is established, the target index is established by adopting a field selection mode, and the configuration is not required to be modified, so that the system is less invasive.
And S140, storing the target index and the Rowkey into a search system for being used as a query index when data is read.
In an embodiment, referring to fig. 4, the step S140 may include steps S141 to S142.
S141, obtaining the type corresponding to the target index from a preset type table according to the target index to obtain the index type.
In this embodiment, the index types include a title type, a field type, a keyword type, and the like, wherein the title type is a life record, the field type is clothing, and the keyword type is environmental protection.
And S142, storing the target index to a corresponding position in the search system according to the index type and the Rowkey so as to be used as a query index when reading data.
The target index and the Rowkey are stored in the search system, when the terminal queries data, the data can be obtained in a Rowkey obtaining mode, the data can also be queried in a target index mode, and the two query modes are combined, so that the data query and reading speed can be improved.
For example: and synchronously acquiring data needing to be synchronized into the HBase from the kafka through spark line, namely the HBase data needing to be synchronized, then acquiring the value of a field needing to establish an index, obtaining data esRDD needing to be inserted into the elastic search through RDD conversion of spark, and then realizing synchronization of the secondary index of the HBase to the elastic search through a saveEs () method.
According to the HBase index synchronization method, after the HBase data needing to be synchronized is obtained, the target index of the HBase data needing to be synchronized is constructed in a field selection mode, and is stored in a search system in a Rowkey combination mode according to the index type corresponding to the target index, configuration does not need to be changed, real-time synchronization of the HBase index is achieved, and invasion to the system is small.
Fig. 5 is a schematic block diagram of an HBase index synchronization apparatus 300 according to an embodiment of the present invention. As shown in fig. 5, the present invention further provides an HBase index synchronization apparatus 300 corresponding to the above HBase index synchronization method. The HBase index synchronization apparatus 300 includes a unit for performing the HBase index synchronization method described above, and the apparatus may be configured in a server.
Specifically, referring to fig. 5, the HBase index synchronization apparatus 300 includes a data obtaining unit 301, a first obtaining unit 302, an index constructing unit 303, and an index storing unit 304.
A data obtaining unit 301, configured to obtain HBase data to be synchronized; a first obtaining unit 302, configured to obtain a Rowkey in the HBase; an index constructing unit 303, configured to establish an index according to HBase data that needs to be synchronized, so as to obtain a target index; and an index storage unit 304, configured to store the target index and the Rowkey in the search system, so as to be used as a query index when reading data.
In one embodiment, as shown in fig. 6, the data obtaining unit 301 includes a stock quantity determining subunit 3011, a stock quantity obtaining subunit 3012, a sending subunit 3013, a data stream determining subunit 3014, a processing subunit 3015, and a importing subunit 3016.
A stock judging subunit 3011, configured to judge whether the HBase data to be synchronized is stock data; a stock obtaining subunit 3012, configured to, if the HBase data to be synchronized is stock data, read the HBase full data by using a MapReduce method, and obtain a field requiring index establishment and a value corresponding to the field requiring index establishment, so as to obtain the HBase data to be synchronized; a sending subunit 3013, configured to, if the HBase data to be synchronized is not stock data, obtain data to be read and send the data to kafka, so as to obtain a data formation result; a data flow judgment subunit 3014 configured to judge whether the data formation result is a formable data flow result; a processing subunit 3015, configured to, if the data forming result is a result that a data stream can be formed, process, by spark timing or flush stream, the message of the kafka received, so as to form HBase data to be synchronized; and an importing subunit 3016, configured to import data to be read if the data forming result is not a result that a data stream can be formed, and read a field that needs to establish an index in the Hbase through spark or flush in combination with a memory computing tool, so as to obtain Hbase data that needs to be synchronized.
In an embodiment, as shown in fig. 7, the index saving unit 304 includes a type obtaining subunit 3041 and a storing subunit 3042.
A type obtaining subunit 3041, configured to obtain, according to the target index, a type corresponding to the target index from a preset type table, so as to obtain an index type; the storage subunit 3042 is configured to store the target index to a corresponding location in the search system according to the index type and the Rowkey, so as to be used as a query index when reading data.
It should be noted that, as can be clearly understood by those skilled in the art, the specific implementation process of the HBase index synchronization apparatus 300 and each unit may refer to the corresponding description in the foregoing method embodiment, and for convenience and brevity of description, no further description is provided herein.
The HBase index synchronizing apparatus 300 may be implemented in the form of a computer program that can be run on a computer device as shown in fig. 8.
Referring to fig. 8, fig. 8 is a schematic block diagram of a computer device according to an embodiment of the present application. The computer device 500 is a server, wherein the server may be an independent server or a server cluster composed of a plurality of servers.
Referring to fig. 8, the computer device 500 includes a processor 502, memory, and a network interface 505 connected by a system bus 501, where the memory may include a non-volatile storage medium 503 and an internal memory 504.
The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer programs 5032 include program instructions that, when executed, cause the processor 502 to perform an HBase index synchronization method.
The processor 502 is used to provide computing and control capabilities to support the operation of the overall computer device 500.
The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503, and when the computer program 5032 is executed by the processor 502, the processor 502 may be caused to execute an HBase index synchronization method.
The network interface 505 is used for network communication with other devices. Those skilled in the art will appreciate that the configuration shown in fig. 8 is a block diagram of only a portion of the configuration relevant to the present teachings and does not constitute a limitation on the computer device 500 to which the present teachings may be applied, and that a particular computer device 500 may include more or less components than those shown, or combine certain components, or have a different arrangement of components.
Wherein the processor 502 is configured to run the computer program 5032 stored in the memory to implement the following steps:
acquiring HBase data to be synchronized; obtaining Rowkey in HBase; establishing an index according to HBase data needing to be synchronized to obtain a target index; and storing the target index and Rowkey into a search system for being used as a query index when reading data.
In an embodiment, when the processor 502 implements the step of acquiring the HBase data to be synchronized, the following steps are specifically implemented:
judging whether HBase data to be synchronized is stock data; if the HBase data to be synchronized is stock data, reading the total HBase data by a MapReduce method, and acquiring a field needing to establish an index and a value corresponding to the field needing to establish the index so as to obtain the HBase data to be synchronized; if the HBase data needing to be synchronized is stock data, acquiring the data needing to be read and sending the data to kafka to obtain a data forming result; judging whether the data forming result is a formable data stream result; if the data forming result is a data stream forming result, receiving the kafka message through spark line or flex stream processing to form HBase data to be synchronized; and if the data forming result is not the result of forming the data stream, importing the data required to be read, and reading the field needing to establish the index in the Hbase to obtain the HBase data required to be synchronized.
In an embodiment, when the processor 502 implements the steps of importing the data to be read and reading a field in the Hbase that needs to establish an index to obtain the Hbase data tool that needs to be synchronized, the following steps are specifically implemented:
and importing data to be read, and reading a field needing to establish an index in the Hbase through at least one of SPARK and Flink to obtain an HBase data tool needing to be synchronized.
In an embodiment, when the processor 502 implements the step of establishing an index according to the HBase data to be synchronized to obtain a target index, the following steps are specifically implemented:
and selecting a field of the HBase second index according to the HBase data to be synchronized to obtain a target index.
Wherein, the number of the fields of the HBase second index is at least one.
In an embodiment, when the processor 502 implements the step of storing the target index and the Rowkey in the search system for being used as a query index when reading data, the following steps are specifically implemented:
acquiring a type corresponding to the target index from a preset type table according to the target index to obtain an index type; and storing the target index to a corresponding position in the search system according to the index type and the Rowkey so as to be used as a query index when data is read.
It should be understood that, in the embodiment of the present Application, the Processor 502 may be a Central Processing Unit (CPU), and the Processor 502 may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, and the like. Wherein a general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
It will be understood by those skilled in the art that all or part of the flow of the method implementing the above embodiments may be implemented by a computer program instructing associated hardware. The computer program includes program instructions, and the computer program may be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the flow steps of the embodiments of the method described above.
Accordingly, the present invention also provides a storage medium. The storage medium may be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program, when executed by a processor, causes the processor to perform the steps of:
acquiring HBase data to be synchronized; obtaining Rowkey in HBase; establishing an index according to HBase data needing to be synchronized to obtain a target index; and storing the target index and Rowkey into a search system for being used as a query index when reading data.
In an embodiment, when the processor executes the computer program to implement the step of acquiring the HBase data to be synchronized, the following steps are specifically implemented:
judging whether HBase data to be synchronized is stock data; if the HBase data to be synchronized is stock data, reading the total HBase data by a MapReduce method, and acquiring a field needing to establish an index and a value corresponding to the field needing to establish the index so as to obtain the HBase data to be synchronized; if the HBase data needing to be synchronized is stock data, acquiring the data needing to be read and sending the data to kafka to obtain a data forming result; judging whether the data forming result is a formable data stream result; if the data forming result is a data stream forming result, receiving the kafka message through spark line or flex stream processing to form HBase data to be synchronized; and if the data forming result is not the result of forming the data stream, importing the data required to be read, and reading the field needing to establish the index in the Hbase to obtain the HBase data required to be synchronized.
In an embodiment, when the processor executes the computer program to implement the steps of importing the data to be read, and reading a field in the Hbase that needs to establish an index to obtain an Hbase data tool that needs to be synchronized, the following steps are specifically implemented:
and importing data to be read, and reading a field needing to establish an index in the Hbase through at least one of SPARK and Flink to obtain an HBase data tool needing to be synchronized.
In an embodiment, when the processor executes the computer program to implement the step of establishing an index according to HBase data to be synchronized to obtain a target index, the following steps are specifically implemented:
and selecting a field of the HBase second index according to the HBase data to be synchronized to obtain a target index.
Wherein, the number of the fields of the HBase second index is at least one.
In an embodiment, when the processor executes the computer program to implement the step of storing the target index and the Rowkey in the search system for being used as a query index when reading data, the following steps are specifically implemented:
acquiring a type corresponding to the target index from a preset type table according to the target index to obtain an index type; and storing the target index to a corresponding position in the search system according to the index type and the Rowkey so as to be used as a query index when data is read.
The storage medium may be a usb disk, a removable hard disk, a Read-Only Memory (ROM), a magnetic disk, or an optical disk, which can store various computer readable storage media.
Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be embodied in electronic hardware, computer software, or combinations of both, and that the components and steps of the examples have been described in a functional general in the foregoing description for the purpose of illustrating clearly the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative. For example, the division of each unit is only one logic function division, and there may be another division manner in actual implementation. For example, various elements or components may be combined or may be integrated into another system, or some features may be omitted, or not implemented.
The steps in the method of the embodiment of the invention can be sequentially adjusted, combined and deleted according to actual needs. The units in the device of the embodiment of the invention can be merged, divided and deleted according to actual needs. In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a storage medium. Based on such understanding, the technical solution of the present invention essentially or partially contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a terminal, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention.
While the invention has been described with reference to specific embodiments, the invention is not limited thereto, and various equivalent modifications and substitutions can be easily made by those skilled in the art within the technical scope of the invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
- The HBase index synchronization method is characterized by comprising the following steps:acquiring HBase data to be synchronized;obtaining Rowkey in HBase;establishing an index according to HBase data needing to be synchronized to obtain a target index;and storing the target index and Rowkey into a search system for being used as a query index when reading data.
- 2. The HBase index synchronization method according to claim 1, wherein the obtaining HBase data to be synchronized comprises:judging whether HBase data to be synchronized is stock data;if the HBase data to be synchronized is stock data, reading the total HBase data by a MapReduce method, and acquiring a field needing to establish an index and a value corresponding to the field needing to establish the index so as to obtain the HBase data to be synchronized;if the HBase data needing to be synchronized is stock data, acquiring the data needing to be read and sending the data to kafka to obtain a data forming result;judging whether the data forming result is a formable data stream result;if the data forming result is a data stream forming result, receiving the kafka message through spark line or flex stream processing to form HBase data to be synchronized;and if the data forming result is not the result of forming the data stream, importing the data required to be read, and reading the field needing to establish the index in the Hbase to obtain the HBase data required to be synchronized.
- 3. The HBase index synchronization method according to claim 2, wherein the importing the data to be read and reading the fields of the HBase requiring index establishment to obtain the HBase data tool to be synchronized comprises:and importing data to be read, and reading a field needing to establish an index in the Hbase through at least one of SPARK and Flink to obtain an HBase data tool needing to be synchronized.
- 4. The HBase index synchronization method according to claim 1, wherein the establishing an index according to HBase data to be synchronized to obtain a target index comprises:and selecting a field of the HBase second index according to the HBase data to be synchronized to obtain a target index.
- 5. The HBase index synchronization method according to claim 4, wherein the number of fields of the HBase second index is at least one.
- 6. The HBase index synchronization method of claim 1, wherein the saving the target index and Rowkey into a search system for use as a query index when reading data comprises:acquiring a type corresponding to the target index from a preset type table according to the target index to obtain an index type;and storing the target index to a corresponding position in the search system according to the index type and the Rowkey so as to be used as a query index when data is read.
- The HBase index synchronization device is characterized by comprising:the data acquisition unit is used for acquiring HBase data needing synchronization;the first acquisition unit is used for acquiring Rowkey in HBase;the index construction unit is used for establishing an index according to HBase data needing to be synchronized so as to obtain a target index;and the index storage unit is used for storing the target index and the Rowkey into the search system so as to be used as a query index when data is read.
- 8. The HBase index synchronization apparatus according to claim 7, wherein the data acquisition unit comprises:the stock judging subunit is used for judging whether the HBase data needing to be synchronized is stock data;the stock obtaining subunit is used for reading the HBase full data by a MapReduce method and obtaining a field needing to establish an index and a value corresponding to the field needing to establish the index to obtain the HBase data needing to be synchronized if the HBase data needing to be synchronized is stock data;the issuing subunit is used for acquiring the data to be read and sending the data to kafka to obtain a data forming result if the HBase data to be synchronized is not stock data;a data flow judgment subunit, configured to judge whether the data formation result is a formable data flow result;the processing subunit is used for receiving the kafka message through spark line or flash stream processing to form HBase data to be synchronized if the data forming result is a data stream forming result;and the importing subunit is used for importing the data to be read if the data forming result is not the result capable of forming the data stream, and reading the field needing to establish the index in the Hbase through spark or flush combined with a memory calculation tool to obtain the HBase data needing to be synchronized.
- 9. A computer device, characterized in that the computer device comprises a memory, on which a computer program is stored, and a processor, which when executing the computer program implements the method according to any of claims 1 to 6.
- 10. A storage medium, characterized in that the storage medium stores a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911229207.5A CN110928954A (en) | 2019-12-04 | 2019-12-04 | HBase index synchronization method, HBase index synchronization device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911229207.5A CN110928954A (en) | 2019-12-04 | 2019-12-04 | HBase index synchronization method, HBase index synchronization device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110928954A true CN110928954A (en) | 2020-03-27 |
Family
ID=69856794
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911229207.5A Pending CN110928954A (en) | 2019-12-04 | 2019-12-04 | HBase index synchronization method, HBase index synchronization device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110928954A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113254445A (en) * | 2021-05-26 | 2021-08-13 | 平安普惠企业管理有限公司 | Real-time data storage method and device, computer equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2013182054A1 (en) * | 2012-06-06 | 2013-12-12 | 腾讯科技（深圳）有限公司 | Memory retrieval, real time retrieval system and method, and computer storage medium |
| CN108667929A (en) * | 2018-05-08 | 2018-10-16 | 浪潮软件集团有限公司 | Method for synchronizing data to elastic search based on HBase coprocessor |
| CN109800222A (en) * | 2018-12-11 | 2019-05-24 | 中国科学院信息工程研究所 | A kind of HBase secondary index adaptive optimization method and system |
| CN110457397A (en) * | 2019-08-16 | 2019-11-15 | 深圳前海微众银行股份有限公司 | A kind of method and device that data are synchronous |
-
2019
- 2019-12-04 CN CN201911229207.5A patent/CN110928954A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2013182054A1 (en) * | 2012-06-06 | 2013-12-12 | 腾讯科技（深圳）有限公司 | Memory retrieval, real time retrieval system and method, and computer storage medium |
| CN108667929A (en) * | 2018-05-08 | 2018-10-16 | 浪潮软件集团有限公司 | Method for synchronizing data to elastic search based on HBase coprocessor |
| CN109800222A (en) * | 2018-12-11 | 2019-05-24 | 中国科学院信息工程研究所 | A kind of HBase secondary index adaptive optimization method and system |
| CN110457397A (en) * | 2019-08-16 | 2019-11-15 | 深圳前海微众银行股份有限公司 | A kind of method and device that data are synchronous |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113254445A (en) * | 2021-05-26 | 2021-08-13 | 平安普惠企业管理有限公司 | Real-time data storage method and device, computer equipment and storage medium |
| CN113254445B (en) * | 2021-05-26 | 2024-01-05 | 禾量科技(上海)有限公司 | Real-time data storage method, device, computer equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| AU2016382908B2 (en) | Short link processing method, device and server | |
| CN110688382B (en) | Data storage query method and device, computer equipment and storage medium | |
| CN109741060B (en) | Information inquiry system, method, device, electronic equipment and storage medium | |
| CN107018042B (en) | Tracking method and tracking system for online service system | |
| CN110309161B (en) | Data synchronization method and device and server | |
| CN111258978B (en) | Data storage method | |
| CN109766176B (en) | Scanning progress calculation method and device based on large-scale network space detection | |
| CN105653198A (en) | Data processing method and device | |
| CN107798135B (en) | Paging query method and device and electronic equipment | |
| US20150234883A1 (en) | Method and system for retrieving real-time information | |
| CN111026774A (en) | Data sequence synchronization method and device, computer equipment and storage medium | |
| CN109241099B (en) | Data query method and terminal equipment | |
| CN106021315A (en) | Log management method and system for application program | |
| US20180321706A1 (en) | Timestamp Alignment Across a Plurality Of Computing Devices | |
| CN110222046B (en) | List data processing method, device, server and storage medium | |
| CN108090056B (en) | Data query method, device and system | |
| CN108206776B (en) | Group history message query method and device | |
| CN113377777B (en) | Data loading method, device, computer program product and storage medium | |
| CN110928954A (en) | HBase index synchronization method, HBase index synchronization device, computer equipment and storage medium | |
| CN110069565B (en) | Distributed database data batch processing method and device | |
| CN112671878B (en) | Block chain information subscription method, device, server and storage medium | |
| CN112527900A (en) | Method, device, equipment and medium for database multi-copy reading consistency | |
| CN111400266B (en) | Data processing method and system, and diagnosis processing method and device for operation event | |
| CN110909072B (en) | Data table establishment method, device and equipment | |
| CN111209263A (en) | Data storage method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200327 |
|
| RJ01 | Rejection of invention patent application after publication |