CN114020750A - A system and method for reading and writing massive data based on distributed storage - Google Patents
A system and method for reading and writing massive data based on distributed storage Download PDFInfo
- Publication number
- CN114020750A CN114020750A CN202111317987.6A CN202111317987A CN114020750A CN 114020750 A CN114020750 A CN 114020750A CN 202111317987 A CN202111317987 A CN 202111317987A CN 114020750 A CN114020750 A CN 114020750A
- Authority
- CN
- China
- Prior art keywords
- data
- hbase database
- information
- row key
- distributed storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 29
- 238000004590 computer program Methods 0.000 claims description 12
- 238000010276 construction Methods 0.000 claims description 5
- 230000011218 segmentation Effects 0.000 claims description 5
- 238000010586 diagram Methods 0.000 description 7
- 238000012545 processing Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 230000000644 propagated effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 230000001105 regulatory effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2219—Large Object storage; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2255—Hash tables
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本发明涉及一种基于分布式存储的海量数据读写方法及系统，其方法包括：获取高精度地图的采集数据，并将其存放到Hbase数据库中；根据采集数据的时间信息、设备信息和空间信息构建所述Hbase数据库的行键；将采集数据的描述信息存放在elasticsearch数据库中，并以Hbase数据库的行键作为通用唯一标识符；根据所述描述信息和所述通用唯一标识符建立数据索引；将用户的数据读写请求与数据索引匹配，对匹配到的Hbase数据库中的数据执行相应的读写操作。本发明首先将数据和索引分别存储，再通过行键实现Hbase与elasticsearch的互通，从而解决了高并发读写场景下性能不足的问题。
The invention relates to a method and system for reading and writing massive data based on distributed storage. The method includes: acquiring the collection data of a high-precision map and storing it in an Hbase database; Information constructs the row key of the HBase database; the description information of the collected data is stored in the elasticsearch database, and the row key of the HBase database is used as a universally unique identifier; According to the description information and the universally unique identifier, a data index is established ; Match the user's data read and write request with the data index, and perform the corresponding read and write operations on the matched data in the HBase database. The invention firstly stores data and indexes separately, and then realizes the intercommunication between Hbase and elastic search through row keys, thereby solving the problem of insufficient performance in high concurrent read and write scenarios.
Description
Technical Field
The invention belongs to the technical field of high-precision maps and big data processing, and particularly relates to a method and a system for keeping the road network data range of an electronic horizon line to be minimized.
Background
Drawing of a high-precision crowdsourcing map requires efficient writing and reading of collected data of a large number of automobiles. Due to the fact that single automobile is high in uploading frequency and large in quantity of automobiles, distributed real-time writing can better meet the data freshness degree than timed batch writing; when data is read, the data reading needs to be dispersed to a plurality of machines due to the high concurrent reading task, and load balancing is realized. The invention provides a distributed read-write processing system device.
The existing collected data is stored in a relational database, all read-write tasks need to be regulated and controlled by a database main node, all client requests need the main node to spend resources for maintaining connection, and when a high-concurrency read-write scene exists, a large resource pressure is caused on the server main node.
Disclosure of Invention
In order to solve the problem of insufficient resources of a server main node under the condition of large concurrent reading and writing, a first aspect of the invention provides a mass data reading and writing method based on distributed storage, which comprises the following steps: acquiring the acquired data of the high-precision map, and storing the acquired data into an Hbase database; constructing a row key of the Hbase database according to the time information, the equipment information and the spatial information of the acquired data; storing description information of acquired data in an elastic search database, and taking a row key of an Hbase database as a universal unique identifier; establishing a data index according to the description information and the universal unique identifier; and matching the data reading and writing request of the user with the data index, and executing corresponding reading and writing operation on the matched data in the Hbase database.
In some embodiments of the invention, said building a data index from said description information and a universally unique identifier comprises: when a concurrent write task exceeds a throughput threshold of an elastic search cluster, the concurrent write task is distributed evenly across different nodes of the elastic search cluster.
In some embodiments of the present invention, the matching the data read-write request of the user with the data index, and performing corresponding read-write operation on the data in the matched Hbase database includes: matching the data reading request of the user with the data index to obtain one or more row key sets meeting the conditions; and sending a query request to the Hbase database by the user through the one or more row key sets meeting the conditions, and returning the matched data to the user by the Hbase database.
Further, the user sends a query request to the Hbase database through the Thift interface.
In some embodiments of the present invention, the constructing the row key of the Hbase database according to the time information, the device information, and the spatial information of the collected data includes: analyzing the collected data; carrying out tile segmentation on the analyzed data to obtain a tile number of each tile; and constructing a row key according to the tile number, the time information and the equipment information.
Furthermore, the prefix of the space information is used as the prefix of the row key, and the acquisition time and the acquisition equipment are used as suffixes.
In a second aspect of the present invention, a mass data read-write system based on distributed storage is provided, including: the acquisition module is used for acquiring the acquired data of the high-precision map and storing the acquired data into the Hbase database; the building module is used for building a row key of the Hbase database according to the time information, the equipment information and the spatial information of the collected data; the establishing module is used for storing the description information of the acquired data in an elastic search database and taking a row key of an Hbase database as a universal unique identifier; establishing a data index according to the description information and the universal unique identifier; and the matching module is used for matching the data reading and writing request of the user with the data index and executing corresponding reading and writing operation on the matched data in the Hbase database.
Furthermore, the construction module comprises an analysis unit, a segmentation unit and a construction unit, wherein the analysis unit is used for analyzing the acquired data; the dividing unit is used for carrying out tile division on the analyzed data to obtain the tile number of each tile; and the construction unit is used for constructing a row key according to the tile number, the time information and the equipment information.
In a third aspect of the present invention, there is provided an electronic device comprising: one or more processors; the storage device is used for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the mass data reading and writing method based on distributed storage provided by the first aspect of the present invention.
In a fourth aspect of the present invention, a computer-readable medium is provided, on which a computer program is stored, where the computer program, when executed by a processor, implements the mass data reading and writing method based on distributed storage provided in the first aspect of the present invention.
The invention has the beneficial effects that:
1. according to the method, the storage data and the index data are respectively stored in different distributed databases, so that the problem that the load of the main node is too high due to concurrent read-write tasks of the high-precision crowdsourcing map in the uploading process is solved;
2. according to the invention, data and indexes are respectively stored, and then the interworking between Hbase and the elastic search is realized through a rowkey, so that the matching problem of a high-distributed storage node and a distributed index node is solved;
3. a row key set constructed by time information, equipment information and space information distinguishes crowdsourcing data sources, and the problem of data inconsistency caused by row key repetition is avoided.
Drawings
Fig. 1 is a schematic basic flow chart of a mass data read-write method based on distributed storage according to some embodiments of the present invention;
fig. 2 is a schematic flowchart of a mass data read-write method based on distributed storage according to some embodiments of the present invention;
FIG. 3 is a timing diagram for data reading in some embodiments of the present invention;
fig. 4 is a schematic structural diagram of a mass data read-write system based on distributed storage in some embodiments of the present invention;
fig. 5 is a schematic structural diagram of an electronic device in some embodiments of the invention.
Detailed Description
The principles and features of this invention are described below in conjunction with the following drawings, which are set forth by way of illustration only and are not intended to limit the scope of the invention.
Referring to fig. 1 and fig. 2, in a first aspect of the present invention, a mass data read-write method based on distributed storage is provided, including: s100, acquiring the acquired data of the high-precision map, and storing the acquired data into an Hbase database; s200, establishing a row key of the Hbase database according to time information, equipment information and spatial information of the acquired data; s300, storing description information of the acquired data in an elastic search database, and taking a row key of an Hbase database as a universal unique identifier; establishing a data index according to the description information and the universal unique identifier; and S400, matching the data read-write request of the user with the data index, and executing corresponding read-write operation on the matched data in the Hbase database.
It is understood that HBase is a Hadoop database, often described as a sparse, distributed, persistent, multidimensional ordered mapping that indexes based on row keys (Rowkey), column keys, and timestamps, and is a randomly accessible platform for storing and retrieving data. The HBase query is typically queried by its Rowkey. The Hbase database or the elastic search database is a distributed database, which is usually deployed in a corresponding cluster (Hbase cluster or elastic search cluster); therefore, the HBase database or the elastic search database described above may also be understood as a corresponding cluster.
In step S200 of some embodiments of the present invention, on the premise that the Rowkey is not repeated, the data is hashed and stored, so as to avoid a server bottleneck of single-point hot writing caused by writing the data into the same database node in a high concurrency scenario; due to the adoption of the hbase distributed cluster, the task of writing data is balanced to different machines. In order to avoid the consistency problem of the row keys, the prefix of the space information is used as the prefix of the row keys, and the acquisition time and the acquisition equipment are used as suffixes. Specifically, the rowkey format of Hbase adopts a mode of 'space > time > acquisition equipment', prefixes of spatial indexes are used as rowkey prefixes, and acquisition time and acquisition equipment are used as suffixes and spliced together.
In view of this, the constructing the row key of the Hbase database according to the time information, the device information, and the spatial information of the collected data includes: analyzing the collected data; carrying out tile segmentation on the analyzed data to obtain a tile number of each tile; and constructing a row key according to the tile number, the time information and the equipment information.
Specifically, the step of constructing the row key of the Hbase database according to the time information, the device information and the spatial information of the collected data comprises:
s201, analyzing the acquired data, and dividing the data according to the longitude and latitude of each element and the NDS tile to obtain the serial number of the tile;
s202, hashing Tile (Tile) numbers by using hash, splicing acquisition time and acquisition equipment numbers, and constructing a rowkey;
s203, after the acquired data are subjected to structuring processing, rowkey is used as a row key value of hbase, and the data are stored into the hbase.
The tile is a square grid picture obtained by cutting a map within a certain range into a plurality of rows and columns according to a certain size and format and a zoom level or a scale, and the square grid picture after being cut is called a tile visually.
In S300 of some embodiments of the present invention, the creating a data index according to the description information and a Universally Unique Identifier (UUID) includes: when a concurrent write task exceeds a throughput threshold of an elastic search cluster, the concurrent write task is distributed evenly across different nodes of the elastic search cluster. The throughput threshold of an elastic search cluster can be understood as the maximum task load of a single or multiple nodes.
In step S400 in some embodiments of the present invention, the matching the data read/write request of the user with the data index, and performing corresponding read/write operation on the data in the matched Hbase database includes: s401, matching a data reading request of a user with a data index to obtain one or more row key sets meeting conditions; s402, the user sends a query request to the Hbase database through the one or more row key sets meeting the conditions, and the Hbase database returns the matched data to the user.
Specifically, using rowkey as uuid value of the elastic search, and using the rowkey and other data needing indexing as key value pair information, writing the rowkey into the elastic search; during query, the rowkey geometry meeting the conditions is queried in the elastic search through the index field; and reading the required structured data from the hbase through rowkey set.
Schematically, fig. 3 shows a timing chart of one data reading process in the above-described embodiment; the method specifically comprises the following steps: a client sends a query ES secondary index to an ES (elastic search) cluster, and the ES cluster returns a rowkey set meeting the conditions; the client is connected to the hbase through the Thift cluster, query is conducted according to the rowkey set, and the hbase finally returns the data set meeting the conditions to the client. It should be understood that the number of concurrent write tasks and concurrent read tasks is not limited in the present invention, i.e., the present invention can be used in one or more concurrent write tasks and concurrent read tasks.
Example 2
Referring to fig. 4, in a second aspect of the present invention, there is provided a mass data read-write system 1 based on distributed storage, including: the acquisition module 11 is used for acquiring the acquired data of the high-precision map and storing the acquired data into an Hbase database; the building module 12 is configured to build a row key of the Hbase database according to the time information, the device information, and the spatial information of the collected data; the establishing module 13 is used for storing the description information of the acquired data in an elastic search database, and taking a row key of an Hbase database as a universal unique identifier; establishing a data index according to the description information and the universal unique identifier; and the matching module 14 is configured to match the data read-write request of the user with the data index, and perform corresponding read-write operation on the data in the matched Hbase database.
Further, the building module 12 includes an analyzing unit, a dividing unit, and a building unit, where the analyzing unit is configured to analyze the collected data; the dividing unit is used for carrying out tile division on the analyzed data to obtain the tile number of each tile; and the construction unit is used for constructing a row key according to the tile number, the time information and the equipment information.
Example 3
Referring to fig. 5, in a third aspect of the present invention, there is provided an electronic apparatus comprising: one or more processors; storage means for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to carry out the method of the first aspect of the invention.
The electronic device 500 may include a processing means (e.g., central processing unit, graphics processor, etc.) 501 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)502 or a program loaded from a storage means 508 into a Random Access Memory (RAM) 503. In the RAM503, various programs and data necessary for the operation of the electronic apparatus 500 are also stored. The processing device 501, the ROM502, and the RAM503 are connected to each other through a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.
The following devices may be connected to the I/O interface 505 in general: input devices 506 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output devices 507 including, for example, a Liquid Crystal Display (LCD), speakers, vibrators, and the like; a storage device 508 including, for example, a hard disk; and a communication device 509. The communication means 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. While fig. 5 illustrates an electronic device 500 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided. Each block shown in fig. 5 may represent one device or may represent multiple devices as desired.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication means 509, or installed from the storage means 508, or installed from the ROM 502. The computer program, when executed by the processing device 501, performs the above-described functions defined in the methods of embodiments of the present disclosure. It should be noted that the computer readable medium described in the embodiments of the present disclosure may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In embodiments of the disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In embodiments of the present disclosure, however, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device. The computer readable medium carries one or more computer programs which, when executed by the electronic device, cause the electronic device to:
computer program code for carrying out operations for embodiments of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, Python, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (10)
1. A mass data read-write method based on distributed storage is characterized in that,
acquiring the acquired data of the high-precision map, and storing the acquired data into an Hbase database;
constructing a row key of the Hbase database according to the time information, the equipment information and the spatial information of the acquired data;
storing description information of acquired data in an elastic search database, and taking a row key of an Hbase database as a universal unique identifier; establishing a data index according to the description information and the universal unique identifier;
and matching the data reading and writing request of the user with the data index, and executing corresponding reading and writing operation on the matched data in the Hbase database.
2. The mass data reading and writing method based on distributed storage according to claim 1, wherein the establishing of the data index according to the description information and the universal unique identifier comprises:
when a concurrent write task exceeds a throughput threshold of an elastic search cluster, the concurrent write task is distributed evenly across different nodes of the elastic search cluster.
3. The mass data read-write method based on distributed storage according to claim 1, wherein the matching of the data read-write request of the user with the data index and the corresponding read-write operation of the data in the matched Hbase database include:
matching the data reading request of the user with the data index to obtain one or more row key sets meeting the conditions;
and sending a query request to the Hbase database by the user through the one or more row key sets meeting the conditions, and returning the matched data to the user by the Hbase database.
4. The mass data read-write method based on distributed storage according to claim 3, further comprising: the user sends a query request to the Hbase database through the Thift interface.
5. The mass data read-write method based on distributed storage according to claim 1, wherein the constructing the row key of the Hbase database according to the time information, the device information and the spatial information of the collected data comprises:
analyzing the collected data;
carrying out tile segmentation on the analyzed data to obtain a tile number of each tile;
and constructing a row key according to the tile number, the time information and the equipment information.
6. The mass data read-write method based on distributed storage according to claim 5, wherein the row key is constructed by the following steps:
and taking the prefix of the spatial information as the prefix of a row key and taking the acquisition time and the acquisition equipment as suffixes.
7. A mass data read-write system based on distributed storage is characterized by comprising:
the acquisition module is used for acquiring the acquired data of the high-precision map and storing the acquired data into the Hbase database;
the building module is used for building a row key of the Hbase database according to the time information, the equipment information and the spatial information of the collected data;
the establishing module is used for storing the description information of the acquired data in an elastic search database and taking a row key of an Hbase database as a universal unique identifier; establishing a data index according to the description information and the universal unique identifier;
and the matching module is used for matching the data reading and writing request of the user with the data index and executing corresponding reading and writing operation on the matched data in the Hbase database.
8. The mass data reading and writing system based on distributed storage according to claim 7, wherein the building module comprises an analysis unit, a segmentation unit, and a building unit,
the analysis unit is used for analyzing the collected data;
the dividing unit is used for carrying out tile division on the analyzed data to obtain the tile number of each tile;
and the construction unit is used for constructing a row key according to the tile number, the time information and the equipment information.
9. An electronic device, comprising: one or more processors; storage means for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the mass data read/write method based on distributed storage according to any one of claims 1 to 6.
10. A computer-readable medium, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the mass data reading and writing method based on distributed storage according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111317987.6A CN114020750A (en) | 2021-11-09 | 2021-11-09 | A system and method for reading and writing massive data based on distributed storage |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111317987.6A CN114020750A (en) | 2021-11-09 | 2021-11-09 | A system and method for reading and writing massive data based on distributed storage |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114020750A true CN114020750A (en) | 2022-02-08 |
Family
ID=80062441
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111317987.6A Pending CN114020750A (en) | 2021-11-09 | 2021-11-09 | A system and method for reading and writing massive data based on distributed storage |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114020750A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114840612A (en) * | 2022-05-09 | 2022-08-02 | 中国人民解放军军事科学院军事医学研究院 | Massive meteorological element data storage and query method and system |
| CN114969036A (en) * | 2022-05-24 | 2022-08-30 | 武汉虹旭信息技术有限责任公司 | Data retrieval method and device |
| CN116610684A (en) * | 2023-04-20 | 2023-08-18 | 北京海致科技集团有限公司 | Data read-write method and device based on distributed consistency protocol |
| CN118409716A (en) * | 2024-07-02 | 2024-07-30 | 成都山莓科技有限公司 | A data writing management method, device and medium based on server hyper-convergence |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104331432A (en) * | 2014-10-22 | 2015-02-04 | 江苏瑞中数据股份有限公司 | Power grid mass time series data access method applicable to section access mode |
| CN109635068A (en) * | 2018-12-20 | 2019-04-16 | 中国地质大学（武汉） | Mass remote sensing data high-efficiency tissue and method for quickly retrieving under cloud computing environment |
| CN110347722A (en) * | 2019-07-11 | 2019-10-18 | 软通智慧科技有限公司 | Data acquisition method, device, equipment and storage medium based on HBase |
| CN111061806A (en) * | 2019-11-21 | 2020-04-24 | 中国航空无线电电子研究所 | Storage method and networked access method for distributed massive geographic tiles |
| CN111352956A (en) * | 2020-02-24 | 2020-06-30 | 交通运输部水运科学研究所 | Acquisition and storage system for shipping big data |
-
2021
- 2021-11-09 CN CN202111317987.6A patent/CN114020750A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104331432A (en) * | 2014-10-22 | 2015-02-04 | 江苏瑞中数据股份有限公司 | Power grid mass time series data access method applicable to section access mode |
| CN109635068A (en) * | 2018-12-20 | 2019-04-16 | 中国地质大学（武汉） | Mass remote sensing data high-efficiency tissue and method for quickly retrieving under cloud computing environment |
| CN110347722A (en) * | 2019-07-11 | 2019-10-18 | 软通智慧科技有限公司 | Data acquisition method, device, equipment and storage medium based on HBase |
| CN111061806A (en) * | 2019-11-21 | 2020-04-24 | 中国航空无线电电子研究所 | Storage method and networked access method for distributed massive geographic tiles |
| CN111352956A (en) * | 2020-02-24 | 2020-06-30 | 交通运输部水运科学研究所 | Acquisition and storage system for shipping big data |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114840612A (en) * | 2022-05-09 | 2022-08-02 | 中国人民解放军军事科学院军事医学研究院 | Massive meteorological element data storage and query method and system |
| CN114969036A (en) * | 2022-05-24 | 2022-08-30 | 武汉虹旭信息技术有限责任公司 | Data retrieval method and device |
| CN116610684A (en) * | 2023-04-20 | 2023-08-18 | 北京海致科技集团有限公司 | Data read-write method and device based on distributed consistency protocol |
| CN116610684B (en) * | 2023-04-20 | 2025-01-28 | 北京海致科技集团有限公司 | A data reading and writing method and device based on distributed consistency protocol |
| CN118409716A (en) * | 2024-07-02 | 2024-07-30 | 成都山莓科技有限公司 | A data writing management method, device and medium based on server hyper-convergence |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114020750A (en) | A system and method for reading and writing massive data based on distributed storage | |
| WO2015078238A1 (en) | Dispatching map matching tasks by cluster server in internet of vehicles | |
| Mehmood et al. | Distributed real-time ETL architecture for unstructured big data | |
| CN113704291A (en) | Data query method and device, storage medium and electronic equipment | |
| CN111258978A (en) | Data storage method | |
| US20150074115A1 (en) | Distributed storage of data | |
| CN106557531B (en) | Method, apparatus and storage medium for converting complex structured objects into flattened data | |
| WO2023056934A1 (en) | Data processing method and apparatus, and electronic device | |
| CN111723161B (en) | A data processing method, device and equipment | |
| CN113918659A (en) | Data manipulation method, device, storage medium and electronic device | |
| CN111950857A (en) | Index system management method and device based on service indexes and electronic equipment | |
| CN111581930A (en) | Online form data processing method and device, electronic equipment and readable medium | |
| CN116108233A (en) | Data processing method, device, equipment and storage medium | |
| CN112307061B (en) | Method and device for querying data | |
| CN111241137B (en) | Data processing method, device, electronic equipment and storage medium | |
| CN113312331B (en) | Data migration method, device, system, electronic device and computer readable medium | |
| US10169083B1 (en) | Scalable method for optimizing information pathway | |
| CN111143355B (en) | Data processing method and device | |
| WO2023001281A1 (en) | Table data processing method and apparatus, terminal, and storage medium | |
| CN117931813A (en) | Lake bin metadata change determining method, device, equipment and medium | |
| CN116821438A (en) | Database-based data processing method, device, medium and equipment | |
| CN113157695B (en) | Data processing method and device, readable medium and electronic equipment | |
| CN115630093A (en) | Data query method, device, medium and electronic equipment | |
| CN110941683B (en) | Method, device, medium and electronic equipment for acquiring object attribute information in space | |
| WO2023065937A1 (en) | Data processing method and apparatus, and readable medium and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |