CN114385079A - Data writing method, data reading method and data reading and writing system - Google Patents
Data writing method, data reading method and data reading and writing system Download PDFInfo
- Publication number
- CN114385079A CN114385079A CN202111568460.0A CN202111568460A CN114385079A CN 114385079 A CN114385079 A CN 114385079A CN 202111568460 A CN202111568460 A CN 202111568460A CN 114385079 A CN114385079 A CN 114385079A
- Authority
- CN
- China
- Prior art keywords
- data
- server
- traffic
- traffic data
- routing server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 47
- 239000012634 fragment Substances 0.000 claims abstract description 67
- 238000012544 monitoring process Methods 0.000 claims abstract description 22
- 238000013467 fragmentation Methods 0.000 claims abstract description 21
- 238000006062 fragmentation reaction Methods 0.000 claims abstract description 21
- 238000012545 processing Methods 0.000 claims description 4
- 230000002776 aggregation Effects 0.000 claims description 3
- 238000004220 aggregation Methods 0.000 claims description 3
- 238000012806 monitoring device Methods 0.000 description 5
- 238000011161 development Methods 0.000 description 4
- 230000018109 developmental process Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 238000001514 detection method Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- 238000003860 storage Methods 0.000 description 2
- 241000566146 Asio Species 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
- G06F3/0613—Improving I/O performance in relation to throughput
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides a data writing method, a data reading method and a data reading and writing system. The method comprises the following steps: the method comprises the steps that a route server receives a data writing request sent by traffic monitoring equipment, wherein the data writing request comprises traffic data; the routing server responds to the data writing request and sends the traffic data to the corresponding fragment servers; the fragmentation server splits the traffic data to generate a plurality of traffic data blocks; the fragment server stores a plurality of traffic data blocks, so that the stability of data writing is improved, and the high efficiency of data writing is realized.
Description
[ technical field ] A method for producing a semiconductor device
The embodiment of the invention relates to the technical field of data storage, in particular to a data writing method, a data reading method and a data reading and writing system.
[ background of the invention ]
With the development of smart cities, smart traffic becomes a service system in smart city construction. In the actual application process of intelligent traffic, a vehicle, monitoring equipment, a cloud end, the Internet and other new-generation information technologies can be involved, and a large amount of real-time data needs to be stored or inquired in the process. As the development of the intelligent traffic is still in an early stage, a lot of problems are generated in the actual application process of the intelligent traffic, for example, the communication cannot be realized better due to the fact that services among vehicles, monitoring equipment and a cloud end are split; the problem of how to quickly implement storage or query of large amounts of real-time data. With the rapid development of modern information technology, data is used as an important component of the information age, and the storage or query of the data becomes more and more important.
In the related art, in the practical application process of intelligent traffic, the efficiency of writing or reading data is very low, and the stability is very poor, which obviously cannot adapt to the rapid development of the information era.
[ summary of the invention ]
In view of this, embodiments of the present invention provide a data writing method, a data reading method, and a data reading and writing system, so as to improve stability of data writing or data reading and achieve high efficiency of data writing or data reading.
In a first aspect, an embodiment of the present invention provides a data writing method, where the method is implemented based on a data reading and writing system, and the system includes a traffic monitoring device, at least one routing server, and multiple sharding servers; the method comprises the following steps:
the routing server receives a data writing request sent by the traffic monitoring equipment, wherein the data writing request comprises traffic data;
the routing server responds to the data writing request and sends the traffic data to a plurality of corresponding fragment servers;
the fragmentation server splits the traffic data to generate a plurality of traffic data blocks;
the fragmentation server stores a plurality of traffic data blocks.
Optionally, the system further comprises a configuration server; the routing server responds to the data writing request and sends the traffic data to a plurality of corresponding fragment servers, and the method comprises the following steps:
the routing server sends the traffic data to a configuration server;
the configuration server inquires out metadata corresponding to the traffic data and sends the metadata to the routing server;
and the routing server determines a plurality of fragment servers corresponding to the metadata and sends the traffic data to the plurality of fragment servers corresponding to the metadata.
In a second aspect, an embodiment of the present invention provides a data reading method, where the method is implemented based on a data reading and writing system, and the system includes at least one routing server, a configuration server, multiple sharding servers, and a client; the method comprises the following steps:
the client sends a data reading request to the routing server, wherein the data reading request comprises a data type;
the routing server sends the data type to the configuration server;
the configuration server inquires out metadata corresponding to the data type and sends the metadata to the routing server;
the routing server determines a plurality of fragment servers corresponding to the metadata and selects a designated fragment server from the plurality of fragment servers;
and the route server reads a plurality of traffic data blocks from the designated fragment server.
Optionally, after reading the corresponding multiple traffic data blocks from the designated fragmentation server, the routing server includes:
the routing server carries out aggregation processing on the plurality of traffic data blocks to generate traffic data;
and the routing server sends the traffic data to the client.
In a third aspect, an embodiment of the present invention provides a data reading and writing system, where the system includes: the traffic monitoring device, at least one of the routing servers, the configuration server, the plurality of sharding servers, and the client; the traffic monitoring equipment, the configuration server, the plurality of fragment servers and the client are connected with the routing server;
the routing server is used for receiving a data writing request sent by the traffic monitoring equipment, wherein the data writing request comprises traffic data; responding to the data writing request, and sending the traffic data to a plurality of corresponding fragment servers;
the fragment server is used for splitting the traffic data to generate a plurality of traffic data blocks; and storing a plurality of traffic data blocks.
Optionally, the system comprises a configuration server;
the routing server is also used for sending the traffic data to a configuration server;
the configuration server is used for inquiring the metadata corresponding to the traffic data and sending the metadata to the routing server;
the routing server is further configured to determine a plurality of segment servers corresponding to the metadata, and send the traffic data to the plurality of segment servers corresponding to the metadata.
Optionally, at least one of the routing server, the configuration server and the plurality of sharding servers are integrated into a MongoDB database.
In a fourth aspect, an embodiment of the present invention provides a data reading and writing system, where the system includes: the traffic monitoring device, at least one of the routing servers, the configuration server, the plurality of sharding servers, and the client; the traffic monitoring equipment, the configuration server, the plurality of fragment servers and the client are connected with the routing server;
the client is used for sending a data reading request to the routing server, wherein the data reading request comprises a data type;
the routing server is used for sending the data type to the configuration server;
the configuration server is used for inquiring the metadata corresponding to the data type and sending the metadata to the routing server;
the routing server is further configured to determine a plurality of fragment servers corresponding to the metadata, and select a designated fragment server from the plurality of fragment servers; and reading a plurality of traffic data blocks from the designated fragmentation server.
Optionally, the routing server is further configured to aggregate the plurality of traffic data blocks to generate traffic data; and sending the traffic data to the client.
Optionally, at least one of the routing server, the configuration server and the plurality of sharding servers are integrated into a MongoDB database.
In the technical scheme provided by the embodiment of the invention, a routing server receives a data writing request sent by traffic monitoring equipment, wherein the data writing request comprises traffic data; the routing server responds to the data writing request and sends the traffic data to the corresponding fragment servers; the fragmentation server splits the traffic data to generate a plurality of traffic data blocks; the fragmentation server stores a plurality of traffic data blocks; the method comprises the steps that a client sends a data reading request to a routing server, wherein the data reading request comprises a data type; the routing server sends the data type to a configuration server; the configuration server inquires out metadata corresponding to the data type and sends the metadata to the routing server; the routing server determines a plurality of fragment servers corresponding to the metadata and selects a designated fragment server from the plurality of fragment servers; the routing server reads a plurality of traffic data blocks from the designated fragment server, so that the stability of data writing or data reading is improved, and the high efficiency of data writing or data reading is realized.
[ description of the drawings ]
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a schematic structural diagram of a data reading and writing system according to an embodiment of the present invention;
fig. 2 is a flowchart of a data writing method according to an embodiment of the present invention;
fig. 3 is a flowchart of a method for sending data to a fragment server according to an embodiment of the present invention;
fig. 4 is a flowchart of a data reading method according to an embodiment of the present invention.
[ detailed description ] embodiments
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be understood that the described embodiments are only some embodiments of the invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The terminology used in the embodiments of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the examples of the present invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be understood that the term "and/or" as used herein is merely one type of associative relationship that describes an associated object, meaning that three types of relationships may exist, e.g., A and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship.
The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination" or "in response to a detection", depending on the context. Similarly, the phrases "if determined" or "if detected (a stated condition or event)" may be interpreted as "when determined" or "in response to a determination" or "when detected (a stated condition or event)" or "in response to a detection (a stated condition or event)", depending on the context.
An embodiment of the present invention provides a data reading and writing system, and fig. 1 is a schematic structural diagram of the data reading and writing system provided in the embodiment of the present invention, as shown in fig. 1, the system includes: the system comprises a traffic monitoring device 1, at least one routing server 2, a configuration server 3, a plurality of sharding servers 4 and a client 5. The traffic monitoring device 1, the configuration server 3, the plurality of slicing servers 4 and the client 5 are connected with the routing server 2.
The routing server 2 is used for receiving a data writing request sent by the traffic monitoring equipment 1, wherein the data writing request comprises traffic data; responding to the data writing request, and sending the traffic data to a plurality of corresponding fragment servers 4; the fragment server 4 is used for splitting the traffic data to generate a plurality of traffic data blocks; a plurality of traffic data blocks are stored.
The route server 2 is also used for sending the traffic data to the configuration server 3; the configuration server 3 is used for inquiring the metadata corresponding to the traffic data and sending the metadata to the routing server 2; the route server 2 is further configured to determine a plurality of segment servers 4 corresponding to the metadata, and send the traffic data to the plurality of segment servers 4 corresponding to the metadata.
At least one routing server 2, a configuration server 3 and a plurality of sharding servers 4 are integrated in the MongoDB database.
In one possible implementation, the client 5 is configured to send a data read request to the routing server 2, where the data read request includes a data type; the route server 2 is used for sending the data type to the configuration server 3; the configuration server 3 is used for inquiring metadata corresponding to the data type and sending the metadata to the routing server 2; the routing server 2 is further configured to determine a plurality of segment servers 4 corresponding to the metadata, and select a designated segment server from the plurality of segment servers 4; and reading a plurality of traffic data blocks from the designated fragmentation server.
The route server 2 is further configured to aggregate the plurality of traffic data blocks to generate traffic data; the traffic data is sent to the client 5.
At least one routing server 2, a configuration server 3 and a plurality of sharding servers 4 are integrated in the MongoDB database.
In the technical scheme of the data reading and writing system provided by the embodiment of the invention, a routing server receives a data writing request sent by traffic monitoring equipment, wherein the data writing request comprises traffic data; the routing server responds to the data writing request and sends the traffic data to the corresponding fragment servers; the fragmentation server splits the traffic data to generate a plurality of traffic data blocks; the fragmentation server stores a plurality of traffic data blocks; the method comprises the steps that a client sends a data reading request to a routing server, wherein the data reading request comprises a data type; the routing server sends the data type to a configuration server; the configuration server inquires out metadata corresponding to the data type and sends the metadata to the routing server; the routing server determines a plurality of fragment servers corresponding to the metadata, and selects a designated fragment server from the plurality of fragment servers; the routing server reads a plurality of traffic data blocks from the designated fragment server, so that the stability of data writing or data reading is improved, and the high efficiency of data writing or data reading is realized.
Fig. 2 is a flowchart of a data writing method according to an embodiment of the present invention, and as shown in fig. 2, the method includes:
In the embodiment of the invention, the traffic monitoring equipment sends the acquired traffic data to the routing server through the data writing request. The traffic monitoring equipment collects traffic data in real time and sends a data writing request through the Internet of things, wherein the data writing request comprises the traffic data. The traffic data includes, but is not limited to, road condition data, vehicle condition data, or vehicle trajectory data.
In the embodiment of the invention, before the routing server receives the data writing request sent by the traffic monitoring equipment, the data reading and writing system carries out optimized configuration according to the optimized configuration request input by the user. The user realizes the dynamic adjustment of the number of network threads by calling a service executor (ServiceExecutor) and self-adaptive (adaptive) configuration; invoking (multithreading) Boost, namely an asio network module, realizing network Input/Output (I/O) multiplexing and realizing separation of network I/O and disk I/O; the data read-write system controls the number of access threads of disk I/O through network I/O multiplexing and lock operation, reduces high system load caused by creation and consumption of a large number of threads, and finally improves concurrent read-write performance through the mode. The data read-write system can improve the throughput capacity of I/O by reading and writing data in parallel. The locking operation comprises a writing locking operation or a reading locking operation, and the priority level of the writing locking operation is greater than that of the reading locking operation.
And 102, the routing server responds to the data writing request and sends the traffic data to a plurality of corresponding fragment servers.
The routing server receives different traffic data and sends the different traffic data to the corresponding fragment servers according to the data configuration rules of the different traffic data.
In the embodiment of the invention, the fragment server is used for storing data, so that the high availability and consistency of the data can be ensured. The sharding server is a copy set in the production environment to prevent single point failure of the data blocks; all sharded servers have a primary shard that includes an undivided data set.
In the embodiment of the invention, when the data volume of the data reading and writing system is large, the data fragmentation needs to be operated in different machines so as to reduce the pressure of a Central Processing Unit (CPU), a memory and an I/O. The fragmentation technique of the MongoDB database includes vertical expansion or horizontal segmentation. The vertical expansion is to perform cluster expansion, and add more CPU, memory and disk space; the horizontal segmentation is to provide services uniformly through the cluster in a data slicing mode. The MongoDB database provides services in a fragmentation clustering mode, and can automatically switch the fragmentation servers when a fault occurs, so that the stability of data writing or data reading is improved.
Fig. 3 is a flowchart of a method for sending data to a shard server according to an embodiment of the present invention, and as shown in fig. 3, step 102 may specifically include:
The configuration server stores therein metadata of the traffic data. The configuration server receives the traffic data, loads the metadata according to the traffic data, and sends the metadata to the routing server. For example, when the traffic data includes the traffic data, the configuration server receives the traffic data, loads the metadata corresponding to the traffic data according to the traffic data, and sends the metadata corresponding to the traffic data to the routing server. The metadata includes an identifier corresponding to the traffic data, and the metadata is used for configuring a plurality of slicing servers.
And 1023, the routing server determines a plurality of fragment servers corresponding to the metadata and sends the traffic data to the plurality of fragment servers corresponding to the metadata.
For example, after the routing server obtains the metadata corresponding to the traffic data, a plurality of segment servers corresponding to the metadata are determined, and the traffic data is sent to the plurality of segment servers corresponding to the metadata.
And 103, the fragmentation server splits the traffic data to generate a plurality of traffic data blocks.
The traffic data is divided into a set as a basic unit, and the traffic data in the set is divided into a plurality of parts through a piece key. Wherein, the slice key is a key in the set, and one key can be placed in only one data block. The value of the slice key can be used as the basis for data splitting, and the value of the slice key is not variable, and the slice key is mainly used as an index for querying the data when the data is read. Wherein the value of the slice key is 512 bytes.
When the traffic data block is too small, the occurrence frequency of a certain value of the fragment key is high, and the traffic data can only be put into one traffic data block and cannot be split, so that the traffic data block cannot be migrated; too large a traffic data block may result in too much data in the traffic data block to migrate the traffic data block.
And 104, storing a plurality of traffic data blocks by the slicing server.
In the technical scheme of the data writing method provided by the embodiment of the invention, a routing server receives a data writing request sent by traffic monitoring equipment, wherein the data writing request comprises traffic data; the routing server responds to the data writing request and sends the traffic data to the corresponding fragment servers; the fragmentation server splits the traffic data to generate a plurality of traffic data blocks; the fragment server stores a plurality of traffic data blocks, so that the stability of data writing is improved, and the high efficiency of data writing is realized.
Fig. 4 is a flowchart of a data reading method according to an embodiment of the present invention, and as shown in fig. 4, the method includes:
After the client and the routing server form wireless network connection, the client sends a data reading request to the routing server through a wireless network. The data type includes, but is not limited to, road conditions, license plates, or vehicle trajectories.
In the embodiment of the present invention, the number of the routing servers may be one or more. When the number of the routing servers is multiple, the client sends the data reading request to the multiple routing servers, and the pressure of the routing servers for receiving the data reading request can be relieved.
The configuration server stores therein metadata of the data type. And the configuration server receives the data type, loads the metadata according to the data type and sends the metadata to the routing server. For example, when the data type includes a license plate, metadata corresponding to the license plate is loaded according to the license plate, and the metadata corresponding to the license plate data is sent to the routing server. Wherein the metadata includes an identification of a data type, and since one metadata may correspond to multiple sharded servers, the metadata is used to query the multiple sharded servers.
For example, after obtaining metadata corresponding to license plate data, the routing server determines a plurality of fragment servers corresponding to the metadata, and selects a designated fragment server from the plurality of fragment servers storing the license plate data.
For example, the routing server may randomly select a designated shard server from a plurality of shard servers corresponding to the metadata; or, the routing server may select a designated sharding server that is the most idle from a plurality of sharding servers corresponding to the metadata.
The fragmentation server stores traffic data in the form of a plurality of traffic data chunks.
In the embodiment of the present invention, after step 205, the method includes:
step 301, the routing server performs aggregation processing on the plurality of traffic data blocks to generate traffic data.
Step 302, the routing server sends the traffic data to the client.
In the technical scheme of the data reading method provided by the embodiment of the invention, a client sends a data reading request to a routing server, wherein the data reading request comprises a data type; the routing server sends the data type to a configuration server; the configuration server inquires out metadata corresponding to the data type and sends the metadata to the routing server; the routing server determines a plurality of fragment servers corresponding to the metadata, and selects a designated fragment server from the plurality of fragment servers; the route server reads a plurality of traffic data blocks from the designated fragment server, so that the stability of data reading is improved, and the high efficiency of data reading is realized.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (10)
1. A data writing method is characterized in that the method is realized based on a data reading and writing system, and the system comprises traffic monitoring equipment, at least one routing server and a plurality of fragment servers; the method comprises the following steps:
the routing server receives a data writing request sent by the traffic monitoring equipment, wherein the data writing request comprises traffic data;
the routing server responds to the data writing request and sends the traffic data to a plurality of corresponding fragment servers;
the fragmentation server splits the traffic data to generate a plurality of traffic data blocks;
the fragmentation server stores a plurality of traffic data blocks.
2. The method of claim 1, wherein the system further comprises a configuration server; the routing server responds to the data writing request and sends the traffic data to a plurality of corresponding fragment servers, and the method comprises the following steps:
the routing server sends the traffic data to a configuration server;
the configuration server inquires out metadata corresponding to the traffic data and sends the metadata to the routing server;
and the routing server determines a plurality of fragment servers corresponding to the metadata and sends the traffic data to the plurality of fragment servers corresponding to the metadata.
3. A data reading method is characterized in that the method is realized based on a data reading and writing system, and the system comprises at least one routing server, a configuration server, a plurality of fragment servers and a client; the method comprises the following steps:
the client sends a data reading request to the routing server, wherein the data reading request comprises a data type;
the routing server sends the data type to the configuration server;
the configuration server inquires out metadata corresponding to the data type and sends the metadata to the routing server;
the routing server determines a plurality of fragment servers corresponding to the metadata and selects a designated fragment server from the plurality of fragment servers;
and the route server reads a plurality of traffic data blocks from the designated fragment server.
4. The method of claim 3, wherein after the routing server reads the corresponding plurality of traffic data blocks from the designated sharding server, the method comprises:
the routing server carries out aggregation processing on the plurality of traffic data blocks to generate traffic data;
and the routing server sends the traffic data to the client.
5. A data reading and writing system, the system comprising: the system comprises traffic monitoring equipment, at least one routing server, a configuration server, a plurality of fragment servers and a client; the traffic monitoring equipment, the configuration server, the plurality of fragment servers and the client are connected with the routing server;
the routing server is used for receiving a data writing request sent by the traffic monitoring equipment, wherein the data writing request comprises traffic data; responding to the data writing request, and sending the traffic data to a plurality of corresponding fragment servers;
the fragment server is used for splitting the traffic data to generate a plurality of traffic data blocks; and storing a plurality of traffic data blocks.
6. The system of claim 5, wherein the routing server is further configured to send the traffic data to a configuration server;
the configuration server is used for inquiring the metadata corresponding to the traffic data and sending the metadata to the routing server;
the routing server is further configured to determine a plurality of segment servers corresponding to the metadata, and send the traffic data to the plurality of segment servers corresponding to the metadata.
7. The system of claim 5, wherein at least one of the routing server, the configuration server, and the sharding servers are integrated into a MongoDB database.
8. A data reading and writing system, the system comprising: the system comprises traffic monitoring equipment, at least one routing server, a configuration server, a plurality of fragment servers and a client; the traffic monitoring equipment, the configuration server, the plurality of fragment servers and the client are connected with the routing server;
the client is used for sending a data reading request to the routing server, wherein the data reading request comprises a data type;
the routing server is used for sending the data type to the configuration server;
the configuration server is used for inquiring the metadata corresponding to the data type and sending the metadata to the routing server;
the routing server is further configured to determine a plurality of fragment servers corresponding to the metadata, and select a designated fragment server from the plurality of fragment servers; and reading a plurality of traffic data blocks from the designated fragmentation server.
9. The system of claim 8, wherein the routing server is further configured to aggregate the plurality of traffic data blocks to generate traffic data; and sending the traffic data to the client.
10. The system of claim 8, wherein at least one of the routing server, the configuration server, and the sharding servers are integrated into a MongoDB database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111568460.0A CN114385079A (en) | 2021-12-21 | 2021-12-21 | Data writing method, data reading method and data reading and writing system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111568460.0A CN114385079A (en) | 2021-12-21 | 2021-12-21 | Data writing method, data reading method and data reading and writing system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114385079A true CN114385079A (en) | 2022-04-22 |
Family
ID=81198894
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111568460.0A Pending CN114385079A (en) | 2021-12-21 | 2021-12-21 | Data writing method, data reading method and data reading and writing system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114385079A (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105138615A (en) * | 2015-08-10 | 2015-12-09 | 北京思特奇信息技术股份有限公司 | Method and system for building big data distributed log |
| CN108829805A (en) * | 2018-06-06 | 2018-11-16 | 福建南威软件有限公司 | A kind of fragment storage method based on MongoDB |
| CN109344198A (en) * | 2018-09-19 | 2019-02-15 | 国网浙江省电力有限公司嘉兴供电公司 | Log system and sharding method based on MongoDB distributed type assemblies framework |
| CN111831663A (en) * | 2020-07-28 | 2020-10-27 | 北京首汽智行科技有限公司 | Data query method based on Mongobb database fragmentation technology |
-
2021
- 2021-12-21 CN CN202111568460.0A patent/CN114385079A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105138615A (en) * | 2015-08-10 | 2015-12-09 | 北京思特奇信息技术股份有限公司 | Method and system for building big data distributed log |
| CN108829805A (en) * | 2018-06-06 | 2018-11-16 | 福建南威软件有限公司 | A kind of fragment storage method based on MongoDB |
| CN109344198A (en) * | 2018-09-19 | 2019-02-15 | 国网浙江省电力有限公司嘉兴供电公司 | Log system and sharding method based on MongoDB distributed type assemblies framework |
| CN111831663A (en) * | 2020-07-28 | 2020-10-27 | 北京首汽智行科技有限公司 | Data query method based on Mongobb database fragmentation technology |
Non-Patent Citations (1)
| Title |
|---|
| 李宏志等: "基于MongoDB集群的遥感数据存储方法的研究", 《四川理工学院学报》, vol. 31, no. 31, pages 89 - 95 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10019459B1 (en) | Distributed deduplication in a distributed system of hybrid storage and compute nodes | |
| WO2017167171A1 (en) | Data operation method, server, and storage system | |
| CN112039979B (en) | Distributed data cache management method, device, equipment and storage medium | |
| CN110147407B (en) | Data processing method and device and database management server | |
| CN111522636B (en) | Application container adjusting method, application container adjusting system, computer readable medium and terminal device | |
| CN103488687A (en) | Searching system and searching method of big data | |
| CN108647266A (en) | A kind of isomeric data is quickly distributed storage, exchange method | |
| CN114238518A (en) | Data processing method, device, equipment and storage medium | |
| US8527565B2 (en) | Selecting and reassigning a blade for a logical partition for service scheduling of a blade server | |
| CN114385079A (en) | Data writing method, data reading method and data reading and writing system | |
| CN110798492B (en) | Data storage method and device and data processing system | |
| CN116866440A (en) | Cluster node selection scheduling method and device, electronic equipment and storage medium | |
| CN111008071A (en) | Task scheduling system, method and server | |
| CN110825732A (en) | Data query method and device, computer equipment and readable storage medium | |
| CN114461407A (en) | Data processing method, data processing device, distribution server, data processing system, and storage medium | |
| CN113905252A (en) | Data storage method and device for live broadcast room, electronic equipment and storage medium | |
| CN108595367B (en) | Server system based on computer cluster in local area network | |
| CN112817987A (en) | Method, device, equipment and storage medium for accessing distributed storage cluster | |
| CN111858657A (en) | Method and equipment for accelerating data parallel query based on high-frequency data processing | |
| CN111428140A (en) | High-concurrency data retrieval method, device, equipment and storage medium | |
| KR102571783B1 (en) | Search processing system performing high-volume search processing and control method thereof | |
| CN115481095A (en) | Heat monitoring-based file access path optimization method in integral system | |
| CN114328604B (en) | Method, device and medium for improving cluster data acquisition capacity | |
| CN114398371B (en) | Multi-copy slicing method, device, equipment and storage medium for database cluster system | |
| CN111797062B (en) | Data processing method, device and distributed database system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |