CN113468259A - Real-time data acquisition and storage method and system - Google Patents
Real-time data acquisition and storage method and system Download PDFInfo
- Publication number
- CN113468259A CN113468259A CN202111021500.XA CN202111021500A CN113468259A CN 113468259 A CN113468259 A CN 113468259A CN 202111021500 A CN202111021500 A CN 202111021500A CN 113468259 A CN113468259 A CN 113468259A
- Authority
- CN
- China
- Prior art keywords
- data
- different
- storage
- different types
- distributed message
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003860 storage Methods 0.000 title claims abstract description 44
- 238000000034 method Methods 0.000 title claims abstract description 20
- 230000005540 biological transmission Effects 0.000 claims abstract description 29
- 238000005096 rolling process Methods 0.000 claims description 17
- 238000013500 data storage Methods 0.000 claims description 9
- 238000004364 calculation method Methods 0.000 claims description 7
- 238000005192 partition Methods 0.000 claims description 4
- 238000009826 distribution Methods 0.000 claims description 3
- 238000012423 maintenance Methods 0.000 abstract description 5
- 230000009286 beneficial effect Effects 0.000 abstract description 2
- 230000006872 improvement Effects 0.000 description 9
- 238000013480 data collection Methods 0.000 description 7
- 238000011161 development Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000011084 recovery Methods 0.000 description 2
- 238000013524 data verification Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000004907 flux Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
Abstract
The invention discloses a real-time data acquisition and storage method and a real-time data acquisition and storage system, wherein the method comprises the following steps: collecting data and acquiring transmission meta information of the data; transmitting the data to a distributed message middleware according to the transmission meta information of the data, and carrying out asynchronous batch data transmission through the distributed message middleware; reading data sent by the distributed message middleware according to a data source list needing to be consumed, and temporarily storing the data after being classified and marked according to different types; and configuring different warehousing strategies according to different types of data, and writing the temporarily stored data into the distributed storage system. The invention has the beneficial effects that: the system can support the access, transmission and storage of large-scale data, and can support a large amount of data of different types when falling to the ground in real time, thereby improving the efficiency of data access; and different warehousing strategies are configured according to different types of data, so that low-cost expansion of access log data sources is realized, and the problems of automatic access of a large amount of classified data and differentiated operation and maintenance are solved.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a real-time data acquisition and storage method and system.
Background
In recent years, with the rapid development of technologies such as artificial intelligence, big data, cloud computing and the like, the digital economic era has brought forth new production elements represented by big data, and the continuous growth and innovative development of data drive are the main lines of enterprise digital transformation. Meanwhile, due to the development of the internet, particularly the mobile internet, data contacts between enterprises and clients are continuously abundant, the integration trend of online and offline is strengthened, and digital marketing becomes the most mature field of current data driving, so that the acquisition, transmission and warehousing of mass data are an important link for accessing the mass data and artificial intelligence of the enterprises, and the throughput, timeliness and demand access efficiency of data acquisition are of great importance.
The existing data acquisition and storage mode does not support consumption of multi-source data, and a data source cannot be updated in time under the condition of not restarting an application; the writing of the multi-target source is not supported, and the target source cannot be updated in time under the condition of not restarting the application. The efficiency of data acquisition and storage with fast daily data access iteration and large number is low, and the data throughput is low.
Disclosure of Invention
In order to solve the above problems, an object of the present invention is to provide a method and a system for real-time data collection and storage, which support access, transmission and landing of large-scale data, and achieve high availability, timeliness and throughput.
The invention provides a real-time data acquisition and storage method, which comprises the following steps:
collecting data and acquiring transmission meta information of the data;
transmitting the data to a distributed message middleware according to the transmission meta information of the data, and carrying out asynchronous batch data transmission through the distributed message middleware;
reading data sent by the distributed message middleware according to a data source list needing to be consumed, and temporarily storing the data after being classified and marked according to different types;
and configuring different warehousing strategies according to different types of data, and writing the temporarily stored data into the distributed storage system.
As a further improvement of the present invention, a binlog file is generated when data is collected and the generated binlog file is written to a local disk.
As a further improvement of the present invention, the transmission meta information includes: address, port, account, password, and data protocol transmitted to the distributed message middleware.
As a further improvement of the invention, the different storage strategies comprise different storage formats and different file rolling and splitting strategies.
As a further improvement of the present invention, the configuring different file rolling division policies includes configuring time windows of different periods, and the time windows are used for updating the file rolling policies.
The invention also provides a real-time data acquisition and storage system, which comprises a data receiving end and a batch computation data storage engine;
the data receiving end comprises a data access engine and a distributed message system, wherein the data access engine is used for providing external data receiving service; the distributed message system is used for receiving the data received by the data receiving end and sending asynchronous batch data to the batch calculation data drop engine;
the batch computation data falling library engine comprises:
the data reading plug-in obtains a data source list needing to be consumed from a metadata center, reads the data sent by the distributed message middleware, and temporarily stores the data after being classified and marked according to different types;
and the data writing plug-in writes the temporarily stored data into the distributed storage system according to different types of data and different storage strategies configured.
As a further improvement of the invention, the system also comprises a configuration strategy center, wherein the configuration strategy center configures different warehousing strategies according to different types and data volume of data to be consumed, and issues the strategy warehousing strategies to the data writing plug-in.
As a further improvement of the invention, the system also comprises a local disk, and the data received by the data receiving end is asynchronously written into the distributed message system and the local disk storage.
As a further improvement of the invention, the system further comprises a stream computation engine, wherein the stream computation engine is used for carrying out message distribution and forwarding to a downstream data application to provide consumption services.
As a further improvement of the present invention, the warehousing policy includes configuring different storage formats and configuring different file rolling and splitting policies, the configuring different file rolling and splitting policies includes configuring time windows of different periods, and the time windows are used for updating the file rolling policies.
As a further improvement of the invention, the system further comprises a data monitor and a configuration monitor, wherein the data monitor is used for monitoring the data read by the data reading plug-in and the data write by the data writing plug-in.
The invention has the beneficial effects that: the method can support access, transmission and storage of large-scale data, and can support a large amount of data of different types when falling to the ground in real time, so that the data access efficiency is improved, and the threshold and the cost for using the large data are reduced. By reading and writing the data into the plug-in based on the Apache flash and configuring different warehousing strategies according to different types of data, the low-cost expansion of the access log data source is realized, and the problems of automatic access of a large amount of classified data and differentiated operation and maintenance are solved. The throughput support of the system can achieve good expansibility, and the integrity of the received data can be kept under extreme conditions; the high-availability scheme of the data receiving service ensures that data is not lost, and the system can automatically return the data after the abnormal fault is solved.
Drawings
Fig. 1 is a schematic flow chart of a method for collecting and storing data in real time according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of a real-time data collection and storage system according to an embodiment of the present invention;
fig. 3 is a schematic diagram of a configuration policy center of a real-time data collection and storage system according to an embodiment of the present invention;
fig. 4 is a schematic system diagram of a real-time data collection and storage system according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present disclosure.
It should be noted that, if directional indications (such as up, down, left, right, front, and back … …) are involved in the embodiment of the present invention, the directional indications are only used to explain the relative positional relationship between the components, the movement situation, and the like in a specific posture (as shown in the drawing), and if the specific posture is changed, the directional indications are changed accordingly.
In addition, in the description of the present invention, the terms used are for illustrative purposes only and are not intended to limit the scope of the present disclosure. The terms "comprises" and/or "comprising" are used to specify the presence of elements, steps, operations, and/or components, but do not preclude the presence or addition of one or more other elements, steps, operations, and/or components. The terms "first," "second," and the like may be used to describe various elements, not necessarily order, and not necessarily limit the elements. In addition, in the description of the present invention, "a plurality" means two or more unless otherwise specified. These terms are only used to distinguish one element from another. These and/or other aspects will become apparent to those of ordinary skill in the art in view of the following drawings, and the description of the embodiments of the present invention will be more readily understood. The drawings are used for the purpose of illustrating embodiments of the disclosure only. One skilled in the art will readily recognize from the following description that alternative embodiments of the illustrated structures and methods of the present invention may be employed without departing from the principles of the present disclosure.
As shown in fig. 1, a method for collecting and storing data in real time in a storage device according to an embodiment of the present invention includes: collecting data and acquiring transmission meta information of the data; the transmission meta information includes, for example, information of an address, a port, an account number, a password, and a data protocol transmitted to the distributed message middleware.
Transmitting the data to a distributed message middleware according to the transmission meta information of the data, and carrying out asynchronous batch data transmission through the distributed message middleware;
reading data sent by the distributed message middleware according to a data source list needing to be consumed, and temporarily storing the data after being classified and marked according to different types; for example, the data types include storage types such as a relational database, a message system, a local file and the like, and data classification marks corresponding to different data storage types;
and configuring different warehousing strategies according to different types of data, and writing the temporarily stored data into the distributed storage system. For example, when different types of data are transmitted to the distributed message middleware, strategies such as partition number and copy set of the message can be configured according to data volume and importance; multiple copies of core data can be configured to prevent the down data of the machine from being lost, and a plurality of partitions can be configured for data with large volume, so that the throughput is increased; when data is written into the distributed storage system from the distributed message middleware, the strategy of rolling and splitting according to the data volume configuration file is as follows: slicing is performed according to a time window or according to the size of a file, so that the data storage efficiency and the data throughput are improved.
An optional implementation manner is that a rule of a buried point is issued, data collection is performed through a client side SDK, data compliance is verified after the data are received, and relevant index calculation is performed, for example: classification, size, format, number, etc., and performing subsequent operations after data verification is correct. According to the transmission meta information of the data, the transmission meta information comprises information such as an address, a port, an account number, a password and a data protocol transmitted to the distributed message middleware; and connecting the distributed message middleware to carry out asynchronous batch data transmission according to the transmission meta information. The success rate and failure rate of data receiving and sending can be used as statistical indexes to carry out data monitoring and early warning. When data is read and written, reading the data sent by the distributed message middleware according to a data source list needing to be consumed, and temporarily storing the data after being classified and marked according to different types; and configuring different warehousing strategies according to different types of data, and writing the temporarily stored data into the distributed storage system.
In an optional implementation mode, during data collection, a binlog file is generated after data is accessed, and the generated binlog file is written into a local disk; if the write fails, the receipt collection service is automatically closed and an early warning is given. The binlog file is mainly used for data backup and disaster recovery, and the data can be played back and additionally recorded after a system failure. Furthermore, data backup can be carried out on data of the local disk through the mixed cloud, and cold and hot hierarchical storage is adopted, so that the safety of the data is improved.
In an optional implementation manner, when data is written, data of different data types need to be configured with different storage formats and different file rolling division strategies according to the data amount, and time windows of different periods can be configured to update the file rolling strategies. For example: when the data volume becomes smaller, closing the target file according to the time window; and when the data volume is increased, closing the target file according to the file size. Different warehousing strategies are configured, so that the efficiency of data access is improved while a large amount of different types of data are landed in real time.
The invention also provides a real-time data acquisition and storage system, as shown in fig. 2-4, the system comprises a data receiving end and a batch computation data storage engine; wherein the content of the first and second substances,
the data receiving end comprises a data access engine and a distributed message system, and the data access engine is used for providing external data receiving service; the distributed message system is used for receiving the data received by the data receiving end and sending asynchronous batch data to the batch computation data drop engine.
The batch computation data falling library engine comprises: a data reading plug-in and a data writing plug-in; the data reading plug-in obtains a data source list needing to be consumed from the metadata center, reads data sent by the distributed message middleware, and temporarily stores the data after the data are classified and marked according to different types; and the data writing plug-in writes the temporarily stored data into the distributed storage system according to different types of data and different configured warehousing strategies. The data reading plug-in and the data writing plug-in are distributed message data source plug-ins and distributed message data source plug-ins based on Apache flash, but the distributed message data source plug-ins of the native Apache flash do not support consumption of multi-source data, and a data source cannot be updated in time under the condition of not restarting application; the distributed message data source plug-in of the native Apache flash does not support multi-target source writing, and the target source cannot be updated in time without restarting the application; reading data source configuration by adopting a native Apache flash, writing the configuration file of target source configuration and transmission pipeline strategy, restarting the service, and then finishing the data persistence of a message middleware; the access iteration is faster and the number of the daily data is larger, such as: thousands of classified data with large differences in volume and message volume become difficult to process when the complex environment meets the standards of operation and maintenance workload of Apache flux and stability of the whole data link. The data reading plug-in and the data writing plug-in of the embodiment of the application can support real-time storage of a large amount of data of different types by configuring different storage strategies. The batch computation data drop engine can ensure that data can be timely persisted to the distributed file storage system and used for offline business computation; through the batch calculation data falling engine, good expansibility can be achieved in the process of supporting the throughput of the system, the received data can be complete under extreme conditions, and a large amount of data of different types can be supported by falling to the ground in real time; therefore, the efficiency of data access and the throughput of the platform are improved, and the threshold and the cost for using big data by enterprises are reduced; and the low-cost expansion of the access log data source is realized, and the problems of automatic access of a large amount of classified data and differentiated operation and maintenance are solved.
According to an optional implementation mode, during data acquisition, a rule of a buried point is issued, data collection is carried out through a client side SDK, a transmission protocol is unified, key data are encrypted, privacy data are desensitized, then the key data are uploaded to a server side, namely, the key data are uploaded to a data receiving end of a system, and the data are asynchronously written into a distributed message system and a local disk for storage after the data receiving end receives the data. When the local disk is stored, a binlog file is generated and written into the local disk, and if the writing fails, the data access service is automatically closed and early warning is performed; the binlog file is mainly used for data backup and disaster recovery, and the data can be played back and additionally recorded after a system failure. The data written into the distributed message system is firstly subjected to message distribution and transfer to downstream data application through a stream calculation engine to provide consumption service, and then is subjected to message real-time library falling through a batch calculation engine.
In an optional implementation manner, the system further includes a configuration policy center, as shown in fig. 3, the configuration policy center configures different warehousing policies according to different types and data amounts of data to be consumed, and issues the policy warehousing policies to the data writing plug-in; the storage strategy comprises configuring different storage formats and different file rolling and dividing strategies, and the configuring of the different file rolling and dividing strategies comprises configuring time windows with different periods, wherein the time windows are used for updating the file rolling strategies. When data is written, the configuration policy center issues the configured warehousing policy to the data reading plug-in of the Apache flash in real time according to the difference of data types and the size of data volume, and can configure time windows of different periods according to different requirements to update the file rolling policy, for example: when the data volume becomes smaller, closing the target file according to the time window; and when the data volume is increased, closing the target file according to the file size.
In an optional implementation manner, the data receiving end may provide two data receiving services, HTTP and RPC, to the outside. The HTTP service performs high-availability proxy to prevent single point of failure; the RPC module is realized by adopting a high concurrency framework (such as Apache Dubbo based on JAVA language), and nodes are deployed for service provision. After the system receives the data, firstly, the data compliance verification and the related index calculation are performed, for example: classification, size, format, number of bars, etc.; and performing subsequent operation after the data is verified to be correct. And the data receiving end asynchronously writes the data into the distributed message system and the local disk for storage after receiving the data. According to the transmission meta information of the data, the transmission meta information comprises information such as an address, a port, an account number, a password and a data protocol transmitted to the distributed message middleware; and connecting the distributed message middleware to carry out asynchronous batch data transmission according to the transmission meta information. The success rate and failure rate of data receiving and sending can be used as statistical indexes to carry out data monitoring and early warning. When the batch computation data drop engine reads and writes data, reading the data sent by the distributed message system according to a data source list needing to be consumed, performing grouping initialization data consumption threads, writing the data into classification marks and then driving the classification marks into a memory pipeline; and configuring different warehousing strategies according to different types of data, and writing the data in the memory pipeline into the distributed storage system. The high-availability scheme of the data receiving service ensures that data is not lost, and the system can automatically return the data after the abnormal fault is solved; by accessing the strategy of uniformly managing and issuing the configuration metadata, the low-cost expansion of the access log data source is realized, and the problems of automatic access of a large amount of classified data and differentiated operation and maintenance are solved.
In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
Furthermore, those of ordinary skill in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments may be used in any combination.
It will be understood by those skilled in the art that while the present invention has been described with reference to exemplary embodiments, various changes may be made and equivalents may be substituted for elements thereof without departing from the scope of the invention. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the invention without departing from the essential scope thereof. Therefore, it is intended that the invention not be limited to the particular embodiment disclosed, but that the invention will include all embodiments falling within the scope of the appended claims.
Claims (10)
1. A real-time data acquisition and storage method is characterized by comprising the following steps:
collecting data and acquiring transmission meta information of the data;
transmitting the data to a distributed message middleware according to the transmission meta information of the data, and carrying out asynchronous batch data transmission through the distributed message middleware;
reading data sent by the distributed message middleware according to a data source list needing to be consumed, and temporarily storing the data after being classified and marked according to different types;
and configuring different warehousing strategies according to different types of data, and writing the temporarily stored data into the distributed storage system.
2. The method of claim 1, wherein a binlog file is generated when collecting data and the generated binlog file is written to a local disk.
3. The method of claim 1, wherein the transmission meta information comprises: address, port, account, password, and data protocol transmitted to the distributed message middleware.
4. The method of claim 1, wherein configuring different warehousing policies comprises configuring different storage formats and configuring different file rolling splitting policies.
5. The method of claim 4, wherein configuring different file scrolling splitting policies comprises configuring time windows of different periods, wherein the time windows are used for updating the file scrolling policies.
6. A real-time data acquisition and storage system is characterized by comprising a data receiving end and a batch computation data storage engine;
the data receiving end comprises a data access engine and a distributed message system, wherein the data access engine is used for providing external data receiving service; the distributed message system is used for receiving the data received by the data receiving end and sending asynchronous batch data to the batch calculation data drop engine;
the batch computation data falling library engine comprises:
the data reading plug-in obtains a data source list needing to be consumed from a metadata center, reads the data sent by the distributed message middleware, and temporarily stores the data after being classified and marked according to different types;
and the data writing plug-in writes the temporarily stored data into the distributed storage system according to different types of data and different storage strategies configured.
7. The system of claim 6, further comprising a configuration policy center, wherein the configuration policy center configures different warehousing policies according to different types and data amounts of data to be consumed, and issues the policy warehousing policies to the data writing plug-in.
8. The system of claim 6, further comprising a local disk, wherein the data received by the data receiver is asynchronously written to the distributed messaging system and the local disk storage.
9. The system of claim 6, further comprising a stream computation engine configured to perform message distribution and forwarding to downstream data applications to provide consumption services.
10. The system of claim 6, wherein the warehousing policy comprises configuring different storage formats and configuring different file rolling partition policies, and the configuring different file rolling partition policies comprises configuring time windows with different periods, and the time windows are used for updating the file rolling policies.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111021500.XA CN113468259A (en) | 2021-09-01 | 2021-09-01 | Real-time data acquisition and storage method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111021500.XA CN113468259A (en) | 2021-09-01 | 2021-09-01 | Real-time data acquisition and storage method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113468259A true CN113468259A (en) | 2021-10-01 |
Family
ID=77867933
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111021500.XA Pending CN113468259A (en) | 2021-09-01 | 2021-09-01 | Real-time data acquisition and storage method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113468259A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114500255A (en) * | 2022-03-01 | 2022-05-13 | 北京百度网讯科技有限公司 | Log data reporting method, device, equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103401934A (en) * | 2013-08-06 | 2013-11-20 | 广州唯品会信息科技有限公司 | Method and system for acquiring log data |
| CN110430232A (en) * | 2019-06-27 | 2019-11-08 | 苏州浪潮智能科技有限公司 | A kind of visual flume configuration operation method and device |
| CN111858833A (en) * | 2020-07-29 | 2020-10-30 | 广州海鹚网络科技有限公司 | Asynchronous indexing method and device, computer equipment and storage medium |
| CN111880930A (en) * | 2020-07-15 | 2020-11-03 | 杭州安恒信息技术股份有限公司 | Data distribution method and system |
-
2021
- 2021-09-01 CN CN202111021500.XA patent/CN113468259A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103401934A (en) * | 2013-08-06 | 2013-11-20 | 广州唯品会信息科技有限公司 | Method and system for acquiring log data |
| CN110430232A (en) * | 2019-06-27 | 2019-11-08 | 苏州浪潮智能科技有限公司 | A kind of visual flume configuration operation method and device |
| CN111880930A (en) * | 2020-07-15 | 2020-11-03 | 杭州安恒信息技术股份有限公司 | Data distribution method and system |
| CN111858833A (en) * | 2020-07-29 | 2020-10-30 | 广州海鹚网络科技有限公司 | Asynchronous indexing method and device, computer equipment and storage medium |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114500255A (en) * | 2022-03-01 | 2022-05-13 | 北京百度网讯科技有限公司 | Log data reporting method, device, equipment and storage medium |
| CN114500255B (en) * | 2022-03-01 | 2024-03-15 | 北京百度网讯科技有限公司 | Log data reporting method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113254466B (en) | Data processing method and device, electronic equipment and storage medium | |
| US20240143587A1 (en) | Temporal optimization of data operations using distributed search and server management | |
| CN109918349B (en) | Log processing method, log processing device, storage medium and electronic device | |
| US7783607B2 (en) | Decentralized record expiry | |
| CN105207826A (en) | Security attack alarm positioning system based on Spark big data platform of Tachyou | |
| CN109558450A (en) | A kind of automobile remote monitoring method and apparatus based on distributed structure/architecture | |
| CN111787055B (en) | Redis-based transaction mechanism and multi-data center oriented data distribution method and system | |
| CN112069264A (en) | Heterogeneous data source acquisition method and device, electronic equipment and storage medium | |
| CN113468259A (en) | Real-time data acquisition and storage method and system | |
| CN108415355A (en) | A kind of efficient identification system of big data | |
| CN114090529A (en) | Log management method, device, system and storage medium | |
| US20200104405A1 (en) | Data warehouse management and synchronization systems and methods | |
| CN109522349B (en) | Cross-type data calculation and sharing method, system and equipment | |
| CN111049898A (en) | Method and system for realizing cross-domain architecture of computing cluster resources | |
| CN107480189A (en) | A kind of various dimensions real-time analyzer and method | |
| CN111049846A (en) | Data processing method and device, electronic equipment and computer readable storage medium | |
| CN111274215A (en) | Distributed log unified management method | |
| US10990611B1 (en) | Adaptive data processing system and method | |
| CN106549983A (en) | The access method and terminal of a kind of database, server | |
| CN115391429A (en) | Time sequence data processing method and device based on big data cloud computing | |
| CN113626457A (en) | Method and system for realizing database and cache consistency by cache deletion retry mechanism | |
| Shona et al. | A survey on the data management in IoT | |
| CN115695587A (en) | Service data processing system, method, device and storage medium | |
| Racka | Apache Nifi As A Tool For Stream Processing Of Measurement Data | |
| CN115374101A (en) | Rail transit station level data management system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20211001 |