CN110825711A - Method and device for transmitting data in quick partitioning mode based on Flume - Google Patents
Method and device for transmitting data in quick partitioning mode based on Flume Download PDFInfo
- Publication number
- CN110825711A CN110825711A CN201910992613.0A CN201910992613A CN110825711A CN 110825711 A CN110825711 A CN 110825711A CN 201910992613 A CN201910992613 A CN 201910992613A CN 110825711 A CN110825711 A CN 110825711A
- Authority
- CN
- China
- Prior art keywords
- file
- data
- directory
- service
- log
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Debugging And Monitoring (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a method and a device for transmitting data in a partitioned mode quickly based on flash. The method comprises the steps of receiving service reporting data, wherein the service reporting data at least comprises one service type; recording the received service reporting data into a corresponding log file according to the service type; monitoring whether the log file directory changes or not; and when the log file directory changes, sending the file to a file directory of the distributed file system. The method and the device solve the technical problem that the data transmission speed is low when data are transmitted based on flash. By the method and the device, the data transmission speed can be improved without configuring the regular expression.
Description
Technical Field
The application relates to the field of data processing, in particular to a method and a device for transmitting data based on flash quick partitioning.
Background
The flash is a distributed, reliable and highly available system for collecting, aggregating and transmitting mass logs, and supports various data senders customized in the log system for collecting data.
The inventor finds that, when the business data needs to be transmitted to the hadoop distributed file system by using flash, for different types of business data, partitions need to be stored in different directories of the distributed file system. Generally, a regular expression configuration mode is adopted, and therefore the file content needs to be analyzed to extract the partition type field, and the data transmission speed is low.
Aiming at the problem that the data transmission speed is low when data is transmitted based on flash in the related technology, an effective solution is not provided at present.
Disclosure of Invention
The application mainly aims to provide a method and a device for transmitting data based on flash fast partitioning, so as to solve the problem that the data transmission speed is slow when the data is transmitted based on flash.
To achieve the above object, according to one aspect of the present application, there is provided a method for transmitting data based on flash fast partition.
The method for transmitting data based on the flash fast partition comprises the following steps: receiving service reporting data, wherein the service reporting data at least comprises one service type; recording the received service reporting data into a corresponding log file according to the service type; monitoring whether the log file directory changes or not; and when the log file directory changes, sending the file to a file directory of the distributed file system.
Further, when the log file directory changes, sending the file to the file directory of the distributed file system includes:
analyzing the file name prefix in the log file directory to obtain a data type;
and setting a file name variable by adopting a file name prefix, and configuring the file name of the directory of the distributed file system according to the file name variable.
Further, when the log file directory changes, sending the file to the file directory of the distributed file system includes:
analyzing the file name prefix in the log file directory to obtain a data type;
and the log files are respectively transmitted to corresponding directories of the distributed file system based on flash according to the file prefix names.
Further, the monitoring whether the log file directory changes includes:
and configuring the flash to monitor whether the log file Directory changes or not by using a boosting Directory Source.
In order to achieve the above object, according to another aspect of the present application, there is provided a method for transmitting data based on flash fast partition, including: generating service data, wherein the service reporting data at least comprises one service type; reporting the service data, and recording the reported service data into a corresponding log file according to the service type; and if the log file directory changes, reporting the file to a file directory of the distributed file system.
To achieve the above object, according to still another aspect of the present application, there is provided an apparatus for transmitting data based on flash fast partition.
The device for transmitting data based on the flash fast partition comprises the following components: a receiving module, configured to receive service reporting data, where the service reporting data at least includes one service type; the recording module is used for recording the received service reporting data into a corresponding log file according to the service type; the monitoring module is used for monitoring whether the log file directory changes; and the sending module is used for sending the file to a file directory of the distributed file system when the log file directory changes.
Further, the sending module is used for
Analyzing the file name prefix in the log file directory to obtain a data type;
and setting a file name variable by adopting a file name prefix, and configuring the file name of the directory of the distributed file system according to the file name variable.
Further, the sending module is used for
Analyzing the file name prefix in the log file directory to obtain a data type;
and the log files are respectively transmitted to corresponding directories of the distributed file system based on flash according to the file prefix names.
Further, the monitoring module is configured to monitor the monitoring signal
And configuring the flash to monitor whether the log file Directory changes or not by using a boosting Directory Source.
In order to achieve the above object, according to still another aspect of the present application, there is provided an apparatus for transmitting data based on flash fast partition, including: a generating module, configured to generate service data, where the service reporting data at least includes one service type; a recording module, configured to report the service data and record the reported service data into a corresponding log file according to the service type; and the reporting module is used for reporting the file to the file directory of the distributed file system when the log file directory changes.
In the method and the device for transmitting data based on flash fast partitioning in the embodiment of the application, the purpose of sending files to the file directories of the distributed file system when the log file directories are changed is achieved by adopting a mode of receiving the service report data and recording the received service report data into the corresponding log files according to the service types and monitoring whether the log file directories are changed, so that the technical effect of increasing the data transmission speed is achieved, and the technical problem of low data transmission speed when the data are transmitted based on flash is solved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, serve to provide a further understanding of the application and to enable other features, objects, and advantages of the application to be more apparent. The drawings and their description illustrate the embodiments of the invention and do not limit it. In the drawings:
FIG. 1 is a flowchart illustrating a method for transmitting data based on flash fast partition according to a first embodiment of the present application;
FIG. 2 is a flowchart illustrating a method for transmitting data based on flash fast partition according to a second embodiment of the present application;
FIG. 3 is a flowchart illustrating a method for transmitting data based on flash fast partition according to a third embodiment of the present application;
FIG. 4 is a flowchart illustrating a method for transmitting data based on flash fast partition according to a fourth embodiment of the present application;
FIG. 5 is a diagram illustrating an apparatus for transmitting data based on flash fast partition according to another embodiment of the present application;
FIG. 6 is a schematic structural diagram of an apparatus for transmitting data based on flash fast partition according to an embodiment of the present application;
fig. 7 is a schematic diagram of an implementation principle according to an embodiment of the present application.
Detailed Description
In order to make the technical solutions better understood by those skilled in the art, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only partial embodiments of the present application, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that the terms "first," "second," and the like in the description and claims of this application and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It should be understood that the data so used may be interchanged under appropriate circumstances such that embodiments of the application described herein may be used. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
In this application, the terms "upper", "lower", "left", "right", "front", "rear", "top", "bottom", "inner", "outer", "middle", "vertical", "horizontal", "lateral", "longitudinal", and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings. These terms are used primarily to better describe the present application and its embodiments, and are not used to limit the indicated devices, elements or components to a particular orientation or to be constructed and operated in a particular orientation.
Moreover, some of the above terms may be used to indicate other meanings besides the orientation or positional relationship, for example, the term "on" may also be used to indicate some kind of attachment or connection relationship in some cases. The specific meaning of these terms in this application will be understood by those of ordinary skill in the art as appropriate.
Furthermore, the terms "mounted," "disposed," "provided," "connected," and "sleeved" are to be construed broadly. For example, it may be a fixed connection, a removable connection, or a unitary construction; can be a mechanical connection, or an electrical connection; may be directly connected, or indirectly connected through intervening media, or may be in internal communication between two devices, elements or components. The specific meaning of the above terms in the present application can be understood by those of ordinary skill in the art as appropriate.
Technical terms in the present application are explained as follows:
the flash is a high-availability, high-reliability and distributed system for acquiring, aggregating and transmitting mass logs, which is provided by Cloudera, and supports various data senders customized in a log system for collecting data; at the same time, flash provides the ability to simply process data and write to various data recipients (customizable).
Server, a piece of computer software that manages resources and provides services to users, is generally divided into file servers (which enable users to access files on other computers), database servers, and application servers.
Regular expressions, a concept of computer science, are commonly used to retrieve and replace text that conforms to a certain pattern (rule).
Hdfs, short for Hadoop distribution File System, is a distributed File System
Source, receiving data from a data generator and passing the received data to one or more channel channels in the event format of Flume, which provides various data reception modes, such as Avro, thread, twitter, Spooling Directory, etc.
Sink, stores data to a centralized storage such as Hbase, HDFS, Kafka, Hive, etc., which consumes data (events) from channals and passes it to the destination.
The interceptor in the interrupt, Flume reads events from Source and sends them to Sink, adds some useful information in the events header, or filters the contents of events to complete the preliminary data cleaning.
The data warehouse is a strategic set which provides all types of data support for decision making processes of all levels of enterprises.
According to the method, the server receives the reported data of different services, the service type is used as the log file name, then the file name prefix is analyzed to obtain the corresponding directory of the data type which is transmitted to hdfs through the flash, and the service data partitioning is realized. The data transmission speed can be improved without configuring a regular expression.
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
As shown in fig. 1, the method includes steps S101 to S104 as follows:
step S101, receiving service reported data,
the service reporting data at least comprises one service type.
Step S102, recording the received service reporting data into a corresponding log file according to the service type;
step S103, monitoring whether the log file directory changes or not;
and step S104, when the log file directory changes, the file is sent to a file directory of the distributed file system.
The method is used for a server side and used for receiving service reporting data, wherein the service reporting data at least comprises one service type; recording the received service reporting data into a corresponding log file according to the service type; monitoring whether the log file directory changes or not; and when the log file directory changes, sending the file to a file directory of the distributed file system.
Specifically, when the server receives the service reporting data, the server records the received data to a log file of a/data/log directory according to the service type of the reporting data. And (3) pre-configuring the Flume to monitor/data/log directory changes by using a boosting directory Source, so that when a new file exists in the directory, the file can be automatically sent to the directory of the distributed file system set by the sink.
From the above description, it can be seen that the following technical effects are achieved by the present application:
in the embodiment of the application, by monitoring whether the log file directory changes or not in a mode of receiving the service report data and recording the received service report data into the corresponding log file according to the service type, the purpose of sending the file to the file directory of the distributed file system when the log file directory changes is achieved, so that the technical effect of accelerating the data transmission speed is achieved, and the technical problem of low data transmission speed when data are transmitted based on flash is solved.
According to the embodiment of the present application, as a preferred embodiment in the present application, as shown in fig. 2, when the log file directory changes, the sending the file to the file directory of the distributed file system includes:
step 201, analyzing the file name prefix in the log file directory to obtain the data type;
step 202, setting a file name variable by using a file name prefix, and configuring the file name of the directory of the distributed file system according to the file name variable.
Specifically, the log file name is parsed, the $ { filename } variable is set using the prefix, and then the Hdfs Sink directory/flume/$ { filename } is configured with the $ { filename } variable.
According to the embodiment of the present application, as a preferred embodiment in the present application, as shown in fig. 3, when the log file directory changes, the sending the file to the file directory of the distributed file system includes:
step 301, analyzing the file name prefix in the log file directory to obtain the data type;
and step 302, respectively transmitting the log files to corresponding directories of the distributed file system based on flash according to the file prefix names.
Specifically, the log file is respectively transmitted to one or more directories of hdfs by using flash according to the prefix name, so that data is stored in the hdfs in a partitioned manner according to the service type.
Taking the detailed description of the operation flow under the records of the three types of log files A.log, B.log and C.log which need to be transmitted to hdfs, such as/flux/A,/flux/B and/flux/C as an example, the three log files A.log, B.log and C.log can be respectively transmitted to three directories of hdfs, flux/A,/flux/B and/flux/C by using flux according to the prefix name A, B, C, and the data can be stored in hdfs in a partitioning manner according to the service type.
In another embodiment of the present application, a method for transmitting data based on flash fast partition is further provided, as shown in fig. 4, specifically including:
step 401, generating service data, wherein the service reporting data at least comprises one service type;
step 402, reporting the service data, and recording the reported service data into a corresponding log file according to the service type;
step 403, reporting the file to a file directory of the distributed file system if the log file directory changes.
The method is used for a client to receive service reporting data, wherein the service reporting data at least comprises one service type; recording the received service reporting data into a corresponding log file according to the service type; monitoring whether the log file directory changes or not; and when the log file directory changes, sending the file to a file directory of the distributed file system.
Specifically, when the server receives the service reporting data, the server records the received data to a log file of a/data/log directory according to the service type of the data reported by the client. And (3) pre-configuring the Flume to monitor/data/log directory changes by using a SpoolingDirectory Source, so that when a new file exists in the directory, the file can be automatically sent to the directory of the distributed file system set by the sink.
It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer-executable instructions and that, although a logical order is illustrated in the flowcharts, in some cases, the steps illustrated or described may be performed in an order different than presented herein.
According to an embodiment of the present application, there is also provided an apparatus for transmitting data based on flash fast partition, which is used for implementing the above method, and as shown in fig. 5, the apparatus includes: a receiving module 10, configured to receive service reporting data, where the service reporting data at least includes one service type; a recording module 11, configured to record the received service reporting data into a corresponding log file according to the service type; the monitoring module 12 is used for monitoring whether the log file directory changes; and the sending module 13 is configured to send the file to a file directory of the distributed file system when the log file directory changes.
According to the embodiment of the present application, as a preferred choice in the embodiment, the sending module 13 is configured to analyze a filename prefix in the log file directory to obtain a data type; and setting a file name variable by adopting a file name prefix, and configuring the file name of the directory of the distributed file system according to the file name variable.
The module is used for a server side and receiving service reporting data, wherein the service reporting data at least comprises one service type; recording the received service reporting data into a corresponding log file according to the service type; monitoring whether the log file directory changes or not; and when the log file directory changes, sending the file to a file directory of the distributed file system.
Specifically, when the server receives the service reporting data, the server records the received data to a log file of a/data/log directory according to the service type of the reporting data. And (3) pre-configuring the Flume to monitor/data/log directory changes by using a boosting directory Source, so that when a new file exists in the directory, the file can be automatically sent to the directory of the distributed file system set by the sink.
According to the embodiment of the present application, as a preferred choice in the embodiment, the sending module 13 is configured to analyze a filename prefix in the log file directory to obtain a data type; and the log files are respectively transmitted to corresponding directories of the distributed file system based on flash according to the file prefix names.
Specifically, the log file name is parsed, the $ { filename } variable is set using the prefix, and then the Hdfs Sink directory/flume/$ { filename } is configured with the $ { filename } variable.
According to the embodiment of the present application, as a preferred preference in the embodiment, the monitoring module 12 is configured to configure the flash to monitor whether the log file Directory changes by using a boosting Directory Source.
Specifically, the log file is respectively transmitted to one or more directories of hdfs by using flash according to the prefix name, so that data is stored in the hdfs in a partitioned manner according to the service type.
Taking the detailed description of the operation flow under the records of the three types of log files A.log, B.log and C.log which need to be transmitted to hdfs, such as/flux/A,/flux/B and/flux/C as an example, the three log files A.log, B.log and C.log can be respectively transmitted to three directories of hdfs, flux/A,/flux/B and/flux/C by using flux according to the prefix name A, B, C, and the data can be stored in hdfs in a partitioning manner according to the service type.
According to another embodiment of the present application, there is also provided an apparatus for transmitting data based on flash fast partition, which is used for implementing the above method, and as shown in fig. 6, the apparatus includes: a generating module 20, configured to generate service data, where the service reporting data at least includes one service type; a recording module 21, configured to report the service data, and record the reported service data into a corresponding log file according to the service type; and the reporting module 22 is configured to report the file to a file directory of the distributed file system when the log file directory changes.
The module is used for a client to receive service reporting data, wherein the service reporting data at least comprises one service type; recording the received service reporting data into a corresponding log file according to the service type; monitoring whether the log file directory changes or not; and when the log file directory changes, sending the file to a file directory of the distributed file system.
Specifically, when the server receives the service reporting data, the server records the received data to a log file of a/data/log directory according to the service type of the data reported by the client. And (3) pre-configuring the Flume to monitor/data/log directory changes by using a SpoolingDirectory Source, so that when a new file exists in the directory, the file can be automatically sent to the directory of the distributed file system set by the sink.
It will be apparent to those skilled in the art that the modules or steps of the present application described above may be implemented by a general purpose computing device, they may be centralized on a single computing device or distributed across a network of multiple computing devices, and they may alternatively be implemented by program code executable by a computing device, such that they may be stored in a storage device and executed by a computing device, or fabricated separately as individual integrated circuit modules, or fabricated as a single integrated circuit module from multiple modules or steps. Thus, the present application is not limited to any specific combination of hardware and software.
As shown in fig. 7, the implementation principle of the present application is as follows:
the flash is a distributed, reliable and highly available system for collecting, aggregating and transmitting mass logs, and supports various data senders customized in the log system for collecting data. In the data warehouse construction process, generally, it is necessary to transmit service data to the hadoop hdfs file system by using flash, and for different types of service data, it is necessary to store different directories of hdfs in a partitioned manner. In the embodiment of the application, the method for transmitting data by using the flash fast partition does not need to analyze the content of the log file any more, and the transmission speed can be increased.
The operation flow is described in detail below by taking the/flux/A,/flux/B,/flux/C directories of three types of log files A.log, B.log and C.log needing to be transmitted to hdfs as examples:
in step S1, the server receives the data.
The server receives data from service report, which contains A, B, C three service types.
And step S2, recording the corresponding log file according to the service type.
And the server records the received data to three log files, namely A.log, B.log and C.log, of a/data/log directory according to the service type.
At step S3, FLUME listens for log directory changes.
And configuring the FLUME to monitor/data/log Directory changes by using a boosting Directory Source, and automatically sending the file to the hdfs Directory set by the sink when the Directory has a new file.
In step S4, the log file name is parsed.
The log filename is parsed, the $ { filename } variable is set using the prefix, and then the Hdfs Sink directory/fluorme/$ { filename } is configured with the $ { filename } variable.
And step S5, mapping to the distributed file system directory according to the file name.
The three log files of A.log, B.log and C.log can be respectively transmitted to three directories of hdfs/flash/A,/flash/B and/flash/C by using flash according to the prefix name A, B, C, so that data can be stored in hdfs in a partitioning manner according to the service type.
The above description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (10)
1. A method for transmitting data based on flash fast partition is characterized by comprising the following steps:
receiving service reporting data, wherein the service reporting data at least comprises one service type;
recording the received service reporting data into a corresponding log file according to the service type;
monitoring whether the log file directory changes or not;
and when the log file directory changes, sending the file to a file directory of the distributed file system.
2. The method for transmitting data based on flash fast partition according to claim 1, wherein when the log file directory changes, sending the file to the file directory of the distributed file system comprises:
analyzing the file name prefix in the log file directory to obtain a data type;
and setting a file name variable by adopting a file name prefix, and configuring the file name of the directory of the distributed file system according to the file name variable.
3. The method for transmitting data based on flash fast partition according to claim 1, wherein when the log file directory changes, sending the file to the file directory of the distributed file system comprises:
analyzing the file name prefix in the log file directory to obtain a data type;
and the log files are respectively transmitted to corresponding directories of the distributed file system based on flash according to the file prefix names.
4. The method for transmitting data based on flash fast partition according to claim 1, wherein the listening whether the log file directory changes comprises:
and configuring the flash to monitor whether the log file Directory changes or not by using a boosting Directory Source.
5. A method for transmitting data based on flash fast partition is characterized by comprising the following steps:
generating service data, wherein the service reporting data at least comprises one service type;
reporting the service data, and recording the reported service data into a corresponding log file according to the service type;
and if the log file directory changes, reporting the file to a file directory of the distributed file system.
6. An apparatus for transmitting data based on flash fast partition, comprising:
a receiving module, configured to receive service reporting data, where the service reporting data at least includes one service type;
the recording module is used for recording the received service reporting data into a corresponding log file according to the service type;
the monitoring module is used for monitoring whether the log file directory changes;
and the sending module is used for sending the file to a file directory of the distributed file system when the log file directory changes.
7. The apparatus for Flume-based fast partition data transmission according to claim 6, wherein the sending module is configured to send the data to the flash memory
Analyzing the file name prefix in the log file directory to obtain a data type;
and setting a file name variable by adopting a file name prefix, and configuring the file name of the directory of the distributed file system according to the file name variable.
8. The apparatus for Flume-based fast partition data transmission according to claim 6, wherein the sending module is configured to send the data to the flash memory
Analyzing the file name prefix in the log file directory to obtain a data type;
and the log files are respectively transmitted to corresponding directories of the distributed file system based on flash according to the file prefix names.
9. The apparatus for flash-based fast partition data transmission according to claim 6, wherein the snooping module is configured to snoop the data according to the instruction
And configuring the flash to monitor whether the log file Directory changes or not by using a boosting Directory Source.
10. An apparatus for transmitting data based on flash fast partition, comprising:
a generating module, configured to generate service data, where the service reporting data at least includes one service type;
a recording module, configured to report the service data and record the reported service data into a corresponding log file according to the service type;
and the reporting module is used for reporting the file to the file directory of the distributed file system when the log file directory changes.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910992613.0A CN110825711A (en) | 2019-10-17 | 2019-10-17 | Method and device for transmitting data in quick partitioning mode based on Flume |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910992613.0A CN110825711A (en) | 2019-10-17 | 2019-10-17 | Method and device for transmitting data in quick partitioning mode based on Flume |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110825711A true CN110825711A (en) | 2020-02-21 |
Family
ID=69549519
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910992613.0A Pending CN110825711A (en) | 2019-10-17 | 2019-10-17 | Method and device for transmitting data in quick partitioning mode based on Flume |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110825711A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130254878A1 (en) * | 2012-03-21 | 2013-09-26 | Owl Computing Technologies, Inc. | Method and apparatus for data transfer reconciliation |
| CN106569936A (en) * | 2016-09-26 | 2017-04-19 | 深圳盒子支付信息技术有限公司 | Method and system for acquiring scrolling log in real time |
| CN107145489A (en) * | 2016-03-01 | 2017-09-08 | 阿里巴巴集团控股有限公司 | A kind of information statistical method and device of the client application based on cloud platform |
| CN107347062A (en) * | 2017-06-19 | 2017-11-14 | 北京开数科技有限公司 | A kind of method, electronic equipment and the readable storage medium storing program for executing of daily record data processing |
| CN109542011A (en) * | 2018-12-05 | 2019-03-29 | 国网江西省电力有限公司信息通信分公司 | A kind of standardized acquisition system of multi-source heterogeneous monitoring data |
| CN109656999A (en) * | 2018-10-11 | 2019-04-19 | 平安科技（深圳）有限公司 | Method of data synchronization, equipment, storage medium and the device of big data quantity |
| CN110032496A (en) * | 2019-04-19 | 2019-07-19 | 杭州玳数科技有限公司 | A kind of log collection method and system for supporting diversified log merging |
-
2019
- 2019-10-17 CN CN201910992613.0A patent/CN110825711A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130254878A1 (en) * | 2012-03-21 | 2013-09-26 | Owl Computing Technologies, Inc. | Method and apparatus for data transfer reconciliation |
| CN107145489A (en) * | 2016-03-01 | 2017-09-08 | 阿里巴巴集团控股有限公司 | A kind of information statistical method and device of the client application based on cloud platform |
| CN106569936A (en) * | 2016-09-26 | 2017-04-19 | 深圳盒子支付信息技术有限公司 | Method and system for acquiring scrolling log in real time |
| CN107347062A (en) * | 2017-06-19 | 2017-11-14 | 北京开数科技有限公司 | A kind of method, electronic equipment and the readable storage medium storing program for executing of daily record data processing |
| CN109656999A (en) * | 2018-10-11 | 2019-04-19 | 平安科技（深圳）有限公司 | Method of data synchronization, equipment, storage medium and the device of big data quantity |
| CN109542011A (en) * | 2018-12-05 | 2019-03-29 | 国网江西省电力有限公司信息通信分公司 | A kind of standardized acquisition system of multi-source heterogeneous monitoring data |
| CN110032496A (en) * | 2019-04-19 | 2019-07-19 | 杭州玳数科技有限公司 | A kind of log collection method and system for supporting diversified log merging |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109245931B (en) | Log management and monitoring alarm realization method of container cloud platform based on kubernets | |
| US11182098B2 (en) | Optimization for real-time, parallel execution of models for extracting high-value information from data streams | |
| US9917735B2 (en) | System and method for big data aggregation in sensor network | |
| CN106571960B (en) | Log collection management system and method | |
| CN111543038B (en) | Network stream splicing using middleware stream splicing | |
| US10178067B1 (en) | Data center portal applications monitoring | |
| US11924240B2 (en) | Mechanism for identifying differences between network snapshots | |
| CN108282514B (en) | Distributed service establishing method and device | |
| US20210279265A1 (en) | Optimization for Real-Time, Parallel Execution of Models for Extracting High-Value Information from Data Streams | |
| CN110515912A (en) | Log processing method, device, computer installation and computer readable storage medium | |
| US20180248709A1 (en) | Managing channels in an open data ecosystem | |
| KR20140072044A (en) | Distributing multi-source push notifications to multiple targets | |
| US10826803B2 (en) | Mechanism for facilitating efficient policy updates | |
| CN107347062A (en) | A kind of method, electronic equipment and the readable storage medium storing program for executing of daily record data processing | |
| JP2013196688A (en) | Cache system and cache service providing method using network switch | |
| CN113839977A (en) | Message pushing method and device, computer equipment and storage medium | |
| US20170262508A1 (en) | Infrastructure management system having scalable storage architecture | |
| EP3744073B1 (en) | Discovery of middleboxes using traffic flow stitching | |
| US20190028501A1 (en) | Anomaly detection on live data streams with extremely low latencies | |
| Eyers et al. | Living in the present: on-the-fly information processing in scalable web architectures | |
| CN110620722B (en) | Order processing method and device | |
| CN112417050A (en) | Data synchronization method and device, system, storage medium and electronic device | |
| CN114443940A (en) | Message subscription method, device and equipment | |
| CN110825711A (en) | Method and device for transmitting data in quick partitioning mode based on Flume | |
| US9917738B2 (en) | Intelligent device data router |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200221 |
|
| RJ01 | Rejection of invention patent application after publication |