CN115268934A - Method and system for realizing real-time acquisition from Binlog to Doris based on Flink - Google Patents

Method and system for realizing real-time acquisition from Binlog to Doris based on Flink Download PDF

Info

Publication number
CN115268934A
CN115268934A CN202210951740.8A CN202210951740A CN115268934A CN 115268934 A CN115268934 A CN 115268934A CN 202210951740 A CN202210951740 A CN 202210951740A CN 115268934 A CN115268934 A CN 115268934A
Authority
CN
China
Prior art keywords
doris
information
data
data source
rowdata
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210951740.8A
Other languages
Chinese (zh)
Inventor
温天柱
陈吉平
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hangzhou Daishu Technology Co ltd
Original Assignee
Hangzhou Daishu Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou Daishu Technology Co ltd filed Critical Hangzhou Daishu Technology Co ltd
Priority to CN202210951740.8A priority Critical patent/CN115268934A/en
Publication of CN115268934A publication Critical patent/CN115268934A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00—Arrangements for software engineering
    • G06F8/40—Transformation of program code
    • G06F8/41—Compilation
    • G06F8/42—Syntactic analysis
    • G06F8/427—Parsing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G06F16/2282—Tablespace storage structures; Management thereof
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F8/00—Arrangements for software engineering
    • G06F8/70—Software maintenance or management
    • G06F8/71—Version control; Configuration management

Abstract

The application discloses a method and a system for realizing real-time acquisition from Binlog to Doris based on Flink, which relate to the technical field of data security and comprise the following steps: acquiring Binlog incremental log information of an original data source, and packaging the log information into a Flink data object RowData; configuring a NameMapping operator in a task script to acquire a mapping relation in a current task, and mapping and recombining information in RowData according to the mapping relation; and configuring Doris data source information, and constructing a DorisStreamLoad for writing data into the Doris data source according to the Doris data source information and the RowData information after mapping and recombination. According to the data transmission method and device, data information of different table structures can be written into the Doris data source table, and the data transmission delay is reduced.

Description

Method and system for realizing real-time acquisition from Binlog to Doris based on Flink
Technical Field
The application relates to the technical field of data security, in particular to a method and a system for achieving Binlog-Doris real-time acquisition based on Flink.
Background
The current real-time data acquisition has the following problems: a single canal component only provides a function of Binlog incremental log analysis of the MySQL database and cannot meet the requirement that data in the real-time digital warehouse model are collected to a Doris data source in real time; writing the data into the Doris data source is to write the data UpSert into the target table in a Doris Stream Load synchronous importing mode, and the throughput and the concurrency of the data cannot be guaranteed under the condition of big data; binlog records incremental logs of all tables in a database, but the prior art cannot write data information of different tables under different libraries and with different table structures into corresponding Doris tables; data synchronization is batch processing and non-streaming, and low delay and real-time performance of data cannot be guaranteed, and data continuous transmission of tasks cannot be guaranteed.
Disclosure of Invention
The method for achieving real-time collection from Billog to Doris based on Flink aims to collect Binlog incremental log information of a MySQL data source through canal, analyze the log information and write data into the Doris data source in a Doris Stream Load synchronous writing mode, and therefore a high-performance low-delay streaming data processing mode is provided, and real-time collection from the Billog to the Doris data is achieved.
In order to achieve the purpose, the following technical scheme is adopted in the application:
the application discloses a method for realizing Binlog-to-Doris real-time acquisition based on Flink, which comprises the following steps:
acquiring Binlog incremental log information of an original data source, and encapsulating the log information into a Flink data object RowData;
configuring a NameMapping operator in a task script to acquire a mapping relation in a current task, and mapping and recombining information in the RowData according to the mapping relation;
and configuring Doris data source information, and constructing DorisStreamLoad for writing data into the Doris data source according to the Doris data source information and the RowData information after mapping and recombination.
Preferably, the collecting Binlog increment log information of an original data source and encapsulating the log information into a Flink data object RowData includes:
starting a canal component, and performing regular matching on configuration parameters by using an openInputFormat method to realize subscription and consumption of a Binlog log of an original database;
calling a nextRecordInternal method, and packaging the collected log information into a Flink data object RowData, wherein the RowData comprises an original database name and a table name.
Preferably, the configuring the NameMapping operator in the task script to obtain the mapping relationship in the current task, and mapping and recombining the information in the RowData according to the mapping relationship includes:
configuring database mapping, schema mapping, tablemapping and field mapping parameters of nameMapping in a task script to obtain a mapping relation in a current task, wherein the mapping relation comprises a database name, a table name and a field name;
disassembling the RowData, and judging whether the disassembled information has a mapping relation;
if so, mapping and recombining the information according to the mapping relation in the current task, otherwise, keeping the original value.
Preferably, the configuring the Doris data source information and constructing a DorisStreamLoad for writing data into the Doris data source according to the Doris data source information and the RowData information after the mapping is recombined, includes:
configuring Doris data source information, wherein the Doris data source information comprises Doris Fe, be connection information, doris login user name and password;
calling a writeRecord method to analyze the mapped and recombined RowData to obtain an original database name, a table name and corresponding table structure information;
and constructing DorisStreamLoad according to the table structure information and the Doris data source information, and writing data into the Doris data source.
Preferably, the RowData further includes a data change type;
and constructing different Http requests according to the data change type and the DorisStreamLoad so as to synchronously send the data to the Doris data source.
A real-time acquisition system for realizing Binlog to Doris based on Flink comprises:
the data acquisition component is used for acquiring Binlog increment log information of an original data source and packaging the log information into a flash data object RowData;
the information matching component is used for configuring a NameMapping operator in the task script so as to obtain a mapping relation in the current task, and mapping and recombining the information in the RowData according to the mapping relation;
and the data writing component is used for configuring the Doris data source information and constructing the DorisStreamLoad for writing the data into the Doris data source according to the Doris data source information and the RowData information after mapping and recombination.
Preferably, the data acquisition assembly comprises:
the subscription unit is used for starting the canal component, performing regular matching on the configuration parameters by using an openInputFormat method, and realizing subscription and consumption of the Binlog log of the original database;
and the packaging unit is used for calling a nextRecordInternal method and packaging the collected log information into a Flink data object RowData, wherein the RowData contains an original database name and a table name.
Preferably, the data writing component includes:
the system comprises a configuration unit, a data processing unit and a data processing unit, wherein the configuration unit is used for configuring Doris data source information, and the Doris data source information comprises Doris Fe, be connection information, a Doris login user name and a password;
the analysis unit is used for calling a writeRecord method to analyze the mapped and recombined RowData to obtain an original database name, a table name and corresponding table structure information;
and the constructing unit is used for constructing the DorisStreamLoad according to the table structure information and the Doris data source information, and writing the data into the Doris data source.
An electronic device comprising a memory and a processor, the memory configured to store one or more computer instructions, wherein the one or more computer instructions are executable by the processor to implement a method for Flink-based Binlog to Doris-based real-time acquisition as any of the above.
A computer readable storage medium storing a computer program, which when executed, causes a computer to implement a method for acquiring Binlog to Doris in real time based on Flink as described in any one of the above.
The invention has the following beneficial effects:
compared with the prior art, the streaming data processing is realized based on the Flink computing framework, so that the delay of data transmission is reduced, and the consistent semantics of data are realized; data are issued in a DorisStreamLoad mode, and high-performance data writing of a Doris data source is achieved; and mapping all the table information acquired by the Binlog through a self-defined NameMappingFlatMap operator to realize writing the data information of different table structures into the Doris table.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive exercise.
Fig. 1 is a flow chart of a method for implementing Binlog to Doris real-time acquisition based on Flink in the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the accompanying drawings, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments obtained by a person skilled in the art based on the embodiments in the present application without making any creative effort belong to the protection scope of the present application.
The terms "first," "second," and the like in the claims and in the description of the present application are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order, it being understood that the terms so used are interchangeable under appropriate circumstances and are merely used to describe a distinguishing manner between similar elements in the embodiments of the present application and that the terms "comprising" and "having" and any variations thereof are intended to cover a non-exclusive inclusion such that a process, method, system, article, or apparatus that comprises a list of elements is not necessarily limited to those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
As shown in fig. 1, a method for realizing Binlog to Doris real-time acquisition based on Flink includes the following steps:
s110, acquiring Binlog incremental log information of an original data source, and packaging the log information into a Flink data object RowData.
According to an embodiment of the present invention, the method further includes: starting a canal component, and performing regular matching on configuration parameters by using an openInputFormat method to realize subscription and consumption of a Binlog log of an original database;
calling a nextRecordInternal method, and packaging the collected log information into a Flink data object RowData, wherein the RowData comprises an original database name and a table name.
Specifically, the invention utilizes a data acquisition component BinlogReader to monitor and acquire Binlog incremental log information in real time, exemplarily:
inheriting and realizing BaseRichInputFormat base classes, and defining binlogReader related parameters;
starting a cana component l, configuring parameters such as table and filter by an openInputFormat method to perform regular matching, subscribing and consuming a Binlog incremental log of a specified data source (namely a target data source) base table, and acquiring the change type of specified data by configuring cat.
Calling a method nextRecordInternal for processing streaming data, analyzing the Binlog incremental log information, and encapsulating the analyzed log information into a Flink data object RowData, wherein the data object RowData comprises but is not limited to the following information contents: database name (schema name), table name (tableme), pre-field-alteration content (before columns), post-field-alteration content (after columns).
And calling a collect method of a Flink framework, and transmitting the data object RowData to the NameMappingoperator of the next execution operator information matching component.
A Canal component is configured in the Binlogreader to monitor an original data source in real time, and Binlog incremental log information is collected, and processing and issuing of the log information are achieved internally.
S120, configuring a NameMapping operator in the task script to acquire a mapping relation in the current task, and mapping and recombining information in the RowData according to the mapping relation.
According to an embodiment of the present invention, the method further includes: configuring database mapping, schema mapping, tablemapping and field mapping parameters of nameMapping in a task script to obtain a mapping relation in a current task, wherein the mapping relation comprises a database name, a table name and a field name;
disassembling the RowData, and judging whether the disassembled information has a mapping relation;
if so, mapping and recombining the information according to the mapping relation in the current task, otherwise, keeping the original value.
Specifically, the invention utilizes an information matching component NameMappingFlatMap to carry out consistency matching on the information of the upstream and downstream data sources, and exemplarily comprises the following steps:
and acquiring a mapping relation in the current task by configuring database mapping, schema mapping, tablemapping and field mapping parameters of nameMapping operators in the task script, wherein the mapping relation comprises a database name, a table name, a field name and the like.
And (4) disassembling RowData of a data object issued by Binlogreader, mapping and recombining data in the RowData according to the obtained mapping relation, and keeping the original value if no information of the mapping relation exists.
And calling the collection method in the Flink framework again, and sending the recombined data such as the database information, the table name information, the table structure information and the like to a downstream data writing component DorisWriter.
The specific implementation of the FlatMapFaction in the Flink API interface realizes the one-to-one matching of information such as upstream and downstream library names, table names, field names and the like through configuration information, and realizes the consistency of upstream and downstream data source synchronous information.
S130, doris data source information is configured, and DorisstramLoad used for writing data into the Doris data source is constructed according to the Doris data source information and the RowData information after mapping and recombination.
According to an embodiment of the present invention, the method further includes: configuring Doris data source information, wherein the Doris data source information comprises Doris Fe, be connection information, doris login user name and password;
calling a writeRecord method to analyze the mapped and recombined RowData to obtain an original database name, a table name and corresponding table structure information;
and constructing DorisStreamLoad according to the table structure information and the Doris data source information, and writing data into the Doris data source.
Specifically, a data write component, dorisWriter, is utilized to write data into a Doris data source, illustratively:
inheriting and realizing a BaseRichOutputFormat base class, defining DorisWriter related parameters, calling an open method to configure related information of a Doris data source, wherein the Doris data source comprises but is not limited to Doris Fe, be connection information, a Doris login user name, a password and the like, establishing connection between a task server and the Doris data source, calling a processing method of streaming data, writeRecord, analyzing a mapped and recombined data object RowData sent by a NameMappingFlatMap at the upstream of the task, acquiring a database name (schema name) and a table name (tableName) in the recombined RowData and corresponding Namnmes, and constructing Doris LomdLomdoad according to the Doris data source information such as the table structure information, the Doris Fe, the Be connection information, the Doris login user name and the password; and finally, constructing different Http requests by combining the data change types (types) in the RowData, and issuing the specific data in the recombined data object RowData to a Doris data source in a synchronous mode.
The method specifically realizes RichOutputFormat in the Flink API by using DorisOutputFormat, thereby reading the recombined data object RowData transmitted from the upstream NameMappingFlatMap operator, and realizing the writing, deleting and updating of Doris data source data by adopting a Doris Stream Load mode.
Compared with the prior art, the method and the device realize the streaming data processing based on the Flink computing framework, reduce the delay of data transmission and realize the consistent semantics of data; meanwhile, data are issued in a DorisStreamLoad mode, and high-performance data writing of a Doris data source is achieved; all the table information collected by the Binlog is mapped through a customized NameMapplingFlatMap operator, and data information of different table structures can be written into the Doris table.
The invention also provides a system for realizing the method for realizing the real-time acquisition from Binlog to Doris based on Flink, which specifically comprises the following steps:
the data acquisition component is used for acquiring Binlog increment log information of an original data source and packaging the log information into a flash data object RowData;
the information matching component is used for configuring a NameMapping operator in the task script so as to obtain a mapping relation in the current task, and mapping and recombining the information in the RowData according to the mapping relation;
and the data writing component is used for configuring Doris data source information and constructing DorisStreamLoad for writing data into the Doris data source according to the Doris data source information and the RowData information after mapping and recombination.
It should be noted that, in the embodiment of the present invention, details that are not disclosed in the system for implementing Binlog to Doris based on Flink are referred to, and details that are disclosed in the method for implementing Binlog to Doris based on Flink are not described herein again.
An electronic device comprising a memory and a processor, the memory configured to store one or more computer instructions, wherein the one or more computer instructions are executable by the processor to implement a method of Flink-based Binlog to Doris real-time acquisition as described above.
It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working process of the electronic device described above may refer to the corresponding process in the foregoing method embodiment, and is not described herein again.
A computer-readable storage medium storing a computer program which, when executed, causes a computer to implement a method for acquiring Binlog to Doris in real time based on Flink as described above.
Illustratively, a computer program may be divided into one or more modules/units, one or more modules/units being stored in a memory and executed by a processor and performing I/O interface transfer of data by an input interface and an output interface to perform the present invention, and one or more modules/units may be a series of computer program instruction segments describing the execution of the computer program in a computer device.
The computer device may be a desktop computer, a notebook, a palm computer, a cloud server, or other computing devices. The computer device may include, but is not limited to, a memory and a processor, and those skilled in the art will appreciate that the present embodiment is only an example of the computer device and does not constitute a limitation of the computer device, and may include more or less components, or combine certain components, or different components, for example, the computer device may further include an input device, a network access device, a bus, and the like.
The Processor may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The storage may be an internal storage unit of the computer device, such as a hard disk or a memory of the computer device. The memory may also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card) and the like provided on the computer device, and further, the memory may also include both an internal storage unit of the computer device and an external storage device, the memory is used for storing a computer program and other programs and data required by the computer device, and the memory may also be used for temporarily storing in the outputter, and the aforementioned storage medium includes various Media capable of storing program codes, such as a usb disk, a removable hard disk, a read only memory ROM, a random access memory RAM, a disk and an optical disk.
The above description is only an embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions within the technical scope of the present invention are included in the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the appended claims.

Claims (10)

1. A method for realizing Binlog-Doris real-time acquisition based on Flink is characterized by comprising the following steps:
acquiring Binlog incremental log information of an original data source, and encapsulating the log information into a Flink data object RowData;
configuring a NameMapping operator in a task script to acquire a mapping relation in a current task, and mapping and recombining information in RowData according to the mapping relation;
and configuring Doris data source information, and constructing DorisStreamLoad for writing data into the Doris data source according to the Doris data source information and the RowData information after mapping and recombination.
2. The method for acquiring Binlog to Doris in real time based on Flink as claimed in claim 1, wherein the acquiring Binlog incremental log information of original data source and packaging the log information into Flink data object RowData comprises:
starting a canal component, and performing regular matching on configuration parameters by using an openInputFormat method to realize subscription and consumption of a Binlog log of an original database;
calling a nextRecordInternal method, and packaging the collected log information into a Flink data object RowData, wherein the RowData comprises an original database name and a table name.
3. The method as claimed in claim 1, wherein the configuring of NameMapping operator in the task script to obtain mapping relationship in the current task, and the mapping and reorganizing of the information in the RowData according to the mapping relationship comprises:
configuring database mapping, schema mapping, tablemapping and field mapping parameters of nameMapping in a task script to obtain a mapping relation in a current task, wherein the mapping relation comprises a database name, a table name and a field name;
disassembling the RowData, and judging whether the disassembled information has a mapping relation;
if yes, mapping and recombining the information according to the mapping relation in the current task, otherwise, keeping the original value.
4. The method of claim 2, wherein the configuring the Doris data source information and constructing the Doris streamload for writing data into the Doris data source according to the Doris data source information and the mapping reorganized RowData information comprises:
configuring Doris data source information, wherein the Doris data source information comprises Doris Fe, be connection information, doris login user name and password;
calling a writeRecord method to analyze the mapped and recombined RowData to obtain an original database name, a table name and corresponding table structure information;
and constructing DorisStreamLoad according to the table structure information and the Doris data source information, and writing data into the Doris data source.
5. The method for acquiring Binlog to Doris in real time based on Flink as claimed in claim 1, wherein the data change type is further included in the RowData;
and constructing different Http requests according to the data change type and the DorisStreamLoad so as to synchronously send the data to the Doris data source.
6. A real-time acquisition system for achieving Binlog to Doris based on Flink, comprising:
the data acquisition component is used for acquiring Binlog incremental log information of an original data source and packaging the log information into a Flink data object RowData;
the information matching component is used for configuring a NameMapping operator in the task script so as to obtain a mapping relation in the current task, and mapping and recombining the information in the RowData according to the mapping relation;
and the data writing component is used for configuring the Doris data source information and constructing the DorisStreamLoad for writing the data into the Doris data source according to the Doris data source information and the RowData information after mapping and recombination.
7. The Flink-based Binlog to Doris real-time acquisition system as claimed in claim 6, wherein the data acquisition component comprises:
the subscription unit is used for starting the canal component, performing regular matching on the configuration parameters by using an openInputFormat method, and subscribing and consuming the Binlog log of the original database;
and the packaging unit is used for calling a nextRecordInternal method and packaging the collected log information into a Flink data object RowData, wherein the RowData contains an original database name and a table name.
8. The Flink-based Binlog-to-Doris real-time acquisition system as recited in claim 1, wherein the data writing component comprises:
the system comprises a configuration unit, a data processing unit and a data processing unit, wherein the configuration unit is used for configuring Doris data source information, and the Doris data source information comprises Doris Fe, be connection information, a Doris login user name and a password;
the analysis unit is used for calling a writeRecord method to analyze the mapped and recombined RowData to obtain an original database name, a table name and corresponding table structure information;
and the constructing unit is used for constructing the DorisStreamLoad according to the table structure information and the Doris data source information, and writing the data into the Doris data source.
9. An electronic device, comprising a memory and a processor, wherein the memory is configured to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement a method for Flink-based Binlog to Doris real-time acquisition as claimed in any one of claims 1 to 5.
10. A computer-readable storage medium storing a computer program, wherein the computer program is configured to enable a computer to execute the method according to any one of claims 1 to 5, to implement a method for acquiring Binlog to Doris in real time based on Flink.
CN202210951740.8A 2022-08-09 2022-08-09 Method and system for realizing real-time acquisition from Binlog to Doris based on Flink Pending CN115268934A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210951740.8A CN115268934A (en) 2022-08-09 2022-08-09 Method and system for realizing real-time acquisition from Binlog to Doris based on Flink

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210951740.8A CN115268934A (en) 2022-08-09 2022-08-09 Method and system for realizing real-time acquisition from Binlog to Doris based on Flink

Publications (1)

Publication Number Publication Date
CN115268934A true CN115268934A (en) 2022-11-01

Family

ID=83751065

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210951740.8A Pending CN115268934A (en) 2022-08-09 2022-08-09 Method and system for realizing real-time acquisition from Binlog to Doris based on Flink

Country Status (1)

Country Link
CN (1) CN115268934A (en)

Similar Documents

Publication Publication Date Title
CN110909063B (en) User behavior analysis method and device, application server and storage medium
CN109997126B (en) Event driven extraction, transformation, and loading (ETL) processing
CN109582660B (en) Data blood margin analysis method, device, equipment, system and readable storage medium
US10496653B1 (en) Data workflow analysis service
JP5298117B2 (en) Data merging in distributed computing
CN111209352B (en) Data processing method and device, electronic equipment and storage medium
CN113360554B (en) Method and equipment for extracting, converting and loading ETL (extract transform load) data
CN104144081A (en) General application log management method, device and system
AU2017254506B2 (en) Method, apparatus, computing device and storage medium for data analyzing and processing
US9971563B2 (en) Systems and methods for low interference logging and diagnostics
CN108664331A (en) Distributed data processing method and device, electronic equipment, storage medium
US11200231B2 (en) Remote query optimization in multi data sources
CN109471893B (en) Network data query method, equipment and computer readable storage medium
CN111274256A (en) Resource control method, device, equipment and storage medium based on time sequence database
CN111367953A (en) Streaming processing method and device for information data
CN111177113A (en) Data migration method and device, computer equipment and storage medium
CN110955674A (en) Asynchronous export method and component based on java service
CN110968592A (en) Metadata acquisition method and device, computer equipment and computer-readable storage medium
CN108616603B (en) Method and system for synchronizing internal and external network data
US10909205B2 (en) System and method for web-session recording
CN111324576B (en) Recording data storage method and device, storage medium and terminal equipment
CN111488386A (en) Data query method and device
CN115268934A (en) Method and system for realizing real-time acquisition from Binlog to Doris based on Flink
CN112506490A (en) Interface generation method and device, electronic equipment and storage medium
Eeda Rendering real-time dashboards using a GraphQL-based UI Architecture

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination