CN114661717A - Multi-data stream management method, system, engine and storage medium - Google Patents

Multi-data stream management method, system, engine and storage medium Download PDF

Info

Publication number
CN114661717A
CN114661717A CN202210307456.7A CN202210307456A CN114661717A CN 114661717 A CN114661717 A CN 114661717A CN 202210307456 A CN202210307456 A CN 202210307456A CN 114661717 A CN114661717 A CN 114661717A
Authority
CN
China
Prior art keywords
data
data source
current
current data
structure information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210307456.7A
Other languages
Chinese (zh)
Inventor
葛丁佳
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Jingkun Computer Technology Co ltd
Original Assignee
Shanghai Jingkun Computer Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Jingkun Computer Technology Co ltd filed Critical Shanghai Jingkun Computer Technology Co ltd
Priority to CN202210307456.7A priority Critical patent/CN114661717A/en
Publication of CN114661717A publication Critical patent/CN114661717A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22—Indexing; Data structures therefor; Storage structures
    • G06F16/2291—User-Defined Types; Storage management thereof
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/25—Integrating or interfacing systems involving database management systems
    • G06F16/258—Data format conversion from or to a database

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The embodiment of the invention discloses a method, a system, an engine and a storage medium for managing multiple data streams, wherein the method comprises the following steps: determining data structure information of current data source data; creating a metadata base and a service database of the current data source data according to the data structure information; and according to the metadata in the metadata database and the service scene corresponding to the current data source data, performing storage definition on the data structure of the current data source data, and importing the current data source data after the storage definition into the service database. The problem that the existing multi-data stream management method cannot meet the real-time requirement is solved.

Description

Multi-data stream management method, system, engine and storage medium
Technical Field
The embodiment of the invention relates to the field of multi-data stream management, in particular to a multi-data stream management method, a multi-data stream management system, an engine and a storage medium.
Background
With the progress of digitization, a large amount of data is generated in various industries in the society at present, the data are different in structure, the redundancy degree is high, and the data use difficulty is high.
The existing data integration method comprises extraction, conversion, dimension reduction and integration, and the scheme modification time is long, so that once a certain link of data source transmission is changed, the scheme needs to be modified again, and the real-time requirement in a production environment cannot be met.
The applicant finds that the existing multi-data stream management method at least cannot meet the requirement of real-time performance in the process of implementing the embodiment of the invention.
Disclosure of Invention
The embodiment of the invention provides a multi-data stream management method, a system, an engine and a storage medium, and solves the problem that the existing multi-data stream management method cannot meet the real-time requirement.
In a first aspect, an embodiment of the present invention provides a multiple data stream management method, where the method includes:
optionally, before the extracting the data structure information of the current data source data, creating according to the data source structure information, the method further includes:
and determining whether the current data source is a newly accessed data source, and if so, finishing the initial definition of the current data source in a data management layer.
Optionally, before creating the metadata database and the service database of the current data source data according to the data structure information, the method further includes:
determining data structure information of current data source data;
determining the acquisition method of the current data source according to the data structure information of the current data source data and the corresponding relation between the predefined data structure information and the acquisition method, and acquiring the data of the current data source by adopting the acquisition method to obtain the current data source data;
and taking the data structure information as the data structure information of the current data source data.
Optionally, before creating the metadata database and the service database of the current data source data according to the data structure information, the method further includes:
determining data structure information of current data source data;
determining an acquisition method of the current data source according to the data structure information of the current data source data, a service scene corresponding to the current data source data and a predefined corresponding relationship between data structure information, the service scene and the acquisition method, and acquiring the data of the current data source by adopting the acquisition method to obtain the current data source data;
and taking the data structure information as the data structure information of the current data source data.
Optionally, the defining, according to the metadata in the metadata database and the service scenario corresponding to the current data source data, a data structure of the current data source data in a storage manner includes:
determining a service data structure required by a service scene corresponding to the current data source data;
and adjusting the data structure of the current data source data according to the attribute relationship between the metadata in the metadata database and the metadata of other data source data in the data management layer and the service data structure so as to finish the storage definition of the current data source data.
Optionally, the method further comprises:
and monitoring the transmission of the current data source data in real time through a data stream pipeline, and synchronizing the processing strategy of the current data source data at each transmission node when monitoring that the transmission state of the current data source data at any transmission node is changed.
Optionally, after the adjusting the data structure of the current data source data to complete the warehousing definition of the current data source data, the method further includes:
if the service data structure or the data structure defined in the warehouse is changed, adjusting the acquisition method of the data structure information corresponding to the current data source in the acquisition method library according to the change result;
and acquiring the data of the current data source by adopting the adjusted acquisition method of the data of the current data source to obtain the data of the current data source.
In a second aspect, an embodiment of the present invention further provides a multiple data stream management system, where the apparatus includes:
the data structure information determining module is used for determining the data structure information of the current data source data;
the database creating module is used for creating a metadata database and a service database of the current data source data according to the data structure information;
and the data import module is used for performing warehousing definition on the data structure of the current data source data according to the metadata in the metadata database and the service scene corresponding to the current data source data, and importing the current data source data after the warehousing definition into the service database.
In a third aspect, an embodiment of the present invention further provides a data engine, where the engine includes:
one or more processors;
a storage device for storing one or more application systems;
when executed by the one or more processors, cause the one or more processors to implement a method of multiple data stream management as in any of the embodiments.
In a fourth aspect, embodiments of the present invention also provide a storage medium containing computer-executable instructions which, when executed by a computer processor, are configured to perform a method of multiple data stream management as in any of the embodiments.
The embodiment of the invention provides a technical scheme of a multi-data stream management method, which determines the data structure information of the current data source data; creating a metadata base and a service database of the current data source data according to the data structure information; and according to the metadata in the metadata database and the service scene corresponding to the current data source data, performing storage definition on the data structure of the current data source data, and importing the current data source data after the storage definition into the service database. The metadata base is created through the data structure information of the current data source, the dynamic customization of the business database is realized through the metadata base, the metadata of the current data source data and the corresponding business scene are combined to carry out storage definition on the data structure of the current data source data, the technical effect of defining the business guide data structure is achieved, the data in the business database can be used without being processed again, the management of multi-source heterogeneous data is more flexible, and meanwhile the real-time requirement of a user on the data is met.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the description below are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on the drawings without creative efforts.
Fig. 1 is a flow chart of a multiple data stream management method according to an embodiment of the present invention;
fig. 2 is a block diagram of a multi-data stream management system according to a second embodiment of the present invention;
fig. 3 is a block diagram of an engine according to a third embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described through embodiments with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example one
Fig. 1 is a flowchart of a multiple data stream management method according to an embodiment of the present invention. The technical scheme of the embodiment is suitable for the situation of multi-source heterogeneous data integration management. The method can be executed by the multi-data stream management system provided by the embodiment of the invention, and the device can be realized in a software and/or hardware mode and is configured to be applied in a processor of an engine. The method specifically comprises the following steps:
s101, determining data structure information of current data source data.
It is understood that when there are at least two data source data of different data structures requiring multiple data stream management simultaneously, the at least two data source data can be processed simultaneously based on a parallel processing manner. For convenience of description of the technical solution, the present embodiment takes a multiple data stream management process of one of the data sources as an example to perform detailed description of the technical solution.
At the beginning of data acquisition of any data source, it is required to determine whether the current data source is a newly accessed data source, and if so, the data source needs to be initially defined in a data management layer in a multi-data stream management system, so as to facilitate data management of different data sources. Wherein the data management layer is used for storing data management information, such as data source definition. The initial definition of a data source is used to define the name of the corresponding data source, the storage path of the data, and the like.
If the current data source is not a newly accessed data source or the initial definition of the current data source is completed in the data management layer, determining the data structure information of the data of the current data source, determining the acquisition method of the current data source according to the data structure information and the corresponding relation between the predefined data structure information and the acquisition method, and completing the data acquisition of the current data source by adopting the acquisition method to obtain the data of the current data source.
The data structure information may be extracted from a header file of the current data source, for example, a structure and a member of the current data source are extracted according to a keyword in the header file, and then attributes and table structures corresponding to the structure and the member are determined to serve as the data structure information of the current data source. It is understood that the data structure information includes attribute name, length, type, etc. information.
Because the structure and the macro defined by the header files of different data sources may be different or the same, the data structure information of different data sources may be the same or different. In order to improve the flexibility and accuracy of data acquisition of different data sources, at least two acquisition methods are stored in the method library, and data structure information corresponding to each acquisition method is predefined. Therefore, when the data structure information of the current data source is determined, the acquisition method corresponding to the current data source can be determined according to the acquisition method corresponding to the data structure information, and then the acquisition method is adopted to acquire the data of the current data source so as to obtain the data of the current data source.
Specifically, for structured, semi-structured, and unstructured data sources, the present embodiment is provided with collection methods such as Java Database connection (JDBC), Open Database connection (ODBC), Agent (Agent), Intelligent Platform Management Interface (IPMI), and the like. During actual data acquisition, which acquisition method is specifically adopted needs to be determined according to data structure information of a data source, and then the determined acquisition method is adopted to acquire data of the corresponding data source.
In one embodiment, the acquisition method corresponding to the MySQL data source is preset to be JDBC or ODBC, so that when the current data source is detected to be MySQL, the JDBC or ODBC is used to acquire data of the current data source.
In one embodiment, in order to deal with a multi-service scene of multi-source heterogeneous data, a corresponding relation of data structure information-service scene-acquisition method is established in advance. And aiming at any data source, acquiring data structure information of the data source and a service scene corresponding to the data source, and then determining a data acquisition method of the data source according to the corresponding relation of the data structure information, the service scene and the data acquisition method. It can be understood that, if a data source corresponds to at least two service scenarios, and service data required by the at least two service scenarios overlap but are not completely the same, the system designs an acquisition method for each service scenario and stores the acquisition method in the method library.
In one embodiment, in order to deal with a multi-service scene of multi-source heterogeneous data, a corresponding relation of data structure information-service scene combination-acquisition method is established in advance. And aiming at any data source, acquiring data structure information of the data source and a service scene combination corresponding to the data source, and then determining a data acquisition method of the data source according to the corresponding relation of the data structure information, the service scene combination and the data acquisition method. It can be understood that, if a data source corresponds to at least two service scenarios, and service data required by the at least two service scenarios are overlapped but not completely identical, at this time, the system determines a data acquisition range according to a union set of the service data required by the at least two service scenarios, then determines an acquisition method of the data source data according to the type of the data source, and stores the acquisition method in a method library.
Preprocessing collected data such as field type processing, date type conversion, enumeration type conversion, data desensitization, non-empty field judgment and the like, and writing the preprocessed data into a data cache library in a batch or real-time mode.
And S102, creating a metadata database and a service database of the current data source data according to the data structure information.
And creating a metadata base according to the data structure information of the current data source, and generating an attribute meta table and a table meta table insertion statement according to the attribute name, length, type and other information in the data structure information of the current data source so as to insert the data structure information of the current data source into the metadata base.
And creating a service database according to the attribute meta table and the table meta table in the meta database corresponding to the current data source, wherein the service database creates the service table of the current data source at the moment.
It will be appreciated that the metadata base may dynamically customize the service database for maintaining the organizational form of the service table, while the service database is used to store the service data. Specifically, the attribute meta table includes information such as an attribute name and a membership table; the table element table comprises information such as the name of the service table, the number of attributes in the table, the use frequency, the recorded data, the last update time and the like. Therefore, the operations of attribute modification, addition, deletion and the like of the service database can be realized through the attribute meta table in the meta database, and the operations of modification, addition, deletion and the like of the table in the service database are carried out through the table meta table.
Wherein the metadata base is also stored in the data management layer.
S103, according to the metadata in the metadata database and the service scene corresponding to the current data source data, performing storage definition on the data structure of the current data source data, and importing the current data source data after the storage definition into the service database.
In one embodiment, a service scenario corresponding to current data source data and a service data structure corresponding to the service scenario are determined, and the service data structure is used as the service data structure corresponding to the current data source data. And according to the metadata of the current data source and the service data structure corresponding to the current data source, performing storage definition on the data structure of the current data source.
Illustratively, the metadata base of the data of the current data source includes table a1, table a2, table A3, table a4 and table a5 of the current data source, and the service data structure includes table a1, table a2, table A3 and table a4, but does not include table a5, so that when the data management layer performs the warehousing definition on the data structure of the current data source, the service data structure includes table a1, table a2, table A3 and table a4, but does not include table a 5.
In one embodiment, a service scenario corresponding to current data source data and a service data structure corresponding to the service scenario are determined, and the service data structure is used as the service data structure corresponding to the current data source data. And according to the attribute relationship between the metadata of the current data source and the metadata of other data sources in the data management layer and the service data structure corresponding to the current data source, performing storage definition on the data structure of the current data source.
Illustratively, the metadata base includes table a1, table a2, table A3, table a4, and table a5 of the current data source, and the service data structure includes table a 1', table a2, table A3, and table a4, but does not include table a 5. Another data source in the data management layer includes Table B1, Table B2, Table B3, Table B4, and Table B1, where Table A1, Table A1', and Table B1 are synonyms. For this situation, when the data management layer performs entry definition on the current data source data, table a1, table a1 'and table B1 are unified as a 1', or unified as table M, that is, unified according to the service data structure, or unified as a new name.
And after the warehousing definition of the current data source data is completed, importing the data corresponding to the data structure after the warehousing definition in the current data source data into a service database. Specifically, the current data source data in the data cache library is loaded into a corresponding table of the service database according to the data structure defined in the database. After the loading in the database is finished, sending the setting parameters of the current data source to the metadata database, wherein the setting parameters include but are not limited to increase and decrease of a table, the record number of the current data source, the use frequency and the final use time. It will be appreciated that when the meta-database updates tables or attributes, the service database responds accordingly, e.g., a service table is deleted from the table meta-table, and the service table is also disappeared from the service database.
In one embodiment, if the service data structure or the data structure defined in the database is changed, which means that the requirement of the service end is changed, the acquisition method corresponding to the data structure information of the current data source in the acquisition method database is adjusted according to the changed service data structure; and acquiring the data of the current data source by adopting the adjusted acquisition method of the data of the current data source to obtain the data of the current data source. Illustratively, the service data structure corresponding to the current data source data or the data structure defined in the database includes table a1, table a2, table A3, and table a4, and the changed service data structure includes table a1, table a2, and table A3, that is, the content of table a4 is not needed at present at the service end, so the acquisition method corresponding to the data structure information of the current data source data in the method database is adjusted, so that the acquisition method does not need to acquire the content of table a4, and only the contents of table a1, table a2, and table A3 are acquired. Thus, if the data structure information of the other data sources is the same as that of the current data source, it is only necessary to collect the contents in table a1, table a2, and table A3.
In one embodiment, the transmission of the current data source data is monitored in real time through a data stream pipeline, and the processing strategy of the current data source data at each transmission node is synchronized when the transmission state of the current data source data at any transmission node is monitored to be changed. Exemplarily, if the data flow pipeline monitors that the service data structure or the data structure defined in the database is changed, the acquisition method of the data structure information corresponding to the current data source in the acquisition method database is adjusted according to the change result; and acquiring the data of the current data source by adopting the adjusted acquisition method to obtain the data of the current data source.
The technical scheme of the multi-data stream management method provided by the embodiment of the invention determines the data structure information of the current data source data; a metadata database and a service database of the current data source data are created according to the data structure information; and according to the metadata in the metadata database and the service scene corresponding to the current data source data, performing storage definition on the data structure of the current data source data, and importing the current data source data after the storage definition into the service database. The metadata base is created through the data structure information of the current data source, the dynamic customization of the business database is realized through the metadata base, the metadata of the current data source data and the corresponding business scene are combined to carry out storage definition on the data structure of the current data source data, the technical effect of defining the business guide data structure is achieved, the data in the business database can be used without being processed again, the management of multi-source heterogeneous data is more flexible, and meanwhile the real-time requirement of a user on the data is met.
Example two
Fig. 2 is a block diagram of a multiple data stream management system according to an embodiment of the present invention. The apparatus for performing the multiple data stream management method provided by any of the above embodiments, optionally implemented in software or hardware. The device includes:
the data acquisition module 11 is configured to determine data structure information of current data source data;
a database creating module 12, configured to create a metadata database and a service database of the current data source data according to the data structure information;
and the data import module 13 is configured to perform storage definition on a data structure of the current data source data according to the metadata in the metadata database and the service scene corresponding to the current data source data, and import the current data source data after the storage definition into the service database.
Optionally, the data acquisition module is further configured to determine whether the current data source is a newly accessed data source, and if so, complete the initial definition of the current data source in the data management layer.
Optionally, the data acquisition module is further configured to determine data structure information of the current data source; determining the acquisition method of the current data source according to the data structure information of the current data source and the corresponding relation between the predefined data structure information and the acquisition method, and acquiring the data of the current data source by adopting the acquisition method to obtain the data of the current data source; and taking the data structure information as the data structure information of the current data source data.
Optionally, the data acquisition module is further configured to preprocess the current data source data to update the current data source data; and storing the updated current data source data to a data cache library.
Optionally, the data import module is configured to determine a service data structure required by a service scenario corresponding to the current data source data; and adjusting the data structure of the current data source data according to the attribute relationship between the metadata in the metadata database and the metadata of other data source data in the data management layer and the service data structure so as to finish the storage definition of the current data source data.
Optionally, the acquisition module is further configured to adjust an acquisition method of the current data source in the acquisition method library according to a change result if the service data structure corresponding to the current data source is changed; and acquiring the data of the current data source by adopting the adjusted acquisition method of the data of the current data source to obtain the data of the current data source.
Optionally, the apparatus further includes a synchronization module, configured to monitor, in real time, transmission of the current data source data through a data stream pipeline, and synchronize a processing policy of the current data source data at each transmission node when it is monitored that a transmission state of any transmission node of the current data source data changes
According to the technical scheme of the multi-data stream management system, the data structure information of the current data source data is determined through the acquisition module; a metadata database and a service database of the current data source data are created through a database creation module according to the data structure information; and performing warehousing definition on the data structure of the current data source data through a data importing module according to the metadata in the metadata database and the service scene corresponding to the current data source data, and importing the current data source data after the warehousing definition into the service database. The metadata base is established through the data structure information of the current data source, the dynamic customization of the business database is realized through the metadata base, the metadata of the current data source data and the corresponding business scene are combined to carry out storage definition on the data structure of the current data source data, the technical effect of defining with the business guide data structure is achieved, the data in the business database can be used without being processed again, the management of the multi-source heterogeneous data is more flexible, and meanwhile, the real-time requirement of a user on the data is met.
The multiple data stream management system provided by the embodiment of the invention can execute the multiple data stream management method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
EXAMPLE III
Fig. 3 is a schematic structural diagram of an engine according to an embodiment of the present invention, and as shown in fig. 3, the apparatus includes a processor 201, a memory 202, an input device 203, and an output device 204; the number of the processors 201 in the device may be one or more, and one processor 201 is taken as an example in fig. 3; the processor 201, the memory 202, the input device 203 and the output device 204 in the apparatus may be connected by a bus or other means, and fig. 3 illustrates the connection by a bus as an example.
The memory 202, which is a computer-readable storage medium, may be used for storing software programs, computer-executable programs, and modules, such as program instructions/modules (e.g., the data collection module 11, the database creation module 12, and the data import module 13) corresponding to the multiple data stream management method in the embodiment of the present invention. The processor 201 executes various functional applications of the device and data processing by executing software programs, instructions and modules stored in the memory 202, i.e. implementing the above-described multiple data stream management method.
The memory 202 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal, and the like. Further, the memory 202 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, the memory 202 may further include memory located remotely from the processor 201, which may be connected to the device over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 203 may be used to receive input numeric or character information and generate key signal inputs relating to user settings and function controls of the apparatus.
The output device 204 may include a display device such as a display screen, for example, of a user terminal.
Example four
Embodiments of the present invention also provide a storage medium containing computer-executable instructions which, when executed by a computer processor, perform a method of multiple data stream management, the method comprising:
determining data structure information of current data source data;
creating a metadata base and a service database of the current data source data according to the data structure information;
according to the metadata in the metadata database and the service scene corresponding to the current data source data, performing warehousing definition on the data structure of the current data source data, and importing the current data source data after the warehousing definition into the service database
Of course, the storage medium provided by the embodiment of the present invention contains computer-executable instructions, and the computer-executable instructions are not limited to the method operations described above, and may also perform related operations in the multiple data stream management method provided by any embodiment of the present invention.
From the above description of the embodiments, it is obvious for those skilled in the art that the present invention can be implemented by software and necessary general hardware, and certainly can be implemented by hardware, but the former is a better embodiment in many cases. Based on such understanding, the technical solutions of the present invention or portions thereof contributing to the prior art may be embodied in the form of a software product, which may be stored in a computer-readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a FLASH Memory (FLASH), a hard disk or an optical disk of a computer, and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the multiple data stream management method according to the embodiments of the present invention.
It should be noted that, in the embodiment of the multiple data stream management system, the units and modules included in the embodiment are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be implemented; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims (10)

1. A method of multiple data stream management, comprising:
determining data structure information of current data source data;
creating a metadata base and a service database of the current data source data according to the data structure information;
and according to the metadata in the metadata database and the service scene corresponding to the current data source data, performing storage definition on the data structure of the current data source data, and importing the current data source data after storage definition into the service database.
2. The method of claim 1, wherein before extracting the data structure information of the current data source data, the method further comprises:
and determining whether the current data source is a newly accessed data source, and if so, finishing the initial definition of the current data source in a data management layer.
3. The method according to claim 1, before creating the metadata database and the service database of the current data source data according to the data structure information, further comprising:
determining data structure information of current data source data;
determining the acquisition method of the current data source according to the data structure information of the current data source data and the corresponding relation between the predefined data structure information and the acquisition method, and acquiring the data of the current data source by adopting the acquisition method to obtain the current data source data;
and taking the data structure information as the data structure information of the current data source data.
4. The method according to claim 1, before creating the metadata database and the service database of the current data source data according to the data structure information, further comprising:
determining data structure information of current data source data;
determining an acquisition method of the current data source according to the data structure information of the current data source data, a service scene corresponding to the current data source data and a predefined corresponding relationship between data structure information, the service scene and the acquisition method, and acquiring the data of the current data source by adopting the acquisition method to obtain the current data source data;
and taking the data structure information as the data structure information of the current data source data.
5. The method according to claim 1, wherein the step of performing a warehousing definition on the data structure of the current data source data according to the metadata in the metadata base and the service scenario corresponding to the current data source data includes:
determining a service data structure required by a service scene corresponding to the current data source data;
and adjusting the data structure of the current data source data according to the attribute relationship between the metadata in the metadata database and the metadata of other data source data in the data management layer and the service data structure so as to finish the storage definition of the current data source data.
6. The method of claim 1, further comprising:
and monitoring the transmission of the current data source data in real time through a data stream pipeline, and synchronizing the processing strategy of the current data source data at each transmission node when monitoring that the transmission state of the current data source data at any transmission node is changed.
7. The method according to any one of claims 1 to 6, wherein after the adjusting the data structure of the current data source data to complete the warehousing definition of the current data source data, further comprises:
if the business data structure or the data structure defined in the storage is changed, adjusting the acquisition method of the data structure information corresponding to the current data source in the acquisition method library according to the change result;
and acquiring the data of the current data source by adopting the adjusted acquisition method of the data of the current data source to obtain the data of the current data source.
8. A multiple data stream management system, comprising:
the data structure information determining module is used for determining the data structure information of the current data source data;
the database creating module is used for creating a metadata database and a service database of the current data source data according to the data structure information;
and the data import module is used for carrying out warehousing definition on the data structure of the current data source data according to the metadata in the metadata database and the service scene corresponding to the current data source data, and importing the current data source data after the warehousing definition into the service database.
9. An engine, characterized in that the engine comprises:
one or more processors;
a storage device for storing one or more application systems;
when executed by the one or more processors, cause the one or more processors to implement a method of multiple data stream management as recited in any of claims 1-7.
10. A storage medium containing computer-executable instructions for performing a multiple data stream management method as claimed in any one of claims 1-7 when executed by a computer processor.
CN202210307456.7A 2022-03-25 2022-03-25 Multi-data stream management method, system, engine and storage medium Pending CN114661717A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210307456.7A CN114661717A (en) 2022-03-25 2022-03-25 Multi-data stream management method, system, engine and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210307456.7A CN114661717A (en) 2022-03-25 2022-03-25 Multi-data stream management method, system, engine and storage medium

Publications (1)

Publication Number Publication Date
CN114661717A true CN114661717A (en) 2022-06-24

Family

ID=82033704

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210307456.7A Pending CN114661717A (en) 2022-03-25 2022-03-25 Multi-data stream management method, system, engine and storage medium

Country Status (1)

Country Link
CN (1) CN114661717A (en)

Similar Documents

Publication Publication Date Title
CN109413127B (en) Data synchronization method and device
US11921749B2 (en) Synchronization adapter for synchronizing application data
JP2004528636A (en) Automatic data update
CN111797121A (en) Strong consistency query method, device and system for read-write separation architecture service system
CN110784498B (en) Personalized data disaster tolerance method and device
CN111858760B (en) Data processing method and device for heterogeneous database
CN110858194A (en) Method and device for expanding database
US8195700B2 (en) Distributed storage for collaboration servers
CN113420026B (en) Database table structure changing method, device, equipment and storage medium
CN111917834A (en) Data synchronization method and device, storage medium and computer equipment
CN111651424B (en) Data processing method, device, data node and storage medium
CN113282611A (en) Method and device for synchronizing stream data, computer equipment and storage medium
CN112506870A (en) Data warehouse increment updating method and device and computer equipment
CN112612850A (en) Data synchronization method and device
CN104462342B (en) database snapshot synchronization processing method and device
CN114020840A (en) Data processing method, device, server, storage medium and product
CN115687503A (en) Method, device and equipment for synchronizing data among databases and storage medium
CN109165259B (en) Index table updating method based on network attached storage, processor and storage device
CN114661717A (en) Multi-data stream management method, system, engine and storage medium
CN110569231B (en) Data migration method, device, equipment and medium
JP7133037B2 (en) Message processing method, device and system
CN114490865A (en) Database synchronization method, device, equipment and computer storage medium
CN113963763A (en) Partition changing method and device for medical data storage
CN113157716A (en) Data processing method, device, equipment and medium
CN112948494A (en) Data synchronization method and device, electronic equipment and computer readable medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination