CN110909061A - Data source processing method and device, electronic equipment and storage medium - Google Patents
Data source processing method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN110909061A CN110909061A CN201911196452.0A CN201911196452A CN110909061A CN 110909061 A CN110909061 A CN 110909061A CN 201911196452 A CN201911196452 A CN 201911196452A CN 110909061 A CN110909061 A CN 110909061A
- Authority
- CN
- China
- Prior art keywords
- data
- rule
- data source
- processing
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2358—Change logging, detection, and notification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24564—Applying rules; Deductive queries
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the application provides a data source processing method, a data source processing device, electronic equipment and a storage medium, wherein the data source processing method comprises the following steps: accessing a plurality of data sources and setting identification information for each data source, wherein the identification information comprises a data source name and a rule identification thereof; acquiring rule change information, wherein the rule change information carries a data source name of a data source of a rule to be changed and a target rule identifier of a changed target rule; modifying the rule of the corresponding data source into the target rule according to the rule change information by the identification information; processing the data source according to the target rule to output target data; therefore, data processing of the data source is realized, and the real-time degree of rule updating of the data source can be improved.
Description
Technical Field
The present application relates to the field of data processing, and in particular, to a data source processing method and apparatus, an electronic device, and a storage medium.
Background
With the deep development of big data related technologies, data sources of large enterprises are various, and data is necessarily required to be collected, standardized and warehoused in the process of supporting company business by using big data technologies. In processing data sources, different data sources may have different rules, that is, different processing logics, and therefore, the processing manners adopted are different. When different data sources are processed, the different data sources need to be preprocessed into a certain specific agreed format and then processed uniformly. However, when the data source rule is changed and the target data is extracted based on the changed rule, the prior art cannot update in real time or update quickly.
In view of the above problems, no effective technical solution exists at present.
Disclosure of Invention
An object of the embodiments of the present application is to provide a data source processing method, an apparatus, an electronic device, and a storage medium, which can improve the update real-time of a data source rule.
In a first aspect, an embodiment of the present application provides a data source processing method, including:
accessing a plurality of data sources and setting identification information for each data source, wherein the identification information comprises a data source name and a rule identification thereof;
acquiring rule change information, wherein the rule change information carries a data source name of a data source of a rule to be changed and a target rule identifier of a changed target rule;
modifying the rule of the corresponding data source into the target rule according to the rule change information by the identification information;
and processing the data source according to the target rule to output target data.
In the embodiment of the application, the data source name and the rule identifier of the data source are associated, so that when the rule of the data source is changed, the rule of the data source can be quickly changed directly based on the data source name and the rule identifier, and the updating real-time degree of the data source rule can be improved.
Optionally, in the data source processing method according to the embodiment of the present application, the step of processing the data source according to the target rule to output target data includes:
analyzing the data source to acquire a data structure of the data source;
acquiring position information of a plurality of data to be processed in the data source according to the target rule and the data structure;
and calling the data to be processed according to the position information, and processing the data to be processed according to the target rule to obtain target data.
According to the data processing method and device, the data structure of the data source is obtained by analyzing the data source, data processing is rapidly performed based on the data structure and the target rule, and the data processing efficiency can be improved.
Optionally, in the data source processing method according to the embodiment of the present application, the step of calling the data to be processed according to the location information and processing the data to be processed according to the target rule to obtain the target data includes:
acquiring a plurality of processing logics according to the target rule;
calling a plurality of corresponding data to be processed according to each processing logic and the position information;
and calculating to obtain target data according to each processing logic and the corresponding data to be processed.
According to the embodiment of the application, the target rule is decomposed into the plurality of processing logics, and the data processing of the target rule with the plurality of processing logics can be realized by performing the rapid data processing based on each processing logic.
Optionally, in the data source processing method according to the embodiment of the present application, the method further includes the following steps:
establishing a data processing process, and inputting the plurality of data sources into the data processing process;
and the step of modifying the rule of the corresponding data source into the target rule according to the rule change information by the identification information comprises the following steps:
sending the rule change information to a preset area for caching;
and the data processing process reads the rule change information of the preset area at regular time so as to modify the rule corresponding to the data source into the target rule.
According to the embodiment of the application, the rule updating information is directly sent to the preset area for caching, and then the rule updating information is read regularly through the data processing process, so that the rule is updated quickly, and the rule updating information can be prevented from being lost.
Optionally, in the data source processing method according to the embodiment of the present application, the method further includes the following steps:
establishing a data processing process, and inputting the plurality of data sources into the data processing process;
and the step of modifying the rule of the corresponding data source into the target rule according to the rule change information by the identification information comprises the following steps:
and inputting the rule change information into the data processing process so as to modify the rule of the data source into the target rule in real time.
The embodiment of the application directly sends the rule updating information to the data processing process, so that the real-time updating of the rule is realized.
In a second aspect, an embodiment of the present application provides a data source processing apparatus, including:
the access module is used for accessing a plurality of data sources and setting identification information for each data source, wherein the identification information comprises a data source name and a rule identification thereof;
the acquisition module is used for acquiring rule change information, and the rule change information carries the data source name of the data source of the rule to be changed and the target rule identifier of the changed target rule;
the modification module is used for modifying the rule of the corresponding data source into the target rule according to the rule change information;
and the processing module is used for processing the data source according to the target rule so as to output target data.
Optionally, in the data source processing apparatus according to an embodiment of the present application, the processing module includes:
the first acquisition unit is used for analyzing the data source to acquire a data structure of the data source;
the second acquisition unit is used for acquiring the position information of a plurality of pieces of data to be processed in the data source according to the target rule and the data structure;
and the processing unit is used for calling the data to be processed according to the position information and processing the data to be processed according to the target rule to obtain target data.
Optionally, in the data source processing apparatus according to the embodiment of the present application, the processing unit is configured to obtain a plurality of processing logics according to the target rule, call a plurality of corresponding data to be processed according to each of the processing logics and the location information, and calculate target data according to each of the processing logics and the corresponding data to be processed.
In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, where the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps in the method as provided in the first aspect are executed.
In a fourth aspect, embodiments of the present application provide a storage medium, on which a computer program is stored, where the computer program, when executed by a processor, performs the steps in the method as provided in the first aspect.
As can be seen from the above, the data source processing method, the data source processing apparatus, the electronic device, and the storage medium provided in the embodiments of the present application access a plurality of data sources, and set an identification information for each of the data sources; acquiring rule change information, wherein the rule change information carries identification information of a data source of a rule to be changed and a changed target rule; modifying the rule of the corresponding data source into the target rule according to the rule change information by the identification information; processing the data source according to the target rule to output target data; therefore, data processing of the data source is realized, and the real-time degree of rule updating of the data source can be improved.
Additional features and advantages of the present application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the embodiments of the present application. The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are required to be used in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and therefore should not be considered as limiting the scope, and that those skilled in the art can also obtain other related drawings based on the drawings without inventive efforts.
Fig. 1 is a scene schematic diagram of a data source processing method and apparatus provided in the embodiment of the present application.
Fig. 2 is a flowchart of a data source processing method according to an embodiment of the present application.
Fig. 3 is a schematic structural diagram of a data source processing apparatus according to an embodiment of the present application.
Fig. 4 is a schematic structural diagram of a second data source processing apparatus according to an embodiment of the present application.
Fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. The components of the embodiments of the present application, generally described and illustrated in the figures herein, can be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present application, presented in the accompanying drawings, is not intended to limit the scope of the claimed application, but is merely representative of selected embodiments of the application. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the present application without making any creative effort, shall fall within the protection scope of the present application.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures. Meanwhile, in the description of the present application, the terms "first", "second", and the like are used only for distinguishing the description, and are not to be construed as indicating or implying relative importance.
Referring to fig. 1, fig. 1 is a schematic view of a data source processing method and apparatus according to some embodiments of the present disclosure. The data source processing method and the data source processing device are integrated in electronic equipment in a computer program mode, and the electronic equipment can be a mobile phone, a computer, a server and the like. For example, the data source processing method and device are integrated in the server C0, the server C0 receives the data sources of the servers C1-Cn and sets an identification information for each of the data sources; then, acquiring rule change information, wherein the rule change information carries identification information of a data source of a rule to be changed and a changed target rule; then, modifying the rule of the corresponding data source into the target rule according to the identification information according to the rule change information; finally, processing the data source according to the target rule to output target data; and the data processing of the data source is realized, and the real-time degree of the rule updating of the data source can be improved.
The present application is described in detail below with reference to specific examples.
Referring to fig. 2, fig. 2 is a flowchart of a data source processing method according to some embodiments of the present application. The data source processing method comprises the following steps:
s101, accessing a plurality of data sources, and setting identification information for each data source, wherein the identification information comprises a data source name and a rule identification thereof.
S102, rule change information is obtained, and the rule change information carries the data source name of the data source of the rule to be changed and the target rule identification of the changed target rule.
S103, modifying the rule of the corresponding data source into the target rule according to the rule change information by the identification information.
And S104, processing the data source according to the target rule to output target data.
In step S101, a plug-in Kafka may be used to implement access to multiple data sources, and each data source that is accessed is assigned with a different identification information, that is, a type tag ID, and the data format of the data source is JSON-formatted data used by most enterprises. Of course, it will be appreciated that other common plug-ins or interfaces may be employed to access the multiple data sources.
The rule of the data source refers to an arithmetic logic for processing the data of the data source.
In some embodiments, after the step S101 is executed, a step of establishing a data processing procedure needs to be executed: and establishing a data processing process, and inputting a plurality of data sources into the data processing process so as to process the data. Specifically, the plug-in Kafka is adopted to distribute the accessed multiple data sources to the data processing process for data processing, and the data processing process may be parallel or serial. According to the data processing method and device, the data structure of the data source is obtained by analyzing the data source, data processing is rapidly performed based on the data structure and the target rule, and the data processing efficiency can be improved.
In step S102, a configuration interface may be used to obtain rule change information, which is input or selected by a user according to requirements. And sends the acquired rule change information to the plug-in Kafka. The rule change information may be rule change information for one data source or rule change information for a plurality of data sources.
In step S103, the plug-in Kafka may be used to distribute the rule change information, and may distribute a plurality of pieces of rule change information or may distribute one piece of rule change information. The data processing process may or may not change the rule change information in real time.
In some embodiments, this step S103 includes: s1031, sending the rule change information to a preset area for caching; s1032, the data processing process reads the rule change information of the preset area at regular time to modify the rule corresponding to the data source into the target rule. Wherein the rule change information is stored in the Redis. Then, the data processing process periodically reads the rule change information by the Redis, so as to modify the rule of the corresponding data source. The data processing process selects proper time to Redis read the target rule according to the aging information of the target rule of the data source.
In some embodiments, this step S103 includes: and inputting the rule change information into a data processing process so as to modify the rule of the corresponding data source into a target rule in real time. The plug-in Kafka is adopted to distribute the rule change information to the data processing process, and the data processing process can realize real-time modification of the rule of the corresponding data source after sensing the rule change information.
In step S104, in the process of processing the data of the data source, each data source is not intentionally distinguished, but is analyzed to obtain the data structure of the data source. Then, the corresponding data source is processed based on the data structure and the target rule, so that the corresponding target data is output.
In some embodiments, this step S104 includes: s1041, analyzing the data source to obtain a data structure of the data source; s1042, acquiring position information of a plurality of data to be processed in the data source according to the target rule and the data structure; and S1043, calling the data to be processed according to the position information, and processing the data to be processed according to the target rule to obtain target data. The data structure is used for describing a data frame of the data source or an index table or a directory table similar to each data in the data source, and the target rule is logic for combining, operating or screening a plurality of data defined in the data structure. After the data processing process analyzes the data structure, a plurality of required data to be processed can be extracted from the data source based on the target rule, and finally, the data to be processed are processed by adopting a corresponding processing mode according to the target rule, so that the target data can be obtained.
Specifically, in some embodiments, the step S1043 specifically includes: acquiring a plurality of processing logics according to the target rule; calling a plurality of corresponding data to be processed according to each processing logic and the position information; and calculating to obtain target data according to each processing logic and the corresponding data to be processed. Because one target rule may correspond to a plurality of different processing logics, each processing logic needs to call data at different positions in the data source to perform respective processing. According to the embodiment of the application, the target rule is decomposed into the plurality of processing logics, and the data processing of the target rule with the plurality of processing logics can be realized by performing the rapid data processing based on each processing logic.
This step S1043 is explained in detail below with reference to specific examples.
For example, the data in data source A includes user names of users B1-Bn, VIP rating, points generated by valid consumption, unit price information and time information of products purchased within two years, etc. The target rule includes three processing logics. The processing logic A is used for screening out active users in one year in the users B1-Bn, wherein the active users refer to users with the consumption amount exceeding X yuan in one year. From this user B1-Bn, VIP ratings are greater than a, and active consumption results in a score greater than y. The processing logic C is to screen out active users in the electronic product field in one year from the users B1-Bn, where the active users in the electronic product field refer to users who consume money larger than z yuan and consume times larger than m times in the electronic product field in one year. The specific processing procedure of each processing logic a will be described in detail below.
And aiming at the processing logic A, the data processing process calls a plurality of users who have consumed within one year and consumption records thereof from the corresponding position of the data source according to the data required to be processed by the processing logic A and the data structure, and then screens out the users whose consumption quota is greater than x elements from the plurality of users who have consumed within the one year.
And for the processing logic B, the data processing process calls a plurality of users with scores larger than y and the VIP grade corresponding to each user from corresponding positions of corresponding data sources according to the data required to be processed by the processing logic B and the data structure, and then screens out the users with the VIP grade larger than a from the users with scores larger than y.
And aiming at the processing logic C, the data processing process calls a plurality of users who have consumed within one year and consumption records thereof from corresponding positions of corresponding data sources according to the data required to be processed by the processing logic A and the data structure, and then screens out the users who have consumed in the field of electronic products, wherein the consumption amount of the users is more than z element and the consumption times of the users are more than m times.
And finally, combining the data obtained by the processing logic A, the processing logic B and the processing logic C to obtain the target data corresponding to the target rule. The target data is output to a demand party, such as an operator, and advertisement placement, reward distribution, member offline activities, and the like are performed according to the target data.
According to the data processing method and device, the data structure of the data source is obtained by analyzing the data source, data processing is rapidly performed based on the data structure and the target rule, and the data processing efficiency can be improved.
As can be seen from the above, the data source processing method provided in the embodiment of the present application accesses a plurality of data sources, and sets an identification information for each of the data sources; acquiring rule change information, wherein the rule change information carries identification information of a data source of a rule to be changed and a changed target rule; modifying the rule of the corresponding data source into the target rule according to the rule change information by the identification information; processing the data source according to the target rule to output target data; therefore, data processing of the data source is realized, and the real-time degree of rule updating of the data source can be improved.
Referring to fig. 3, fig. 3 is a block diagram of a data source processing device according to some embodiments of the present disclosure. The data source processing device comprises an access module 201, an acquisition module 202, a modification module 203 and a processing module 204.
The access module 201 is configured to access a plurality of data sources, and set identification information for each of the data sources. The access module 20 may use the plug-in Kafka to realize access to multiple data sources, and assign a different identification information, i.e. a type tag ID, to each data source accessed, where the data format of the data source is the JSON format used by most enterprises.
In some embodiments, after performing the step S101, the data source processing apparatus further includes a building module: the data processing system is used for establishing a data processing process and inputting a plurality of data sources into the data processing process so as to process data. Specifically, the establishing module adopts a plug-in Kafka to distribute the accessed multiple data sources to the data processing process for data processing, and the data processing process may be parallel or serial.
The obtaining module 202 is configured to obtain rule change information, where the rule change information carries identification information of a data source of a rule to be changed and a target rule after change. The obtaining module 202 may use a configuration interface to obtain rule change information, which is input or selected by a user according to requirements. And sends the acquired rule change information to the plug-in Kafka. The rule change information may be rule change information for one data source or rule change information for a plurality of data sources.
The modification module 203 is configured to modify the rule of the corresponding data source into the target rule according to the rule change information. The plug-in Kafka may be used to distribute the rule change information, may distribute a plurality of pieces of rule change information, or may distribute one piece of rule change information. The data processing process may or may not change the rule change information in real time. The modification module 203 is configured to send the rule change information to a preset region for caching, and periodically read the rule change information of the preset region by using a data processing process to modify the rule corresponding to the data source into the target rule. Wherein the rule change information is stored in the Redis. Then, the data processing process periodically reads the rule change information by the Redis, so as to modify the rule of the corresponding data source.
In some embodiments, the modification module 203 is configured to input rule change information into the data processing process to modify the rule of the corresponding data source into the target rule in real time. The plug-in Kafka is adopted to distribute the rule change information to the data processing process, and the data processing process can realize real-time modification of the rule of the corresponding data source after sensing the rule change information.
The processing module 204 is configured to process the data source according to the target rule to output target data. In the process of processing the data of the data source, each data source is not intentionally distinguished, but each data source is analyzed to obtain the data structure of the data source. Then, the corresponding data source is processed based on the data structure and the target rule, so that the corresponding target data is output.
Referring to fig. 4, in some embodiments, the processing module 204 includes a first obtaining unit 2041, a second obtaining unit 2042 and a processing unit 2043. The first obtaining unit 2041 is configured to parse the data source to obtain a data structure of the data source; the second obtaining unit 2042 is configured to obtain location information of multiple pieces of to-be-processed data in the data source according to the target rule and the data structure; the processing unit 2043 is configured to invoke the to-be-processed data according to the location information, and process the to-be-processed data according to the target rule to obtain target data. The data structure is used for describing a data frame of the data source or an index table or a directory table similar to each data in the data source, and the target rule is logic for combining, operating or screening a plurality of data defined in the data structure. After the data processing process analyzes the data structure, a plurality of required data to be processed can be extracted from the data source based on the target rule, and finally, the data to be processed are processed by adopting a corresponding processing mode according to the target rule, so that the target data can be obtained.
In particular, in some embodiments, the processing unit 2043 is configured to obtain a plurality of processing logics according to the target rule; calling a plurality of corresponding data to be processed according to each processing logic and the position information; and calculating to obtain target data according to each processing logic and the corresponding data to be processed. Because one target rule may correspond to a plurality of different processing logics, each processing logic needs to call data at different positions in the data source to perform respective processing. The following examples are given.
For example, the data in data source A includes user names of users B1-Bn, VIP rating, points generated by valid consumption, unit price information and time information of products purchased within two years, etc. The target rule includes three processing logics. The processing logic A is used for screening out active users in one year in the users B1-Bn, wherein the active users refer to users with the consumption amount exceeding X yuan in one year. From this user B1-Bn, VIP ratings are greater than a, and active consumption results in a score greater than y. The processing logic C is to screen out active users in the electronic product field in one year from the users B1-Bn, where the active users in the electronic product field refer to users who consume money larger than z yuan and consume times larger than m times in the electronic product field in one year.
And for the processing logic A, the data processing process calls a plurality of users who have consumed within one year and consumption records thereof from the corresponding position of the data source according to the data required to be processed by the processing logic A and the data structure, and then screens out the users whose consumption amount is larger than x elements from the plurality of users who have consumed within one year.
For the processing logic B, the data processing process calls a plurality of users with points greater than y and the VIP level corresponding to each user from the corresponding position of the corresponding data source according to the data to be processed by the processing logic B and the data structure, and then screens out the users with the VIP level greater than a from the users with points greater than y.
And aiming at the processing logic C, the data processing process calls a plurality of users who generate consumption within one year and consumption records thereof from corresponding positions of corresponding data sources according to the data required to be processed by the processing logic A and the data structure, and then screens out the users who generate consumption and have the consumption quota more than z yuan and the consumption times more than m times in the field of electronic products from the plurality of users who generate consumption. And finally, combining the data obtained by the processing logic A, the processing logic B and the processing logic C to obtain the target data corresponding to the target rule. The target data is output to a demand party, such as an operator, and advertisement placement and the like can be performed according to the target data.
As can be seen from the above, the data source processing apparatus provided in the embodiment of the present application accesses a plurality of data sources, and sets an identification information for each of the data sources; acquiring rule change information, wherein the rule change information carries identification information of a data source of a rule to be changed and a changed target rule; modifying the rule of the corresponding data source into the target rule according to the rule change information by the identification information; processing the data source according to the target rule to output target data; therefore, data processing of the data source is realized, and the real-time degree of rule updating of the data source can be improved.
Referring to fig. 5, fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present application, and the present application provides an electronic device 3, including: the processor 301 and the memory 302, the processor 301 and the memory 302 being interconnected and communicating with each other via a communication bus 303 and/or other form of connection mechanism (not shown), the memory 302 storing a computer program executable by the processor 301, the processor 301 executing the computer program when the computing device is running to perform the method of any of the alternative implementations of the embodiments described above.
The embodiment of the present application provides a storage medium, and when being executed by a processor, the computer program performs the method in any optional implementation manner of the above embodiment. The storage medium may be implemented by any type of volatile or nonvolatile storage device or combination thereof, such as a Static Random Access Memory (SRAM), an Electrically Erasable Programmable Read-Only Memory (EEPROM), an Erasable Programmable Read-Only Memory (EPROM), a Programmable Read-Only Memory (PROM), a Read-Only Memory (ROM), a magnetic Memory, a flash Memory, a magnetic disk, or an optical disk.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions when actually implemented, and for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some communication interfaces, and may be in an electrical, mechanical or other form.
In addition, units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
Furthermore, the functional modules in the embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.
In this document, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions.
The above description is only an example of the present application and is not intended to limit the scope of the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (10)
1. A data source processing method, comprising:
accessing a plurality of data sources and setting identification information for each data source, wherein the identification information comprises a data source name and a rule identification thereof;
acquiring rule change information, wherein the rule change information carries a data source name of a data source of a rule to be changed and a target rule identifier of a changed target rule;
modifying the rule of the corresponding data source into the target rule according to the rule change information by the identification information;
and processing the data source according to the target rule to output target data.
2. The data source processing method of claim 1, wherein the step of processing the data source according to the target rule to output target data comprises:
analyzing the data source to acquire a data structure of the data source;
acquiring position information of a plurality of data to be processed in the data source according to the target rule and the data structure;
and calling the data to be processed according to the position information, and processing the data to be processed according to the target rule to obtain target data.
3. The data source processing method according to claim 2, wherein the step of calling the data to be processed according to the location information and processing the data to be processed according to the target rule to obtain target data comprises:
acquiring a plurality of processing logics according to the target rule;
calling a plurality of corresponding data to be processed according to each processing logic and the position information;
and calculating to obtain target data according to each processing logic and the corresponding data to be processed.
4. The data source processing method of claim 1, further comprising the steps of:
establishing a data processing process, and inputting the plurality of data sources into the data processing process;
and the step of modifying the rule of the corresponding data source into the target rule according to the rule change information by the identification information comprises the following steps:
sending the rule change information to a preset area for caching;
and the data processing process reads the rule change information of the preset area at regular time so as to modify the rule corresponding to the data source into the target rule.
5. The data source processing method of claim 1, further comprising the steps of:
establishing a data processing process, and inputting the plurality of data sources into the data processing process;
and the step of modifying the rule of the corresponding data source into the target rule according to the rule change information by the identification information comprises the following steps:
and inputting the rule change information into the data processing process so as to modify the rule of the data source into the target rule in real time.
6. A data source processing apparatus, comprising:
the access module is used for accessing a plurality of data sources and setting identification information for each data source, wherein the identification information comprises a data source name and a rule identification thereof;
the acquisition module is used for acquiring rule change information, and the rule change information carries the data source name of the data source of the rule to be changed and the target rule identifier of the changed target rule;
the modification module is used for modifying the rule of the corresponding data source into the target rule according to the rule change information;
and the processing module is used for processing the data source according to the target rule so as to output target data.
7. The data source processing device of claim 6, wherein the processing module comprises:
the first acquisition unit is used for analyzing the data source to acquire a data structure of the data source;
the second acquisition unit is used for acquiring the position information of a plurality of pieces of data to be processed in the data source according to the target rule and the data structure;
and the processing unit is used for calling the data to be processed according to the position information and processing the data to be processed according to the target rule to obtain target data.
8. The data source processing apparatus according to claim 7, wherein the processing unit is configured to obtain a plurality of processing logics according to the target rule, call a plurality of corresponding data to be processed according to each of the processing logics and the location information, and calculate target data according to each of the processing logics and the corresponding data to be processed.
9. An electronic device comprising a processor and a memory, said memory storing computer readable instructions which, when executed by said processor, perform the steps of the method of any of claims 1-5.
10. A storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, performs the steps of the method according to any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911196452.0A CN110909061A (en) | 2019-11-27 | 2019-11-27 | Data source processing method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911196452.0A CN110909061A (en) | 2019-11-27 | 2019-11-27 | Data source processing method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110909061A true CN110909061A (en) | 2020-03-24 |
Family
ID=69820452
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911196452.0A Pending CN110909061A (en) | 2019-11-27 | 2019-11-27 | Data source processing method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110909061A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112100205A (en) * | 2020-09-27 | 2020-12-18 | 北京有竹居网络技术有限公司 | Data processing method, device, equipment and computer readable medium |
| CN113778502A (en) * | 2020-06-29 | 2021-12-10 | 北京沃东天骏信息技术有限公司 | Data processing method, device, system and storage medium |
| CN116431688A (en) * | 2022-11-14 | 2023-07-14 | 北京远舢智能科技有限公司 | Data processing method and device, electronic equipment and storage medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105989445A (en) * | 2015-02-25 | 2016-10-05 | 阿里巴巴集团控股有限公司 | Rule management method and system |
| US20160306827A1 (en) * | 2015-04-15 | 2016-10-20 | International Business Machines Corporation | Synchronizing data rules and corresponding metadata to implement data governance |
| CN106484595A (en) * | 2016-10-09 | 2017-03-08 | 华青融天（北京）技术股份有限公司 | A kind of event-handling method and device |
| CN106557307A (en) * | 2015-09-29 | 2017-04-05 | 腾讯科技（深圳）有限公司 | The processing method and processing system of business datum |
| CN109408481A (en) * | 2018-11-06 | 2019-03-01 | 北京字节跳动网络技术有限公司 | Update method, device, electronic equipment and the readable medium of log collection rule |
| CN109710604A (en) * | 2019-01-09 | 2019-05-03 | 北京京东金融科技控股有限公司 | Data processing method, device, system, computer readable storage medium |
| CN110457348A (en) * | 2018-05-02 | 2019-11-15 | 北京三快在线科技有限公司 | A kind of data processing method and device |
| CN110502546A (en) * | 2019-08-22 | 2019-11-26 | 郑州阿帕斯科技有限公司 | A kind of data processing method and device |
-
2019
- 2019-11-27 CN CN201911196452.0A patent/CN110909061A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105989445A (en) * | 2015-02-25 | 2016-10-05 | 阿里巴巴集团控股有限公司 | Rule management method and system |
| US20160306827A1 (en) * | 2015-04-15 | 2016-10-20 | International Business Machines Corporation | Synchronizing data rules and corresponding metadata to implement data governance |
| CN106557307A (en) * | 2015-09-29 | 2017-04-05 | 腾讯科技（深圳）有限公司 | The processing method and processing system of business datum |
| CN106484595A (en) * | 2016-10-09 | 2017-03-08 | 华青融天（北京）技术股份有限公司 | A kind of event-handling method and device |
| CN110457348A (en) * | 2018-05-02 | 2019-11-15 | 北京三快在线科技有限公司 | A kind of data processing method and device |
| CN109408481A (en) * | 2018-11-06 | 2019-03-01 | 北京字节跳动网络技术有限公司 | Update method, device, electronic equipment and the readable medium of log collection rule |
| CN109710604A (en) * | 2019-01-09 | 2019-05-03 | 北京京东金融科技控股有限公司 | Data processing method, device, system, computer readable storage medium |
| CN110502546A (en) * | 2019-08-22 | 2019-11-26 | 郑州阿帕斯科技有限公司 | A kind of data processing method and device |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113778502A (en) * | 2020-06-29 | 2021-12-10 | 北京沃东天骏信息技术有限公司 | Data processing method, device, system and storage medium |
| CN112100205A (en) * | 2020-09-27 | 2020-12-18 | 北京有竹居网络技术有限公司 | Data processing method, device, equipment and computer readable medium |
| CN116431688A (en) * | 2022-11-14 | 2023-07-14 | 北京远舢智能科技有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN116431688B (en) * | 2022-11-14 | 2024-05-03 | 北京远舢智能科技有限公司 | Data processing method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110909061A (en) | Data source processing method and device, electronic equipment and storage medium | |
| US20170221080A1 (en) | Brand Analysis | |
| CN110300084B (en) | IP address-based portrait method and apparatus, electronic device, and readable medium | |
| CN109657998B (en) | Resource allocation method, device, equipment and storage medium | |
| US20150332316A1 (en) | System and method for unifying user-level data across different media platforms | |
| RU2622850C2 (en) | Method and server for processing product identifiers and machine-readable storage medium | |
| CN111444287A (en) | Graph database construction method, associated information query method, device and computing equipment | |
| CN113835874A (en) | Deep learning service scheduling method, system, terminal and storage medium | |
| CN111858278A (en) | Log analysis method and system based on big data processing and readable storage device | |
| CN108304990A (en) | A kind of sensitive pre-judging method of power failure and system | |
| CN114741392A (en) | Data query method and device, electronic equipment and storage medium | |
| CN110618911A (en) | Data monitoring method and device, storage medium and server | |
| CN110727885A (en) | Internet global uniform identifier generation system and generation method thereof | |
| CN114238777A (en) | Negative feedback flow distribution method, device, equipment and medium based on behavior analysis | |
| CN109949090A (en) | Lead referral method, apparatus, electronic equipment and medium | |
| CN113190587A (en) | Data processing method and device for realizing service data processing | |
| CN115130959A (en) | Method, system, terminal and storage medium for generating spare part BOM | |
| CN112231369A (en) | Arrival rate curve fitting method and device | |
| US20150100515A1 (en) | Customer data unification | |
| CN112256689A (en) | Service data cleaning method and device and electronic equipment | |
| CN112836971A (en) | Quota resource determination method and device, electronic equipment and storage medium | |
| CN111831698A (en) | Data auditing method, system and electronic equipment | |
| CN113779017A (en) | Method and apparatus for data asset management | |
| CN111105270A (en) | Method and device for managing push data | |
| CN110032703A (en) | A kind of enterprise web site trustship method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20200324 |