CN116860898B - Data processing method and device - Google Patents

Data processing method and device Download PDF

Info

Publication number
CN116860898B
CN116860898B CN202311137102.3A CN202311137102A CN116860898B CN 116860898 B CN116860898 B CN 116860898B CN 202311137102 A CN202311137102 A CN 202311137102A CN 116860898 B CN116860898 B CN 116860898B
Authority
CN
China
Prior art keywords
data
incremental
message middleware
engine
database
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202311137102.3A
Other languages
Chinese (zh)
Other versions
CN116860898A (en
Inventor
纪涛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
CCB Finetech Co Ltd
Original Assignee
CCB Finetech Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by CCB Finetech Co Ltd filed Critical CCB Finetech Co Ltd
Priority to CN202311137102.3A priority Critical patent/CN116860898B/en
Publication of CN116860898A publication Critical patent/CN116860898A/en
Application granted granted Critical
Publication of CN116860898B publication Critical patent/CN116860898B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28Databases characterised by their database models, e.g. relational or object models
    • G06F16/284Relational databases
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22Indexing; Data structures therefor; Storage structures
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/242Query formulation
    • G06F16/2433Query languages

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a data processing method and device, and relates to the technical field of data processing. One embodiment of the method comprises the following steps: receiving incremental business data transmitted by an acquisition engine; identifying the service type of the incremental service data, and calling a processing model corresponding to the service type to perform service processing on the incremental service data; transmitting the processed incremental business data to a target database, and displaying the processed incremental business data through a display engine configured in the target database. According to the embodiment, the acquisition engine is configured in the source database, and the display engine is configured in the target database, so that the problems of manual acquisition and manual visual display of service data depending on manpower in the prior art are solved, the original structure of the databases cannot be affected, the message middleware is arranged between the two databases, the development cost of data processing of the databases is further reduced, and the data processing efficiency is improved.

Description

Data processing method and device
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data processing method and apparatus.
Background
In the traditional method, service data is extracted from a relational database, manual operation is often carried out by a developer, visual display of the service data is carried out manually, the developer needs to consider front-end display and back-end access logic, the whole database development process is time-consuming and labor-consuming, and the maintenance cost is high.
Disclosure of Invention
In view of the above, embodiments of the present invention provide a data processing method and apparatus, which at least can solve the problem in the prior art that the manual collection and manual display of service data on a database are relied on.
To achieve the above object, according to an aspect of an embodiment of the present invention, there is provided a data processing method including:
receiving incremental business data transmitted by an acquisition engine; the acquisition engine is configured in the source database and is used for calling a preconfigured acquisition statement when the existence of the incremental service data in the source database is monitored, and acquiring the incremental service data;
identifying the service type of the incremental service data, and calling a processing model corresponding to the service type to perform service processing on the incremental service data;
Transmitting the processed incremental business data to a target database, and displaying the processed incremental business data through a display engine configured in the target database.
Optionally, when the collection engine is configured in the source database and is used for calling a pre-configured collection sentence when the existence of the incremental business data in the source database is monitored, before the incremental business data is collected, the collection engine further comprises:
Receiving an operation of configuring an acquisition engine on the source database;
Receiving an operation of configuring a first connection string and an acquisition statement connected to a source database in a configuration file of an acquisition engine; the first connection string is used for establishing access rights of the acquisition engine to the source database.
Optionally, before the displaying engine configured in the target database performs displaying processing on the processed incremental service data, the method further includes:
receiving an operation of a target database configuration display engine;
Receiving an operation of configuring a second connection string connected to the target database in a configuration file of the presentation engine; the second connection string is used for establishing the access authority of the display engine to the target database.
Optionally, the method is applied to message middleware, and the message middleware comprises a first message middleware and a second message middleware;
The receiving the incremental service data transmitted by the acquisition engine comprises the following steps: the first message middleware receives incremental business data which are transmitted by the acquisition engine one by one to generate stream data;
the identifying the service type of the incremental service data comprises the following steps: the second message middleware reads the stream data from the first message middleware one by one, and identifies the service type of each stream data.
Optionally, the collection engine is further configured to determine a service table where the incremental service data is located in the source database, obtain a table name of the service table, and extract a feature word from the table name;
The first message middleware receives incremental service data transmitted by the acquisition engine one by one to generate stream data, and the method further comprises the following steps: querying a first partition corresponding to the feature word, and transmitting the incremental business data to the first partition to generate stream data through the first partition;
The second message middleware reads stream data from the first message middleware one by one, and the method further comprises the following steps: and inquiring a second partition corresponding to the characteristic word, and transmitting the stream data to the second partition.
Optionally, the displaying the processed incremental service data by a display engine configured in the target database includes:
determining indexes which are pre-configured in the target database and correspond to the feature words through a display engine configured in the target database;
inquiring a display mode corresponding to the index, and displaying the processed incremental service data by using the display mode.
Optionally, the source database is a relational database.
To achieve the above object, according to another aspect of an embodiment of the present invention, there is provided a data processing apparatus including:
The receiving module is used for receiving the incremental business data transmitted by the acquisition engine; the acquisition engine is configured in the source database and is used for calling a preconfigured acquisition statement when the existence of the incremental service data in the source database is monitored, and acquiring the incremental service data;
The processing module is used for identifying the service type of the incremental service data, and calling a processing model corresponding to the service type so as to perform service processing on the incremental service data;
and the transmission module is used for transmitting the processed incremental business data to the target database so as to display the processed incremental business data through a display engine configured in the target database.
Optionally, the method further comprises a first configuration module for:
Receiving an operation of configuring an acquisition engine on the source database;
Receiving an operation of configuring a first connection string and an acquisition statement connected to a source database in a configuration file of an acquisition engine; the first connection string is used for establishing access rights of the acquisition engine to the source database.
Optionally, the method further comprises a second configuration module for:
receiving an operation of a target database configuration display engine;
Receiving an operation of configuring a second connection string connected to the target database in a configuration file of the presentation engine; the second connection string is used for establishing the access authority of the display engine to the target database.
Optionally, the method is applied to message middleware, and the message middleware comprises a first message middleware and a second message middleware;
the receiving module is used for: the first message middleware receives incremental business data which are transmitted by the acquisition engine one by one to generate stream data;
the processing module is used for: the second message middleware reads the stream data from the first message middleware one by one, and identifies the service type of each stream data.
Optionally, the collection engine is further configured to determine a service table where the incremental service data is located in the source database, obtain a table name of the service table, and extract a feature word from the table name;
the receiving module is further configured to: querying a first partition corresponding to the feature word, and transmitting the incremental business data to the first partition to generate stream data through the first partition;
The processing module is further configured to: and inquiring a second partition corresponding to the characteristic word, and transmitting the stream data to the second partition.
Optionally, the target database is configured to:
determining indexes which are pre-configured in the target database and correspond to the feature words through a display engine configured in the target database;
inquiring a display mode corresponding to the index, and displaying the processed incremental service data by using the display mode.
Optionally, the source database is a relational database.
To achieve the above object, according to still another aspect of an embodiment of the present invention, there is provided a data processing electronic device.
The electronic equipment of the embodiment of the invention comprises: one or more processors; and a storage device for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement any of the data processing methods described above.
To achieve the above object, according to still another aspect of the embodiments of the present invention, there is provided a computer-readable medium having stored thereon a computer program which, when executed by a processor, implements any of the above-described data processing methods.
To achieve the above object, according to still another aspect of an embodiment of the present invention, there is provided a computer program product. A computer program product according to an embodiment of the present invention includes a computer program that, when executed by a processor, implements a data processing method according to an embodiment of the present invention.
According to the solution provided by the present invention, one embodiment of the above invention has the following advantages or beneficial effects: the method comprises the steps of configuring an acquisition engine in a source database, configuring an acquisition statement in a configuration file of the acquisition engine, extracting incremental service data from the source database through the acquisition engine, transmitting the incremental service data to a message middleware for processing, and finally storing the processed data into a target database for visual display through a display engine configured in the target database, so that the incremental service data of the source database are subjected to service monitoring visualization, the effectiveness of service monitoring is improved, and the operation and maintenance cost is reduced.
Further effects of the above-described non-conventional alternatives are described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a schematic flow diagram of a data processing method according to an embodiment of the present invention;
FIG. 2 is a flow chart of an alternative data processing method according to an embodiment of the invention;
FIG. 3 is a flow chart of another alternative data processing method according to an embodiment of the invention;
FIG. 4 is a flow chart of yet another alternative data processing method according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of the main modules of a data processing apparatus according to an embodiment of the present invention;
FIG. 6 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;
Fig. 7 is a schematic diagram of a computer system suitable for use in implementing a mobile device or server of an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention will now be described with reference to the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding, and are to be considered merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
It is noted that embodiments of the invention and features of the embodiments may be combined with each other without conflict. In the technical scheme of the invention, the related aspects of acquisition, analysis, use, transmission, storage and the like of the personal information of the user accord with the regulations of related laws and regulations, are used for legal and reasonable purposes, are not shared, leaked or sold outside the legal use aspects and the like, and are subjected to supervision and management of a supervision department. Necessary measures should be taken for the personal information of the user to prevent illegal access to such personal information data, ensure that personnel having access to the personal information data comply with the regulations of the relevant laws and regulations, and ensure the personal information of the user.
Once these user personal information data are no longer needed, the risk should be minimized by limiting or even prohibiting the data collection and/or deletion. User privacy is protected, when applicable, by de-identifying the data, including in some related applications, such as by removing a particular identifier (e.g., date of birth, etc.), controlling the amount or specificity of stored data (e.g., collecting location data at a city level rather than at a specific address level), controlling how the data is stored, and/or other methods.
Referring to fig. 1, a main flowchart of a data processing method provided by an embodiment of the present invention is shown, including the following steps:
S101: receiving incremental business data transmitted by an acquisition engine; the acquisition engine is configured in the source database and is used for calling a preconfigured acquisition statement when the existence of the incremental service data in the source database is monitored, and acquiring the incremental service data;
s102: identifying the service type of the incremental service data, and calling a processing model corresponding to the service type to perform service processing on the incremental service data;
S103: transmitting the processed incremental business data to a target database, and displaying the processed incremental business data through a display engine configured in the target database.
In the above embodiment, for step S101, the data extraction means that the data required for the target data source system is extracted from the source data source system. The source database is a relational database, such as an Oracle database, the relational database refers to a database which adopts a relational model to organize data, the data is stored in a form of rows and columns so as to be convenient for users to understand, the series of rows and columns of the relational database are called tables, and a group of tables form the database.
The method comprises the steps of pre-installing the Flume software (namely an acquisition engine) in a relational database, and after the Flume software is installed, configuring a first connection string for connecting the relational database and SQL (Structured Query Language ) acquisition sentences, such as agent test.
Wherein, the fly is a high-availability, high-reliability, distributed system for collecting, aggregating and transmitting mass logs, which is provided by Cloudera, and supports customizing various data transmitters in the log system for collecting data; meanwhile, the jume provides the ability to simply process data and write to various data recipients (customizable).
In the development of various application programs of the database, the connection database is the first step of the development of the application program of the database, and is also the most important step, and the connection modes of the connection database and the corresponding connection strings are different from database to database. Such as agent test.sources.source test.hibernate. Connection. User: designating a user name of the connection database, agent test.source test.hibernate.connection.password: a password to connect to the database is specified.
After configuration is completed, the jump software monitors whether incremental service data exist in the relational database in real time, including but not limited to new and modified service transaction quantity, transaction success, transaction response time, transaction duration and the like, if yes, data acquisition is performed according to a preconfigured SQL statement to extract the service data in the relational database in real time, and then the acquired incremental service data are output to the target database in real time.
For step S102, the number of incremental service data collected by SQL is usually large, for example, 100 ten thousand incremental service data are 1 second, if the incremental service data are directly transmitted to the target database, a certain network bandwidth pressure exists, and in order to alleviate this problem, message middleware is set between the source database and the target database in the scheme. The incremental business data collected by the Flume software of the source database is firstly sent to the message middleware, the message middleware carries out business processing on the incremental business data, and then the processed results are sent to the target database one by one.
Different processing models, such as asset service, liability service, intermediate service and off-list service, are preset in the message middleware aiming at different service types. 1) Liability business refers to business that a commercial bank raised funds and forms a funding source, and is the basis of commercial bank asset business and other business, mainly including deposit business. 2) The property business refers to business that commercial banks acquire profits by using funds in various ways, and mainly includes cash business, loan business and investment business. 3) The intermediate business is a business that the commercial bank does not directly accept or form the liability and debt, does not use own funds to transact commissions such as payment for the clients and charges the commissions to the clients, and mainly comprises a settlement business, an agency business, a consultation business, a hosting business and the like. 4) Excellent transactions refer to transactions undertaken by commercial banks that are not included in the balance sheet and do not affect the total assets and total liabilities of the bank, including guaranty, financial derivative transactions such as long-term foreign exchange contracts, currency exchanges, currency futures, currency options, interest rate options, fingers, futures, options, and the like.
The process models herein include, but are not limited to, data cleansing, machining, and formatting processes. For example, it is determined whether the data format is the same as the format required by the target database, and if it is different, format conversion is required. For example, some data include junk data, which requires string interception, and finally forms an input result meeting the target database requirement jason.
For step S103, the message middleware transmits the processed incremental business data to a target database, such as an elastomer search, which is a distributed search and analysis engine located at the ELASTIC STACK core. In the scheme, a display engine, such as Kibana software, is also pre-configured in the elastic search to perform visual display processing on the incremental service data through Kibana software, so as to realize corresponding service monitoring.
Wherein Kibana is an open source analysis and visualization platform designed for use with the elastomer search, which can use Kibana to search, view, interact with data stored in the elastomer search index, and can easily demonstrate advanced data analysis and visualization using a variety of different charts, tables, maps, etc., kibana.
In the scheme, after the software is pre-configured and installed Kibana in the elastomer search, the configuration Kibana is connected to a second connection string of the elastomer search in a configuration file of the Kibana software, and the second connection string is used for providing Kibana software with access rights to the elastomer search.
According to the method provided by the embodiment, the problems of the existing manual collection and visual display are solved by configuring the collection engine in the source database and the display engine in the target database, the original structure of the databases is not affected, and the message middleware is arranged between the two databases, so that the development cost of data processing of the databases is further reduced, and the data processing speed is increased.
Referring to fig. 2, an alternative flow chart of a data processing method according to an embodiment of the invention is shown, comprising the following steps:
S201: the first message middleware receives incremental business data which are transmitted by the acquisition engine one by one to generate stream data; the acquisition engine is configured in the source database and is used for calling a preconfigured acquisition statement when the existence of the incremental service data in the source database is monitored, and acquiring the incremental service data;
S202: the second message middleware reads the stream data from the first message middleware one by one, identifies the service type of each stream data, and invokes a processing model corresponding to the service type so as to process the service of each stream data;
S203: the second message middleware transmits the processed incremental business data to a target database so as to display the processed incremental business data through a display engine configured in the target database.
In the above embodiment, for steps S201 to S203, the message middleware set in the scheme is divided into a first message middleware and a second message middleware, such as Kafka and logstack, where Kafka is a high throughput distributed publish-subscribe message system, and is written by Scala and Java, and can process all action stream data of consumers. The Logstar is a real-time pipeline open-source log collection engine, and can dynamically clean, process and format data from different sources so as to be used for data analysis and visualization.
The amount of incremental service data collected by the jump software installed in the relational database is large, if the incremental service data are directly transmitted to the logstack, the logstack processing efficiency can be affected because the amount of data processed per second by the logstack is limited. Therefore, in order to effectively avoid data loss, the scheme preferably transmits the collected incremental service data to the entrance of Kafka one by one, and the agent test.sinks.sinktest.type: the type of receiver is designated KAFKASINK. The Kafka receives incremental service data acquired by the Flume one by one, the incremental service data are ordered through channels of the Kafka to generate stream data, and then the Logstar receives the data one by one from an outlet of the channels of the Kafka in real time by using an input module.
And likewise, after the Logstar acquires the stream data, cleaning, processing and formatting the stream data through a Filter configuration. For example, the format of the queried data is different from that of the target database, and the format of the queried data needs to be converted. For example, some data include junk data, and it is necessary to perform string interception processing. And then ultimately forms an input result that meets the elastic search requirement jason. The processed stream data is output to the elastic search in real time through the output configuration of the Logstar.
The method provided by the embodiment can realize the purposes of collecting data in a source database and visually displaying in a target database by using open-source free software on the market, does not influence the original structure of the database, and can be replaced by developing software with similar functions by a developer, such as logstar, and can be replaced by using spark software by combining with self-developed codes.
Referring to FIG. 3, another alternative flow chart of a data processing method according to an embodiment of the invention is shown, comprising the steps of:
S301: the acquisition engine is configured in the source database, and is used for calling a preconfigured acquisition statement when the existence of incremental service data in the source database is monitored, acquiring the incremental service data, determining a service table in which the incremental service data is positioned in the source database, acquiring a table name of the service table, and extracting feature words from the table name;
s302: the method comprises the steps that a first message middleware receives incremental business data which are transmitted by an acquisition engine one by one, inquires a first partition corresponding to the feature words, and transmits the incremental business data to the first partition so as to generate stream data through the first partition;
S303: the second message middleware reads the stream data from the first message middleware one by one, inquires a second partition corresponding to the feature word, and transmits the stream data to the second partition;
S304: in the second partition, identifying a service type of each stream data, and calling a processing model corresponding to the service type to perform service processing on each stream data;
S305: the second message middleware transmits the processed incremental business data to a target database so as to display the processed incremental business data through a display engine configured in the target database.
In the above embodiment, the description of step S305 may be referred to as fig. 2, and will not be repeated here.
For steps S301 to S304, since the acquisition engine acquires more incremental service data per second, the incremental service data transmitted to Kafka and logstack are also larger, and in this scheme, partitions are set in Kafka, and different partitions are directed against different feature words, such as A, B, C. Specifically, the agent test.sinks.sinktest.topic: the name of the Kafka theme to be transmitted to is specified.
And according to the characteristics of the relational database, the data are stored in the form of rows and columns for the convenience of the user to understand, the series of rows and columns of the relational database is called a table, and a group of tables forms the database. The incremental business data collected by the collection engine is essentially the data in the business table, so that the business table in which each incremental business data is located in the source database also needs to be determined. Corresponding feature words are preset in the table names of each service table, such as A, B, C.
After Kafka receives incremental traffic data transmitted by the acquisition engine one by one, it also needs to query the partition (i.e., the first partition) where the data is located according to its feature words to create stream data through the partition. For example, in the acquired incremental service data, the data amount of the corresponding feature word a is 200, the data amount of the corresponding feature word B is 300, the data amount of the corresponding feature word C is 200, if a partition mode is not adopted, 700 data are included in the generated stream data, after the partition mode is adopted, the stream data of the corresponding feature word a only include 200 data, the stream data of the corresponding feature word B only include 300 data, and the stream data of the corresponding feature word C only include 200 data, so that the data splitting purpose is realized.
The corresponding logstack is also provided with partitions (namely second partitions) corresponding to the feature words in advance, and each partition is provided with different processing models according to different service types in advance. After the logstack acquires stream data from the Kafka one by one, data distribution is also needed according to the characteristic words, so that parallel processing of a plurality of increment business data is realized, and the data processing efficiency is improved. And then, the processed data can be simultaneously sent to the target database for visual display by different partitions.
According to the method provided by the embodiment, the partitions corresponding to the feature words are preset in different message middleware, so that the multiple incremental business data are processed in different partitions at the same time, and the data processing efficiency is improved.
Referring to FIG. 4, a flowchart of yet another alternative data processing method according to an embodiment of the present invention is shown, comprising the steps of:
s401: the acquisition engine is configured in the source database, and is used for calling a preconfigured acquisition statement when the existence of incremental service data in the source database is monitored, acquiring the incremental service data, determining a service table in which the incremental service data is positioned in the source database, acquiring a table name of the service table, and extracting feature words from the table name;
S402: the method comprises the steps that a first message middleware receives incremental business data which are transmitted by an acquisition engine one by one, inquires a first partition corresponding to the feature words, and transmits the incremental business data to the first partition so as to generate stream data through the first partition;
s403: the second message middleware reads the stream data from the first message middleware one by one, inquires a second partition corresponding to the feature word, and transmits the stream data to the second partition;
s404: in the second partition, identifying the service type of each stream data, and calling a processing model corresponding to the service type to perform service processing on each stream data;
S405: the second message middleware transmits the processed incremental business data to a target database so as to determine indexes which are pre-configured in the target database and correspond to the feature words through a display engine configured in the target database;
s406: inquiring a display mode corresponding to the index, and displaying the processed incremental service data by using the display mode.
In the above embodiment, the steps S401 to S404 may be described with reference to fig. 3, and are not described herein.
For steps S405 and S406, the second message middleware stores the processed incremental service data into an elastic search, and the elastic search queries data from an elastic search distributed database through Kibana software, supports the situation that the service is displayed in real time by making a column diagram, a pie diagram, a line diagram, an area diagram and other various presentation modes by self definition, and can clearly check the service operation situation through a chart form instead of checking the service operation situation in a relational database, thereby realizing service monitoring.
However, different incremental service data may support different display forms, such as a transaction data support histogram, a payment data support pie chart and a query data support line chart, and according to the scheme, different indexes are preset in the target database for different feature words, and the indexes are used for query display modes, such as a histogram, a pie chart and a line chart, for the display modes corresponding to the feature words A, B, C, so that the adaptive display of different incremental service data is realized, and the method has the characteristic of flexible customization.
According to the method provided by the embodiment, different display modes are preset for different feature words by the target database, and the feature words are arranged in the table names of the service table, so that diversified display of different incremental service data is realized, and the follow-up service monitoring and using are facilitated.
According to the method provided by the embodiment of the invention, the Flume is configured in the relational database, the SQL acquisition statement is configured in the configuration file of the Flume, incremental service data is extracted from the relational database and transmitted to the Kafka for stream processing, the Logflash reads the stream data from the Kafka, the stream data is cleaned, processed and formatted, and finally the processed data is stored in the target database ES, so that the data is visually displayed through the open source software kibana configured in the ES, and compared with the traditional mode, the method can realize related service monitoring more efficiently, more flexibly and intelligently, thereby improving the operation and maintenance efficiency and reducing the operation and maintenance cost.
Referring to fig. 5, a schematic diagram of main modules of a data processing apparatus 500 according to an embodiment of the present invention is shown, including:
The receiving module 501 is configured to receive incremental service data transmitted by the acquisition engine; the acquisition engine is configured in the source database and is used for calling a preconfigured acquisition statement when the existence of the incremental service data in the source database is monitored, and acquiring the incremental service data;
The processing module 502 is configured to identify a service type of the incremental service data, and call a processing model corresponding to the service type to perform service processing on the incremental service data;
and the transmission module 503 is configured to transmit the processed incremental service data to a target database, so as to perform display processing on the processed incremental service data through a display engine configured in the target database.
The implementation device of the invention further comprises a first configuration module for:
Receiving an operation of configuring an acquisition engine on the source database;
Receiving an operation of configuring a first connection string and an acquisition statement connected to a source database in a configuration file of an acquisition engine; the first connection string is used for establishing access rights of the acquisition engine to the source database.
The implementation device of the invention further comprises a second configuration module for:
receiving an operation of a target database configuration display engine;
Receiving an operation of configuring a second connection string connected to the target database in a configuration file of the presentation engine; the second connection string is used for establishing the access authority of the display engine to the target database.
The implementation device is applied to the message middleware, and the message middleware comprises a first message middleware and a second message middleware;
The receiving module 501 is configured to: the first message middleware receives incremental business data which are transmitted by the acquisition engine one by one to generate stream data;
The processing module 502 is configured to: the second message middleware reads the stream data from the first message middleware one by one, and identifies the service type of each stream data.
In the embodiment of the invention, the acquisition engine is also used for determining a service table in which the incremental service data is located in a source database, acquiring a table name of the service table, and extracting feature words from the table name;
The receiving module 501 is further configured to: querying a first partition corresponding to the feature word, and transmitting the incremental business data to the first partition to generate stream data through the first partition;
The processing module 502 is further configured to: and inquiring a second partition corresponding to the characteristic word, and transmitting the stream data to the second partition.
In the implementation device of the present invention, the target database is configured to:
determining indexes which are pre-configured in the target database and correspond to the feature words through a display engine configured in the target database;
inquiring a display mode corresponding to the index, and displaying the processed incremental service data by using the display mode.
In the implementation device of the invention, the source database is a relational database.
In addition, the implementation of the apparatus in the embodiments of the present invention has been described in detail in the above method, so that the description is not repeated here.
Fig. 6 shows an exemplary system architecture 600 in which embodiments of the invention may be applied, including terminal devices 601, 602, 603, a network 604, and a server 605 (by way of example only).
The terminal devices 601, 602, 603 may be various electronic devices having a display screen and supporting web browsing, are installed with various communication client applications, and a user may interact with the server 605 through the network 604 using the terminal devices 601, 602, 603 to receive or transmit messages, etc.
The network 604 is used as a medium to provide communication links between the terminal devices 601, 602, 603 and the server 605. The network 604 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
The server 605 may be a server providing various services, and it should be noted that, the method provided by the embodiment of the present invention is generally executed by the server 605, and accordingly, the apparatus is generally disposed in the server 605.
It should be understood that the number of terminal devices, networks and servers in fig. 6 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 7, there is illustrated a schematic diagram of a computer system 700 suitable for use in implementing an embodiment of the present invention. The terminal device shown in fig. 7 is only an example, and should not impose any limitation on the functions and the scope of use of the embodiment of the present invention.
As shown in fig. 7, the computer system 700 includes a Central Processing Unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 702 or a program loaded from a storage section 708 into a Random Access Memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, ROM 702, and RAM 703 are connected to each other through a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.
The following components are connected to the I/O interface 705: an input section 706 including a keyboard, a mouse, and the like; an output portion 707 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, a speaker, and the like; a storage section 708 including a hard disk or the like; and a communication section 709 including a network interface card such as a LAN card, a modem, or the like. The communication section 709 performs communication processing via a network such as the internet. The drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that a computer program read therefrom is mounted into the storage section 708 as necessary.
In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 709, and/or installed from the removable medium 711. The above-described functions defined in the system of the present invention are performed when the computer program is executed by a Central Processing Unit (CPU) 701.
The computer readable medium shown in the present invention may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules involved in the embodiments of the present invention may be implemented in software or in hardware. The described modules may also be provided in a processor, for example, as: a processor comprises a receiving module, a processing module and a transmission module. The names of these modules do not in any way constitute a limitation of the module itself, for example, a transmission module may also be described as a "module transmitted to the target database".
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be present alone without being fitted into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to perform any of the data processing methods described above.
The computer program product of the invention comprises a computer program which, when being executed by a processor, implements the data processing method in the embodiments of the invention.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives can occur depending upon design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.

Claims (9)

1. The data processing method is characterized by being applied to message middleware, wherein the message middleware comprises a first message middleware and a second message middleware, the first message middleware is kafka, the second message middleware is logstar, and the method comprises the following steps:
The first message middleware receives incremental business data transmitted by the acquisition engine to generate stream data; the acquisition engine is configured in the source database and is used for calling a preconfigured acquisition statement when the existence of the incremental service data in the source database is monitored, and acquiring the incremental service data; the acquisition engine is Flume software and is also used for determining a service table in which the incremental service data are located in a source database, acquiring a table name of the service table and extracting feature words from the table name; the first message middleware is further used for inquiring a first partition corresponding to the feature word, and transmitting the incremental business data to the first partition so as to generate stream data through the first partition;
The second message middleware reads the stream data from the first message middleware one by one, identifies the service type of each stream data, and invokes a processing model corresponding to the service type to process the incremental service data; the second message middleware is further used for inquiring a second partition corresponding to the feature word and transmitting the stream data to the second partition; and
Transmitting the processed incremental business data to a target database so as to display the processed incremental business data through a display engine configured in the target database, wherein the method comprises the following steps of: determining indexes which are pre-configured in the target database and correspond to the feature words through a display engine configured in the target database; inquiring a display mode corresponding to the index, and displaying the processed incremental business data by using the display mode;
The display engine is Kibana software and receives the operation of configuring the display engine to the target database; receiving an operation of configuring a second connection string connected to the target database in a configuration file of the presentation engine; the second connection string is used for establishing the access authority of the display engine to the target database.
2. The method of claim 1, wherein when the collection engine is configured in the source database for invoking a pre-configured collection statement when it is monitored that incremental traffic data exists in the source database, further comprising, prior to collecting the incremental traffic data:
Receiving an operation of configuring an acquisition engine on the source database;
Receiving an operation of configuring a first connection string and an acquisition statement connected to a source database in a configuration file of an acquisition engine; the first connection string is used for establishing access rights of the acquisition engine to the source database.
3. A method according to claim 1 or 2, wherein the source database is a relational database.
4. A data processing apparatus, applied to a message middleware, the message middleware including a first message middleware and a second message middleware, the first message middleware being kafka, the second message middleware being logstack, comprising:
The receiving module is used for receiving the incremental service data transmitted by the acquisition engine by the first message middleware to generate stream data; the acquisition engine is configured in the source database and is used for calling a preconfigured acquisition statement when the existence of the incremental service data in the source database is monitored, and acquiring the incremental service data; the acquisition engine is Flume software and is also used for determining a service table in which the incremental service data are located in a source database, acquiring a table name of the service table and extracting feature words from the table name; the first message middleware is further used for inquiring a first partition corresponding to the feature word, and transmitting the incremental business data to the first partition so as to generate stream data through the first partition;
The processing module is used for reading the stream data from the first message middleware one by the second message middleware, identifying the service type of each stream data, and calling a processing model corresponding to the service type so as to perform service processing on the incremental service data; the second message middleware is further used for inquiring a second partition corresponding to the feature word and transmitting the stream data to the second partition;
the transmission module is used for transmitting the processed incremental business data to the target database so as to display the processed incremental business data through a display engine configured in the target database, and comprises the following steps: determining indexes which are pre-configured in the target database and correspond to the feature words through a display engine configured in the target database; inquiring a display mode corresponding to the index, and displaying the processed incremental business data by using the display mode;
The display engine is Kibana software, and the second configuration module is used for receiving the operation of configuring the display engine on the target database; receiving an operation of configuring a second connection string connected to the target database in a configuration file of the presentation engine; the second connection string is used for establishing the access authority of the display engine to the target database.
5. The apparatus of claim 4, further comprising a first configuration module configured to:
Receiving an operation of configuring an acquisition engine on the source database;
Receiving an operation of configuring a first connection string and an acquisition statement connected to a source database in a configuration file of an acquisition engine; the first connection string is used for establishing access rights of the acquisition engine to the source database.
6. The apparatus of claim 4 or 5, wherein the source database is a relational database.
7. An electronic device, comprising:
One or more processors;
storage means for storing one or more programs,
When executed by the one or more processors, causes the one or more processors to implement the method of any of claims 1-3.
8. A computer readable medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the method according to any of claims 1-3.
9. A computer program product comprising a computer program which, when executed by a processor, implements the method of any of claims 1-3.
CN202311137102.3A 2023-09-05 2023-09-05 Data processing method and device Active CN116860898B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311137102.3A CN116860898B (en) 2023-09-05 2023-09-05 Data processing method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311137102.3A CN116860898B (en) 2023-09-05 2023-09-05 Data processing method and device

Publications (2)

Publication Number Publication Date
CN116860898A CN116860898A (en) 2023-10-10
CN116860898B true CN116860898B (en) 2024-04-23

Family

ID=88223805

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311137102.3A Active CN116860898B (en) 2023-09-05 2023-09-05 Data processing method and device

Country Status (1)

Country Link
CN (1) CN116860898B (en)

Citations (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018113580A1 (en) * 2016-12-19 2018-06-28 腾讯科技(深圳)有限公司 Data management method and server
CN110334070A (en) * 2019-05-21 2019-10-15 中国人民财产保险股份有限公司 Data processing method, system, equipment and storage medium
CN110879813A (en) * 2019-11-20 2020-03-13 浪潮软件股份有限公司 Binary log analysis-based MySQL database increment synchronization implementation method
CN111723160A (en) * 2020-08-24 2020-09-29 国网浙江省电力有限公司 Multi-source heterogeneous incremental data synchronization method and system
CN111930385A (en) * 2020-07-28 2020-11-13 苏州亿歌网络科技有限公司 Data acquisition method, device, equipment and storage medium
CN111984728A (en) * 2020-08-14 2020-11-24 北京人大金仓信息技术股份有限公司 Heterogeneous database data synchronization method, device, medium and electronic equipment
CN112231274A (en) * 2020-10-16 2021-01-15 京东数字科技控股股份有限公司 Log summarizing method and device, electronic equipment and storage medium
CN112883118A (en) * 2021-03-31 2021-06-01 浪潮云信息技术股份公司 Method and system for synchronously acquiring incremental data based on sql
CN113407601A (en) * 2020-03-17 2021-09-17 北京国双科技有限公司 Data acquisition method and device, storage medium and electronic equipment
CN113553310A (en) * 2021-09-22 2021-10-26 深圳市信润富联数字科技有限公司 Data acquisition method and device, storage medium and electronic equipment
CN114048217A (en) * 2021-10-21 2022-02-15 微民保险代理有限公司 Incremental data synchronization method and device, electronic equipment and storage medium
KR20220060429A (en) * 2020-11-04 2022-05-11 아토리서치(주) System for collecting log data of remote network switches and method for constructing big-data thereof
CN114661823A (en) * 2022-04-01 2022-06-24 中国人民财产保险股份有限公司 Data synchronization method and device, electronic equipment and readable storage medium
CN114691700A (en) * 2020-12-28 2022-07-01 广东飞企互联科技股份有限公司 Kafaka cluster-based intelligent park retrieval method
CN115391403A (en) * 2022-08-29 2022-11-25 中电金信软件有限公司 Data integration method and data integration device based on rule engine
CN116126238A (en) * 2022-12-30 2023-05-16 中国电信股份有限公司 Data storage method, system, device and nonvolatile storage medium
CN116644136A (en) * 2023-05-30 2023-08-25 中国银行股份有限公司 Data acquisition method, device, equipment and medium for increment and full data

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040064442A1 (en) * 2002-09-27 2004-04-01 Popovitch Steven Gregory Incremental search engine
US10983895B2 (en) * 2018-06-05 2021-04-20 Unravel Data Systems, Inc. System and method for data application performance management

Patent Citations (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018113580A1 (en) * 2016-12-19 2018-06-28 腾讯科技(深圳)有限公司 Data management method and server
CN110334070A (en) * 2019-05-21 2019-10-15 中国人民财产保险股份有限公司 Data processing method, system, equipment and storage medium
CN110879813A (en) * 2019-11-20 2020-03-13 浪潮软件股份有限公司 Binary log analysis-based MySQL database increment synchronization implementation method
CN113407601A (en) * 2020-03-17 2021-09-17 北京国双科技有限公司 Data acquisition method and device, storage medium and electronic equipment
CN111930385A (en) * 2020-07-28 2020-11-13 苏州亿歌网络科技有限公司 Data acquisition method, device, equipment and storage medium
CN111984728A (en) * 2020-08-14 2020-11-24 北京人大金仓信息技术股份有限公司 Heterogeneous database data synchronization method, device, medium and electronic equipment
CN111723160A (en) * 2020-08-24 2020-09-29 国网浙江省电力有限公司 Multi-source heterogeneous incremental data synchronization method and system
CN112231274A (en) * 2020-10-16 2021-01-15 京东数字科技控股股份有限公司 Log summarizing method and device, electronic equipment and storage medium
KR20220060429A (en) * 2020-11-04 2022-05-11 아토리서치(주) System for collecting log data of remote network switches and method for constructing big-data thereof
CN114691700A (en) * 2020-12-28 2022-07-01 广东飞企互联科技股份有限公司 Kafaka cluster-based intelligent park retrieval method
CN112883118A (en) * 2021-03-31 2021-06-01 浪潮云信息技术股份公司 Method and system for synchronously acquiring incremental data based on sql
CN113553310A (en) * 2021-09-22 2021-10-26 深圳市信润富联数字科技有限公司 Data acquisition method and device, storage medium and electronic equipment
CN114048217A (en) * 2021-10-21 2022-02-15 微民保险代理有限公司 Incremental data synchronization method and device, electronic equipment and storage medium
CN114661823A (en) * 2022-04-01 2022-06-24 中国人民财产保险股份有限公司 Data synchronization method and device, electronic equipment and readable storage medium
CN115391403A (en) * 2022-08-29 2022-11-25 中电金信软件有限公司 Data integration method and data integration device based on rule engine
CN116126238A (en) * 2022-12-30 2023-05-16 中国电信股份有限公司 Data storage method, system, device and nonvolatile storage medium
CN116644136A (en) * 2023-05-30 2023-08-25 中国银行股份有限公司 Data acquisition method, device, equipment and medium for increment and full data

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
Design and Implementation of Air Quality Data Processing System Based on Big Data Technology;Xin Zhu.et al;2018 IEEE 4th International Conference on Computer and Communications: ICCC 2018;20201210;第1846-1850页 *

Also Published As

Publication number Publication date
CN116860898A (en) 2023-10-10

Similar Documents

Publication Publication Date Title
US11720527B2 (en) API for implementing scoring functions
CN109034988B (en) Accounting entry generation method and device
CN111177231A (en) Report generation method and report generation device
WO2020248883A1 (en) Debt collection method, system and apparatus
CN111429241A (en) Accounting processing method and device
CN110795478A (en) Data warehouse updating method and device applied to financial business and electronic equipment
CN112417274A (en) Message pushing method and device, electronic equipment and storage medium
CN113722433A (en) Information pushing method and device, electronic equipment and computer readable medium
CN116860898B (en) Data processing method and device
CN112837149A (en) Method and device for identifying enterprise credit risk
CN110930238A (en) Method, device, equipment and computer readable medium for improving audit task efficiency
CN115330540A (en) Method and device for processing transaction data
CN115422202A (en) Service model generation method, service data query method, device and equipment
US20190164094A1 (en) Risk rating analytics based on geographic regions
CN112783615B (en) Data processing task cleaning method and device
CN114723455A (en) Service processing method and device, electronic equipment and storage medium
CN114723548A (en) Data processing method, apparatus, device, medium, and program product
CN113450208A (en) Loan risk change early warning and model training method and device
CN113779017A (en) Method and apparatus for data asset management
CN113704222A (en) Method and device for processing service request
CN114997977B (en) Data processing method, device, electronic equipment and computer readable medium
CN115829753B (en) Cross-border securities business exchange method and system based on blockchain
CN114529382A (en) Account checking data processing method and device
CN115795508A (en) Method, device, equipment and computer readable medium for processing business data
CN112862608A (en) Transaction data matching method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant