CN115391407A

CN115391407A - Data stream loose coupling statistical method, device, equipment, medium and program product

Info

Publication number: CN115391407A
Application number: CN202211059808.8A
Authority: CN
Inventors: 韦东杰; 贾国琛; 靳国栋; 田媛; 张汉至
Original assignee: China Construction Bank Corp; CCB Finetech Co Ltd
Current assignee: China Construction Bank Corp; CCB Finetech Co Ltd
Priority date: 2022-08-31
Filing date: 2022-08-31
Publication date: 2022-11-25

Abstract

The embodiment of the disclosure provides a data flow loose coupling statistical method, which relates to the field of financial science and technology and big data, and the method comprises the following steps: acquiring a service data stream, wherein the service data stream is the sum of real-time data of various types of services; extracting basic data and real-time service data of various types of services from the service data stream, wherein the basic data comprises associated data among the services with dependency relationship; supplementing the basic data to the real-time service data of each type of service to obtain loosely coupled service data of each type of service; and respectively storing the loosely coupled service data of each type of service into a distributed deployed message queue to perform classified statistics. The present disclosure also provides a data flow loose coupling statistical apparatus, a device, a storage medium, and a program product.

Description

Data stream loose coupling statistical method, device, equipment, medium and program product

Technical Field

The present disclosure relates to the field of big data, and more particularly, to a data flow loose coupling statistical method, apparatus, device, medium, and program product.

Background

In the financial industry, transaction data generated by transaction can be recorded in a service database in real time, and generally, the data is imported into an analysis database through an Extract-Transform-Load (ETL) technology in the later period when a service system stops service, so that the data is summarized, and powerful data support is provided for daily operation activities of branches.

With the development of the financial industry, a plurality of application scenarios need to acquire real-time statistical data, and precious time is won for implementation of financial strategies. Because the business logic of the financial industry is relatively complex, the degree and complexity of interdependence of various business data can be gradually increased along with the expansion of the business logic, namely the coupling degree between the data is increased. The high data coupling degree gradually increases the difficulty of system expandability and business expandability, and limits the system expandability to a certain extent. Therefore, a service partition loose coupling processing solution needs to be researched for the complex situation.

Disclosure of Invention

In view of the foregoing, the present disclosure provides data stream loose coupling statistical methods, apparatuses, devices, media, and program products that improve the degree of data loose coupling.

According to a first aspect of the present disclosure, there is provided a data flow loose coupling statistical method, the method comprising: acquiring a service data stream, wherein the service data stream is a real-time data sum of various types of services; extracting basic data and real-time service data of various types of services from the service data stream, wherein the basic data comprises associated data among the services with dependency relationship; supplementing the basic data to the real-time service data of each type of service to obtain loosely coupled service data of each type of service; and storing the loosely coupled service data of each type of service into a distributed deployed message queue respectively to perform classified statistics.

According to an embodiment of the present disclosure, the extracting basic data and real-time service data of each type of service from the service data stream includes: copying the service data stream into a first service data stream and a second service data stream based on a preset copying command; filtering the first service data stream based on a preset basic keyword, and screening out basic data corresponding to the basic keyword from the service data stream; and classifying the second service data stream according to the service type to obtain the real-time service data of each type of service.

According to an embodiment of the present disclosure, the filtering the first service data stream based on a preset basic keyword to obtain the basic data further includes: and constructing a basic data key value pair by using the basic key words and the basic data values, and storing the basic data key value pair into a basic data retrieval base of a Redis data structure.

According to an embodiment of the present disclosure, the supplementing the basic data to the real-time service data of each type of service to obtain the loosely-coupled service data of each type of service includes: acquiring basic keywords of basic data on which each type of service depends based on the dependency relationship among the services; calling the basic data search library to acquire basic data corresponding to the basic keywords; and supplementing the basic data into real-time service data of the service depending on the basic data to obtain the loosely-coupled service data.

According to the embodiment of the present disclosure, the storing the loosely coupled service data of each type of service into the message queues deployed in a distributed manner respectively to perform classification statistics includes: reading the loosely coupled service data of each type of service from the distributed deployed message queue by adopting a distributed quasi-real-time computing engine, and respectively counting the loosely coupled service data of each type of service; summarizing the counted loosely coupled service data of each type of service to obtain loosely coupled statistical data of the service data stream.

According to an embodiment of the present disclosure, the method further comprises: setting a Redis server cluster, wherein the Redis server cluster comprises 1 main Redis server group and a plurality of auxiliary Redis server groups, the main Redis server group is used for executing the data stream loose coupling statistical method, and the auxiliary Redis server groups are used for backing up data generated on a main Redis server; monitoring the running state of the Redis server cluster; and when the main Redis server group fails to operate, upgrading one of the plurality of slave Redis server groups to a Redis server.

A second aspect of the present disclosure provides a data flow loose coupling statistical apparatus, including: the data flow acquisition module is used for acquiring a service data flow, wherein the service data flow is the sum of real-time data of various types of services; the data extraction module is used for extracting basic data and real-time service data of various types of services from the service data stream, wherein the basic data comprises associated data among the services with dependency relationship; the data complementing module is used for complementing the basic data to the real-time service data of each type of service to obtain loosely coupled service data of each type of service; and the distributed statistical module is used for respectively storing the loosely coupled service data of each type of service into the distributed deployed message queues to execute classified statistics.

A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the data flow loose coupling statistics method described above.

A fourth aspect of the present disclosure also provides a computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the above-described data flow loose coupling statistical method.

A fifth aspect of the present disclosure also provides a computer program product comprising a computer program which, when executed by a processor, implements the above-mentioned data flow loose coupling statistical method.

The at least one technical scheme adopted in the embodiment of the disclosure can achieve the following beneficial effects:

according to the statistical method for loose coupling of data streams, provided by the embodiment of the disclosure, when the data streams are acquired, the basic data depended by each service requirement is obtained in real time, so that the service data of each type of service is completed, and the data dependency relationship among the services is removed, so that loose coupling of data among various services is realized, independence and calculation isolation of subsequent service logic are facilitated, further, quick expansion of the services can be realized, and service deployment is facilitated.

Drawings

The foregoing and other objects, features and advantages of the disclosure will be apparent from the following description of embodiments of the disclosure, which proceeds with reference to the accompanying drawings, in which:

FIG. 1 schematically illustrates an application scenario diagram of a data flow loose coupling statistical method, apparatus, device, medium, and program product according to embodiments of the present disclosure;

FIG. 2 schematically illustrates a flow chart of a data flow loose coupling statistical method according to an embodiment of the present disclosure;

FIG. 3 schematically illustrates a detailed flow chart of a data flow loose coupling statistical method according to an embodiment of the present disclosure;

FIG. 4 schematically illustrates an application of a data flow loose coupling statistical method according to an embodiment of the present disclosure;

FIG. 5 schematically illustrates a schematic diagram of a Redis server cluster according to an embodiment of the disclosure;

FIG. 6 schematically shows a block diagram of a data flow loose coupling statistics apparatus according to an embodiment of the present disclosure; and

fig. 7 schematically illustrates a block diagram of an electronic device suitable for implementing a data flow loose coupling statistical method according to an embodiment of the present disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that these descriptions are illustrative only and are not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs, unless otherwise defined. It is noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of this specification and should not be interpreted in an idealized or overly formal sense.

Where a convention analogous to "A, B and at least one of C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B and C" would include, but not be limited to, systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).

It should be noted that the data flow loose coupling statistical method and apparatus disclosed in the present disclosure may be used for data statistics in the financial field, and may also be used for data statistics in any field other than the financial field.

The embodiment of the disclosure provides a data flow loose coupling statistical method, which is used for acquiring a service data flow, wherein the service data flow is a real-time data sum of various types of services; extracting basic data and real-time service data of various types of services from the service data stream, wherein the basic data comprises associated data among the services with dependency relationship; supplementing the basic data to the real-time service data of each type of service to obtain loosely coupled service data of each type of service; and respectively storing the loosely coupled service data of each type of service into a distributed deployed message queue to perform classified statistics.

Fig. 1 schematically illustrates an application scenario diagram of a data flow loose coupling statistical method, apparatus, device, medium, and program product according to embodiments of the present disclosure.

As shown in fig. 1, the application scenario 100 according to this embodiment may include data statistics of the financial field. The network 104 serves as a medium for providing communication links between the

terminal devices

101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.

The user may use the

terminal devices

101, 102, 103 to interact with the server 105 via the network 104 to receive or send messages or the like. The

terminal devices

101, 102, 103 may have installed thereon various communication client applications, such as shopping applications, web browser applications, search applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only).

The

terminal devices

101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.

The server 105 may be a server providing various services, such as a background management server (for example only) providing support for websites browsed by users using the

terminal devices

101, 102, 103. The background management server may analyze and perform other processing on the received data such as the user request, and feed back a processing result (e.g., a webpage, information, or data obtained or generated according to the user request) to the terminal device.

It should be noted that the data flow loose coupling statistical method provided by the embodiment of the present disclosure may be generally performed by the server 105. Accordingly, the data flow loose coupling statistical apparatus provided by the embodiment of the present disclosure may be generally disposed in the server 105. The statistical method for loose coupling of data streams provided by the embodiments of the present disclosure may also be performed by a server or a server cluster different from the server 105 and capable of communicating with the

terminal devices

101, 102, 103 and/or the server 105. Accordingly, the data flow loose coupling statistical device provided by the embodiment of the present disclosure may also be disposed in a server or a server cluster different from the server 105 and capable of communicating with the

terminal devices

101, 102, 103 and/or the server 105.

It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

The data flow loose coupling statistical method of the disclosed embodiment will be described in detail through fig. 2 to 3 based on the scenario described in fig. 1.

Fig. 2 schematically illustrates a flow chart of a data flow loose coupling statistical method according to an embodiment of the present disclosure.

As shown in fig. 2, the data flow loose coupling statistical method of this embodiment includes operations S210 to S240, and the transaction processing method may be performed sequentially.

In operation S210, a service data stream is obtained, where the service data stream is a real-time data sum of multiple types of services.

In this embodiment, the service data flow may be service ledger data of an enterprise, which includes data streams of various services inside the enterprise.

For example, for the loan transaction of a bank, whether the loan approval is passed or not needs to depend on credit data of a borrower to judge and examine, and when the approval is passed, the loan transaction can be smoothly executed. This results in that the processing of a single service may require orchestration between data across multiple services, resulting in increased computational complexity of the server. According to operations S220 to S240 of the embodiment of the present disclosure, loose coupling of data between different services may be achieved.

In operation S220, basic data and real-time service data of each type of service are extracted from the service data stream, where the basic data includes association data between services having a dependency relationship.

In the embodiment of the present disclosure, data that each service depends on is screened out as basic data from the service data stream, for example, in a financial service, a user needs to use account information of the user in consuming services such as car purchasing, decoration, installment payment, and the like, that is, a plurality of consuming services all depend on account opening service information of the user, and account information in the account opening service of the user is basic information that the several consuming services all need to depend on.

In the embodiment of the present disclosure, different service data are separated from a service data stream according to service types, so as to prepare for implementing data loose coupling of different types of services subsequently.

In operation S230, the basic data is supplemented to the real-time service data of each type of service, so as to obtain loosely-coupled service data of each type of service.

In the embodiment of the disclosure, the basic data that each type of service depends on is supplemented to each service data, thereby removing the dependency relationship between different services, no longer relying on other services to obtain the basic data, and realizing the data loose coupling of a single service.

In operation S240, the loosely coupled service data of each type of service is stored in the message queues of the distributed deployment to perform classification statistics.

According to the statistical method for loose coupling of data flows provided by the embodiment of the disclosure, service data of various types of services can be complemented, additional basic data on which service requirements depend can be obtained in real time, the data dependency relationship among the services can be removed, loose coupling of data among various services can be realized, independence and calculation isolation of subsequent service logics can be facilitated, rapid expansion of the services can be further realized, and service deployment can be facilitated.

The data loose coupling statistical method of the embodiments of the present disclosure will be described in detail below.

Fig. 3 schematically illustrates a detailed flow chart of a data flow loose coupling statistical method according to an embodiment of the present disclosure.

As shown in fig. 3, the statistical method for data stream loose coupling according to the embodiment of the present disclosure may specifically include operations S310 to S340.

In operation S310, a service data flow is obtained, where the service data flow is a real-time data sum of multiple types of services.

In operation S320, basic data and real-time service data of each type of service are extracted from the service data stream, where the basic data includes association data between services having a dependency relationship. The specific operation S320 includes S321 to S323.

In operation S321, the traffic data stream is copied into a first traffic data stream and a second traffic data stream based on a preset copy command.

In this embodiment, a copier program may be preset to execute a copy command, and copy data in a service data stream into two parts in real time, one part being used to extract basic keywords, and the other part being used to classify service data, so that the basic keywords and the classified service data may be obtained simultaneously without affecting data of an original service.

In operation S322, the first service data stream is filtered based on the preset basic keyword, and basic data corresponding to the basic keyword is screened from the service data stream.

In this embodiment, a basic data real-time filter is used to filter a first service data stream, and write basic data into a key-value real-time basic data search library in real time, at the first time, a basic data key value pair is constructed by using a basic keyword and a basic data value, and is stored into a basic data search library of a Redis data structure, so as to update the database in real time, thereby ensuring that data is up-to-date. The basic data are stored in a basic data retrieval base of a Redis data structure in a key-value form, the key-value is substantially stored in a memory, invalid data are periodically cleaned, and the core basic data can be rapidly called.

For example, when the current data of the first service data stream is card opening information, the information includes information such as a card number and a recommender, and since the information is used by a plurality of services such as payment and loan, the key-value mapping of the card number and the recommender is stored in the basic data search base in real time.

In operation S323, the second service data stream is classified according to the service type, so as to obtain real-time service data of each type of service.

In this embodiment, the second service data stream may be classified according to a preset service category label or a keyword, so as to obtain service data of different service types.

In operation S330, the basic data is supplemented to the real-time service data of each type of service, so as to obtain loosely-coupled service data of each type of service. Specifically, operation S330 includes operations S331 to S333.

In operation S331, a basic keyword of basic data on which each type of service depends is obtained based on a dependency relationship between the services.

In this embodiment, the dependency relationship between the services may be cleared up in the preparation of implementing the statistical method for loose coupling of data streams provided by the present disclosure, and the source and the basic keyword of the basic data depended on by each service are sorted out for direct use in the process of real-time completion of subsequent data.

In operation S332, the basic data search library is called to obtain basic data corresponding to the basic keyword. The key-value real-time basic data retrieval library can be used for efficiently accessing mass basic data, can meet the real-time calling of the data and completes the real-time completion of the service data.

In operation S333, the basic data is supplemented to the real-time service data of the service depending on the basic data, resulting in loosely-coupled service data.

According to the steps, the real-time completion of the service data can be completed through the efficient retrieval of the basic data retrieval database, so that the loose coupling of the data among the services is realized.

In operation S340, the loosely coupled service data of each type of service is stored in the message queue of the distributed deployment to perform classification statistics.

Operation S340 includes operations S341 to S342.

In operation S341, the distributed near real-time computing engine is used to read the loosely coupled service data of each type of service from the distributed deployed message queue, and count the loosely coupled service data of each type of service respectively.

In operation S342, the counted loosely coupled service data of each type of service is summarized to obtain loosely coupled statistical data of the service data stream.

According to the method, the basic data is updated in real time, the service data is completed in real time, and the classified distribution of the service data is carried out in real time, so that necessary conditions are provided for realizing the real-time processing of the end-to-end data. The loosely coupled service data of each type of service is respectively stored in different message queues, and in subsequent statistical calculation, a distributed quasi-real-time calculation engine can be conveniently adopted, or independent calculation engines are deployed on different servers, so that continuous and extensible deployment of the service is realized.

In the subsequent processing or data analysis links, each piece of data contains necessary elements for statistical analysis, so that real-time accumulation statistics can be realized.

Fig. 4 schematically shows an application diagram of a data flow loose coupling statistical method according to an embodiment of the present disclosure.

As shown in fig. 4, the real-time account of the bank card includes data streams of various services, such as data streams of services of purchasing vehicles, staging, using credit cards, and fitting-up staging using credit cards, and data streams of services of issuing credit cards, and there is a dependency relationship between the services, for example, basic data of issuing cards depending on using credit cards, and the like. With the dependency relationship, independent partitioning and loosely coupled calculation are required, so that services can be independently deployed, which is somewhat contradictory.

According to the statistical method for data flow loose coupling provided by the embodiment of the present disclosure, after the service data flow is obtained, the service data flow can be used as data to perform pre-application processing, and the statistical method for data loose coupling shown in fig. 2 to 3 is executed to implement loose coupling between services in advance, so as to obtain each classified service data. As shown in fig. 4, after the pre-application processing, data of different service types such as vehicle purchasing installment topic, decoration installment topic, installment general topic and the like can be extracted from the real-time standing book data. And then, data processing can be respectively carried out on different service types to complete service handling, and further statistical data of various types of data can be obtained.

In the data processing of the front-end application, when a new data message stream arrives at a service data complementing node, a complementing program acquires basic information in real time, for example, when a message of credit card consumption information arrives and contains short fields of card numbers, consumption amount, time and the like, the program of the complementing node acquires basic information (such as identification numbers and the like) of a card holder from a key-value real-time basic information base according to the card numbers in the message, and can acquire information of a marketing person in real time according to the card numbers. Therefore, when the short message is supplemented by necessary service data, a field set with more information is formed in real time, and the information is synchronously distributed to downstream service topic according to the type of the message (such as vehicle purchase stage topic, decoration stage topic, cash topic and the like), thereby realizing the real-time supplement and distribution of the message.

In the embodiment, the multidimensional key-value real-time basic data search library is designed to at least meet the following requirements: firstly, mass basic data can be efficiently accessed, for example, basic information such as credit card opening, authorization and the like is mainly stored in banking business, and the basic number of the basic information is relatively large due to numerous users of a bank system; secondly, real-time data analysis of various services such as consumption and the like is required to be carried out immediately after the card is opened, and the data analysis work is real-time, so that the multidimensional key-value real-time basic data retrieval library is required to be real-time due to prepositive dependence; and thirdly, the high availability requirement of the core service is met. Therefore, in this embodiment, a Redis server cluster is adopted to ensure stable implementation of the data stream loose coupling statistical method provided by the embodiment of the present disclosure.

In the technical scheme of the present disclosure, the processes of collecting, storing, using, processing, transmitting, providing, disclosing and applying the personal information of the related users all conform to the regulations of related laws and regulations, and necessary security measures are taken without violating the good customs of the public order.

Fig. 5 schematically illustrates an application diagram of a Redis server cluster according to an embodiment of the present disclosure.

As shown in fig. 5, the data stream loose coupling statistical method provided by the embodiment of the present disclosure may be implemented by a Redis server cluster, and the implementation manner includes: setting a Redis server cluster, wherein the Redis server cluster comprises 1 main Redis server group and a plurality of slave Redis server groups, the main Redis server group is used for executing a data stream loose coupling statistical method, the slave Redis server groups are used for backing up data generated on the main Redis server in real time, and the data comprises a service data stream, basic data and service data extracted from the service data stream and a message queue for storing slave coupling data of various services; setting a sentry program to monitor the running state of the Redis server cluster; when the main Redis server group runs in fault, the sentinel program upgrades one of the multiple slave Redis server groups to the Redis server, and then the data stream loose coupling statistical method provided in the embodiment of the disclosure is executed, so that real-time processing of data is not affected.

Based on the data flow loose coupling statistical device method, the disclosure also provides a data flow loose coupling statistical device. The apparatus will be described in detail below with reference to fig. 5.

Fig. 6 schematically shows a block diagram of a data flow loose coupling statistical apparatus according to an embodiment of the present disclosure.

As shown in fig. 6, the data flow loose coupling statistical apparatus 600 of this embodiment includes a data flow obtaining module 610, a data extracting module 620, a data completing module 630, and a distributed statistical module 640.

The data flow obtaining module 610 is configured to obtain a service data flow, where the service data flow is a real-time data sum of multiple types of services. In an embodiment, the data stream obtaining module 610 may be configured to perform the operation S210 described above, which is not described herein again.

The data extraction module 620 is configured to extract basic data and real-time service data of each type of service from the service data stream, where the basic data includes associated data between services having a dependency relationship. In an embodiment, the data extraction module 620 may be configured to perform the operation S220 described above, which is not described herein again.

The data complementing module 630 is configured to complement the basic data to the real-time service data of each type of service, so as to obtain loosely-coupled service data of each type of service. In an embodiment, the data completing module 630 may be configured to perform the operation S230 described above, which is not described herein again.

The distributed statistics module 640 is configured to store the loosely-coupled service data of each type of service into the message queues of the distributed deployment respectively to perform classification statistics. In an embodiment, the distributed statistics module 640 may be configured to perform the operation S240 described above, which is not described herein again.

According to the embodiment of the present disclosure, any plurality of the data stream obtaining module 610, the data extracting module 620, the data completing module 630, and the distributed statistics module 640 may be combined and implemented in one module, or any one of the modules may be split into a plurality of modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of the other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the data stream obtaining module 610, the data extracting module 620, the data completing module 630 and the distributed statistics module 640 may be implemented at least partially as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented by hardware or firmware in any other reasonable manner of integrating or packaging a circuit, or implemented by any one of three implementations of software, hardware and firmware, or implemented by a suitable combination of any of them. Alternatively, at least one of the data stream acquisition module 610, the data extraction module 620, the data completion module 630, and the distributed statistics module 640 may be implemented at least in part as a computer program module that, when executed, may perform a corresponding function.

Fig. 7 schematically illustrates a block diagram of an electronic device adapted to implement a data flow loose coupling statistical method according to an embodiment of the present disclosure.

As shown in fig. 7, an electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 702 or a program loaded from a storage section 708 into a Random Access Memory (RAM) 703. The processor 701 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or associated chipset, and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), among others. The processor 701 may also include on-board memory for caching purposes. The processor 701 may comprise a single processing unit or a plurality of processing units for performing the different actions of the method flows according to embodiments of the present disclosure.

In the RAM 703, various programs and data necessary for the operation of the electronic apparatus 700 are stored. The processor 701, the ROM 702, and the RAM 703 are connected to each other by a bus 704. The processor 701 performs various operations of the method flows according to the embodiments of the present disclosure by executing programs in the ROM 702 and/or the RAM 703. It is noted that the programs may also be stored in one or more memories other than the ROM 702 and RAM 703. The processor 701 may also perform various operations of method flows according to embodiments of the present disclosure by executing programs stored in the one or more memories.

Electronic device 700 may also include input/output (I/O) interface 705, which input/output (I/O) interface 705 is also connected to bus 704, according to an embodiment of the present disclosure. The electronic device 700 may also include one or more of the following components connected to the I/O interface 705: an input portion 706 including a keyboard, a mouse, and the like; an output section 707 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 708 including a hard disk and the like; and a communication section 709 including a network interface card such as a LAN card, a modem, or the like. The communication section 709 performs communication processing via a network such as the internet. A drive 710 is also connected to the I/O interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 710 as necessary, so that a computer program read out therefrom is mounted into the storage section 708 as necessary.

The present disclosure also provides a computer-readable storage medium, which may be contained in the apparatus/device/system described in the above embodiments; or may exist alone without being assembled into the device/apparatus/system. The computer-readable storage medium carries one or more programs which, when executed, implement the method according to an embodiment of the disclosure.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, a computer-readable storage medium may include the ROM 702 and/or the RAM 703 and/or one or more memories other than the ROM 702 and the RAM 703 described above.

Embodiments of the present disclosure also include a computer program product comprising a computer program containing program code for performing the method illustrated by the flow chart. When the computer program product runs in a computer system, the program code is used for causing the computer system to realize the data flow loose coupling statistical method provided by the embodiment of the disclosure.

The computer program performs the above-described functions defined in the system/apparatus of the embodiments of the present disclosure when executed by the processor 701. The systems, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.

In one embodiment, the computer program may be hosted on a tangible storage medium such as an optical storage device, a magnetic storage device, or the like. In another embodiment, the computer program may also be transmitted in the form of a signal over a network medium, distributed, and downloaded and installed via the communication section 709, and/or installed from the removable medium 711. The computer program containing program code may be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 709, and/or installed from the removable medium 711. The computer program, when executed by the processor 701, performs the above-described functions defined in the system of the embodiment of the present disclosure. The systems, devices, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.

In accordance with embodiments of the present disclosure, program code for executing computer programs provided by embodiments of the present disclosure may be written in any combination of one or more programming languages, and in particular, these computer programs may be implemented using high level procedural and/or object oriented programming languages, and/or assembly/machine languages. The programming language includes, but is not limited to, programming languages such as Java, C + +, python, the "C" language, or the like. The program code may execute entirely on the user computing device, partly on the user device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Those skilled in the art will appreciate that various combinations and/or combinations of features recited in the various embodiments and/or claims of the present disclosure can be made, even if such combinations or combinations are not expressly recited in the present disclosure. In particular, various combinations and/or combinations of the features recited in the various embodiments and/or claims of the present disclosure may be made without departing from the spirit or teaching of the present disclosure. All such combinations and/or associations are within the scope of the present disclosure.

The embodiments of the present disclosure have been described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described separately above, this does not mean that the measures in the embodiments cannot be used advantageously in combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be devised by those skilled in the art without departing from the scope of the present disclosure, and such alternatives and modifications are intended to be within the scope of the present disclosure.

Claims

1. A statistical method for loose coupling of data streams, the method comprising:

acquiring a service data stream, wherein the service data stream is a real-time data sum of various types of services;

extracting basic data and real-time service data of various types of services from the service data stream, wherein the basic data comprises associated data among the services with dependency relationship;

supplementing the basic data to the real-time service data of each type of service to obtain loosely coupled service data of each type of service;

and respectively storing the loosely coupled service data of each type of service into a distributed deployed message queue to perform classified statistics.

2. The method of claim 1, wherein the extracting the basic data and the real-time service data of each type of service from the service data stream comprises:

copying the service data stream into a first service data stream and a second service data stream based on a preset copying command;

filtering the first service data stream based on a preset basic keyword, and screening basic data corresponding to the basic keyword from the service data stream;

and classifying the second service data stream according to the service type to obtain the real-time service data of each type of service.

3. The method of claim 2, wherein the filtering the first service data stream based on a preset basic keyword to obtain the basic data further comprises:

and constructing a basic data key value pair by using the basic key words and the basic data values, and storing the basic data key value pair into a basic data retrieval base of a Redis data structure.

4. The method of claim 3, wherein the supplementing the basic data to the real-time service data of each type of service to obtain loosely-coupled service data of each type of service comprises:

acquiring basic keywords of basic data on which each type of service depends based on the dependency relationship among the services;

calling the basic data search library to acquire basic data corresponding to the basic keywords;

and supplementing the basic data into real-time service data of the service depending on the basic data to obtain the loosely-coupled service data.

5. The method of claim 1, wherein the step of separately storing the loosely-coupled service data of each type of service in a message queue of a distributed deployment to perform classification statistics comprises:

reading the loosely coupled service data of each type of service from the distributed deployed message queue by adopting a distributed quasi-real-time computing engine, and respectively counting the loosely coupled service data of each type of service;

and summarizing the counted loosely coupled service data of each type of service to obtain loosely coupled statistical data of the service data stream.

6. The method of claim 1, further comprising:

setting a Redis server cluster, wherein the Redis server cluster comprises 1 main Redis server group and a plurality of auxiliary Redis server groups, the main Redis server group is used for executing the data stream loose coupling statistical method, and the auxiliary Redis server groups are used for backing up data generated on a main Redis server in real time;

monitoring the running state of the Redis server cluster;

and when the main Redis server group fails to operate, upgrading one of the plurality of slave Redis server groups to a Redis server.

7. A data flow loose coupling statistics apparatus, comprising:

the data flow acquisition module is used for acquiring a service data flow, wherein the service data flow is the sum of real-time data of various types of services;

the data extraction module is used for extracting basic data and real-time service data of various types of services from the service data stream, wherein the basic data comprises associated data among the services with dependency relationship;

the data complementing module is used for complementing the basic data to the real-time service data of each type of service to obtain loosely coupled service data of each type of service;

and the distributed statistical module is used for respectively storing the loosely coupled service data of each type of service into the distributed deployed message queues to execute classified statistics.

8. An electronic device, comprising:

one or more processors;

a storage device for storing one or more programs,

wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of any of claims 1-6.

9. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 6.