CN111459986B - Data computing system and method - Google Patents
Data computing system and method Download PDFInfo
- Publication number
- CN111459986B CN111459986B CN202010266751.3A CN202010266751A CN111459986B CN 111459986 B CN111459986 B CN 111459986B CN 202010266751 A CN202010266751 A CN 202010266751A CN 111459986 B CN111459986 B CN 111459986B
- Authority
- CN
- China
- Prior art keywords
- calculation
- data
- computing
- synchronous
- engine
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24568—Data stream processing; Continuous queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Information Transfer Between Computers (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a data computing system and a method, wherein the system comprises the following steps: the system comprises a data acquisition module, a message queue, a calculation engine module and a calculation result storage module, wherein: the data acquisition module is used for acquiring data from a plurality of preset data sources in real time and sending the acquired data to the message queue; the calculation engine module is used for acquiring data from the message queue and performing data calculation; the calculation engine module includes: the system comprises a streaming computing engine and a synchronous computing engine, wherein the streaming computing engine is used for carrying out real-time streaming computing, and the synchronous computing engine is used for carrying out synchronous computing; the calculation result storage module is used for receiving and storing the calculation result sent by the calculation engine module. The invention provides a one-stop real-time computing system for streaming computing and synchronous computing, which meets the computing demands of the financial industry in multiple scenes such as real-time risk monitoring, real-time anti-fraud and the like.
Description
Technical Field
The invention relates to the field of big data, in particular to a data computing system and a data computing method.
Background
The data in the financial industry has the characteristics of wide sources, large throughput, complex calculation model and the like. The data can be applied in the context of mobile application advertising, fraud detection, credit card transactions, financial transactions, etc. by processing the data in real time in order to make a fast and viable decision. The data application scene types in the financial industry are more, the calculation to be performed on the data in different application scenes is different, for example, the real-time calculation is required to be performed on the data in fraud detection, and the calculation processing can be performed on the data in a batch calculation mode in scenes with low real-time requirements. The prior art lacks a data processing system for multiple application scenario types in the financial industry.
Disclosure of Invention
The invention provides a data computing system and a data computing method for solving at least one technical problem in the background art.
To achieve the above object, according to one aspect of the present invention, there is provided a data computing system including: the system comprises a data acquisition module, a message queue, a calculation engine module and a calculation result storage module, wherein:
the data acquisition module is used for acquiring data from a plurality of preset data sources in real time and sending the acquired data to the message queue;
the calculation engine module is used for acquiring data from the message queue and performing data calculation; the calculation engine module includes: the system comprises a streaming computing engine and a synchronous computing engine, wherein the streaming computing engine is used for carrying out real-time streaming computing, and the synchronous computing engine is used for carrying out synchronous computing;
the calculation result storage module is used for receiving and storing the calculation result sent by the calculation engine module.
Optionally, the data computing system further comprises: the computing rule configuration module is used for receiving real-time streaming computing tasks set by a user;
the streaming computing engine is specifically configured to obtain data from the message queue according to the real-time streaming computing task and perform real-time streaming computing, and send a computing result to a corresponding storage node in the computing result storage module according to the real-time streaming computing task.
Optionally, the data computing system further comprises: the synchronous computing service module is used for receiving a synchronous computing request set by a user;
the synchronous calculation engine is specifically configured to acquire data from the message queue according to the synchronous calculation request, perform synchronous calculation, and send a calculation result to a corresponding storage node in the calculation result storage module according to the synchronous calculation request.
Optionally, the data computing system further comprises: and the real-time analysis module is used for carrying out real-time anti-fraud analysis, real-time risk monitoring, real-time marketing and personalized recommendation according to the calculation result and a plurality of preset analysis tasks.
Optionally, the calculation result storage module includes: hbase database, relational database, HIVE, and Redis cache.
In order to achieve the above object, according to another aspect of the present invention, there is provided a data calculation method, which:
the data acquisition module acquires data from a plurality of preset data sources in real time and sends the acquired data to the message queue;
the calculation engine module obtains data from the message queue and performs data calculation, wherein the calculation engine module comprises: the system comprises a streaming computing engine and a synchronous computing engine, wherein the streaming computing engine is used for carrying out real-time streaming computing, and the synchronous computing engine is used for carrying out synchronous computing;
and the calculation result storage module receives and stores the calculation result sent by the calculation engine module.
Optionally, the calculation engine module obtains data from the message queue and performs data calculation, including:
the streaming computing engine acquires data from the message queue according to a preset real-time streaming computing task, performs real-time streaming computing, and sends a computing result to a corresponding storage node in the computing result storage module according to the real-time streaming computing task.
Optionally, the calculation engine module obtains data from the message queue and performs data calculation, including:
and the synchronous calculation engine acquires data from the message queue according to a preset synchronous calculation request, performs synchronous calculation, and sends a calculation result to a corresponding storage node in the calculation result storage module according to the synchronous calculation request.
To achieve the above object, according to another aspect of the present invention, there is also provided a computer apparatus including a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the steps in the above data calculation method when executing the computer program.
In order to achieve the above object, according to another aspect of the present invention, there is also provided a computer-readable storage medium storing a computer program which, when executed in a computer processor, implements the steps of the above-described data calculation method.
The beneficial effects of the invention are as follows: the invention provides a one-stop real-time computing system for streaming computing and synchronous computing, which meets the computing demands of the financial industry in multiple scenes such as real-time risk monitoring, real-time anti-fraud and the like.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art. In the drawings:
FIG. 1 is a block diagram of a data computing system in accordance with an embodiment of the present invention;
FIG. 2 is a flow chart of a data calculation method according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a computer device according to an embodiment of the invention.
Detailed Description
In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
It is noted that the terms "comprises" and "comprising," and any variations thereof, in the description and claims of the present invention and in the foregoing figures, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed or inherent to such process, method, article, or apparatus.
It should be noted that, without conflict, the embodiments of the present invention and features of the embodiments may be combined with each other. The invention will be described in detail below with reference to the drawings in connection with embodiments.
FIG. 1 is a block diagram of a data computing system according to an embodiment of the present invention, and as shown in FIG. 1, the data computing system according to the present embodiment includes: the system comprises a data acquisition module, a message queue, a calculation engine module and a calculation result storage module.
In the embodiment of the invention, the data acquisition module is used for acquiring data from a plurality of preset data sources in real time and sending the acquired data to the message queue. In an alternative embodiment of the present invention, the data source may include: APP, WEB, database, log, etc. The data acquisition module adopts different modes to acquire data in real time aiming at different data sources, the data acquisition module can extract data from a log in real time in a manner of flame, the data acquisition module can acquire data from a real-time database in a manner of data extraction, and the data acquisition module can also acquire data of APP and WEB in real time from an APP and WEB proxy server.
In an alternative embodiment of the invention, the data acquisition module places the acquired data into a message queue in a message manner. The message queues may employ various message queues of the prior art, such as kafka message queues, rocketMQ message queues, and the like.
In an embodiment of the present invention, a calculation engine module includes: the system comprises a streaming computing engine and a synchronous computing engine, wherein the streaming computing engine is used for carrying out real-time streaming computing, and the synchronous computing engine is used for carrying out synchronous computing. In an alternative embodiment of the present invention, both the streaming computing engine and the synchronous computing engine employ a Flink engine. The stream computing engine and the synchronous computing engine perform computation according to the computing tasks set by the user, and send the computing results to the computing result storage module for storage.
The computing result storage module comprises a plurality of storage nodes for storing data of different data types, and the streaming computing engine and the synchronous computing engine send computing results to the different storage nodes for storage according to computing tasks set by users. In an alternative embodiment of the present invention, the storage node may include: hbase database, relational database, HIVE, redis cache, etc.
As shown in fig. 1, the data computing system according to the embodiment of the present invention further includes a computation rule configuration module. The calculation rule configuration module is used for receiving real-time streaming calculation tasks set by a user. The real-time streaming computing task records information such as data required for computation, a computing model, a data storage node for storing a computing result, and the like. The streaming computing engine acquires corresponding data from the message queue according to the real-time streaming computing task set by the user, performs real-time streaming computing, and sends a computing result to a corresponding storage node in the computing result storage module according to the real-time streaming computing task.
As shown in fig. 1, the data computing system according to the embodiment of the present invention further includes a synchronous computing service module. The synchronous calculation service module receives a synchronous calculation request set by a user, wherein the synchronous calculation request comprises information such as data required by synchronous calculation, a calculation model, a data storage node for storing a calculation result and the like. The synchronous calculation engine acquires corresponding data from the message queue according to the synchronous calculation request of the user, performs synchronous calculation, and sends a calculation result to a corresponding storage node in the calculation result storage module according to the synchronous calculation request. In an alternative embodiment of the invention, the synchronous computing engine and the synchronous computing service module rely on each other for data synchronization.
As shown in fig. 1, the data computing system according to the embodiment of the present invention further includes a real-time analysis module. The real-time analysis module is used for performing real-time anti-fraud analysis, real-time risk monitoring, real-time marketing, personalized recommendation and other services according to the calculation result in the calculation result storage module and a plurality of preset analysis tasks. In an alternative embodiment of the present invention, the analysis task includes information such as calculation result data required for analysis and an analysis algorithm.
The data computing system of the embodiment of the invention supports multiple data sources, supports acquisition of message flows from various message middleware, extracts full/incremental data from databases such as a relational database and the like, and provides a service interface for other systems to push data in real time. The integrity and diversity of the calculation data are ensured.
The data computing system of the embodiment of the invention can provide unified asynchronous/synchronous computing service: the Flink-based streaming computing can only meet the real-time asynchronous computing requirements of clients, but cannot meet the computing scene requiring real-time return of computing results. The invention comprises a high-performance synchronous calculation engine, and can meet the application scene of synchronously returning calculation results. The invention combines with asynchronous flow type computing engine to provide unified computing service, and covers all real-time computing demands.
The data computing system of the embodiment of the invention can provide visual computing configuration and real-time analysis: in order to improve the usability of the system, business personnel without programming experience can also quickly realize business targets through stream computation, the platform provides a visual computation rule configuration engine, and corresponding computation rules can be defined for various application scenes. Index calculation rules for various types of wind controlled events may be defined, as for a wind controlled scenario. Meanwhile, the platform provides a real-time online analysis function, business personnel select data sources online, report statistics rules are set, charts with various dimensions are automatically generated and refreshed in real time, and real-time visual analysis of data is achieved.
As can be seen from the above description, the invention is based on the Flink engine, performs a great deal of function and availability design, designs a set of one-stop real-time computing system capable of supporting wind control, anti-fraud, stream computing and synchronous computing, and meets the computing requirements of the financial industry in a plurality of scenes such as real-time wind control, real-time anti-fraud and the like.
The term "module" as used above may be a combination of software and/or hardware that implements the intended function. While the system described in the above embodiments is preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.
Based on the same inventive concept, the embodiment of the invention also provides a data calculation method, as described in the following embodiment. Since the principle of solving the problem by the data computing method is similar to that of the data computing system, the embodiments of the data computing method can refer to the embodiments of the data computing system, and the repetition is omitted.
Fig. 2 is a flowchart of a data calculation method according to an embodiment of the present invention, and as shown in fig. 2, the data calculation method according to an embodiment of the present invention includes steps S101 to S103.
Step S101, a data acquisition module acquires data from a plurality of preset data sources in real time and sends the acquired data to a message queue.
Step S102, a calculation engine module obtains data from the message queue and performs data calculation, where the calculation engine module includes: the system comprises a streaming computing engine and a synchronous computing engine, wherein the streaming computing engine is used for carrying out real-time streaming computing, and the synchronous computing engine is used for carrying out synchronous computing.
Step S103, a calculation result storage module receives and stores the calculation result sent by the calculation engine module.
In an optional embodiment of the present invention, the performing data calculation in step S102 specifically includes:
the streaming computing engine acquires data from the message queue according to a preset real-time streaming computing task, performs real-time streaming computing, and sends a computing result to a corresponding storage node in the computing result storage module according to the real-time streaming computing task.
In an optional embodiment of the present invention, the performing data calculation in step S102 specifically further includes:
and the synchronous calculation engine acquires data from the message queue according to a preset synchronous calculation request, performs synchronous calculation, and sends a calculation result to a corresponding storage node in the calculation result storage module according to the synchronous calculation request.
It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer executable instructions, and that although a logical order is illustrated in the flowcharts, in some cases the steps illustrated or described may be performed in an order other than that illustrated herein.
To achieve the above object, according to another aspect of the present application, there is also provided a computer apparatus. As shown in fig. 3, the computer device comprises a memory, a processor, a communication interface and a communication bus, on which a computer program is stored which can be run on the processor, said processor implementing the steps in the method of the above embodiments when executing said computer program.
The processor may be a central processing unit (Central Processing Unit, CPU). The processor may also be any other general purpose processor, digital signal processor (Digital Signal Processor, DSP), application specific integrated circuit (Application Specific Integrated Circuit, ASIC), field programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof.
The memory is used as a non-transitory computer readable storage medium for storing non-transitory software programs, non-transitory computer executable programs, and units, such as corresponding program units in the above-described method embodiments of the invention. The processor executes the various functional applications of the processor and the processing of the composition data by running non-transitory software programs, instructions and modules stored in the memory, i.e., implementing the methods of the method embodiments described above.
The memory may include a memory program area and a memory data area, wherein the memory program area may store an operating system, at least one application program required for a function; the storage data area may store data created by the processor, etc. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory may optionally include memory located remotely from the processor, the remote memory being connectable to the processor through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The one or more units are stored in the memory, which when executed by the processor, performs the method in the above embodiments.
The details of the computer device may be correspondingly understood by referring to the corresponding relevant descriptions and effects in the above embodiments, and will not be repeated here.
To achieve the above object, according to another aspect of the present application, there is also provided a computer-readable storage medium storing a computer program which, when executed in a computer processor, implements the steps of the above-described data calculation method. It will be appreciated by those skilled in the art that implementing all or part of the above-described embodiment method may be implemented by a computer program to instruct related hardware, where the program may be stored in a computer readable storage medium, and the program may include the above-described embodiment method when executed. Wherein the storage medium may be a magnetic Disk, an optical Disk, a Read-Only Memory (ROM), a random access Memory (RandomAccessMemory, RAM), a Flash Memory (Flash Memory), a Hard Disk (HDD), a Solid State Drive (SSD), or the like; the storage medium may also comprise a combination of memories of the kind described above.
It will be apparent to those skilled in the art that the modules or steps of the invention described above may be implemented in a general purpose computing device, they may be concentrated on a single computing device, or distributed across a network of computing devices, or they may alternatively be implemented in program code executable by computing devices, such that they may be stored in a memory device for execution by the computing devices, or they may be separately fabricated into individual integrated circuit modules, or multiple modules or steps within them may be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.
The above description is only of the preferred embodiments of the present invention and is not intended to limit the present invention, but various modifications and variations can be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (8)
1. A data computing system, comprising: the system comprises a data acquisition module, a message queue, a calculation engine module, a synchronous calculation service module and a calculation result storage module, wherein:
the data acquisition module is used for acquiring data from a plurality of preset data sources in real time and sending the acquired data to the message queue;
the calculation engine module is used for acquiring data from the message queue and performing data calculation; the calculation engine module includes: the system comprises a streaming computing engine and a synchronous computing engine, wherein the streaming computing engine is used for carrying out real-time streaming computing, and the synchronous computing engine is used for carrying out synchronous computing;
the synchronous calculation service module is used for receiving a synchronous calculation request set by a user, wherein the synchronous calculation request comprises information of data required by synchronous calculation, calculation model information and data storage node information stored by a calculation result; the synchronous calculation engine is specifically configured to acquire data from the message queue according to the synchronous calculation request, perform synchronous calculation, and send a calculation result to a corresponding storage node in the calculation result storage module according to the synchronous calculation request; the synchronous calculation engine and the synchronous calculation service module mutually depend on each other to perform data synchronization;
the calculation result storage module is used for receiving and storing the calculation result sent by the calculation engine module.
2. The data computing system of claim 1, further comprising: the computing rule configuration module is used for receiving real-time streaming computing tasks set by a user;
the streaming computing engine is specifically configured to obtain data from the message queue according to the real-time streaming computing task and perform real-time streaming computing, and send a computing result to a corresponding storage node in the computing result storage module according to the real-time streaming computing task.
3. The data computing system of claim 1, further comprising: and the real-time analysis module is used for carrying out real-time anti-fraud analysis, real-time risk monitoring, real-time marketing and personalized recommendation according to the calculation result and a plurality of preset analysis tasks.
4. The data computing system of claim 1, wherein the computation result storage module comprises: hbase database, relational database, HIVE, and Redis cache.
5. A data computing method, comprising:
the data acquisition module acquires data from a plurality of preset data sources in real time and sends the acquired data to the message queue;
the calculation engine module obtains data from the message queue and performs data calculation, wherein the calculation engine module comprises: the system comprises a streaming computing engine and a synchronous computing engine, wherein the streaming computing engine is used for carrying out real-time streaming computing, and the synchronous computing engine is used for carrying out synchronous computing;
the calculation result storage module receives and stores the calculation result sent by the calculation engine module;
the calculation engine module obtains data from the message queue and performs data calculation, including:
the synchronous calculation engine acquires data from the message queue according to a synchronous calculation request acquired from a synchronous calculation service module, performs synchronous calculation, and transmits a calculation result to a corresponding storage node in a calculation result storage module according to the synchronous calculation request; the synchronous calculation service module is used for receiving a synchronous calculation request set by a user, wherein the synchronous calculation request comprises information of data required by synchronous calculation, calculation model information and data storage node information stored by a calculation result; the synchronous computing engine and the synchronous computing service module mutually depend on each other to perform data synchronization.
6. The data computing method of claim 5, wherein the computing engine module obtains data from the message queue and performs data computation, comprising:
the streaming computing engine acquires data from the message queue according to a preset real-time streaming computing task, performs real-time streaming computing, and sends a computing result to a corresponding storage node in the computing result storage module according to the real-time streaming computing task.
7. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method of any of claims 5 to 6 when executing the computer program.
8. A computer readable storage medium storing a computer program, characterized in that the computer program when executed in a computer processor implements the method of any one of claims 5 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010266751.3A CN111459986B (en) | 2020-04-07 | 2020-04-07 | Data computing system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010266751.3A CN111459986B (en) | 2020-04-07 | 2020-04-07 | Data computing system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111459986A CN111459986A (en) | 2020-07-28 |
| CN111459986B true CN111459986B (en) | 2023-07-21 |
Family
ID=71680504
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010266751.3A Active CN111459986B (en) | 2020-04-07 | 2020-04-07 | Data computing system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111459986B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112217893A (en) * | 2020-10-12 | 2021-01-12 | 广州欢网科技有限责任公司 | Frequency and quantity control method, device and equipment for advertisement delivery and advertisement delivery system |
| CN112365355B (en) * | 2020-12-10 | 2023-12-26 | 深圳迅策科技有限公司 | Method, device and readable medium for calculating foundation valuation and risk index in real time |
| CN112507029B (en) * | 2020-12-18 | 2022-11-04 | 上海哔哩哔哩科技有限公司 | Data processing system and data real-time processing method |
| CN112364063B (en) * | 2021-01-12 | 2021-06-04 | 北京智慧星光信息技术有限公司 | Stream computing system, data processing method thereof, electronic device, and medium |
| CN113220530B (en) * | 2021-05-14 | 2022-07-19 | 上海哔哩哔哩科技有限公司 | Data quality monitoring method and platform |
| CN113485694B (en) * | 2021-07-06 | 2023-04-28 | 算话信息科技(上海)有限公司 | Variable data intelligent middle platform system of algorithm |
| CN113553320B (en) * | 2021-07-29 | 2022-09-02 | 上海哔哩哔哩科技有限公司 | Data quality monitoring method and device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102253820A (en) * | 2011-06-16 | 2011-11-23 | 华中科技大学 | Stream type repetitive data detection method |
| CN102819599A (en) * | 2012-08-15 | 2012-12-12 | 华数传媒网络有限公司 | Method for constructing hierarchical catalogue based on consistent hashing data distribution |
| CN103905537A (en) * | 2014-03-20 | 2014-07-02 | 冶金自动化研究设计院 | System for managing industry real-time data storage in distributed environment |
| US10013158B1 (en) * | 2012-09-22 | 2018-07-03 | Sitting Man, Llc | Methods, systems, and computer program products for sharing a data object in a data store via a communication |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| DE102013226977B3 (en) * | 2013-12-20 | 2015-02-05 | Cetitec GmbH | Communication node for a packet-switched data network and method for its operation |
| CN105574205B (en) * | 2016-01-18 | 2019-03-19 | 国家电网公司 | The log dynamic analysis system of distributed computing environment |
| CN105959151B (en) * | 2016-06-22 | 2019-05-07 | 中国工商银行股份有限公司 | A kind of Stream Processing system and method for High Availabitity |
| CN107943840B (en) * | 2017-10-30 | 2022-01-11 | 深圳前海微众银行股份有限公司 | Data processing method, system and computer readable storage medium |
| CN108920948A (en) * | 2018-05-25 | 2018-11-30 | 众安信息技术服务有限公司 | A kind of anti-fraud streaming computing device and method |
| CN110334070A (en) * | 2019-05-21 | 2019-10-15 | 中国人民财产保险股份有限公司 | Data processing method, system, equipment and storage medium |
-
2020
- 2020-04-07 CN CN202010266751.3A patent/CN111459986B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102253820A (en) * | 2011-06-16 | 2011-11-23 | 华中科技大学 | Stream type repetitive data detection method |
| CN102819599A (en) * | 2012-08-15 | 2012-12-12 | 华数传媒网络有限公司 | Method for constructing hierarchical catalogue based on consistent hashing data distribution |
| US10013158B1 (en) * | 2012-09-22 | 2018-07-03 | Sitting Man, Llc | Methods, systems, and computer program products for sharing a data object in a data store via a communication |
| CN103905537A (en) * | 2014-03-20 | 2014-07-02 | 冶金自动化研究设计院 | System for managing industry real-time data storage in distributed environment |
Non-Patent Citations (2)
| Title |
|---|
| medical data visual synchronization and information interaction using internet-based graphics rendering and message-oriented streaming;Qi Zhang;informatics in medicine unlocked;第17卷;1-14 * |
| 面向海量小文件的分布式存储系统设计与实现;李洪奇;朱丽萍;孙国玉;王露;;计算机工程与设计;第37卷(第01期);86-92 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111459986A (en) | 2020-07-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111459986B (en) | Data computing system and method | |
| CN108776934B (en) | Distributed data calculation method and device, computer equipment and readable storage medium | |
| CN103024014B (en) | By the method and system of the mass data distribution processor of message queue | |
| CN111124819B (en) | Method and device for full link monitoring | |
| CN108600300B (en) | Log data processing method and device | |
| US11188443B2 (en) | Method, apparatus and system for processing log data | |
| CN111459689A (en) | Message processing system and method based on distributed queue | |
| US20200159764A1 (en) | Method for Processing and Displaying Real-Time Social Data on Map | |
| CN103297291A (en) | Method and system for monitoring website real-time statuses | |
| CN106815254A (en) | A kind of data processing method and device | |
| CN110334072A (en) | A kind of distributed file system, file updating method and device | |
| CN113360554A (en) | Method and equipment for extracting, converting and loading ETL (extract transform load) data | |
| CN111479095B (en) | Service processing control system, method and device | |
| CN111770022B (en) | Capacity expansion method, system, equipment and computer storage medium based on link monitoring | |
| CN112181678A (en) | Service data processing method, device and system, storage medium and electronic device | |
| Ravindra et al. | Latency aware elastic switching-based stream processing over compressed data streams | |
| CN113014608A (en) | Flow distribution control method and device, electronic equipment and storage medium | |
| US20210258349A1 (en) | System and method for data extraction, processing, and management across multiple communication platforms | |
| CN107480189A (en) | A kind of various dimensions real-time analyzer and method | |
| CN111049898A (en) | Method and system for realizing cross-domain architecture of computing cluster resources | |
| US9912545B2 (en) | High performance topology resolution for non-instrumented nodes | |
| US20210092199A1 (en) | Cachability of single page applications | |
| CN114048512A (en) | Method and device for processing sensitive data | |
| CN115695587A (en) | Service data processing system, method, device and storage medium | |
| CN111988368A (en) | Data interaction system and interaction method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220913 Address after: 25 Financial Street, Xicheng District, Beijing 100033 Applicant after: CHINA CONSTRUCTION BANK Corp. Address before: 25 Financial Street, Xicheng District, Beijing 100033 Applicant before: CHINA CONSTRUCTION BANK Corp. Applicant before: Jianxin Financial Science and Technology Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |