CN112995306A - Storm-based real-time accounting information processing method and system - Google Patents
Storm-based real-time accounting information processing method and system Download PDFInfo
- Publication number
- CN112995306A CN112995306A CN202110173593.1A CN202110173593A CN112995306A CN 112995306 A CN112995306 A CN 112995306A CN 202110173593 A CN202110173593 A CN 202110173593A CN 112995306 A CN112995306 A CN 112995306A
- Authority
- CN
- China
- Prior art keywords
- information
- data
- obtaining
- instruction
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/12—Accounting
- G06Q40/125—Finance or payroll
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/10—Network architectures or network communication protocols for network security for controlling access to devices or network resources
- H04L63/105—Multiple levels of security
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L2209/00—Additional information or applications relating to cryptographic mechanisms or cryptographic arrangements for secret or secure communication H04L9/00
- H04L2209/56—Financial cryptography, e.g. electronic payment or e-cash
Abstract
The invention discloses a storm-based real-time accounting information processing method and system, wherein the method is applied to an accounting information processing system, and the accounting information processing system comprises a data receiving layer, a data processing layer and a data storage layer, wherein the method comprises the following steps: acquiring first accounting information sent by a message center through a data receiving layer; according to the first sending instruction, the data receiving layer sends the first financial information to the data processing layer; obtaining first memory resource information of a current node, wherein the first memory resource information comprises a plurality of execution machines; judging whether the first memory resource information meets a first preset condition or not through a resource monitoring and scheduling module; and if so, distributing the first financial information to a plurality of execution machines based on a storm parallel mode according to the first distribution instruction, and then obtaining first processing data information and storing the first processing data information in a data storage layer. The technical problems that in the prior art, the financial processing result is not updated timely and the data processing efficiency is low are solved.
Description
Technical Field
The invention relates to the technical field of bank account data processing, in particular to a storm-based real-time account information processing method and system.
Background
The credit accounting calculation mode of a large commercial bank is complex, the data volume is large, the dimensionality is high, the processing flow relates to multiple centers and multiple assemblies, the traditional batch processing mode is adopted to process the accounting data at present, the data needs to be collected at regular time, and the accounting data processing is completed by scheduling tasks at regular time, so that the problem that the data updating is not timely is brought to a customer manager, and the latest accounting data and the accounting state cannot be obtained at the first time.
However, in the process of implementing the technical solution of the invention in the embodiments of the present application, the inventors of the present application find that the above-mentioned technology has at least the following technical problems:
in the prior art, when accounting data processing is performed in a batch processing mode, due to the fact that data needs to be collected regularly and tasks need to be scheduled regularly, the technical problems that accounting processing results are not updated timely and data processing efficiency is low exist.
Disclosure of Invention
The embodiment of the application provides a storm-based real-time accounting information processing method and system, and solves the technical problems that in the prior art, when accounting data are processed in a batch processing mode, due to the fact that data need to be collected regularly and tasks need to be scheduled regularly, the updating of accounting processing results is not timely and the data processing efficiency is low.
In view of the foregoing problems, embodiments of the present application provide a storm-based real-time accounting information processing method and system.
In a first aspect, an embodiment of the present application provides a storm-based real-time accounting information processing method, where the method is applied to an accounting information processing system, and the accounting information processing system includes a data receiving layer, a data processing layer, and a data storage layer, where the method includes: the method comprises the steps that first financial information sent by a message center is obtained through a data receiving layer, wherein the data receiving layer is in communication connection with the message center; obtaining a first sending instruction; sending, by the data receiving layer, the first accounting information to the data processing layer according to the first sending instruction; obtaining first memory resource information of a current node, wherein the first memory resource information comprises a plurality of execution machines; after receiving the first financial information, the data processing layer judges whether the first memory resource information meets a first preset condition through a resource monitoring and scheduling module; if the first preset condition is met, obtaining a first distribution instruction; according to the first distribution instruction, on the basis of a storm parallel mode, after the first financial information is distributed to the execution machines according to a preset strategy, first processing data information is obtained; obtaining a first storage instruction; and storing the first processing data information in the data storage layer according to the first storage instruction.
On the other hand, the application also provides a storm-based real-time accounting information processing system, which comprises: the first obtaining unit is used for obtaining first financial information sent by a message center through the data receiving layer, wherein the data receiving layer is in communication connection with the message center; a second obtaining unit configured to obtain a first sending instruction; a first sending unit, configured to send, by the data receiving layer, the first accounting information to the data processing layer according to the first sending instruction; a third obtaining unit, configured to obtain first memory resource information of a current node, where the first memory resource information includes multiple execution machines; the first judging unit is used for judging whether the first memory resource information meets a first preset condition through a resource monitoring and scheduling module after the data processing layer receives the first financial information; a fourth obtaining unit, configured to obtain a first allocation instruction if the first preset condition is met; a fifth obtaining unit, configured to obtain first processing data information after distributing the first accounting information to the multiple execution machines according to a storm parallel manner based on the first distribution instruction and according to a preset policy; a sixth obtaining unit to obtain a first store instruction; and the first storage unit is used for storing the first processing data information in the data storage layer according to the first storage instruction.
In a third aspect, the present invention provides a storm-based real-time accounting information processing system, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the method of the first aspect when executing the program.
One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages:
the embodiment of the application provides a storm-based real-time accounting information processing method and system, the method is applied to an accounting information processing system, the accounting information processing system comprises a data receiving layer, a data processing layer and a data storage layer, and the method comprises the following steps: the method comprises the steps that first financial information sent by a message center is obtained through a data receiving layer, wherein the data receiving layer is in communication connection with the message center; obtaining a first sending instruction; sending, by the data receiving layer, the first accounting information to the data processing layer according to the first sending instruction; obtaining first memory resource information of a current node, wherein the first memory resource information comprises a plurality of execution machines; after receiving the first financial information, the data processing layer judges whether the first memory resource information meets a first preset condition through a resource monitoring and scheduling module; if the first preset condition is met, obtaining a first distribution instruction; according to the first distribution instruction, on the basis of a storm parallel mode, after the first financial information is distributed to the execution machines according to a preset strategy, first processing data information is obtained; obtaining a first storage instruction; and storing the first processing data information in the data storage layer according to the first storage instruction. The technical problems that in the prior art, when accounting data processing is carried out in a batch processing mode, due to the fact that data needs to be collected regularly and tasks need to be scheduled regularly, the updating of accounting processing results is not timely and the data processing efficiency is low are solved.
The foregoing description is only an overview of the technical solutions of the present application, and the present application can be implemented according to the content of the description in order to make the technical means of the present application more clearly understood, and the following detailed description of the present application is given in order to make the above and other objects, features, and advantages of the present application more clearly understandable.
Drawings
Fig. 1 is a schematic flowchart of a storm-based real-time accounting information processing method according to an embodiment of the present application;
fig. 2 is a schematic structural diagram of a storm-based real-time accounting information processing system according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of an exemplary electronic device according to an embodiment of the present application.
Description of reference numerals: a first obtaining unit 11, a second obtaining unit 12, a first sending unit 13, a third obtaining unit 14, a first judging unit 15, a fourth obtaining unit 16, a fifth obtaining unit 17, a sixth obtaining unit 18, a first storage unit 19, a bus 300, a receiver 301, a processor 302, a sender 303, a memory 304, and a bus interface 306.
Detailed Description
The embodiment of the application provides a storm-based real-time accounting information processing method and system, and solves the technical problems that in the prior art, when accounting data are processed in a batch processing mode, due to the fact that data need to be collected regularly and tasks need to be scheduled regularly, the updating of accounting processing results is not timely and the data processing efficiency is low. Hereinafter, example embodiments according to the present application will be described in detail with reference to the accompanying drawings. It should be apparent that the described embodiments are merely some embodiments of the present application and not all embodiments of the present application, and it should be understood that the present application is not limited to the example embodiments described herein.
Summary of the application
The credit accounting calculation mode of a large commercial bank is complex, the data volume is large, the dimensionality is high, the processing flow relates to multiple centers and multiple assemblies, the traditional batch processing mode is usually adopted to process the accounting data, but in the prior art, due to the need of regularly collecting data and regularly scheduling tasks, the technical problems that the accounting processing result is not timely updated, the latest accounting data and the accounting state cannot be obtained at the first time and the data processing efficiency is low exist.
In view of the above technical problems, the technical solution provided by the present application has the following general idea:
the embodiment of the application provides a storm-based real-time accounting information processing method and system, the method is applied to an accounting information processing system, the accounting information processing system comprises a data receiving layer, a data processing layer and a data storage layer, and the method comprises the following steps: the method comprises the steps that first financial information sent by a message center is obtained through a data receiving layer, wherein the data receiving layer is in communication connection with the message center; obtaining a first sending instruction; sending, by the data receiving layer, the first accounting information to the data processing layer according to the first sending instruction; obtaining first memory resource information of a current node, wherein the first memory resource information comprises a plurality of execution machines; after receiving the first financial information, the data processing layer judges whether the first memory resource information meets a first preset condition through a resource monitoring and scheduling module; if the first preset condition is met, obtaining a first distribution instruction; according to the first distribution instruction, on the basis of a storm parallel mode, after the first financial information is distributed to the execution machines according to a preset strategy, first processing data information is obtained; obtaining a first storage instruction; and storing the first processing data information in the data storage layer according to the first storage instruction.
Having thus described the general principles of the present application, various non-limiting embodiments thereof will now be described in detail with reference to the accompanying drawings.
Example one
As shown in fig. 1, an embodiment of the present application provides a storm-based real-time accounting information processing method, where the method is applied to an accounting information processing system, and the accounting information processing system includes a data receiving layer, a data processing layer, and a data storage layer, where the method includes:
step S100: the method comprises the steps that first financial information sent by a message center is obtained through a data receiving layer, wherein the data receiving layer is in communication connection with the message center;
further, the data receiving layer is an EDA data receiving module.
Specifically, as the name implies, the data receiving layer is used for receiving financial data of a to-be-processed process, the data processing layer is used for processing the financial data, and the data storage layer is used for storing the processed financial data, wherein the data receiving layer is a data receiving module based on an Event-Driven Architecture (EDA), and specifically, the data receiving layer receives accounting information from the message center in a publish-subscribe mode.
Further, before the obtaining, by the data receiving layer, the first accounting information sent from the message center, step S100 in this embodiment of the present application further includes:
step S110: obtaining a first subscription theme of a first user;
step S120: obtaining a second sending instruction;
step S130: and sending the first subscription theme to the message center according to the second sending instruction.
Specifically, the user is used as a subscriber, determines a subscription theme according to service needs, sends the subscription theme to the data receiving layer, and the publisher pushes a theme message subscribed by the user to the information center according to a sending instruction of the data receiving layer, so that the data receiving layer obtains the theme data subscribed by the user from the information center.
Further, before obtaining the first subscription topic of the first user, step S110 in this embodiment of the present application further includes:
step S111: obtaining a first permission level of the first user;
step S112: judging whether the first permission level meets a preset permission level or not;
step S113: and if the preset authority level is met, obtaining the first subscription theme of the first user.
Specifically, in the accounting information processing system, in order to ensure the security and confidentiality management of financial data, different users have different permission levels, each user cannot access data resources irrelevant to the work of the user, and accordingly, the financial data has a preset permission level, and if the permission level of a certain user is consistent with the preset permission level of theme data to be subscribed, the data receiving layer receives the subscription theme of the user. The technical effect of obtaining the user subscription theme according to the permission level is achieved.
Further, after the sending the first subscription topic to the message center according to the second sending instruction, step S130 in this embodiment of the present application further includes:
step S131: after receiving the first subscription topic, the publisher acquires first financial information according to the first subscription topic;
step S132: obtaining a first push instruction;
step S133: and pushing the first accounting information to the message center according to the first pushing instruction.
Specifically, after receiving a user subscription topic sent by the data receiving layer, the publisher acquires the accounting information corresponding to the subscription topic, and then pushes the accounting information to the information center in time according to a push instruction.
Step S200: obtaining a first sending instruction;
step S300: sending, by the data receiving layer, the first accounting information to the data processing layer according to the first sending instruction;
step S400: obtaining first memory resource information of a current node, wherein the first memory resource information comprises a plurality of execution machines;
further, the number of the plurality of execution machines is at least two.
Specifically, the data receiving layer serves as a data source of the storm, sends the received account information to the downstream data processing layer, and the data processing layer processes the account information, wherein the data processing layer is provided with two or more actuators and is used for processing credit account information, states, account information states and the like.
Step S500: after receiving the first financial information, the data processing layer judges whether the first memory resource information meets a first preset condition through a resource monitoring and scheduling module;
step S600: if the first preset condition is met, obtaining a first distribution instruction;
further, after receiving the first accounting information, the data processing layer determines, by a resource monitoring and scheduling module, whether the first memory resource information satisfies a first preset condition, where step S500 in this embodiment of the present application specifically includes:
step S510: acquiring first preset memory requirement information of the first account information;
step S520: judging whether the first memory resource information meets the first preset memory requirement information or not;
step S530: and if the first preset memory requirement information is met, obtaining a first allocation instruction.
Specifically, in the embodiment of the present application, a resource monitoring and scheduling module is arranged on the data processing layer, where the resource monitoring and scheduling module is configured to monitor usage conditions of CPUs and memory resources of multiple actuators in real time, and after receiving the accounting information sent by the data receiving layer, the data processing layer first determines preset memory requirement information corresponding to the accounting information, in short, the size of a memory occupied by the accounting information is processed, then determines whether the memory resource information of a current node is sufficient, that is, whether the preset memory requirement information of the accounting information is satisfied, and if so, the system allocates a corresponding memory resource to the accounting information.
Further, the determining whether the first memory resource information meets the first preset memory requirement information, in step S520 of this embodiment of the present application, further includes:
step S521: if the first memory resource information does not meet the first preset memory requirement information, a first operation instruction is obtained;
step S522: according to the first operation instruction, after first resource use information is inquired, node information to be distributed is obtained;
step S523: obtaining a second allocation instruction;
step S524: and distributing the first accounting information to the nodes to be distributed according to the second distribution instruction.
Specifically, if the memory resource information of the current node is insufficient, that is, the preset memory requirement information of the accounting information is not satisfied, the system obtains the information of the node to be allocated by querying the memory resource usage information occupied by the current node, and finally allocates the task of processing the accounting information to the node to be allocated for execution.
Further, after querying the first resource usage information according to the first operation instruction, the obtaining information of the node to be allocated includes:
obtaining second memory resource information of other nodes, wherein the other nodes are nodes except the current node;
acquiring a set of all nodes meeting the first preset memory requirement information in the other nodes according to the first memory resource information;
and acquiring information of nodes to be distributed from the set of all nodes meeting the first preset memory requirement information.
Specifically, if the memory resource information of the current node does not satisfy the preset memory requirement information of the accounting information, the system obtains all node sets satisfying the preset memory requirement information of the accounting information by obtaining memory resource information of other nodes except the current node, and then can obtain the to-be-allocated node information for processing the accounting information from the set.
Step S700: according to the first distribution instruction, on the basis of a storm parallel mode, after the first financial information is distributed to the execution machines according to a preset strategy, first processing data information is obtained;
further, in step S700 of the embodiment of the present application, the allocating, according to the first allocation instruction and based on a storm parallel manner, the first accounting information to the multiple execution machines according to a preset policy includes:
step S710: acquiring a first task list according to the first account information;
step S720: acquiring task data to be distributed and quantity information of the tasks to be distributed according to the first task list;
step S730: obtaining first distribution task data corresponding to each execution machine according to the quantity information of the tasks to be distributed, the task data to be distributed and the execution machines;
step S740: sequentially distributing the first distributed task data to each of the plurality of execution machines.
In particular, Storm is a distributed real-time big data processing framework of Twitter open source, the earliest open source being github, from 0.9.1 version onwards, attributed to the Apache community, known in the industry as the real-time edition Hadoop. With more and more scenes, such as website statistics, recommendation systems, early warning systems, financial systems (high frequency trading, stock) and the like, which cannot tolerate the MapReduce high delay of Hadoop, the application of large data real-time processing solutions (flow calculation) is becoming more and more extensive, and the large data real-time processing solutions are the latest explosion point in the field of distributed technology at present, while the Storm is the outstanding and mainstream in the flow calculation technology. The embodiment of the application is based on a storm parallel mode, the processing flow of the first financial information is divided into a plurality of small tasks, a task list is formed, and therefore task data to be distributed and quantity information of the tasks to be distributed can be obtained, then the tasks to be distributed are reasonably distributed to corresponding execution machines, each task data to be distributed is sequentially distributed to the corresponding execution machines, and processing results of the first financial information, namely the first processing data information, are obtained after the execution tasks are finished.
Further, after the allocating the first allocation task data to each of the plurality of execution machines in sequence, step S740 in this embodiment of the present application further includes:
step S741: sequentially traversing each execution machine in the plurality of execution machines, and judging whether the first distribution task data distributed by each actuator is processed;
step S742: if the processing is finished, first processing data information is obtained.
Specifically, after the memory resources are allocated according to the preset policy, the processed data information of the accounting information can be obtained only after the task data of each actuator is processed.
Step S800: obtaining a first storage instruction;
step S900: and storing the first processing data information in the data storage layer according to the first storage instruction.
Further, after distributing the first accounting information to the plurality of execution machines according to a preset policy based on the storm parallel manner according to the first distribution instruction and before obtaining first processing data information, step S800 in this embodiment of the present application further includes:
step S810: obtaining a first database;
step S820: and connecting the first database with the data processing layer through JDBC.
Further, the first database is an oracle database, wherein the first database is used for storing the first processing data information.
Specifically, before the first processed data information is obtained, the data processing layer is connected with an oracle database through JDBC, so that the data processing layer can conveniently store the processed data into the data storage layer, JDBC is an abbreviation of Java Database Connectivity in english, translated into Java Database connection, is an application program interface in Java language for specifying how a client program accesses a Database, and provides methods such as querying and updating data in a Database, and is generally said to be a relational Database oriented JDBC, in this embodiment, an oracle Database, the oracle database is a relational database management system of the oracle corporation, is a popular relational database management system in the world at present, has good system portability, convenient use and strong function, and is suitable for various large, medium and small microcomputer environments. The method is a high-efficiency and high-reliability database scheme suitable for high throughput.
In summary, the method and the system for processing the real-time accounting information based on storm provided by the embodiment of the application have the following technical effects:
1. whether the first memory resource information of the current node is sufficient or not is judged through the resource monitoring and scheduling module, if the memory resources are insufficient, the node information to be distributed is obtained from a set of all nodes meeting the first preset memory requirement information, so that the technical effects of monitoring the resource use condition and dynamically distributing node tasks are achieved, the first financial information is distributed to the execution machines according to a preset strategy based on a storm parallel mode, the technical effects of processing financial data in real time in parallel, updating financial processing results in time and improving data processing efficiency are achieved.
Example two
Based on the same inventive concept as the storm-based real-time accounting information processing method in the foregoing embodiment, the present invention further provides a storm-based real-time accounting information processing system, as shown in fig. 2, the system includes:
a first obtaining unit 11, where the first obtaining unit 11 is configured to obtain, through the data receiving layer, first accounting information sent from a message center, where the data receiving layer is in communication connection with the message center;
a second obtaining unit 12, the second obtaining unit 12 being configured to obtain a first sending instruction;
a first sending unit 13, where the first sending unit 13 is configured to send, by the data receiving layer, the first accounting information to the data processing layer according to the first sending instruction;
a third obtaining unit 14, where the third obtaining unit 14 is configured to obtain first memory resource information of a current node, where the first memory resource information includes multiple execution machines;
a first determining unit 15, where the first determining unit 15 is configured to determine, by a resource monitoring and scheduling module, whether the first memory resource information satisfies a first preset condition after the data processing layer receives the first accounting information;
a fourth obtaining unit 16, wherein the fourth obtaining unit 16 is configured to obtain a first allocation instruction if the first preset condition is met;
a fifth obtaining unit 17, where the fifth obtaining unit 17 is configured to obtain first processing data information after distributing the first financial information to the multiple execution machines according to a storm parallel manner based on the first distribution instruction and according to a preset policy;
a sixth obtaining unit 18, the sixth obtaining unit 18 being configured to obtain the first store instruction;
a first storage unit 19, wherein the first storage unit 19 is configured to store the first processing data information in the data storage layer according to the first storage instruction.
Further, the system further comprises:
a seventh obtaining unit, configured to obtain a first subscription topic of a first user;
an eighth obtaining unit configured to obtain a second sending instruction;
and the second sending unit is used for sending the first subscription theme to the message center according to the second sending instruction.
Further, the system further comprises:
a ninth obtaining unit, configured to obtain a first permission level of the first user;
a second judging unit, configured to judge whether the first permission level meets a preset permission level;
a tenth obtaining unit, configured to obtain the first subscription theme of the first user if the preset permission level is met.
Further, the system further comprises:
an eleventh obtaining unit, configured to, after receiving the first subscription topic, a publisher obtain first accounting information according to the first subscription topic;
a twelfth obtaining unit, configured to obtain a first push instruction;
the first pushing unit is used for pushing the first financial information to the message center according to the first pushing instruction.
Further, the data receiving layer is an EDA data receiving module.
Further, the first determining unit 15 specifically includes:
a thirteenth obtaining unit, configured to obtain first preset memory requirement information of the first financial information;
a third determining unit, configured to determine whether the first memory resource information meets the first preset memory requirement information;
a fourteenth obtaining unit, configured to obtain a first allocation instruction if the first preset memory requirement information is met.
Further, the system further comprises:
a fifteenth obtaining unit, configured to obtain a first operation instruction if the first memory resource information does not satisfy the first preset memory requirement information;
a sixteenth obtaining unit, configured to obtain information of a node to be allocated after querying first resource usage information according to the first operation instruction;
a seventeenth obtaining unit to obtain a second allocation instruction;
a first allocation unit, configured to allocate the first accounting information to the node to be allocated according to the second allocation instruction.
Further, the sixteenth obtaining unit further includes:
an eighteenth obtaining unit, configured to obtain second memory resource information of other nodes, where the other nodes are nodes other than the current node;
a nineteenth obtaining unit, configured to obtain, according to the first memory resource information, a set of all nodes that meet the first preset memory requirement information in the other nodes;
a twentieth obtaining unit, configured to obtain information of a node to be allocated from the set of all nodes that satisfy the first preset memory requirement information.
Further, the system comprises:
a twenty-first obtaining unit, configured to obtain a first task list according to the first accounting information;
a twenty-second obtaining unit, configured to obtain task data to be allocated and quantity information of the tasks to be allocated according to the first task list;
a twenty-third obtaining unit, configured to obtain, according to the quantity information of the tasks to be allocated, the task data to be allocated, and the multiple execution machines, first allocation task data corresponding to each of the execution machines;
a second assigning unit to sequentially assign the first assigned task data to each of the plurality of execution machines.
Further, the system further comprises:
a fourth judging unit, configured to sequentially traverse each of the multiple execution machines, and judge whether the first allocation task data allocated by each of the actuators is processed;
a twenty-fourth obtaining unit configured to obtain the first machining data information if the completion is made.
Further, the number of the plurality of execution machines is at least two.
Further, the system further comprises:
a twenty-fifth obtaining unit, configured to obtain a first database;
a first connection unit configured to connect the first database with the data processing layer by JDBC.
Further, the first database is an oracle database, wherein the first database is used for storing the first processing data information.
Various changes and specific examples of the storm-based real-time accounting information processing method in the first embodiment of fig. 1 are also applicable to the storm-based real-time accounting information processing system in the present embodiment, and through the foregoing detailed description of the storm-based real-time accounting information processing method, those skilled in the art can clearly know the implementation method of the storm-based real-time accounting information processing system in the present embodiment, so for the brevity of the description, detailed descriptions are omitted here.
Exemplary electronic device
The electronic device of the embodiment of the present application is described below with reference to fig. 3.
Fig. 3 illustrates a schematic structural diagram of an electronic device according to an embodiment of the present application.
Based on the inventive concept of the storm-based real-time accounting information processing method in the foregoing embodiment, the invention further provides a storm-based real-time accounting information processing system, on which a computer program is stored, and the program, when executed by a processor, realizes the steps of any one of the methods of the storm-based real-time accounting information processing method described above.
Where in fig. 3 a bus architecture (represented by bus 300), bus 300 may include any number of interconnected buses and bridges, bus 300 linking together various circuits including one or more processors, represented by processor 302, and memory, represented by memory 304. The bus 300 may also link together various other circuits such as peripherals, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described any further herein. A bus interface 306 provides an interface between the bus 300 and the receiver 301 and transmitter 303. The receiver 301 and the transmitter 303 may be the same element, i.e., a transceiver, providing a means for communicating with various other systems over a transmission medium.
The processor 302 is responsible for managing the bus 300 and general processing, and the memory 304 may be used for storing data used by the processor 302 in performing operations.
The embodiment of the application provides a storm-based real-time accounting information processing method and system, the method is applied to an accounting information processing system, the accounting information processing system comprises a data receiving layer, a data processing layer and a data storage layer, and the method comprises the following steps: the method comprises the steps that first financial information sent by a message center is obtained through a data receiving layer, wherein the data receiving layer is in communication connection with the message center; obtaining a first sending instruction; sending, by the data receiving layer, the first accounting information to the data processing layer according to the first sending instruction; obtaining first memory resource information of a current node, wherein the first memory resource information comprises a plurality of execution machines; after receiving the first financial information, the data processing layer judges whether the first memory resource information meets a first preset condition through a resource monitoring and scheduling module; if the first preset condition is met, obtaining a first distribution instruction; according to the first distribution instruction, on the basis of a storm parallel mode, after the first financial information is distributed to the execution machines according to a preset strategy, first processing data information is obtained; obtaining a first storage instruction; and storing the first processing data information in the data storage layer according to the first storage instruction. The technical problems that in the prior art, when accounting data processing is carried out in a batch processing mode, due to the fact that data needs to be collected regularly and tasks need to be scheduled regularly, the updating of accounting processing results is not timely and the data processing efficiency is low are solved.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a system for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction system which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks. While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all such alterations and modifications as fall within the scope of the invention.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
Claims (15)
1. A storm-based real-time accounting information processing method is applied to an accounting information processing system, and the accounting information processing system comprises a data receiving layer, a data processing layer and a data storage layer, wherein the method comprises the following steps:
the method comprises the steps that first financial information sent by a message center is obtained through a data receiving layer, wherein the data receiving layer is in communication connection with the message center;
obtaining a first sending instruction;
sending, by the data receiving layer, the first accounting information to the data processing layer according to the first sending instruction;
obtaining first memory resource information of a current node, wherein the first memory resource information comprises a plurality of execution machines;
after receiving the first financial information, the data processing layer judges whether the first memory resource information meets a first preset condition through a resource monitoring and scheduling module;
if the first preset condition is met, obtaining a first distribution instruction;
according to the first distribution instruction, on the basis of a storm parallel mode, after the first financial information is distributed to the execution machines according to a preset strategy, first processing data information is obtained;
obtaining a first storage instruction;
and storing the first processing data information in the data storage layer according to the first storage instruction.
2. The method of claim 1, wherein prior to obtaining the sent first accounting information from the message center by the data receiving layer, the method further comprises:
obtaining a first subscription theme of a first user;
obtaining a second sending instruction;
and sending the first subscription theme to the message center according to the second sending instruction.
3. The method of claim 2, wherein prior to obtaining the first subscription topic for the first user, the method further comprises:
obtaining a first permission level of the first user;
judging whether the first permission level meets a preset permission level or not;
and if the preset authority level is met, obtaining the first subscription theme of the first user.
4. The method of claim 2, wherein after sending the first subscription topic to the message center according to the second sending instruction, the method further comprises:
after receiving the first subscription topic, the publisher acquires first financial information according to the first subscription topic;
obtaining a first push instruction;
and pushing the first accounting information to the message center according to the first pushing instruction.
5. The method of claim 1, wherein the data receiving layer is an EDA data receiving module.
6. The method according to claim 1, wherein after receiving the first accounting information, the data processing layer determines, by a resource monitoring and scheduling module, whether the first memory resource information satisfies a first preset condition, and specifically includes:
acquiring first preset memory requirement information of the first account information;
judging whether the first memory resource information meets the first preset memory requirement information or not;
and if the first preset memory requirement information is met, obtaining a first allocation instruction.
7. The method of claim 6, wherein the method further comprises:
if the first memory resource information does not meet the first preset memory requirement information, a first operation instruction is obtained;
according to the first operation instruction, after first resource use information is inquired, node information to be distributed is obtained;
obtaining a second allocation instruction;
and distributing the first accounting information to the nodes to be distributed according to the second distribution instruction.
8. The method of claim 7, wherein the obtaining of the information of the nodes to be allocated after querying the first resource usage information according to the first operation instruction further comprises:
obtaining second memory resource information of other nodes, wherein the other nodes are nodes except the current node;
acquiring a set of all nodes meeting the first preset memory requirement information in the other nodes according to the first memory resource information;
and acquiring information of nodes to be distributed from the set of all nodes meeting the first preset memory requirement information.
9. The method of claim 1, wherein the allocating the first accounting information to the plurality of execution machines according to a preset policy based on a storm parallel manner according to the first allocation instruction comprises:
acquiring a first task list according to the first account information;
acquiring task data to be distributed and quantity information of the tasks to be distributed according to the first task list;
obtaining first distribution task data corresponding to each execution machine according to the quantity information of the tasks to be distributed, the task data to be distributed and the execution machines;
sequentially distributing the first distributed task data to each of the plurality of execution machines.
10. The method of claim 9, wherein, subsequent to the assigning the first assigned task data to each of the plurality of execution machines in turn, the method further comprises:
sequentially traversing each execution machine in the plurality of execution machines, and judging whether the first distribution task data distributed by each actuator is processed;
if the processing is finished, first processing data information is obtained.
11. The method of claim 1, wherein the plurality of execution machines is at least two in number.
12. The method as claimed in claim 1, wherein, after the distributing the first accounting information to the plurality of execution machines according to the first distributing instruction based on storm parallel mode according to the preset strategy, before obtaining the first processing data information, the method further comprises:
obtaining a first database;
and connecting the first database with the data processing layer through JDBC.
13. The method of claim 12, wherein the first database is an oracle database, wherein the first database is used to store the first process data information.
14. A storm-based real-time accounting information processing system, wherein the system comprises:
the first obtaining unit is used for obtaining first financial information sent by a message center through the data receiving layer, wherein the data receiving layer is in communication connection with the message center;
a second obtaining unit configured to obtain a first sending instruction;
a first sending unit, configured to send, by the data receiving layer, the first accounting information to the data processing layer according to the first sending instruction;
a third obtaining unit, configured to obtain first memory resource information of a current node, where the first memory resource information includes multiple execution machines;
the first judging unit is used for judging whether the first memory resource information meets a first preset condition through a resource monitoring and scheduling module after the data processing layer receives the first financial information;
a fourth obtaining unit, configured to obtain a first allocation instruction if the first preset condition is met;
a fifth obtaining unit, configured to obtain first processing data information after distributing the first accounting information to the multiple execution machines according to a storm parallel manner based on the first distribution instruction and according to a preset policy;
a sixth obtaining unit to obtain a first store instruction;
and the first storage unit is used for storing the first processing data information in the data storage layer according to the first storage instruction.
15. A storm-based real-time accounting information processing system comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the system of any one of claims 1-13 when executing the program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110173593.1A CN112995306B (en) | 2021-02-05 | 2021-02-05 | Real-time accounting information processing method and system based on storm |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110173593.1A CN112995306B (en) | 2021-02-05 | 2021-02-05 | Real-time accounting information processing method and system based on storm |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112995306A true CN112995306A (en) | 2021-06-18 |
| CN112995306B CN112995306B (en) | 2023-10-20 |
Family
ID=76347752
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110173593.1A Active CN112995306B (en) | 2021-02-05 | 2021-02-05 | Real-time accounting information processing method and system based on storm |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112995306B (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107748711A (en) * | 2017-10-17 | 2018-03-02 | 厦门市美亚柏科信息股份有限公司 | Method, terminal device and the storage medium of Automatic Optimal Storm degree of parallelisms |
| CN108459919A (en) * | 2018-03-29 | 2018-08-28 | 中信百信银行股份有限公司 | A kind of distributed transaction processing method and device |
| CN109254842A (en) * | 2017-07-12 | 2019-01-22 | 腾讯科技（深圳）有限公司 | Method for managing resource, device and the readable storage medium storing program for executing of distributive type system |
| CN110109986A (en) * | 2018-01-16 | 2019-08-09 | 阿里巴巴集团控股有限公司 | Task processing method, system, server and task scheduling system |
| CN110688399A (en) * | 2019-08-26 | 2020-01-14 | 远光软件股份有限公司 | Stream type calculation real-time report system and method |
| CN110807595A (en) * | 2019-11-05 | 2020-02-18 | 杭州安恒信息技术股份有限公司 | Task distribution method and system |
| CN111274052A (en) * | 2020-01-19 | 2020-06-12 | 中国平安人寿保险股份有限公司 | Data distribution method, server, and computer-readable storage medium |
| CN112286664A (en) * | 2020-10-30 | 2021-01-29 | 中国平安财产保险股份有限公司 | Task scheduling method and device, computer equipment and readable storage medium |
-
2021
- 2021-02-05 CN CN202110173593.1A patent/CN112995306B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109254842A (en) * | 2017-07-12 | 2019-01-22 | 腾讯科技（深圳）有限公司 | Method for managing resource, device and the readable storage medium storing program for executing of distributive type system |
| CN107748711A (en) * | 2017-10-17 | 2018-03-02 | 厦门市美亚柏科信息股份有限公司 | Method, terminal device and the storage medium of Automatic Optimal Storm degree of parallelisms |
| CN110109986A (en) * | 2018-01-16 | 2019-08-09 | 阿里巴巴集团控股有限公司 | Task processing method, system, server and task scheduling system |
| CN108459919A (en) * | 2018-03-29 | 2018-08-28 | 中信百信银行股份有限公司 | A kind of distributed transaction processing method and device |
| CN110688399A (en) * | 2019-08-26 | 2020-01-14 | 远光软件股份有限公司 | Stream type calculation real-time report system and method |
| CN110807595A (en) * | 2019-11-05 | 2020-02-18 | 杭州安恒信息技术股份有限公司 | Task distribution method and system |
| CN111274052A (en) * | 2020-01-19 | 2020-06-12 | 中国平安人寿保险股份有限公司 | Data distribution method, server, and computer-readable storage medium |
| CN112286664A (en) * | 2020-10-30 | 2021-01-29 | 中国平安财产保险股份有限公司 | Task scheduling method and device, computer equipment and readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112995306B (en) | 2023-10-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110290189B (en) | Container cluster management method, device and system | |
| US11704144B2 (en) | Creating virtual machine groups based on request | |
| WO2018149221A1 (en) | Device management method and network management system | |
| Tao et al. | Study on manufacturing grid resource service QoS modeling and evaluation | |
| CN109117252B (en) | Method and system for task processing based on container and container cluster management system | |
| CN103810045A (en) | Resource allocation method, resource manager, resource server and system | |
| CN112698952A (en) | Unified management method and device for computing resources, computer equipment and storage medium | |
| CN115567251A (en) | Multi-service isolation method and system for micro-service cluster | |
| US8819239B2 (en) | Distributed resource management systems and methods for resource management thereof | |
| US20050089063A1 (en) | Computer system and control method thereof | |
| CN112860387A (en) | Distributed task scheduling method and device, computer equipment and storage medium | |
| CN105338037A (en) | Dynamic scheduling method and system | |
| CN108696559B (en) | Stream processing method and device | |
| CN113034178A (en) | Multi-system integral calculation method and device, terminal equipment and storage medium | |
| CN112995306B (en) | Real-time accounting information processing method and system based on storm | |
| US20230336557A1 (en) | Resource allocating and management in a computer network | |
| CN109308219B (en) | Task processing method and device and distributed computer system | |
| CN116719623A (en) | Job scheduling method, job result processing method and device | |
| CN114675954A (en) | Task scheduling method and device | |
| CN110247802B (en) | Resource configuration method and device for cloud service single-machine environment | |
| CN114443293A (en) | Deployment system and method for big data platform | |
| CN117193956A (en) | Distributed data processing system, method, server, and readable storage medium | |
| CN112860292A (en) | Configuration management method and device based on application | |
| CN111367680A (en) | Job task allocation method, device, server, system and storage medium | |
| US20060015593A1 (en) | Three dimensional surface indicating probability of breach of service level |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |