CN115952239A - Distributed hierarchical computing system based on expression, electronic device and storage medium - Google Patents
Distributed hierarchical computing system based on expression, electronic device and storage medium Download PDFInfo
- Publication number
- CN115952239A CN115952239A CN202310215546.8A CN202310215546A CN115952239A CN 115952239 A CN115952239 A CN 115952239A CN 202310215546 A CN202310215546 A CN 202310215546A CN 115952239 A CN115952239 A CN 115952239A
- Authority
- CN
- China
- Prior art keywords
- message
- data
- calculation
- expression
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a distributed hierarchical computing system, electronic equipment and a storage medium based on an expression, belonging to the field of distributed hierarchical computing, wherein the system comprises: and aiming at an expression calculation scene based on the service data, realizing distributed asynchronous calculation of the expression by using a message mechanism. The newly added data is preprocessed through the calculation preprocessing module to confirm the calculation task, the stock data is recalculated through the task scheduling module to process mass data, the calculation message is aggregated and redistributed through the message aggregation and distribution module, the overall calculation efficiency is improved by aggregating the repeated calculation tasks, meanwhile, the appointed calculation task is given higher priority to be executed through the priority distribution method, and the calculation efficiency of the key service is further improved.
Description
Technical Field
The invention belongs to the field of distributed hierarchical computation, and particularly relates to a distributed hierarchical computation system based on an expression, electronic equipment and a storage medium.
Background
The calculation of the expression generally includes two schemes of landing and non-landing, the scheme of non-landing has obvious short boards, the expression needs to be calculated in real time when data is inquired each time, the scheme has a large influence on the performance of the interface, and the value of the expression cannot be screened and counted.
Landing schemes traditional schemes are implemented by updating the values of expressions through synchronous or asynchronous interfaces when data is created or changed, and the pain points of the schemes are as follows:
firstly, the low efficiency of the synchronous interface can affect the operation experience of the user, and the problem of concurrent updating of the asynchronous interface can cause data inconsistency;
secondly, the operation of stock data can not be supported generally, so that only new data can be influenced after a calculation formula is changed; even if the operation of the stock data is supported, the operation efficiency and the final data consistency are still very large challenges under the condition of mass data; moreover, a large amount of data may cause calculation congestion, so that data needing real-time calculation cannot be calculated in time, and user services are affected;
thirdly, the sources of the data change are various, including but not limited to direct submission from an interface, writing through an API, and direct warehousing through an import tool, and the difficulty of computing coverage is greatly improved due to the variety of the sources of the data change.
Disclosure of Invention
In order to solve the above technical problems, the present invention provides a technical solution of a distributed hierarchical computing system, an electronic device, and a storage medium based on an expression, so as to solve the above technical problems.
The invention discloses a distributed hierarchical computing system based on expression in the first aspect, the system includes:
the system comprises: the system comprises an expression management module, a service data module, a task scheduling module, a calculation preprocessing module, a message aggregation distribution module, a calculation module, a service library, a task library and a message library;
the service library: a database storage, the business data information of structured storage;
the task library: the database storage is used for structurally storing the information of the calculation tasks of the task scheduling module;
the message library: the database storage is used for structurally storing the calculation aggregation messages of the message aggregation distribution module;
the expression management module: configuring an expression rule, submitting a request of a full amount of calculation tasks of stock data to the task scheduling module after the expression configuration is completed, and sending the configured expression rule to the service library;
the service data module: maintaining service data, sending a data change message to the calculation preprocessing module after the data change is successful, and storing the changed data into the service library;
the task scheduling module: according to the request of the computing task, sending a second computing message CLMsg2 for submitting the task and a task canceling to the message aggregation distribution module, and making a task call-back to the computing module; sending information of the calculation task to the task library;
the calculation preprocessing module: receiving a data change message sent by a business data module, generating a first calculation message CLmsg1 according to the dependency relationship of data analysis expressions before and after data change, and sending the first calculation message CLmsg1 to a message aggregation distribution module;
the message aggregation and distribution module: receiving the first calculation message CLMsg1 sent by the calculation preprocessing module and a second calculation message CLMsg2 sent by the task scheduling module, aggregating the first calculation message CLMsg1 or the second calculation message CLMsg2 to obtain a calculation aggregation message, storing the calculation aggregation message in a message library, regularly inquiring the calculation aggregation message in the message library and distributing the calculation aggregation message to the calculation module;
the calculation module: and receiving the calculation aggregation message sent by the message aggregation distribution module, calculating the value of the expression, storing the value of the expression in a service library, and returning a processing result.
According to the system of the first aspect of the present invention, the service data information includes: object description information, field description information, and object instance information;
the specific content of the object description information includes: object ID, object name, and field list;
the expression rule belongs to field description information, and the specific content of the expression rule comprises the following contents: the method comprises the following steps of (1) field ID, field name, belonged object ID, field type, return value type of expression and calculation formula of expression;
the specific content of the object instance information includes: data ID, belonging object ID, version number, and value of each field.
According to the system of the first aspect of the present invention, the data change message includes: data operation type, data ID, data before change, and data after change;
the method for maintaining the service data, sending a data change message to the calculation preprocessing module after the data change is successful, and storing the changed data in the service library comprises the following steps:
a user submits a request for operating service data through a webpage or a terminal, wherein parameters of the request comprise a data operation type, a data ID, an object ID and data content;
inquiring service data from a service library according to the data ID and the object ID, comparing the inquired data content with the data content in the requested parameter, and generating data before change and data after change;
according to the requested parameters, storing the changed data in the requested parameters to a service library according to the data operation type and the data ID, and adding one to the version number of the data;
and generating a data change message according to the data before and after the change, and sending the data change message to the calculation preprocessing module.
According to the system of the first aspect of the present invention, the method for sending the second computation message CLMsg2 for submitting the task to the message aggregation distribution module according to the request of the computation task includes:
receiving a request of a calculation task of an expression management module, generating a calculation task Job1 and storing the calculation task Job1 in a task library;
the method comprises the steps that one or a group of threads exist, a task state is found out by means of timing training and is a to-be-processed calculation task Job1, and meanwhile the task state is set to be processed;
inquiring data ID in batches from a business library according to the object ID in the trained calculation task Job1, generating a plurality of second calculation messages CLmsg2 and sending the second calculation messages CLmsg2 to a message aggregation distribution module; the message body of the second calculation message CLMsg2 comprises: task ID, field ID list, belonged object ID and data ID;
the method for making task call-back to the computing module comprises the following steps:
and receiving a request of task callback from the computing module, and updating the computing state of the computing task in the task library to be completed according to the task ID.
According to the system of the first aspect of the present invention, the method for generating the first calculation message CLMsg1 by analyzing the expression dependency relationship according to the data before and after the data change, and sending the first calculation message CLMsg1 to the message aggregation distribution module includes:
a plurality of processes of a plurality of computer devices receive data change messages simultaneously, and a changed field list FL1 is analyzed from data before and after the change of a message body of the data change messages;
inquiring the object description information and the associated object description information of the data according to the ID of the object to which the data change message belongs, searching all expression rule fields in the object description information, analyzing variables in a calculation formula of the expression rule fields, and generating a directed acyclic graph GP1 according to the dependency relationship among the variables; the nodes of the directed acyclic graph GP1 are fields with incidence relations in objects, edges are dependency relations, and the directions of the edges point to depended parties from the depended parties;
according to the generated directed acyclic graph GP1 and the analyzed field list FL1, traversing all nodes in the directed acyclic graph GP1 to search nodes belonging to FL1, and obtaining a front node field list FL2 of the nodes of FL1;
inquiring data ID required to be calculated according to a preposed node field list FL2 and a data change message, generating a first calculation message CLMsg1, and sending the first calculation message CLMsg1 to a message aggregation distribution module;
the message body of the first computation message CLMsg1 comprises: a list of field IDs, an object ID to which it belongs, and a data ID.
According to the system of the first aspect of the invention, the method of aggregating the first computation message CLMsg1 or the second computation message CLMsg2 comprises:
parsing a data ID, a belonging object ID and a field ID list from a message body of the first computation message CLMsg1 or the second computation message CLMsg 2;
calculating the value of the message priority based on the size of the data ID, the object ID or the field type in the field ID list;
inquiring the calculation aggregation message to be processed from the message library according to the analyzed data ID, if the inquiry result is empty, setting the message priority as P, and storing the calculation aggregation message to be processed into the message library; if the query result is not null, comparing the query result with the data of the query result to obtain the highest priority value P1, calculating a union set with the field ID list of the data of the query result, and storing the new priority P1 and the field ID list after the union set calculation into a message library.
According to the system of the first aspect of the present invention, the method for regularly querying and distributing the calculation aggregation messages in the message library to the calculation module comprises:
dividing the message priority into several zones: r1, R2.. Rn, each interval having a start priority SP and an end priority EP, the intervals being non-overlapping, the intervals adding together all priorities, each interval performing the following process;
allocating thread resources according to the interval, wherein each thread executes the following processes;
the thread periodically inquires the calculation aggregation messages with priorities belonging to the interval and the message states to be processed, the inquired calculation aggregation messages are sorted from high to low according to the priorities, and the calculation module can calculate from high to low according to the priorities;
updating the message state of the inquired calculation aggregation message into processing, sending the inquired calculation aggregation message to a calculation module for processing, and returning a processing result after calculation by the calculation module;
and according to the processing result, if the processing is successful, deleting the processed calculation aggregation message from the message library, otherwise, updating the state of the calculation aggregation message to be pending, and waiting for retrying again.
According to the system of the first aspect of the present invention, the method for receiving the calculation aggregation message sent by the message aggregation distribution module, calculating the value of the expression, storing the value in the service library, and returning the processing result includes:
a plurality of processes of a plurality of computer devices simultaneously receive calculation aggregation messages sent by a message aggregation distribution module, and analyze data, data IDs, belonged object IDs and field ID lists from message bodies of the calculation aggregation messages;
inquiring the service data information, the expression information and the information of the data of the incidence relation from the service library according to the separated data;
analyzing variables in a calculation formula of an expression according to the inquired result, searching the values of the variables from the service data information of the data, substituting the values into the calculation formula, and calculating the values of the expression;
generating a new version number V2 based on the version number V1 of the data of the service library according to the inquired result, updating the value of the expression and the new version number V2 to the service library by using the V1 as a condition, returning to the service library for inquiry if the updating fails to indicate that the data is modified by other threads in the service library, and recalculating the current data until the data is successfully updated to the service library;
and after the value of the expression and the new version number V2 are successfully updated to the service library, the processing result is successfully returned.
A second aspect of the invention provides an electronic device comprising a memory and a processor, the memory having stored thereon a computer program which, when executed by the processor, performs a method in an expression-based distributed hierarchical computing system according to the first aspect of the invention.
A third aspect of the present invention provides a storage medium storing a computer program executable by one or more processors and operable to implement a method in an expression-based distributed hierarchical computing system according to the first aspect of the present invention.
The scheme provided by the invention aims at an expression calculation scene based on the service data, and realizes the distributed asynchronous calculation of the expression by using a message mechanism. The newly added data is preprocessed through the calculation preprocessing module to confirm the calculation task, the stock data is recalculated through the task scheduling module to process mass data, the calculation message is aggregated and redistributed through the message aggregation and distribution module, the designated calculation task is given higher priority to be executed through the priority distribution method while the overall calculation efficiency is improved through aggregating the repeated calculation task, and the calculation efficiency of key services is further improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a block diagram of an expression-based distributed hierarchical computing system according to an embodiment of the present invention;
FIG. 2 is a directed acyclic graph generated from field dependencies according to an embodiment of the present invention;
fig. 3 is a block diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
A first aspect of the present invention discloses a distributed hierarchical computing system based on expressions, and fig. 1 is a structural diagram of a distributed hierarchical computing system based on expressions according to an embodiment of the present invention, specifically as shown in fig. 1, the system includes: the system comprises an expression management module, a service data module, a task scheduling module, a calculation preprocessing module, a message aggregation distribution module, a calculation module, a service library, a task library and a message library;
the service library: the database storage is used for storing business data information in a structured mode;
the task library: the database storage is used for structurally storing the information of the calculation tasks of the task scheduling module;
the message library: a database storage for storing the calculation aggregate message of the message aggregation distribution module in a structured manner;
the expression management module: configuring an expression rule, submitting a request of a full amount of calculation tasks of stock data to the task scheduling module after the expression configuration is completed, and sending the configured expression rule to the service library;
the service data module: maintaining service data, sending a data change message to the calculation preprocessing module after the data change is successful, and storing the changed data in the service library;
the task scheduling module: according to the request of the computing task, sending a second computing message CLmsg2 for submitting the task and a task canceling to the message aggregation distribution module, and making a task call-back to the computing module; sending information of the calculation task to the task library;
the calculation preprocessing module is used for: receiving a data change message DCMsg sent by a service data module, generating a first calculation message CLMsg1 according to the data analysis expression dependency relationship before and after data change, and sending the first calculation message CLMsg1 to a message aggregation distribution module;
the message aggregation and distribution module: receiving the first calculation message CLmsg1 sent by the calculation preprocessing module and a second calculation message CLmsg2 sent by the task scheduling module, aggregating the first calculation message CLmsg1 or the second calculation message CLmsg2 to obtain a calculation aggregation message, storing the calculation aggregation message in a message library, regularly inquiring the calculation aggregation message in the message library, and distributing the calculation aggregation message to the calculation module;
the calculation module: and receiving the calculation aggregation message sent by the message aggregation distribution module, calculating the value of the expression, storing the value of the expression in a service library, and returning a processing result.
A service library: a database stores business data information in a structured manner.
In some embodiments, the service data information comprises: object description information, field description information, and object instance information;
the specific content of the object description information includes: object ID, object name, and field list;
the expression rule belongs to field description information, and the specific content of the expression rule comprises the following contents: the field ID, the field name, the belonged object ID, the field type, the return value type of the expression and the calculation formula of the expression;
the specific content of the object instance information includes: data ID, belonging object ID, version number, and value of each field.
Taking the example of the object A and the object B, the object A is associated with the object B, namely, the relationship between the object A and the object B is N:1. The example is used only for the patent description model, and the practical application is not limited to the object ID, name, field name and type in the example.
Object a is defined as shown in table 1.
TABLE 1
Object B definition is shown in table 2.
TABLE 2
An example of the object a data is shown in table 3.
TABLE 3
An example of the object B data is shown in table 4.
TABLE 4
A task library: a database storage is used for storing information of a computing task of a task scheduling module in a structured mode, and the specific content of the computing task comprises: the method comprises the following steps of task ID, task state, field ID to be calculated, creation time, completion time and other information, wherein the task state at least comprises three states of waiting to be processed, in processing and completed.
A message library: a database storage is used for storing the calculation aggregation messages of the message aggregation distribution module in a structured mode, and the specific content of the calculation aggregation messages comprises: the information comprises information such as message ID, message priority, message state, data ID, object ID, field ID list and the like, wherein the message state at least comprises two states of waiting processing and processing.
A service data module: and maintaining the service data, sending a data change message to the calculation preprocessing module after the data change is successful, and storing the changed data in the service library.
In some embodiments, the data change message comprises: data operation type, data ID, data before change and data after change;
the method for maintaining the service data, sending a data change message to the calculation preprocessing module after the data change is successful, and storing the changed data into the service library comprises the following steps:
a user submits a request for operating service data through a webpage or a terminal, wherein parameters of the request comprise a data operation type, a data ID, an object ID and data content;
taking object A-Table 3 data A-1 as an example, the N2 field is changed from 10 to 20 in the request, as shown in Table 5.
TABLE 5
Inquiring service data from a service library according to the data ID and the belonged object ID, comparing the inquired data content with the data content in the request parameter, and generating data before change and data after change;
according to the requested parameters, storing the changed data in the requested parameters to a service library according to the data operation type and the data ID, and adding one to the version number of the data;
taking data a-1 as an example, the data saved in the service repository after modification is shown in table 6.
TABLE 6
And generating a data change message DCMsg according to the data before and after the change, and sending the DCMsg to a calculation preprocessing module.
Taking data A-1 as an example, the DCMsg requested to be generated in Table 5 is shown in Table 7.
TABLE 7
A task scheduling module: according to the request of the computing task, sending a second computing message CLMsg2 for submitting the task and a task canceling to the message aggregation distribution module, and making a task call-back to the computing module; and sending the information of the calculation task to the task library.
In some embodiments, the method for sending a second computing message CLMsg2 for submitting a task to the message aggregation distribution module according to the request of the computing task includes:
receiving a request of a calculation task of an expression management module, generating a calculation task Job1 and storing the calculation task Job1 in a task library;
the method comprises the steps that one or a group of threads exist, a task state is inquired to be a to-be-processed calculation task Job1 in a timing round training mode, and meanwhile the task state is set to be processed, so that the same Job1 is prevented from being scheduled repeatedly;
inquiring data IDs in batches from a business library according to the object IDs in the trained calculation task Job1, generating a plurality of second calculation messages CLMsg2 and sending the second calculation messages CLMsg2 to a message aggregation distribution module; the message body of the second calculation message CLMsg2 comprises: task ID, field ID list, belonged object ID and data ID;
the method for making task call-back to the computing module comprises the following steps:
and receiving a request of task callback from the computing module, and updating the computing state of the computing task in the task library to be completed according to the task ID.
The calculation preprocessing module: receiving a data change message DCMsg sent by a service data module, generating a first calculation message CLMsg1 according to the data analysis expression dependency relationship before and after data change, and sending the first calculation message CLMsg1 to a message aggregation distribution module.
In some embodiments, the method for generating the first computation message CLMsg1 according to the data analysis expression dependency relationship before and after the data change and sending the first computation message CLMsg1 to the message aggregation distribution module includes:
a plurality of processes of a plurality of computer devices receive the data change message DCMsg at the same time, and a changed field list FL1 is analyzed from data before and after the change of a message body of the data change message DCMsg;
according to the object ID of the data change message DCMsg, inquiring object description information and associated object description information of the data, searching all expression rule fields, analyzing variables in a calculation formula, and generating a directed acyclic graph GP1 according to the dependency relationship among the variables; the nodes of the directed acyclic graph GP1 are fields with incidence relations in the object, the edges are dependency relations, and the directions of the edges are pointed to depended parties by the dependency parties;
taking the example of the object a and the object B in table 1 and table 2, the field dependency relationship thereof generates a directed acyclic graph as shown in fig. 2.
According to the generated directed acyclic graph GP1 and the analyzed field list FL1, traversing all nodes in the directed acyclic graph GP1 to search nodes belonging to the FL1, and obtaining a front node field list FL2 of the nodes;
inquiring data ID required to be calculated according to a preposed node field list FL2 and a data change message, generating a first calculation message CLmsg1, and sending the first calculation message CLmsg1 to a message aggregation distribution module;
taking the DCMsg of table 7 as an example, FL1 is [ obja.n2], FL2 is [ obja.f4, ojbb.c1], and the CLMsg generated is shown in table 8.
TABLE 8
The message body of the first computation message CLMsg1 comprises: a field ID list, an object ID to which it belongs, and a data ID.
The message aggregation and distribution module: receiving the first calculation message CLMsg1 sent by the calculation preprocessing module and the second calculation message CLMsg2 sent by the task scheduling module, aggregating the first calculation message CLMsg1 or the second calculation message CLMsg2 to obtain a calculation aggregation message, storing the calculation aggregation message in a message library, regularly inquiring the calculation aggregation message in the message library, and distributing the calculation aggregation message to the calculation module.
In some embodiments, the method of aggregating the first computation message CLMsg1 or the second computation message CLMsg2 comprises:
analyzing a data ID, a belonging object ID and a field ID list from a message body of the first calculation message CLMsg1 or the second calculation message CLMsg 2;
calculating the value of the message priority based on the size of the data ID, the object ID or the field type in the field ID list; the calculation method includes but is not limited to methods based on the size of the data ID, the object ID, the field type or other information in the message, and the value of P can be calculated according to the actual application scene; for example, the calculation result is expected to be seen as soon as possible from the data submitted by the business data module user through a webpage or a terminal, and the calculation result can be set to be higher priority, the calculation data quantity of the stock data after the calculation expression is modified by the task scheduling module is large, the delay tolerance of the calculation result is higher, and the calculation result can be set to be lower priority;
inquiring the calculation aggregation message to be processed from the message library according to the analyzed data ID, if the inquiry result is empty, setting the message priority as P, and storing the calculation aggregation message to be processed into the message library; if the query result is not empty, comparing the query result with the data of the query result to obtain the highest priority value P1, calculating a union set with the field ID list of the data of the query result, and storing the new priority P1 and the field ID list after calculating the union set into a message library.
Taking CLMsg of table 8 as an example, the data from page submission assuming ObjA priority P of 1 and ObjB of 100, the calculation aggregation message is generated as shown in table 9.
TABLE 9
In some embodiments, the method for regularly querying and distributing the calculation aggregate message in the message library to the calculation module includes:
dividing the message priority into several zones: r1, R2.. Rn, each interval having a start priority SP and an end priority EP, the intervals not overlapping, the intervals together being all priorities, for example: r1[0,99], R2[100, 999), R3[1000, + ∞), wherein R1 represents 0 to 99, R2 represents 100 to 999, R3 represents 1000 to infinity, each interval implementing the following scheme;
allocating thread resources according to the intervals, allocating the number of threads of each interval according to the application scene and hardware resources, for example, allocating thread pools T1, T2 and T3 to the R1, R2 and R3 intervals of the step a respectively, and executing the following processes by each thread;
the method comprises the steps that a thread periodically inquires calculation aggregation messages with priorities belonging to the interval and the message states to be processed, namely the data of which the message priorities are more than or equal to an initial priority SP and less than or equal to a termination priority EP, the inquired calculation aggregation messages are sorted according to the priorities from high to low, and a calculation module can calculate according to the priorities from high to low; taking the calculation aggregate message in table 9 as an example, the T1 thread pool may query the message M2, and the T2 thread pool may query the message M2:
updating the message state of the inquired calculation aggregation message into a processing state, sending the inquired calculation aggregation message to a calculation module for processing, and returning a processing result after calculation by the calculation module;
and according to the processing result, if the processing is successful, deleting the processed calculation aggregation message from the message library, otherwise, updating the state of the calculation aggregation message to be processed, and waiting for retrying the processing again.
The method for receiving the calculation aggregation message sent by the message aggregation distribution module, calculating the value of the expression, storing the value of the expression in the service library and returning the processing result comprises the following steps:
a plurality of processes of a plurality of computer devices simultaneously receive calculation aggregation messages sent by a message aggregation distribution module, and analyze data, data IDs, object IDs and field ID lists from message bodies of the calculation aggregation messages;
inquiring the service data information, the expression information and the data information of the incidence relation of the data from a service library according to the separated data;
analyzing variables in a calculation formula of an expression according to the inquired result, searching the values of the variables from the service data information of the data, substituting the values into the calculation formula, and calculating the values of the expression;
generating a new version number V2 based on the version number V1 of the data of the service library according to the inquired result, updating the value of the expression and the new version number V2 to the service library by using the V1 as a condition, returning to the service library for inquiry if the updating fails to indicate that the data is modified by other threads in the service library, and recalculating the current data until the data is successfully updated to the service library; the optimistic lock principle is used for updating the calculation result, so that the final consistency of the calculation result is ensured;
and after the value of the expression and the new version number V2 are successfully updated to the service library, the processing result is successfully returned.
If the above steps are abnormal conditions such as timeout and the like, the returned processing result fails, and the message aggregation distribution module tries to distribute again. If the current calculation data comes from a second calculation message CLMsg2 of the task scheduling module and is the last batch of data to be calculated, the processing result successfully calls a task callback interface of the task scheduling module to notify that the task calculation is completed.
The object a data is calculated as exemplified in table 10.
Watch 10
The object B data is calculated as exemplified in table 11.
TABLE 11
In summary, the technical solutions of the aspects of the present invention have the following advantages compared with the prior art: and aiming at the expression calculation scene based on the service data, a message mechanism is used for realizing the distributed asynchronous calculation of the expression. The newly added data is preprocessed through the calculation preprocessing module to confirm the calculation task, the stock data is recalculated through the task scheduling module to process mass data, the calculation message is aggregated and redistributed through the message aggregation and distribution module, the designated calculation task is given higher priority to be executed through the priority distribution method while the overall calculation efficiency is improved through aggregating the repeated calculation task, and the calculation efficiency of key services is further improved.
In a second aspect, the present invention discloses an electronic device, which includes a memory and a processor, where the memory stores a computer program, and the processor, when executing the computer program, implements the steps of the method in the expression-based distributed hierarchical computing system according to any one of the first aspect of the present disclosure.
Fig. 3 is a block diagram of an electronic device according to an embodiment of the present invention, and as shown in fig. 3, the electronic device includes a processor, a memory, a communication interface, a display screen, and an input device, which are connected by a system bus. Wherein the processor of the electronic device is configured to provide computing and control capabilities. The memory of the electronic equipment comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The communication interface of the electronic device is used for carrying out wired or wireless communication with an external terminal, and the wireless communication can be realized through WIFI, an operator network, near Field Communication (NFC) or other technologies. The display screen of the electronic equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the electronic equipment, an external keyboard, a touch pad or a mouse and the like.
It will be understood by those skilled in the art that the structure shown in fig. 3 is only a partial block diagram related to the technical solution of the present disclosure, and does not constitute a limitation of the electronic device to which the solution of the present application is applied, and a specific electronic device may include more or less components than those shown in the drawings, or combine some components, or have a different arrangement of components.
In a third aspect, the present invention discloses a storage medium, and in particular relates to a computer readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method in the distributed hierarchical computing system based on expressions in any one of the first aspect of the present invention are implemented.
It should be noted that the technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, however, as long as there is no contradiction between the combinations of the technical features, the scope of the present description should be considered. The above examples only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent application shall be subject to the appended claims.
Claims (10)
1. An expression-based distributed hierarchical computing system, the system comprising: the system comprises an expression management module, a service data module, a task scheduling module, a calculation preprocessing module, a message aggregation distribution module, a calculation module, a service library, a task library and a message library;
the service library: the database storage is used for storing business data information in a structured mode;
the task library: the database storage is used for structurally storing the information of the calculation tasks of the task scheduling module;
the message library: the database storage is used for structurally storing the calculation aggregation messages of the message aggregation distribution module;
the expression management module: configuring an expression rule, submitting a request of a calculation task of the total amount of stock data to the task scheduling module after the expression configuration is completed, and sending the configured expression rule to the service library;
the service data module: maintaining service data, sending a data change message to the calculation preprocessing module after the data change is successful, and storing the changed data in the service library;
the task scheduling module: according to the request of the computing task, sending a second computing message CLmsg2 for submitting the task and a task canceling to the message aggregation distribution module, and making a task call-back to the computing module; sending information of the calculation task to the task library;
the calculation preprocessing module: receiving a data change message sent by a service data module, generating a first calculation message CLMsg1 according to the data analysis expression dependency relationship before and after data change, and sending the first calculation message CLMsg1 to a message aggregation distribution module;
the message aggregation and distribution module: receiving the first calculation message CLMsg1 sent by the calculation preprocessing module and a second calculation message CLMsg2 sent by the task scheduling module, aggregating the first calculation message CLMsg1 or the second calculation message CLMsg2 to obtain a calculation aggregation message, storing the calculation aggregation message in a message library, regularly inquiring the calculation aggregation message in the message library and distributing the calculation aggregation message to the calculation module;
the calculation module: and receiving the calculation aggregation message sent by the message aggregation distribution module, calculating the value of the expression, storing the value of the expression in a service library, and returning a processing result.
2. The expression-based distributed hierarchical computation system of claim 1, wherein the service data information comprises: object description information, field description information, and object instance information;
the specific content of the object description information includes: object ID, object name, and field list;
the expression rule belongs to field description information, and the specific content of the expression rule comprises the following contents: the field ID, the field name, the belonged object ID, the field type, the return value type of the expression and the calculation formula of the expression;
the specific content of the object instance information includes: data ID, belonging object ID, version number, and value of each field.
3. An expression-based distributed hierarchical computing system according to claim 2 wherein said data change message comprises: data operation type, data ID, data before change, and data after change;
the method for maintaining the service data, sending a data change message to the calculation preprocessing module after the data change is successful, and storing the changed data in the service library comprises the following steps:
a user submits a request for operating service data through a webpage or a terminal, wherein parameters of the request comprise a data operation type, a data ID, an object ID and data content;
inquiring service data from a service library according to the data ID and the object ID, comparing the inquired data content with the data content in the requested parameter, and generating data before change and data after change;
according to the requested parameters, storing the changed data in the requested parameters to a service library according to the data operation type and the data ID, and adding one to the version number of the data;
and generating a data change message according to the data before and after the change, and sending the data change message to the calculation preprocessing module.
4. The expression-based distributed hierarchical computing system according to claim 2, wherein the method for sending a second computation message CLMsg2 submitting a task to the message aggregation distribution module according to the request of the computation task comprises:
receiving a request of a calculation task of an expression management module, generating a calculation task Job1 and storing the calculation task Job1 in a task library;
the method comprises the steps that one or a group of threads exist, a task state is found out by means of timing training and is a to-be-processed calculation task Job1, and meanwhile the task state is set to be processed;
inquiring data IDs in batches from a business library according to the object IDs in the trained calculation task Job1, generating a plurality of second calculation messages CLMsg2 and sending the second calculation messages CLMsg2 to a message aggregation distribution module; the message body of the second calculation message CLMsg2 comprises: task ID, field ID list, belonged object ID and data ID;
the method for making task call-back to the computing module comprises the following steps:
and receiving a request of task callback from the computing module, and updating the computing state of the computing task in the task library to be completed according to the task ID.
5. The expression-based distributed hierarchical computation system according to claim 2, wherein the method for generating the first computation message CLMsg1 by parsing the expression dependency according to the data before and after the data modification and sending the first computation message CLMsg1 to the message aggregation distribution module comprises:
a plurality of processes of a plurality of computer devices receive data change messages simultaneously, and a changed field list FL1 is analyzed from data before and after the change of a message body of the data change messages;
according to the ID of the object to which the data change message belongs, inquiring the description information of the object to which the data belongs and the description information of the associated object, searching all expression rule fields in the description information, analyzing variables in a calculation formula of the expression rule fields, and generating a directed acyclic graph GP1 according to the dependency relationship among the variables; the nodes of the directed acyclic graph GP1 are fields with incidence relations in the object, the edges are dependency relations, and the directions of the edges are pointed to depended parties by the dependency parties;
according to the generated directed acyclic graph GP1 and the analyzed field list FL1, traversing all nodes in the directed acyclic graph GP1 to search nodes belonging to FL1, and obtaining a front node field list FL2 of the nodes of FL1;
inquiring data ID required to be calculated according to a preposed node field list FL2 and a data change message, generating a first calculation message CLMsg1, and sending the first calculation message CLMsg1 to a message aggregation distribution module;
the message body of the first computation message CLMsg1 comprises: a field ID list, an object ID to which it belongs, and a data ID.
6. An expression-based distributed hierarchical computing system according to claim 2, wherein said method of aggregating said first computation message CLMsg1 or said second computation message CLMsg2 comprises:
parsing a data ID, a belonging object ID and a field ID list from a message body of the first computation message CLMsg1 or the second computation message CLMsg 2;
calculating the value of the message priority based on the size of the data ID, the object ID or the field type in the field ID list;
inquiring the calculation aggregation message to be processed from the message library according to the analyzed data ID, if the inquiry result is empty, setting the message priority as P, and storing the calculation aggregation message to be processed into the message library; if the query result is not null, comparing the query result with the data of the query result to obtain the highest priority value P1, calculating a union set with the field ID list of the data of the query result, and storing the new priority P1 and the field ID list after the union set calculation into a message library.
7. The distributed hierarchical expression-based computing system according to claim 6, wherein the method for regularly querying the computing aggregate message in the message library and distributing the computing aggregate message to the computing module comprises:
dividing the message priority into several zones: r1, R2.. Rn, each interval has a start priority SP and an end priority EP, the intervals do not overlap, the intervals are added together to be all priorities, and each interval executes the following process;
allocating thread resources according to the interval, wherein each thread executes the following processes;
the thread periodically inquires the calculation aggregation messages with priorities belonging to the interval and the message states to be processed, the inquired calculation aggregation messages are sorted from high to low according to the priorities, and the calculation module can calculate from high to low according to the priorities;
updating the message state of the inquired calculation aggregation message into a processing state, sending the inquired calculation aggregation message to a calculation module for processing, and returning a processing result after calculation by the calculation module;
and according to the processing result, if the processing is successful, deleting the processed calculation aggregation message from the message library, otherwise, updating the state of the calculation aggregation message to be pending, and waiting for retrying again.
8. The expression-based distributed hierarchical computation system according to claim 2, wherein the method for receiving the calculation aggregation message sent by the message aggregation distribution module, calculating the value of the expression, storing the value in the service library, and returning the processing result includes:
a plurality of processes of a plurality of computer devices simultaneously receive calculation aggregation messages sent by a message aggregation distribution module, and analyze data, data IDs, object IDs and field ID lists from message bodies of the calculation aggregation messages;
inquiring the service data information, the expression information and the information of the data of the incidence relation from the service library according to the separated data;
analyzing variables in a calculation formula of an expression according to the inquired result, searching the values of the variables from the service data information of the data, substituting the values into the calculation formula, and calculating the values of the expression;
generating a new version number V2 based on the version number V1 of the data of the service library according to the inquired result, updating the value of the expression and the new version number V2 to the service library by using the V1 as a condition, returning to the service library for inquiry if the updating fails to indicate that the data is modified by other threads in the service library, and recalculating the current data until the data is successfully updated to the service library;
and after the value of the expression and the new version number V2 are successfully updated to the service library, the processing result is successfully returned.
9. An electronic device comprising a memory and a processor, the memory having stored thereon a computer program that, when executed by the processor, performs a method in an expression-based distributed hierarchical computing system according to any of claims 1 to 8.
10. A storage medium storing a computer program executable by one or more processors and operable to implement a method in an expression-based distributed hierarchical computing system as claimed in any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310215546.8A CN115952239B (en) | 2023-03-08 | 2023-03-08 | Expression-based distributed hierarchical computing system, electronic device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310215546.8A CN115952239B (en) | 2023-03-08 | 2023-03-08 | Expression-based distributed hierarchical computing system, electronic device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115952239A true CN115952239A (en) | 2023-04-11 |
| CN115952239B CN115952239B (en) | 2023-06-23 |
Family
ID=85903260
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310215546.8A Active CN115952239B (en) | 2023-03-08 | 2023-03-08 | Expression-based distributed hierarchical computing system, electronic device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115952239B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018205853A1 (en) * | 2017-05-10 | 2018-11-15 | 腾讯科技（深圳）有限公司 | Distributed computing system and method and storage medium |
| US20210357397A1 (en) * | 2020-05-15 | 2021-11-18 | Vmware, Inc. | Efficient event-type-based distributed log-analytics system |
| CN114817389A (en) * | 2022-04-21 | 2022-07-29 | 平安科技（深圳）有限公司 | Data processing method, data processing device, storage medium and electronic equipment |
| CN115718776A (en) * | 2022-07-26 | 2023-02-28 | 上海仪电鑫森科技发展有限公司 | Big data application platform system |
-
2023
- 2023-03-08 CN CN202310215546.8A patent/CN115952239B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018205853A1 (en) * | 2017-05-10 | 2018-11-15 | 腾讯科技（深圳）有限公司 | Distributed computing system and method and storage medium |
| US20210357397A1 (en) * | 2020-05-15 | 2021-11-18 | Vmware, Inc. | Efficient event-type-based distributed log-analytics system |
| CN114817389A (en) * | 2022-04-21 | 2022-07-29 | 平安科技（深圳）有限公司 | Data processing method, data processing device, storage medium and electronic equipment |
| CN115718776A (en) * | 2022-07-26 | 2023-02-28 | 上海仪电鑫森科技发展有限公司 | Big data application platform system |
Non-Patent Citations (3)
| Title |
|---|
| XU ZHAO等: "Iprof: A Non-intrusive Request Flow Profiler for Distributed Systems", 《USENIX》 * |
| 何翔;张明明;黄高攀;: "基于分级存储环境的数据即时应用产品", 计算机系统应用, no. 03 * |
| 朱昭萌;张功萱;张永平;郭箭;张巍;: "一种基于内容的数据分发网络及算法", 计算机科学, no. 04 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115952239B (en) | 2023-06-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11190618B2 (en) | Scheduling method, scheduler, storage medium, and system | |
| CN108920222B (en) | Business processing method and device based on rule engine | |
| CN108549583B (en) | Big data processing method and device, server and readable storage medium | |
| CN108280150B (en) | Distributed asynchronous service distribution method and system | |
| CN107113341B (en) | System for high throughput processing of transactions in a distributed relational database management system for data partitioning | |
| US11475006B2 (en) | Query and change propagation scheduling for heterogeneous database systems | |
| WO2014062637A2 (en) | Systems and methods for fault tolerant, adaptive execution of arbitrary queries at low latency | |
| CN110058940B (en) | Data processing method and device in multi-thread environment | |
| CN110795479A (en) | Method and device for distributed ETL scheduling based on data | |
| CN106407021A (en) | Queue-based business request processing method | |
| US20120224482A1 (en) | Credit feedback system for parallel data flow control | |
| Wolf et al. | On the optimization of schedules for MapReduce workloads in the presence of shared scans | |
| CN113157710A (en) | Block chain data parallel writing method and device, computer equipment and storage medium | |
| CN114691658A (en) | Data backtracking method and device, electronic equipment and storage medium | |
| CN112099937A (en) | Resource management method and device | |
| CN114116149A (en) | Task scheduling method, device, equipment and medium | |
| CN111258726A (en) | Task scheduling method and device | |
| CN110389976B (en) | Multi-interface data scheduling method and device | |
| CN117331674A (en) | Task scheduling execution method, electronic device and computer readable storage medium | |
| Kang et al. | An experimental analysis of limitations of MapReduce for iterative algorithms on Spark | |
| CN112817992A (en) | Method, device, electronic equipment and readable storage medium for executing change task | |
| CN115952239A (en) | Distributed hierarchical computing system based on expression, electronic device and storage medium | |
| CN111831424B (en) | Task processing method, system and device | |
| CN111580733A (en) | Task processing method and device, computing equipment and medium | |
| CN115712572A (en) | Task testing method and device, storage medium and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |