CN108073624B - Service data processing system and method - Google Patents

Service data processing system and method Download PDF

Info

Publication number
CN108073624B
CN108073624B CN201611005030.7A CN201611005030A CN108073624B CN 108073624 B CN108073624 B CN 108073624B CN 201611005030 A CN201611005030 A CN 201611005030A CN 108073624 B CN108073624 B CN 108073624B
Authority
CN
China
Prior art keywords
statistical
service data
type
state
preset
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201611005030.7A
Other languages
Chinese (zh)
Other versions
CN108073624A (en
Inventor
郭少林
肖彭燕
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ping An Technology Shenzhen Co Ltd
Original Assignee
Ping An Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ping An Technology Shenzhen Co Ltd filed Critical Ping An Technology Shenzhen Co Ltd
Priority to CN201611005030.7A priority Critical patent/CN108073624B/en
Publication of CN108073624A publication Critical patent/CN108073624A/en
Application granted granted Critical
Publication of CN108073624B publication Critical patent/CN108073624B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
    • G06F16/2462—Approximate or statistical queries

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Probability & Statistics with Applications (AREA)
  • Computing Systems (AREA)
  • Fuzzy Systems (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Computational Linguistics (AREA)
  • Document Processing Apparatus (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a service data processing system and a method thereof, wherein the system comprises: the determining module is used for determining whether the service data before and after the operation on the service data meet the preset statistical conditions when the operation of the user on the service data in the database is detected; the conversion module is used for converting the state of the business data meeting the statistical conditions before and after operation into an operation state bit according to a preset first rule; and the statistical module is used for determining the statistical operation type corresponding to the operation state bit obtained by conversion according to the mapping relation between the predetermined operation state bit and the statistical operation type, and performing statistical operation on the database according to the determined statistical operation type. The invention does not need to compile multilayer nested complex logic codes, is simpler, does not need to carry out complicated code maintenance, and reduces the maintenance cost.

Description

Service data processing system and method
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a system and a method for processing service data.
Background
At present, business data analysis becomes an important reference for enterprises to make business directions and develop and plan. Generally, an enterprise sets different conditions to count the business data meeting the conditions, for example, when the sales data are counted, several sales indexes are preset, and then only the sales data meeting the sales indexes in the sales records of each salesperson are counted to obtain the sales data required by the enterprise.
However, since the condition setting of the existing enterprise business data statistics product is generally realized by the multi-layer nested coding logic of simple if … else, the product realized by the coding logic is extremely difficult to code when the business requirement changes, and the maintenance cost (the code needs to be modified in a large area) is high. Therefore, a new business data statistics product is needed to fulfill the business data statistics requirement of the enterprise.
Disclosure of Invention
The invention mainly aims to provide a business data processing system and a business data processing method, aiming at counting business data in a simpler mode and reducing maintenance cost.
In order to achieve the above object, the present invention provides a service data processing system, which includes:
the determining module is used for determining whether the service data meet preset statistical conditions before and after the service data are operated when the operation of a user on the service data in the database is detected;
the conversion module is used for converting the state of the business data meeting the statistical conditions before and after operation into an operation state bit according to a preset first rule;
and the statistical module is used for determining the statistical operation type corresponding to the operation state bit obtained by conversion according to the mapping relation between the predetermined operation state bit and the statistical operation type, and performing statistical operation on the database according to the determined statistical operation type.
Preferably, the state of satisfaction of the statistical condition by the service data before and after operation includes a state of satisfaction, non-satisfaction or no change, and the conversion module is further configured to:
converting the satisfied state of the statistical condition of the service data before operation into a ternary first state bit according to a preset rule; and converting the satisfied state of the statistical conditions of the service data after operation into a ternary second state bit according to a preset rule, and combining the first state bit and the second state bit to obtain a ternary operation state bit.
Preferably, if the statistical condition includes a plurality of judgment conditions related to each other, the determining module is further configured to:
when the operation of a user on the service data in the database is detected, determining whether the service data before and after the operation on the service data meet a plurality of preset judgment conditions;
the conversion module is further configured to:
and converting a plurality of satisfied states corresponding to a plurality of judgment conditions before and after the operation of the service data into a plurality of state bits according to a preset first rule, and combining the plurality of state bits to obtain the operation state bit.
Preferably, if the statistical condition includes a plurality of judgment conditions that are or-related to each other, the determining module is further configured to:
determining whether the service data meet each preset judgment condition before and after the operation on the service data;
the conversion module is further configured to:
obtaining each corresponding statistical operation type according to the change of the state meeting of each judgment condition before and after the operation of the service data, converting each statistical operation type into a type state bit according to a preset second rule, and combining a plurality of type state bits to obtain a type combination state bit;
the statistics module is further configured to:
and determining the obtained statistical operation type corresponding to the type combination state bit according to the mapping relation between the predetermined type combination state bit and the statistical operation type.
Preferably, if the statistical condition includes a plurality of determination conditions, and the plurality of determination conditions include an or relationship and an and relationship, the determining module is further configured to:
determining whether the service data meet a plurality of preset first judgment conditions before and after the service data are operated, wherein the first judgment conditions are judgment conditions of mutual and relational relationship in the statistical conditions; determining whether the service data meet each preset second judgment condition before and after the operation is performed on the service data, wherein the second judgment condition is a judgment condition of mutual or relationship in the statistical conditions;
the conversion module is further configured to:
obtaining corresponding statistical operation types according to the change of the satisfied states of the first judgment conditions before and after the operation of the service data, obtaining each corresponding statistical operation type according to the change of the satisfied states of the second judgment conditions before and after the operation of the service data, converting the obtained statistical operation types into type state bits according to a preset second rule, and combining the type state bits to obtain a type combination state bit;
the statistics module is further configured to:
and determining the obtained statistical operation type corresponding to the type combination state bit according to the mapping relation between the predetermined type combination state bit and the statistical operation type.
In addition, in order to achieve the above object, the present invention further provides a service data processing method based on the service data processing system, including the following steps:
A. when the operation of a user on the service data in the database is detected, determining whether the service data before and after the operation on the service data meet a preset statistical condition;
B. converting the state of the business data meeting the statistical conditions before and after operation into an operation state bit according to a preset first rule;
C. and determining the statistical operation type corresponding to the operation state bit obtained by conversion according to the mapping relation between the predetermined operation state bit and the statistical operation type, and performing statistical operation on the database according to the determined statistical operation type.
Preferably, the state of satisfaction of the statistical condition by the service data before and after operation includes a state of satisfaction, non-satisfaction or no change, and the step B includes:
and converting the satisfied state of the statistical condition of the service data before operation into a ternary first state bit according to a preset rule, converting the satisfied state of the statistical condition of the service data after operation into a ternary second state bit according to a preset rule, and combining the first state bit and the second state bit to obtain a ternary operation state bit.
Preferably, if the statistical condition includes a plurality of determination conditions that are related to each other, the step a is replaced with:
when the operation of a user on the service data in the database is detected, determining whether the service data before and after the operation on the service data meet a plurality of preset judgment conditions;
the step B is replaced by the following steps:
and converting a plurality of satisfied states corresponding to a plurality of judgment conditions before and after the operation of the service data into a plurality of state bits according to a preset first rule, and combining the plurality of state bits to obtain the operation state bit.
Preferably, if the statistical condition includes a plurality of mutually or related determination conditions, the step a is replaced by:
determining whether the service data meet each preset judgment condition before and after the operation on the service data;
the step B is replaced by the following steps:
obtaining each corresponding statistical operation type according to the change of the state meeting of each judgment condition before and after the operation of the service data, converting each statistical operation type into a type state bit according to a preset second rule, and combining a plurality of type state bits to obtain a type combination state bit;
the step C is replaced by the following steps:
and determining the obtained statistical operation type corresponding to the type combination state bit according to the mapping relation between the predetermined type combination state bit and the statistical operation type.
Preferably, if the statistical condition includes a plurality of determination conditions, and the determination conditions include an or relationship and an and relationship, the step a is replaced by:
determining whether the service data meet a plurality of preset first judgment conditions before and after the service data are operated, wherein the first judgment conditions are judgment conditions of mutual and relation in the statistical conditions, and determining whether the service data meet each preset second judgment condition before and after the service data are operated, and the second judgment conditions are judgment conditions of mutual or relation in the statistical conditions;
the step B is replaced by the following steps:
obtaining corresponding statistical operation types according to the change of the satisfied states of the first judgment conditions before and after the operation of the service data, obtaining each corresponding statistical operation type according to the change of the satisfied states of the second judgment conditions before and after the operation of the service data, converting the obtained statistical operation types into type state bits according to a preset second rule, and combining the type state bits to obtain a type combination state bit;
the step C is replaced by the following steps:
and determining the obtained statistical operation type corresponding to the type combination state bit according to the mapping relation between the predetermined type combination state bit and the statistical operation type.
In addition, in order to achieve the above object, the present invention further provides a service data processing method based on the service data processing system, including the following steps:
configuring a preset data synchronization tool, and creating a target table for storing business data by using the data synchronization tool; creating a distributed publishing and subscribing message system, and integrating a real-time computing frame by using the distributed publishing and subscribing message system;
when the operation of the user on the service data in the target table is detected, the distributed publishing and subscribing message system determines the statistical operation type of the service data based on a preset calculation rule;
and the real-time computing frame carries out statistical operation on the service data in the target table according to the determined statistical operation type, acquires the real-time statistical data of the service data in the target table and displays the real-time statistical data to a user.
Preferably, the method further comprises:
integrating an open source database by using the distributed publish-subscribe message system, wherein the open source database is used for storing historical data with the same service data identifier as a main key;
setting a table for recording the statistical times of the same service data identifier in the open source database;
if different types of indexes in the service data of the same service data identifier are recorded, maintaining the counting times of the same service data identifier unchanged; and if one piece of service data of the same service data identifier is deleted, subtracting 1 from the number of times of counting the same service data identifier.
In addition, in order to achieve the above object, the present invention further provides a service data processing system, where the service data processing system includes a data synchronization tool, a distributed publish-subscribe message system, and a real-time computation framework, where:
the data synchronization tool is used for creating a target table for storing business data;
the distributed publishing and subscribing message system is used for integrating a real-time computing framework and determining a statistical operation type of the business data based on a preset computing rule when detecting the operation of a user on the business data in the target table;
and the real-time computing frame is used for carrying out statistical operation on the service data in the target table according to the determined statistical operation type, acquiring the real-time statistical data of the service data in the target table and displaying the real-time statistical data to a user.
Preferably, the distributed publish-subscribe messaging system is further configured to integrate a source database;
the open source database is used for storing historical data with the same service data identifier as a main key, setting a table for recording the statistical frequency of the same service data identifier, if different types of indexes in the service data of the same service data identifier are recorded, maintaining the statistical frequency of the same service data identifier unchanged, and if one piece of service data of the same service data identifier is deleted, reducing the statistical frequency of the same service data identifier by 1.
When the operation of a user on the service data in the database is detected, the satisfied state of the service data before and after the operation on the preset statistical condition is converted into an operation state bit according to a preset first rule, and the statistical operation type corresponding to the operation state bit is determined according to the mapping relation between the predetermined operation state bit and the statistical operation type so as to carry out the statistical operation. The corresponding statistical operation type can be determined only by converting the satisfied state of the preset statistical conditions of the service data before and after operation into the corresponding operation state bit, and the writing of multi-layer nested complex logic codes is not needed, so that the method is simpler, complex code maintenance is not needed, and the maintenance cost is reduced.
Drawings
FIG. 1 is a functional block diagram of an embodiment of a business data processing system according to the present invention;
fig. 2 is a schematic flow chart of an embodiment of a service data processing method according to the present invention;
fig. 3 is a schematic flow chart of another embodiment of a service data processing method according to the present invention;
fig. 4 is a schematic structural diagram of another embodiment of the service data processing system according to the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
In order to make the technical problems, technical solutions and advantageous effects to be solved by the present invention clearer and clearer, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The invention provides a service data processing system.
Referring to fig. 1, fig. 1 is a functional module schematic diagram of an embodiment of a service data processing system according to the present invention.
In one embodiment, the business data processing system comprises:
the determining module 01 is configured to determine whether the service data before and after the operation on the service data meets a preset statistical condition when the operation on the service data in the database by the user is detected.
In this embodiment, when it is detected that a user performs operations such as inserting, modifying, updating, and deleting on service data in a database, it is determined whether the service data before the operation on the service data and after the operation on the service data satisfy a preset statistical condition. The statistical condition may include one or more determination conditions, and the determination conditions may be related to each other, or a relationship, and the like, which is not limited herein.
For example, in one embodiment, the business data in the database is a sales record of a salesperson, and for the real-time statistical requirements of the sales business data, mainly the indexes of interest are counted, which are given by SQL, such as:
select index 1, index 2, case where judgment condition index 3 …
from table
where Condition 1, Condition 2, …
Wherein, the conditions in case where and where determine whether the index needs to be counted. These conditions are embodied in a sales record of the sales staff, and if the sales record contains a field ORDER _ STATUS _ CODE, the ORDER _ STATUS _ CODE belongs to {08,09,20,14,09,01,02,05} for calculating the judgment conditions of the pre-underwriting premium and the pre-underwriting number, and the premium and the number can be calculated only when the ORDER _ STATUS _ CODE belongs to {08,09,20,14,09,01,02,05} in the real-time statistics of the sales service data. There are three situations here:
1. insert, that is, a sales record is newly entered by a salesperson and stored in a database by fields, and at this time, it is only necessary to determine whether or not the ORDER _ STATUS _ CODE satisfies the condition. If yes, the premium and the number of the pieces are counted; if not, not counting;
2. update, i.e., the sales person changes an already existing sales record, such as changing the field ORDER _ STATUS _ CODE, history 08, change to
Figure GDA0002674226200000071
It can be seen that when ORDER _ STATUS _ CODE does not satisfy the condition, the premium and the number of records cannot be counted. Since the premium and the number of pieces have already been counted in the old state since the change from 08 to 03 (satisfaction to non-satisfaction), the counted premium and the number of pieces are subtracted in the new state;
3. delete, i.e. the salesperson deletes a sales record, and a row of data is deleted corresponding to the database. Similarly, taking the ORDER _ STATUS _ CODE as 08 as an example, the STATUS change at this time is from 08 satisfied to no data record, and can be regarded as not satisfied, i.e. from satisfied to not satisfied, the premium and the number of pieces have been counted in the old state, so the counted premium and the number of pieces are subtracted in the new state.
In the case of updating, etc., it is necessary to determine how to perform statistics finally by determining whether the sales records in the state before and after the operations of updating, etc. on the sales records in the database satisfy the statistical conditions. Therefore, in this embodiment, when it is detected that the user performs operations such as inserting, modifying, updating, and deleting on the sales record in the database, it is determined whether the sales record before the operation on the sales record and after the operation on the sales record satisfies a preset statistical condition.
And the conversion module 02 is configured to convert the state of the service data, which satisfies the statistical condition before and after the operation, into an operation state bit according to a preset first rule.
And after determining whether the service data meet preset statistical conditions in the old and new states before the service data are operated and after the service data are operated, converting the meeting state of the statistical conditions of the service data in the old and new states into an operation state bit according to a preset first rule. The state of the service data in the old state and the new state meeting the statistical condition may include meeting, not meeting, and the like, which is not limited herein. When the satisfied state of the statistical condition of the service data in the old and new states is converted into the operation state bit, the conversion can be performed according to different systems, such as ternary system, decimal system, and the like, which is not limited herein.
The statistical module 03 is configured to determine a statistical operation type corresponding to the converted operation state bit according to a predetermined mapping relationship between the operation state bit and the statistical operation type, and perform statistical operation on the database according to the determined statistical operation type.
The mapping relationship between different operation status bits and the statistical operation type is determined in advance, for example, the different operation status bits may be mapped with different statistical operation types such as add operation, subtract operation, and the like one to one, or many to one, so that after the satisfied status of the statistical conditions in the new and old states of the service data is converted into the operation status bits, the statistical operation type corresponding to the converted operation status bits may be determined according to the predetermined mapping relationship between the different operation status bits and the statistical operation type, and then the statistical operations such as add operation, subtract operation, and the like may be performed on the database according to the determined statistical operation type.
In this embodiment, when an operation of a user on service data in a database is detected, a state that the service data meets a preset statistical condition before and after the operation is converted into an operation state bit according to a preset first rule, and a statistical operation type corresponding to the operation state bit is determined according to a mapping relationship between the predetermined operation state bit and the statistical operation type, so as to perform a statistical operation. The corresponding statistical operation type can be determined only by converting the satisfied state of the preset statistical conditions of the service data before and after operation into the corresponding operation state bit, and the writing of multi-layer nested complex logic codes is not needed, so that the method is simpler, complex code maintenance is not needed, and the maintenance cost is reduced.
Further, in other embodiments, the state of the service data before and after the operation, which satisfies the statistical condition, includes a state of satisfying, not satisfying, or not changing, and the conversion module 02 is further configured to:
and converting the satisfied state of the statistical condition of the service data before operation into a ternary first state bit according to a preset rule, converting the satisfied state of the statistical condition of the service data after operation into a ternary second state bit according to a preset rule, and combining the first state bit and the second state bit to obtain a ternary operation state bit.
In this embodiment, the service data in the database is taken as the sales record of the salesperson, and the operation of the sales record in the database is update operation is specifically described as an example:
for update, it needs to be considered whether the sales data of the business data before updating, i.e. at time t, and the sales data of the business data after updating, i.e. at time t +1, meet statistical conditions. For the final statistical operation on the database, the statistical conditions, that is, the states of the judgment conditions at the new time (time t +1) and the old time (time t), are mainly depended on, and if the time t is not met and the time t +1 is met, the sales data needs to be calculated. In update operation, there are six situations in which the state of the service data changes between new (time t +1) and old (time t), as shown in table 1 below:
Figure GDA0002674226200000091
TABLE 1
Converting the satisfied states of the service data to the statistical conditions such as "unsatisfied", "unchanged" and "satisfied" at the new (time t +1) and the old (time t) into ternary state bits, which are respectively represented by 0,1 and 2, so that table 1 can be converted into the following table 2:
Figure GDA0002674226200000101
TABLE 2
In one embodiment, taking the example of statistics of the sales policy premium of a salesperson, assume that at time t the premium of the salesperson is summoney (t) s, where s > is 0.
1. For the newly added passing policy A at the moment t +1, the premium is a1, the policy premium sumMoney (t +1) is s + a1, and the statistical operation type at this time is premium adding operation;
2. if a passing policy B is added at the moment t +2, the premium is B1, the policy premium is sumMoney (t +2) s + a1+ B1, and the statistical operation type at the moment is premium adding operation;
3. if the policy B is deleted at the moment t +3, the sales person deletes the policy B, the policy premium sumMoney (t +3) is s + a1, and the statistical operation type at the moment is the premium reduction operation;
4. if the policy C is not passed, the salesperson updates the policy C which is created before but not passed at the time of t +4, so that the policy C passes the examination, the premium of the policy is C1, the premium of the policy is sumMoney (t +4) ═ s + a1+ C1, and the statistical operation type at this time is premium adding operation;
5. at time t +5, the salesperson modifies policy A again to change the premium from original a1 to a2, the premium of the policy at this time is sumMoney (t +5) ═ s + a1+ c1+ (a1-a2), and the statistical operation type at this time is the premium reduction operation.
Therefore, although the indexes needing to be counted are various, the counting operation types of the counting indexes in the service scene only comprise the charging and discharging operations of the insurance policy, and when the insurance policy number is further counted, the counting operation types comprise charging and discharging operations of the insurance policy, charging and discharging operations of the insurance policy number and the insurance policy, and no operation. As shown in Table 2, when statistics on the number of policy and premium are required, cases 1,2, 3, 4, 5, and 6 correspond to a type of statistical operation: the cases 1 and 2 correspond to simple operation of adding and subtracting premium; the situation 3 corresponds to the number of the insurance policies and the insurance fee to reduce the operation; case 4 corresponds to the number of insurance policies and the insurance fee to be added; case 5 and case 6 correspond to no operation.
Due to the finite state bits, the state bits at t +1 and t are converted into ternary representations, and taking table 2 as an example, six cases can be respectively represented as follows: 22. 12, 02, 20, 10, 00, optionally, the state bits of the ternary system can be converted into decimal systems, and the corresponding decimal systems are respectively: 8. 5, 2, 6, 3 and 0. It can be predetermined that decimal numbers 0 and 3 correspond to no operation, 5 and 8 correspond to plus or minus of the simple premium, 2 corresponds to minus of the number of policies and the premium, and 6 corresponds to plus of the number of policies and the premium. Converting the satisfied state of the statistical condition before and after the operation of the service data into an operation state bit, for example, the conversion can be performed according to the following formula:
y＝a0β2n-1+a1β2n-2+…+a2n-2β+a2n-1
wherein: beta, n epsilon [1,2, … ]]β is a system number, and n is the number of judgment conditions. a is2i-2And a2i-1Respectively representing the states of a judgment index at the time t +1 and the time t. Determining the statistical operation type corresponding to the converted operation status bit according to the predetermined mapping relationship between the operation status bit and the statistical operation type, that is, according to the determined operation status bitAnd carrying out statistical operation on the database according to the statistical operation type.
Further, in another embodiment, if the statistical condition includes a plurality of determination conditions related to each other, the determining module 01 is further configured to:
when the operation of a user on the service data in the database is detected, whether the service data meet a plurality of preset judgment conditions before and after the operation on the service data is determined.
The conversion module 02 is further configured to:
and converting a plurality of satisfied states corresponding to a plurality of judgment conditions before and after the operation of the service data into a plurality of state bits according to a preset first rule, and combining the plurality of state bits to obtain the operation state bit.
If the statistical condition includes a plurality of mutually related determination conditions, the following description is given by taking the following conditions 1 and 2 as examples in table 3:
Figure GDA0002674226200000111
TABLE 3
The conditions 1 and 2 are in an inclusive relationship, and β is 3 and n is 2, so that the ternary of the state bit is 2020, and the conversion is performed, and y is 2 × 33+0*32And +2 × 3+0 is 60, the corresponding decimal is 60, and 60 is the type combination state bit of the decimal, and the statistical operation type corresponding to the obtained type combination state bit is determined according to the mapping relation between the predetermined type combination state bit and the statistical operation type, such as addition operation, so that the statistical operation is performed on the database according to the determined statistical operation type.
Further, in another embodiment, if the statistical condition includes a plurality of determination conditions that are or-related to each other, the determining module 01 is further configured to:
and determining whether the service data meet each preset judgment condition before and after the operation on the service data.
The conversion module 02 is further configured to:
and obtaining each corresponding statistical operation type according to the change of the state meeting of each judgment condition before and after the operation of the service data, converting each statistical operation type into a type state bit according to a preset second rule, and combining a plurality of type state bits to obtain a type combination state bit.
The statistical module 03 is further configured to:
and determining the obtained statistical operation type corresponding to the type combination state bit according to the mapping relation between the predetermined type combination state bit and the statistical operation type.
Four types of statistical operation are defined in the above embodiments: the method comprises the steps of setting no operation, adding operation, subtracting operation and simple adding and subtracting as 0,1, 2 and 3 respectively, determining a corresponding statistical operation type according to the change of the satisfied state of each judgment condition before and after operation of the service data, and converting the statistical operation type corresponding to each judgment condition in a plurality of judgment conditions which are mutually or in a relationship into type state bits such as 0,1, 2 and 3 by adopting a quaternary system. Here, conditions 1 and 2 in the following table 4 are explained as examples:
Figure GDA0002674226200000121
TABLE 4
If the condition 1 and the condition 2 are in an or relationship, the type status bits of the statistical operation type conversion corresponding to the condition 1 may be 0,1, 2, and 3, and the type status bits of the statistical operation type conversion corresponding to the condition 2 may also be 0,1, 2, and 3, then in all cases including the condition 1 and the condition 2 which are in an or relationship with each other, there are 256 types of combined status bits obtained by combining the type status bits of the condition 1 and the condition 2, as follows: 00. 01,02, 03 … … 30, 31, 32, 33. Specific analyses are here performed with 00, 11, 22, 33 and 12 as examples:
1. 00, condition 1 is no operation, condition 2 is also no operation, both conditions or also no operation.
2. 11, the condition 1 is an addition operation, which is from unsatisfied to satisfied; condition 2 is an add operation, from unsatisfied to satisfied; two conditional or also additive operations.
3. At 22, condition 1 is a subtract operation, from satisfied to unsatisfied; condition 2 is a subtract operation, from satisfied to unsatisfied; two conditional or also subtract operations.
4. 33, the conditions 1 and 2 are simply added or subtracted from the satisfied state to the satisfied state or are unchanged from the satisfied state to the satisfied state; two conditions or simple addition and subtraction.
5. 12, the condition 1 is addition operation, which is from unsatisfied to satisfied; condition 2 is a subtraction operation, from satisfied to occasionally satisfied; however, if the two conditions are yes, the time t +1 is satisfied or unsatisfied, that is, the time t is unsatisfied or satisfied, and is also satisfied, so that the corresponding operation at this time is simply added or subtracted.
Therefore, each type combination status bit corresponds to a statistical operation type, such as no operation, add operation, subtract operation, or simply add or subtract. Therefore, the mapping relationship between different types of combination status bits and the statistical operation type may be determined in advance, and after the type status bits of the plurality of mutually or related judgment conditions in the statistical condition are combined to obtain the type combination status bit, the statistical operation type corresponding to the obtained type combination status bit may be determined according to the predetermined mapping relationship between the different types of combination status bits and the statistical operation type, that is, the statistical operations such as the addition operation and the subtraction operation may be performed on the database according to the determined statistical operation type.
Further, the type combination status bit can also be converted into decimal by the following formula:
y＝a0β2n-1+a1β2n-2+…+a2n-2β+a2n-1
the mapping setting and confirmation of different types of combination state bits and statistical operation types are carried out through decimal values, and the method is more convenient. For example, with (33)4For example, the decimal is y-3 x 42+3*4＝60。
Further, in another embodiment, if the statistical condition includes a plurality of determination conditions, and the plurality of determination conditions include an or relationship and an and relationship, the determining module 01 is further configured to:
determining whether the service data meet a plurality of preset first determination conditions before and after the service data are operated, wherein the first determination conditions are determination conditions of mutual and relation in the statistical conditions, and determining whether the service data meet each preset second determination condition before and after the service data are operated, and the second determination conditions are determination conditions of mutual or relation in the statistical conditions.
The conversion module 02 is further configured to:
obtaining corresponding statistical operation types according to the change of the satisfied states of the first judgment conditions before and after the operation of the service data, obtaining each corresponding statistical operation type according to the change of the satisfied states of the second judgment conditions before and after the operation of the service data, converting the obtained statistical operation types into type state bits according to a preset second rule, and combining the type state bits to obtain a type combination state bit.
The statistical module 03 is further configured to:
and determining the obtained statistical operation type corresponding to the type combination state bit according to the mapping relation between the predetermined type combination state bit and the statistical operation type.
In this embodiment, when the statistical conditions include a plurality of or-relationship determination conditions and a plurality of or-relationship determination conditions, determining the or-relationship, determining all statistical operation types corresponding to the combination of the conditions, determining the or-relationship, determining the statistical operation type corresponding to the determination condition of each or-relationship, converting all the obtained statistical operation types into type status bits according to a preset second rule, and combining the type status bits to obtain type combination status bits, that is, determining the obtained statistical operation type corresponding to the type combination status bits according to a predetermined mapping relationship between the type combination status bits and the statistical operation types. The specific process of determining the statistical operation type corresponding to the condition, determining the statistical operation type corresponding to the condition or the condition, and determining the corresponding statistical operation type according to the predetermined mapping relationship between the type combination status bit and the statistical operation type is as described in the above embodiments, and is not described herein again.
The invention further provides a service data processing method based on the service data processing system.
Referring to fig. 2, fig. 2 is a flowchart illustrating a service data processing method according to an embodiment of the present invention.
In an embodiment, the service data processing method includes:
step S10, when detecting the operation of the user on the service data in the database, determining whether the service data before and after the operation on the service data meets a preset statistical condition.
In this embodiment, when it is detected that a user performs operations such as inserting, modifying, updating, and deleting on service data in a database, it is determined whether the service data before the operation on the service data and after the operation on the service data satisfy a preset statistical condition. The statistical condition may include one or more determination conditions, and the determination conditions may be related to each other, or a relationship, and the like, which is not limited herein.
For example, in one embodiment, the business data in the database is a sales record of a salesperson, and for the real-time statistical requirements of the sales business data, mainly the indexes of interest are counted, which are given by SQL, such as:
select index 1, index 2, case where judgment condition index 3 …
from table
where Condition 1, Condition 2, …
Wherein, the conditions in case where and where determine whether the index needs to be counted. These conditions are embodied in a sales record of the sales staff, and if the sales record contains a field ORDER _ STATUS _ CODE, the ORDER _ STATUS _ CODE belongs to {08,09,20,14,09,01,02,05} for calculating the judgment conditions of the pre-underwriting premium and the pre-underwriting number, and the premium and the number can be calculated only when the ORDER _ STATUS _ CODE belongs to {08,09,20,14,09,01,02,05} in the real-time statistics of the sales service data. There are three situations here:
1. insert, that is, a sales record is newly entered by a salesperson and stored in a database by fields, and at this time, it is only necessary to determine whether or not the ORDER _ STATUS _ CODE satisfies the condition. If yes, the premium and the number of the pieces are counted; if not, not counting;
2. update, i.e., the sales person changes an already existing sales record, such as changing the field ORDER _ STATUS _ CODE, history 08, change to
Figure GDA0002674226200000151
It can be seen that when ORDER _ STATUS _ CODE does not satisfy the condition, the premium and the number of records cannot be counted. Since the premium and the number of pieces have already been counted in the old state since the change from 08 to 03 (satisfaction to non-satisfaction), the counted premium and the number of pieces are subtracted in the new state;
3. delete, i.e. the salesperson deletes a sales record, and a row of data is deleted corresponding to the database. Similarly, taking the ORDER _ STATUS _ CODE as 08 as an example, the STATUS change at this time is from 08 satisfied to no data record, and can be regarded as not satisfied, i.e. from satisfied to not satisfied, the premium and the number of pieces have been counted in the old state, so the counted premium and the number of pieces are subtracted in the new state.
In the case of updating, etc., it is necessary to determine how to perform statistics finally by determining whether the sales records in the state before and after the operations of updating, etc. on the sales records in the database satisfy the statistical conditions. Therefore, in this embodiment, when it is detected that the user performs operations such as inserting, modifying, updating, and deleting on the sales record in the database, it is determined whether the sales record before the operation on the sales record and after the operation on the sales record satisfies a preset statistical condition.
Step S20, converting the satisfied state of the statistical condition before and after the operation of the service data into an operation status bit according to a preset first rule.
And after determining whether the service data meet preset statistical conditions in the old and new states before the service data are operated and after the service data are operated, converting the meeting state of the statistical conditions of the service data in the old and new states into an operation state bit according to a preset first rule. The state of the service data in the old state and the new state meeting the statistical condition may include meeting, not meeting, and the like, which is not limited herein. When the satisfied state of the statistical condition of the service data in the old and new states is converted into the operation state bit, the conversion can be performed according to different systems, such as ternary system, decimal system, and the like, which is not limited herein.
Step S30, determining the statistical operation type corresponding to the converted operation status bit according to the predetermined mapping relationship between the operation status bit and the statistical operation type, and performing statistical operation on the database according to the determined statistical operation type.
The mapping relationship between different operation status bits and the statistical operation type is determined in advance, for example, the different operation status bits may be mapped with different statistical operation types such as add operation, subtract operation, and the like one to one, or many to one, so that after the satisfied status of the statistical conditions in the new and old states of the service data is converted into the operation status bits, the statistical operation type corresponding to the converted operation status bits may be determined according to the predetermined mapping relationship between the different operation status bits and the statistical operation type, and then the statistical operations such as add operation, subtract operation, and the like may be performed on the database according to the determined statistical operation type.
In this embodiment, when an operation of a user on service data in a database is detected, a state that the service data meets a preset statistical condition before and after the operation is converted into an operation state bit according to a preset first rule, and a statistical operation type corresponding to the operation state bit is determined according to a mapping relationship between the predetermined operation state bit and the statistical operation type, so as to perform a statistical operation. The corresponding statistical operation type can be determined only by converting the satisfied state of the preset statistical conditions of the service data before and after operation into the corresponding operation state bit, and the writing of multi-layer nested complex logic codes is not needed, so that the method is simpler, complex code maintenance is not needed, and the maintenance cost is reduced.
Further, in other embodiments, the state of satisfaction of the statistical condition by the service data before and after the operation includes a state of satisfaction, non-satisfaction, or no change, and the step S20 may include:
and converting the satisfied state of the statistical condition of the service data before operation into a ternary first state bit according to a preset rule, converting the satisfied state of the statistical condition of the service data after operation into a ternary second state bit according to a preset rule, and combining the first state bit and the second state bit to obtain a ternary operation state bit.
In this embodiment, the service data in the database is taken as the sales record of the salesperson, and the operation of the sales record in the database is update operation is specifically described as an example:
for update, it needs to be considered whether the sales data of the business data before updating, i.e. at time t, and the sales data of the business data after updating, i.e. at time t +1, meet statistical conditions. For the final statistical operation on the database, the statistical conditions, that is, the states of the judgment conditions at the new time (time t +1) and the old time (time t), are mainly depended on, and if the time t is not met and the time t +1 is met, the sales data needs to be calculated. In update operation, there are six situations in which the state of the service data changes between new (time t +1) and old (time t), as shown in table 1 below:
Figure GDA0002674226200000171
TABLE 1
Converting the satisfied states of the service data to the statistical conditions such as "unsatisfied", "unchanged" and "satisfied" at the new (time t +1) and the old (time t) into ternary state bits, which are respectively represented by 0,1 and 2, so that table 1 can be converted into the following table 2:
Figure GDA0002674226200000172
TABLE 2
In one embodiment, taking the example of statistics of the sales policy premium of a salesperson, assume that at time t the premium of the salesperson is summoney (t) s, where s > is 0.
6. For the newly added passing policy A at the moment t +1, the premium is a1, the policy premium sumMoney (t +1) is s + a1, and the statistical operation type at this time is premium adding operation;
7. if a passing policy B is added at the moment t +2, the premium is B1, the policy premium is sumMoney (t +2) s + a1+ B1, and the statistical operation type at the moment is premium adding operation;
8. if the policy B is deleted at the moment t +3, the sales person deletes the policy B, the policy premium sumMoney (t +3) is s + a1, and the statistical operation type at the moment is the premium reduction operation;
9. if the policy C is not passed, the salesperson updates the policy C which is created before but not passed at the time of t +4, so that the policy C passes the examination, the premium of the policy is C1, the premium of the policy is sumMoney (t +4) ═ s + a1+ C1, and the statistical operation type at this time is premium adding operation;
10. at time t +5, the salesperson modifies policy A again to change the premium from original a1 to a2, the premium of the policy at this time is sumMoney (t +5) ═ s + a1+ c1+ (a1-a2), and the statistical operation type at this time is the premium reduction operation.
Therefore, although the indexes needing to be counted are various, the counting operation types of the counting indexes in the service scene only comprise the charging and discharging operations of the insurance policy, and when the insurance policy number is further counted, the counting operation types comprise charging and discharging operations of the insurance policy, charging and discharging operations of the insurance policy number and the insurance policy, and no operation. As shown in Table 2, when statistics on the number of policy and premium are required, cases 1,2, 3, 4, 5, and 6 correspond to a type of statistical operation: the cases 1 and 2 correspond to simple operation of adding and subtracting premium; the situation 3 corresponds to the number of the insurance policies and the insurance fee to reduce the operation; case 4 corresponds to the number of insurance policies and the insurance fee to be added; case 5 and case 6 correspond to no operation.
Due to the finite state bits, the state bits at t +1 and t are converted into ternary representations, and taking table 2 as an example, six cases can be respectively represented as follows: 22. 12, 02, 20, 10, 00, optionally, the state bits of the ternary system can be converted into decimal systems, and the corresponding decimal systems are respectively: 8. 5, 2, 6, 3 and 0. It can be predetermined that decimal numbers 0 and 3 correspond to no operation, 5 and 8 correspond to plus or minus of the simple premium, 2 corresponds to minus of the number of policies and the premium, and 6 corresponds to plus of the number of policies and the premium. Converting the satisfied state of the statistical condition before and after the operation of the service data into an operation state bit, for example, the conversion can be performed according to the following formula:
y＝a0β2n-1+a1β2n-2+…+a2n-2β+a2n-1
wherein: beta, n epsilon [1,2, … ]]β is a system number, and n is the number of judgment conditions. a is2i-2And a2i-1Respectively representing the states of a judgment index at the time t +1 and the time t. And determining the statistical operation type corresponding to the operation state bit obtained by conversion according to the mapping relation between the predetermined operation state bit and the statistical operation type, namely performing statistical operation on the database according to the determined statistical operation type.
Further, in another embodiment, if the statistical condition includes a plurality of mutually related determination conditions, the step S10 may be replaced by:
when the operation of a user on the service data in the database is detected, whether the service data meet a plurality of preset judgment conditions before and after the operation on the service data is determined.
The above step S20 may be replaced by:
and converting a plurality of satisfied states corresponding to a plurality of judgment conditions before and after the operation of the service data into a plurality of state bits according to a preset first rule, and combining the plurality of state bits to obtain the operation state bit.
If the statistical condition includes a plurality of mutually related determination conditions, the following description is given by taking the following conditions 1 and 2 as examples in table 3:
Figure GDA0002674226200000191
TABLE 3
The conditions 1 and 2 are in an inclusive relationship, and β is 3 and n is 2, so that the ternary of the state bit is 2020, and the conversion is performed, and y is 2 × 33+0*32And +2 × 3+0 is 60, the corresponding decimal is 60, and 60 is the type combination state bit of the decimal, and the statistical operation type corresponding to the obtained type combination state bit is determined according to the mapping relation between the predetermined type combination state bit and the statistical operation type, such as addition operation, so that the statistical operation is performed on the database according to the determined statistical operation type.
Further, in another embodiment, if the statistical condition includes a plurality of mutually or related determination conditions, the step S10 is replaced with:
and determining whether the service data meet each preset judgment condition before and after the operation on the service data.
The above step S20 is replaced with:
and obtaining each corresponding statistical operation type according to the change of the state meeting of each judgment condition before and after the operation of the service data, converting each statistical operation type into a type state bit according to a preset second rule, and combining a plurality of type state bits to obtain a type combination state bit.
The above step S30 is replaced with:
and determining the obtained statistical operation type corresponding to the type combination state bit according to the mapping relation between the predetermined type combination state bit and the statistical operation type.
Four types of statistical operation are defined in the above embodiments: the method comprises the steps of setting no operation, adding operation, subtracting operation and simple adding and subtracting as 0,1, 2 and 3 respectively, determining a corresponding statistical operation type according to the change of the satisfied state of each judgment condition before and after operation of the service data, and converting the statistical operation type corresponding to each judgment condition in a plurality of judgment conditions which are mutually or in a relationship into type state bits such as 0,1, 2 and 3 by adopting a quaternary system. Here, conditions 1 and 2 in the following table 4 are explained as examples:
Figure GDA0002674226200000201
TABLE 4
If the condition 1 and the condition 2 are in an or relationship, the type status bits of the statistical operation type conversion corresponding to the condition 1 may be 0,1, 2, and 3, and the type status bits of the statistical operation type conversion corresponding to the condition 2 may also be 0,1, 2, and 3, then in all cases including the condition 1 and the condition 2 which are in an or relationship with each other, there are 256 types of combined status bits obtained by combining the type status bits of the condition 1 and the condition 2, as follows: 00. 01,02, 03 … … 30, 31, 32, 33. Specific analyses are here performed with 00, 11, 22, 33 and 12 as examples:
6. 00, condition 1 is no operation, condition 2 is also no operation, both conditions or also no operation.
7. 11, the condition 1 is an addition operation, which is from unsatisfied to satisfied; condition 2 is an add operation, from unsatisfied to satisfied; two conditional or also additive operations.
8. At 22, condition 1 is a subtract operation, from satisfied to unsatisfied; condition 2 is a subtract operation, from satisfied to unsatisfied; two conditional or also subtract operations.
9. 33, the conditions 1 and 2 are simply added or subtracted from the satisfied state to the satisfied state or are unchanged from the satisfied state to the satisfied state; two conditions or simple addition and subtraction.
10. 12, the condition 1 is addition operation, which is from unsatisfied to satisfied; condition 2 is a subtraction operation, from satisfied to occasionally satisfied; however, if the two conditions are yes, the time t +1 is satisfied or unsatisfied, that is, the time t is unsatisfied or satisfied, and is also satisfied, so that the corresponding operation at this time is simply added or subtracted.
Therefore, each type combination status bit corresponds to a statistical operation type, such as no operation, add operation, subtract operation, or simply add or subtract. Therefore, the mapping relationship between different types of combination status bits and the statistical operation type may be determined in advance, and after the type status bits of the plurality of mutually or related judgment conditions in the statistical condition are combined to obtain the type combination status bit, the statistical operation type corresponding to the obtained type combination status bit may be determined according to the predetermined mapping relationship between the different types of combination status bits and the statistical operation type, that is, the statistical operations such as the addition operation and the subtraction operation may be performed on the database according to the determined statistical operation type.
Further, the type combination status bit can also be converted into decimal by the following formula:
y＝a0β2n-1+a1β2n-2+…+a2n-2β+a2n-1
the mapping setting and confirmation of different types of combination state bits and statistical operation types are carried out through decimal values, and the method is more convenient. For example, with (33)4For example, the decimal is y-3 x 42+3*4＝60。
Further, in another embodiment, if the statistical condition includes a plurality of determination conditions, and the determination conditions include an or relationship and an and relationship, the step S10 is replaced with:
determining whether the service data meet a plurality of preset first determination conditions before and after the service data are operated, wherein the first determination conditions are determination conditions of mutual and relation in the statistical conditions, and determining whether the service data meet each preset second determination condition before and after the service data are operated, and the second determination conditions are determination conditions of mutual or relation in the statistical conditions.
The above step S20 is replaced with:
obtaining corresponding statistical operation types according to the change of the satisfied states of the first judgment conditions before and after the operation of the service data, obtaining each corresponding statistical operation type according to the change of the satisfied states of the second judgment conditions before and after the operation of the service data, converting the obtained statistical operation types into type state bits according to a preset second rule, and combining the type state bits to obtain a type combination state bit.
The above step S30 is replaced with:
and determining the obtained statistical operation type corresponding to the type combination state bit according to the mapping relation between the predetermined type combination state bit and the statistical operation type.
In this embodiment, when the statistical conditions include a plurality of or-relationship determination conditions and a plurality of or-relationship determination conditions, determining the or-relationship, determining all statistical operation types corresponding to the combination of the conditions, determining the or-relationship, determining the statistical operation type corresponding to the determination condition of each or-relationship, converting all the obtained statistical operation types into type status bits according to a preset second rule, and combining the type status bits to obtain type combination status bits, that is, determining the obtained statistical operation type corresponding to the type combination status bits according to a predetermined mapping relationship between the type combination status bits and the statistical operation types. The specific process of determining the statistical operation type corresponding to the condition, determining the statistical operation type corresponding to the condition or the condition, and determining the corresponding statistical operation type according to the predetermined mapping relationship between the type combination status bit and the statistical operation type is as described in the above embodiments, and is not described herein again.
The invention further provides a service data processing method based on the service data processing system.
Referring to fig. 3, fig. 3 is a schematic flow chart of another embodiment of the service data processing method of the present invention.
In another embodiment, the service data processing method includes:
step S40, a preset data synchronization tool (Oracle Golden Gate, OGG for short) is configured, the Oracle Golden Gate software is log-based structured data replication backup software, and it obtains incremental changes of data by analyzing online logs or archive logs of a source database, and then applies these changes to a target database, thereby implementing synchronization between the source database and the target database. Creating a target table for storing business data by using the configured OGG, wherein the target table can store policy data of sales personnel; a distributed publish-subscribe messaging system (kafka topic) is created, a kafka server is started, and a kafka integrated real-time computing framework spark is used.
Step S50, when detecting the operation of the user on the service data in the target table, the distributed publish-subscribe message system determines the statistical operation type of the service data based on a preset calculation rule. Specifically, when detecting that a user operates the service data in the target table, the distributed publish-subscribe message system kafka determines whether the service data before and after operating the service data meet a preset statistical condition; converting the state of the business data meeting the statistical conditions before and after operation into an operation state bit according to a preset first rule; and determining the statistical operation type corresponding to the operation state bit obtained by conversion according to the mapping relation between the predetermined operation state bit and the statistical operation type. Further, the distributed publish-subscribe message system kafka may further determine the statistical operation type when the statistical condition includes a plurality of judgment conditions for mutual and relationship, a plurality of judgment conditions for mutual or relationship, or a plurality of judgment conditions for mutual or relationship and judgment conditions for relation, and the specific process is as described in the above embodiment and is not described herein again.
For example, in one embodiment, the array corresponding to the four operations is returned according to the number of judgment conditions for calculating the statistical index and the number of and or conditions, and it is assumed that the arrays 1,2, 3 and 4 store the decimal numbers corresponding to no operation, simple addition and subtraction, subtraction and addition operations, respectively.
After a salesman operates on a piece of sales data, kafka consumes the data, judges the judgment conditions contained in the data and the data at the previous moment to determine the state bit to which the data belongs, and according to a mapping function:
y＝a0β2n-1+a1β2n-2+…+a2n-2β+a2n-1
the corresponding decimal value is calculated, and then the operation corresponding to the moment can be obtained by judging which one of array1, array2, array3 and array4 the decimal value belongs to.
And step S60, the real-time computing frame spark performs statistical operation on the service data in the target table according to the determined statistical operation type, obtains the real-time statistical data of the service data in the target table, and displays the real-time statistical data to the user.
In the embodiment, the OGG, the kafka and the spark are combined to form a real-time system; kafka is a distributed publish/subscribe based messaging system, Spark is a real-time computing framework, and the two are integrated to process business data. The kafka converts the satisfied state of the service data to the preset statistical condition before and after the operation into the corresponding operation state bit to determine the corresponding statistical operation type, so that a multi-layer nested complex logic code is not required to be written, the index calculation complexity is reduced, the operation is simpler, and the real-time statistics and display of the service data are realized. The performance of a source end database is not influenced, and the requirement of a user for checking various service indexes in real time can be met.
Further, in other embodiments, the method further comprises:
integrating an open source database by using the distributed publish-subscribe message system, wherein the open source database is used for storing historical data with the same service data identifier as a main key;
setting a table for recording the statistical times of the same service data identifier in the open source database;
if different types of indexes in the service data of the same service data identifier are recorded, maintaining the counting times of the same service data identifier unchanged; and if one piece of service data of the same service data identifier is deleted, subtracting 1 from the number of times of counting the same service data identifier.
In this embodiment, a kafka integrated open source database Hbase is used, and the open source database Hbase is used to store historical data using the same service data identifier as a primary key. For example, in one embodiment, the same policy of the seller has several different policies recorded at different times, and the number of policies for the seller can only be counted once. That is, the number of insurance policies of the salesperson needs to be removed in statistics. If different types of indexes in the same insurance policy of the same salesperson are recorded, keeping the counting times of the number of the insurance policies unchanged; and if the business data of the same insurance policy of the same salesperson is deleted, subtracting 1 from the statistical frequency of the number of the insurance policies of the same salesperson.
The invention further provides a service data processing system.
Referring to fig. 4, fig. 4 is a schematic structural diagram of another embodiment of the service data processing system of the present invention.
In another embodiment, the business data processing system comprises: data synchronization tool 04(Oracle Golden Gate, OGG for short), distributed publish-subscribe messaging system (kafka)05, and real-time computing framework (spark) 06. In this embodiment, a preset data synchronization tool is configured, and the Oracle Golden Gate software is log-based structured data replication backup software, which obtains incremental changes of data by analyzing online logs or archived logs of a source database and applies the incremental changes to a target database, thereby synchronizing the source database and the target database. Creating a target table for storing business data by using the configured OGG, wherein the target table can store policy data of sales personnel; a distributed publish-subscribe messaging system (kafka topic) is created, a kafka server is started, and a kafka integrated real-time computing framework spark is used.
And when the operation of the user on the service data in the target table is detected, the distributed publish-subscribe message system determines the statistical operation type of the service data based on a preset calculation rule. Specifically, when detecting that a user operates the service data in the target table, the distributed publish-subscribe message system kafka determines whether the service data before and after operating the service data meet a preset statistical condition; converting the state of the business data meeting the statistical conditions before and after operation into an operation state bit according to a preset first rule; and determining the statistical operation type corresponding to the operation state bit obtained by conversion according to the mapping relation between the predetermined operation state bit and the statistical operation type. Further, the distributed publish-subscribe message system kafka may further determine the statistical operation type when the statistical condition includes a plurality of judgment conditions for mutual and relationship, a plurality of judgment conditions for mutual or relationship, or a plurality of judgment conditions for mutual or relationship and judgment conditions for relation, and the specific process is as described in the above embodiment and is not described herein again.
For example, in one embodiment, the array corresponding to the four operations is returned according to the number of judgment conditions for calculating the statistical index and the number of and or conditions, and it is assumed that the arrays 1,2, 3 and 4 store the decimal numbers corresponding to no operation, simple addition and subtraction, subtraction and addition operations, respectively.
After a salesman operates on a piece of sales data, kafka consumes the data, judges the judgment conditions contained in the data and the data at the previous moment to determine the state bit to which the data belongs, and according to a mapping function:
y＝a0β2n-1+a1β2n-2+…+a2n-2β+a2n-1
the corresponding decimal value is calculated, and then the operation corresponding to the moment can be obtained by judging which one of array1, array2, array3 and array4 the decimal value belongs to.
And the real-time computing frame spark performs statistical operation on the service data in the target table according to the determined statistical operation type, acquires the real-time statistical data of the service data in the target table, and displays the real-time statistical data to a user.
In the embodiment, the OGG, the kafka and the spark are combined to form a real-time system; kafka is a distributed publish/subscribe based messaging system, Spark is a real-time computing framework, and the two are integrated to process business data. The kafka converts the satisfied state of the service data to the preset statistical condition before and after the operation into the corresponding operation state bit to determine the corresponding statistical operation type, so that a multi-layer nested complex logic code is not required to be written, the index calculation complexity is reduced, the operation is simpler, and the real-time statistics and display of the service data are realized. The performance of a source end database is not influenced, and the requirement of a user for checking various service indexes in real time can be met.
Further, in other embodiments, the distributed publish-subscribe message system is further configured to integrate a source database; the open source database is used for storing historical data with the same service data identifier as a main key; setting a table for recording the statistical times of the same service data identifier in the open source database; if different types of indexes in the service data of the same service data identifier are recorded, maintaining the counting times of the same service data identifier unchanged; and if one piece of service data of the same service data identifier is deleted, subtracting 1 from the number of times of counting the same service data identifier.
In this embodiment, a kafka integrated open source database Hbase is used, and the open source database Hbase is used to store historical data using the same service data identifier as a primary key. For example, in one embodiment, the same policy of the seller has several different policies recorded at different times, and the number of policies for the seller can only be counted once. That is, the number of insurance policies of the salesperson needs to be removed in statistics. If different types of indexes in the same insurance policy of the same salesperson are recorded, keeping the counting times of the number of the insurance policies unchanged; and if the business data of the same insurance policy of the same salesperson is deleted, subtracting 1 from the statistical frequency of the number of the insurance policies of the same salesperson.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (such as a mobile phone, a computer, a server, an air conditioner, or a network device) to execute the method according to the embodiments of the present invention.
The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, and are not to be construed as limiting the scope of the invention. The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments. Additionally, while a logical order is shown in the flow diagrams, in some cases, the steps shown or described may be performed in an order different than here.
Those skilled in the art can implement the invention in various modifications, such as features from one embodiment can be used in another embodiment to yield yet a further embodiment, without departing from the scope and spirit of the invention. Any modification, equivalent replacement and improvement made within the technical idea of using the present invention should be within the scope of the right of the present invention.

Claims (10)

1. A business data processing system, comprising:
the determining module is used for determining whether the service data meet preset statistical conditions before and after the service data are operated when the operation of a user on the service data in the database is detected;
the conversion module is used for converting the state of the business data meeting the statistical conditions before and after operation into an operation state bit according to a preset first rule;
the statistical module is used for determining the statistical operation type corresponding to the operation state bit obtained by conversion according to the mapping relation between the predetermined operation state bit and the statistical operation type, and performing statistical operation on the database according to the determined statistical operation type;
the state of the service data before and after the operation, which satisfies the statistical condition, includes a state of satisfying, not satisfying, or not changing, and the conversion module is further configured to:
converting the satisfied state of the statistical condition of the service data before operation into a ternary first state bit according to a preset rule; and converting the satisfied state of the statistical conditions of the service data after operation into a ternary second state bit according to a preset rule, and combining the first state bit and the second state bit to obtain a ternary operation state bit.
2. The business data processing system of claim 1, wherein if the statistical condition comprises a plurality of judgment conditions for an and relationship, the determining module is further configured to:
when the operation of a user on the service data in the database is detected, determining whether the service data before and after the operation on the service data meet a plurality of preset judgment conditions;
the conversion module is further configured to:
and converting a plurality of satisfied states corresponding to a plurality of judgment conditions before and after the operation of the service data into a plurality of state bits according to a preset first rule, and combining the plurality of state bits to obtain the operation state bit.
3. The business data processing system of claim 1, wherein if the statistical condition comprises a plurality of judgment conditions that are or relationships to each other, the determining module is further configured to:
determining whether the service data meet each preset judgment condition before and after the operation on the service data;
the conversion module is further configured to:
obtaining each corresponding statistical operation type according to the change of the state meeting of each judgment condition before and after the operation of the service data, converting each statistical operation type into a type state bit according to a preset second rule, and combining a plurality of type state bits to obtain a type combination state bit;
the statistics module is further configured to:
and determining the obtained statistical operation type corresponding to the type combination state bit according to the mapping relation between the predetermined type combination state bit and the statistical operation type.
4. The business data processing system of claim 1, wherein if the statistical condition comprises a plurality of determination conditions including an or relationship and an and relationship therebetween, the determining module is further configured to:
determining whether the service data meet a plurality of preset first judgment conditions before and after the service data are operated, wherein the first judgment conditions are judgment conditions of mutual and relational relationship in the statistical conditions; determining whether the service data meet each preset second judgment condition before and after the operation is performed on the service data, wherein the second judgment condition is a judgment condition of mutual or relationship in the statistical conditions;
the conversion module is further configured to:
obtaining corresponding statistical operation types according to the change of the satisfied states of the first judgment conditions before and after the operation of the service data, obtaining each corresponding statistical operation type according to the change of the satisfied states of the second judgment conditions before and after the operation of the service data, converting the obtained statistical operation types into type state bits according to a preset second rule, and combining the type state bits to obtain a type combination state bit;
the statistics module is further configured to:
and determining the obtained statistical operation type corresponding to the type combination state bit according to the mapping relation between the predetermined type combination state bit and the statistical operation type.
5. A method for processing service data, the method comprising the steps of:
A. when the operation of a user on the service data in the database is detected, determining whether the service data before and after the operation on the service data meet a preset statistical condition;
B. converting the state of the business data meeting the statistical conditions before and after operation into an operation state bit according to a preset first rule;
C. determining a statistical operation type corresponding to the operation state bit obtained by conversion according to a predetermined mapping relation between the operation state bit and the statistical operation type, and performing statistical operation on the database according to the determined statistical operation type;
the state of the service data before and after the operation, which satisfies the statistical condition, includes a state of satisfying, not satisfying, or not changing, and the step B includes:
and converting the satisfied state of the statistical condition of the service data before operation into a ternary first state bit according to a preset rule, converting the satisfied state of the statistical condition of the service data after operation into a ternary second state bit according to a preset rule, and combining the first state bit and the second state bit to obtain a ternary operation state bit.
6. The service data processing method according to claim 5, wherein if the statistical condition includes a plurality of judgment conditions that are related to each other, the step a is replaced with:
when the operation of a user on the service data in the database is detected, determining whether the service data before and after the operation on the service data meet a plurality of preset judgment conditions;
the step B is replaced by the following steps:
and converting a plurality of satisfied states corresponding to a plurality of judgment conditions before and after the operation of the service data into a plurality of state bits according to a preset first rule, and combining the plurality of state bits to obtain the operation state bit.
7. The service data processing method according to claim 5, wherein if the statistical condition includes a plurality of judgment conditions that are or-related to each other, the step a is replaced with:
determining whether the service data meet each preset judgment condition before and after the operation on the service data;
the step B is replaced by the following steps:
obtaining each corresponding statistical operation type according to the change of the state meeting of each judgment condition before and after the operation of the service data, converting each statistical operation type into a type state bit according to a preset second rule, and combining a plurality of type state bits to obtain a type combination state bit;
the step C is replaced by the following steps:
and determining the obtained statistical operation type corresponding to the type combination state bit according to the mapping relation between the predetermined type combination state bit and the statistical operation type.
8. The business data processing method according to claim 5, wherein if the statistical condition includes a plurality of determination conditions, and the plurality of determination conditions include an or relationship and an and relationship, the step A is replaced with:
determining whether the service data meet a plurality of preset first judgment conditions before and after the service data are operated, wherein the first judgment conditions are judgment conditions of mutual and relation in the statistical conditions, and determining whether the service data meet each preset second judgment condition before and after the service data are operated, and the second judgment conditions are judgment conditions of mutual or relation in the statistical conditions;
the step B is replaced by the following steps:
obtaining corresponding statistical operation types according to the change of the satisfied states of the first judgment conditions before and after the operation of the service data, obtaining each corresponding statistical operation type according to the change of the satisfied states of the second judgment conditions before and after the operation of the service data, converting the obtained statistical operation types into type state bits according to a preset second rule, and combining the type state bits to obtain a type combination state bit;
the step C is replaced by the following steps:
and determining the obtained statistical operation type corresponding to the type combination state bit according to the mapping relation between the predetermined type combination state bit and the statistical operation type.
9. A service data processing method based on the service data processing system according to any one of claims 1 to 4, characterized in that the method comprises the following steps:
configuring a preset data synchronization tool, and creating a target table for storing business data by using the data synchronization tool; creating a distributed publishing and subscribing message system, and integrating a real-time computing frame by using the distributed publishing and subscribing message system;
when the operation of the user on the service data in the target table is detected, the distributed publishing and subscribing message system determines the statistical operation type of the service data based on a preset calculation rule;
and the real-time computing frame carries out statistical operation on the service data in the target table according to the determined statistical operation type, acquires the real-time statistical data of the service data in the target table and displays the real-time statistical data to a user.
10. The traffic data processing method according to claim 9, characterized in that the method further comprises:
integrating an open source database by using the distributed publish-subscribe message system, wherein the open source database is used for storing historical data with the same service data identifier as a main key;
setting a table for recording the statistical times of the same service data identifier in the open source database;
if different types of indexes in the service data of the same service data identifier are recorded, maintaining the counting times of the same service data identifier unchanged; and if one piece of service data of the same service data identifier is deleted, subtracting 1 from the number of times of counting the same service data identifier.
CN201611005030.7A 2016-11-15 2016-11-15 Service data processing system and method Active CN108073624B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201611005030.7A CN108073624B (en) 2016-11-15 2016-11-15 Service data processing system and method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201611005030.7A CN108073624B (en) 2016-11-15 2016-11-15 Service data processing system and method

Publications (2)

Publication Number Publication Date
CN108073624A CN108073624A (en) 2018-05-25
CN108073624B true CN108073624B (en) 2020-11-24

Family

ID=62162811

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201611005030.7A Active CN108073624B (en) 2016-11-15 2016-11-15 Service data processing system and method

Country Status (1)

Country Link
CN (1) CN108073624B (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109241185B (en) * 2018-08-27 2021-03-30 武汉达梦数据库股份有限公司 Data synchronization method and data synchronization device
CN110555028A (en) * 2019-08-22 2019-12-10 上海数禾信息科技有限公司 data display method and device

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101546395A (en) * 2008-03-24 2009-09-30 国际商业机器公司 Method and equipment for generating service process model and method for identifying status field
CN101639842A (en) * 2008-07-29 2010-02-03 阿里巴巴集团控股有限公司 Characteristics data statistic method and system of members
CN102129619A (en) * 2011-03-08 2011-07-20 国网信息通信有限公司 Service data processing method and system based on cloud storage
US20150355847A1 (en) * 2014-06-10 2015-12-10 International Business Machines Corporation Transfer size monitor, determination, and optimization engine for storage devices

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101546395A (en) * 2008-03-24 2009-09-30 国际商业机器公司 Method and equipment for generating service process model and method for identifying status field
CN101639842A (en) * 2008-07-29 2010-02-03 阿里巴巴集团控股有限公司 Characteristics data statistic method and system of members
CN102129619A (en) * 2011-03-08 2011-07-20 国网信息通信有限公司 Service data processing method and system based on cloud storage
US20150355847A1 (en) * 2014-06-10 2015-12-10 International Business Machines Corporation Transfer size monitor, determination, and optimization engine for storage devices

Also Published As

Publication number Publication date
CN108073624A (en) 2018-05-25

Similar Documents

Publication Publication Date Title
CN112699175A (en) Data management system and method thereof
US20120072464A1 (en) Systems and methods for master data management using record and field based rules
US8645332B1 (en) Systems and methods for capturing data refinement actions based on visualized search of information
KR20080002941A (en) Adaptive data cleaning
US20120254254A1 (en) Directed Graph Transitive Closure
CN112364004A (en) Policy data processing method and device based on data warehouse and storage medium
CN112527886A (en) Data warehouse system based on urban brain
US20190073394A1 (en) Automatically aggregating data in database tables
CN108415990B (en) Data quality monitoring method and device, computer equipment and storage medium
CN116719799A (en) Environment-friendly data management method, device, computer equipment and storage medium
CN108073624B (en) Service data processing system and method
CN114416703A (en) Method, device, equipment and medium for automatically monitoring data integrity
CN116483822B (en) Service data early warning method, device, computer equipment and storage medium
CN111611230A (en) Method and device for establishing main data system, computer equipment and storage medium
CN111125045A (en) Lightweight ETL processing platform
US10664501B2 (en) Deriving and interpreting users collective data asset use across analytic software systems
CN114925919A (en) Service resource processing method and device, computer equipment and storage medium
CN114860851A (en) Data processing method, device, equipment and storage medium
CN113656445A (en) Data processing method and device, electronic equipment and storage medium
US9489438B2 (en) Systems and methods for visualizing master data services information
KR20220115859A (en) Edge table representation of the process
US20150100515A1 (en) Customer data unification
CN111949743A (en) Method, device and equipment for acquiring network operation data
EP3759665A1 (en) Multi-dimensional organization of data for efficient analysis
CN111026763A (en) Data processing method, device, equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant