CN117493460A - Data processing method, device and equipment applied to long-protection fund - Google Patents

Data processing method, device and equipment applied to long-protection fund Download PDF

Info

Publication number
CN117493460A
CN117493460A CN202311524468.6A CN202311524468A CN117493460A CN 117493460 A CN117493460 A CN 117493460A CN 202311524468 A CN202311524468 A CN 202311524468A CN 117493460 A CN117493460 A CN 117493460A
Authority
CN
China
Prior art keywords
data
processed
ignite
auditing
cluster
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311524468.6A
Other languages
Chinese (zh)
Inventor
逯燕芳
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Taikang Insurance Group Co Ltd
Taikang Pension Insurance Co Ltd
Original Assignee
Taikang Insurance Group Co Ltd
Taikang Pension Insurance Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Taikang Insurance Group Co Ltd, Taikang Pension Insurance Co Ltd filed Critical Taikang Insurance Group Co Ltd
Priority to CN202311524468.6A priority Critical patent/CN117493460A/en
Publication of CN117493460A publication Critical patent/CN117493460A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22Indexing; Data structures therefor; Storage structures
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00Administration; Management
    • G06Q10/10Office automation; Time management
    • G06Q10/103Workflow collaboration or project management
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q40/00Finance; Insurance; Tax strategies; Processing of corporate or income taxes
    • G06Q40/06Asset management; Financial planning or analysis

Landscapes

  • Engineering & Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Theoretical Computer Science (AREA)
  • Strategic Management (AREA)
  • Human Resources & Organizations (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • General Business, Economics & Management (AREA)
  • General Engineering & Computer Science (AREA)
  • Finance (AREA)
  • Economics (AREA)
  • Marketing (AREA)
  • Operations Research (AREA)
  • Development Economics (AREA)
  • Accounting & Taxation (AREA)
  • Computing Systems (AREA)
  • Tourism & Hospitality (AREA)
  • Quality & Reliability (AREA)
  • Software Systems (AREA)
  • Game Theory and Decision Science (AREA)
  • Technology Law (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

The application provides a data processing method, device and equipment applied to long-protection funds, which can be used in the technical field of big data. The method is applied to an ignite data grid, wherein the ignite data grid is deployed in an ignite cluster, and comprises the following steps: acquiring data to be processed, and storing the data to be processed into a data storage module of each node included in the ignite cluster; responding to a data auditing instruction, and determining data to be audited from data storage modules of all nodes included in the ignite cluster; based on a preset fund settlement list template, carrying out standardized processing on the data to be checked to obtain standardized data to be checked; and sending the standardized data to be checked to a data checking module so that the data checking module can check the standardized data to be checked to obtain a data checking result. The method saves the dependence of hardware resources, simplifies the data processing flow and improves the efficiency and the performance of data processing.

Description

Data processing method, device and equipment applied to long-protection fund
Technical Field
The application relates to the technical field of big data, in particular to a data processing method, device and equipment applied to long-protection funds.
Background
The existing long-protection fund intelligent auditing system stores the data to be processed into a data platform after acquiring the data to be processed, extracts the data to be audited, which needs to be audited, from the data platform, and writes the data to be audited into a standard relational database. At this time, the data to be processed in the standard relational database can be operated to obtain standardized data to be audited. And then, based on the auditing module, auditing the standardized data to be audited to obtain a data auditing result.
On one hand, the data processing mode not only consumes too much hardware resources, but also depends on the operation capability of a data platform or a relational database, so that the requirement on hardware resources is too high. On the other hand, the data processing mode needs to process the data to be processed for multiple times, the processing flow is complex, errors are easy to occur, and the data auditing efficiency is also influenced.
Disclosure of Invention
The application provides a data processing method, device and equipment applied to long-protection funds, which are used for solving the technical problems of more hardware resource dependence, complex processing flow and lower data processing efficiency when auditing and processing the fund settlement data in the prior art.
In a first aspect, the present application provides a data processing method applied to a long-guard fund, the method being applied to an igite data grid, the igite data grid being deployed in an igite cluster, the method comprising:
acquiring data to be processed, and storing the data to be processed into a data storage module of each node included in the ignite cluster; the data to be processed represents fund settlement related data of the long-protection fund; the data storage modules included by the nodes in the ignite cluster correspond to a plurality of data storage levels;
responding to a data auditing instruction, and determining data to be audited from data storage modules of all nodes included in the ignite cluster; the data auditing instruction is used for auditing the data to be processed; the data to be checked represents the data to be processed which needs to be checked;
based on a preset fund settlement list template, carrying out standardized processing on the data to be checked to obtain standardized data to be checked; the preset fund settlement list template is used for determining the representation format of the data to be audited;
transmitting the standardized data to be checked to a data checking module so that the data checking module can check the standardized data to be checked to obtain a data checking result; the data auditing module is used for auditing the standardized data to be audited; and the data auditing result is used for representing whether the data to be audited passes or not.
In one example, storing the data to be processed in a data storage module of each node included in the ignite cluster includes:
and uniformly storing the data to be processed into the data storage modules of which the data storage levels are magnetic disks in each node included in the ignite cluster according to a stream processing mode.
In one example, uniformly storing the data to be processed into a data storage module with a data storage level being a disk in each node included in the ignite cluster, where the data storage module includes:
storing the data to be processed with the association relation in a data storage module under the same node in the ignite cluster; the data to be processed with the association relation represents the data to be processed with the same main key information.
In one example, the determining, in response to the data auditing instruction, data to be audited from the data storage modules of the nodes included in the ignite cluster includes:
responding to the data auditing instruction, and determining to-be-processed data matched with the data auditing instruction from a data storage module with a data storage hierarchy of a disk in each node included in the ignite cluster;
And loading the matched data to be processed into a data storage module of which the data storage level is a cache in the ignite cluster to obtain the data to be checked.
In one example, the determining, in response to the data auditing instruction, data to be audited from the data storage modules of the nodes included in the ignite cluster includes:
responding to a data auditing instruction, and determining at least one primary key information matched with the data auditing instruction;
and determining the data to be checked from the data storage modules of all the nodes included in the ignite cluster based on the at least one primary key information.
In one example, the normalizing the pending data based on a preset fund settlement manifest template to obtain normalized pending data includes:
and determining an SQL instruction corresponding to the preset fund settlement list template, and carrying out standardized processing on the data to be checked based on the SQL instruction to obtain the standardized data to be checked.
In one example, the storing the data to be processed in the data storage module of each node included in the ignite cluster includes:
Determining data attribute information of the data to be processed; wherein the data attribute information characterizes the data amount of the data to be processed and/or the association degree between the data to be processed;
determining a data storage mode matched with the data attribute information; the data storage mode is a partition storage mode or at least one of a copy storage mode;
and storing the data to be processed into a data storage module of each node included in the ignite cluster based on the data storage mode.
In a second aspect, the present application provides a data processing apparatus for use with a long-guard fund, comprising:
the data storage unit is used for acquiring data to be processed and storing the data to be processed into the data storage modules of all nodes included in the ignite cluster; the data to be processed represents fund settlement related data of the long-protection fund; the data storage modules included by the nodes in the ignite cluster correspond to a plurality of data storage levels;
the data determining unit is used for responding to a data auditing instruction and determining data to be audited from the data storage modules of all nodes included in the ignite cluster; the data auditing instruction is used for auditing the data to be processed; the data to be checked represents the data to be processed which needs to be checked;
The standardized unit is used for carrying out standardized processing on the data to be checked based on a preset fund settlement list template to obtain standardized data to be checked; the preset fund settlement list template is used for determining the representation format of the data to be audited;
the auditing processing unit is used for sending the standardized data to be audited to the data auditing module so that the data auditing module can audit the standardized data to obtain a data auditing result; the data auditing module is used for auditing the standardized data to be audited; and the data auditing result is used for representing whether the data to be audited passes or not.
In one example, a data storage unit is configured to:
and uniformly storing the data to be processed into the data storage modules of which the data storage levels are magnetic disks in each node included in the ignite cluster according to a stream processing mode.
In one example, a data storage unit is configured to:
storing the data to be processed with the association relation in a data storage module under the same node in the ignite cluster; the data to be processed with the association relation represents the data to be processed with the same main key information.
In one example, the data determining unit is configured to:
responding to the data auditing instruction, and determining to-be-processed data matched with the data auditing instruction from a data storage module with a data storage hierarchy of a disk in each node included in the ignite cluster;
and loading the matched data to be processed into a data storage module of which the data storage level is a cache in the ignite cluster to obtain the data to be checked.
In one example, the data determining unit is configured to:
responding to a data auditing instruction, and determining at least one primary key information matched with the data auditing instruction;
and determining the data to be checked from the data storage modules of all the nodes included in the ignite cluster based on the at least one primary key information.
In one example, the normalization unit is configured to:
and determining an SQL instruction corresponding to the preset fund settlement list template, and carrying out standardized processing on the data to be checked based on the SQL instruction to obtain the standardized data to be checked.
In one example, a data storage unit is configured to:
determining data attribute information of the data to be processed; wherein the data attribute information characterizes the data amount of the data to be processed and/or the association degree between the data to be processed;
Determining a data storage mode matched with the data attribute information; the data storage mode is a partition storage mode or at least one of a copy storage mode;
and storing the data to be processed into a data storage module of each node included in the ignite cluster based on the data storage mode.
In a third aspect, the present application provides a computer device comprising: a processor, and a memory communicatively coupled to the processor;
the memory stores computer-executable instructions;
the processor executes computer-executable instructions stored in the memory to implement the method of the first aspect.
In a fourth aspect, the present application provides a computer-readable storage medium having stored therein computer-executable instructions for performing the method of the first aspect when executed by a processor.
In a fifth aspect, the present application provides a computer program product comprising: computer-executable instructions stored in a readable storage medium from which at least one processor of a computer device can read, the at least one processor executing the computer-executable instructions causing the computer device to perform the method of the first aspect.
The data processing method, the device and the equipment applied to the long-protection foundation can store the data to be processed into the data storage modules of all the nodes included in the ignite cluster after the data to be processed are acquired, and at the moment, the data to be processed can be stored in a distributed mode through the ignite cluster, so that the hardware resource cost is saved. And then, the data to be audited can be determined from the data storage modules of all the nodes included in the ignite cluster in response to the data auditing instruction, and the standardized processing is carried out on the data to be audited based on a preset fund settlement list template to obtain standardized data to be audited. In summary, according to the embodiment of the application, the data processing function and the data storage function can be considered through the ignite cluster, so that the input and output times of data are reduced, the probability of data processing errors is further reduced, and the running stability of the data auditing system is improved.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application.
Fig. 1 is a schematic flow chart of data processing on a long-protection fund in the prior art according to an embodiment of the present application;
fig. 2 is a flow chart of a data processing method applied to long-protection fund according to an embodiment of the present application;
FIG. 3 is a flowchart illustrating another data processing method applied to long-term fund according to an embodiment of the present disclosure;
FIG. 4 is a schematic flow chart of data processing according to an embodiment of the present disclosure;
fig. 5 is a flowchart of an application scenario of a data processing method applied to a long-protection fund according to an embodiment of the present application;
fig. 6 is a schematic diagram of hardware resource consumption comparison of a data processing method applied to long-protection funds according to an embodiment of the present application with a data processing method in the prior art when data processing is performed on data to be processed;
FIG. 7 is a schematic diagram of a data processing device for long-term fund according to an embodiment of the present disclosure;
FIG. 8 is a schematic diagram of another data processing apparatus for long-term fund according to an embodiment of the present disclosure;
fig. 9 is a schematic structural diagram of a computer device according to an embodiment of the present application;
FIG. 10 is a block diagram of a computer device, according to an example embodiment.
Specific embodiments thereof have been shown by way of example in the drawings and will herein be described in more detail. These drawings and the written description are not intended to limit the scope of the inventive concepts in any way, but to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments.
Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present application as detailed in the accompanying claims.
The term "and/or" is used herein to describe only one relationship, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist together, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, C, and may mean including any one or more elements selected from the group consisting of A, B and C.
It should be noted that, the user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) referred to in the present application are information and data authorized by the user or fully authorized by each party, and the collection, use and processing of the related data need to comply with the related laws and regulations and standards, and provide corresponding operation entries for the user to select authorization or rejection.
It should be noted that the data processing method, device and equipment applied to long-protection funds provided in the embodiments of the present application may be used in the technical field of big data, or may be used in the technical field of finance, or may be used in other related fields than the above-mentioned technology, where the application fields of the data processing method, device and equipment applied to long-protection funds of the present application are not limited.
The existing intelligent audit system for funds mainly adopts the following two design modes.
Mode one: the audit engine topic data (i.e., the data to be processed) is extracted from the nursing foundation database by way of ETL (Extract-Transform-Load) and placed in the relational database. And then, in the process of auditing, reading the data to be audited from the relational database, and sending the data to an auditing engine for auditing.
In this embodiment, when the data volume of the data to be audited is large, the data needs to be queried in a correlated way, so that the requirement on the relational database is high, the query time is too long, and the performance of the auditing process is poor. Moreover, under different software platforms, the performance of many relational databases cannot reach the standard, so that the auditing performance of the data to be audited cannot meet the actual use requirement.
Mode two: and extracting the subject data of the auditing engine (namely, the data to be processed) from the nursing foundation database in an ETL mode, and placing the data to be processed into a large data platform. The large data platform can then be used for further processing of the data. At this time, in the process of auditing, required data can be extracted from the big data platform to perform auditing processing.
In the implementation mode, more middle platform components are needed to be relied on, and the requirements on the existing basic software platform and hardware resources are high.
In addition, when the data processing is performed on the data related to the fund settlement in the two modes, the data to be processed needs to be processed for multiple times, so that program abnormality phenomenon caused by data processing errors is easy to occur in the actual application process, the problem is difficult to be checked, and a large amount of time cost and labor cost are consumed for processing and solving once the data problem occurs.
In the prior art, according to the two design manners, a data processing manner of the long-protection fund is designed, and fig. 1 is an exemplary schematic flow chart of data processing of the long-protection fund in the prior art according to the embodiment of the present application. As shown in fig. 1, the flow may be specifically described as the following process:
first, visit and settlement data for long care funds are extracted from a care fund database into a data platform standard table. The data platform standard table can be a large data platform or a heavy relational database.
Next, using ETL, business data (i.e., data to be audited) required by the audit engine is extracted from the data platform standard table and written into the audit engine subject library.
The business data required by the auditing engine can be fund settlement data, and the auditing engine subject database can be a standard relational database.
And then, extracting service data from the audit engine theme database, and encapsulating the service data into a data format processed by a standard audit engine after operation processing, namely, engine standard input, namely, standardized data to be audited.
Finally, the auditing engine reads standardized data to be audited, performs auditing processing, and outputs a data auditing result.
In summary, the data processing method in the prior art not only consumes too many hardware resources, but also depends on the computing capability of the data platform or the relational database, which results in too high requirements on hardware resources. On the other hand, the data processing mode needs to process the data to be processed for multiple times, the processing flow is complex, errors are easy to occur, and the data auditing efficiency is also influenced.
The data processing method applied to the long-protection fund aims at solving the technical problems in the prior art.
The following describes the technical solutions of the present application and how the technical solutions of the present application solve the above technical problems in detail with specific embodiments. The following embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.
Fig. 2 is a flow chart of a data processing method applied to long-guard fund according to an embodiment of the present application, where the method may be applied to an igite data grid, and the igite data grid is deployed in an igite cluster.
As shown in fig. 2, the method includes:
S201, obtaining data to be processed, and storing the data to be processed into a data storage module of each node included in the ignite cluster.
The data to be processed represents fund settlement related data of the long-protection fund; the nodes in the ignite cluster include data storage modules corresponding to a plurality of data storage tiers.
In one example, the data to be processed may include at least the following data, fund settlement data, and fund visit data. At this time, the data to be processed may be obtained from the nursing-fund database.
In one example, the number of nodes included in the ignite cluster may be 3 or 5, and the number of nodes included in the ignite cluster is not specifically limited herein, so as to meet the actual needs.
In one example, the data storage modules may include data storage modules with data storage tiers that are disks, and data storage modules with data storage tiers that are caches.
S202, responding to a data auditing instruction, and determining data to be audited from data storage modules of all nodes included in the ignite cluster.
The data auditing instruction is used for auditing the data to be processed; the data to be audited characterizes the data to be processed which needs to be audited.
In one example, the data to be audited may be understood as settlement data, e.g., the data to be audited may be fund settlement data in the data to be processed.
S203, based on a preset fund settlement list template, standardized processing is carried out on the data to be audited, and standardized data to be audited is obtained.
The preset fund settlement list template is used for determining the representation format of the data to be audited.
In one example, a preset fund settlement inventory template may be used to determine the data format of the data to be audited as a data format that matches the input data of the data auditing module.
In one example, the data to be audited may be normalized by a preset fund statement template to obtain normalized data to be audited in a data format that matches the data auditing module.
S204, the standardized data to be checked is sent to a data checking module, so that the data checking module performs checking processing on the standardized data to be checked, and a data checking result is obtained.
The data auditing module is used for auditing standardized data to be audited; the data auditing result is used for representing whether the data to be audited pass the auditing.
In one example, the data auditing results may characterize the wind control results of the data to be audited, e.g., the data auditing results may be passed for data auditing or may be failed for data auditing, wherein the data auditing may characterize the passing of the data to be audited wind control and the data auditing may characterize the failing of the data to be audited wind control.
As can be seen from the above description, in the embodiment of the present application, after the data to be processed is acquired, the data to be processed may be stored in the data storage modules of each node included in the ignite cluster, and at this time, the data to be processed may be stored in a distributed manner through the ignite cluster, so that the hardware resource cost is saved. And then, the data to be audited can be determined from the data storage modules of all the nodes included in the ignite cluster in response to the data auditing instruction, and the standardized processing is carried out on the data to be audited based on a preset fund settlement list template to obtain standardized data to be audited. In summary, according to the embodiment of the application, the data processing function and the data storage function can be considered through the ignite cluster, so that the input and output times of data are reduced, the probability of data processing errors is further reduced, and the running stability of the data auditing system is improved.
Fig. 3 is a flowchart of another data processing method applied to long-protection funds according to an embodiment of the present application, as shown in fig. 3, the method includes:
s301, acquiring data to be processed.
Wherein the data to be processed characterizes fund settlement related data of the long-guard fund, for example, the data to be processed may include the data fund settlement data described above, and fund visit data.
In one example, the nodes in the ignite cluster include data storage modules corresponding to multiple data storage tiers. At this time, the data to be processed may be stored in the ignite cluster in accordance with the steps described in S302 below.
S302, uniformly storing data to be processed into data storage modules of which the data storage levels are magnetic disks in all nodes included in the ignite cluster according to a stream processing mode.
In one example, the data to be processed can be obtained from the nursing foundation database in a streaming processing mode, and the data to be processed is stored, so that the ordered and effective auditing processing of the data to be processed can be realized.
In one example, the data to be processed can be uniformly stored in the data storage module of which the data storage level is a disk in the ignite cluster, and the implementation mode can effectively maintain the balance relation among all nodes of the ignite cluster, so that the stability of the ignite cluster is ensured.
In one example, for the above-mentioned to-be-processed data, the to-be-processed data is uniformly stored in the data storage modules of which the data storage hierarchy is a disk in each node included in the ignite cluster, and the following process is specifically included.
And storing the data to be processed with the association relation in a data storage module under the same node in the ignite cluster. The data to be processed with the association relation represents the data to be processed with the same main key information.
According to the embodiment, the data to be processed with the association relationship can be stored in the data storage module under the same node, so that the inquiry and acquisition time of the data to be processed can be saved, and the data processing efficiency is improved.
In one example, the data storage module for storing the data to be processed to each node included in the ignite cluster specifically includes the following steps.
Step one, determining data attribute information of data to be processed.
Wherein the data attribute information characterizes the data amount of the data to be processed and/or the association degree between the data to be processed.
And step two, determining a data storage mode matched with the data attribute information.
The data storage mode is a partition storage mode or at least one of the copy storage modes.
In one example, when the data size of the data to be processed is large, it may be determined that the data storage mode matched with the data attribute information is a partition storage mode according to the association relationship between the data to be processed, and at this time, the data to be processed may be stored in the partition in each node. When the data size of the data to be processed is small or the correlation between the data to be processed is strong, the data storage mode matched with the data attribute information can be determined to be the copy storage mode, and at this time, the data to be processed can be copied and stored in each node.
And thirdly, storing the data to be processed into the data storage modules of all the nodes included in the ignite cluster based on the data storage mode.
According to the embodiment, the data to be processed can be stored according to the data attribute information of the data to be processed, so that the speed of acquiring the data to be processed from the data storage module can be improved as much as possible, and further the data processing performance is improved.
In one example, after the data to be processed is stored in the ignite cluster, the data to be audited may be determined from the data storage modules of the nodes included in the ignite cluster according to the steps described in S303 to S304 below.
S303, responding to the data auditing instruction, and determining at least one primary key information matched with the data auditing instruction.
S304, determining data to be audited from the data storage modules of all nodes included in the ignite cluster based on at least one primary key information.
In one example, the data to be audited can be determined from the local data set corresponding to each node included in the ignite cluster according to the juxtaposition mode provided by the ignite data grid, so that a large amount of data can be prevented from moving between networks, and the expansibility and the processing performance of data processing are better.
In the above embodiment, each node included in the ignite cluster can realize the data processing flow without higher performance, so that the resource consumption is greatly saved, and the resource allocation benefit can be maximized.
In one example, the data storage modules included by the nodes in the ignite cluster in the embodiments of the present application correspond to multiple data storage tiers, i.e., disks and caches. At this time, in response to the data auditing instruction, the data to be processed matched with the data auditing instruction can be determined from the data storage modules of which the data storage hierarchy is a disk in each node included in the ignite cluster. And then, the matched data to be processed are loaded into a data storage module of which the data storage level is a cache in the ignite cluster, so that the data to be audited is obtained.
In one example, fig. 4 is a schematic flow chart of a data processing provided in an embodiment of the present application. As shown in fig. 4, the ignit data grid provided in the embodiment of the present application supports a distributed SQL (Structured Query Language ) database, and at this time, the data to be audited may be obtained from each node in the ignit cluster by using a data query manner supported by the distributed SQL (i.e., the SQL shown in fig. 4), that is, the data lacking in the cache may be automatically loaded from the disk, so as to obtain the data to be audited.
In one example, after obtaining the data to be audited, the data to be audited may be normalized based on a preset fund settlement manifest template, to obtain normalized data to be audited, see in particular the steps described in S305 below.
S305, determining an SQL instruction corresponding to a preset fund settlement list template, and carrying out standardized processing on data to be audited based on the SQL instruction to obtain standardized data to be audited.
The preset fund settlement list template is used for determining the representation format of the data to be audited.
According to the embodiment, the standardized data to be checked can be obtained by directly utilizing SQL, so that the complexity of data processing can be reduced, and the probability of data processing errors is further reduced.
S306, sending the standardized data to be checked to a data checking module so that the data checking module can check the standardized data to obtain a data checking result.
The data auditing module is used for auditing standardized data to be audited; the data auditing result is used for representing whether the data to be audited pass the auditing.
In an example, this step may be described in S204 above, and will not be described in detail herein.
Fig. 5 is a flowchart of an application scenario of a data processing method applied to a long-protection fund according to an embodiment of the present application. As shown in fig. 5, the embodiment of the present application may obtain the data to be processed from the nursing foundation database, copy the data to be processed into the ignite cluster (i.e., the data copying process shown in fig. 5), and store the data to be processed into the data storage modules of the nodes in the ignite cluster. The data to be processed may be subjected to data processing to obtain standardized data to be audited, and then the result after the data processing, that is, the standardized data to be audited may be sent to a data auditing module (for example, an auditing engine shown in fig. 5) to perform auditing processing to obtain a data auditing result.
In one example, it is assumed that the data processing method applied to long-care funds provided in the embodiments of the present application is applied to a funds clearing house in a large city with more than 500 ten thousand people, and at this time, an ignite cluster may be deployed for storing and calculating service data extracted from a nursing funds database. At this time, in the flow of data processing (i.e., data auditing) of the business data, it can be described as the following procedure.
(1) Based on the streaming processing of the ignite data grid, the data to be processed is injected into the ignite cluster from the nursing foundation database for storage.
(2) And starting a hierarchical storage function of the ignite data grid, loading data to be processed to a disk layer, and loading data to be checked to a cache layer, so that the ignite cluster can store large-scale fund settlement related data.
(3) And starting an ignite distributed deployment function, deploying 4 nodes, and uniformly distributing the data to be processed to each node according to the primary key information of the service data.
(4) When the auditing engine is started to audit, a simple SQL instruction can be written by utilizing the real-time computing function of the ignite based on the memory, standardized data to be audited corresponding to the data to be audited is calculated, and the standardized data to be audited is transmitted into the auditing engine to audit, so that a data auditing result is obtained.
In an example, fig. 6 is a schematic diagram comparing hardware resource consumption when data is processed by a data processing method applied to long-protection fund and a data processing method in the prior art according to the embodiment of the present application. As shown in fig. 6, in the prior art, the data processing is performed on the data to be processed mainly through the big data platform and the auditing engine subject library, at this time, as can be seen from fig. 6, in the prior art, the hardware resource consumption corresponding to the big data platform is "CPU:16c+; memory: 32GB+ "; the hardware resource consumption corresponding to the auditing engine subject library is' CPU:8c+; memory: 16GB+ ". The application can process through the ignite cluster, and the corresponding hardware resource consumption is "CPU:2c 4; memory: 8gb 4".
In one example, after obtaining standardized data to be checked corresponding to the data to be processed, the standardized data to be checked may be sent to a data checking module to perform checking processing to obtain a data checking result, where, as shown in fig. 6, in the prior art, the hardware resources consumed by the data checking module are "CPU:4C; memory: 16 GB'; in this application, the hardware resources consumed by the data auditing module are "CPU:4C; memory: 8 GB). As is clear from this, in the related art, the total number of hardware resources (i.e., the total) consumed for performing data processing is "28C/64GB", and in the present application, the total number of hardware resources consumed for performing data processing is "12C/40GB". Where "C" represents the number of processor cores.
In summary, the data processing method applied to the long-protection fund provided by the embodiment of the application can reduce dependence on hardware resources, simplify data processing flow and improve data processing performance.
It will be appreciated by those skilled in the art that in the above-described method of the specific embodiments, the written order of steps is not meant to imply a strict order of execution but rather should be construed according to the function and possibly inherent logic of the steps.
Fig. 7 is a schematic structural diagram of a data processing device applied to a long-protection fund according to an embodiment of the present application, and as shown in fig. 7, the data processing device 700 applied to a long-protection fund includes:
the data storage unit 701 is configured to obtain data to be processed, and store the data to be processed in a data storage module of each node included in the ignite cluster; the data to be processed represents fund settlement related data of the long-protection fund; the nodes in the ignite cluster include data storage modules corresponding to a plurality of data storage tiers.
The data determining unit 702 is configured to determine data to be audited from data storage modules of each node included in the ignite cluster in response to a data auditing instruction; the data auditing instruction is used for auditing the data to be processed; the data to be audited characterizes the data to be processed which needs to be audited.
A standardization unit 703, configured to perform standardization processing on data to be audited based on a preset fund settlement list template, so as to obtain standardized data to be audited; the preset fund settlement list template is used for determining the representation format of the data to be audited.
The auditing processing unit 704 is configured to send the standardized data to be audited to the data auditing module, so that the data auditing module performs auditing processing on the standardized data to obtain a data auditing result; the data auditing module is used for auditing standardized data to be audited; the data auditing result is used for representing whether the data to be audited pass the auditing.
Fig. 8 is a schematic structural diagram of another data processing apparatus applied to long-protection funds according to an embodiment of the present application, and as shown in fig. 8, the data processing apparatus 800 applied to long-protection funds includes:
a data storage unit 801, configured to obtain data to be processed, and store the data to be processed in a data storage module of each node included in the ignite cluster; the data to be processed represents fund settlement related data of the long-protection fund; the nodes in the ignite cluster include data storage modules corresponding to a plurality of data storage tiers.
A data determining unit 802, configured to determine data to be audited from data storage modules of each node included in the ignite cluster in response to a data auditing instruction; the data auditing instruction is used for auditing the data to be processed; the data to be audited characterizes the data to be processed which needs to be audited.
A standardization unit 803, configured to perform standardization processing on data to be audited based on a preset fund settlement list template, so as to obtain standardized data to be audited; the preset fund settlement list template is used for determining the representation format of the data to be audited.
The auditing processing unit 804 is configured to send the standardized data to be audited to the data auditing module, so that the data auditing module performs auditing processing on the standardized data to obtain a data auditing result; the data auditing module is used for auditing standardized data to be audited; the data auditing result is used for representing whether the data to be audited pass the auditing.
In one example, the data storage unit 801 is configured to:
and uniformly storing the data to be processed into the data storage modules of which the data storage hierarchy is a disk in each node included in the ignite cluster according to a stream processing mode.
In one example, the data storage unit 801 is configured to:
storing the data to be processed with the association relation in a data storage module under the same node in the ignite cluster; the data to be processed with the association relation represents the data to be processed with the same main key information.
In one example, the data determining unit 802 is configured to:
responding to the data auditing instruction, and determining the data to be processed matched with the data auditing instruction from the data storage modules of which the data storage hierarchy is a disk in all nodes included in the ignite cluster;
and loading the matched data to be processed into a data storage module with a data storage level being a cache in the ignite cluster to obtain the data to be audited.
In one example, the data determining unit 802 is configured to:
responding to the data auditing instruction, and determining at least one primary key information matched with the data auditing instruction;
and determining data to be audited from the data storage modules of all nodes included in the ignite cluster based on at least one primary key information.
In one example, the normalization unit 803 is configured to:
and determining an SQL instruction corresponding to the preset fund settlement list template, and carrying out standardized processing on data to be audited based on the SQL instruction to obtain standardized data to be audited.
In one example, the data storage unit 801 is configured to:
determining data attribute information of data to be processed; the data attribute information characterizes the data quantity of the data to be processed and/or the association degree between the data to be processed;
determining a data storage mode matched with the data attribute information; the data storage mode is a partition storage mode or at least one of the copy storage modes;
based on the data storage mode, the data to be processed is stored in the data storage modules of the nodes included in the ignite cluster.
Fig. 9 is a schematic structural diagram of a computer device according to an embodiment of the present application, and as shown in fig. 9, a computer device 900 includes: memory 901, processor 902.
A memory 901; a memory for storing instructions executable by the processor 902.
Wherein the processor 902 is configured to perform the method as provided by the above embodiments.
The computer device further comprises a receiver 903 and a transmitter 904. The receiver 903 is configured to receive instructions and data transmitted from an external device, and the transmitter 904 is configured to transmit instructions and data to the external device.
Fig. 10 is a block diagram of a computer device, which may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, an exercise device, a personal digital assistant, etc., in accordance with an exemplary embodiment.
The apparatus 1000 may include one or more of the following components: a processing component 1002, a memory 1004, a power component 1006, a multimedia component 1008, an audio component 1010, an input/output (I/O) interface 1012, a sensor component 1014, and a communications component 1016.
The processing component 1002 generally controls overall operation of the apparatus 1000, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 1002 can include one or more processors 1020 to execute instructions to perform all or part of the steps of the methods described above. Further, the processing component 1002 can include one or more modules that facilitate interaction between the processing component 1002 and other components. For example, the processing component 1002 can include a multimedia module to facilitate interaction between the multimedia component 1008 and the processing component 1002.
The memory 1004 is configured to store various types of data to support operations at the apparatus 1000. Examples of such data include instructions for any application or method operating on the device 1000, contact data, phonebook data, messages, pictures, videos, and the like. The memory 1004 may be implemented by any type or combination of volatile or nonvolatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk.
The power supply component 1006 provides power to the various components of the device 1000. The power components 1006 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 1000.
The multimedia component 1008 includes a screen between the device 1000 and the user that provides an output interface. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensor may sense not only the boundary of a touch or sliding action, but also the duration and pressure associated with the touch or sliding operation. In some embodiments, the multimedia assembly 1008 includes a front-facing camera and/or a rear-facing camera. The front camera and/or the rear camera may receive external multimedia data when the apparatus 1000 is in an operation mode, such as a photographing mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have focal length and optical zoom capabilities.
The audio component 1010 is configured to output and/or input audio signals. For example, the audio component 1010 includes a Microphone (MIC) configured to receive external audio signals when the device 1000 is in an operational mode, such as a call mode, a recording mode, and a speech recognition mode. The received audio signals may be further stored in memory 1004 or transmitted via communication component 1016. In some embodiments, the audio component 1010 further comprises a speaker for outputting audio signals.
The I/O interface 1012 provides an interface between the processing assembly 1002 and peripheral interface modules, which may be a keyboard, click wheel, buttons, and the like. These buttons may include, but are not limited to: homepage button, volume button, start button, and lock button.
The sensor assembly 1014 includes one or more sensors for providing status assessment of various aspects of the device 1000. For example, the sensor assembly 1014 may detect an on/off state of the device 1000, a relative positioning of the components, such as a display and keypad of the device 1000, the sensor assembly 1014 may also detect a change in position of the device 1000 or a component of the device 1000, the presence or absence of user contact with the device 1000, an orientation or acceleration/deceleration of the device 1000, and a change in temperature of the device 1000. The sensor assembly 1014 may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact. The sensor assembly 1014 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 1014 can also include an acceleration sensor, a gyroscopic sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
The communication component 1016 is configured to facilitate communication between the apparatus 1000 and other devices, either wired or wireless. The device 1000 may access a wireless network based on a communication standard, such as WiFi,2G or 3G, or a combination thereof. In one exemplary embodiment, the communication component 1016 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component 1016 further includes a Near Field Communication (NFC) module to facilitate short range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra Wideband (UWB) technology, bluetooth (BT) technology, and other technologies.
In an exemplary embodiment, the apparatus 800 may be implemented by one or more Application Specific Integrated Circuits (ASICs), digital Signal Processors (DSPs), digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic elements for executing the methods described above.
The embodiments of the present application also provide a computer readable storage medium having stored thereon computer executable instructions that when executed by a processor perform the steps of the data processing method for long-term funds in the method embodiments described above. Wherein the storage medium may be a volatile or nonvolatile computer readable storage medium.
The embodiment of the present application further provides a computer program product, where the computer program product carries computer execution instructions, and the instructions included in the computer execution instructions may be used to execute the steps of the data processing method applied to the long-protection fund in the method embodiment, and specifically, reference may be made to the method embodiment, and details are not repeated herein.
Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the application pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.
It is to be understood that the present application is not limited to the precise arrangements and instrumentalities shown in the drawings, which have been described above, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the application is limited only by the appended claims.

Claims (10)

1. A data processing method applied to a long-guard fund, characterized in that the method is applied to an igite data grid, and the igite data grid is deployed in an igite cluster; the method comprises the following steps:
Acquiring data to be processed, and storing the data to be processed into a data storage module of each node included in the ignite cluster; the data to be processed represents fund settlement related data of the long-protection fund; the data storage modules included by the nodes in the ignite cluster correspond to a plurality of data storage levels;
responding to a data auditing instruction, and determining data to be audited from data storage modules of all nodes included in the ignite cluster; the data auditing instruction is used for auditing the data to be processed; the data to be checked represents the data to be processed which needs to be checked;
based on a preset fund settlement list template, carrying out standardized processing on the data to be checked to obtain standardized data to be checked; the preset fund settlement list template is used for determining the representation format of the data to be audited;
transmitting the standardized data to be checked to a data checking module so that the data checking module can check the standardized data to be checked to obtain a data checking result; the data auditing module is used for auditing the standardized data to be audited; and the data auditing result is used for representing whether the data to be audited passes or not.
2. The method of claim 1, wherein storing the data to be processed in a data storage module of each node included in the ignite cluster comprises:
and uniformly storing the data to be processed into the data storage modules of which the data storage levels are magnetic disks in each node included in the ignite cluster according to a stream processing mode.
3. The method according to claim 2, wherein uniformly storing the data to be processed in the data storage modules of which the data storage hierarchy is a disk in each node included in the ignite cluster includes:
storing the data to be processed with the association relation in a data storage module under the same node in the ignite cluster; the data to be processed with the association relation represents the data to be processed with the same main key information.
4. The method of claim 3, wherein the determining, in response to the data auditing instruction, data to be audited from the data storage modules of the nodes included in the ignite cluster includes:
responding to the data auditing instruction, and determining to-be-processed data matched with the data auditing instruction from a data storage module with a data storage hierarchy of a disk in each node included in the ignite cluster;
And loading the matched data to be processed into a data storage module of which the data storage level is a cache in the ignite cluster to obtain the data to be checked.
5. The method of claim 3, wherein the determining, in response to the data auditing instruction, data to be audited from the data storage modules of the nodes included in the ignite cluster includes:
responding to a data auditing instruction, and determining at least one primary key information matched with the data auditing instruction;
and determining the data to be checked from the data storage modules of all the nodes included in the ignite cluster based on the at least one primary key information.
6. The method according to claim 1, wherein the normalizing the pending data based on a preset fund settlement manifest template to obtain normalized pending data includes:
and determining an SQL instruction corresponding to the preset fund settlement list template, and carrying out standardized processing on the data to be checked based on the SQL instruction to obtain the standardized data to be checked.
7. The method according to any of claims 1-6, wherein storing the data to be processed into a data storage module of each node comprised by an ignite cluster comprises:
Determining data attribute information of the data to be processed; wherein the data attribute information characterizes the data amount of the data to be processed and/or the association degree between the data to be processed;
determining a data storage mode matched with the data attribute information; the data storage mode is a partition storage mode or at least one of a copy storage mode;
and storing the data to be processed into a data storage module of each node included in the ignite cluster based on the data storage mode.
8. A data processing apparatus for use with a long-guard fund, comprising:
the data storage unit is used for acquiring data to be processed and storing the data to be processed into the data storage modules of all nodes included in the ignite cluster; the data to be processed represents fund settlement related data of the long-protection fund; the data storage modules included by the nodes in the ignite cluster correspond to a plurality of data storage levels;
the data determining unit is used for responding to a data auditing instruction and determining data to be audited from the data storage modules of all nodes included in the ignite cluster; the data auditing instruction is used for auditing the data to be processed; the data to be checked represents the data to be processed which needs to be checked;
The standardized unit is used for carrying out standardized processing on the data to be checked based on a preset fund settlement list template to obtain standardized data to be checked; the preset fund settlement list template is used for determining the representation format of the data to be audited;
the auditing processing unit is used for sending the standardized data to be audited to the data auditing module so that the data auditing module can audit the standardized data to obtain a data auditing result; the data auditing module is used for auditing the standardized data to be audited; and the data auditing result is used for representing whether the data to be audited passes or not.
9. A computer device, comprising: a processor, and a memory communicatively coupled to the processor;
the memory stores computer-executable instructions;
the processor executes computer-executable instructions stored in the memory to implement the data processing method of any one of claims 1 to 7 applied to long-guard funds.
10. A computer-readable storage medium, wherein computer-executable instructions are stored in the computer-readable storage medium, which when executed by a processor, is configured to implement the data processing method of any one of claims 1 to 7 applied to a long-term fund.
CN202311524468.6A 2023-11-15 2023-11-15 Data processing method, device and equipment applied to long-protection fund Pending CN117493460A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311524468.6A CN117493460A (en) 2023-11-15 2023-11-15 Data processing method, device and equipment applied to long-protection fund

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311524468.6A CN117493460A (en) 2023-11-15 2023-11-15 Data processing method, device and equipment applied to long-protection fund

Publications (1)

Publication Number Publication Date
CN117493460A true CN117493460A (en) 2024-02-02

Family

ID=89679832

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311524468.6A Pending CN117493460A (en) 2023-11-15 2023-11-15 Data processing method, device and equipment applied to long-protection fund

Country Status (1)

Country Link
CN (1) CN117493460A (en)

Similar Documents

Publication Publication Date Title
US20170150290A1 (en) Method and device for information push
US20170249934A1 (en) Electronic device and method for operating the same
US10269347B2 (en) Method for detecting voice and electronic device using the same
US10817702B2 (en) Method, terminal, and computer-readable storage medium for processing fingerprint
CN117493460A (en) Data processing method, device and equipment applied to long-protection fund
CN114186894A (en) Project risk detection method and device, electronic equipment and storage medium
CN114428737A (en) Communication method, device, equipment and storage medium based on micro service system
CN112015973B (en) Relationship reasoning method and terminal of heterogeneous network
CN108021808B (en) Account book security reinforcing method and device and electronic equipment
CN113362180B (en) Customer information identification method, device and equipment for suspected nonstandard financial behavior
CN112102081B (en) Method, device, readable storage medium and blockchain network for generating blockchain
CN111831749B (en) Data processing method and device, storage medium and electronic equipment
CN112733141B (en) Information processing method and device
CN117033246A (en) Transaction playback processing method, device, equipment and storage medium
CN117149649A (en) Transaction testing method and device and electronic equipment
CN117035973A (en) Method, device, computer equipment and storage medium for identifying public clients
CN117075857A (en) Processing method, device, equipment and storage medium of internet banking system
CN116009946A (en) Git-based code statistics method and device
CN115658476A (en) Automatic testing method and device, electronic equipment and storage medium
CN116468566A (en) Method, device, equipment and storage medium for processing flow information
CN117149524A (en) Software version restoration method and device, electronic equipment and storage medium
CN115510107A (en) Data preheating caching method, device, equipment and storage medium
CN117093486A (en) Interface testing method, device, equipment and storage medium
CN111752896A (en) Method and device for cleaning junk files in mobile terminal and mobile terminal
CN117273649A (en) Method, device, equipment and storage medium for processing approval request

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination