CN111897856A - Supervision message generation method and device, electronic equipment and readable storage medium - Google Patents
Supervision message generation method and device, electronic equipment and readable storage medium Download PDFInfo
- Publication number
- CN111897856A CN111897856A CN202010761344.XA CN202010761344A CN111897856A CN 111897856 A CN111897856 A CN 111897856A CN 202010761344 A CN202010761344 A CN 202010761344A CN 111897856 A CN111897856 A CN 111897856A
- Authority
- CN
- China
- Prior art keywords
- data
- reported
- data set
- supervision
- standard
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 46
- 238000012545 processing Methods 0.000 claims abstract description 64
- 238000006243 chemical reaction Methods 0.000 claims abstract description 47
- 238000000638 solvent extraction Methods 0.000 claims abstract description 13
- 238000005538 encapsulation Methods 0.000 claims abstract description 12
- 238000005192 partition Methods 0.000 claims description 28
- 238000003780 insertion Methods 0.000 claims description 25
- 230000037431 insertion Effects 0.000 claims description 25
- 238000013145 classification model Methods 0.000 claims description 18
- 230000002159 abnormal effect Effects 0.000 claims description 16
- 238000004590 computer program Methods 0.000 claims description 6
- 238000004806 packaging method and process Methods 0.000 claims description 6
- 239000000126 substance Substances 0.000 claims 1
- 238000005516 engineering process Methods 0.000 abstract description 2
- 230000006870 function Effects 0.000 description 10
- 238000012544 monitoring process Methods 0.000 description 10
- 238000007726 management method Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000003066 decision tree Methods 0.000 description 2
- 238000012217 deletion Methods 0.000 description 2
- 230000037430 deletion Effects 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000001105 regulatory effect Effects 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
- G06F16/24558—Binary matching operations
- G06F16/2456—Join operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/258—Data format conversion from or to a database
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Mathematical Physics (AREA)
- Fuzzy Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to the field of big data, and discloses a supervision message generation method, which comprises the following steps: acquiring a scene to be reported, dynamically splicing query statements according to the scene to be reported, and querying in a preset database by using the query statements to obtain a reported data set; carrying out format conversion processing on data contained in the reported data set to obtain a standard reported data set; the standard reported data set is subjected to batch warehousing and table partitioning processing to obtain a target reported data table; and carrying out file encapsulation configuration processing on the target reported data table to obtain a supervision message. The embodiment of the invention also relates to a block chain technology, and the target reporting data table can be stored in the block chain. The invention also provides a supervision message generation device, electronic equipment and a computer readable storage medium. The invention can improve the generation speed of the supervision message.
Description
Technical Field
The invention relates to the field of big data, in particular to a supervision message generation method and device, electronic equipment and a readable storage medium.
Background
At present, along with the development of internet economy, the internet economy needs to be better supervised, and the main supervision mode at present is to supervise by using a supervision message, so that the generation of the supervision message is more and more important, and the supervision message is a file formed by information needing to be supervised, such as: the method comprises the following steps that a plurality of business entity companies such as loan, guarantee, insurance and the like need to report information such as business data, financial data and the like according to supervision requirements, and a file formed by the information is a supervision message.
However, in the generation process of the current supervision message, the supervised data needs to be manually screened, so that the formats are not uniform, and the speed of storing the data into the data table is slow, which affects the generation efficiency of the supervision message.
Disclosure of Invention
The invention provides a supervision message generation method, a supervision message generation device, electronic equipment and a computer readable storage medium, and mainly aims to improve the generation speed of supervision messages.
In order to achieve the above object, a method for generating supervision messages provided by the present invention includes:
acquiring a scene to be reported, dynamically splicing query statements according to the scene to be reported, and querying in a preset database by using the query statements to obtain a reported data set;
carrying out format conversion processing on data contained in the reported data set to obtain a standard reported data set;
the standard reported data set is subjected to batch warehousing and table partitioning processing to obtain a target reported data table;
and carrying out file encapsulation configuration processing on the target reported data table to obtain a supervision message.
Optionally, the dynamically splicing query statements according to the scene to be reported includes:
acquiring all fields to be reported in the scene to be reported to obtain query fields of the scene to be reported;
generating a query condition according to the scene to be reported;
and splicing the query field and the query condition to obtain the query statement.
Optionally, the performing format conversion processing on the data included in the reported data set to obtain a standard reported data set includes:
creating a thread pool containing at least two threads;
creating a format conversion task according to the data contained in the reported data set to obtain a segmented task set;
and importing the format conversion task set into the thread pool to perform json format conversion to obtain the standard reporting data set.
Optionally, before the standard report data set is batched into a warehouse and partitioned into tables to obtain a standard report data table, the method further includes:
and identifying and deleting abnormal data in the reported data set by using a pre-constructed data classification model.
Optionally, before the abnormal data in the reported data set is identified and deleted by using the pre-constructed data classification model, the method further includes:
acquiring a historical reported data set;
taking each information field contained in the historical reported data set as a root node;
and carrying out corresponding data correctness classification on all information field data of the historical reported data set to serve as leaf nodes corresponding to the root nodes, and obtaining the data classification model.
Optionally, the step of performing batch warehousing and table partitioning processing on the standard reporting data set to obtain a target reporting data table includes:
grouping the data contained in the standard reported data set according to a preset grouping number to obtain a grouped data set;
and inserting each group of data contained in the grouped data set into a pre-constructed partition table in batch by using a pre-constructed insertion statement to obtain the target reporting data table.
Optionally, the performing file encapsulation configuration processing on the target reporting data table to obtain a supervision packet includes:
packaging the target reporting data table into a preset format file by using a preset programming language file to obtain an initial supervision message;
and carrying out file configuration processing on the initial supervision message to obtain the supervision message.
In order to solve the above problem, the present invention further provides a supervision packet generating apparatus, including:
the report data generation module is used for acquiring a scene to be reported, dynamically splicing query statements according to the scene to be reported, and querying in a preset database by using the query statements to obtain a report data set;
the format conversion module is used for carrying out format conversion processing on the data contained in the reported data set to obtain a standard reported data set;
the supervision message generation module is used for carrying out batch warehousing and table partition processing on the standard reporting data set to obtain a target reporting data table; and carrying out file encapsulation configuration processing on the target reported data table to obtain a supervision message.
In order to solve the above problem, the present invention also provides an electronic device, including:
a memory storing at least one instruction; and
and the processor executes the instructions stored in the memory to realize the supervision message generation method.
In order to solve the above problem, the present invention further provides a computer-readable storage medium storing a computer program, which is executed by a processor to implement the supervision message generating method described above.
In the embodiment of the invention, query sentences are dynamically spliced according to the scene to be reported and are used for querying in a preset database to obtain a reported data set, so that the data screening speed is improved; carrying out format conversion processing on the data contained in the reported data set to obtain a standard reported data set, and then, classifying and processing the data with different formats in the reported data set; the standard reported data set is subjected to batch warehousing and table partitioning processing to obtain a target reported data table, so that the data insertion speed is improved, and meanwhile, the data can be conveniently modified; and carrying out file packaging configuration processing on the target report data table to obtain a supervision message, so that the template generation of the supervision message is realized, and the generation speed of the supervision message is improved.
Drawings
Fig. 1 is a schematic flow chart of a supervision packet generation method according to an embodiment of the present invention;
fig. 2 is a schematic block diagram of a supervision packet generation apparatus according to an embodiment of the present invention;
fig. 3 is a schematic diagram of an internal structure of an electronic device implementing a supervision packet generation method according to an embodiment of the present invention;
the implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
The invention provides a supervision message generation method. Fig. 1 is a schematic flow chart of a supervision packet generation method according to an embodiment of the present invention. The method may be performed by an apparatus, which may be implemented by software and/or hardware.
In this embodiment, the supervision packet generating method includes:
s1, acquiring a scene to be reported, dynamically splicing query statements according to the scene to be reported, and querying in a preset database by using the query statements to obtain a reported data set;
in the embodiment of the present invention, the scene to be reported is a data scene type of data to be reported, for example: loan transaction scenarios, and guarantee transaction scenarios.
Further, in the embodiment of the present invention, the dynamically splicing query statements according to the scene to be reported includes: step A, acquiring all information fields to be reported in the scene to be reported to obtain query fields of the scene to be reported;
in the embodiment of the invention, the supervisor specifies that different information fields need to be reported in different scenes to be reported, so that all the information fields needing to be reported in the scenes to be reported can be acquired from the database required by the supervisor to report.
B, generating a query condition according to the scene to be reported;
in the embodiment of the invention, the data to be reported in different scenes to be reported correspond to different data requirements such as data ranges, data sizes and the like, so that the query conditions are generated according to the scenes to be reported.
And step C, splicing the query field and the query condition to obtain the query statement.
Preferably, the Query statement is an SQL (Structured Query Language) statement, and the data set of all Query fields that meet the Query condition can be obtained by executing the Query statement.
Further, the embodiment of the present invention executes the query statement, and queries in a preset database to obtain a message data set corresponding to the reporting scenario.
The preset database may include different service system databases, such as a home E credit database, a credit insurance life insurance database, and the like.
Preferably, the data in the service system database may include service data generated in an actual service system, the quantity scale is large, and the monthly report order can reach millions. The business data may include data generated when products are traded on the market, such as product information, product trading time, product trading manner, product trading price, customer and customer information for purchasing the products, and the like.
Preferably, in the embodiment of the present invention, the query statement is dynamically spliced according to the scene to be reported, and the reported data set is obtained by querying from different service system databases, so that the service data of different service system databases can be uniformly processed, the data processing performance is improved, and the generation efficiency of the message is improved.
S2, performing format conversion processing on the data contained in the reported data set to obtain a standard reported data set;
in the embodiment of the invention, the data in the subsequent supervision message needs a uniform data format, so that the data contained in the reported data set is subjected to format conversion processing, the data in the reported data set is converted into a uniform format, the data in different formats does not need to be processed respectively subsequently, and the generation speed of the subsequent supervision message is improved.
In detail, the embodiment of the present invention performs format conversion processing on data included in the reported data set, including:
step a, creating a thread pool containing at least two threads;
in the embodiment of the invention, in order to accelerate the speed of format conversion processing on the data contained in the reported data set, the data contained in the reported data set is processed by utilizing multiple threads, so that the data processing speed is improved. For example: the reported data set comprises data of the A information segment and data of the B information segment, the data of the A information segment and the data of the B information segment can be converted simultaneously by utilizing multithreading, and the data of the A information segment and the data of the B information segment are not required to be converted firstly and then converted like a single thread. Preferably, the thread pool according to the embodiment of the present invention may be constructed by Executors class of JAVA language.
B, creating a format conversion task according to the data contained in the reported data set to obtain a segmented task set;
in detail, in the embodiment of the present invention, data included in the reported data set is segmented to obtain multiple segment data groups, and the multiple segment data groups are subjected to group sorting and marking according to an order of the data included in the reported data set to obtain a segment task set, where each marked segment data group is a segment task.
And c, importing the format conversion task set into the thread pool to perform json format conversion to obtain the standard reporting data set.
In the embodiment of the present invention, the format conversion task set is imported into the thread pool, each thread in the thread pool executes a format conversion task, data in a segmented data group corresponding to the format conversion task is converted into a json format, and a task conversion result is obtained. In the embodiment of the invention, the thread pool is utilized to convert the data of each information segment contained in the reported data set into the json format in parallel, so that the threads can be recycled, frequent creation and cancellation of the threads are not needed, and the efficiency of data format conversion is improved.
S3, performing batch warehousing and table partition processing on the standard reported data set to obtain a target reported data table;
in the embodiment of the present invention, in order to facilitate query and modification of data in the standard reported data set, the standard reported data set needs to be converted into a format of a data table, so that the standard reported data set is subjected to batch storage and table partitioning processing to obtain a standard reported data table; the method comprises the following steps: inserting the data contained in the standard reported data set into a pre-constructed partition table in batches according to a preset data insertion quantity to obtain the standard reported data table; for example: and inserting the data contained in the standard reported data set into the pre-constructed partition table in batches according to the insertion speed of inserting 1000 data at a time, so that the insertion speed is greatly improved compared with the method of inserting the data into the pre-constructed partition table one by one.
In detail, in the embodiment of the present invention, inserting the data included in the standard reported data set into the pre-constructed partition table in batches according to the preset data insertion number includes:
step I, grouping data contained in the standard reported data set according to a preset grouping number to obtain a grouped data set;
for example: the data contained in the standard reported data set is 2000 pieces of data, the preset grouping number is 1000 pieces, the data in the standard reported data set are divided into two groups, the data volume of each group of data is 1000 pieces, and the two groups of data are collected to obtain a grouped data set.
And step II, inserting each group of data contained in the grouped data set into the pre-constructed partition table in batch by using the pre-constructed insertion statement to obtain a target reporting data table.
For example: the grouped data set comprises A, B groups of data, the total amount of data contained in each group of data is 1000, 1000 pieces of data contained in each group of A, B groups of data are respectively inserted into a pre-constructed partition table in batches at one time, and a target reported data table is obtained.
In the embodiment of the present invention, each group of data included in the grouped data set is inserted into the pre-constructed partition table by using a preset insertion statement, and preferably, in the embodiment of the present invention, the preset insertion statement may be constructed by using an insert function of SQL language, and by using the method of inserting each group of data included in the grouped data set into the pre-constructed partition table in batches, the frequency of insertion connection with the pre-constructed partition table may be reduced, and the insertion speed may be increased.
Further, in order to improve the accuracy of data in subsequent supervision messages, before the standard reported data set is subjected to batch warehousing and table partitioning processing, the invention also needs to perform abnormal data deletion processing on the reported data set, and the method comprises the following steps: and identifying and deleting abnormal data in the reported data set by using a pre-constructed data classification model.
In detail, in the embodiment of the present invention, each piece of data included in the reported data set is input into a pre-constructed data classification model for classification, and abnormal data in the reported data set is identified and obtained.
Preferably, in the embodiment of the present invention, the data classification model is a decision tree model.
Further, in the embodiment of the present invention, before each piece of data included in the reported data set is input into a pre-constructed data classification model for classification, the model needs to be trained, which includes:
step one, acquiring a historical reported data set;
in the embodiment of the present invention, the historical reported data set is data that includes all information fields that need to be reported and has the same data format as the reported data set. The historical reporting data set may be obtained from a reported regulatory expression database of a company.
Step two, taking each information field contained in the historical reported data set as a root node;
and thirdly, carrying out corresponding data correctness classification by using all information field data of the historical reported data set to obtain leaf nodes corresponding to the root nodes, and obtaining the data classification model.
In detail, in the embodiment of the present invention, the data correctness is classified into a classification of the information segment data into a correct data leaf node and an abnormal data leaf node.
Furthermore, the abnormal data in the reported data set are identified and deleted by utilizing the pre-constructed data classification model, so that the identification speed of the abnormal data is improved.
In the embodiment of the invention, the data contained in the standard reported data set is inserted into the pre-constructed partition table in batches according to the preset data insertion speed, and compared with the method of inserting the data contained in the standard reported data set into the partition table into a common data table, the method improves the data query speed and is convenient for modifying the data in the table.
In another embodiment of the present invention, in order to ensure that the data of the target reporting data table is not leaked, the target reporting data table may be stored in a block chain.
And S4, performing file encapsulation configuration processing on the target reported data table to obtain a supervision message.
In the embodiment of the present invention, the converting the target reported data table into a file format required by a monitoring party, and the performing file configuration and encapsulation processing on the target reported data table includes: packaging the target reporting data table into a preset format file by using a preset programming language type file to obtain an initial supervision message, so that the templating of supervision message generation is realized, and the speed of supervision message generation is improved, wherein the preset format comprises: TXT format, XML format. Preferably, the embodiment of the present invention encapsulates the target reporting data table into the preset format file by using a servlet class in Java language.
Further, according to the requirement of the monitoring party, the embodiment of the present invention performs file configuration processing on the initial monitoring message to obtain the monitoring message. Wherein the file configuration process comprises: setting file names and file formats, and the like.
In the embodiment of the invention, query sentences are dynamically spliced according to the scene to be reported and are used for querying in a preset database to obtain a reported data set, so that the data screening speed is improved; carrying out format conversion processing on the data contained in the reported data set to obtain a standard reported data set, and then, classifying and processing the data with different formats in the reported data set; the standard reported data set is subjected to batch warehousing and table partitioning processing to obtain a target reported data table, so that the data insertion speed is improved, and meanwhile, the data can be conveniently modified; and carrying out file packaging configuration processing on the target report data table to obtain a supervision message, so that the template generation of the supervision message is realized, and the generation speed of the supervision message is improved.
Fig. 2 is a functional block diagram of the supervision message generating apparatus according to the present invention.
The supervision message generating apparatus 100 according to the present invention may be installed in an electronic device. According to the implemented functions, the supervision packet generating apparatus 100 may include a report data generating module 101, a format converting module 102, and a supervision packet generating module 103. A module according to the present invention, which may also be referred to as a unit, refers to a series of computer program segments that can be executed by a processor of an electronic device and that can perform a fixed function, and that are stored in a memory of the electronic device.
In the present embodiment, the functions regarding the respective modules/units are as follows:
the report data generation module 101 is configured to acquire a scene to be reported, dynamically splice query statements according to the scene to be reported, and query in a preset database by using the query statements to obtain a report data set.
In the embodiment of the present invention, the scene to be reported is a data scene type of data to be reported, for example: loan transaction scenarios, and guarantee transaction scenarios.
Further, in the embodiment of the present invention, the report data generating module 101 dynamically splices query statements according to the scene to be reported by using the following means:
step A, acquiring all information fields to be reported in the scene to be reported to obtain query fields of the scene to be reported;
in the embodiment of the invention, the supervisor specifies that different information fields need to be reported in different scenes to be reported, so that all the information fields needing to be reported in the scenes to be reported can be acquired from the database required by the supervisor to report.
B, generating a query condition according to the scene to be reported;
in the embodiment of the invention, the data to be reported in different scenes to be reported correspond to different data requirements such as data ranges, data sizes and the like, so that the query conditions are generated according to the scenes to be reported.
And step C, splicing the query field and the query condition to obtain the query statement.
Preferably, the Query statement is an SQL (Structured Query Language) statement, and the data set of all Query fields that meet the Query condition can be obtained by executing the Query statement.
Further, in the embodiment of the present invention, the report data generation module 101 executes the query statement, and queries in a preset database to obtain a packet data set corresponding to a report scenario.
The preset database may include different service system databases, such as a home E credit database, a credit insurance life insurance database, and the like.
Preferably, the message data set may include service data generated in an actual service system, and the number scale is large, and the monthly report order may reach millions. The business data may include data generated when products are traded on the market, such as product information, product trading time, product trading manner, product trading price, customer and customer information for purchasing the products, and the like.
Preferably, the report data generating module 101 of the embodiment of the present invention obtains the report data set by querying from different service system databases by dynamically splicing query statements according to the scene to be reported, so as to implement unified processing on the service data of different service system databases, thereby improving data processing performance and improving message generating efficiency.
The format conversion module 102 is configured to perform format conversion processing on data included in the reported data set to obtain a standard reported data set.
In the embodiment of the invention, the data in the subsequent supervision message needs a uniform data format, so that the data contained in the reported data set is subjected to format conversion processing, the data in the reported data set is converted into a uniform format, the data in different formats does not need to be processed respectively subsequently, and the generation speed of the subsequent supervision message is improved.
In detail, the format conversion module 102 in the embodiment of the present invention performs format conversion processing on data included in the reported data set by using the following method:
step a, creating a thread pool containing at least two threads;
in the embodiment of the invention, in order to accelerate the speed of format conversion processing on the data contained in the reported data set, the data contained in the reported data set is processed by utilizing multiple threads, so that the data processing speed is improved. For example: the reported data set comprises data of the A information segment and data of the B information segment, the data of the A information segment and the data of the B information segment can be converted simultaneously by utilizing multithreading, and the data of the A information segment and the data of the B information segment are not required to be converted firstly and then converted like a single thread. Preferably, the thread pool according to the embodiment of the present invention may be constructed by Executors class of JAVA language.
B, creating a format conversion task according to the data contained in the reported data set to obtain a segmented task set;
in detail, in the embodiment of the present invention, data included in the reported data set is segmented to obtain multiple segment data groups, and the multiple segment data groups are subjected to group sorting and marking according to an order of the data included in the reported data set to obtain a segment task set, where each marked segment data group is a segment task.
And c, importing the format conversion task set into the thread pool to perform json format conversion to obtain the standard reporting data set.
In this embodiment of the present invention, the format conversion module 102 imports the format conversion task set into the thread pool, each thread in the thread pool executes one format conversion task, converts data in a segmented data group corresponding to the format conversion task into a json format, and obtains a task conversion result, further, when the format conversion task executed by the thread is completed, the thread pool is used to recycle the thread, and the thread is reused, and when the task in the format conversion task set is completed, all the task results are sequentially collected according to a sorting flag of the corresponding segmented task, so as to obtain the standard report data set. In the embodiment of the invention, the thread pool is utilized to convert the data of each information segment contained in the reported data set into the json format in parallel, so that the threads can be recycled, frequent creation and cancellation of the threads are not needed, and the efficiency of data format conversion is improved.
The supervision message generating module 103 is configured to perform batch warehousing and table partitioning processing on the standard reporting data set to obtain a target reporting data table, and perform file encapsulation configuration processing on the target reporting data table to obtain a supervision message.
In the embodiment of the present invention, in order to facilitate the query and modification of the data in the standard reported data set, the standard reported data set needs to be converted into a format of a data table, so that the supervision packet generation module 103 performs batch storage and table partitioning processing on the standard reported data set to obtain a standard reported data table; the method comprises the following steps: inserting the data contained in the standard reported data set into a pre-constructed partition table in batches according to a preset data insertion quantity to obtain the standard reported data table; for example: and inserting the data contained in the standard reported data set into the pre-constructed partition table in batches according to the insertion speed of inserting 1000 data at a time, so that the insertion speed is greatly improved compared with the method of inserting the data into the pre-constructed partition table one by one.
In detail, in the embodiment of the present invention, the supervision packet generating module 103 inserts the data included in the standard reported data set into the pre-constructed partition table in batches according to the preset data insertion amount by using the following means:
step I, grouping data contained in the standard reported data set according to a preset grouping number to obtain a grouped data set;
for example: the data contained in the standard reported data set is 2000 pieces of data, the preset grouping number is 1000 pieces, the data in the standard reported data set are divided into two groups, the data volume of each group of data is 1000 pieces, and the two groups of data are collected to obtain a grouped data set.
And step II, inserting each group of data contained in the grouped data set into the pre-constructed partition table in batch by using a pre-constructed insertion function to obtain a target reported data table.
For example: the grouped data set comprises A, B groups of data, the total amount of data contained in each group of data is 1000, 1000 pieces of data contained in each group of A, B groups of data are respectively inserted into a pre-constructed partition table in batches at one time, and a target reported data table is obtained.
In the embodiment of the present invention, the supervision packet generating module 103 inserts each group of data included in the grouped data set into the pre-constructed partition table by using a preset insertion statement, and preferably, in the embodiment of the present invention, the preset insertion statement may be constructed by using an insert function of SQL language, and by inserting each group of data included in the grouped data set into the pre-constructed partition table in batches, the frequency of insertion connection with the pre-constructed partition table may be reduced and the insertion speed may be increased by using a batch insertion manner.
Further, in order to improve the accuracy of data in subsequent supervision messages, before the supervision message generating module 103 performs batch warehousing and table partitioning processing on the standard reported data set, the method also needs to perform abnormal data deletion processing on the reported data set, and includes: and identifying and deleting abnormal data in the reported data set by using a pre-constructed data classification model.
In detail, in the embodiment of the present invention, the supervision packet generating module 103 inputs each piece of data included in the reported data set into a pre-constructed data classification model for classification, and identifies abnormal data in the reported data set.
Preferably, in the embodiment of the present invention, the data classification model is a decision tree model.
Further, in the embodiment of the present invention, before each piece of data included in the reported data set is input into a pre-constructed data classification model for classification, the supervision packet generation module 103 needs to train the model by using the following means:
step one, acquiring a historical reported data set;
in the embodiment of the present invention, the historical reported data set is data that includes all information fields that need to be reported and has the same data format as the reported data set. The historical reporting data set may be obtained from a reported regulatory expression database of a company.
Step two, taking each information field contained in the historical reported data set as a root node;
and thirdly, carrying out corresponding data correctness classification by using all information field data of the historical reported data set to obtain leaf nodes corresponding to the root nodes, and obtaining the data classification model.
In detail, in the embodiment of the present invention, the data correctness is classified into a classification of the information segment data into a correct data leaf node and an abnormal data leaf node.
Further, in the embodiment of the present invention, the supervision packet generating module 103 identifies and deletes the abnormal data in the reported data set by using a pre-constructed data classification model, so as to improve the identification speed of the abnormal data.
In the embodiment of the present invention, the supervision packet generating module 103 inserts the data included in the standard reported data set into the pre-constructed partition table in batches according to a preset data insertion speed, and compared with inserting the data included in the standard reported data set into the partition table into a common data table, the data in the table can be modified conveniently.
In another embodiment of the present invention, in order to ensure that the data of the target reporting data table is not leaked, the target reporting data table may be stored in a block chain.
In the embodiment of the present invention, the target reported data table needs to be converted into a file format required by a monitoring party, and the monitoring message generation module 103 performs file configuration and encapsulation processing on the target reported data table, including: packaging the target reporting data table into a preset format file by using a preset programming language type file to obtain an initial supervision message, so that the templating of supervision message generation is realized, and the speed of supervision message generation is improved, wherein the preset format comprises: TXT format, XML format. Preferably, the embodiment of the present invention encapsulates the target reporting data table into the preset format file by using a servlet class in Java language.
Further, according to the requirement of the monitoring party, the monitoring message generation module 103 performs file configuration processing on the initial monitoring message to obtain the monitoring message in the embodiment of the present invention. Wherein the file configuration process comprises: setting file names and file formats and the like.
Fig. 3 is a schematic structural diagram of an electronic device implementing the supervision packet generation method according to the present invention.
The electronic device 1 may comprise a processor 10, a memory 11 and a bus, and may further comprise a computer program, such as a supervision message generating program, stored in the memory 11 and executable on the processor 10.
The memory 11 includes at least one type of readable storage medium, which includes flash memory, removable hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk, etc. The memory 11 may in some embodiments be an internal storage unit of the electronic device 1, such as a removable hard disk of the electronic device 1. The memory 11 may also be an external storage device of the electronic device 1 in other embodiments, such as a plug-in mobile hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the electronic device 1. Further, the memory 11 may also include both an internal storage unit and an external storage device of the electronic device 1. The memory 11 may be used not only for storing application software installed in the electronic device 1 and various types of data, such as codes of a supervision message generating program, but also for temporarily storing data that has been output or is to be output.
The processor 10 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or may be composed of a plurality of integrated circuits packaged with the same or different functions, including one or more Central Processing Units (CPUs), microprocessors, digital Processing chips, graphics processors, and combinations of various control chips. The processor 10 is a Control Unit (Control Unit) of the electronic device, connects various components of the whole electronic device by using various interfaces and lines, and executes various functions and processes data of the electronic device 1 by running or executing programs or modules (e.g., a supervision message generating program, etc.) stored in the memory 11 and calling data stored in the memory 11.
The bus may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is arranged to enable connection communication between the memory 11 and at least one processor 10 or the like.
Fig. 3 shows only an electronic device with components, and it will be understood by those skilled in the art that the structure shown in fig. 3 does not constitute a limitation of the electronic device 1, and may comprise fewer or more components than those shown, or some components may be combined, or a different arrangement of components.
For example, although not shown, the electronic device 1 may further include a power supply (such as a battery) for supplying power to each component, and preferably, the power supply may be logically connected to the at least one processor 10 through a power management device, so as to implement functions of charge management, discharge management, power consumption management, and the like through the power management device. The power supply may also include any component of one or more dc or ac power sources, recharging devices, power failure detection circuitry, power converters or inverters, power status indicators, and the like. The electronic device 1 may further include various sensors, a bluetooth module, a Wi-Fi module, and the like, which are not described herein again.
Further, the electronic device 1 may further include a network interface, and optionally, the network interface may include a wired interface and/or a wireless interface (such as a WI-FI interface, a bluetooth interface, etc.), which are generally used for establishing a communication connection between the electronic device 1 and other electronic devices.
Optionally, the electronic device 1 may further comprise a user interface, which may be a Display (Display), an input unit (such as a Keyboard), and optionally a standard wired interface, a wireless interface. Alternatively, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, or the like. The display, which may also be referred to as a display screen or display unit, is suitable for displaying information processed in the electronic device 1 and for displaying a visualized user interface, among other things.
It is to be understood that the described embodiments are for purposes of illustration only and that the scope of the appended claims is not limited to such structures.
The supervision message generating program 12 stored in the memory 11 of the electronic device 1 is a combination of instructions that, when executed in the processor 10, may implement:
acquiring a scene to be reported, dynamically splicing query statements according to the scene to be reported, and querying in a preset database by using the query statements to obtain a reported data set;
carrying out format conversion processing on data contained in the reported data set to obtain a standard reported data set;
the standard reported data set is subjected to batch warehousing and table partitioning processing to obtain a target reported data table;
and carrying out file encapsulation configuration processing on the target reported data table to obtain a supervision message.
Specifically, the specific implementation method of the processor 10 for the instruction may refer to the description of the relevant steps in the embodiment corresponding to fig. 1, which is not described herein again.
Further, the integrated modules/units of the electronic device 1, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. The computer-readable medium may include: any entity or device capable of carrying said computer program code, recording medium, U-disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM).
Further, the computer usable storage medium may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created according to the use of the blockchain node, and the like.
In the embodiments provided in the present invention, it should be understood that the disclosed apparatus, device and method can be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is only one logical functional division, and other divisions may be realized in practice.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment.
In addition, functional modules in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional module.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof.
The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference signs in the claims shall not be construed as limiting the claim concerned.
The block chain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.
Furthermore, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or means recited in the system claims may also be implemented by one unit or means in software or hardware. The terms second, etc. are used to denote names, but not any particular order.
Finally, it should be noted that the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting, and although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims (10)
1. A supervision message generation method is characterized by comprising the following steps:
acquiring a scene to be reported, dynamically splicing query statements according to the scene to be reported, and querying in a preset database by using the query statements to obtain a reported data set;
carrying out format conversion processing on data contained in the reported data set to obtain a standard reported data set;
the standard reported data set is subjected to batch warehousing and table partitioning processing to obtain a target reported data table;
and carrying out file encapsulation configuration processing on the target reported data table to obtain a supervision message.
2. The supervision packet generating method according to claim 1, wherein dynamically concatenating the query statement according to the scene to be reported includes:
acquiring all fields to be reported in the scene to be reported to obtain query fields of the scene to be reported;
generating a query condition according to the scene to be reported;
and splicing the query field and the query condition to obtain the query statement.
3. The supervision packet generating method according to claim 1, wherein performing format conversion processing on the data included in the reported data set to obtain a standard reported data set includes:
creating a thread pool containing at least two threads;
creating a format conversion task according to the data contained in the reported data set to obtain a segmented task set;
and importing the format conversion task set into the thread pool to perform json format conversion to obtain the standard reporting data set.
4. The supervision packet generating method according to claim 1, wherein before the standard report data set is batched into a warehouse and processed in table partitions to obtain a standard report data table, the method further comprises:
and identifying and deleting abnormal data in the reported data set by using a pre-constructed data classification model.
5. The supervision packet generation method according to claim 4, wherein before the abnormal data in the reported data set is identified and deleted by using the pre-constructed data classification model, the method further comprises:
acquiring a historical reported data set;
taking each information field contained in the historical reported data set as a root node;
and carrying out corresponding data correctness classification on all information field data of the historical reported data set to serve as leaf nodes corresponding to the root nodes, and obtaining the data classification model.
6. The supervision packet generation method according to claim 1, wherein the batch-in-storage and table-partitioning processing of the standard reporting data set to obtain a target reporting data table includes:
grouping the data contained in the standard reported data set according to a preset grouping number to obtain a grouped data set;
and inserting each group of data contained in the grouped data set into a pre-constructed partition table in batch by using a pre-constructed insertion statement to obtain the target reporting data table.
7. The supervision packet generating method according to claim 1, wherein the performing the file encapsulation configuration processing on the target reporting data table to obtain the supervision packet includes:
packaging the target reporting data table into a preset format file by using a preset programming language file to obtain an initial supervision message;
and carrying out file configuration processing on the initial supervision message to obtain the supervision message.
8. An apparatus for supervision packet generation, the apparatus comprising:
the report data generation module is used for acquiring a scene to be reported, dynamically splicing query statements according to the scene to be reported, and querying in a preset database by using the query statements to obtain a report data set;
the format conversion module is used for carrying out format conversion processing on the data contained in the reported data set to obtain a standard reported data set;
the supervision message generation module is used for carrying out batch warehousing and table partition processing on the standard reporting data set to obtain a target reporting data table; and carrying out file encapsulation configuration processing on the target reported data table to obtain a supervision message.
9. An electronic device, characterized in that the electronic device comprises:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a supervision message generation method according to any of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the supervision message generation method according to any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010761344.XA CN111897856A (en) | 2020-07-31 | 2020-07-31 | Supervision message generation method and device, electronic equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010761344.XA CN111897856A (en) | 2020-07-31 | 2020-07-31 | Supervision message generation method and device, electronic equipment and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111897856A true CN111897856A (en) | 2020-11-06 |
Family
ID=73184133
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010761344.XA Pending CN111897856A (en) | 2020-07-31 | 2020-07-31 | Supervision message generation method and device, electronic equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111897856A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112540959A (en) * | 2020-12-14 | 2021-03-23 | 建信金融科技有限责任公司 | Data processing method and device |
| CN112581276A (en) * | 2020-12-24 | 2021-03-30 | 中国农业银行股份有限公司 | Data processing method and device |
| CN112632176A (en) * | 2020-12-31 | 2021-04-09 | 中国农业银行股份有限公司 | Interaction method and device for supervising submission database |
| CN112785284A (en) * | 2020-12-31 | 2021-05-11 | 银清科技有限公司 | Message storage method and device based on structured document |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107391744A (en) * | 2017-08-10 | 2017-11-24 | 东软集团股份有限公司 | Data storage, read method, device and its equipment |
| CN109388636A (en) * | 2017-08-11 | 2019-02-26 | 中国移动通信集团重庆有限公司 | Business datum is inserted into database method, apparatus, computer equipment and storage medium |
| CN110597842A (en) * | 2019-07-22 | 2019-12-20 | 石化盈科信息技术有限责任公司 | Service data query method and system |
| CN111177231A (en) * | 2019-12-20 | 2020-05-19 | 中国建设银行股份有限公司 | Report generation method and report generation device |
| CN111277465A (en) * | 2020-01-20 | 2020-06-12 | 支付宝(杭州)信息技术有限公司 | Abnormal data message detection method and device and electronic equipment |
| CN111897831A (en) * | 2020-07-31 | 2020-11-06 | 平安普惠企业管理有限公司 | Service message generation method and device, electronic equipment and storage medium |
-
2020
- 2020-07-31 CN CN202010761344.XA patent/CN111897856A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107391744A (en) * | 2017-08-10 | 2017-11-24 | 东软集团股份有限公司 | Data storage, read method, device and its equipment |
| CN109388636A (en) * | 2017-08-11 | 2019-02-26 | 中国移动通信集团重庆有限公司 | Business datum is inserted into database method, apparatus, computer equipment and storage medium |
| CN110597842A (en) * | 2019-07-22 | 2019-12-20 | 石化盈科信息技术有限责任公司 | Service data query method and system |
| CN111177231A (en) * | 2019-12-20 | 2020-05-19 | 中国建设银行股份有限公司 | Report generation method and report generation device |
| CN111277465A (en) * | 2020-01-20 | 2020-06-12 | 支付宝(杭州)信息技术有限公司 | Abnormal data message detection method and device and electronic equipment |
| CN111897831A (en) * | 2020-07-31 | 2020-11-06 | 平安普惠企业管理有限公司 | Service message generation method and device, electronic equipment and storage medium |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112540959A (en) * | 2020-12-14 | 2021-03-23 | 建信金融科技有限责任公司 | Data processing method and device |
| CN112581276A (en) * | 2020-12-24 | 2021-03-30 | 中国农业银行股份有限公司 | Data processing method and device |
| CN112632176A (en) * | 2020-12-31 | 2021-04-09 | 中国农业银行股份有限公司 | Interaction method and device for supervising submission database |
| CN112785284A (en) * | 2020-12-31 | 2021-05-11 | 银清科技有限公司 | Message storage method and device based on structured document |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111897856A (en) | Supervision message generation method and device, electronic equipment and readable storage medium | |
| CN112418798A (en) | Information auditing method and device, electronic equipment and storage medium | |
| CN112541745A (en) | User behavior data analysis method and device, electronic equipment and readable storage medium | |
| CN111428458A (en) | Universal report generation method and device and computer readable storage medium | |
| CN111897831A (en) | Service message generation method and device, electronic equipment and storage medium | |
| CN112364107A (en) | System analysis visualization method and device, electronic equipment and computer readable storage medium | |
| CN114979120B (en) | Data uploading method, device, equipment and storage medium | |
| CN112115145A (en) | Data acquisition method and device, electronic equipment and storage medium | |
| CN112528616A (en) | Business form generation method and device, electronic equipment and computer storage medium | |
| CN113434674A (en) | Data analysis method and device, electronic equipment and readable storage medium | |
| CN114185895A (en) | Data import and export method and device, electronic equipment and storage medium | |
| CN113961584A (en) | Method and device for analyzing field blood relationship, electronic equipment and storage medium | |
| CN114610747A (en) | Data query method, device, equipment and storage medium | |
| CN115408399A (en) | Blood relationship analysis method, device, equipment and storage medium based on SQL script | |
| CN112949278A (en) | Data checking method and device, electronic equipment and readable storage medium | |
| CN114881616A (en) | Business process execution method and device, electronic equipment and storage medium | |
| CN112579621A (en) | Data display method and device, electronic equipment and computer storage medium | |
| CN113722533A (en) | Information pushing method and device, electronic equipment and readable storage medium | |
| CN113505273A (en) | Data sorting method, device, equipment and medium based on repeated data screening | |
| CN112541688A (en) | Service data checking method and device, electronic equipment and computer storage medium | |
| CN115759746A (en) | Configurable user early warning method and device, electronic equipment and storage medium | |
| CN111553133B (en) | Report generation method and device, electronic equipment and storage medium | |
| CN114490137A (en) | Service data real-time statistical method and device, electronic equipment and readable storage medium | |
| CN114547011A (en) | Data extraction method and device, electronic equipment and storage medium | |
| CN113313211A (en) | Text classification method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |