CN110471917B - Intelligent customs declaration and customs clearance filling method based on historical data mining - Google Patents

Abstract

The invention provides a customs declaration intelligent filling method based on historical data mining. The method comprises the following steps: preprocessing historical data of customs declaration form, merging the header and the body data, and removing irrelevant fields; designing and implementing a correlation analysis algorithm of each field based on a specific value; designing a tree structure, and constructing and storing a dynamic tree based on the correlation of each field; according to the generated dynamic tree, intelligently recommending filling contents; and regularly and automatically maintaining and updating the data according to the newly input data. The correlation analysis algorithm and the dynamic tree designed by the invention can be used for intelligently filling the other filling fields according to the current input content of the user, and the accuracy is high. The customs clearance reporting system has strong generalization and self-learning capabilities, can greatly improve the customs clearance reporting efficiency, and saves manpower and material resources for clearance reporting organizations and clearance reporting enterprises.

Description

Intelligent customs declaration and customs clearance filling method based on historical data mining

The technical field is as follows:

the invention relates to the field of customs clearance information reporting, in particular to a customs clearance intelligent reporting method based on historical data mining.

Background art:

the customs of the people's republic of China is used as a supervision and management organ for the entry and exit of China, and the declaration and management of import and export commodities is an important and fundamental work in daily business of the customs. At present, import and export commodity declaration is mainly that a declaration company sorts and gathers all related paper receipts and then inputs the paper receipts into a customs related system, and because input fields are numerous and the content-to-content relationship is small, the input efficiency is low, the error rate is high, and manpower and material resources are wasted.

The invention content is as follows:

the invention provides a customs declaration and customs clearance intelligent filling method based on historical data mining. The method and the device fully consider the correlation among the fields in the filling information based on specific values, provide an intelligent filling strategy, and have the characteristics of higher efficiency and higher accuracy compared with the traditional complete manual entry.

The invention provides the following scheme, and discloses a customs declaration form intelligent reporting method based on historical data mining, which comprises the following steps:

s1: and preprocessing the historical data of the customs declaration form, merging the header and the body data, and removing irrelevant fields.

S1.1: merging data using Spark distributed computation

The real customs declaration form data is divided into header data and body data, and the header data and the body data are respectively stored in two data tables, wherein the header data describes certain order information, such as an entrance port, a declaration unit, a transaction mode and the like; the form data describes specific information of the goods in a certain order, such as a goods number, a goods name, a declaration unit price and the like. The invention uses Spark distributed computation method, and connects the two tables by using order number as main key, to obtain data table containing all filling information fields, and store it in Hive database.

S1.2: and (5) counting the null value condition of each field and removing irrelevant fields.

In the customs clearance process, some fields belong to user-selected filling items, so that the situation that the fields are empty can occur, the ratio of the number of empty value data pieces of a certain field to the total number of data pieces is counted in the preprocessing process, and if the number of empty value data pieces of a certain field is larger than 90%, the field is removed. And removing fields without actual recommended value, such as time, serial number, manual number and the like.

Through the preprocessing, a field data table with recommendation value is obtained.

S2: and designing and implementing a correlation analysis algorithm of each field based on specific values.

Conventional relevance discovery algorithms rely solely on the inherent linkage between fields themselves to determine the magnitude of relevance between certain fields. The invention combines the real customs clearance data with the fields, and judges the correlation between the field and other fields under the condition that the value of a certain field is determined, thereby simulating the continuous interaction process with the user in the actual input scene.

The defined correlation is that given a field A and a value a thereof, when the value of a certain field B is determined, the value of other fields needing to be recorded except A, B is unique or the selected items are minimum, and the correlation between the field A and the field B is called to be maximum.

The input of the algorithm is a historical data set, a user input field A and a field value a, and the output is a field B with the maximum relevance under the current input condition. The algorithm is implemented as follows:

5. segmenting the historical data set according to the input field A and the field value a to obtain a subdata set under the specific input condition;

6. carrying out deduplication processing on the subdata sets;

7. calculating the correlation size of the rest fields needing to be input and the field A, and selecting the field B with the maximum correlation with the field A

8. And outputting the field B.

S3: and designing a tree structure, and constructing and storing a dynamic tree based on the correlation of each field.

The invention realizes the intelligent customs declaration filling method based on historical data mining, designs a tree structure after finding the field correlation relation based on values through S2, and constructs a tree with the historical shortest filling path.

S3.1: tree structure design

The tree structure includes nodes and edges. Wherein the nodes are divided into partition nodes (non-leaf nodes) and recommendation nodes (leaf nodes). Dividing nodes into a field attribute name and the attribute value with the highest occurrence frequency of the field, recommending the nodes to be of a Map structure, and storing the field name and the corresponding attribute value; and all the nodes are connected through edges, and the edges store the attribute values corresponding to the father node fields.

S3.2: dynamic tree generation

Firstly reading in a field data table after preprocessing of S1, defining a first-layer division node (root node) field as a declaration row, respectively defining a second-layer division node and a third-layer division node as a management unit and a commodity name, and connecting each layer of nodes through each filled value of the corresponding field of a father node.

And selecting the nodes of the fourth layer and the later layers according to the corresponding attribute values of the nodes and the edges of the upper layer as input, selecting the field with the maximum correlation as a lower layer node until the value of other input fields is unique or all the fields are input after a certain node, generating a recommended node (leaf node), and storing the other input fields and field values or generating an empty node.

Through the steps, a dynamic tree with the history shortest filling path is generated.

S3.3: dynamic tree storage

And storing the dynamic tree generated by the S3.2 in a MySql database, wherein the table structure is shown as the following table.

TABLE 1 partition node table structure

Name of field	Type of field	Description of the invention
			id	int	Self-growth, main key
Field_name	String	Filling field names
			Best_value	String	Value of field with highest frequency of occurrence
Name	String	Node name, unique
			Level	int	Hierarchy of the tree structure
Agent_code	String	Corresponding declaration row name
			Date	Date	The recording insertion time

TABLE 2 leaf node Table Structure

Name of field	Type of field	Description of the invention
			id	int	Self-growth, main key
Value	String	Field name + field value stitching
			Level	int	Hierarchy of the tree structure
Agent_code	String	Corresponding declaration row name
			Date	Date	The recording insertion time

Watch 3 edge structure

Name of field	Type of field	Description of the invention
			id	int	Self-growth, main key
Father	String	Parent node field name
			Son	String	Child node field name
Value	String	A certain fill value of father node
			Agent_code	String	Corresponding declaration row name
Date	Date	The recording insertion time

S4: and intelligently recommending the filling content according to the generated dynamic tree.

Reading each structure of the dynamic tree stored in the MySql database into a memory, and constructing a corresponding dynamic tree in the memory again. Firstly, searching a dynamic tree according to declaration rows, operation units and commodity names input by a user, then performing depth-first search according to BestValue attributes stored in each node, and outputting all other fields and recommended filling values in order; and interacting with the user, modifying or inputting partial fields, and finally searching and recommending again according to the new input value until the input value meets the input expectation.

S5: and regularly and automatically maintaining and updating the data according to the newly input data.

Firstly, storing the input condition of each time of a user in a database and adding a time tag, then adding new data into a historical data set at fixed time every week according to the time tag, and constructing and storing a new dynamic tree according to the newly generated historical data set, thereby realizing automatic maintenance and updating of the data.

The invention has the following technical effects:

1. the method can be used for intelligently filling and reporting the other filled fields according to the current input content of the user, and the accuracy is high;

2. the customs clearance reporting system has strong generalization and self-learning capabilities, can greatly improve the customs clearance reporting efficiency, and saves manpower and material resources for clearance reporting mechanisms and clearance reporting enterprises;

3. the invention has the function of automatic updating at regular intervals.

Description of the drawings:

in order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive labor.

Fig. 1 is a flowchart of an implementation of a customs declaration and customs clearance intelligent reporting method based on historical data mining according to an embodiment of the present invention.

Fig. 2 is a dynamic tree structure diagram of a customs declaration and customs clearance intelligent reporting method based on historical data mining according to an embodiment of the present invention.

The specific implementation mode is as follows:

the invention provides a customs clearance report intelligent filling method based on the mining of customs clearance history data. The whole scheme comprises:

preprocessing historical data of customs declaration form, merging the header and the body data, and removing irrelevant fields; designing and implementing a correlation analysis algorithm of each field based on a specific value; designing a tree structure, and constructing and storing a dynamic tree based on the correlation of each field; according to the generated dynamic tree, intelligently recommending filling contents; and regularly and automatically maintaining and updating the data according to the newly input data. FIG. one is a flow chart of the present invention.

The correlation analysis algorithm and the dynamic tree designed by the invention can be used for intelligently and dynamically filling the rest filling fields according to the user input, and the accuracy is high. The customs clearance reporting system has strong generalization and self-learning capabilities, can greatly improve the customs clearance reporting efficiency, and saves manpower and material resources for clearance reporting organizations and clearance reporting enterprises.

The embodiments of the invention are illustrated:

take the real data set provided by a customs port in China as an example, wherein the data set comprises the data completely declared by the port in 2015 and 2016. 14,423,930 records in the header data table and 36,578,318 records in the body data table in 2015; in 2016, there are 14,832,866 records in the header data sheet and 38,673,224 records in the body data sheet. The data relates to 7000 declaration lines. The table head data has 43 fields for describing information of a certain order, such as an entrance port, a declaration unit, a transaction mode and the like, and the table body data has 15 fields for describing specific information of various commodities in a certain order, such as a commodity number, a commodity name, a declaration unit price and the like.

Step S1: and preprocessing the historical data of the customs declaration form, merging the header and the body data, and removing irrelevant fields.

S1.1: merging data using Spark distributed computation

And connecting the two tables by using a Spark distributed calculation method and taking the order number as a main key to obtain a data table containing all filled information fields, and storing the data table in the Hive database.

S1.2: and (5) counting the null value condition of each field and removing irrelevant fields.

And counting and removing fields with null value ratio larger than 90%, and removing time, serial numbers, manual numbers and the like to obtain a field data table with recommendation value.

Step S2: and designing and implementing a correlation analysis algorithm of each field based on specific values.

And combining the real customs clearance data with the fields, and judging the correlation between the field and other fields under the condition that the value of the field is determined, thereby simulating the continuous interaction process with the user in the actual input scene.

And (3) realizing a field correlation analysis algorithm, inputting a field name A, a field value a and a data set under current input by a design program, and outputting a field B with the maximum correlation under the specific input condition.

Step S3: and designing a tree structure, and constructing and storing a dynamic tree based on the correlation of each field.

S3.1: tree structure design

The tree structure includes nodes and edges. Wherein the nodes are divided into partition nodes (non-leaf nodes) and recommendation nodes (leaf nodes). Dividing nodes into a field attribute name and the attribute value with the highest occurrence frequency of the field, recommending the nodes to be of a Map structure, and storing the field name and the corresponding attribute value; and all the nodes are connected through edges, and the edges store the attribute values corresponding to the father node fields.

S3.2: dynamic tree generation

And (4) reading in the data table generated in the step (S1), constructing a dynamic tree with the first-layer division node as a reporting line, the second-layer division node as a management unit and the reporting commodity, and connecting the nodes of each layer through each reported value of the corresponding field of the parent node. And (4) selecting the field with the maximum correlation obtained in the step S2 by the rest layer division nodes, and recording the recommendation node as the rest to be unique or null.

S3.3: dynamic tree storage

And respectively storing the generated division nodes, the recommendation nodes and the edges of the dynamic tree into a Mysql division node table, a leaf node table and an edge table.

Step S4: and intelligently recommending the filling content according to the generated dynamic tree.

Through the steps, the dynamic tree obtained based on historical data mining and correlation discovery algorithm is stored in the Mysql database. This step reads the database data into the memory and reconstructs the dynamic tree. Firstly, searching a dynamic tree according to declaration rows, operation units and commodity names input by a user, then performing depth-first search according to BestValue attributes stored in each node, and outputting all other fields and recommended filling values in order; and interacting with the user, modifying or inputting partial fields, and finally searching and recommending again according to the new input value until the input value meets the input expectation.

Step S5: and regularly and automatically maintaining and updating the data according to the newly input data.

Firstly, storing the input condition of each time of the user in a database and adding a time tag, then adding new data into a historical data set at fixed time every week according to the time tag, and constructing and storing a new dynamic tree according to the newly generated historical data set.

Claims (5)

1. A customs declaration and customs clearance intelligent filling method based on historical data mining is characterized by comprising the following steps:

s1: preprocessing historical data of customs declaration form, merging the header and the body data, and removing irrelevant fields:

s1.1: merging data using Spark distributed computation

Real customs declaration data is divided into header data and body data, and the header data and the body data are respectively stored in two data tables, wherein the header data describes information of a certain order, the body data describes specific commodity information in the certain order, a Spark distributed calculation method is used, the two tables are connected by taking an order number as a main key, a data table containing all filled information fields is obtained, and the data table is stored in a Hive database;

s1.2: counting null value of each field, removing irrelevant fields

Counting the ratio of the number of null data pieces in a certain field to the total number of data pieces in the preprocessing process, and removing the field if the ratio is more than 90%; meanwhile, removing the fields without actual recommendation values;

through the preprocessing, a field data table with a recommendation value is obtained;

s2: design and implement each field correlation analysis algorithm based on specific values

By combining the real customs clearance data with the fields, the relevance between the fields and other fields is judged under the condition that the value of one field is determined, so that the continuous interaction process with the user in the actual input scene is simulated;

defining the correlation to be that given a field A and a value a thereof, when the value of a certain field B is determined, the value of other fields needing to be input except A, B is unique or the selected items are minimum, and then the correlation between the field A and the field B is maximum;

the input of the algorithm is a historical data set, a user input field A and a field value a, the output is a field B with the maximum relevance under the current input condition, and the algorithm is realized as follows:

firstly, segmenting a historical data set according to an input field A and a field value a to obtain a subdata set under a specific input condition;

secondly, carrying out duplicate removal processing on the sub-data sets;

thirdly, calculating the correlation size of the rest fields needing to be input and the field A, and selecting the field B with the maximum correlation with the field A;

fourthly, outputting a field B;

s3: designing a tree structure, constructing a dynamic tree based on the correlation of each field and storing

After finding the field correlation relation based on specific values through S2, designing a tree structure, and constructing a tree with the historical shortest filling path:

s3.1: tree structure design

The tree structure comprises nodes and edges, wherein the nodes are divided into a division node and a recommendation node, the division node is a field attribute name and a highest attribute value of the occurrence frequency of the field, and the recommendation node is a Map structure and stores the field name and a corresponding attribute value; all nodes are connected through edges, and the edges store attribute values corresponding to the father node fields;

s3.2: dynamic tree generation

Firstly, reading in a field data table preprocessed by S1, defining a first-layer division node field as a declaration line, respectively defining a second-layer division node and a third-layer division node as a management unit and a commodity name, and connecting each layer of nodes with each node through each declaration value of a field corresponding to a father node;

selecting the nodes of the fourth layer and later layers according to the corresponding attribute values of the nodes and edges of the upper layer as input, selecting the field with the maximum correlation as a node of the lower layer until other input fields have unique values or all fields are input after a certain node, generating a recommended node, and storing the rest input fields and field values or generating an empty node;

through the steps, a dynamic tree with the history shortest filling path is generated;

s3.3: dynamic tree storage

Storing the dynamic tree generated by the S3.2 in a MySql database;

s4: intelligently recommending the filling content according to the generated dynamic tree

Reading each structure of the dynamic tree stored in the MySql database into a memory, and constructing a corresponding dynamic tree in the memory again: firstly, searching a dynamic tree according to declaration rows, operation units and commodity names input by a user, then performing depth-first search according to BestValue attributes stored in each node, and outputting all other fields and recommended filling values in order; interacting with a user, modifying or inputting partial fields, and finally searching and recommending again according to a new input value until the input value is in accordance with input expectation;

s5: regularly and automatically maintaining and updating data according to newly input data

Firstly, storing the input condition of each time of a user in a database and adding a time tag, then adding new data into a historical data set at fixed time every week according to the time tag, and constructing and storing a new dynamic tree according to the newly generated historical data set, thereby realizing automatic maintenance and updating of the data.

2. The customs declaration intelligent reporting method based on the historical data mining as claimed in claim 1, wherein: in S1.1, the certain order information is an entrance and exit port, a declaration unit and a transaction mode.

3. The customs declaration intelligent reporting method based on the historical data mining as claimed in claim 1, wherein: in S1.1, the specific information of the commodity in the certain order is a commodity number, a commodity name, and a declaration unit price.

4. The customs declaration intelligent reporting method based on the historical data mining as claimed in claim 1, wherein: in S1.2, the fields without actual recommended values refer to time, serial numbers and manual numbers.

5. The customs declaration intelligent reporting method based on the historical data mining as claimed in claim 1, wherein: the table structure in S3.3 is shown in the following table:

TABLE 1 partition node table structure

Name of field Type of field Description of the invention id int Self-growth, main key Field_name String Filling field names Best_value String Value of field with highest frequency of occurrence Name String Node name, unique Level int Hierarchy of the tree structure Agent_code String Corresponding declaration row name Date Date The recording insertion time

TABLE 2 leaf node Table Structure

Name of field Type of field Description of the invention id int Self-growth, main key Value String Field name + field value stitching Level int Hierarchy of the tree structure Agent_code String Corresponding declaration row name Date Date The recording insertion time

Watch 3 edge structure

Name of field Type of field Description of the invention id int Self-growth, main key Father String Parent node field name Son String Child node field name Value String A certain fill value of father node Agent_code String Corresponding declaration row name Date Date The recording insertion time

。

Images