CN117056372A

CN117056372A - Logistics data aggregation method, device, equipment and storage medium

Info

Publication number: CN117056372A
Application number: CN202311023629.3A
Authority: CN
Inventors: 王震东; 王建军; 杨周龙
Original assignee: Dongpu Software Co Ltd
Current assignee: Dongpu Software Co Ltd
Priority date: 2023-08-14
Filing date: 2023-08-14
Publication date: 2023-11-14

Abstract

The application relates to the technical field of data processing, in particular to a method, a device, equipment and a storage medium for aggregating logistics data, wherein the method comprises the following steps: acquiring data to be aggregated, and performing marking treatment and writing treatment to obtain preprocessed data; extracting and analyzing file types included in the preprocessed data, judging whether an abnormal part exists or not, and if not, carrying out data aggregation on the preprocessed data according to a preset aggregation rule; judging whether an interception related instruction exists or not, if so, executing an interception action, confirming an interception aggregation layer according to the instruction content of the interception related instruction, and performing interception operation on the data aggregation process according to the interception aggregation layer; if the interception action is not required to be executed, acquiring the aggregated data to generate an aggregation chart; the method disclosed by the application can acquire the data to be aggregated fed back by a plurality of front ends for unified aggregation treatment, is convenient for overall management of the data, can correspondingly generate an aggregation chart according to the aggregated data, and is convenient for consulting the aggregated data.

Description

Logistics data aggregation method, device, equipment and storage medium

Technical Field

The present invention relates to the field of data processing technologies, and in particular, to a method, an apparatus, a device, and a storage medium for aggregating physical distribution data.

Background

With the continuous development of the internet, it is becoming more and more common to store and count service data in a cloud environment, for example, store and count different types of service data such as QPS (query per second) and PV (page view); because the amount of data stored in the cloud environment is large, when the mass business data stored in the cloud environment is counted, the mass business data needs to be aggregated.

In the existing logistics industry, priority schemes are distributed in each distribution network point and are not uniformly uploaded to a headquarter, the problems of high operation cost and poor overall management and control force exist, the data structures of the distribution network points are inconsistent, the aggregation structure lacks uniform standards, and if the data of the distribution network points are uniformly aggregated, the problems of large workload and large aggregation difficulty exist.

It can be seen that there is a need for improvements and improvements in the art.

Disclosure of Invention

In order to overcome the defects of the prior art, the invention aims to provide a method, a device, equipment and a storage medium for polymerizing logistics data, which are used for uniformly polymerizing front-end logistics data based on preset polymerization rules and are convenient for overall management of the logistics data.

The first aspect of the invention provides a method for polymerizing logistics data, which comprises the following steps: when a data aggregation request is fed back, acquiring data to be aggregated, and performing marking processing and writing processing on the data to be aggregated to obtain preprocessed data; extracting and analyzing file types included in the preprocessing data, judging whether an abnormal part exists or not, and if so, storing the preprocessing data into an aggregation database; if the data does not exist, data aggregation is carried out on the preprocessed data according to a preset aggregation rule; judging whether an interception related instruction exists or not, and if so, confirming whether to execute an interception action according to the interception related instruction; if the interception action is required to be executed, confirming an interception aggregation layer according to the instruction content of the interception related instruction, and performing interception operation on the data aggregation process according to the interception aggregation layer; and if the interception action is not required to be executed, acquiring the aggregated data generated after the data aggregation is completed, and generating an aggregation chart according to the aggregated data.

Optionally, in a first implementation manner of the first aspect of the present invention, when the data aggregation request is fed back, obtaining data to be aggregated, and performing marking processing and writing processing on the data to be aggregated to obtain preprocessed data, where the method specifically includes: when a data aggregation request is fed back, acquiring data to be aggregated and splitting the data to be aggregated to obtain a plurality of allocated files; acquiring a single number, a data source and data content of each allocated file, and confirming attribute tags of the allocated files by adopting a regular expression according to the single number and the data source; inputting the acquired data content into a pre-trained recognition model to obtain data values, and respectively confirming priorities of a plurality of allocated files according to the data values; and calling a CSV interface to convert the allocated file into a CSV file to obtain the preprocessing data.

Optionally, in a second implementation manner of the first aspect of the present invention, the extracting and analyzing the file type included in the preprocessed data, judging whether an abnormal piece exists, if so, storing the preprocessed data in an aggregation database, specifically includes: acquiring each allocated file and corresponding file description thereof included in the preprocessing data; inputting the file description into a pre-trained word segmentation model to obtain a first word segmentation result; matching a preset file type table and an obtained first word segmentation result by adopting a regular expression so as to confirm the file type; judging whether an abnormal part exists according to the file type, wherein the abnormal part comprises a wrong part and a retreated part; when an abnormal part exists, the preprocessing data is stored in an aggregation database.

Optionally, in a third implementation manner of the first aspect of the present invention, if the pre-processing data does not exist, the data aggregation is performed on the pre-processing data according to a preset aggregation rule, which specifically includes: if the file is not present, acquiring a plurality of allocated files and corresponding single numbers thereof included in the preprocessing data; judging whether the allocated files with the consistent single numbers exist or not based on a violence method, and if so, acquiring the establishment time of the allocated files with the consistent single numbers; confirming time sequence of the establishment time based on an oracle function, saving the allocated file with the time closest to the current time and deleting other allocated files with consistent single numbers; acquiring a preset aggregation rule, wherein the aggregation rule comprises an aggregation sequence, and aggregating the preprocessed data according to the aggregation sequence.

Optionally, in a fourth implementation manner of the first aspect of the present invention, the determining whether an interception related instruction exists, if so, determining whether to execute the interception action according to the interception related instruction, includes: judging whether an interception related instruction exists or not, wherein the interception related instruction comprises an interception instruction and an interception cancellation instruction; when only the interception instruction is received, executing an interception action; when the interception instruction and the cancellation interception instruction coexist, comparing the receiving time of the interception instruction with the receiving time of the cancellation interception instruction; when the receiving time of the interception instruction is cancelled before the receiving time of the interception instruction, the interception action is not executed, otherwise, the interception action is executed.

Optionally, in a fifth implementation manner of the first aspect of the present invention, if the intercepting action is to be performed, the intercepting aggregation layer is confirmed according to the instruction content of the intercepting related instruction, and the intercepting operation is performed on the data aggregation process according to the intercepting aggregation layer, which specifically includes: when the interception action needs to be executed, acquiring the instruction content of the interception instruction; inputting instruction content into a pre-trained word segmentation model to obtain a second word segmentation result; matching a preset interception type configuration table and an obtained second word segmentation result by adopting a regular expression so as to confirm an interception aggregation layer; and intercepting the data aggregation process according to the interception aggregation layer.

Optionally, in a sixth implementation manner of the first aspect of the present invention, if the intercepting action is not required to be performed, acquiring aggregated data generated after the data aggregation is completed, and generating an aggregation chart according to the aggregated data, specifically includes: acquiring aggregated data generated after data aggregation is completed when the interception action is not required to be executed; acquiring a chart attribute to be generated, wherein the chart attribute comprises a chart type, a chart name and a coordinate axis format of a chart; extracting chart data from the aggregated data by adopting a KMP algorithm according to the chart attribute; an aggregate chart is generated from the chart data based on the echartis function.

The second aspect of the present invention provides a device for aggregating physical distribution data, comprising: the acquisition module is used for acquiring data to be aggregated when a data aggregation request is fed back, and performing marking processing and writing processing on the data to be aggregated to obtain preprocessed data; the analysis module is used for extracting and analyzing file types included in the preprocessed data, judging whether an abnormal part exists or not, and storing the preprocessed data into the aggregation database if the abnormal part exists; the aggregation module is used for carrying out data aggregation on the preprocessed data according to a preset aggregation rule if the preprocessed data does not exist; the judging module is used for judging whether an interception related instruction exists or not, and if so, confirming whether to execute the interception action according to the interception related instruction; the interception module is used for confirming an interception aggregation layer according to the instruction content of the interception related instruction if the interception action is required to be executed, and performing interception operation on the data aggregation process according to the interception aggregation layer; and the generation module is used for acquiring the aggregated data generated after the data aggregation is completed if the interception action is not required to be executed, and generating an aggregation chart according to the aggregated data.

Optionally, in a first implementation manner of the second aspect of the present invention, the acquiring module includes: the splitting unit is used for acquiring data to be aggregated and splitting the data to be aggregated when a data aggregation request is fed back to obtain a plurality of allocated files; the first acquisition unit is used for acquiring the single number, the data source and the data content of each allocated file, and confirming the attribute label of the allocated file by adopting a regular expression according to the single number and the data source; the identification unit is used for inputting the acquired data content into a pre-trained identification model to obtain data values, and respectively confirming the priorities of a plurality of allocated files according to the data values; and the conversion unit is used for calling the CSV interface to convert the allocated file into the CSV file to obtain the preprocessing data.

Optionally, in a second implementation manner of the second aspect of the present invention, the analysis module includes: the second acquisition unit is used for acquiring each allocated file and the corresponding file description included in the preprocessing data; the first word segmentation unit is used for inputting the file description into a pre-trained word segmentation model to obtain a first word segmentation result; the first matching unit is used for matching a preset file type table and an obtained first word segmentation result by adopting a regular expression so as to confirm the file type; the first judging unit is used for judging whether an abnormal part exists according to the file type, and the abnormal part comprises a wrong part and a returning part; and the storage unit is used for storing the preprocessing data to the aggregation database when the abnormal part exists.

Optionally, in a third implementation manner of the second aspect of the present invention, the aggregation module includes: a third obtaining unit, configured to obtain, if the plurality of allocation files and corresponding single numbers included in the preprocessed data do not exist; the second judging unit is used for judging whether the allocated files with the consistent single numbers exist or not based on a violence method, and if so, acquiring the establishment time of the allocated files with the consistent single numbers; the deleting unit is used for confirming the time sequence of the establishment time based on the oracle function, saving the allocated file with the time closest to the current time and deleting other allocated files with consistent single numbers; the aggregation unit is used for acquiring a preset aggregation rule, wherein the aggregation rule comprises an aggregation sequence, and the preprocessed data is aggregated according to the aggregation sequence.

Optionally, in a fourth implementation manner of the second aspect of the present invention, the determining module includes: the third judging unit is used for judging whether an interception related instruction exists or not, wherein the interception related instruction comprises an interception instruction and an interception cancellation instruction; the first interception unit is used for executing interception actions when only interception instructions exist; the comparison unit is used for comparing the receiving time of the interception instruction and the receiving time of the cancellation interception instruction when the interception instruction and the cancellation interception instruction coexist; and the second interception unit is used for not executing the interception action when the receiving time of the interception instruction is earlier than the receiving time of the interception instruction, and otherwise, executing the interception action.

Optionally, in a fifth implementation manner of the second aspect of the present invention, the interception module includes: a fourth obtaining unit, configured to obtain instruction content of the interception instruction when the interception action needs to be performed; the second word segmentation unit is used for inputting the instruction content into a pre-trained word segmentation model to obtain a second word segmentation result; the second matching unit is used for matching a preset interception type configuration table and an obtained second word result by adopting a regular expression so as to confirm an interception aggregation layer; and the third interception unit is used for intercepting the data aggregation process according to the interception aggregation layer.

Optionally, in a sixth implementation manner of the second aspect of the present invention, the generating module includes: a fifth acquisition unit for acquiring aggregated data generated after completion of data aggregation when the interception action is not required to be performed; a sixth obtaining unit, configured to obtain a chart attribute to be generated, where the chart attribute includes a chart type, a chart name, and a coordinate axis format of the chart; the extraction unit is used for extracting chart data from the aggregated data by adopting a KMP algorithm according to the chart attribute; and the generating unit is used for generating an aggregation chart according to the chart data based on the Echarts function.

A third aspect of the present application provides a logistics data aggregation apparatus, comprising: a memory and at least one processor, the memory having instructions stored therein; at least one of the processors invokes the instructions in the memory to cause the logistics data aggregation apparatus to perform the steps of the logistics data aggregation method of any one of the preceding claims.

A fourth aspect of the present application provides a computer readable storage medium having instructions stored thereon which when executed by a processor perform the steps of the method of aggregation of logistical data as described in any one of the preceding claims.

In the technical scheme, preprocessing data is obtained by acquiring data to be aggregated, performing marking processing and writing processing; extracting and analyzing file types included in the preprocessed data, judging whether an abnormal part exists or not, and if not, carrying out data aggregation on the preprocessed data according to a preset aggregation rule; judging whether an interception related instruction exists or not, if so, executing an interception action, confirming an interception aggregation layer according to the instruction content of the interception related instruction, and performing interception operation on the data aggregation process according to the interception aggregation layer; if the interception action is not required to be executed, acquiring the aggregated data to generate an aggregation chart; according to the method disclosed by the application, the data to be aggregated fed back by a plurality of front ends can be obtained, and unified aggregation treatment is carried out on the data to be aggregated based on the preset aggregation rule, so that the aggregation difficulty is reduced, and the overall management of logistics data is facilitated; in addition, the aggregation chart can be correspondingly generated according to the aggregated data, the aggregated data can be intuitively presented, the manager can conveniently review the aggregated data, and the use experience of the manager is improved.

Drawings

FIG. 1 is a first flowchart of a method for aggregating physical distribution data according to an embodiment of the present invention;

FIG. 2 is a second flow chart of a method for aggregating physical distribution data according to an embodiment of the present invention;

FIG. 3 is a third flow chart of a method for aggregating physical distribution data according to an embodiment of the present invention;

FIG. 4 is a fourth flowchart of a method for aggregating physical distribution data according to an embodiment of the present invention;

FIG. 5 is a fifth flowchart of a method for aggregating physical distribution data according to an embodiment of the present invention;

FIG. 6 is a sixth flowchart of a method for aggregating physical distribution data according to an embodiment of the present invention;

FIG. 7 is a seventh flowchart of a method for aggregating physical distribution data according to an embodiment of the present invention;

FIG. 8 is a schematic structural diagram of a device for aggregating physical distribution data according to an embodiment of the present invention;

fig. 9 is a schematic structural diagram of a physical distribution data aggregation device according to an embodiment of the present invention.

Detailed Description

The present invention provides a method, apparatus, device and storage medium for aggregating physical distribution data, and the terms "first", "second", "third", "fourth", etc. (if any) in the description and claims of the present invention and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments described herein may be implemented in other sequences than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed or inherent to such process, method, article, or apparatus.

For ease of understanding, a specific flow of an embodiment of the present invention is described below, referring to fig. 1, and an embodiment of a method for aggregating physical distribution data in an embodiment of the present invention includes:

101. when a data aggregation request is fed back, acquiring data to be aggregated, and performing marking processing and writing processing on the data to be aggregated to obtain preprocessed data;

in this embodiment, the front end feeds back the data aggregation request, and at the same time, the front end that feeds back the data aggregation request may include multiple front ends, and after receiving the data aggregation request, the aggregation system acquires the logistics data fed back by multiple front ends, that is, the data to be aggregated, so as to uniformly aggregate the data to be aggregated of the multiple front ends; the aggregation system includes an aggregation database.

102. Extracting and analyzing file types included in the preprocessing data, judging whether an abnormal part exists or not, and if so, storing the preprocessing data into an aggregation database;

in this embodiment, the preprocessing data includes a plurality of allocated files, each allocated file corresponds to a file type, and the file type is used to determine whether any allocated file is an abnormal part, and when any allocated file is an abnormal part, data aggregation is not performed.

103. If the data does not exist, data aggregation is carried out on the preprocessed data according to a preset aggregation rule;

in this embodiment, the data aggregation is performed on the preprocessed data based on the preset aggregation rule, so that the data aggregation process is standardized, that is, the aggregation difficulty is reduced and the data aggregation effect is improved even if the aggregation structure is standardized.

104. Judging whether an interception related instruction exists or not, and if so, confirming whether to execute an interception action according to the interception related instruction;

105. if the interception action is required to be executed, confirming an interception aggregation layer according to the instruction content of the interception related instruction, and performing interception operation on the data aggregation process according to the interception aggregation layer;

in this embodiment, in the process of executing data aggregation, if an interception related instruction occurs, whether an interception action is required is first determined, so that unnecessary data aggregation actions are avoided, the possibility of occurrence of unnecessary data is reduced, and the data processing effect and the data processing flexibility are improved.

106. If the interception action is not required to be executed, acquiring the aggregated data generated after the data aggregation is completed, and generating an aggregation chart according to the aggregated data;

in this embodiment, if no interception related instruction exists in the process of executing data aggregation, the data aggregation is normally performed until all the data to be aggregated are aggregated; after the aggregated data is generated, the relevant data can be extracted from the aggregated data according to the chart generation instruction fed back by the front end to generate an aggregated chart, the aggregated data is intuitively and clearly presented, and the use experience of a user is improved.

The application discloses a logistics data aggregation method, which comprises the steps of obtaining data to be aggregated, and performing marking treatment and writing treatment to obtain preprocessed data; extracting and analyzing file types included in the preprocessed data, judging whether an abnormal part exists or not, and if not, carrying out data aggregation on the preprocessed data according to a preset aggregation rule; judging whether an interception related instruction exists or not, if so, executing an interception action, confirming an interception aggregation layer according to the instruction content of the interception related instruction, and performing interception operation on the data aggregation process according to the interception aggregation layer; if the interception action is not required to be executed, acquiring the aggregated data to generate an aggregation chart; according to the method disclosed by the application, the data to be aggregated fed back by a plurality of front ends can be obtained, and unified aggregation treatment is carried out on the data to be aggregated based on the preset aggregation rule, so that the aggregation difficulty is reduced, and the overall management of logistics data is facilitated; in addition, the aggregation chart can be correspondingly generated according to the aggregated data, the aggregated data can be intuitively presented, the manager can conveniently review the aggregated data, and the use experience of the manager is improved.

Referring to fig. 2, a second embodiment of a method for aggregating physical distribution data according to an embodiment of the present application includes:

201. When a data aggregation request is fed back, acquiring data to be aggregated and splitting the data to be aggregated to obtain a plurality of allocated files;

in this embodiment, the data to be aggregated includes one or more compressed packets, where the compressed packets include one or more allocated files, and after decompression and decryption processing are performed on the compressed packets, a plurality of allocated files may be obtained.

202. Acquiring a single number, a data source and data content of each allocated file, and confirming attribute tags of the allocated files by adopting a regular expression according to the single number and the data source;

in this embodiment, the single number and the data source are determined as unique values, where the data source refers to the front end, i.e. the front end of which allocation network point the data to be aggregated originates from; further, a Grep command can be adopted to realize confirmation of the allocated file attribute tag by matching with a preset file type table, and the Grep command supports three regular expression grammars, namely Basic, extended and Perl-compatible; when the regular expression type is not specified, the grep command defaults the search pattern to a basic regular expression; when multiple matching modes need to be searched, OR (alternation) operators can be used, different matching items can be specified through OR operators, the matching items can be text strings OR expression sets, and regular expressions are bracketed through single quotation marks; the Grep command may be a few bits in a single number

The number of each file label is correspondingly set in the file type table, # [ wid ] + [ \ swd ]; the regular expression uses meta characters #, w, d, +, and \s; wherein # denotes matching a well number character, w and d denote matching any letter or number character, respectively, + denotes matching one or more preceding characters or character combinations, # denotes matching zero or more characters or character combinations, \s denotes matching blank characters such as space characters or tab characters.

203. Inputting the acquired data content into a pre-trained recognition model to obtain data values, and respectively confirming priorities of a plurality of allocated files according to the data values;

in this embodiment, after the recognition model is trained based on the AHP-fuzzy comprehensive evaluation method and the data content is input into the recognition model, the sensitive fields included in the data content can be confirmed, the data value of the data content is confirmed according to the specific content of the sensitive fields and the number of the sensitive fields, and the priority of the allocated files is confirmed according to the data value.

204. Calling a CSV interface to convert the allocated file into a CSV file to obtain preprocessing data;

In this embodiment, the generated CSV file is stored into a full data table of the aggregation system.

Referring to fig. 3, a third embodiment of a method for aggregating physical distribution data according to an embodiment of the present invention includes:

301. acquiring each allocated file and corresponding file description thereof included in the preprocessing data;

in this embodiment, the allocated file corresponds to a file log, and the file description is recorded in the file log.

302. Inputting the file description into a pre-trained word segmentation model to obtain a first word segmentation result;

in the embodiment, the word segmentation model realizes training based on a Chinese word segmentation algorithm, and specifically, performs word segmentation on the name by adopting a forward maximum matching method; the forward maximum matching method (FMM) is a word segmentation algorithm based on a word list, for an input descending message text sequence, a word with the largest length at the current position is segmented by a greedy algorithm, word-direct and a character string s of the word to be segmented are firstly set, the length of the longest word in the word-direct is calculated to be m, a word length segment with the largest length is selected from the first position of the character string, and if the length of the character string is less than the maximum word length, all the character strings are selected; judging whether the selected character string segment is in a word stock, if so, separating the word, and if not, starting from the right, reducing one character one by one until the segment is ended in a dictionary or only the last character is remained; and obtaining a plurality of file descriptors by a forward maximum matching method, namely, the first word segmentation result consists of one or more file descriptors.

303. Matching a preset file type table and an obtained first word segmentation result by adopting a regular expression so as to confirm the file type;

in this embodiment, file types are set in the file type table, each file type corresponds to one or more descriptors, and regular expressions are adopted to match the preset descriptors in the file type table and the file descriptors included in the first word segmentation result so as to confirm the file type.

304. Judging whether an abnormal part exists according to the file type, wherein the abnormal part comprises a wrong part and a retreated part;

in this embodiment, when the file descriptor includes 98, 99, refund, 96 or error, an exception is indicated.

305. When an abnormal part exists, storing the preprocessed data to an aggregation database;

in this embodiment, when an abnormal piece exists, data aggregation is not performed, and the preprocessed data is directly stored in an aggregation database of the aggregation system.

Referring to fig. 4, a fourth embodiment of a method for aggregating physical distribution data according to an embodiment of the present invention includes:

401. if the file is not present, acquiring a plurality of allocated files and corresponding single numbers thereof included in the preprocessing data;

402. judging whether the allocated files with the consistent single numbers exist or not based on a violence method, and if so, acquiring the establishment time of the allocated files with the consistent single numbers;

In this embodiment, the violent method is also called enumeration method, which uses the characteristics of fast computing speed and high accuracy of computer to test all possible situations of the problem to be solved, and find out the answer meeting the requirement, so that the enumeration method is comprehensive in terms of replacing the answer by sacrificing time.

403. Confirming time sequence of the establishment time based on an oracle function, saving the allocated file with the time closest to the current time and deleting other allocated files with consistent single numbers;

in this embodiment, the order of creation of the allocated files is confirmed based on the oracle time comparison function, and for a plurality of allocated files with the same single number, only the allocated file with the creation time closest to the current time is reserved, and other allocated files with the same single number are deleted, so that the data amount of data aggregation can be reduced, and the efficiency of data aggregation is improved.

404. Acquiring a preset aggregation rule, wherein the aggregation rule comprises an aggregation sequence, and aggregating the preprocessed data according to the aggregation sequence;

in this embodiment, the polymerization sequence may be: address zone-one-section code-destination node-destination division-dispatch code home site-end post-cabinet-other-service type-destination allocation-originating node; the preprocessing data are sequentially subjected to data aggregation according to the aggregation sequence, when the aggregation of the destination nodes is completed, the destination nodes with the highest priority are acquired when the destination nodes are required to be aggregated, the corresponding upper-level stations are searched by the association basic information company table, whether the upper-level stations are consistent with the destination nodes with the aggregated front-end data or not is judged, and if so, the destination nodes are normally aggregated; if the data are inconsistent, the target subsection aggregation data are set to 0, and the dispatch code belongs to the site field 0; and after the target subsection is aggregated, judging whether the six-bit codes of the target subsection and the dispatch code attribution site are consistent, if so, normally aggregating the dispatch code attribution site, and if not, locating the dispatch code attribution site to be 0.

Referring to fig. 5, a fifth embodiment of a method for aggregating physical distribution data according to an embodiment of the present invention includes:

501. judging whether an interception related instruction exists or not, wherein the interception related instruction comprises an interception instruction and an interception cancellation instruction;

502. when only the interception instruction is received, executing an interception action;

in the present embodiment, when only the intercept instruction is canceled, the intercept action is not performed.

503. When the interception instruction and the cancellation interception instruction coexist, comparing the receiving time of the interception instruction with the receiving time of the cancellation interception instruction;

504. when the receiving time of the interception instruction is cancelled and is earlier than that of the interception instruction, the interception action is not executed, otherwise, the interception action is executed;

in the embodiment, in the data aggregation process, whether to execute the interception action is confirmed according to the interception related instruction, so that the timely interception of the data aggregation process is realized, generation of aggregated data needing to be invalidated is avoided, and the workload and the storage load of an aggregation system are reduced.

Referring to fig. 6, a sixth embodiment of a method for aggregating physical distribution data according to an embodiment of the present invention includes:

601. when the interception action needs to be executed, acquiring the instruction content of the interception instruction;

602. inputting instruction content into a pre-trained word segmentation model to obtain a second word segmentation result;

In this embodiment, the second word segmentation result includes one or more interception keywords.

603. Matching a preset interception type configuration table and an obtained second word segmentation result by adopting a regular expression so as to confirm an interception aggregation layer;

in this embodiment, a plurality of aggregation layer keywords are set in the interception type configuration table, and the interception aggregation layer can be confirmed by matching the aggregation layer keywords and the interception keywords in the interception type configuration table through the regular expression.

604. Intercepting the data aggregation process according to the interception aggregation layer;

in this embodiment, for example, when the interception keyword includes 5 or 10 or 13 or 14 or 15 or 17 or 20 or 24, the interception and aggregation layer is to sort to the website, and when the interception and aggregation layer is to sort to the website, the end salesman, the end post, the end cabinet and the end other fields are set to 0, data aggregation is not performed, the service category replaces the secondary priority, and the other fields are aggregated according to the sorting type priority; when the interception aggregation layer is used for sorting to a salesman, the terminal post, the terminal cabinet and other fields of the terminal are set to 0, data aggregation is not performed, the service type replaces secondary priority, and other fields are aggregated according to sorting type priority; when the interception aggregation layer intercepts the front end, all fields do not participate in aggregation, and the preprocessed data is directly stored in an aggregation database.

Referring to fig. 7, a seventh embodiment of a method for aggregating physical distribution data according to an embodiment of the present invention includes:

701. acquiring aggregated data generated after data aggregation is completed when the interception action is not required to be executed;

in this embodiment, when no interception related instruction occurs in the data aggregation process, the data aggregation process is performed normally, and after the aggregation of all the preprocessed data is completed according to the preset aggregation rule, the aggregated data is output.

702. Acquiring a chart attribute to be generated, wherein the chart attribute comprises a chart type, a chart name and a coordinate axis format of a chart;

in this embodiment, the coordinate axis format of the chart refers to the abscissa and the ordinate of the chart and the corresponding data types thereof.

703. Extracting chart data from the aggregated data by adopting a KMP algorithm according to the chart attribute;

in this embodiment, extraction of chart data is implemented by adopting a KMP algorithm, specifically, the type of data to be extracted is confirmed according to the chart name, and the name corresponding to the specific data to be extracted is confirmed according to the coordinate axis format of the chart; the basic idea of KMP algorithm is: starting from the first character of the main string test and the pattern string pattern, comparing the characters of the two character strings one by one, if a certain character is not matched, backtracking the main string to the second character, backtracking the sub-string to the first character, then comparing one by one, if a certain character is not matched, backtracking the main string to the third character, backtracking the sub-string to the first character, then comparing one by one, and circulating until all the sub-string characters are successfully matched, thus completing the matching of the abnormal type keywords and word segmentation results.

704. Generating an aggregation chart according to chart data based on the Echarts function;

in this embodiment, according to the extracted chart data, an aggregation chart can be generated by substituting the Option of the return value into the corresponding parameter interface based on the echartis function in combination with the import component.

The method for aggregating physical distribution data in the embodiment of the present invention is described above, and the apparatus for aggregating physical distribution data in the embodiment of the present invention is described below, referring to fig. 8, where an embodiment of the apparatus for aggregating physical distribution data in the embodiment of the present invention includes:

the acquiring module 801 is configured to acquire data to be aggregated when a data aggregation request is fed back, and perform marking processing and writing processing on the data to be aggregated to obtain preprocessed data; the analysis module 802 is configured to extract and analyze a file type included in the preprocessed data, determine whether an abnormal piece exists, and store the preprocessed data to the aggregation database if the abnormal piece exists; the aggregation module 803 is configured to aggregate the preprocessed data according to a preset aggregation rule if the preprocessed data does not exist; a judging module 804, configured to judge whether an interception related instruction exists, and if so, confirm whether to execute the interception action according to the interception related instruction; the interception module 805 is configured to, if an interception action is required to be performed, confirm an interception aggregation layer according to instruction content of an interception related instruction, and perform an interception operation on a data aggregation process according to the interception aggregation layer; and a generating module 806, configured to obtain the aggregated data generated after the data aggregation is completed, and generate an aggregation chart according to the aggregated data if the intercepting action is not required to be performed.

In this embodiment, the obtaining module 801 includes: the splitting unit 8011 is configured to obtain data to be aggregated and perform splitting processing when a data aggregation request is fed back, so as to obtain a plurality of allocated files; the first obtaining unit 8012 is configured to obtain a single number, a data source and data content of each allocated file, and confirm an attribute tag of the allocated file by adopting a regular expression according to the single number and the data source; the identification unit 8013 is configured to input the acquired data content into a pre-trained identification model, obtain a data value, and respectively confirm priorities of a plurality of allocated files according to the data value; and the conversion unit 8014 is configured to call the CSV interface to convert the allocated file into a CSV file, thereby obtaining the preprocessed data.

In this embodiment, the analysis module 802 includes: a second obtaining unit 8021, configured to obtain each allocated file and a file description corresponding to each allocated file included in the preprocessed data; the first word segmentation unit 8022 is used for inputting the file description into a pre-trained word segmentation model to obtain a first word segmentation result; a first matching unit 8023, configured to match a preset file type table and an obtained first word segmentation result by using a regular expression, so as to confirm a file type; the first judging unit 8024 is used for judging whether an abnormal part exists according to the file type, wherein the abnormal part comprises a wrong part and a returning part; a storage unit 8025 for storing the preprocessed data to the syndication database when there is an abnormal piece.

In this embodiment, the aggregation module 803 includes: a third obtaining unit 8031, configured to obtain, if the plurality of allocated files and corresponding single numbers included in the preprocessed data do not exist; a second judging unit 8032, configured to judge whether a file with a consistent single number exists based on a violence method, and if so, obtain an establishment time of the file with the consistent single number; the pruning unit 8033 is used for confirming the time sequence of the establishment time based on the oracle function, saving the allocated file with the time closest to the current time and deleting other allocated files with consistent single numbers; the aggregation unit 8034 is configured to obtain a preset aggregation rule, where the aggregation rule includes an aggregation order, and aggregate the preprocessed data according to the aggregation order.

In this embodiment, the determining module 804 includes: a third judging unit 8041, configured to judge whether an interception-related instruction exists, where the interception-related instruction includes an interception instruction and a cancellation interception instruction; a first interception unit 8042 for performing an interception action when only an interception instruction is received; a comparing unit 8043 for comparing the reception time of the interception instruction and the reception time of the cancellation interception instruction when the interception instruction and the cancellation interception instruction coexist; the second intercepting unit 8044 is configured to not perform the intercepting action when the receiving time of the intercepting instruction is earlier than the receiving time of the intercepting instruction, and otherwise, perform the intercepting action.

In this embodiment, the interception module 805 includes: a fourth obtaining unit 8051, configured to obtain instruction content of an interception instruction when the interception action needs to be performed; the second word segmentation unit 8052 is used for inputting the instruction content into a pre-trained word segmentation model to obtain a second word segmentation result; a second matching unit 8053, configured to match a preset interception type configuration table and an obtained second word result by using a regular expression, so as to confirm an interception aggregation layer; and a third interception unit 8054, configured to intercept the data aggregation process according to the interception aggregation layer.

In this embodiment, the generating module 806 includes: a fifth acquiring unit 8061 configured to acquire, when the intercepting action does not need to be performed, aggregated data generated after the data aggregation is completed; a sixth obtaining unit 8062, configured to obtain a chart attribute to be generated, where the chart attribute includes a chart type, a chart name, and a coordinate axis format of the chart; an extracting unit 8063, configured to extract chart data from the aggregated data by adopting a KMP algorithm according to the chart attribute; a generating unit 8064 for generating an aggregation chart from chart data based on the echartis function.

The above fig. 8 describes the physical distribution data aggregation device in the embodiment of the present invention in detail from the point of view of the modularized functional entity, and the following describes the physical distribution data aggregation device in the embodiment of the present invention in detail from the point of view of hardware processing.

Fig. 9 is a schematic structural diagram of a physical distribution data aggregation apparatus according to an embodiment of the present invention, where the physical distribution data aggregation apparatus 900 may have relatively large differences according to different configurations or performances, and may include one or more processors (central processing units, CPU) 910 (e.g., one or more processors) and a memory 920, and one or more storage media 930 (e.g., one or more mass storage devices) storing application programs 933 or data 932. Wherein the memory 920 and storage medium 930 may be transitory or persistent storage. The program stored on the storage medium 930 may include one or more modules (not shown), each of which may include a series of instruction operations for the logistics data aggregation apparatus 900. Still further, the processor 910 may be configured to communicate with a storage medium 930 and execute a series of instruction operations in the storage medium 930 on the logistics data aggregation apparatus 900 to implement the steps of the logistics data aggregation method provided by the above-described method embodiments.

The logistics data aggregation apparatus 900 may also include one or more power supplies 940, one or more wired or wireless network interfaces 950, one or more input/output interfaces 960, and/or one or more operating systems 931, such as Windows Serve, mac OS X, unix, linux, freeBSD, etc. It will be appreciated by those skilled in the art that the illustrated construction of the physical distribution data aggregation device of the present application is not limiting and may include more or less components than illustrated, or may be combined with certain components, or may be arranged with different components.

The present application also provides a computer readable storage medium, which may be a non-volatile computer readable storage medium, and which may also be a volatile computer readable storage medium, having stored therein instructions that, when executed on a computer, cause the computer to perform the steps of the method of logistical data aggregation.

It will be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working process of the system or apparatus and unit described above may refer to the corresponding process in the foregoing method embodiment, which is not repeated herein.

The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied essentially or in part or all of the technical solution or in part in the form of a software product stored in a storage medium, including instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a read-only memory (ROM), a random access memory (random access memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

Finally, it should be noted that: the foregoing is merely a preferred example of the present invention, and the present invention is not limited thereto, but it is to be understood that modifications and equivalents of some of the technical features described in the foregoing embodiments may be made by those skilled in the art, although the present invention has been described in detail with reference to the foregoing embodiments. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method of aggregating physical distribution data, comprising:

when a data aggregation request is fed back, acquiring data to be aggregated, and performing marking processing and writing processing on the data to be aggregated to obtain preprocessed data;

extracting and analyzing file types included in the preprocessing data, judging whether an abnormal part exists or not, and if so, storing the preprocessing data into an aggregation database;

if the data does not exist, data aggregation is carried out on the preprocessed data according to a preset aggregation rule;

judging whether an interception related instruction exists or not, and if so, confirming whether to execute an interception action according to the interception related instruction;

if the interception action is required to be executed, confirming an interception aggregation layer according to the instruction content of the interception related instruction, and performing interception operation on the data aggregation process according to the interception aggregation layer;

and if the interception action is not required to be executed, acquiring the aggregated data generated after the data aggregation is completed, and generating an aggregation chart according to the aggregated data.

2. The method for aggregating physical distribution data according to claim 1, wherein when the request for aggregating the physical distribution data is fed back, obtaining the data to be aggregated, and performing marking processing and writing processing on the data to be aggregated to obtain the preprocessed data, the method specifically comprises:

When a data aggregation request is fed back, acquiring data to be aggregated and splitting the data to be aggregated to obtain a plurality of allocated files;

acquiring a single number, a data source and data content of each allocated file, and confirming attribute tags of the allocated files by adopting a regular expression according to the single number and the data source;

inputting the acquired data content into a pre-trained recognition model to obtain data values, and respectively confirming priorities of a plurality of allocated files according to the data values;

and calling a CSV interface to convert the allocated file into a CSV file to obtain the preprocessing data.

3. The method for aggregating physical distribution data according to claim 1, wherein the extracting and analyzing the file types included in the preprocessed data, judging whether an abnormal piece exists, and if so, storing the preprocessed data in an aggregation database, specifically includes:

acquiring each allocated file and corresponding file description thereof included in the preprocessing data;

inputting the file description into a pre-trained word segmentation model to obtain a first word segmentation result;

matching a preset file type table and an obtained first word segmentation result by adopting a regular expression so as to confirm the file type;

judging whether an abnormal part exists according to the file type, wherein the abnormal part comprises a wrong part and a retreated part;

When an abnormal part exists, the preprocessing data is stored in an aggregation database.

4. The method for aggregating physical distribution data according to claim 1, wherein if the physical distribution data does not exist, the method for aggregating the pretreated data according to a preset aggregation rule specifically comprises:

if the file is not present, acquiring a plurality of allocated files and corresponding single numbers thereof included in the preprocessing data;

judging whether the allocated files with the consistent single numbers exist or not based on a violence method, and if so, acquiring the establishment time of the allocated files with the consistent single numbers;

confirming time sequence of the establishment time based on an oracle function, saving the allocated file with the time closest to the current time and deleting other allocated files with consistent single numbers;

acquiring a preset aggregation rule, wherein the aggregation rule comprises an aggregation sequence, and aggregating the preprocessed data according to the aggregation sequence.

5. The method for aggregating physical distribution data according to claim 1, wherein the determining whether an interception related instruction exists, and if so, determining whether to execute the interception according to the interception related instruction, specifically comprises:

judging whether an interception related instruction exists or not, wherein the interception related instruction comprises an interception instruction and an interception cancellation instruction;

When only the interception instruction is received, executing an interception action;

when the interception instruction and the cancellation interception instruction coexist, comparing the receiving time of the interception instruction with the receiving time of the cancellation interception instruction;

when the receiving time of the interception instruction is cancelled before the receiving time of the interception instruction, the interception action is not executed, otherwise, the interception action is executed.

6. The method for aggregating physical distribution data according to claim 5, wherein if the intercepting operation is required, the intercepting aggregation layer is confirmed according to the instruction content of the intercepting related instruction, and the intercepting operation is performed according to the intercepting aggregation layer, which specifically comprises:

when the interception action needs to be executed, acquiring the instruction content of the interception instruction;

inputting instruction content into a pre-trained word segmentation model to obtain a second word segmentation result;

matching a preset interception type configuration table and an obtained second word segmentation result by adopting a regular expression so as to confirm an interception aggregation layer;

and intercepting the data aggregation process according to the interception aggregation layer.

7. The method for aggregating physical distribution data according to claim 1, wherein if the intercepting action is not required, acquiring the aggregated data generated after the data aggregation is completed, and generating an aggregation chart according to the aggregated data, specifically comprising:

Acquiring aggregated data generated after data aggregation is completed when the interception action is not required to be executed;

acquiring a chart attribute to be generated, wherein the chart attribute comprises a chart type, a chart name and a coordinate axis format of a chart;

extracting chart data from the aggregated data by adopting a KMP algorithm according to the chart attribute;

an aggregate chart is generated from the chart data based on the echartis function.

8. A device for aggregating physical distribution data, comprising:

the acquisition module is used for acquiring data to be aggregated when a data aggregation request is fed back, and performing marking processing and writing processing on the data to be aggregated to obtain preprocessed data;

the analysis module is used for extracting and analyzing file types included in the preprocessed data, judging whether an abnormal part exists or not, and storing the preprocessed data into the aggregation database if the abnormal part exists;

the aggregation module is used for carrying out data aggregation on the preprocessed data according to a preset aggregation rule if the preprocessed data does not exist;

the judging module is used for judging whether an interception related instruction exists or not, and if so, confirming whether to execute the interception action according to the interception related instruction;

the interception module is used for confirming an interception aggregation layer according to the instruction content of the interception related instruction if the interception action is required to be executed, and performing interception operation on the data aggregation process according to the interception aggregation layer;

And the generation module is used for acquiring the aggregated data generated after the data aggregation is completed if the interception action is not required to be executed, and generating an aggregation chart according to the aggregated data.

9. A logistics data aggregation apparatus, characterized in that the logistics data aggregation apparatus comprises: a memory and at least one processor, the memory having instructions stored therein;

at least one of the processors invokes the instructions in the memory to cause the logistics data aggregation apparatus to perform the steps of the logistics data aggregation method of any one of claims 1-7.

10. A computer readable storage medium having instructions stored thereon, which when executed by a processor, perform the steps of the method of stream data aggregation as claimed in any one of claims 1 to 7.