CN113298469A - Abnormal data detection method and device, electronic equipment and storage medium - Google Patents
Abnormal data detection method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN113298469A CN113298469A CN202110578754.5A CN202110578754A CN113298469A CN 113298469 A CN113298469 A CN 113298469A CN 202110578754 A CN202110578754 A CN 202110578754A CN 113298469 A CN113298469 A CN 113298469A
- Authority
- CN
- China
- Prior art keywords
- data
- abnormal
- full
- information
- detection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000002159 abnormal effect Effects 0.000 title claims abstract description 234
- 238000001514 detection method Methods 0.000 title claims abstract description 224
- 238000003860 storage Methods 0.000 title claims abstract description 19
- 238000000034 method Methods 0.000 claims abstract description 96
- 230000032683 aging Effects 0.000 claims description 86
- 230000008569 process Effects 0.000 claims description 31
- 238000003908 quality control method Methods 0.000 claims description 18
- 238000004590 computer program Methods 0.000 claims description 3
- 238000004806 packaging method and process Methods 0.000 claims description 3
- 230000002547 anomalous effect Effects 0.000 abstract description 4
- 238000012545 processing Methods 0.000 description 17
- 238000010586 diagram Methods 0.000 description 16
- 239000013589 supplement Substances 0.000 description 15
- 230000005856 abnormality Effects 0.000 description 10
- 230000000153 supplemental effect Effects 0.000 description 9
- 230000001502 supplementing effect Effects 0.000 description 8
- 238000012544 monitoring process Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 238000012216 screening Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000007405 data analysis Methods 0.000 description 2
- 230000009469 supplementation Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004140 cleaning Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000004883 computer application Methods 0.000 description 1
- 238000013075 data extraction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/08—Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
- G06Q10/083—Shipping
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Business, Economics & Management (AREA)
- Economics (AREA)
- Software Systems (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Human Resources & Organizations (AREA)
- Development Economics (AREA)
- Entrepreneurship & Innovation (AREA)
- General Engineering & Computer Science (AREA)
- Marketing (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Strategic Management (AREA)
- Tourism & Hospitality (AREA)
- General Business, Economics & Management (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses an abnormal data detection method, an abnormal data detection device, electronic equipment and a storage medium, wherein the method comprises the following steps: acquiring full-process data of cargo transportation; determining corresponding abnormal detection logic according to the transportation data type of the full-flow data; and detecting the full-flow data according to the abnormal detection logic to obtain an abnormal data detection result. According to the embodiment of the invention, automatic detection is realized through the anomaly detection logic corresponding to the transportation data type of the full-flow data, the influence of human factors on anomaly detection is reduced, and the accuracy of detection of the anomalous transportation data is improved.
Description
Technical Field
The embodiment of the invention relates to the technical field of computer application, in particular to an abnormal data detection method and device, electronic equipment and a storage medium.
Background
With the development of the internet, the internet gradually plays an important role in the production, sales, transportation and other links of enterprises. In order to improve the accuracy of the transportation information, most enterprises adopt an information quality control system to manage abnormal information in the transportation process, and transportation personnel report the abnormal freight notes to the quality control system in a manual mode when the freight note information is abnormal in the transportation process. However, the following technical problems exist in the processing mode:
1) the detection of the abnormal waybill mainly depends on the professional knowledge of the transport staff, is greatly influenced by the subjective understanding of the transport staff, the accuracy of the detection cannot be guaranteed, and the abnormal waybill fails to report or misrereports.
2) After the abnormal freight note is detected, the transportation personnel are required to upload the abnormal freight note to the quality control system through a system website or a terminal, the processing efficiency of the abnormal freight note is low, a large amount of normal working time of the transportation personnel is occupied, and the cargo transportation efficiency of enterprises is reduced.
Disclosure of Invention
The invention provides an abnormal data detection method, an abnormal data detection device, electronic equipment and a storage medium, which are used for realizing automatic detection of cargo transportation data, reducing the influence of human factors on abnormal detection and improving the accuracy of the abnormal transportation data detection.
In a first aspect, an embodiment of the present invention provides an abnormal data detection method, where the method includes:
acquiring full-process data of cargo transportation;
determining corresponding abnormal detection logic according to the transportation data type of the full-flow data;
and detecting the full-flow data according to the abnormal detection logic to obtain an abnormal data detection result.
In a second aspect, an embodiment of the present invention provides an abnormal data detecting apparatus, including:
the data acquisition module is used for acquiring the full-process data of cargo transportation;
the logic determination module is used for determining corresponding abnormal detection logic according to the transportation data type of the full-flow data;
and the detection execution module is used for detecting the full-flow data according to the abnormal detection logic so as to obtain an abnormal data detection result.
In a third aspect, an embodiment of the present invention further provides an electronic device, where the electronic device includes:
one or more processors;
a memory for storing one or more programs,
when the one or more programs are executed by the one or more processors, the one or more processors implement the abnormal data detecting method according to any one of the embodiments of the present invention.
In a fourth aspect, the embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the abnormal data detection method according to any one of the embodiments of the present invention.
According to the embodiment of the invention, the full-flow data of the freight transportation is collected, the abnormal detection logic corresponding to the full-flow data is determined according to the type of the transportation data, the full-flow data is detected according to the abnormal detection logic, the abnormal data detection result is obtained, the full-flow data of the freight transportation is collected, and the detection is performed according to the abnormal data detection logic, so that the automatic detection of the freight transportation data is realized, the influence of human factors on the detection of the freight transportation data is reduced, and the accuracy of the abnormal detection is improved.
Drawings
FIG. 1 is a flow chart of a method for detecting abnormal data according to an embodiment of the present invention;
FIG. 2 is a flow chart of a method for detecting abnormal data according to an embodiment of the present invention;
FIG. 3 is an exemplary diagram of a full flow data acquisition provided by an embodiment of the present invention;
FIG. 4 is a flow chart of a method for detecting abnormal data according to an embodiment of the present invention;
FIG. 5 is an exemplary diagram of a method for detecting abnormal data according to an embodiment of the present invention;
FIG. 6 is a flow chart of a method for detecting abnormal data according to an embodiment of the present invention;
FIG. 7 is an exemplary diagram of a data supplement provided by an embodiment of the present invention;
FIG. 8 is a flow chart of a method for detecting abnormal data according to an embodiment of the present invention;
FIG. 9 is an exemplary diagram of aging information detection provided by embodiments of the present invention;
FIG. 10 is an exemplary diagram of data monitoring provided by embodiments of the present invention;
FIG. 11 is an exemplary diagram of a method for anomalous data detection in accordance with an embodiment of the present invention;
fig. 12 is a schematic structural diagram of an abnormal data detecting apparatus according to an embodiment of the present invention;
fig. 13 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be noted that, for convenience of description, only a part of the structures related to the present invention, not all of the structures, are shown in the drawings, and furthermore, embodiments of the present invention and features of the embodiments may be combined with each other without conflict.
Example one
Fig. 1 is a flowchart of an abnormal data detection method provided in an embodiment of the present invention, where the present embodiment is applicable to a case of detecting abnormal transportation data in a cargo transportation process, and the method may be executed by an abnormal data detection device, and the device may be implemented in a hardware and/or software manner, referring to fig. 1, the method provided in an embodiment of the present invention specifically includes the following steps:
and step 110, acquiring the full-process data of the cargo transportation.
The full-flow data may be digital information formed in the process of cargo transportation, the full-flow data may cover digital information of all links of cargo transportation, the full-flow data may be information acquired in different data systems of cargo transportation, and the full-flow data may include, for example, order data, express delivery site data, cargo identification data, delivery status data, and the like.
Specifically, the system can be interfaced with different data systems for cargo transportation, and the digitized information in each system is collected as the full-flow data for cargo transportation, for example, the data system for cargo transportation can send the digitized information to message channels such as a kafka message queue, a RabbitMQ message queue, a ZeroMQ message queue, and the like, and can collect the digitized information as the full-flow data in the message channels.
And step 120, determining corresponding abnormal detection logic according to the transportation data type of the full-flow data.
The type of the transportation data may be information representing an attribute of the full-flow data, for example, the full-flow data may represent a transportation state of the goods or a state of the goods during the transportation of the goods, and when the full-flow data may be divided into two types according to the transportation state of the goods and the state of the goods, the type of the transportation data may include order information and a package state. The anomaly detection logic can be a judgment logic for judging whether the full-flow data is anomalous data or not, the anomaly detection logic can be preset, and different anomaly detection logics can be set for different types of transportation data.
In the embodiment of the present invention, the received full-flow data may be classified according to a preset transportation data type, and a corresponding abnormality detection logic is determined for each classified transportation data type, it may be understood that the transportation data type and the abnormality detection logic may have an association relationship, for example, may have the same identification number or may be stored in an associated storage space.
And step 130, detecting the full-flow data according to the abnormal detection logic to obtain an abnormal data detection result.
The abnormal data detection result may be result information generated after performing abnormal detection on the full-process data, and the abnormal data detection result may include the full-process data and an identifier of whether the full-process data is abnormal data.
In the embodiment of the present invention, whether the full-flow data in the corresponding transportation data type is in accordance may be determined according to the obtained abnormality detection logic, and if it is determined that the full-flow data is in accordance with the corresponding abnormality detection logic, the full-flow data may be identified as abnormal data as an abnormality detection result.
According to the embodiment of the invention, the full-flow data of the freight transportation is obtained, the corresponding abnormal detection logic is determined based on the operation data type of the full-flow data, the full-flow data is detected according to the determined abnormal detection logic, and the abnormal detection result is obtained, so that the automatic detection of the full-flow data in the freight transportation process is realized, the influence of human factors on the abnormal detection is reduced, and the accuracy of the abnormal data detection can be improved.
Further, on the basis of the above embodiment of the present invention, the acquiring full-flow data of cargo transportation includes: extracting service data from at least one freight transportation service system through a message channel, wherein the service data comprises at least one of order data, waybill data, package data and delivery data; and taking each service data as full-flow data.
The message channel may be a message transmission channel connected to a service system for freight transportation, and may be one or more of message channels such as kafka message queue, RabbitMQ message queue, ZeroMQ message queue, and the like, and the service system may be a digital system for realizing freight transportation, and may include a sorting and shipping system, a transportation system, a quality control system, and the like. The order data can be contract data or receipt data for carrying out goods transportation; the waybill data can be data generated for consignment goods in the goods transportation process, and can be filled by a shipper and used as a basis for goods carrying by a transportation department; the package data may be related information of the consignment in the freight transportation process, for example, information of the weight, size, position and the like of the consignment, and the delivery data may be information of the consignment in the transportation state, which may reflect the progress of the freight transportation.
Specifically, the cargo systems for cargo transportation may use a message channel as a medium for message transmission, and may collect service data such as order data, waybill data, package data, delivery data, and the like in the message channel as full-flow data.
Fig. 2 is a flowchart of an abnormal data detection method according to an embodiment of the present invention, which is embodied on the basis of the foregoing embodiment of the present invention, and referring to fig. 2, the method according to the embodiment of the present invention specifically includes the following steps:
and step 210, acquiring the full-process data of the cargo transportation.
And step 220, dividing the full-flow data into waybill information and parcel information according to the types of the transportation data.
The transportation data type is divided into waybill information and parcel information according to information represented by full-flow data, the waybill information can represent cargo transportation state information, and the parcel information can represent cargo state information.
In the embodiment of the application, the full-process data can be classified according to the type of the transportation data, one type of the full-process data information can be waybill information, and the other type of the full-process data can be parcel information.
And step 230, respectively sending the waybill information and the parcel information to a waybill processor and a parcel processor, wherein the waybill processor and the parcel processor are respectively preset with abnormal detection logics corresponding to the types of the transportation data.
The waybill processor can be a software and/or hardware device for processing waybill information, the parcel processor can be a software and/or hardware device for processing parcel information, the waybill processor and the parcel processor can be respectively provided with abnormality detection logic for processing the waybill information and the parcel information, and the abnormality detection logic can be added to the waybill processor and the parcel processor in a predefined mode or a user input mode.
Specifically, the waybill information can be sent to the waybill processor, so that the waybill processor can detect corresponding waybill information according to preset abnormality detection logic, and correspondingly, the parcel information can be sent to the parcel processor, so that the parcel processor detects corresponding parcel information according to preset abnormality detection logic.
And step 240, judging whether the full-flow data accords with the abnormal detection logic.
In the embodiment of the invention, the full-flow data can be detected by using the exception detection logics in the shipping processor and the parcel processor respectively, and whether the full-flow data in the shipping processor and the parcel processor meet the corresponding exception detection logics is judged. For example, the package processor may determine whether the full-flow data is the report loss information, whether the full-flow data is the shipping cancellation information, or whether the full-flow data includes the new waybill number character string on the premise that the full-flow data is the invoice breaking print information.
Specifically, if the full-flow data in the manifest processor or the package processor conforms to the corresponding anomaly detection logic, the full-flow data is considered as the anomaly data, and the full-flow data can be stored as the anomaly detection result. If the full-flow data in the manifest processor or the package processor does not conform to the corresponding anomaly detection logic, the full-flow data is not abnormal data, and the full-flow data can be discarded without processing.
According to the embodiment of the invention, the full-flow data of cargo transportation is collected, the full-flow data is divided into the waybill information and the parcel information according to the type of the transportation data, the waybill information and the parcel information are respectively sent to the waybill processor and the parcel processor which are preset with corresponding different abnormal detection logics, whether the full-flow data accords with the abnormal detection logics is respectively determined by the waybill processor and the parcel processor, the satisfied full-flow data is stored as the abnormal data detection result, and the unsatisfied full-flow data is discarded.
Further, on the basis of the above embodiment of the invention, before acquiring the full-flow data of the transportation of the goods, the method further includes:
adding an anomaly detection logic corresponding to at least one service order type to the operation order processor; and adding an abnormal detection logic corresponding to at least one abnormal parcel state to the parcel processor, wherein the abnormal parcel transportation state comprises at least one of bill-breaking printing, proper delivery, loss reporting and shipment cancellation.
The order splitting printing can be a process of splitting an original order into a plurality of sub-orders and printing the sub-orders, the appropriate delivery can be a process of completing the consignment of goods according to a goods consignment address and a receiver according to a specified procedure, the damage reporting can be a process of processing the damaged goods in the consignment process, and the shipment canceling can be a process of canceling the consignment of goods.
In the embodiment of the invention, the exception handling logic in the waybill processor can be added according to different business order types, different exception detection logics can be set for waybill information of different businesses, different exception detection logics can be set in the parcel processor according to different exception parcel states of parcels, and the exception parcel states can include one or more of list splitting printing, proper delivery, loss reporting and shipment cancellation.
Exemplarily, in the abnormal data detection method provided by the embodiment of the present invention, the acquired full-process data may be preprocessed, fig. 3 is an exemplary diagram of the full-process data acquisition provided by the embodiment of the present invention, referring to fig. 3, the full-process data to be processed is actively captured, and abnormal data may be selected according to actual business logic to detect a specific concerned business line, waybill process and waybill package; useless data can be cleaned through the identification information of the waybill package, and only the needed data is concerned; the data screening identifies data through waybill information, screens reasonable and normal data through the data passing state and the creation time, and ensures the integrity and the time sequence of the data; the data processing is logic processing, a logic processor is added in a pluggable mode, the logic processor can be divided into two types of processors, namely an order type processor and a package state processor, and the self processor can be customized according to needs; after the full-flow data processing is finished, the abnormal data detection result can be stored in the database ES.
Fig. 4 is a flowchart of an abnormal data detection method according to an embodiment of the present invention, which is embodied on the basis of the foregoing embodiment of the present invention, and corrects an abnormal detection result based on prediction information, referring to fig. 4, the method according to the embodiment of the present invention specifically includes the following steps:
and step 310, acquiring the full-process data of the cargo transportation.
And step 320, determining corresponding abnormal detection logic according to the transportation data type of the full-flow data.
And 330, detecting the full-flow data according to the abnormal detection logic to obtain an abnormal data detection result.
And 340, collecting prediction information of a prediction data source, wherein the prediction information at least comprises sorting delivery information, routing information, batch information and quality control hotspot information.
The forecast data source may be a data source for collecting forecast information, the forecast data source may be a business system for freight transportation, the forecast information may be information for forecasting and supplementing existing full-flow data, and may include sorting shipment information, routing information, batch information, and quality control hotspot information, where the sorting shipment information may be information reflecting different packages in the same package in the freight transportation process, the routing information may be path information for package transportation in the freight transportation process, the batch information may be information of a batch to which a package belongs, and the quality control hotspot information may be information of a key area of interest or a vehicle set by a user.
In the embodiment of the invention, different freight transportation business systems can be used as prediction data sources, and sorting and delivery information, routing information, batch information, quality control hotspot information and the like in each business system can be collected as prediction information.
And step 350, determining a prediction detection condition corresponding to the full-process data according to the prediction information.
The prediction detection condition may be a condition for screening abnormal data detection results based on the prediction information, and the prediction detection condition may sort out full-flow data that may be erroneously determined as abnormal data in the abnormal data detection results.
Specifically, one or more prediction detection conditions may be determined according to the prediction information, for example, a predicted position of the package may be generated according to the sorting delivery message, the predicted position may be used as the prediction detection condition, an abnormal route of the package may be predicted according to the route information, the abnormal route may be used as the prediction detection condition, an arrival situation of the package may be predicted according to the batch information, and the arrival situation may be used as the prediction detection condition.
And 360, eliminating the full-flow data which accords with the prediction detection condition in the abnormal data detection result.
In the embodiment of the invention, whether the full-flow data in the abnormal detection data result meets the prediction detection condition or not can be judged, if the full-flow data meets the prediction detection condition, the full-flow data is judged to be the abnormal data by mistake, the full-flow data can be removed from the abnormal data detection result, and if the full-flow data does not meet the prediction detection condition, the prediction information can be added to the corresponding full-flow data.
According to the embodiment of the invention, the full-flow data of the cargo transportation is obtained, the corresponding abnormal detection logic is determined based on the operation data type of the full-flow data, the full-flow data is detected according to the determined abnormal detection logic, the abnormal detection result is obtained, the prediction information is collected from the prediction data source, the prediction detection condition corresponding to the prediction information is generated, the full-flow information in the abnormal detection result is removed according to the prediction detection condition, the misjudgment rate of the abnormal data detection is reduced, and the accuracy of the abnormal data detection in the cargo transportation process is improved.
In an exemplary embodiment, after abnormal data detection is performed on full-flow data, the misjudgment rate of the abnormal data detection may also be reduced by using prediction information, fig. 5 is an exemplary diagram of an abnormal data detection method provided by an embodiment of the present invention, and referring to fig. 5, data prediction mainly improves the accuracy of the abnormal data detection by using processed data, and supplements additional data. Acquiring current information of the collection package through a box number or a collection package number concerned by the sorting delivery message, and judging whether the collection package bill and the current position information are met or not by comparing the box number or the collection package number with the full-flow information; the routing information acquires an abnormal route, and whether the abnormal route is met or not is judged by comparing the site provided by the abnormal route information and the expected arrival timeliness with the full-flow data; the batch information can obtain the package information of the same batch, and whether the full-flow data normally flows is judged by back checking the package; the quality control hot spot is manually reported for a quality control system, processed data is compared through a hot spot area or a vehicle, whether the data of the whole process is abnormal or not is judged, and the data is updated in time.
Fig. 6 is a flowchart of an abnormal data detection method according to an embodiment of the present invention, which is embodied on the basis of the above embodiment of the present invention, and enriches an abnormal data detection result by collecting supplementary information, with reference to fig. 6, the method according to the embodiment of the present invention specifically includes the following steps:
and step 410, acquiring the full-process data of the cargo transportation.
And step 420, determining corresponding abnormal detection logic according to the transportation data type of the full-flow data.
And 430, detecting the full-flow data according to the abnormal detection logic to obtain an abnormal data detection result.
And 440, collecting supplementary information of a data supplementary source, wherein the supplementary information at least comprises one of supplementary waybill information, supplementary abnormal order flow information, supplementary abnormal site information and supplementary abnormal aging information.
The data supplementary source may be a data source for supplementing the full-flow data in the anomaly detection result, and may be different business systems in the cargo transportation process, the supplementary information may be data for supplementing the full-flow data, and the acquisition time of the supplementary information may be different from the acquisition time of the full-flow data, for example, the information of the data source may be continuously acquired after the full-flow data is acquired as the supplementary information.
In the embodiment of the invention, a business system of a cargo transportation process can be used as a data supplement source for supplementing the full-flow data of the abnormal detection result, and supplemental waybill information, supplemental abnormal order flow information, supplemental abnormal site information and supplemental abnormal aging information in the data supplement source can be regularly or continuously acquired as the supplemental information, wherein the supplemental waybill information can be waybill information for supplementing the full-flow data, the supplemental abnormal order flow information can be order flow data for supplementing the full-flow data, the supplemental abnormal site information can be cargo transportation site information for supplementing the full-flow data, and the supplemental abnormal aging information can be aging information for supplementing the full-flow data.
And step 450, adding the supplementary information to corresponding full-flow data in the abnormal data detection result.
Specifically, the corresponding full-process data can be searched in the abnormal data detection result according to the unique identification information of each supplementary information, and the supplementary information can be added to the searched full-process data to realize the supplement of the abnormal data detection result, so that the monitoring and the processing of the abnormal data are facilitated.
According to the embodiment of the invention, the full-flow data of the cargo transportation is obtained, the corresponding abnormal detection logic is determined based on the operation data type of the full-flow data, the full-flow data is detected according to the determined abnormal detection logic, the abnormal detection result is obtained, the supplementary information of the data supplementary source is collected, the supplementary information is added to the corresponding full-flow data in the abnormal detection result, the completeness of the full-flow data in the abnormal data detection result is improved, and the data monitoring and processing in the cargo transportation process are facilitated.
In an exemplary implementation manner, after the abnormal data detection result is determined, the abnormal detection method provided in the embodiment of the present invention may further complement the full-flow data in the abnormal data detection result, fig. 7 is an exemplary diagram of data supplementation provided in the embodiment of the present invention, and referring to fig. 7, a data supplementation mechanism may supplement data loss of the full-flow data by accessing waybill information, sorting and shipping information, transportation site data information, and quality control information, update the abnormal data in the ES database, make the reported abnormal data more complete, reduce the system pressure of the quality control system, and ensure the data integrity.
Fig. 8 is a flowchart of an abnormal data detection method according to an embodiment of the present invention, where the embodiment of the present invention is to screen full-flow data in an abnormal data detection result through aging information on the basis of the embodiment of the present invention, so as to improve accuracy of the abnormal data detection result, and referring to fig. 8, the method according to the embodiment of the present invention specifically includes the following steps:
and step 510, acquiring the full-process data of the cargo transportation.
And step 520, determining corresponding abnormal detection logic according to the transportation data type of the full-flow data.
And step 530, detecting the full-flow data according to the abnormal detection logic to obtain an abnormal data detection result.
And 540, collecting aging information of an aging data source, wherein the aging information at least comprises waybill aging information, running aging information, site aging information and abnormal aging information.
The source of the aging data can be a service system with aging information in a cargo transportation system, the aging information can be information representing the effective time range of the full-flow data, and the aging information can comprise waybill aging information, running aging information, waybill aging information and abnormal aging information according to the dimension of cargo transportation. The waybill timeliness can be an effective time deadline set for a waybill dimension, the operation timeliness information can be an effective time deadline set for a cargo operation process, the waybill timeliness information can be an effective time deadline set for the waybill dimension, and the abnormal timeliness information can be an effective time deadline set for different types of cargo transportation abnormity.
In the embodiment of the invention, the service system with the aging information can be used as an aging data source, and the waybill aging information, the operation aging information, the waybill aging information and the abnormal aging information of the aging data source can be collected as the aging information.
And 550, acquiring an abnormal aging rule matched with the aging information.
The abnormal aging rule may be an aging rule for determining whether the full-flow data is abnormal data, for example, it may be determined whether the time range of the full-flow data is not within the time range of the aging information.
Specifically, the corresponding abnormal aging rule can be determined according to the aging information, the abnormal aging rule can be set with one or more rules according to different dimensions of the aging information in the cargo transportation process, each rule can correspond to one or more of waybill aging information, operation aging information, waybill aging information and abnormal aging information, and the matched abnormal aging rule can be determined according to the dimension of the aging information.
And 560, eliminating the full-flow data which does not accord with the abnormal aging rule in the abnormal detection result.
In the embodiment of the present invention, the abnormal aging rule may be used to determine the full-flow data in the abnormal detection result, if the time range of the full-flow data conforms to the abnormal aging rule, the full-flow data is determined to be abnormal data, if the time range of the full-flow data does not conform to the abnormal aging rule, the full-flow data is determined to be erroneously determined to be abnormal data, and the full-flow data may be deleted from the abnormal data detection result.
According to the embodiment of the invention, the full-flow data of cargo transportation is acquired, the corresponding abnormal detection logic is determined based on the operation data type of the full-flow data, the full-flow data is detected according to the determined abnormal detection logic, the abnormal data detection result is acquired, the aging information is collected, the abnormal aging rule matched with the aging information is determined, the full-flow data which is not matched with the abnormal aging rule in the abnormal data detection result is eliminated, and the accuracy of the abnormal data detection result is improved through the aging information.
In an exemplary implementation manner, after abnormal data detection is performed, aging information of the full-flow data may also be collected, and abnormal data detection results are screened according to the aging information in the embodiment of the present invention, fig. 9 is an exemplary diagram of aging information detection provided in the embodiment of the present invention, and referring to fig. 9, capturing of the aging information is controlled, and aging control is performed on the full-flow data according to the aging information. The time efficiency control is mainly the reporting time efficiency defined by obtaining the promised time efficiency of the freight note, the estimated downstream time of arrival and the quality control time efficiency. The aging dispatcher can define that different aging is set according to areas, sites, abnormal reasons and single documents, the classification management is flexible to match, different aging ranges can be accurately controlled, and more accurate reported data can be obtained.
Further, on the basis of the above embodiment of the present invention, after the detecting the full-flow data according to the abnormal detection logic to obtain the abnormal data detection result, the method further includes: acquiring newly-added full-process data, and acquiring target process data matched with the newly-added full-process data in an abnormal data detection result; and adding the newly added full process data and the target process data into the abnormal detection result after packaging the data.
In the embodiment of the invention, newly added full-flow data can be acquired, the newly added full-flow data is matched with the full-flow data in the abnormal data detection result to obtain target full-flow data with an association relation, the newly added full-flow data and the target full-flow data can be packaged, and the packaged data is used as new full-flow data to be added to the abnormal detection result.
In an exemplary implementation manner, the full-flow data may also be monitored, fig. 10 is an exemplary diagram of data monitoring provided in an embodiment of the present invention, and referring to fig. 10, a reported waybill exception and package exception are obtained, where the waybill exception and the package exception are stored in a Redis storage, then the finalization information, the order change information, the warehouse returning information, and the site acquisition termination information of the full-flow data are obtained, the obtained new full-flow data is compared with the waybill exception and the package exception, the data conforming to the logic is encapsulated and sent to the quality control system, and the quality control system performs exception data detection processing on the exception documents such as the waybill exception and the package exception again.
In an exemplary implementation, fig. 11 is an exemplary diagram of an abnormal data detection method according to an embodiment of the present invention, and referring to fig. 11, an abnormal data detection system mainly accesses a package full-flow data message channel, filters out data that does not need to be focused and error data by cleaning data, screens out focused state data after reaching a data screening step, where the data has own processing logic according to different states, divides different state data into different business logics, and stores the data in an ES storage after the processing is completed. The asynchronous data updating logic updates the ES storage data by accessing the new message, is used for compensating the data and updating the abnormal package state, and reduces the false alarm and the multiple reports; capturing stored abnormal package information by adding different aging models, and reporting the abnormal package information to a quality control system; the data prediction is to predict whether the downstream station of the package flow is normally operated in time through routing information and the like; the reported data can be monitored continuously, and when new circulation information arrives, the abnormal single information is notified and updated in time.
Fig. 12 is a schematic structural diagram of an abnormal data detection apparatus provided in an embodiment of the present invention, which is capable of executing an abnormal data detection method provided in any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution method, where the apparatus may be implemented by software and/or hardware, and specifically includes: a data acquisition module 601, a logic determination module 602, and a detection execution module 603.
The data acquisition module 601 is used for acquiring the full-process data of the cargo transportation.
A logic determining module 602, configured to determine a corresponding anomaly detection logic according to the transportation data type of the full-flow data.
The detection execution module 603 is configured to detect the full-flow data according to the anomaly detection logic to obtain an anomaly data detection result.
According to the embodiment of the invention, the data acquisition module is used for acquiring the full-flow data of the freight, the logic determination module is used for determining the corresponding abnormal detection logic based on the operation data type of the full-flow data, and the detection execution module is used for detecting the full-flow data according to the determined abnormal detection logic and acquiring the abnormal detection result, so that the automatic detection of the full-flow data in the freight transportation process is realized, the influence of human factors on the abnormal detection is reduced, and the accuracy of the abnormal data detection can be improved.
Further, on the basis of the above embodiment of the invention, the data acquisition module 601 includes:
the data extraction unit is used for extracting service data from at least one service system for freight transportation through a message channel, wherein the service data comprises at least one of order data, waybill data, package data and delivery data; and taking each service data as full-flow data.
Further, on the basis of the above embodiment of the present invention, the logic determining module 602 includes:
and the data classification unit is used for dividing the full-flow data into waybill information and parcel information according to the types of the transportation data.
The information sending unit is used for respectively sending the waybill information and the package information to a waybill processor and a package processor; and the freight list processor and the parcel processor are respectively preset with abnormal detection logics corresponding to the types of the transportation data.
Further, on the basis of the above embodiment of the present invention, the logic presetting module is configured to add an abnormality detection logic corresponding to at least one service order type to the waybill processor; and adding an abnormal detection logic corresponding to at least one abnormal parcel state to the parcel processor, wherein the abnormal parcel transportation state comprises at least one of bill-breaking printing, proper delivery, loss reporting and shipment cancellation.
Further, on the basis of the above embodiment of the invention, the apparatus further includes: the data prediction unit is used for acquiring prediction information of a prediction data source, wherein the prediction information at least comprises sorting and delivery information, routing information, batch information and quality control hotspot information; determining a prediction detection condition corresponding to the full-flow data according to the prediction information; and eliminating the full-flow data which accords with the prediction detection condition in the abnormal data detection result.
Further, on the basis of the above embodiment of the invention, the apparatus further includes: the system comprises a data supplement module, a data analysis module and a data analysis module, wherein the data supplement module is used for acquiring supplement information of a data supplement source, and the supplement information at least comprises one of supplement waybill information, supplement abnormal order flow information, supplement abnormal site information and supplement abnormal aging information; and adding the supplementary information to corresponding full-flow data in the abnormal data detection result.
Further, on the basis of the above embodiment of the invention, the apparatus further includes: the aging processing module is used for acquiring aging information of an aging data source, wherein the aging information at least comprises waybill aging information, running aging information, site aging information and abnormal aging information; acquiring an abnormal aging rule matched with the aging information; and eliminating the full-flow data which does not accord with the abnormal aging rule in the abnormal detection result.
Further, on the basis of the embodiment of the present invention, the data adding module is configured to obtain added full-process data, and obtain target process data in the abnormal data detection result, where the target process data matches the added full-process data; and adding the newly added full-process data and the target process data into the abnormal detection result after packaging the data.
Further, on the basis of the above embodiment of the present invention, the detection executing module 603 includes:
the detection unit is used for judging whether the full-flow data conforms to the abnormal detection logic or not;
the execution unit is used for storing the full-flow data as an abnormal data detection result if the detection result is met; and if not, losing the full-flow data.
Fig. 13 is a schematic structural diagram of an electronic device according to an embodiment of the present invention, as shown in fig. 13, the electronic device includes a processor 70, a memory 71, an input device 72, and an output device 73; the number of the processors 70 in the electronic device may be one or more, and one processor 70 is taken as an example in fig. 13; the processor 70, the memory 71, the input device 72 and the output device 73 in the electronic apparatus may be connected by a bus or other means, and the bus connection is exemplified in fig. 13.
The memory 71 is a computer-readable storage medium, and can be used for storing software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the abnormal data detection method in the embodiment of the present invention (for example, the data acquisition module 601, the logic determination module 602, and the detection execution module 603 in the abnormal data detection apparatus). The processor 70 executes various functional applications and data processing of the electronic device by running software programs, instructions, and modules stored in the memory 71, that is, implements the above-described abnormal data detection method.
The memory 71 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal, and the like. Further, the memory 71 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, the memory 71 may further include memory located remotely from the processor 70, which may be connected to the electronic device through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The input device 72 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function controls of the electronic apparatus. The output device 73 may include a display device such as a display screen.
Embodiments of the present invention also provide a storage medium containing computer-executable instructions, which when executed by a computer processor, perform a method for anomalous data detection, the method comprising:
acquiring full-process data of cargo transportation;
determining corresponding abnormal detection logic according to the transportation data type of the full-flow data;
and detecting the full-flow data according to the abnormal detection logic to obtain an abnormal data detection result.
Of course, the storage medium provided by the embodiment of the present invention contains computer-executable instructions, and the computer-executable instructions are not limited to the method operations described above, and may also perform related operations in the abnormal data detection method provided by any embodiment of the present invention.
From the above description of the embodiments, it is obvious for those skilled in the art that the present invention can be implemented by software and necessary general hardware, and certainly, can also be implemented by hardware, but the former is a better embodiment in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a FLASH Memory (FLASH), a hard disk or an optical disk of a computer, and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute the methods according to the embodiments of the present invention.
It should be noted that, in the embodiment of the abnormal data detection apparatus, the included units and modules are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be implemented; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (12)
1. An abnormal data detection method, comprising:
acquiring full-process data of cargo transportation;
determining corresponding abnormal detection logic according to the transportation data type of the full-flow data;
and detecting the full-flow data according to the abnormal detection logic to obtain an abnormal data detection result.
2. The method of claim 1, wherein the obtaining full flow data for the shipment of the cargo comprises:
extracting service data from at least one freight transportation service system through a message channel, wherein the service data comprises at least one of order data, waybill data, package data and delivery data;
and taking each service data as full-flow data.
3. The method of claim 1, wherein determining the corresponding anomaly detection logic according to the transportation data type of the full-flow data comprises:
dividing the full-flow data into waybill information and package information according to the types of the transportation data;
respectively sending the waybill information and the parcel information to a waybill processor and a parcel processor;
and the freight list processor and the parcel processor are respectively preset with abnormal detection logics corresponding to the types of the transportation data.
4. The method of claim 3, further comprising, prior to obtaining full flow data for the shipment of the goods:
adding an anomaly detection logic corresponding to at least one service order type to the operation order processor;
and adding an abnormal detection logic corresponding to at least one abnormal parcel state to the parcel processor, wherein the abnormal parcel transportation state comprises at least one of bill-breaking printing, proper delivery, loss reporting and shipment cancellation.
5. The method of claim 1, further comprising:
collecting prediction information of a prediction data source, wherein the prediction information at least comprises sorting and delivery information, routing information, batch information and quality control hotspot information;
determining a prediction detection condition corresponding to the full-flow data according to the prediction information;
and eliminating the full-flow data which accords with the prediction detection condition in the abnormal data detection result.
6. The method of claim 1, further comprising:
acquiring supplementary information of a data supplementary source, wherein the supplementary information at least comprises one of supplementary waybill information, supplementary abnormal order flow information, supplementary abnormal site information and supplementary abnormal aging information;
and adding the supplementary information to corresponding full-flow data in the abnormal data detection result.
7. The method of claim 1, further comprising:
acquiring aging information of an aging data source, wherein the aging information at least comprises waybill aging information, running aging information, site aging information and abnormal aging information;
acquiring an abnormal aging rule matched with the aging information;
and eliminating the full-flow data which does not accord with the abnormal aging rule in the abnormal detection result.
8. The method of claim 1, after said detecting the full flow data according to the anomaly detection logic to obtain an anomaly data detection result, further comprising:
acquiring newly-added full-process data, and acquiring target process data matched with the newly-added full-process data in the abnormal data detection result;
and adding the newly added full-process data and the target process data into the abnormal detection result after packaging the data.
9. The method according to any of claims 1-8, wherein said detecting the full flow data according to the anomaly detection logic to obtain an anomaly data detection result comprises:
determining whether the full-flow data conforms to the anomaly detection logic;
if yes, storing the full-flow data as an abnormal data detection result;
and if not, losing the full-flow data.
10. An abnormal data detecting apparatus, comprising:
the data acquisition module is used for acquiring the full-process data of cargo transportation;
the logic determination module is used for determining corresponding abnormal detection logic according to the transportation data type of the full-flow data;
and the detection execution module is used for detecting the full-flow data according to the abnormal detection logic so as to obtain an abnormal data detection result.
11. An electronic device, characterized in that the electronic device comprises:
one or more processors;
a memory for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the anomaly data detection method as recited in any of claims 1-9.
12. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the abnormal data detecting method according to any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110578754.5A CN113298469A (en) | 2021-05-26 | 2021-05-26 | Abnormal data detection method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110578754.5A CN113298469A (en) | 2021-05-26 | 2021-05-26 | Abnormal data detection method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113298469A true CN113298469A (en) | 2021-08-24 |
Family
ID=77325178
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110578754.5A Pending CN113298469A (en) | 2021-05-26 | 2021-05-26 | Abnormal data detection method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113298469A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115221151A (en) * | 2022-07-13 | 2022-10-21 | 小米汽车科技有限公司 | Vehicle data transmission method and device, vehicle, storage medium and chip |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100082151A1 (en) * | 2008-09-30 | 2010-04-01 | Young Eric C | Systems and methods for receiving shipment parcels |
| CN102232222A (en) * | 2008-09-30 | 2011-11-02 | 亚马逊技术股份有限公司 | Systems and methods for receiving shipment parcels |
| CN108694633A (en) * | 2017-04-06 | 2018-10-23 | 北京京东尚科信息技术有限公司 | Order abnormality detection and facture, device, electronic equipment and readable storage medium storing program for executing |
| CN109840736A (en) * | 2018-12-29 | 2019-06-04 | 浙江天畅供应链管理有限公司 | A kind of intelligent transport management process |
| US20200100115A1 (en) * | 2018-09-22 | 2020-03-26 | Fedex Corporate Services, Inc. | Systems and methods for securely monitoring a shipping container for an environmental anomaly |
-
2021
- 2021-05-26 CN CN202110578754.5A patent/CN113298469A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100082151A1 (en) * | 2008-09-30 | 2010-04-01 | Young Eric C | Systems and methods for receiving shipment parcels |
| CN102232222A (en) * | 2008-09-30 | 2011-11-02 | 亚马逊技术股份有限公司 | Systems and methods for receiving shipment parcels |
| CN108694633A (en) * | 2017-04-06 | 2018-10-23 | 北京京东尚科信息技术有限公司 | Order abnormality detection and facture, device, electronic equipment and readable storage medium storing program for executing |
| US20200100115A1 (en) * | 2018-09-22 | 2020-03-26 | Fedex Corporate Services, Inc. | Systems and methods for securely monitoring a shipping container for an environmental anomaly |
| CN109840736A (en) * | 2018-12-29 | 2019-06-04 | 浙江天畅供应链管理有限公司 | A kind of intelligent transport management process |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115221151A (en) * | 2022-07-13 | 2022-10-21 | 小米汽车科技有限公司 | Vehicle data transmission method and device, vehicle, storage medium and chip |
| CN115221151B (en) * | 2022-07-13 | 2024-02-02 | 小米汽车科技有限公司 | Vehicle data transmission method and device, vehicle, storage medium and chip |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108764808A (en) | Data Analysis Services system and its on-time model dispositions method | |
| AU2019275633B2 (en) | System and method of automated fault correction in a network environment | |
| CN109635292B (en) | Work order quality inspection method and device based on machine learning algorithm | |
| CN112184115A (en) | Logistics early warning method, logistics recommendation method, logistics early warning device and related equipment | |
| CN115271617B (en) | Intelligent information processing system for bulk commodity logistics transportation transaction | |
| CN113469857A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN113902370A (en) | Information processing method, device, equipment and storage medium | |
| CN114726654A (en) | Data analysis method and server for coping with cloud computing network attack | |
| CN114723543B (en) | Financial archive big data management system and method for cross-border e-commerce | |
| CN113298469A (en) | Abnormal data detection method and device, electronic equipment and storage medium | |
| CN111291051A (en) | Freight rate data processing method and device, storage medium and electronic equipment | |
| CN113947076A (en) | Policy data detection method and device, computer equipment and storage medium | |
| US20210232995A1 (en) | System and methods for risk assessment in a multi-tenant cloud environment | |
| JP2005536340A (en) | Method and apparatus for reading a delivery or document overwrite | |
| CN114612029A (en) | Risk waybill data identification method and device, computer equipment and storage medium | |
| KR101984257B1 (en) | Cloud service based big data analysing system and method therein | |
| CN113034037B (en) | Logistics monitoring method and device, storage medium and electronic equipment | |
| CN115600950A (en) | Logistics track information processing method, medium, system, device and computing equipment | |
| CN114418236A (en) | Information prediction method, information prediction device, storage medium and electronic equipment | |
| JP7108566B2 (en) | Digital evidence management method and digital evidence management system | |
| CN113971539A (en) | Express package interception management method, device, equipment and storage medium | |
| CN112465149A (en) | Same-city part identification method and device, electronic equipment and storage medium | |
| CN116503887B (en) | Goods generation order processing method based on image recognition | |
| CN113037570B (en) | Alarm processing method and equipment | |
| CN111149120A (en) | Safety system for a transportation facility |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |