CN115080553B - Intelligent monitoring method for export goods - Google Patents
Intelligent monitoring method for export goods Download PDFInfo
- Publication number
- CN115080553B CN115080553B CN202210855751.6A CN202210855751A CN115080553B CN 115080553 B CN115080553 B CN 115080553B CN 202210855751 A CN202210855751 A CN 202210855751A CN 115080553 B CN115080553 B CN 115080553B
- Authority
- CN
- China
- Prior art keywords
- data
- container
- field
- data field
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/40—Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/04—Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem"
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
- G06Q10/06395—Quality analysis or management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q30/00—Commerce
- G06Q30/02—Marketing; Price estimation or determination; Fundraising
- G06Q30/0201—Market modelling; Market analysis; Collecting market data
- G06Q30/0206—Price or cost determination based on market factors
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Quality & Reliability (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The application discloses an intelligent monitoring method for export goods, which relates to big data and artificial intelligence technology and comprises the following steps: establishing a data transmission channel between the data center and an external service system; matching in an input data field pool of an available model according to data of each field contained in the original data to obtain a standard data field corresponding to the original data; matching the model in the available model according to the standard data field corresponding to the original data to obtain a candidate model; selecting a candidate model according to the manual instruction, and determining the corresponding relation and the conversion relation between the original data field and the data field in the selected candidate model; cleaning the original data based on the conversion relation to obtain standard data; and inputting the standard data into the candidate model to execute a prediction task, and monitoring the target parameter according to a prediction result. The data cleaning and model selection efficiency can be increased by implementing the method and the device.
Description
Technical Field
The application relates to big data and artificial intelligence technology, in particular to an intelligent monitoring method for export goods.
Background
The export of goods presents commercial and political risks, and the same vehicle faces different consumer groups and local regulations in different countries or regions, for example, used cars. Often requiring a large number of people to perform valuation, etc. operations to assess the benefits and risks of the exports.
With the development of big data technology, price evaluation can be performed through an artificial intelligence model, but not all exporters have related research and development capabilities, and data held by the exporters are different. Therefore, there is a need to provide monitoring services for export goods for different exporters.
However, since the data grasped by different exporters are different, if modeling is performed by using the data given by the exporter, a large amount of manual processing is required, and the efficiency is extremely low.
Disclosure of Invention
The present invention is directed to solving at least one of the problems of the prior art. Therefore, the invention provides an intelligent monitoring method for export goods, which aims to improve the efficiency of data cleaning and model selection.
The embodiment of the application provides an intelligent monitoring method for export goods, which comprises the following steps:
establishing a data transmission channel between a data center and an external service system, wherein the external service system reports original data based on a user-defined interface mode, and the original data comprises a plurality of fields;
matching in an input data field pool of available models according to data of each field contained in the original data to obtain a standard data field corresponding to the original data, wherein the input data field Chi Zhongbao contains a plurality of input data fields of the available models;
matching the model in the available model according to the standard data field corresponding to the original data to obtain a candidate model;
selecting a candidate model according to the manual instruction, and determining the corresponding relation and the conversion relation between the original data field and the data field in the selected candidate model;
cleaning the original data based on the conversion relation to obtain standard data;
and inputting the standard data into the candidate model to execute a prediction task, and monitoring the target parameter according to a prediction result.
In some embodiments, the selecting a candidate model according to a manual instruction, and determining a corresponding relationship and a transformation relationship between an original data field and a data field in the selected candidate model specifically includes:
displaying an operation interface, wherein a first container, a second container, a third container and a model list are loaded in the operation interface, the sequence of candidate models in the model list is arranged in a descending order according to the matching degree of the standard data field and the candidate models, the first container, the second container and the third container respectively comprise a plurality of sub-containers, and each sub-container is used for storing data field information;
receiving a first manual instruction, and determining a selected candidate model based on the first manual instruction;
loading each first data field information of the candidate model in a first container according to the candidate model;
loading each second data field information of the original data in a second container;
receiving a second manual instruction, and moving at least part of data field information of the original data from the second container to a third container according to the second manual instruction, so that the number of the field information in the third container is the same as that of the field information in the second container;
and receiving a third manual instruction, and configuring a conversion relation between the fields in the third container and the fields in the first container according to the third manual instruction.
In some embodiments, in the step of loading each piece of second data field information of the original data in the second container, the order of the second data field information is determined according to a matching relationship between the second data field information and the first data field information.
In some embodiments, the second container is labeled with second data field information according to data quality;
the marking means that different colors, fonts and textures are adopted to represent the data quality of the corresponding field, the data quality is determined according to the integrity rate of the data of the field, and the integrity rate means the proportion of the residual data in all the data after the abnormal data in the field are removed.
In some embodiments, the method further comprises the following steps:
and monitoring the position of the mouse, and displaying the data summary information and the data sample corresponding to the data field information in the sub-container when the mouse is judged to stay on the sub-container of the first container, the sub-container of the second container or the sub-container of the third container.
In some embodiments, the method further comprises the steps of:
establishing 3*N array a [ ], for storing data summary information address and data sample address;
in the step of displaying the data summary information and the data samples corresponding to the data field information in the sub-container, acquiring data summary information addresses and data samples according to the information in the array a [ ] [ ] corresponding to the sub-container;
wherein, a 0 [ ] is used to store the data abstract information address and data sample address corresponding to the first data field information; a 1 [ ] is used to store the data abstract information address and data sample address corresponding to the second data field information; a [2] [ ] is used for storing the data summary information address and the data sample address corresponding to the second data field information moved to the third container;
when the second data field information corresponding to a [1] [ i ] is moved to the third container, the data in the a [1] [ i ] is not deleted;
when the second data field information in the a [1] [ y ] corresponding sub-container and the second data field information in the a [1] [ x ] corresponding sub-container are exchanged, exchanging the data of the a [1] [ y ] and the a [1] [ x ];
updating the data in a [2] [ j ] when the corresponding sub-container of a [2] [ j ] is moved into the new second data field information;
when the information of the second data field which is moved in the sub container corresponding to the a [2] [ j ] is removed, clearing the data in the a [2] [ j ];
wherein i, j, x and y are integers belonging to 0~N-1.
In some embodiments, the matching in the input data field pool of the available model to obtain the standard data field corresponding to the original data specifically includes:
screening candidate standard data fields in an input data field pool according to the data types of the original data fields;
and matching the corresponding standard data field in the candidate standard data fields according to the field description of the original data field.
In some embodiments, matching the corresponding standard data field in the candidate standard data field according to the field description of the original data field specifically includes:
respectively processing the field description of the original data field and the field description of the standard data field as follows:
extracting keywords from the field description according to the high-frequency word bank, and converting the keywords into a first semantic vector;
converting the field description into a second semantic vector;
weighting the first semantic vector and the second semantic vector to obtain a third semantic vector;
and performing similarity calculation on the third semantic vector corresponding to the original data field and the third semantic vector of the standard data field, and taking the standard data field with the similarity larger than a threshold value and the maximum similarity as the standard data field corresponding to the original data field.
In some embodiments, multiple of the available models are used to perform the same task, with different available models differing in the input data field.
In some embodiments, the matching a model in an available model according to a standard data field corresponding to the original data to obtain a candidate model specifically includes:
taking input data fields of available models as a first set, and taking standard data fields corresponding to original data as a second set;
when available models exist, wherein the first set belongs to the subset of the second set, the corresponding available models are used as candidate models;
when there is no available model of the first set that belongs to the second set, the candidate model is determined according to the proportion of elements in the intersection of the first set and the second set to the first set.
According to the method and the device, the data transmission channel between the data center and the external service system is established, and the external service system is supported to report the original data based on a user-defined interface mode; the data center has the advantages that the reporting party can provide related data according to the capacity of the reporting party without being restricted by specific data requirements, so that an operator of the data center can receive the data of the reporting party under the condition of providing a certain interface specification and then perform data cleaning and model selection, and the flexibility and the universality of services are improved; then, according to the data of each field contained in the original data, matching in an input data field pool of the available model to obtain a standard data field corresponding to the original data, wherein the input data field Chi Zhongbao contains a plurality of input data fields of the available model, and determining a standard field possibly corresponding to the field in a field matching mode; matching the model in the available model according to the standard data field corresponding to the original data to obtain a candidate model; then, selecting a candidate model according to the manual instruction, and determining the corresponding relation and the conversion relation between the original data field and the data field in the selected candidate model; cleaning the original data based on the conversion relation to obtain standard data; finally, inputting the standard data into the candidate model to execute a prediction task, and monitoring a target parameter according to a prediction result; through the method, the existing models which are possibly applicable can be matched according to the data structure reported by the user, and then the processing personnel of the data center make specific selection on the selection of the models and the adaptation of the data conversion relation, so that the flexibility is increased, meanwhile, compared with the traditional negotiation mode, the technical requirements of the data provider are reduced, and meanwhile, the data center personnel are more convenient in model adaptation and data cleaning.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a flowchart of an intelligent monitoring method for export goods according to an embodiment of the present disclosure;
FIG. 2 is a block diagram of a system to which the method provided by the embodiments of the present application is applied;
FIG. 3 is a schematic diagram of a software interface provided by an embodiment of the present application;
fig. 4 is a schematic diagram of a software interface and a corresponding array relationship provided in the embodiment of the present application.
Detailed Description
In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below through embodiments with reference to the accompanying drawings in the embodiments of the present application, and it is obvious that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without making any creative effort belong to the protection scope of the present application.
Referring to fig. 1, an embodiment of the present application provides an intelligent monitoring method for export goods, which can be applied to the system architecture shown in fig. 2, wherein the method is mainly applied at a data center, and includes:
s1, establishing a data transmission channel between a data center and an external service system, wherein the external service system reports original data based on a user-defined interface mode, and the original data comprises a plurality of fields.
For example, 10 data, such as vehicle picture, color, model, frame number, year, mileage, and selling price, are held in the hands of the first second-hand vehicle exit dealer. And the second used vehicle exporter only has 8 data. The two second-hand vehicle exporters can report several kinds of data (for example, the first exporter reports 9, and the second exporter reports six kinds of data) according to the reporting interface specification provided by the data center according to the data grasped by the two second-hand vehicle exporters, and simultaneously select related monitoring tasks, for example, the predicted price to analyze whether the current export price is reasonable or not. The data center may match whether a suitable model is available for relevant prediction based on the existing input fields of some models. It means that the data center sets a number of pre-trained available models for a certain task, and the input parameters of these models are different. For example, also for the price prediction model, the input fields for the first model could be model, mileage, year, number of repairs, and the input fields for the second model could be model, mileage, color, area of use.
And S2, matching in an input data field pool of the available model according to data of each field contained in the original data to obtain a standard data field corresponding to the original data, wherein the input data field Chi Zhongbao comprises a plurality of input data fields of the available model.
In this step, the data center may put input fields of all models of a model executing a certain task into the data field pool, and at this time, each field of the original data may be matched in the data field pool based on the field description and the data type of the field. In general, data types may be matched first, and then matched based on field descriptions, because it is less likely that one data will be described across data types. The types of fields may be text, numerical, image, audio, video, etc., but for general prediction tasks, text, numerical, and image are the main. For example, the field pool includes: mileage (numerical value), color (text), color (image), year of factory (numerical value), insurance accident (text) … …. If the reported data includes a certain image field, a field of color in a field pool is matched according to the field type, and then whether the field describes color by using a picture is judged based on the description of the reported field. The same data field in different models is common in the field pool, even though there may be differences in the value units in different models.
And S3, matching the model in the available model according to the standard data field corresponding to the original data to obtain a candidate model.
Specifically, step S3 includes:
the input data fields of the available models are used as a first set, and the standard data fields corresponding to the original data are used as a second set. For example, the input data field of the first available model is S1= { A, B, C, D, E, F }, the input data field of the second available model is S2= { B, C, D, E, F, X }, and the standard field corresponding to the original data is S3= { A, B, C, D, E, F, G, H, I, J }.
When there are available models of which the first set belongs to the subset of the second set, the corresponding available models are taken as candidate models. Obviously, S1 belongs to a subset of S3, whereas S2 is not. At this time, S1 may be used as a first candidate model, and of course, although S2 is not a subset of S3, it actually only differs by one field, and if the field can be converted from some fields, the second model may be used to perform the prediction task.
When there is no own available model for which the first set belongs to the second set, the candidate model is determined according to the proportion of elements in the intersection of the first set and the second set to the first set. Of course, there may be only a model such as the second model, that is, there is no model that completely matches the input type in the reported data, but similarly, if the number of fields that differ is not large, for example, one or two, the relevant fields can be obtained by mining other data fields, and the model can still be applied. Therefore, in this embodiment, a certain matching percentage may be set to determine the candidate model, so that no available model can be matched.
And S4, selecting a candidate model according to the manual instruction, and determining the corresponding relation and the conversion relation between the original data field and the data field in the selected candidate model.
Referring to fig. 3, this embodiment discloses a related interface, and this step specifically includes:
and displaying an operation interface, wherein a first container, a second container, a third container and a model list are loaded in the operation interface, the sequence of the candidate models in the model list is arranged in a descending order according to the matching degree of the standard data field and the candidate models, the first container, the second container and the third container respectively comprise a plurality of sub-containers, and each sub-container is used for storing data field information. The matching degree of the model is sorted based on the coincidence degree of the fields, and if the matching degree can be represented by dividing the number of matched fields by the number of fields of the model. If a plurality of models are matched with the degree of 1 or are the same, the models are arranged in descending order according to the number of the parameters of the models. The container referred to in this embodiment refers to a UI component for accommodating a text/image UI component, such a component being provided to facilitate visualization operation.
Receiving a first manual instruction (namely an instruction input by a user through a peripheral device), and determining the selected candidate model based on the first manual instruction. For example, the user may select any of the models in the candidate model list.
And loading each piece of first data field information of the candidate model in the first container according to the candidate model. After a user selects a model, the first container and data thereof are reloaded, the number of the sub-containers of the first container is adjusted according to the number of the fields of the model, and then the data field information of the model is loaded in the sub-containers according to the input parameter fields of the model.
And loading each second data field information of the original data in the second container. It should be understood that, in this step, the second data field information is a field in the original data, and the order of the second data field information is determined according to the matching relationship between the second data field information and the first data field information, where each first data field information corresponds to one second data field information with the highest matching degree. For example, in the matching of the fields, the mileage distribution in the raw data is matched with the total mileage in the model, the mileage distribution may be mileage per year, and the total mileage of the model is the total mileage of the driving, and there is a certain conversion relationship between the mileage distribution and the total mileage of the model. An accumulated function may be configured such that the two are uniform. Some fields may only have unit conversion relationship, for example, some data tanks use liter as unit, some data uses gallon as unit, and both can be converted by a certain function. In fig. 3, a configuration column of the class transformation function is also provided, and the user can edit the related function or select an existing function for processing.
And receiving a second manual instruction, and moving at least part of data field information of the original data from the second container to a third container according to the second manual instruction, so that the number of the field information in the third container is the same as that of the field information in the second container. As shown in fig. 3, the user may adjust the corresponding relationship of the fields according to the actual contents of the fields by referring to the result of the system matching. For example, in the example of FIG. 3, the user has replaced the insurance field with the number of incidents, perhaps because the number of incidents matches the input field definition of the model more closely than the insurance. It is also possible that the data quality of the insurance item in the original data is too poor, for example, there are many abnormal data, or the data missing degree is high, which may make the user select a more appropriate field.
In some embodiments, the second container labels second data field information according to data quality;
the marking means that different colors, fonts and textures are adopted to represent the data quality of the corresponding field, the data quality is determined according to the integrity rate of the data of the field, and the integrity rate means the proportion of the residual data in all the data after the abnormal data in the field are removed.
The user can observe the quality of the data visually through the labels during operation, can adopt different colors to represent the quality of the data in a grading way, and can convert the data with poor quality from other data with better quality in time, so that the accuracy of a prediction result is optimized.
And receiving a third manual instruction, and configuring a conversion relation between the fields in the third container and the fields in the first container according to the third manual instruction. As shown in fig. 3, the user can edit the conversion function, select a cleaning model to convert the data, and so on. Through the auxiliary mode, professionals of the data center can select the optimal model, the matching relation between the data and the model and the data cleaning mode for the data reported by the user to solve the problem based on the matching relation of the system. On one hand, the data provider only needs to follow a certain interface rule to transmit data, and does not need to negotiate the content of the data with the data center side in advance, so that the efficiency of model adaptation and data cleaning is higher. It should be understood that the first manual command, the second manual command and the third manual command are all commands input by the user, and the commands include, but are not limited to, clicking, dragging, etc., and the commands may refer to a set of multiple operations.
And S5, cleaning the original data based on the conversion relation to obtain standard data.
After the conversion relation is selected by the staff of the data center, the original data can be cleaned to obtain the standard data which can be input into the model.
And S6, inputting the standard data into the candidate model to execute a prediction task, and monitoring the target parameters according to a prediction result. The tasks herein may be based on user-selected tasks such as price forecasting, quality assessment, and the like.
It can be known from the above-mentioned embodiment that, through implementing this scheme, on the one hand can reduce the requirement of data provider, and on the other hand when carrying out model matching and field matching, has shortened processing personnel's processing time greatly, has promoted efficiency. The willingness of the data provider to use services or access a platform can be increased by reducing the technical threshold of the data provider, so that the data center can obtain more data and continuously optimize a prediction model.
In some embodiments, to facilitate user determination, the method further comprises the following steps:
and monitoring the position of the mouse, and displaying the data summary information and the data sample corresponding to the data field information in the sub-container when the mouse is judged to stay on the sub-container of the first container, the sub-container of the second container or the sub-container of the third container.
Specifically, in order to realize the display of the data summary information address and the data sample and facilitate the operation, the present solution stores the data summary information address and the data sample address in a manner of configuring data, and referring to fig. 4, the method further includes the following steps:
an array a [ ] [ ] of 3*N is established for storing the data summary information address and the data sample address. N is determined by the number of fields of the original data. For the first container and the third container, the number of the sub-containers is determined according to the number of the input fields of the model, and the number of the sub-containers is generally less than or equal to N, which means that spare array units exist after the arrays a [0] [ ] and a [2] [ ], and the spare array units can be used for making temporary storage units when two array units in a [1] [ ] are exchanged.
In the step of displaying the data summary information and the data samples corresponding to the data field information in the sub-container, acquiring data summary information addresses and data samples according to the information in the array a [ ] [ ] corresponding to the sub-container;
wherein, a 0 [ ] is used to store the data abstract information address and data sample address corresponding to the first data field information; a 1 [ ] is used to store the data abstract information address and data sample address corresponding to the second data field information; and a [2] [ ] is used for storing the data summary information address and the data sample address corresponding to the second data field information moved to the third container. It should be understood that different locations of the memory can be pointed to by storing addresses, so that the data can be read only by recording the start address and the end address of the data without considering the length and the type of the data in the data when setting the array. It should be understood that, in the present embodiment, the data fields in the containers are different from each other, and data exchange between the sub-containers is also required, so the manner of employing the present embodiment is simple and convenient.
When the second data field information corresponding to a [1] [ i ] is moved to the third container, the data in a [1] [ i ] is not deleted. Since the second data field information is also not deleted in the corresponding container. This means that one second data field information can be moved into a sub-container of a plurality of third containers. A partial model may use two data having a certain conversion relationship as input data.
When the second data field information in the a [1] [ y ] corresponding sub-container and the second data field information in the a [1] [ x ] corresponding sub-container are exchanged, the data of a [1] [ y ] and a [1] [ x ] are exchanged. In this embodiment, when exchanging a [1] [ y ] and a [1] [ x ] (or exchanging a [2] [ y ] and a [2] [ x ]), the spare a [0] [ z ] (or a [2] [ z ]) can be used as the intermediate storage unit, that is, a [1] [ y ] is stored in a [0] [ z ], then a [1] [ x ] is covered on a [1] [ y ], and then a [0] [ z ] is covered on a [1] [ x ]. By the method, an array is not required to be additionally arranged, the allocated memory is utilized, and the processing steps are reduced.
And updating the data in a [2] [ j ] when the corresponding sub-container of a [2] [ j ] is moved into the new second data field information. For the third container, if the second data field information is already stored, the content can be overwritten by moving in a new second data field information.
When the information of the second data field which has been moved in the corresponding sub-container of a [2] [ j ] is removed, the data in a [2] [ j ] is cleared. When the data field information is removed, the contents of the container are also removed, so that the data in a [2] [ j ] is not necessary to be retained.
Wherein i, j, x, y and z are integers belonging to 0~N-1.
In some embodiments, in S2, the matching in the input data field pool of the available model to obtain a standard data field corresponding to the original data specifically includes:
and S21, screening candidate standard data fields in an input data field pool according to the data types of the original data fields.
And S22, matching the corresponding standard data field in the candidate standard data field according to the field description of the original data field. Specifically, the method specifically comprises the following steps:
s221, respectively performing the following processing on the field description of the original data field and the field description of the standard data field:
and S2211, extracting keywords from the field description according to the high-frequency word bank, and converting the keywords into a first semantic vector. In this embodiment, a high frequency thesaurus may be provided, and a large number of thesaurus for describing fields are stored in the high frequency thesaurus. Useless descriptions can be filtered out through the word stock, and then high-frequency words are converted into semantic vectors in the NLP model, so that the weight of the keywords can be increased.
And S2212, converting the field description into a second semantic vector. And then, converting the whole description into a semantic vector through an NLP (non line segment) model, and analyzing by synthesizing the keywords and the whole sentence.
S2213, weighting the first semantic vector and the second semantic vector to obtain a third semantic vector. In this embodiment, weighting may be performed by way of weight 1:1, and the weighted semantic vector actually enhances the weight of the keyword.
S222, similarity calculation is carried out on the third semantic vector corresponding to the original data field and the third semantic vector of the standard data field, and the standard data field with the similarity larger than a threshold value and the maximum similarity is used as the standard data field corresponding to the original data field. By the method, the corresponding vector can be matched more accurately, and the accuracy of the optimized model is improved. The degree of matching between fields can be expressed in terms of similarity, with higher similarity indicating a higher degree of matching.
In some embodiments, multiple of the available models are used to perform the same task, with different available models having different input data fields.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present application and the technical principles employed. It will be understood by those skilled in the art that the present application is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the application. Therefore, although the present application has been described in more detail with reference to the above embodiments, the present application is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present application, and the scope of the present application is determined by the scope of the appended claims.
Claims (9)
1. An intelligent monitoring method for exported goods is characterized by comprising the following steps:
establishing a data transmission channel between a data center and an external service system, wherein the external service system reports original data based on a self-defined data structure, and the original data comprises a plurality of fields;
matching in an input data field pool of available models according to data of each field contained in the original data to obtain a standard data field corresponding to the original data, wherein the input data field Chi Zhongbao contains a plurality of input data fields of the available models;
matching the model in the available model according to the standard data field corresponding to the original data to obtain a candidate model;
selecting a candidate model according to the manual instruction, and determining the corresponding relation and the conversion relation between the original data field and the data field in the selected candidate model;
cleaning the original data based on the conversion relation to obtain standard data;
inputting standard data into the candidate model to execute a prediction task, and monitoring a target parameter according to a prediction result;
selecting a candidate model according to the manual instruction, and determining the corresponding relation and the conversion relation between the original data field and the data field in the selected candidate model, wherein the method specifically comprises the following steps:
displaying an operation interface, wherein a first container, a second container, a third container and a model list are loaded in the operation interface, the sequence of candidate models in the model list is arranged in a descending order according to the matching degree of the standard data field and the candidate models, the first container, the second container and the third container respectively comprise a plurality of sub-containers, and each sub-container is used for storing data field information;
receiving a first manual instruction, and determining a selected candidate model based on the first manual instruction;
loading each first data field information of the candidate model in a first container according to the candidate model;
loading each second data field information of the original data in a second container;
receiving a second manual instruction, and moving at least part of data field information of the original data from the second container to a third container according to the second manual instruction, so that the number of the field information in the third container is the same as that of the field information in the second container;
and receiving a third manual instruction, and configuring a conversion relation between fields in the third container and fields in the first container according to the third manual instruction.
2. The intelligent monitoring method for exported goods according to claim 1, wherein in the step of loading each second data field information of the original data in the second container, the order of the second data field information is determined according to the matching relationship between the second data field information and the first data field information, and each first data field information corresponds to one second data field information with the highest matching degree.
3. The intelligent monitoring method for exported goods according to claim 1, wherein the second container is labeled with second data field information according to data quality;
the marking means that different colors, fonts and textures are adopted to represent the data quality of the corresponding field, the data quality is determined according to the integrity rate of the data of the field, and the integrity rate means the proportion of the residual data in all the data after the abnormal data in the field are removed.
4. The intelligent monitoring method for export cargo of claim 2, further comprising the steps of:
and monitoring the position of the mouse, and displaying the data summary information and the data sample corresponding to the data field information in the sub-container when the mouse is judged to stay on the sub-container of the first container, the sub-container of the second container or the sub-container of the third container.
5. The intelligent monitoring method for export cargo of claim 4, wherein the method further comprises the steps of:
establishing 3*N array a [ ], for storing data summary information address and data sample address;
in the step of displaying the data summary information and the data samples corresponding to the data field information in the sub-container, acquiring data summary information addresses and data samples according to the information in the array a [ ] [ ] corresponding to the sub-container;
wherein, a 0 [ ] is used to store the data abstract information address and data sample address corresponding to the first data field information; a 1 [ ] is used to store the data abstract information address and data sample address corresponding to the second data field information; and a [2] [ ] is used for storing the data summary information address and the data sample address corresponding to the second data field information moved to the third container.
6. The intelligent monitoring method for exported goods according to claim 1, wherein the matching in the pool of input data fields of available models to obtain the standard data fields corresponding to the raw data specifically comprises:
screening candidate standard data fields in an input data field pool according to the data types of the original data fields;
and matching the corresponding standard data field in the candidate standard data field according to the field description of the original data field.
7. The intelligent monitoring method for exported goods according to claim 6, wherein matching the corresponding standard data field in the candidate standard data field according to the field description of the original data field specifically comprises:
respectively processing the field description of the original data field and the field description of the standard data field as follows:
extracting keywords from the field description according to the high-frequency word bank, and converting the keywords into a first semantic vector;
converting the field description into a second semantic vector;
weighting the first semantic vector and the second semantic vector to obtain a third semantic vector;
and performing similarity calculation on the third semantic vector corresponding to the original data field and the third semantic vector of the standard data field, and taking the standard data field with the similarity larger than a threshold value and the maximum similarity as the standard data field corresponding to the original data field.
8. The intelligent monitoring method for export goods as claimed in claim 1, wherein a plurality of said available models are used to perform the same task, and the input data fields of different available models are different.
9. The intelligent monitoring method for exported goods according to claim 1, wherein the matching of the model in the available model according to the standard data field corresponding to the original data to obtain the candidate model specifically comprises:
taking input data fields of available models as a first set, and taking standard data fields corresponding to original data as a second set;
when available models exist, wherein the first set belongs to the subset of the second set, the corresponding available models are used as candidate models;
when there is no own available model for which the first set belongs to the second set, the candidate model is determined according to the proportion of elements in the intersection of the first set and the second set to the first set.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210855751.6A CN115080553B (en) | 2022-07-21 | 2022-07-21 | Intelligent monitoring method for export goods |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210855751.6A CN115080553B (en) | 2022-07-21 | 2022-07-21 | Intelligent monitoring method for export goods |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115080553A CN115080553A (en) | 2022-09-20 |
| CN115080553B true CN115080553B (en) | 2022-11-08 |
Family
ID=83260682
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210855751.6A Active CN115080553B (en) | 2022-07-21 | 2022-07-21 | Intelligent monitoring method for export goods |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115080553B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103036911A (en) * | 2013-01-09 | 2013-04-10 | 深圳中兴网信科技有限公司 | Business data processing system and business data processing method |
| CN107240014A (en) * | 2017-04-28 | 2017-10-10 | 天合泽泰（厦门）征信服务有限公司 | A kind of credit rating method based on enterprise's reference business |
| CN113807066A (en) * | 2021-09-16 | 2021-12-17 | 东软集团股份有限公司 | Chart generation method and device and electronic equipment |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| FI120755B (en) * | 2003-06-06 | 2010-02-15 | Tieto Oyj | Processing of data record to find correspondingly a reference data set |
| US20090326995A1 (en) * | 2007-04-23 | 2009-12-31 | Sorisi Joseph A | Apparatuses, methods and systems for a trade business card |
| US10387882B2 (en) * | 2015-07-01 | 2019-08-20 | Klarna Ab | Method for using supervised model with physical store |
| US20180096018A1 (en) * | 2016-09-30 | 2018-04-05 | Microsoft Technology Licensing, Llc | Reducing processing for comparing large metadata sets |
| US10908788B2 (en) * | 2019-05-16 | 2021-02-02 | Oracle International Corporation | Automated process discovery and facilitation within a cloud business application |
-
2022
- 2022-07-21 CN CN202210855751.6A patent/CN115080553B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103036911A (en) * | 2013-01-09 | 2013-04-10 | 深圳中兴网信科技有限公司 | Business data processing system and business data processing method |
| CN107240014A (en) * | 2017-04-28 | 2017-10-10 | 天合泽泰（厦门）征信服务有限公司 | A kind of credit rating method based on enterprise's reference business |
| CN113807066A (en) * | 2021-09-16 | 2021-12-17 | 东软集团股份有限公司 | Chart generation method and device and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115080553A (en) | 2022-09-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP0591615B1 (en) | Information management apparatus dealing with waste and waste recycle planning supporting apparatus | |
| CN110888857A (en) | Data label generation method, device, terminal and medium based on neural network | |
| Sun et al. | Economic lot scheduling problem in a remanufacturing system with returns at different quality grades | |
| CN106991175A (en) | A kind of customer information method for digging, device, equipment and storage medium | |
| CN115526652A (en) | Client loss early warning method and system based on machine learning | |
| US20180260774A1 (en) | System and Method for Providing an Intelligent Configuration Rationalization Solution | |
| CN112435153A (en) | Cargo twisting platform and method for transport ship | |
| CN116128254A (en) | Embedded intelligent manufacturing resource configuration method and terminal system based on edge calculation | |
| CN115080553B (en) | Intelligent monitoring method for export goods | |
| Jia et al. | Normalized projection-based group decision making method and application to marine equipment reliability assessment | |
| CN111738870B (en) | Method and platform for identifying insurance risk of engineering performance guarantee based on characteristic engineering | |
| CN117273400A (en) | Enterprise resource planning intelligent merging and upgrading early warning method and system | |
| US20100153382A1 (en) | Systems and methods for the matching of materials data to parts data | |
| CN111611353B (en) | Screening method, screening device, electronic equipment and computer readable storage medium | |
| Wirth et al. | Detecting Early Indicator Cars in an Automotive Database: A Multi-Strategy Approach. | |
| Gao et al. | Synchronized entry-traffic flow prediction for regional expressway system based on multidimensional tensor | |
| CN114066055A (en) | Method, device and server for predicting late-stage approach of vehicle in logistics transportation | |
| CN113762415A (en) | Neural network-based intelligent matching method and system for automobile financial products | |
| EP0606474B1 (en) | Apparatus and method for creating membership functions | |
| CN111931065A (en) | Business opportunity recommendation method, system, electronic device and medium based on LSTM model | |
| CN113256180B (en) | Customer service work order information intelligent dynamic loading method and system based on machine learning | |
| CN111626666A (en) | Distributed storage yard mode | |
| CN115640335B (en) | Enterprise portrait-based enterprise analysis method, system and cloud platform | |
| CN117194356A (en) | Renewable resource supply chain data sharing system and method | |
| CN117788044A (en) | Method, system, medium and electronic equipment for predicting vehicle change intention |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |