CN114331299A - Data processing method and device, electronic equipment and computer readable storage medium - Google Patents
Data processing method and device, electronic equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN114331299A CN114331299A CN202210237655.5A CN202210237655A CN114331299A CN 114331299 A CN114331299 A CN 114331299A CN 202210237655 A CN202210237655 A CN 202210237655A CN 114331299 A CN114331299 A CN 114331299A
- Authority
- CN
- China
- Prior art keywords
- data
- model
- target
- local
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The embodiment of the invention discloses a data processing method, a data processing device, electronic equipment and a computer readable storage medium. Each local model corresponds to different data sources, and each data source has an incidence relation. Therefore, on the basis of not transmitting data source information, the target task amount is determined by processing the target data in different data sources through each local model and each global model, and the accuracy of determining the target task amount is improved.
Description
Technical Field
The invention relates to the technical field of transportation, in particular to a data processing method, a data processing device, electronic equipment and a computer readable storage medium.
Background
Currently, in the transportation field, due to data isolation among business systems, task prediction can be performed based on historical data of a single scene. However, since data between business systems have a certain correlation, there is a certain error if a single data source is used to guide future task situations.
Disclosure of Invention
In view of this, embodiments of the present invention provide a data processing method, an apparatus, an electronic device, and a computer-readable storage medium, so as to determine a task amount using data in different data sources on the basis of not transferring data source information, and improve an accuracy rate of determining a target task amount.
In a first aspect, an embodiment of the present invention provides a data processing method, where the method includes:
acquiring target data in data sources corresponding to local models, wherein the local models correspond to different data sources, and the data sources have an incidence relation;
inputting the target data into corresponding local models for processing so as to obtain initial task quantities corresponding to the local models;
and inputting each preliminary task quantity into a global model for processing so as to obtain a target task quantity.
Further, the target data includes historical task data within a predetermined time period within a target area, the historical task data including task location information and time information.
Further, the historical task data is transportation task data, and the transportation task data comprises two-wheel vehicle task data, network appointment task data and/or freight task data.
Further, the target task amount is used for characterizing the number of target objects required by the target area in the corresponding time period.
Further, each of the local model and the global model is determined based on the following steps:
responding to the fact that model training does not reach preset conditions, and adjusting local model parameters based on current global model parameters;
inputting the training data of each local model into the corresponding local model for processing, and adjusting the parameters of each local model according to the data result of each local model;
adjusting global model parameters according to the adjusted local model parameters;
and responding to the fact that model training reaches a preset condition, and obtaining each trained local model and global model based on each adjusted local model parameter and global model parameter.
Further, the preset condition is that the number of times of model training reaches a preset value.
Further, the method further comprises:
and determining the contribution value of each local model according to the trained global model and the output result of each local model, wherein the contribution value is used for evaluating the data source quality of the corresponding local model.
In a second aspect, an embodiment of the present invention provides a data processing apparatus, where the apparatus includes:
the acquisition unit is used for acquiring target data in data sources corresponding to local models, wherein the local models correspond to different data sources, and the data sources have an incidence relation;
the preliminary determining unit is used for inputting the target data into corresponding local models to be processed so as to obtain preliminary task quantities corresponding to the local models;
and the target determining unit is used for inputting each preliminary task quantity into the global model for processing so as to obtain a target task quantity.
In a third aspect, embodiments of the present invention provide a computer program product comprising a computer program/instructions which, when executed by a processor, implement the method as defined in any one of the above.
In a fourth aspect, an embodiment of the present invention provides an electronic device, including a memory and a processor, the memory being configured to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of the above.
In a fifth aspect, the present invention provides a computer-readable storage medium, in which a computer program is stored, and the computer program, when executed by a processor, implements the method steps as described in any one of the above.
According to the technical scheme of the embodiment of the invention, the target data in the data source corresponding to each local model is obtained, the target data is input to the corresponding local model for processing, the initial task amount corresponding to each local model is obtained, and then the initial task amount is input to the global model for processing, so that the target task amount is obtained. Therefore, on the basis of not transmitting data source information, the target task amount is determined by processing the target data in different data sources through each local model and each global model, and the accuracy of determining the target task amount is improved.
Drawings
The above and other objects, features and advantages of the present invention will become more apparent from the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:
FIG. 1 is a schematic diagram of a target task volume determination system according to an embodiment of the invention;
FIG. 2 is an architecture diagram of a target workload model according to an embodiment of the invention;
FIG. 3 is a flow chart of a data processing method of an embodiment of the present invention;
FIG. 4 is another flow chart of a data processing method of an embodiment of the present invention;
FIG. 5 is a flow chart for determining a local model and a global model according to an embodiment of the present invention;
FIG. 6 is a flow chart of adjusting model parameters according to an embodiment of the present invention;
FIG. 7 is a schematic diagram of a data processing apparatus according to an embodiment of the present invention;
fig. 8 is a schematic diagram of an electronic device of an embodiment of the invention.
Detailed Description
The present invention will be described below based on examples, but the present invention is not limited to only these examples. In the following detailed description of the present invention, certain specific details are set forth. It will be apparent to one skilled in the art that the present invention may be practiced without these specific details. Well-known methods, procedures, components and circuits have not been described in detail so as not to obscure the present invention.
Further, those of ordinary skill in the art will appreciate that the drawings provided herein are for illustrative purposes and are not necessarily drawn to scale.
Unless the context clearly requires otherwise, throughout the description, the words "comprise", "comprising", and the like are to be construed in an inclusive sense as opposed to an exclusive or exhaustive sense; that is, what is meant is "including, but not limited to".
In the description of the present invention, it is to be understood that the terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. In addition, in the description of the present invention, "a plurality" means two or more unless otherwise specified.
The problem of data information island exists all the time, and how to effectively solve some technical problems by using data in different data sources has important significance on the premise of ensuring data safety and specification.
In the field of transportation, taking the use scenes of the network car booking business and the two-wheeled vehicle business (including a single vehicle and a shared single vehicle) as an example, when the supply and demand of the network car booking order in a specific time period in a target area are unbalanced, a plurality of network car booking users can select the two-wheeled vehicle to meet travel demands due to overlong waiting time. That is, the order demand data of the network contract order will affect the two-wheel vehicle business. Based on the method, the order quantity of the two-wheel vehicle can be predicted through the network appointment vehicle and the service data corresponding to the two-wheel vehicle, so that operation and maintenance personnel can distribute a proper number of vehicles in a target area in advance, the circulation rate of the vehicles is improved, and the user can go out conveniently. However, since the net appointment vehicle and the two-wheeled vehicle belong to different service lines, and data isolation exists between the different service lines, in the prior art, the target task condition at the future time can only be determined based on the data corresponding to a single service line, and the accuracy of the determined target task amount needs to be improved.
Based on the above, the embodiment of the present invention provides a data processing method, which is based on the federal learning thought, and can predict the target task volume by using data information in different data sources while not directly transmitting data in different data sources, so as to improve the accuracy of determining the target task volume. In addition, the data processing method of the embodiment can also determine accurate urban global OD (traffic start and stop point) demand information by using data in different data sources on the basis of not acquiring data in data sources of each network car booking company.
In the following, the data processing method in this embodiment is described with the network appointment car and two-wheel vehicle business, and it should be understood that the technical solution in this embodiment is not limited to be applied to the network appointment car and the two-wheel vehicle, but may also be applied to transportation scenes such as designated driving, freight transportation, and tailgating.
Fig. 1 is a schematic diagram of a target task amount determination system according to an embodiment of the present invention. As shown in fig. 1, the target task amount determination system includes a data source layer 10, a data layer 20, a model layer 30, and a scheduling layer 40. The data source layer 10 includes a plurality of different data sources, and is used for storing all data (i.e., source data) corresponding to each data source. The data layer 20 is used for storing target data which is screened out from each data source and is related to the target task amount. The model layer 30 is used to process the target data from different data layers to determine the target task amount under the service line to be determined. The scheduling layer 40 is configured to reasonably allocate resources according to the target task amount after the model layer outputs the target task amount.
Optionally, the data source layer, the data layer, the model layer, and the scheduling layer in this embodiment may be deployed in the same server, and different execution modules implement corresponding functions; or may be servers distributed independently, and the data transfer is realized through communication connection between the servers.
In an optional implementation manner, the type of the data source in this embodiment may correspond to different business lines in the same enterprise, or may correspond to the same business line in different enterprises, or may correspond to different business lines in different enterprises. Suppose that in the same field, an enterprise A and an enterprise B exist, and the enterprise A and the enterprise B operate a network car booking service line, a shared bicycle service line and a shared electric bicycle service line. Since all service lines between the enterprise a and the enterprise B may have mutual influence, taking the finally determined target task amount as the number of the shared single vehicles as an example, the data source in this embodiment may be a data source corresponding to the network car booking service line, the shared single vehicle service line, and the shared electric single vehicle service line in the company a, a data source corresponding to the shared single vehicle service line in the company a and the shared single vehicle service line in the company B, a data source corresponding to the network car booking service line, the shared single vehicle service line, and the shared electric single vehicle service line in the company a, and a data source corresponding to the network car booking service line, the shared single vehicle service line, and the shared electric single vehicle service line in the company B.
Meanwhile, due to different storage modes, the number of the data sources of the same type in the same enterprise can be one or more. In view of this, the number of data sources in the present embodiment is determined according to the actual usage scenario of the target task amount determination system, and is not limited herein.
For the sake of understanding, the following description will be made with respect to the processing procedure of the target task amount system in a specific usage scenario. When the single-vehicle throwing quantity of a target area in a preset time period is determined according to network appointment vehicle and two-wheel vehicle (including shared single vehicles and shared electric single vehicles) service data in the same company, a data source layer comprises three types of data sources, the quantity of each type of data source is one, and all order data corresponding to the network appointment vehicle service, all order data corresponding to the shared single vehicle service and all order data corresponding to the shared electric single vehicle service are stored in each data source respectively. The data layer stores target data which are screened from all data sources and are related to the network appointment orders, the shared bicycle orders and the shared electric bicycles, wherein the target data comprise time information, position information, identification information and other information. The model layer respectively processes the target data from different sources and determines the target task amount of the target area in a predetermined period of time later (i.e. the number of single-vehicle shots in the target area in the predetermined period of time). And finally, the dispatching layer puts in corresponding number of bicycles according to the target task amount determined by the model layer.
FIG. 2 is an architecture diagram of a target workload model according to an embodiment of the invention. As shown in fig. 2, the model layer 30 in the present embodiment includes a target task volume model adopting the federal learning system architecture, and the architecture of the target task volume model includes at least two training nodes 31 and an aggregation node 32. Each training node 31 acquires target data from a corresponding data source, processes the target data through an internally deployed local model, and determines a preliminary task amount. The aggregation node 32 receives the preliminary task quantities sent by the training nodes 31, processes the preliminary task quantities through an internally deployed global model, and determines a target task quantity.
It should be understood that the number of training nodes corresponds to the number of data sources, and may be adjusted according to actual use requirements, and is not limited herein.
Further, continuing with the example above, the data source layer includes three types of data sources, one for each type of data source. As shown in fig. 2, the target task amount model architecture in this embodiment includes three training nodes 31 and one aggregation node 32, each training node 31 is deployed with a local model, and the aggregation node 32 is deployed with a global model. And each local model acquires corresponding target data from corresponding source data, processes the target data and determines the corresponding preliminary task amount. And the global model acquires the initial task amount determined by each local model, processes each initial task amount and determines a target task amount. Therefore, the target task quantity is determined through the processing process on the premise of protecting the data privacy security in each data source. Meanwhile, when the target task amount is determined, different influence factors corresponding to the target data in different data sources are considered, so that the accuracy of the determined target task amount can be improved.
Because the user groups corresponding to different data sources are not completely overlapped, when the target task quantity determining system is used, the data security is considered, firstly, the encryption-based user sample alignment technology is utilized, common users of the two parties are confirmed on the premise that data in the data sources are not disclosed, and the users which are not mutually overlapped are not exposed, so that the local models corresponding to the training nodes are modeled by combining the characteristics of the users. After the common user group is determined, the local models in the model layer can be trained by using the target data corresponding to the common user group. Meanwhile, in order to ensure the confidentiality of data in the model training process, encryption training needs to be performed by means of a global model. For the sake of understanding, taking a linear regression model as an example, the training process includes the following steps: the global model distributes the public key to different local models for encrypting the data to be exchanged in the training process; the intermediate result of the gradient is interactively calculated among the local models in an encryption mode; and each local model is calculated based on the encrypted gradient value, and the calculation result is summarized to the global model. The global model calculates the total gradient value through the summary result and decrypts the total gradient value; after decryption, the global model respectively returns the decrypted gradient values to different local models, and each local model updates the parameters of each model according to the gradient. And (5) iterating the steps until the loss function is skilled, namely completing the whole model training process.
It should be understood that in the sample pair and model training process, the target data corresponding to each local model is kept in the local data source, and data interaction in the model training process does not cause data privacy disclosure. Therefore, each data source realizes a cooperative training model in the use process of the target task quantity determination system. In addition, in the use process of each local model and the global model after the training is finished, the use effect of each local model and the global model can be expressed in the use process and is recorded on the permanent data recording mechanism in the form of a contribution value, the model effect obtained by providing a data source with a large amount of data is better, and the contribution value of a data provider corresponding to the data source with the better model effect is larger. Through the feedback of the contribution value, not only can the contribution value be used as a profit distribution standard to distribute profits for each data provider, but also more data providers can be stimulated to be added into the target mission volume control system.
Fig. 3 is a flowchart of a data processing method of an embodiment of the present invention. As shown in fig. 3, the data processing method in the present embodiment includes the following steps.
In step S110, target data in the data source corresponding to each local model is obtained. Each local model corresponds to different data sources, and each data source has an incidence relation.
Optionally, the target data in this embodiment includes historical task data within a predetermined time period within the target area. The historical task data comprises task position information and time information.
Further, when determining the target area, the embodiment first performs mesh division on the actual spatial area according to a preset area size, determines a plurality of mesh areas, and determines a space corresponding to each mesh area as a corresponding target area. The preset area size of each grid area is set to be 3km x 3 km. The predetermined time may be set to one week, one month, or one quarter. It should be understood that the specific duration corresponding to the size of the grid area and the predetermined time in this embodiment may also be adjusted according to an actual usage scenario.
Further, the historical task data in this embodiment is transportation task data, and the transportation task data includes two-wheel vehicle task data, net appointment task data, and/or freight task data. The two-wheel vehicle task data comprises shared single-vehicle task data and shared electric single-vehicle task data.
Specifically, the target data in this embodiment includes network appointment task data, shared bicycle task data, and shared electric bicycle task data within a predetermined time period in the target area.
In step S120, the target data is input to the corresponding local model for processing, so as to obtain a preliminary task amount corresponding to each local model.
In this embodiment, different target data are respectively input into corresponding local models for processing, and an output result of each local model is determined as a corresponding preliminary task amount. Specifically, each local model receives corresponding target data respectively, performs data on the corresponding target data, outputs the shared bicycle task amount under the influence of the historical network booking order, the shared bicycle task amount under the influence of the historical shared bicycle order and the shared bicycle task amount under the influence of the historical shared electric bicycle order respectively, and determines the shared bicycle task amount under the influence of the historical network booking order, the shared bicycle task amount under the influence of the historical shared bicycle order and the shared bicycle task amount under the influence of the historical shared electric bicycle order as initial task amounts output under the action of the corresponding local models respectively.
Optionally, each local model in this embodiment adopts the same model structure, and each local model may adopt a Recurrent Neural Network (RNN), a Long Short-term Memory Network (LSTM), or a Graph Neural Network (GNN) and Point of Interest (POI) information for modeling. The recurrent neural network is a recurrent neural network in which sequence data is input, recursion (recursion) is performed in the evolution direction of the sequence, and all nodes (recurrent units) are connected in a chain manner. Compared with a basic neural network, the recurrent neural network not only establishes the right connection between layers, but also establishes the right connection between neurons between the layers, and can effectively process data with sequence characteristics. The long-short term memory network is a variant of the recurrent neural network, can well model the time sequence, and is suitable for processing and predicting important events with very long intervals and delays in the time sequence. Meanwhile, compared with a cyclic neural network, a plurality of activation functions are added in the long-term and short-term memory network, so that the probability of gradient explosion is reduced, and the prediction result is more accurate. The Graph neural network provides a Graph Embedding (Graph Embedding) technology which can be used for Graph characteristic learning, and by introducing traditional Graph analysis, various operations such as multilayer Graph convolution and the like and activation functions are carried out to finally obtain the representation of each node, so that tasks such as node classification, link prediction, Graph and sub-Graph generation and the like are facilitated, the processing capacity of deep learning on non-Euclidean data can be further expanded, and a method for extracting characteristics on the non-regular data is provided.
In step S130, each preliminary task amount is input to the global model for processing, so as to obtain a target task amount.
In this embodiment, the target task amount is used to represent the number of target objects required by the target area in the corresponding time period. Specifically, the target object in the present embodiment is a shared bicycle. And processing the shared bicycle task quantity under the influence of the historical network car booking orders, the shared bicycle task quantity under the influence of the historical shared bicycle orders and the shared bicycle task quantity under the influence of the historical shared electric bicycle orders output by each local model through the global model, and determining the final shared bicycle quantity. Thus, the determination of the target task amount is realized through the above processing procedure.
According to the technical scheme of the embodiment of the invention, the target data in the data source corresponding to each local model is obtained, the target data is input to the corresponding local model for processing, the initial task amount corresponding to each local model is obtained, and then the initial task amount is input to the global model for processing, so that the target task amount is obtained. Therefore, the target task amount is determined by processing the target data in different data sources through the local models and the global model. Meanwhile, influence factors corresponding to target data in different data sources are comprehensively considered when the target task amount is determined, so that the accuracy of the determined target task amount is higher.
Fig. 4 is another flowchart of a data processing method according to an embodiment of the present invention. As shown in fig. 4, the data processing method in the present embodiment includes the following steps.
In step S210, target data in the data source corresponding to each local model is obtained. Each local model corresponds to different data sources, and each data source has an incidence relation.
Optionally, the target data in this embodiment includes historical task data within a predetermined time period within the target area. The historical task data comprises task position information and time information.
Further, the historical task data in this embodiment is transportation task data, and the transportation task data includes two-wheel vehicle task data, net appointment task data, and/or freight task data.
In step S220, the target data is input to the corresponding local model for processing, so as to obtain a preliminary task amount corresponding to each local model.
In step S230, each preliminary task amount is input to the global model for processing, so as to obtain a target task amount.
In this embodiment, the target task amount is used to represent the number of target objects required by the target area in the corresponding time period.
In step S240, a first contribution value corresponding to each local model is determined according to the preliminary task amount output by each local model and the target task amount output by the global model.
In this embodiment, after the target task amount is obtained, a first contribution value corresponding to each local model is determined according to output results of each local model and the global model, and the contribution degree of training nodes where different local models are located to the target task amount is determined through the first contribution value.
In step S250, the corresponding profit is allocated according to the first contribution value corresponding to each local model.
In this embodiment, the contribution degree of the data provider corresponding to each training node is determined according to the first contribution value, and the corresponding profit is distributed to each data provider based on the contribution degree of each data provider, so that the target task amount determiner can conveniently and fairly distribute the profit according to the contribution degree of the data provider corresponding to each training node, and the improvement of the service enthusiasm of the data provider corresponding to each training node is facilitated.
According to the technical scheme of the embodiment of the invention, the target data in the data source corresponding to each local model is obtained, the target data is input to the corresponding local model for processing, the initial task amount corresponding to each local model is obtained, and then the initial task amount is input to the global model for processing, so that the target task amount is obtained. Therefore, the target task quantity with higher accuracy is determined by processing the target data in different data sources through each local model and the global model. Meanwhile, after the target task amount is determined, earnings are distributed according to the contribution values corresponding to the local models, and the service enthusiasm of the data providers corresponding to the training nodes is improved while fair earnings distribution is achieved.
FIG. 5 is a flow chart for determining a local model and a global model according to an embodiment of the present invention. As shown in fig. 5, each of the local model and the global model in the present embodiment is determined based on the following steps.
In step S310, in response to the model training not reaching the preset condition, the local model parameters are adjusted based on the current global model parameters.
In step S320, the training data of each local model is input into the corresponding local model for processing, and the parameters of each local model are adjusted according to the data result of each local model.
In step S330, global model parameters are adjusted according to the adjusted local model parameters.
In step S340, in response to that the model training reaches a preset condition, each trained local model and global model is obtained based on each adjusted local model parameter and global model parameter.
In an optional implementation manner, the preset condition in this embodiment is that the number of times of model training reaches a preset value.
In another alternative implementation manner, after adjusting the parameters of the global model and the local models, the second contribution values of the local models are determined according to the output results of the global model and the local models after training. The second contribution value is used to evaluate the data source quality of the corresponding local model. Further, the preset condition in this embodiment is that the second contribution value corresponding to each local model does not change any more. Meanwhile, a contribution degree preset value can be set, when a second contribution value corresponding to the local model is smaller than the contribution degree preset value, the data source quality of the local model is determined to be poor, and the global model can be retrained by replacing other types of data sources, so that the accuracy of determining the target task quantity output by the global model is higher.
According to the technical scheme of the embodiment, each local model parameter is adjusted based on the current global model parameter, after the training data of each local model is input to the corresponding local model for processing, each local model parameter is further adjusted according to the data result of each local model, the global model parameter is adjusted based on each adjusted local model parameter until the model training reaches the preset adjustment, each adjusted local model parameter and each adjusted global model parameter are determined as the final local model and the final global model, the model training process of each local model and the global model is realized, and the target task quantity is determined based on the target data in different data sources and each trained local model and global model.
FIG. 6 is a flow chart of adjusting model parameters according to an embodiment of the present invention. As shown in fig. 6, each local model and global model parameter in the present embodiment is implemented by the following steps.
In step S410, initial global model parameters of the global model are obtained.
In this embodiment, the initial global model parameters are manually set based on experience.
In step S420, local model parameters corresponding to the local models are determined based on the current global model parameters.
In this embodiment, the current global model parameter may be an initial global model parameter when training starts, or may be a global model parameter corresponding to the global model when the last training process ends.
In step S430, local model parameters corresponding to each local model are further adjusted according to the training data.
In step S440, global model parameters corresponding to the current global model are adjusted according to the adjusted local model parameters corresponding to each local model.
In step S450, it is determined whether the current parameter adjustment times reaches a preset value.
In this embodiment, when the current parameter adjustment times reaches the preset value, the step S460 is continuously executed. And when the current parameter adjustment times does not reach the preset value, returning to execute the step S420.
In step S460, each of the local model and the global model that has been trained is determined.
In this embodiment, in response to that the current parameter adjustment times reach a preset value, the local model corresponding to each local model parameter and the global model corresponding to the global model parameter after the current parameter adjustment are determined as each trained local model and global model.
Therefore, the training of each local model and the global model is completed through the processing procedure, and each trained local model and global model are determined and then applied to the data processing procedure so as to determine the target task amount according to the target data in different data sources.
Fig. 7 is a schematic diagram of a data processing apparatus according to an embodiment of the present invention. As shown in fig. 7, the data processing apparatus in the present embodiment includes an acquisition unit 1, a preliminary determination unit 2, and a target determination unit 3. The obtaining unit 1 is configured to obtain target data in a data source corresponding to each local model. The preliminary determining unit 2 is configured to input the target data into the corresponding local model for processing, so as to obtain a preliminary task amount corresponding to each local model. The target determination unit 3 is configured to input each preliminary task amount to the global model for processing, so as to obtain a target task amount. Each local model corresponds to different data sources, and each data source has an incidence relation. The target data includes historical task data within a predetermined time period within the target area, the historical task data including task location information and time information. Furthermore, the historical task data is traffic transportation task data, the traffic transportation task data comprises two-wheel vehicle task data, network appointment task data and/or freight transportation task data, and the two-wheel vehicle task data comprises shared single-vehicle task data and shared electric single-vehicle task data. The target task amount is used for representing the number of target objects required by the target area in the corresponding time period.
According to the technical scheme of the embodiment of the invention, the target data in the data source corresponding to each local model is obtained through the obtaining unit, the target data is input to the corresponding local model for processing through the preliminary determining unit, the preliminary task quantity corresponding to each local model is obtained, and the preliminary task quantity is input to the global model for processing through the target determining unit, so that the target task quantity is obtained. Therefore, the target task amount is determined by processing the target data in different data sources through the local models and the global model.
Fig. 8 is a schematic diagram of an electronic device of an embodiment of the invention. As shown in fig. 8, the electronic device shown in fig. 8 is a general-purpose data processing apparatus including a general-purpose computer hardware structure including at least a processor 81 and a memory 82. The processor 81 and the memory 82 are connected by a bus 83. The memory 82 is adapted to store instructions or programs executable by the processor 81. Processor 81 may be a stand-alone microprocessor or a collection of one or more microprocessors. Thus, the processor 81 implements the processing of data and the control of other devices by executing instructions stored by the memory 82 to perform the method flows of embodiments of the present invention as described above. The bus 83 connects the above components together, and also connects the above components to a display controller 84 and a display device and an input/output (I/O) device 85. Input/output (I/O) devices 85 may be a mouse, keyboard, modem, network interface, touch input device, motion sensing input device, printer, and other devices known in the art. Typically, the input/output devices 85 are coupled to the system through an input/output (I/O) controller 86.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, apparatus (device) or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may employ a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations of methods, apparatus (devices) and computer program products according to embodiments of the application. It will be understood that each flow in the flow diagrams can be implemented by computer program instructions.
These computer program instructions may be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows.
These computer program instructions may also be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows.
Another embodiment of the invention is directed to a non-transitory storage medium storing a computer-readable program for causing a computer to perform some or all of the above-described method embodiments.
That is, as can be understood by those skilled in the art, all or part of the steps in the method for implementing the embodiments described above may be accomplished by specifying the relevant hardware through a program, where the program is stored in a storage medium and includes several instructions to enable a device (which may be a single chip, a chip, or the like) or a processor (processor) to execute all or part of the steps of the method described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A method of data processing, the method comprising:
acquiring target data in data sources corresponding to local models, wherein the local models correspond to different data sources, and the data sources have an incidence relation;
inputting the target data into corresponding local models for processing so as to obtain initial task quantities corresponding to the local models;
and inputting each preliminary task quantity into a global model for processing so as to obtain a target task quantity.
2. The method of claim 1, wherein the target data comprises historical task data over a predetermined period of time within a target area, the historical task data comprising task location information and time information.
3. The method of claim 2, wherein the historical mission data is transportation mission data, the transportation mission data including two-wheel vehicle mission data, net appointment mission data, and/or freight mission data.
4. The method of claim 1, wherein the target task volume is used to characterize a number of target objects required by a target area over a corresponding time period.
5. The method of claim 1, wherein each of the local model and the global model is determined based on the steps of:
responding to the fact that model training does not reach preset conditions, and adjusting local model parameters based on current global model parameters;
inputting the training data of each local model into the corresponding local model for processing, and adjusting the parameters of each local model according to the data result of each local model;
adjusting global model parameters according to the adjusted local model parameters;
and responding to the fact that model training reaches a preset condition, and obtaining each trained local model and global model based on each adjusted local model parameter and global model parameter.
6. The method of claim 5, wherein the predetermined condition is that the number of model training times reaches a predetermined value.
7. The method of claim 5, further comprising:
and determining the contribution value of each local model according to the trained global model and the output result of each local model, wherein the contribution value is used for evaluating the data source quality of the corresponding local model.
8. A data processing apparatus, characterized in that the apparatus comprises:
the acquisition unit is used for acquiring target data in data sources corresponding to local models, wherein the local models correspond to different data sources, and the data sources have an incidence relation;
the preliminary determining unit is used for inputting the target data into corresponding local models to be processed so as to obtain preliminary task quantities corresponding to the local models;
and the target determining unit is used for inputting each preliminary task quantity into the global model for processing so as to obtain a target task quantity.
9. An electronic device comprising a memory and a processor, wherein the memory is configured to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method of any of claims 1-7.
10. A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, which computer program, when being executed by a processor, carries out the method steps of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210237655.5A CN114331299B (en) | 2022-03-11 | 2022-03-11 | Data processing method and device, electronic equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210237655.5A CN114331299B (en) | 2022-03-11 | 2022-03-11 | Data processing method and device, electronic equipment and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114331299A true CN114331299A (en) | 2022-04-12 |
| CN114331299B CN114331299B (en) | 2022-07-22 |
Family
ID=81033508
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210237655.5A Active CN114331299B (en) | 2022-03-11 | 2022-03-11 | Data processing method and device, electronic equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114331299B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108960476A (en) * | 2018-03-30 | 2018-12-07 | 山东师范大学 | Shared bicycle method for predicting and device based on AP-TI cluster |
| WO2019109340A1 (en) * | 2017-12-08 | 2019-06-13 | Telefonaktiebolaget Lm Ericsson (Publ) | Method and apparatus for managing sharing vehicle |
| CN110047279A (en) * | 2019-04-04 | 2019-07-23 | 东南大学 | A method of shared bicycle scheduling quantum is determined based on order data |
| CN112419729A (en) * | 2020-11-23 | 2021-02-26 | 北京嘀嘀无限科技发展有限公司 | Method and system for determining decision strategy according to predicted traffic flow |
| CN112766591A (en) * | 2021-01-27 | 2021-05-07 | 同济大学 | Shared bicycle scheduling method |
| CN113163366A (en) * | 2021-04-25 | 2021-07-23 | 武汉理工大学 | Privacy protection model aggregation system and method based on federal learning in Internet of vehicles |
| CN113570065A (en) * | 2021-07-08 | 2021-10-29 | 国网河北省电力有限公司信息通信分公司 | Data management method, device and equipment based on alliance chain and federal learning |
-
2022
- 2022-03-11 CN CN202210237655.5A patent/CN114331299B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019109340A1 (en) * | 2017-12-08 | 2019-06-13 | Telefonaktiebolaget Lm Ericsson (Publ) | Method and apparatus for managing sharing vehicle |
| CN108960476A (en) * | 2018-03-30 | 2018-12-07 | 山东师范大学 | Shared bicycle method for predicting and device based on AP-TI cluster |
| CN110047279A (en) * | 2019-04-04 | 2019-07-23 | 东南大学 | A method of shared bicycle scheduling quantum is determined based on order data |
| CN112419729A (en) * | 2020-11-23 | 2021-02-26 | 北京嘀嘀无限科技发展有限公司 | Method and system for determining decision strategy according to predicted traffic flow |
| CN112766591A (en) * | 2021-01-27 | 2021-05-07 | 同济大学 | Shared bicycle scheduling method |
| CN113163366A (en) * | 2021-04-25 | 2021-07-23 | 武汉理工大学 | Privacy protection model aggregation system and method based on federal learning in Internet of vehicles |
| CN113570065A (en) * | 2021-07-08 | 2021-10-29 | 国网河北省电力有限公司信息通信分公司 | Data management method, device and equipment based on alliance chain and federal learning |
Non-Patent Citations (3)
| Title |
|---|
| XINYU LI等: "Improving short-term bike sharing demand forecast through an irregular convolutional neural network", 《ARXIV》 * |
| 张徐: "共享单车需求量预测及调度优化算法的研究与应用", 《中国优秀博硕士学位论文全文数据库（硕士）工程科技Ⅱ辑》 * |
| 滴滴云说运维: "滴滴共享两轮车供需策略探讨", 《HTTPS://BLOG.CSDN.NET/DIDI_CLOUD/ARTICLE/DETAILS/117811700》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114331299B (en) | 2022-07-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Schuijbroek et al. | Inventory rebalancing and vehicle routing in bike sharing systems | |
| Xu et al. | Large-scale order dispatch in on-demand ride-hailing platforms: A learning and planning approach | |
| Liu et al. | Context-aware taxi dispatching at city-scale using deep reinforcement learning | |
| Ma et al. | Development of a data-driven platform for transit performance measures using smart card and GPS data | |
| CN106250987A (en) | A kind of machine learning method, device and big data platform | |
| CN107094165A (en) | Distribution capacity is determined, dispatching task obtains, dispenses resource regulating method and equipment | |
| Nuzzolo et al. | Individual utility‐based path suggestions in transit trip planners | |
| CN109118224A (en) | Proof of work method, apparatus, medium and the electronic equipment of block chain network | |
| CN114303162A (en) | The reinforcement learning method for rewarding drivers: generative countermeasure network for driver-system interaction | |
| CN107392389A (en) | Taxi dispatching processing method based on ARIMA models | |
| Flouris et al. | FERARI: a prototype for complex event processing over streaming multi-cloud platforms | |
| CN111753006A (en) | Prediction system and method based on federal learning | |
| CN109685603A (en) | A kind of we rent tools method, system and terminal | |
| Box et al. | An automated signalized junction controller that learns strategies from a human expert | |
| Fabbiani et al. | Distributed big data analysis for mobility estimation in intelligent transportation systems | |
| Talukder et al. | Trajectory-based signal control in mixed connected vehicle environments | |
| Lotfi et al. | Modeling framework and decomposition scheme for on‐demand mobility services with ridesharing and transfer | |
| US10824956B1 (en) | System and method for price estimation of reports before execution in analytics | |
| CN117561517A (en) | Computer-implemented apparatus and method for predicting traffic conditions in a route planning application | |
| CN103119617B (en) | The automatic flow of ticketing system produces | |
| Rodriguez-Deniz et al. | Robust real-time delay predictions in a network of high-frequency urban buses | |
| TW201738776A (en) | Real-time streaming record data analysis system and method using a computed result obtained by a computing process as the system and method for increasing the future computing efficiency | |
| CN114331299B (en) | Data processing method and device, electronic equipment and computer readable storage medium | |
| Wang et al. | GPS data in urban online Car-hailing: simulation on optimization and prediction in reducing void cruising distance | |
| KR102477717B1 (en) | Big data-based carbon emission calculation for each situation and automatic trading method, device and system based on artificial intelligence |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20220829 Address after: Room 203, 2 / F, building 34, yard 8, Dongbei Wangxi Road, Haidian District, Beijing 100193 Patentee after: Beijing Qisheng Technology Co.,Ltd. Patentee after: Xiamen Qiwen Technology Co.,Ltd. Address before: Room 203, 2 / F, building 34, yard 8, Dongbei Wangxi Road, Haidian District, Beijing 100193 Patentee before: Beijing Qisheng Technology Co.,Ltd. |