CN110737535B - Data processing method and device based on message queue and computer equipment - Google Patents
Data processing method and device based on message queue and computer equipment Download PDFInfo
- Publication number
- CN110737535B CN110737535B CN201910849176.7A CN201910849176A CN110737535B CN 110737535 B CN110737535 B CN 110737535B CN 201910849176 A CN201910849176 A CN 201910849176A CN 110737535 B CN110737535 B CN 110737535B
- Authority
- CN
- China
- Prior art keywords
- data
- target
- result
- information
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Multi Processors (AREA)
Abstract
The invention provides a data processing method, a data processing device, computer equipment and a storage medium based on a message queue, wherein the method comprises the following steps: receiving initial data input from a designated input node; inquiring a preset data processing flow chart to find an operation type corresponding to the current operation node, and adding information of the operation type into a data head of the initial data to obtain target data; dividing target data according to a preset number of parts and putting the target data into a message queue; distributing multiple copies of first data from the message queue to each target receiving end in turn, starting a current operation node, performing operation on the first data to obtain a first target result combined by multiple copies of results, and sending the first target result to the message queue; and the data are repeatedly partitioned and distributed to a plurality of target receiving terminals for operation, and the like until the operation of the target data is completed according to the data processing flow chart, so that the automatic distribution and parallel operation of the data are realized, and the efficiency is greatly improved.
Description
Technical Field
The present invention relates to the field of data processing technologies, and in particular, to a data processing method and apparatus based on a message queue, a computer device, and a storage medium.
Background
Currently, enterprises generally process data obtained from the outside through a data processing system to obtain data required by the enterprises. The existing data processing system is basically realized based on a relational database, the flow is firstly divided into a plurality of stages, each stage uses fixed codes to carry out batch processing, then, staged data is landed on the relational database, and then, the processing of the next link is carried out. For example, in a multi-link processing task, a traditional mode heavily depends on a relational database, namely, each link outputs a result to a database such as oracle, and data is read from the database again when the next link starts.
When the data is processed in the above manner, because an automatic waiting and triggering mechanism between different data is absent, all the required data generally need to be equalized to ensure that the flow can be smoothly performed, the flow is initiated together, and the reading sequence of various data has strict regulations, so that the processing can be started after all the input data are collected, and the single-pen real-time processing cannot be realized, which not only causes poor real-time performance of data processing, but also causes low efficiency of data processing.
Disclosure of Invention
The invention mainly aims to provide a data processing method, a data processing device, computer equipment and a storage medium based on a message queue, and aims to solve the technical problem of low data processing efficiency in the prior art.
Based on the above object, the present invention provides a data processing method based on message queue, including: receiving initial data input from a designated input node;
inquiring a preset data processing flow chart according to the initial data to find the operation type of the current operation node corresponding to the initial data, and adding the information of the operation type into a data header of the initial data to obtain target data;
dividing the target data according to a preset number of parts to obtain a plurality of parts of first data, and putting the first data into a message queue;
distributing a plurality of copies of the first data from the message queue to a specified number of target receiving ends in turn;
starting the current operation node in the target receiving end through each target receiving end according to the information of the operation type, and operating the first data according to the operation type of the current operation node to obtain a plurality of pieces of result data;
receiving multiple copies of result data sent from the target receiving end, combining the multiple copies of result data to obtain a first target result, and sending the first target result to the message queue;
dividing the first target result in the message queue according to the preset number of parts to obtain a plurality of pieces of first result data, and distributing the plurality of pieces of first result data to a specified number of target receiving ends in turn;
inquiring the data processing flow chart according to the first result data through the target receiving terminal to find the operation type of a next operation node corresponding to the target data, starting the next operation node, and operating the first result data according to the operation type of the next operation node to obtain a plurality of pieces of result data after the operation of the next operation node;
and receiving a plurality of copies of result data after next operation node operation sent by the target receiving end, combining the plurality of copies of result data after next operation node operation to obtain a second target result, sending the second result to the message queue, and so on until the operation of the target data is completed according to the data processing flow chart.
The invention also provides a data processing device based on the message queue, which comprises:
a receiving data unit for receiving initial data input from a designated input node;
the searching type unit is used for searching a preset data processing flow chart according to the initial data to find the operation type of the current operation node corresponding to the initial data, and adding the information of the operation type into a data header of the initial data to obtain target data;
the data dividing unit is used for dividing the target data according to a preset number of parts to obtain a plurality of parts of first data and putting the first data into a message queue;
the distribution data unit is used for distributing a plurality of copies of the first data from the message queue to a specified number of target receiving ends in turn;
the operation data unit is used for starting the current operation node in the target receiving end according to the information of the operation type through each target receiving end and operating the first data according to the operation type of the current operation node to obtain a plurality of pieces of result data;
a receiving result unit, configured to receive multiple copies of result data sent from the target receiving end, combine multiple copies of results to obtain a first target result, and send the first target result to the message queue;
a division result unit, configured to divide the first target result in the message queue according to the preset number of shares to obtain multiple pieces of first result data, and distribute the multiple pieces of first result data in turn to a specified number of target receiving ends in a share-by-share manner;
a search result unit, configured to query the data processing flow chart according to the first result data through the target receiving end, to find an operation type of a next operation node corresponding to the target data, and start the next operation node, to perform an operation on the first result data according to the operation type of the next operation node, so as to obtain multiple pieces of result data after the operation of the next operation node;
and the operation result unit is used for receiving a plurality of copies of result data after the operation of the next operation node sent by the target receiving end, combining the plurality of copies of result data after the operation of the next operation node to obtain a second target result, sending the second result to the message queue, and so on until the operation of the target data is completed according to the data processing flow chart.
Further, the message queue-based data processing apparatus further includes:
the judging data unit is used for judging whether the conveying completion information corresponding to the initial data is received or not;
the first triggering unit is used for triggering and inquiring the instruction of the data processing flow chart when the delivery completion information is received;
the judgment operation unit is used for judging whether the operation types of the operation nodes corresponding to the target data are single operation or not according to the target data when the transmission completion information is not received, wherein the single operation is an independent operation for one piece of data;
and the judging and receiving unit is used for triggering the instruction for inquiring the data processing flow chart if the operation types of the operation nodes are the single operation.
The invention also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method when executing the computer program.
The invention also provides a computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned method.
The invention has the beneficial effects that: the data input is controlled only through an external program, the subsequent action is triggered by the slice data distributed by the message queue, each operation server inquires and controls the data flow direction and processes the data flow direction, and the next round of slice data transmission is carried out after the processing is finished, so that the operation performance and the processing efficiency are greatly improved by iteration.
Drawings
FIG. 1 is a diagram illustrating steps of a message queue-based data processing method according to an embodiment of the present invention;
FIG. 2 is a block diagram of a message queue-based data processing apparatus according to an embodiment of the present invention;
fig. 3 is a block diagram schematically illustrating a structure of a computer device according to an embodiment of the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the invention.
Referring to fig. 1, the data processing method based on a message queue in this embodiment includes:
step S1: receiving initial data input from a designated input node;
step S2: inquiring a preset data processing flow chart according to the initial data to find the operation type of the current operation node corresponding to the initial data, and adding the information of the operation type into a data header of the initial data to obtain target data;
and step S3: dividing the target data according to a preset number of parts to obtain a plurality of parts of first data, and putting the first data into a message queue;
and step S4: distributing multiple copies of the first data from the message queue to a specified number of target receiving ends in turn;
step S5: starting the current operation node in the target receiving end through each target receiving end according to the information of the operation type, and operating the first data according to the operation type of the current operation node to obtain multiple pieces of result data;
step S6: receiving multiple copies of result data sent from the target receiving end, combining the multiple copies of result data to obtain a first target result, and sending the first target result to the message queue;
step S7: dividing the first target result in the message queue according to the preset number of parts to obtain a plurality of pieces of first result data, and distributing the plurality of pieces of first result data to a specified number of target receiving ends in turn;
step S8: inquiring the data processing flow chart according to the first result data through the target receiving terminal so as to find the operation type of a next operation node corresponding to the target data, starting the next operation node, and operating the first result data according to the operation type of the next operation node to obtain a plurality of pieces of result data after operation of the next operation node;
step S9: and receiving a plurality of copies of result data after next operation node operation sent by the target receiving end, combining the plurality of copies of result data after next operation node operation to obtain a second target result, sending the second result to the message queue, and so on until the operation of the target data is completed according to the data processing flow chart.
In this embodiment, the data processing method is implemented by a data processing system, and may be applied to any scene of batch processing of data, and the data processing system may be implemented based on cooperation of a plurality of servers, such as a main server and a plurality of secondary servers. As described in step S1, the designated input node is a virtual node in the data processing system, and specifically, the data input program may input data to the input node using an API (API is a tool kit for inputting data), where the data input program is a program developed by a developer for inputting data, and when initial data is input from the input node, that is, the initial data is received from an external device by the host server through the data input program, the initial data is data that starts to enter a data processing flow and is not processed. The input nodes can be arranged in a plurality, and each input node corresponds to one type of data.
As shown in step S2, it is known that each piece of data includes a data header, where the data header is a header message of the data, and the header message includes information of each dimension of the data, for example, the data header includes information of a type and a source of the piece of data, and a data processing flowchart corresponding to the initial data can be found through the information of the type and the source of the initial data, and the data processing flowcharts corresponding to different initial data are different. The data processing flow chart is a flow chart for recording data processed by each operation node, that is, for each target data, the data flow chart records a process of processing the target data, and may include a plurality of operation nodes and an operation type of each operation node. Specifically, the user can draw corresponding data processing flow charts for various data, that is, each data has a one-to-one corresponding processing flow chart, and the processing flow charts are stored in a preset database and queried as needed. When the initial data is received, the corresponding data processing flow chart can be searched according to the initial data, so that the operation type of the current operation node corresponding to the initial data can be known, at this time, the information of the operation type can be added into the data header of the initial data, which is equivalent to that the data carries the identification information of the operation type of the current operation node, so as to be used for subsequent searching identification, and the initial data is marked as target data for the convenience of distinguishing.
The invention provides four types of different data which are respectively single-stroke operation, connection operation, dimensionality reduction operation and collection operation, and various different data can be processed by the single or the combination of the four operations. The single-stroke operation is the operation of a certain piece of data, and the single-stroke operation does not need to be connected with other data and combined for calculation; the connection operation is opposite, a plurality of data are required to be connected and then operated together, the parameter content required by the operation comes from the plurality of data, and the data required to be connected with each other can be matched through fields among the data; the dimensionality reduction operation is to convert the dimensionality with the upper-level and lower-level relationships in the data and then perform operation, and in the process of converting the dimensionality, multiple pieces of different data may be needed, where the dimensionality refers to information on one aspect of the data, and each piece of data has multiple different dimensionalities, for example, recording a piece of data of a client, which may include information on the aspects of age, academic history, gender, marital, and the like, that is, information on multiple dimensionalities. The collection operation refers to an operation of collecting a plurality of data groups together.
For example, the target data a and the corresponding data processing flow diagram include three operation nodes, where the initial operation node is a single operation, the next operation node is a connection operation, and the next operation node is a single operation, and then a sequentially passes through the single operation, the connection operation, and the single operation according to the data processing flow diagram.
As described in the foregoing steps S3 to S4, the target data is divided according to the preset fraction, that is, the target data is sliced according to the preset number of copies, so as to obtain the first data with the same amount of each copy, where the preset number of copies is preset by the developer. And then putting the first data into a message queue, wherein the data in the message queue carries a message of a corresponding input node type. For example, dividing 100 pieces of data into 10 pieces, each piece of data includes 10 pieces of data, sending each piece of first data to a message queue, then distributing each piece of first data to a plurality of target receiving ends in turn, and starting a current operation node, namely starting to enter an operation state, wherein the number of the plurality of target receiving ends is 4, the first piece of the 10 pieces of data is distributed to a first target receiving end, the second piece of data is distributed to a second target receiving end, the third piece of data is distributed to a third target receiving end, the fourth piece of data is distributed to a fourth target receiving end, the fifth piece of data is distributed to the first target receiving end, the sixth piece of data is distributed to the second target receiving end, and so on until the 10 pieces of data are distributed, the target receiving end is the secondary server and is used for performing operation on the target data, so that automatic distribution and parallel operation of the data are realized, and the efficiency is greatly improved.
As described in the foregoing steps S5 to S6, it can be known that the first data includes the data header, that is, the information of the operation type of the current operation node can be found from the data header, so in each target receiving end, the operation type can be obtained from the data header of the first data, then the current operation node is started according to the operation type, that is, the target data is operated according to the operation type to obtain multiple copies of result data, and the multiple copies of result data are combined and returned to the message queue.
As described in the foregoing steps S7 to S9, after the first target result is returned to the message queue, the first target result may be divided again according to the preset number of parts to obtain the preset number of parts of first result data, and then the first result data is distributed to the multiple target receiving ends in turn again, where a data header of the first result data includes a data type, source information from a previous node, and the like, and then, in the target receiving end, a corresponding data processing flow diagram is queried according to the data header of the first result data, a corresponding next operation node and an operation type corresponding to the next operation node are found according to the source information, then, the next operation node is started, the first result data is operated according to the operation type of the next operation node to obtain multiple results, and after combination, a second target result is obtained, the message queue is returned again, the foregoing steps are repeated, and iterative computation is performed until the data processing flow diagram is completed, that the operation of the target data is completed according to the data processing flow diagram, and a final result is obtained.
Further, after the final result corresponding to the target data is obtained, the final result can be sliced according to the specified record quantity to obtain a plurality of pieces of result data with the same record quantity, and then the result data are sent to the message queue.
In one embodiment, before the step S2, the method further includes:
step S11: judging whether the transmission completion information corresponding to the initial data is received or not;
step S12: if the delivery completion information is received, triggering and inquiring the instruction of the data processing flow chart;
step S13: if the information of finished transmission is not received, judging whether the operation types of all operation nodes corresponding to the target data are single operation according to the target data, wherein the single operation is single operation for one piece of data;
step S14: if yes, the operation type of each operation node is the single operation, and the instruction for inquiring the data processing flow chart is triggered.
In this embodiment, when each external program interfacing with the input node completes data transmission, a "data transmission completion" message may be sent to the data processing system through the API to declare that all data of the input node has been completely transmitted, and then the driving system completes related operations, so that before querying the data processing flowchart, it may be determined whether the transmission completion message corresponding to the initial data is received, and if the data transmission completion message sent by the external program is received, the data processing system triggers the instruction for querying the data processing flowchart, queries the data processing flowchart according to the instruction, and then executes the steps S2 to S9.
When the delivery completion information is not received, the type of each operation node corresponding to the target data can be checked firstly, and since the single operation is the operation of a certain data per se, the single operation can be directly entered into the process for operation without being connected with other data and combined for calculation, that is, other data does not need to be waited, whether the operation types of each operation node corresponding to the target data are the single operation or not is judged firstly, specifically, the corresponding data processing flow chart can be found according to the target data, so that the type of each operation node corresponding to the target data is obtained, when the operation types of each operation node in the flow chart are the single operation, in order to improve the real-time performance, the instruction of the data processing flow chart is triggered to be inquired, the inquiry is carried out according to the instruction data processing flow chart, and the steps S2-S7 are executed. Or after a preset time, for example, 60 seconds, the judgment is performed, that is, whether the delivery completion information sent by the external program is received within the 60 seconds is judged again, if the information is not received, the instruction for inquiring the data processing flow chart is also triggered, the inquiry is performed according to the instruction data processing flow chart, and the steps S2 to S7 are performed, so that the data can be 'calculated at any time', and the real-time performance of data processing is greatly improved.
In one embodiment, the step S2 includes:
step S21: acquiring a data type corresponding to the initial data according to the initial data;
step S22: calling a preset data processing flow chart corresponding to the data type, wherein different data types correspond to different data processing flow charts;
step S23: and finding out the operation type of the current operation node corresponding to the initial data from the data processing flow chart.
In this embodiment, different types of initial data correspond to different types of initial data, and corresponding data processing flowcharts are different for different types of initial data, for example, the processing flow of the first type of initial data is sequentially a single operation, a dimension reduction operation, and a collection operation, and the processing flow of the second type of initial data is sequentially a single operation, a join operation, and a dimension reduction operation. The data head of the initial data comprises information of the data type, so that the data type can be obtained, then the corresponding data processing flow chart is extracted from the preset database according to the data type, then the current operation node is found in the data flow chart according to the node information in the data head, the operation type of the current operation node is obtained, the operation node is configured according to the operation type, and the target data is operated.
In one embodiment, the dividing of the known first target result into a preset number of pieces of first result data, each piece of the first result data including a plurality of pieces of data to be operated, and the number of the pieces of data of each piece of the first result data being consistent, when the current operation node is a connection operation, before the step S5, includes:
step S51: acquiring corresponding associated information from each piece of data to be operated, wherein the associated information comprises first information and second information matched with the first information, the first information is positioned in a start field of the data, and the second information is positioned in an end field of the data;
step S52: traversing all the data to be operated according to a mode of butting a starting field carrying the first information in the data to be operated with an ending field carrying the second information matched with the first information in another data to be operated to obtain connected data for performing connection operation.
In this embodiment, for each piece of data to be subjected to connection operation, each piece of data carries corresponding associated information, and specifically, the associated information includes two pieces of information, namely first information and second information matched with the first information, where the first information is located in a start field of the data, and the second information is located in an end field of the data. Before connection operation is carried out, first information and second information corresponding to each data to be operated are obtained, and it is worth noting that the first information and the second information in the same data to be operated are not matched, after relevant information corresponding to each data to be operated is obtained, all the data to be operated are traversed in a mode that a starting field carrying the first information in a certain data to be operated is butted with an ending field carrying the second information matched with the first information in another data to be operated, namely the starting field of the first data to be operated is butted with the ending field of the second data to be operated, wherein the starting field of the first data to be operated carries the first information, and traversing all the data to be operated in such a way to obtain data for connection operation, marking the data as connected data, and operating the connected data, wherein for example, three pieces of data are ABC respectively, the start field of A has first information a and the end field has second information B, the start field of B has first information B and the end field has second information C, the start field of C has first information C and the end field has second information d, the start field of B is sequentially butted to the end field of A and the start field of C is butted to the end field of B according to corresponding matching relations, so as to obtain the connected data.
In an embodiment, the step of querying, by the target receiving end, the data processing flowchart according to the first result data to find the operation type of the next operation node corresponding to the target data includes:
step S81: finding a data header of the first result data from the first result data through the target receiving end, wherein the data header comprises node information of the current operation node, and the node information comprises relationship information of the current operation node and operation nodes adjacent up and down;
step S82: and finding the operation type of the next node corresponding to the current operation node from the data processing flow chart according to the node information.
In this embodiment, the specific process of finding the operation type of the next operation node is as follows, it is to be understood that each piece of data has a data header, the data header includes a type of the piece of data and node information of the operation node, the node information includes relationship information between the current operation node and vertically adjacent operation nodes, such as a source and a type of the previous operation node and a type of the next node, and according to the source and the type of the previous node, a corresponding structural block diagram and structural block diagrams of upper and lower steps associated with the structural block diagram can be found from the data flow diagram, so that a structural block diagram corresponding to the operation node of the next step is obtained, and the corresponding operation type is also obtained.
In one embodiment, after the step S6, the method further includes:
step S61: judging whether the operation type of the current operation node is any one of connection operation, dimension reduction operation and collection operation;
step S62: if the operation type is any one of connection operation, dimension reduction operation and collection operation, judging whether a control signal carrying information of required data transmission completion is received, wherein the required data is any one of all data required when the operation is carried out according to the corresponding operation type in the operation node and is result data after the operation of the operation node;
step S63: if a control signal carrying information that the transmission of the required data is finished is received, a data dividing instruction is triggered;
step S64: and if the control signal carrying the information that the transmission of the required data is finished is not received, storing the received required data into a memory database until the control signal is received, and triggering a data dividing instruction after the control signal is received.
In this embodiment, it is known that there are four types of operation types provided, which are a single operation, a connection operation, a dimension reduction operation, and a collection operation, and for the single operation, it is only necessary to perform an operation on target data itself, but for the connection operation, the dimension reduction operation, and the collection operation, it may be necessary to receive multiple pieces of target data before performing the operation, so before the operation, it is first determined which type of current operation is, if any one of the connection operation, the dimension reduction operation, and the collection operation is, it is necessary to know whether all data required for the operation is received, after the operation of the last operation node is completed, a result is returned to a message queue, and a control signal is sent out, where the control signal carries information that the required data is transmitted, where the required data is all data required when the operation node operates according to the corresponding operation type, and is also result data after the operation of the operation node, when the control signal is received, it is stated that the required data is transmitted, the operation can be started, but if the control signal is not received, it is stated that the required data is not transmitted, at this time, the received required data can be stored in a memory database, and then the required data is stored in a control signal is not received, and a control signal S-7 is executed each time, and a control signal is executed, and a control signal S is not received. The traditional relational database is replaced by the memory database, the IO performance of the memory database is far higher than that of the relational database, and the operation performance of the system is further improved.
In summary, the data processing method only needs to control data input through an external program, subsequent actions are triggered by slice data distributed by a message queue, each operation server automatically inquires flow configuration information to control data flow direction and process, and then sends the next round of slice data after processing, and the process is iterated, all data in the whole operation process are imported from input nodes, and no data is allowed to be inquired from the outside in the operation process of each link, so that the method is more intelligent, can ensure that a result is completely determined by input data, is not influenced by the external environment, and is convenient for searching abnormal reasons when problems occur. On the other hand, the operation performance is greatly improved, the processing process of about two million data in a field is finished for about 20 minutes according to the currently realized Demo rough estimation, while about one and a half hours is needed in the traditional mode, the overall aging is improved by more than 4 times compared with the traditional mode, and the data is processed in real time along with the calculation, so that the overall processing efficiency is greatly improved.
Referring to fig. 2, the present invention provides a message queue-based data processing apparatus corresponding to the message queue-based data processing method, the message queue-based data processing apparatus including:
a receive data unit 100 for receiving initial data input from a designated input node;
a type searching unit 200, configured to search a preset data processing flow chart according to the initial data to find an operation type of a current operation node corresponding to the initial data, and add information of the operation type to a data header of the initial data to obtain target data;
a data dividing unit 300, configured to divide the target data by a preset number of portions to obtain multiple first data, and place the multiple first data into a message queue;
a distribution data unit 400, configured to distribute multiple copies of the first data from the message queue to a specified number of target receiving ends in turn;
an operation data unit 500, configured to start the current operation node in each target receiving end according to the information of the operation type, and perform an operation on the first data according to the operation type of the current operation node to obtain multiple pieces of result data;
a result receiving unit 600, configured to receive multiple copies of result data sent from the target receiving end, combine the multiple copies of result data to obtain a first target result, and send the first target result to the message queue;
a dividing result unit 700, configured to divide the first target result in the message queue according to the preset number of shares to obtain multiple pieces of first result data, and distribute the multiple pieces of first result data in turn to a specified number of target receiving ends in a share-by-share manner;
a search result unit 800, configured to query, by the target receiving end, the data processing flowchart according to the first result data, to find an operation type of a next operation node corresponding to the target data, and start the next operation node, so as to perform an operation on the first result data according to the operation type of the next operation node, so as to obtain multiple pieces of result data after the operation of the next operation node;
and an operation result unit 900, configured to receive multiple copies of result data after operation of the next operation node sent from the target receiving end, combine the multiple copies of result data after operation of the next operation node to obtain a second target result, send the second result to the message queue, and so on until the operation of the target data is completed according to the data processing flow chart.
In one embodiment, the message queue-based data processing apparatus further includes:
a data judging unit for judging whether the information corresponding to the initial data is received;
the first triggering unit is used for triggering and inquiring the instruction of the data processing flow chart when the delivery completion information is received;
the judgment operation unit is used for judging whether the operation types of the operation nodes corresponding to the target data are single operation or not according to the target data when the transmission completion information is not received, wherein the single operation is an independent operation for one piece of data;
and the judging and receiving unit is used for triggering the instruction for inquiring the data processing flow chart if the operation types of the operation nodes are the single operation.
In one embodiment, the lookup type unit 200 includes:
an obtaining type subunit, configured to obtain, according to the initial data, a data type corresponding to the initial data;
the calling flow subunit is used for calling a preset data processing flow chart corresponding to the data type, wherein different data types correspond to different data processing flow charts;
and the type finding subunit is used for finding out the operation type of the current operation node corresponding to the initial data from the data processing flow chart.
In one embodiment, given that the first target result is divided into a preset number of pieces of first result data, each piece of first result data includes multiple pieces of data to be operated on, and the number of data in each piece of first result data is consistent, when the current operation node is a connection operation, the message queue-based data processing apparatus further includes:
the acquiring information subunit is configured to acquire corresponding associated information from each piece of data to be operated, where the associated information includes first information and second information matched with the first information, the first information is located in a start field of the data, and the second information is located in an end field of the data;
and the connection data subunit is configured to traverse all the data to be operated in a manner of butting a start field, which carries the first information, in the data to be operated with an end field, which carries the second information matched with the first information, in another data to be operated, so as to obtain connected data for performing connection operation.
In an embodiment, the search result unit 800 includes:
a searching header subunit, configured to find a data header of the first result data from the first result data through the target receiving end, where the data header includes node information of the current operation node, and the node information includes relationship information between the current operation node and an operation node adjacent to the current operation node;
and the type finding subunit is used for finding the operation type of the next node corresponding to the current operation node from the data processing flow chart according to the node information.
In an embodiment, the message queue-based data processing apparatus further includes:
the judging type unit is used for judging whether the operation type of the current operation node is any one of connection operation, dimension reduction operation and collection operation;
a signal receiving unit, configured to determine whether a control signal carrying information indicating that transmission of required data is completed is received if the operation type is any one of a connection operation, a dimensionality reduction operation, and a set operation, where the required data is any one of all data required when an operation is performed in the operation node according to the corresponding operation type and is result data after operation of the operation node;
the trigger segmentation unit is used for triggering a data segmentation instruction when receiving a control signal carrying information that the required data transmission is finished;
and the data storage unit is used for storing the received required data into the memory database until the control signal is received when the control signal carrying the information that the transmission of the required data is finished is not received, and triggering a data dividing instruction after the control signal is received.
Referring to fig. 3, an embodiment of the present invention further provides a computer device, where the computer device may be a server, and an internal structure of the computer device may be as shown in fig. 3. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the computer designed processor is used to provide computational and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The memory provides an environment for the operating system and the running of computer programs in the non-volatile storage medium. The database of the computer device is used for storing all data required for the above-mentioned data processing. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a message queue based data processing method.
The processor executes the steps of the message queue-based data processing method: receiving initial data input from a designated input node; inquiring a preset data processing flow chart according to the initial data to find the operation type of the current operation node corresponding to the initial data, and adding the information of the operation type into a data header of the initial data to obtain target data; dividing the target data according to a preset number of parts to obtain a plurality of parts of first data, and putting the first data into a message queue; distributing a plurality of copies of the first data from the message queue to a specified number of target receiving ends in turn; starting the current operation node in the target receiving end through each target receiving end according to the information of the operation type, and operating the first data according to the operation type of the current operation node to obtain multiple pieces of result data; receiving multiple copies of result data sent from the target receiving end, combining the multiple copies of result data to obtain a first target result, and sending the first target result to the message queue; dividing the first target result in the message queue according to the preset number of parts to obtain a plurality of pieces of first result data, and distributing the plurality of pieces of first result data to a specified number of target receiving ends in turn; inquiring the data processing flow chart according to the first result data through the target receiving terminal to find the operation type of a next operation node corresponding to the target data, starting the next operation node, and operating the first result data according to the operation type of the next operation node to obtain a plurality of pieces of result data after the operation of the next operation node; and receiving a plurality of copies of result data after next operation node operation sent by the target receiving end, combining the plurality of copies of result data after next operation node operation to obtain a second target result, sending the second result to the message queue, and so on until the operation of the target data is completed according to the data processing flow chart.
After the step of querying a preset data processing flow chart according to the initial data to find the operation type of the current operation node corresponding to the initial data, the computer device includes: judging whether the transmission completion information corresponding to the initial data is received or not; if the delivery completion information is received, triggering and inquiring the instruction of the data processing flow chart; if the information of finished transmission is not received, judging whether the operation types of all operation nodes corresponding to the target data are single operation according to the target data, wherein the single operation is single operation for one piece of data; and if the operation type of each operation node is the single operation, triggering the instruction for inquiring the data processing flow chart.
In an embodiment, the step of querying a preset data processing flow chart according to the initial data to find the operation type of the current operation node corresponding to the target data includes: acquiring a data type corresponding to the initial data according to the initial data; calling a preset data processing flow chart corresponding to the data type, wherein different data types correspond to different data processing flow charts; and finding out the operation type of the current operation node corresponding to the initial data from the data processing flow chart.
In an embodiment, before the step of starting the current operation node in the target receiving end according to the information of the operation type in the data header by each target receiving end and operating the first data according to the operation type of the current operation node to obtain a plurality of pieces of result data, the method includes: acquiring corresponding associated information from each piece of data to be operated, wherein the associated information comprises first information and second information matched with the first information, the first information is located in a start field of the data, and the second information is located in an end field of the data; traversing all the data to be operated according to a mode of butting a starting field carrying the first information in the data to be operated with an ending field carrying the second information matched with the first information in another data to be operated to obtain connected data for performing connection operation.
In an embodiment, the step of querying, by the target receiving end, the data processing flowchart according to the first result data to find the operation type of the next operation node corresponding to the target data includes: finding a data header of the first result data from the first result data through the target receiving end, wherein the data header comprises node information of the current operation node, and the node information comprises relationship information of the current operation node and operation nodes adjacent up and down; and finding the operation type of the next node corresponding to the current operation node from the data processing flow chart according to the node information.
In an embodiment, after the step of receiving multiple copies of result data sent from the target receiving end, combining the multiple copies of result data to obtain a first target result, and sending the first target result to the message queue, the method includes: judging whether the operation type of the current operation node is any one of connection operation, dimension reduction operation and collection operation; if the operation type is any one of connection operation, dimension reduction operation and collection operation, judging whether a control signal carrying information of required data transmission completion is received, wherein the required data is any one of all data required when the operation is carried out according to the corresponding operation type in the operation node and is result data after the operation of the operation node; if a control signal carrying information that the transmission of the required data is finished is received, a data dividing instruction is triggered; and if the control signal carrying the information that the transmission of the required data is finished is not received, storing the received required data into a memory database until the control signal is received, and triggering a data dividing instruction after the control signal is received.
Those skilled in the art will appreciate that the architecture shown in fig. 3 is only a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects may be applied.
An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements a data processing method based on a message queue, and the implementing method specifically includes: receiving initial data input from a designated input node; inquiring a preset data processing flow chart according to the initial data to find the operation type of the current operation node corresponding to the initial data, and adding the information of the operation type into a data header of the initial data to obtain target data; dividing the target data according to a preset number of parts to obtain a plurality of parts of first data, and putting the first data into a message queue; distributing a plurality of copies of the first data from the message queue to a specified number of target receiving ends in turn; starting the current operation node in the target receiving end through each target receiving end according to the information of the operation type, and operating the first data according to the operation type of the current operation node to obtain multiple pieces of result data; receiving multiple copies of result data sent from the target receiving end, combining the multiple copies of result data to obtain a first target result, and sending the first target result to the message queue; dividing the first target result in the message queue according to the preset number of parts to obtain a plurality of pieces of first result data, and distributing the plurality of pieces of first result data to a specified number of target receiving ends in turn; inquiring the data processing flow chart according to the first result data through the target receiving terminal to find the operation type of a next operation node corresponding to the target data, starting the next operation node, and operating the first result data according to the operation type of the next operation node to obtain a plurality of pieces of result data after the operation of the next operation node; and receiving a plurality of copies of result data after next operation node operation sent by the target receiving end, combining the plurality of copies of result data after next operation node operation to obtain a second target result, sending the second result to the message queue, and so on until the operation of the target data is completed according to the data processing flow chart.
The computer-readable storage medium, after the step of querying a preset data processing flow chart according to the initial data to find the operation type of the current operation node corresponding to the initial data, includes: judging whether the transmission completion information corresponding to the initial data is received or not; if the delivery completion information is received, triggering and inquiring the instruction of the data processing flow chart; if the information of finished transmission is not received, judging whether the operation types of all operation nodes corresponding to the target data are single operation according to the target data, wherein the single operation is single operation for one piece of data; and if the operation type of each operation node is single operation, triggering the instruction for inquiring the data processing flow chart.
In an embodiment, the step of searching the preset data processing flow chart according to the initial data query to find the operation type of the current operation node corresponding to the target data includes: acquiring a data type corresponding to the initial data according to the initial data; calling a preset data processing flow chart corresponding to the data type, wherein different data types correspond to different data processing flow charts; and finding out the operation type of the current operation node corresponding to the initial data from the data processing flow chart.
In an embodiment, before the step of starting the current operation node in the target receiving end according to the information of the operation type in the data header by each target receiving end and operating the first data according to the operation type of the current operation node to obtain a plurality of pieces of result data, the method includes: acquiring corresponding associated information from each piece of data to be operated, wherein the associated information comprises first information and second information matched with the first information, the first information is located in a start field of the data, and the second information is located in an end field of the data; traversing all the data to be operated according to a mode of butting a starting field carrying the first information in the data to be operated with an ending field carrying the second information matched with the first information in another data to be operated to obtain connected data for performing connection operation.
In an embodiment, the step of querying, by the target receiving end, the data processing flowchart according to the first result data to find the operation type of the next operation node corresponding to the target data includes: finding a data head of the first result data from the first result data through the target receiving end, wherein the data head comprises node information of the current operation node, and the node information comprises relationship information of the current operation node and upper and lower adjacent operation nodes; and finding the operation type of the next node corresponding to the current operation node from the data processing flow chart according to the node information.
In an embodiment, after the step of receiving multiple copies of result data sent from the target receiving end, combining the multiple copies of result data to obtain a first target result, and sending the first target result to the message queue, the method includes: judging whether the operation type of the current operation node is any one of connection operation, dimension reduction operation and collection operation; if the operation type is any one of connection operation, dimension reduction operation and collection operation, judging whether a control signal carrying information of required data transmission completion is received, wherein the required data is any one of all data required when the operation is carried out according to the corresponding operation type in the operation node and is result data after the operation of the operation node; if a control signal carrying information that the transmission of the required data is finished is received, triggering a data dividing instruction; and if the control signal carrying the information that the transmission of the required data is finished is not received, storing the received required data into a memory database until the control signal is received, and triggering a data dividing instruction after the control signal is received.
It will be understood by those skilled in the art that all or part of the processes of the methods of the above embodiments may be implemented by hardware associated with instructions of a computer program, which may be stored on a non-volatile computer-readable storage medium, and when executed, may include processes of the above embodiments of the methods. Any reference to memory, storage, database, or other medium provided herein and used in the examples may include non-volatile and/or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link (Synchlink) DRAM (SLDRAM), rambus (Rambus) direct RAM (RDRAM), direct bused dynamic RAM (DRDRAM), and bused dynamic RAM (RDRAM).
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, apparatus, article, or method. Without further limitation, an element defined by the phrases "comprising a," "8230," "8230," or "comprising" does not exclude the presence of another identical element in a process, apparatus, article, or method comprising the element.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (10)
1. A data processing method based on a message queue is characterized by comprising the following steps:
receiving initial data input from a designated input node;
judging whether the transmission completion information corresponding to the initial data is received or not;
if the delivery completion information is not received, and the operation type of each operation node in the flow chart is single operation, after the preset time, whether the delivery completion information sent by an external program is received within the preset time is judged again;
if the information is not received, triggering an instruction for inquiring the data processing flow chart, and inquiring according to the instruction data processing flow chart;
inquiring a preset data processing flow chart according to the initial data to find the operation type of the current operation node corresponding to the initial data, and adding the information of the operation type into a data header of the initial data to obtain target data;
dividing the target data according to a preset number of parts to obtain multiple parts of first data, and putting the multiple parts of first data into a message queue;
distributing a plurality of copies of the first data from the message queue to a specified number of target receiving ends in turn;
starting the current operation node in the target receiving end through each target receiving end according to the information of the operation type, and operating the first data according to the operation type of the current operation node to obtain a plurality of pieces of result data;
receiving multiple copies of result data sent from the target receiving end, combining the multiple copies of result data to obtain a first target result, and sending the first target result to the message queue;
dividing the first target result in the message queue according to the preset number of parts to obtain a plurality of pieces of first result data, and distributing the plurality of pieces of first result data to a specified number of target receiving ends in turn;
inquiring the data processing flow chart according to the first result data through the target receiving terminal to find the operation type of a next operation node corresponding to the target data, starting the next operation node, and operating the first result data according to the operation type of the next operation node to obtain a plurality of pieces of result data after the operation of the next operation node;
and receiving a plurality of copies of result data after next operation node operation sent by the target receiving end, combining the plurality of copies of result data after next operation node operation to obtain a second target result, sending the second result to the message queue, and so on until the operation of the target data is completed according to the data processing flow chart.
2. The message queue-based data processing method according to claim 1, wherein the step of determining whether the delivery completion information corresponding to the initial data is received comprises:
if the delivery completion information is received, triggering and inquiring the instruction of the data processing flow chart;
if the information of finished transmission is not received, judging whether the operation types of all operation nodes corresponding to the target data are single operation according to the target data, wherein the single operation is single operation for one piece of data;
and if the operation type of each operation node is the single operation, triggering the instruction for inquiring the data processing flow chart.
3. The message queue-based data processing method according to claim 1, wherein the step of querying a preset data processing flow chart according to the initial data to find the operation type of the current operation node corresponding to the target data comprises:
acquiring a data type corresponding to the initial data according to the initial data;
calling a preset data processing flow chart corresponding to the data type, wherein different data types correspond to different data processing flow charts;
and finding out the operation type of the current operation node corresponding to the initial data from the data processing flow chart.
4. The message queue-based data processing method according to claim 1, wherein an operation type of the current operation node is a connection operation, each piece of the first result data includes a plurality of pieces of data to be operated, and before the step of starting the current operation node in the target receiving end according to information of the operation type by each target receiving end and operating the first data according to the operation type of the current operation node to obtain a plurality of pieces of result data, the method comprises:
acquiring corresponding associated information from each piece of data to be operated, wherein the associated information comprises first information and second information matched with the first information, the first information is positioned in a start field of the data, and the second information is positioned in an end field of the data;
traversing all the data to be operated according to a mode of butting a starting field carrying the first information in the data to be operated with an ending field carrying the second information matched with the first information in another data to be operated to obtain connected data for performing connection operation.
5. The message queue-based data processing method as claimed in claim 1, wherein the step of querying the data processing flow chart by the target receiving end according to the first result data to find the operation type of the next operation node corresponding to the target data comprises:
finding a data head of the first result data from the first result data through the target receiving end, wherein the data head comprises node information of the current operation node, and the node information comprises relationship information of the current operation node and upper and lower adjacent operation nodes;
and finding the operation type of the next node corresponding to the current operation node from the data processing flow chart according to the node information.
6. The message queue-based data processing method of claim 1, wherein the step of receiving multiple copies of result data sent from the target receiving end, combining the multiple copies of result data to obtain a first target result, and sending the first target result to the message queue is followed by the step of:
judging whether the operation type of the current operation node is any one of connection operation, dimension reduction operation and collection operation;
if the operation type is any one of connection operation, dimension reduction operation and collection operation, judging whether a control signal carrying information of required data transmission completion is received, wherein the required data is any one of all data required when the operation is carried out according to the corresponding operation type in the operation node and is result data after the operation of the operation node;
if a control signal carrying information that the transmission of the required data is finished is received, a data dividing instruction is triggered;
and if the control signal carrying the information that the transmission of the required data is finished is not received, storing the received required data into a memory database until the control signal is received, and triggering a data dividing instruction after the control signal is received.
7. A message queue-based data processing apparatus, comprising:
a receiving data unit for receiving initial data input from a designated input node;
the judging data unit is used for judging whether the conveying completion information corresponding to the initial data is received or not;
the judging and receiving unit is used for judging whether the conveying completion information sent by an external program is received within the preset time again after the preset time if the conveying completion information is not received and the operation types of all the operation nodes in the flow chart are single operation; if the information is not received, triggering an instruction for inquiring the data processing flow chart, and inquiring according to the instruction data processing flow chart;
the searching type unit is used for searching a preset data processing flow chart according to the initial data to find the operation type of the current operation node corresponding to the initial data, and adding the information of the operation type into a data header of the initial data to obtain target data;
the data dividing unit is used for dividing the target data according to a preset number of parts to obtain a plurality of first data, and putting the first data into a message queue;
the distribution data unit is used for distributing a plurality of copies of the first data from the message queue to a specified number of target receiving ends in turn;
the operation data unit is used for starting the current operation node in the target receiving end through each target receiving end according to the information of the operation type, and operating the first data according to the operation type of the current operation node to obtain a plurality of pieces of result data;
a receiving result unit, configured to receive multiple copies of result data sent from the target receiving end, combine the multiple copies of result data to obtain a first target result, and send the first target result to the message queue;
a division result unit, configured to divide the first target result in the message queue according to the preset number of shares to obtain multiple pieces of first result data, and distribute the multiple pieces of first result data in turn to a specified number of target receiving ends in a share-by-share manner;
a search result unit, configured to query, by the target receiving end, the data processing flowchart according to the first result data to find an operation type of a next operation node corresponding to the target data, and start the next operation node, so as to perform an operation on the first result data according to the operation type of the next operation node, to obtain multiple sets of result data after operation on the next operation node;
and the operation result unit is used for receiving a plurality of copies of result data after the operation of the next operation node sent by the target receiving end, combining the plurality of copies of result data after the operation of the next operation node to obtain a second target result, sending the second result to the message queue, and so on until the operation of the target data is completed according to the data processing flow chart.
8. The message queue-based data processing apparatus of claim 7, further comprising:
the first triggering unit is used for triggering and inquiring the instruction of the data processing flow chart when the delivery completion information is received;
the judging and operating unit is used for judging whether the operation types of the operation nodes corresponding to the target data are single operation according to the target data when the information of finished transmission is not received, wherein the single operation is independent operation for one piece of data;
and the judging and receiving unit is used for triggering the instruction for inquiring the data processing flow chart when the operation type of each operation node is the single operation.
9. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor implements the steps of the method of any one of claims 1 to 6 when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910849176.7A CN110737535B (en) | 2019-09-09 | 2019-09-09 | Data processing method and device based on message queue and computer equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910849176.7A CN110737535B (en) | 2019-09-09 | 2019-09-09 | Data processing method and device based on message queue and computer equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110737535A CN110737535A (en) | 2020-01-31 |
| CN110737535B true CN110737535B (en) | 2023-02-07 |
Family
ID=69268167
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910849176.7A Active CN110737535B (en) | 2019-09-09 | 2019-09-09 | Data processing method and device based on message queue and computer equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110737535B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104778074A (en) * | 2014-01-14 | 2015-07-15 | 腾讯科技（深圳）有限公司 | Calculation task processing method and device |
| CN108089915A (en) * | 2016-11-22 | 2018-05-29 | 北京京东尚科信息技术有限公司 | The method and system of business controlization processing based on message queue |
| CN109344682A (en) * | 2018-08-02 | 2019-02-15 | 平安科技（深圳）有限公司 | Classroom monitoring method, device, computer equipment and storage medium |
| CN109857565A (en) * | 2019-01-18 | 2019-06-07 | 深圳壹账通智能科技有限公司 | Data processing method, device, computer equipment and storage medium |
| CN110113387A (en) * | 2019-04-17 | 2019-08-09 | 深圳前海微众银行股份有限公司 | A kind of processing method based on distributed batch processing system, apparatus and system |
-
2019
- 2019-09-09 CN CN201910849176.7A patent/CN110737535B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104778074A (en) * | 2014-01-14 | 2015-07-15 | 腾讯科技（深圳）有限公司 | Calculation task processing method and device |
| CN108089915A (en) * | 2016-11-22 | 2018-05-29 | 北京京东尚科信息技术有限公司 | The method and system of business controlization processing based on message queue |
| CN109344682A (en) * | 2018-08-02 | 2019-02-15 | 平安科技（深圳）有限公司 | Classroom monitoring method, device, computer equipment and storage medium |
| CN109857565A (en) * | 2019-01-18 | 2019-06-07 | 深圳壹账通智能科技有限公司 | Data processing method, device, computer equipment and storage medium |
| CN110113387A (en) * | 2019-04-17 | 2019-08-09 | 深圳前海微众银行股份有限公司 | A kind of processing method based on distributed batch processing system, apparatus and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110737535A (en) | 2020-01-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20200057958A1 (en) | Identification and application of hyperparameters for machine learning | |
| CN110807085B (en) | Fault information query method and device, storage medium and electronic device | |
| CN105243169B (en) | A kind of data query method and system | |
| CN108182258B (en) | Distributed data analysis system and method | |
| CN112035599A (en) | Query method and device based on vertical search, computer equipment and storage medium | |
| CN112307374A (en) | Jumping method, device and equipment based on backlog and storage medium | |
| CN110598109A (en) | Information recommendation method, device, equipment and storage medium | |
| CN108154024B (en) | Data retrieval method and device and electronic equipment | |
| CN110795171A (en) | Service data processing method and device, computer equipment and storage medium | |
| CN113067853A (en) | Data pushing method and device, electronic equipment and storage medium | |
| CN111723097A (en) | Application program interface configuration method and device, computer equipment and storage medium | |
| CN110555165B (en) | Information identification method and device, computer equipment and storage medium | |
| CN110737535B (en) | Data processing method and device based on message queue and computer equipment | |
| CN113255742A (en) | Policy matching degree calculation method and system, computer equipment and storage medium | |
| CN104750806A (en) | Large data query method and system | |
| CN110648052B (en) | Wind control decision method and device, computer equipment and storage medium | |
| CN112785180A (en) | Business operation management system and method with autonomous prompt function | |
| CN108241758B (en) | Data query method and related equipment | |
| CN115147020B (en) | Decoration data processing method, device, equipment and storage medium | |
| CN107784588A (en) | Insurance user information merging method and device | |
| CN111339170A (en) | Data processing method and device, computer equipment and storage medium | |
| CN116418705A (en) | Network asset identification method, system, terminal and medium based on machine learning | |
| CN110895538A (en) | Data retrieval method, device, storage medium and processor | |
| CN111460268B (en) | Method and device for determining database query request and computer equipment | |
| CN113761043A (en) | Data extraction method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |