CN117708214A - Data asynchronous submission production method and device, electronic equipment and recording medium - Google Patents
Data asynchronous submission production method and device, electronic equipment and recording medium Download PDFInfo
- Publication number
- CN117708214A CN117708214A CN202311786331.8A CN202311786331A CN117708214A CN 117708214 A CN117708214 A CN 117708214A CN 202311786331 A CN202311786331 A CN 202311786331A CN 117708214 A CN117708214 A CN 117708214A
- Authority
- CN
- China
- Prior art keywords
- data
- production
- state
- node
- flow table
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000004519 manufacturing process Methods 0.000 title claims abstract description 381
- 238000012545 processing Methods 0.000 claims abstract description 56
- 238000000034 method Methods 0.000 claims abstract description 44
- 238000012795 verification Methods 0.000 claims abstract description 23
- 238000004590 computer program Methods 0.000 claims description 3
- 238000011144 upstream manufacturing Methods 0.000 abstract description 17
- 238000012360 testing method Methods 0.000 description 46
- 238000013101 initial test Methods 0.000 description 13
- 238000012015 optical character recognition Methods 0.000 description 6
- 230000001360 synchronised effect Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 238000010606 normalization Methods 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000004140 cleaning Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 238000011835 investigation Methods 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000003860 storage Methods 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Abstract
The invention discloses a data asynchronous submission production method, a device, electronic equipment and a recording medium, wherein the data asynchronous submission production method comprises the following steps: basic verification is carried out when initial data is submitted to production, and a data flow table and a data state table are established after verification is passed; the submitted initial data is written into a data flow table, the data flow table is initialized, and a data state table records state nodes in the data production process; acquiring a current state node of data production and a process for controlling the data production according to the data state table; the data flow table records the production data and the production record of each node in the data production process. The invention realizes decoupling of upstream submitted data and intermediate data production service, the intermediate data production service independently carries out data production processing, and the data production process has compact service logic by combining the data flow table and the data state table, so that the data production is efficient and stable.
Description
Technical Field
The invention relates to the technical field of data processing, and particularly provides a data asynchronous submission production method and device, electronic equipment and a recording medium.
Background
Along with the development of internet big data technology, in order to promote the accuracy and the comprehensiveness of retrieval results, the data magnitude of database is constantly increasing. Thus, the database needs to write new data periodically/aperiodically, update the data magnitude of the extended database, and the process flow generally includes: the upstream business submits the original data to a data production system, the data production system carries out production processing on the original data into data conforming to a database, the produced data is written into the database, and then downstream use is provided. Taking a test question database as an example, in order to write original test question data into the test question database, the original test question data needs to be submitted to a test question production system, the test question production system performs basic verification on the original test question, and after the basic verification is passed, production processing operations such as OCR (optical character recognition) processing, normalization processing and the like are performed on the original test question data, so that the generated test question data can be written into the test question database.
Because the prior upstream service submits the original data and then needs to carry out the production processing of the data to write the data into the database, the synchronous processing takes a long time, and the upstream service and the data production system cannot bear the data. Therefore, a common process flow for writing raw data into a database is that an upstream service submits raw data, an intermediate data production service processes the data separately, and the produced data is stored into the database and then provided for downstream users to use. The intermediate data production service processes data in an asynchronous script or manual cleaning mode, the asynchronous script processes submitted original data piece by piece, the next data can be processed after the previous data is completely produced, and once a certain data production fails, blocking can be caused, so that the asynchronous script realizes asynchronous processing of the original data submission-production, but lacks compact service logic, and the intermediate data production service stability is poor and the production efficiency is low. The manual cleaning mode depends on professional operation, and the processing efficiency is poor, and the production requirement of a large amount of original data can not be met.
In summary, the present application aims to solve the problem of asynchronous processing of data submission-production, and provides a patent of the present invention.
Disclosure of Invention
In order to solve the technical problems, the invention provides a data asynchronous submission production method, a device, electronic equipment and a recording medium, and specifically adopts the following technical scheme:
in a first aspect, the present invention provides a data asynchronous commit production method, including:
basic verification is carried out when initial data is submitted to production, and a data flow table and a data state table are established after verification is passed;
the submitted initial data is written into a data flow table, and the data flow table is initialized, and the data state table records state nodes in the data production process;
acquiring a current state node of data production and a process for controlling the data production according to the data state table;
the data flow table records the production data and the production record of each node in the data production process.
As an optional embodiment of the present invention, in a data asynchronous submission production method of the present invention, the process of obtaining a current node of data production and controlling data production according to a data state table includes:
traversing the data to be processed to obtain a current state node in a data state table;
and judging the next production node for data production according to the current state node, pushing the data to be processed, which are the same as the next production node, into a processing queue corresponding to the processing node, and carrying out data production.
As an optional implementation manner of the invention, the data asynchronous submission production method of the invention comprises the following steps:
judging the production state of the data according to the state nodes in the data state table;
when the data is in an unprocessed, overtime processed or processing failure production state, the data is fished back.
As an optional implementation manner of the invention, in the data asynchronous submission production method, according to the data state table, if the initial data is judged to be in the initial state node for more than the first preset time, the initial data is in an unprocessed production state;
fishing back the unprocessed initial data, and automatically submitting again;
optionally, traversing the data to be processed, and when the unprocessed data judged according to the data state table exceeds a preset threshold, reducing or stopping submitting the production flow request of the initial data and reporting.
As an optional implementation manner of the invention, in the data asynchronous submission production method, according to the data state table, if the data production is judged to be in the intermediate production state node for more than the second preset time, the data production is in the production state of overtime processing, and the overtime processing production data is fished back;
and inquiring historical production data and historical production records of the data production process according to the data flow table, and analyzing the reason of the overtime of data production.
As an optional implementation manner of the invention, in the data asynchronous submission production method, when the current node of the data production is judged to be in a failure state according to the data state table, the data production is in a production state of failure processing, and the production data of the failure processing is fished back;
and inquiring historical production data and historical production records of the data production process according to the data flow table to analyze reasons of data production failure.
As an optional implementation manner of the invention, the data asynchronous submission production method of the invention comprises the following steps:
subscribing and informing the production result of the initial data submission production through a message queue;
when the data production fails, feeding back a production state node and a failure reason of the data production failure;
the data flow table supports the query acquisition of historical production data and historical production records.
In a second aspect, the present invention provides a data asynchronous commit production device, comprising:
the data submitting module performs basic verification during initial data submitting production;
the data flow table module establishes a data flow table after verification is passed, and the submitted initial data is written into the data flow table and the data flow table is initialized;
the data state table module is used for checking and establishing a data state table through post-synchronization, and the data state table records state nodes in the data production process;
the scheduling module acquires current state nodes of data production and processes for controlling the data production according to a data state table, and the data flow table records production data and production records of all nodes in the data production process.
In a third aspect, the invention provides an electronic device comprising a processor and a memory for storing a computer executable program, which when executed by the processor performs the one data asynchronous commit production method.
In a fourth aspect, the present invention provides a computer-readable recording medium storing a computer-executable program which, when executed, implements the data asynchronous commit production method.
Compared with the prior art, the invention has the beneficial effects that:
according to the data asynchronous submitting production method, an asynchronous processing mode is adopted for submitting and producing initial data, the initial data is written into a data flow table when submitted, a data submitting result is fed back to a data submitting party, synchronous production is not needed after the initial data is submitted, and the production of the initial data is controlled by a scheduling module according to state nodes in a data state table; the invention adopts the data state table to record the state node of the data production, and the scheduling module schedules the data production to enter the next production state node according to the completion condition of the current production state node, so that the data production process has compact service logic, and the high efficiency of the data production is ensured.
Therefore, the data asynchronous submitting production method realizes the decoupling of the upstream submitting data and the intermediate data production service, the upstream simply submitting data does not need to pay attention to the data production and warehousing results, the intermediate data production service independently carries out the production processing of the data, and the data production process has compact service logic by combining a data flow table and a data state table, so that the data production is efficient and stable.
Drawings
FIG. 1 is a flow chart of a data asynchronous commit production method in accordance with an embodiment of the present invention;
FIG. 2 is a schematic diagram of an electronic device according to an embodiment of the present invention;
fig. 3 is a schematic diagram of a computer-readable recording medium of an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It will be apparent that the described embodiments are some, but not all, embodiments of the invention.
Thus, the following detailed description of the embodiments of the invention is not intended to limit the scope of the invention, as claimed, but is merely representative of some embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
It should be noted that, under the condition of no conflict, the embodiments of the present invention and the features and technical solutions in the embodiments may be combined with each other.
It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures.
In the description of the present invention, it should be noted that, the terms "upper", "lower", and the like indicate an azimuth or a positional relationship based on the azimuth or the positional relationship shown in the drawings, or an azimuth or a positional relationship conventionally put in use of the inventive product, or an azimuth or a positional relationship conventionally understood by those skilled in the art, such terms are merely for convenience of describing the present invention and simplifying the description, and do not indicate or imply that the apparatus or element to be referred must have a specific azimuth, be constructed and operated in a specific azimuth, and thus should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and the like, are used merely to distinguish between descriptions and should not be construed as indicating or implying relative importance.
Example 1
Referring to fig. 1, a data asynchronous commit production method of the present embodiment includes:
basic verification is carried out when initial data is submitted to production, and a data flow table and a data state table are established after verification is passed;
the submitted initial data is written into a data flow table, and the data flow table is initialized, and the data state table records state nodes in the data production process;
acquiring a current state node of data production and a process for controlling the data production according to the data state table;
the data flow table records the production data and the production record of each node in the data production process.
According to the data asynchronous submitting production method, an asynchronous processing mode is adopted for submitting and producing initial data, the initial data is written into a data flow table when submitted, a data submitting result is fed back to a data submitting party, synchronous production is not needed after the initial data is submitted, and the production of the initial data is controlled by a scheduling module according to state nodes in a data state table; in the embodiment, the data state table is adopted to record the state node of the data production, and the scheduling module schedules the data production to enter the next production state node according to the completion condition of the current production state node, so that the data production process has compact service logic, and the high efficiency of the data production is ensured.
Therefore, the data asynchronous submitting production method of the embodiment realizes decoupling of the upstream submitting data and the intermediate data production service, the upstream simply submitting data does not need to pay attention to data production and warehousing results, the intermediate data production service independently carries out data production processing, and the data production process has compact service logic by combining a data flow table and a data state table, so that the data production process is efficient and stable.
As an optional implementation manner of the present embodiment, in a data asynchronous submission production method of the present embodiment, the process of obtaining a current node of data production and controlling data production according to a data state table includes:
traversing the data to be processed to obtain a current state node in a data state table;
and judging the next production node for data production according to the current state node, pushing the data to be processed, which are the same as the next production node, into a processing queue corresponding to the processing node, and carrying out data production.
According to the data asynchronous submitting production method, each data production node adopts a queue mode, so that data is scheduled to be queued in the corresponding queue according to the completion condition of the current state node in the data state table, the production of the data is performed in a synchronous and orderly mode according to the production nodes, the compact service logic is provided, and the production efficiency of the data is improved.
As an optional implementation manner of the embodiment, a data asynchronous submission production method of the embodiment includes:
judging the production state of the data according to the state nodes in the data state table;
when the data is in an unprocessed, overtime processed or processing failure production state, the data is fished back.
Therefore, according to the data asynchronous submission production method, unprocessed, overtime processed or processing failure data are fished back in time, so that the problem that the abnormal data occupy resources for a long time, so that data production is blocked and data production efficiency is affected is avoided.
Specifically, the data asynchronous submission production method of the embodiment is as follows for an unprocessed data judgment party: judging that the initial data is in an initial state node for more than a first preset time according to the data state table, and then the initial data is in an unprocessed production state; and (5) carrying out drag-back processing on the unprocessed initial data, and automatically submitting again.
For unprocessed data, the reason for the unprocessed initial data may be that the current load of the data production service is large because the unprocessed data has not entered into the production process, and therefore, the data should be automatically tried to be submitted again after being fished back, so as to see whether the data production service can be entered.
Optionally, a data asynchronous submission production method of the embodiment includes: traversing the data to be processed, and when the unprocessed data judged according to the data state table exceeds a preset threshold value, reducing or stopping submitting the production flow request of the initial data and reporting. In this embodiment, the unprocessed data is monitored, and when the unprocessed data exceeds a preset threshold, it is indicated that the current data production service has an excessive load and insufficient processing capacity, or the data production service has a fault, so that the initial data is reduced or stopped to submit the production flow request, thereby avoiding further increasing the pressure of the data production service, and reporting the data so as to perform fault investigation in time.
As an optional implementation manner of the embodiment, a judging method of the data asynchronous submission production method for timeout processing data in the embodiment is as follows: judging that the data production is in the production state of overtime processing if the node in the intermediate production state exceeds a second preset time according to the data state table; and carrying out a bailing process on the production data processed in the time-out process. For data processed by overtime, there may be a problem that data production cannot be completed, and therefore, in order to avoid that overtime data occupies production resources for a long time, the overtime data is fished back.
Meanwhile, according to the data asynchronous submitting production method, historical production data and historical production records of the data production process are inquired according to the data flow table, and the reason of overtime data production is analyzed. Because the data flow table is written with the data and the production record produced by each production node in the data production process, the data flow table of the overtime data is consulted to help analyze the reason of the overtime data production so as to provide a targeted solution.
The method for judging production failure data by using the data asynchronous submission production method in the embodiment comprises the following steps: when the current node of the data production is judged to be in a failure state according to the data state table, the data production is in a production state of failure processing, and the production data of the failure processing is fished back. By monitoring the data state table, the embodiment can timely discover and drag back the data in the production failure state node.
Meanwhile, according to the data asynchronous submitting production method, historical production data and historical production records of the data production process are inquired according to the data flow table, and reasons of data production failure are analyzed.
As an optional implementation manner of the embodiment, a data asynchronous submission production method of the embodiment includes:
subscribing and informing the production result of the initial data submission production through a message queue;
when the data production fails, feeding back a production state node and a failure reason of the data production failure;
the data flow table supports the query acquisition of historical production data and historical production records.
Because the data asynchronous submitting production method is provided by the patent application, the submitting production result of the data cannot be obtained immediately after the upstream data submitting party submits the data, if the upstream data submitting party has the requirement of obtaining the submitting production result of the data, the production result of the submitted initial data can be obtained through subscribing through a message queue; meanwhile, for the data with the data production failure, the nodes with the data production failure can be fed back so as to analyze the reasons of the data production failure in a targeted manner.
As a specific application example of the data asynchronous submission production method in this embodiment, the submission production of the test questions in the test question library specifically includes:
upstream submitting initial test question resources needing to be put in storage;
the basic verification is carried out on the initial test question resource, and specifically comprises whether the content of the test question is complete (for example, whether the test question has a complete question stem part, an answer part and an analysis part) and whether the test question information is complete (for example, whether the test question information comprises subject information and grade information);
for the initial test question resources which pass the verification, a data flow table and a data state table are established, the submitted initial test question resources are written into the data flow table, the data flow table is initialized, and the data state table records state nodes of the initial test question resource production process;
the main production nodes for producing the test questions comprise OCR (optical character recognition) nodes, information extraction nodes and normalization processing nodes;
the scheduling module acquires a current state node of test question production and a process for controlling the test question production according to a data state table of the initial test question resource;
in the test question production process, the data flow table records production data and production records of all nodes in the data production process.
For the test questions with the production completion, the test questions are directly written into the test question database, and for the test questions with the production failure, the data flow table of the test questions can be queried, and the reasons of the test questions with the production failure are analyzed, so that the targeted modification of the test questions is conveniently carried out, and the probability of the test questions submitted to the production success is improved.
The embodiment also provides a data asynchronous submitting production device, which comprises:
the data submitting module performs basic verification during initial data submitting production;
the data flow table module establishes a data flow table after verification is passed, and the submitted initial data is written into the data flow table and the data flow table is initialized;
the data state table module is used for checking and establishing a data state table through post-synchronization, and the data state table records state nodes in the data production process;
the scheduling module acquires current state nodes of data production and processes for controlling the data production according to a data state table, and the data flow table records production data and production records of all nodes in the data production process.
According to the data asynchronous submitting production device, an asynchronous processing mode is adopted for submitting and producing initial data, a data submitting module performs basic verification when the initial data is submitted, and a data submitting result is fed back; the data flow table module writes the initial data into the data flow table after the verification is passed, synchronous production of the initial data is not needed after the initial data is submitted, and the production of the initial data is controlled by the scheduling module according to the state nodes in the data state table of the data state table module; the data state table module of the embodiment adopts the data state table to record the state nodes of data production, and the scheduling module schedules the data production to enter the next production state node according to the completion condition of the current production state node, so that the data production process has compact service logic, and the high efficiency of the data production is ensured.
Therefore, the data asynchronous submitting production device realizes decoupling of the upstream submitting data and the intermediate data production service, the upstream simply submitting data does not need to pay attention to data production and warehousing results, the intermediate data production service independently carries out data production processing, and the data production process has compact service logic by combining a data flow table and a data state table, so that the data production process is efficient and stable.
As an optional implementation manner of this embodiment, the obtaining, by the scheduling module of this embodiment, a current node for data production and a process for controlling data production according to the data state table includes:
the scheduling module traverses the data to be processed to acquire a current state node in a data state table;
the scheduling module judges the next production node for data production according to the current state node, pushes the data to be processed, which is the same as the next production node, into a processing queue corresponding to the processing node, and performs data production.
According to the data asynchronous submitting production device, each data production node adopts a queue mode, so that the scheduling module schedules data to the corresponding queue for queuing according to the completion condition of the current state node in the data state table, the production of the data is performed in a synchronous and orderly mode according to the production nodes, the data production device has compact service logic, and the production efficiency of the data is improved.
As an optional implementation manner of the embodiment, a data asynchronous submitting production device of the embodiment comprises a data fishing back module:
the data retrieval module judges the production state of the data according to the state nodes in the data state table;
when the data is in an unprocessed, overtime processed or processing failure production state, the data fishing back module carries out data fishing back processing.
Therefore, according to the data asynchronous submitting production device, the data retrieval module retrieves unprocessed, overtime processed or failed processed data in time, so that the problem that the abnormal data occupy resources for a long time to cause data production blockage and influence the data production efficiency is avoided.
Specifically, the data fetch module in this embodiment determines, for the unprocessed data, that: judging that the initial data is in an initial state node for more than a first preset time according to the data state table, and then the initial data is in an unprocessed production state; and (5) carrying out drag-back processing on the unprocessed initial data, and automatically submitting again.
For unprocessed data, the reason for unprocessed initial data may be that the current load of the data production service is large because the unprocessed data has not entered the production process, so the data retrieval module should automatically attempt to re-submit after retrieving the unprocessed data, so as to see if the data production service can be entered.
Optionally, the data retrieval module of this embodiment traverses the data to be processed, and reduces or stops submitting the production flow request of the initial data and reports the production flow request when the unprocessed data determined according to the data state table exceeds a preset threshold. The data retrieval module in this embodiment monitors unprocessed data, and when the unprocessed data exceeds a preset threshold, it indicates that the current data production service load is too large, the processing capacity is insufficient, or the data production service has a fault, so the data retrieval module adopts to reduce or stop the submitting of the initial data to the production flow request, thereby avoiding further increasing the pressure of the data production service. And meanwhile, the reporting module reports so as to conduct fault investigation in time.
As an optional implementation manner of this embodiment, the method for determining the timeout processing data by the data fishing module of this embodiment is: judging that the data production is in the production state of overtime processing if the node in the intermediate production state exceeds a second preset time according to the data state table; and carrying out a bailing process on the production data processed in the time-out process. For data processed by overtime, there may be a problem that data production cannot be completed, and therefore, in order to avoid that overtime data occupies production resources for a long time, the overtime data is fished back.
Meanwhile, the data retrieval module of the embodiment queries historical production data and historical production record analysis data production overtime reasons of the data production process according to the data flow table. Because the data flow table is written with the data and the production record produced by each production node in the data production process, the data retrieval module is beneficial to analyzing the reason of the overtime data production by referring to the data flow table of the overtime data so as to provide a targeted solution.
The method for judging production failure data by the data retrieval module in this embodiment is as follows: when the current node of the data production is judged to be in a failure state according to the data state table, the data production is in a production state of failure processing, and the production data of the failure processing is fished back. By monitoring the data state table, the embodiment can timely discover and drag back the data in the production failure state node.
Meanwhile, the data retrieval module of the embodiment queries historical production data and historical production records of the data production process according to the data flow table to analyze reasons of data production failure.
As an optional implementation manner of this embodiment, a data asynchronous submission production apparatus of this embodiment includes a message queue subscription module:
the message queue subscription module notifies the production result of initial data submission production through message queue subscription;
when the data production fails, feeding back a production state node and a failure reason of the data production failure;
the data flow table supports the query acquisition of historical production data and historical production records.
Because the data asynchronous submitting production device provided by the patent application adopts the data asynchronous submitting production method, an upstream data submitting party cannot acquire the submitting production result of the data immediately after submitting the data, and therefore, if the upstream data submitting party has the requirement of acquiring the submitting production result of the data, the production result of the submitted initial data can be acquired through subscribing through a message queue; meanwhile, for the data with the data production failure, the nodes with the data production failure can be fed back so as to analyze the reasons of the data production failure in a targeted manner.
As a specific implementation example of the data asynchronous submission production device, which applies the embodiment, the submission production of the test questions in the test question library specifically includes:
upstream submitting initial test question resources needing to be put in storage;
the data submitting module performs basic verification on the initial test question resource, and specifically comprises whether the content of the test question is complete (for example, whether the test question has a complete question stem part, an answer part and an analysis part) and whether the test question information is complete (for example, whether the test question information comprises subject information and grade information);
the data flow table module establishes a data flow table for the initial test question resources which pass the verification, the data state table module establishes a data state table for the initial test question resources which pass the verification, the submitted initial test question resources are written into the data flow table, the data flow table is initialized, and the data state table records state nodes of the initial test question resource production process;
the data production module is used for producing the main production nodes of the test questions, wherein the main production nodes comprise an OCR (optical character recognition) node, an information extraction node and a normalization processing node;
the scheduling module acquires a current state node of test question production and a process for controlling the test question production according to a data state table of the initial test question resource;
in the test question production process, the data flow table records production data and production records of all nodes in the data production process.
For the test questions with the production completion, the test questions are directly written into the test question database, and for the test questions with the production failure, the data flow table of the test questions can be queried, and the reasons of the test questions with the production failure are analyzed, so that the targeted modification of the test questions is conveniently carried out, and the probability of the test questions submitted to the production success is improved.
Example two
The following describes an embodiment of an electronic device according to the present invention, which may be regarded as a specific physical implementation of the above-described embodiment of the method and apparatus according to the present invention. Details described in relation to the embodiments of the electronic device of the present invention should be considered as additions to the embodiments of the method or apparatus described above; for details not disclosed in the embodiments of the electronic device of the present invention, reference may be made to the above-described method or apparatus embodiments.
Fig. 2 is a schematic structural diagram of an electronic device according to an embodiment of the present invention, the electronic device including a processor and a memory for storing a computer-executable program, the processor executing the data acquisition method of embodiment one when the computer program is executed by the processor.
As shown in fig. 2, the electronic device is in the form of a general purpose computing device. The processor may be one or a plurality of processors and work cooperatively. The invention does not exclude that the distributed processing is performed, i.e. the processor may be distributed among different physical devices. The electronic device of the present invention is not limited to a single entity, but may be a sum of a plurality of entity devices.
The memory stores a computer executable program, typically machine readable code. The computer readable program may be executable by the processor to enable an electronic device to perform the method, or at least some of the steps of the method, of the present invention.
The memory includes volatile memory, such as Random Access Memory (RAM) and/or cache memory, and may be non-volatile memory, such as Read Only Memory (ROM).
Optionally, in this embodiment, the electronic device further includes an I/O interface, which is used for exchanging data between the electronic device and an external device. The I/O interface may be a bus representing one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.
It should be understood that the electronic device shown in fig. 2 is only one example of the present invention, and the electronic device of the present invention may further include elements or components not shown in the above examples. For example, some electronic devices further include a display unit such as a display screen, and some electronic devices further include a man-machine interaction element such as a button, a keyboard, and the like. The electronic device may be considered as covered by the invention as long as the electronic device is capable of executing a computer readable program in a memory for carrying out the method or at least part of the steps of the method.
Fig. 3 is a schematic diagram of a computer-readable recording medium of an embodiment of the present invention. As shown in fig. 3, a computer-readable recording medium stores a computer-executable program that, when executed, implements a data acquisition method according to the first embodiment of the present invention. The computer readable recording medium may include a data signal propagated in baseband or as part of a carrier wave, with readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. The readable recording medium can also be any readable medium that can communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The program code embodied on the readable recording medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).
From the above description of embodiments, those skilled in the art will readily appreciate that the present invention may be implemented by hardware capable of executing a specific computer program, such as the system of the present invention, as well as electronic processing units, servers, clients, handsets, control units, processors, etc. included in the system. The invention may also be implemented by computer software executing the method of the invention, e.g. by control software executed by a microprocessor, an electronic control unit, a client, a server, etc. It should be noted, however, that the computer software for performing the method of the present invention is not limited to being executed by one or a specific hardware entity, but may also be implemented in a distributed manner by unspecified specific hardware. For computer software, the software product may be stored on a computer readable recording medium (which may be a CD-ROM, a usb disk, a removable hard disk, etc.), or may be stored distributed over a network, as long as it enables the electronic device to perform the method according to the invention.
The above embodiments are only for illustrating the present invention and not for limiting the technical solutions described in the present invention, and although the present invention has been described in detail in the present specification with reference to the above embodiments, the present invention is not limited to the above specific embodiments, and thus any modifications or equivalent substitutions are made to the present invention; all technical solutions and modifications thereof that do not depart from the spirit and scope of the invention are intended to be included in the scope of the appended claims.
Claims (10)
1. A method for asynchronously submitting and producing data, comprising the steps of:
basic verification is carried out when initial data is submitted to production, and a data flow table and a data state table are established after verification is passed;
the submitted initial data is written into a data flow table, and the data flow table is initialized, and the data state table records state nodes in the data production process;
acquiring a current state node of data production and a process for controlling the data production according to the data state table;
the data flow table records the production data and the production record of each node in the data production process.
2. The method of claim 1, wherein the step of obtaining the current node for data production and controlling the process of data production according to the data state table comprises:
traversing the data to be processed to obtain a current state node in a data state table;
and judging the next production node for data production according to the current state node, pushing the data to be processed, which are the same as the next production node, into a processing queue corresponding to the processing node, and carrying out data production.
3. The method for asynchronously submitting and producing data according to claim 1, comprising:
judging the production state of the data according to the state nodes in the data state table;
when the data is in an unprocessed, overtime processed or processing failure production state, the data is fished back.
4. A data asynchronous submission production method according to claim 3, wherein if the initial data is in the initial state node for more than a first preset time according to the data state table, the initial data is in an unprocessed production state;
fishing back the unprocessed initial data, and automatically submitting again;
optionally, traversing the data to be processed, and when the unprocessed data judged according to the data state table exceeds a preset threshold, reducing or stopping submitting the production flow request of the initial data and reporting.
5. The method for asynchronously submitting data to production according to claim 3, wherein if the data production is judged to be in the intermediate production state node for more than the second preset time according to the data state table, the data production is in the production state of overtime processing, and the overtime processing production data is fished back;
and inquiring historical production data and historical production records of the data production process according to the data flow table, and analyzing the reason of the overtime of data production.
6. The method for asynchronously submitting and producing data according to claim 3, wherein when the current node of the data production is judged to be in a failure state according to the data state table, the data production is in a production state of failure processing, and the production data of the failure processing is fished back;
and inquiring historical production data and historical production records of the data production process according to the data flow table to analyze reasons of data production failure.
7. A method of asynchronous commit production of data according to any one of claims 3 to 6, comprising:
subscribing and informing the production result of the initial data submission production through a message queue;
when the data production fails, feeding back a production state node and a failure reason of the data production failure;
the data flow table supports the query acquisition of historical production data and historical production records.
8. A data asynchronous submission production device, comprising:
the data submitting module performs basic verification during initial data submitting production;
the data flow table module establishes a data flow table after verification is passed, and the submitted initial data is written into the data flow table and the data flow table is initialized;
the data state table module is used for checking and establishing a data state table through post-synchronization, and the data state table records state nodes in the data production process;
the scheduling module acquires current state nodes of data production and processes for controlling the data production according to a data state table, and the data flow table records production data and production records of all nodes in the data production process.
9. Electronic device comprising a processor and a memory for storing a computer executable program, characterized in that the processor performs a data asynchronous commit production method according to any of claims 1-7 when the computer program is executed by the processor.
10. A computer-readable recording medium, characterized in that a computer-executable program is stored, which, when executed, implements a data asynchronous commit production method as claimed in any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311786331.8A CN117708214A (en) | 2023-12-22 | 2023-12-22 | Data asynchronous submission production method and device, electronic equipment and recording medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311786331.8A CN117708214A (en) | 2023-12-22 | 2023-12-22 | Data asynchronous submission production method and device, electronic equipment and recording medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117708214A true CN117708214A (en) | 2024-03-15 |
Family
ID=90153119
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311786331.8A Pending CN117708214A (en) | 2023-12-22 | 2023-12-22 | Data asynchronous submission production method and device, electronic equipment and recording medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117708214A (en) |
-
2023
- 2023-12-22 CN CN202311786331.8A patent/CN117708214A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107506451B (en) | Abnormal information monitoring method and device for data interaction | |
| US10216821B2 (en) | Methods and systems for bulk uploading of data in an on-demand service environment | |
| CN110825767A (en) | Data query method, platform, equipment and storage medium | |
| CN111400011B (en) | Real-time task scheduling method, system, equipment and readable storage medium | |
| CN111045911B (en) | Performance test method, performance test device, storage medium and electronic equipment | |
| US20170005904A1 (en) | System and method for monitoring performance of applications for an entity | |
| CN110569214A (en) | Index construction method and device for log file and electronic equipment | |
| EP3937022B1 (en) | Method and apparatus of monitoring interface performance of distributed application, device and storage medium | |
| CN110147470B (en) | Cross-machine-room data comparison system and method | |
| US11775504B2 (en) | Computer estimations based on statistical tree structures | |
| CN111782452A (en) | Method, system, device and medium for interface contrast test | |
| CN113076729A (en) | Method and system for importing report, readable storage medium and electronic equipment | |
| CN112052082A (en) | Task attribute optimization method, device, server and storage medium | |
| US11099960B2 (en) | Dynamically adjusting statistics collection time in a database management system | |
| CN113238815B (en) | Interface access control method, device, equipment and storage medium | |
| CN114090529A (en) | Log management method, device, system and storage medium | |
| CN116842090A (en) | Accounting system, method, equipment and storage medium | |
| CN110033242B (en) | Working time determining method, device, equipment and medium | |
| CN117708214A (en) | Data asynchronous submission production method and device, electronic equipment and recording medium | |
| CN111274215A (en) | Distributed log unified management method | |
| US8024301B2 (en) | Automatic database diagnostic usage models | |
| CN113468218A (en) | Method and device for monitoring and managing database slow SQL | |
| CN113779117A (en) | Data monitoring method and device, storage medium and electronic equipment | |
| Koschel et al. | Evaluating time series database management systems for insurance company | |
| CN115334153B (en) | Data processing method and device for service grid |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication |